Method and system for automated registration and login of business carriers and acquisition of funds accounts

Abstract

The present invention provides a method and system for automated registration and login of a business carrier and acquisition of a fund account, and relates to the field of data processing technology. The method comprises: for an element located within a frame in an interactive task queue, converting its coordinates into coordinates of a relative frame, and controlling the browser to switch to the corresponding frame; traversing the interactive task queue, and sequentially performing the following operations: performing a simulated click operation on a button-type element; injecting virtual identity information into an input box-type element; calling a corresponding recognition model to complete verification of a verification code-type element; page scrolling and submission detection, after completing the interactive task queue operation, detecting whether there is a visible submit button on the current page, if not; after submission, if jumping to a recharge page, extracting payment account information through a regular expression, and storing it in a database in association with the virtual identity information. The present invention can improve the robustness of automated operations under dynamic pages.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and system for automated registration and login of business carriers and acquisition of funds accounts. Background Art

[0002] In the field of automated web interaction technology, existing solutions still have room for improvement when handling dynamic page environments and complex interaction scenarios, mainly in the following aspects:

[0003] Due to the limited adaptability of page structure, current mainstream automation tools (such as script engines based on DOM parsing) usually rely on static element location paths (such as XPath / CSS selectors). When the target page appears:

[0004] Common changes such as interface layout version iteration, dynamic generation of element identifiers, or adaptive adjustment of interactive component positions can easily lead to positioning failure and process interruption.

[0005] For the use of frame nesting technology (such as <iframe>), the existing solutions do not have a coordinate system conversion mechanism, which makes it difficult to accurately calculate the position of elements within the sub-frame, lack automatic context switching functions, or have compatibility issues in cross-frame operations. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method for automatically registering and logging in to a business carrier and obtaining a fund account, which can improve the robustness of automated operations under dynamic pages. In order to solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0008] A method for automatically registering and logging in to a business carrier and obtaining a fund account, the method comprising:

[0009] Step S1: Access the target website domain name through a headless browser, take a screenshot of the current visible area after loading, and generate a page image;

[0010] Step S2: Input the page image into a pre-trained object detection model to identify the category, location coordinates and confidence level of UI elements in the page; sort the recognition results based on a preset priority rule to generate an interactive task queue arranged in descending order of priority;

[0011] Step S3, for the elements in the interactive task queue that are located in the frame, convert their coordinates to the coordinates of the relative frame, and control the browser to switch to the corresponding frame;

[0012] Step S4, traverse the interactive task queue and perform the following operations in sequence: perform a simulated click operation on the button type element; inject virtual identity information into the input box type element; call the corresponding recognition model to complete the verification of the verification code type element;

[0013] Step S5, page scrolling and submission detection, after completing the interactive task queue operation, detect whether there is a visible submit button on the current page. If not, scroll the page and re-execute steps S1 to S2 until a visible submit button is identified. If visible, trigger the submit operation;

[0014] Step S6: After submission, if the page jumps to the recharge page, the payment account information is extracted through a regular expression and associated with the virtual identity information and stored in the database.A business carrier automated registration and login and fund account acquisition system, comprising:

[0016] An acquisition module is used to access the target website domain name through a headless browser, take a screenshot of the current visible area after loading, and generate a page image;

[0017] A generation module is used to input the page image into a pre-trained object detection model to identify the category, location coordinates and confidence level of UI elements in the page; sort the recognition results based on preset priority rules and generate an interactive task queue arranged in descending order of priority;

[0018] A conversion module, for converting the coordinates of elements in the interactive task queue that are located within a frame into coordinates of a relative frame, and controlling the browser to switch to the corresponding frame;

[0019] A verification module is used to traverse the interactive task queue and perform the following operations in sequence: perform a simulated click operation on the button type element; inject virtual identity information into the input box type element; call the corresponding recognition model to complete the verification of the verification code type element;

[0020] A processing module is used for page scrolling and submission detection. After completing the interactive task queue operation, it detects whether there is a visible submit button on the current page until a visible submit button is identified. If it is visible, the submission operation is triggered; if it jumps to the recharge page after submission, the payment account information is extracted through a regular expression and associated with the virtual identity information and stored in the database. The above-mentioned scheme of the present invention at least includes the following beneficial effects:

[0022] The visually driven UI recognition mechanism is adopted, and the page image is directly parsed through the target detection model, eliminating the dependence on the DOM structure. It can effectively deal with random changes in page layout, dynamic generation of element identifiers and adaptive adjustment of component positions, and can solve the process interruption problem caused by path failure in traditional scripts.

[0023] A dynamic conversion chain of coordinate systems is established, which can automatically detect the nested structure of frames, calculate relative coordinates in real time, and intelligently switch browser contexts, thus breaking through the error rate of cross-frame element operations.

[0024] Building a classification recognition-dynamic execution system can improve the average recognition rate of complex verification codes; by designing a scrolling-redetection closed-loop mechanism, scrolling can be triggered based on the state of the visible area, and the screenshot and element recognition will be automatically re-taken after scrolling. A locking rule for the operated elements can be established to ensure that the bottom operation component is 100% triggerable and avoid page state conflicts.

[0025] Develop a hierarchical parsing - standardized storage process, adapt regular expressions to fragmented texts, automatically associate virtual identities with payment accounts, and perform background parsing of unstructured data (such as QR codes), which can improve the extraction integrity rate of key account information.BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a schematic flowchart of a method for automatically registering and logging in a service carrier and obtaining a fund account provided by an embodiment of the present invention.

[0027] Figure 2 is a schematic diagram of a system for automatically registering and logging in a service carrier and obtaining a fund account provided by an embodiment of the present invention.DETAILED DESCRIPTION

[0028] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art.

[0029] As shown in Figure 1, an embodiment of the present invention proposes a method for automatic registration and login of service carriers and obtaining a fund account. The method includes the following steps:

[0030] Step S1, access the target website domain name through a headless browser, take a screenshot of the current visible area after loading is completed, and generate a page image;

[0031] Step S2, input the page image into a pre-trained object detection model to identify the UI element categories, position coordinates, and confidence levels in the page; sort the recognition results based on a preset priority rule, and generate an interaction task queue arranged in descending order of priority;

[0032] Step S3, for the elements located within the frame in the interaction task queue, convert their coordinates to coordinates relative to the frame, and control the browser to switch to the corresponding frame;

[0033] Step S4, traverse the interaction task queue, and perform the following operations in sequence: perform a simulated click operation on button elements; inject virtual identity information into input box elements; call the corresponding recognition model to complete verification for verification code elements;

[0034] Step S5, page scrolling and submission detection. After completing the operations in the interaction task queue, detect whether there is a visible submission button on the current page. If it is not visible, scroll the page and re-execute steps S1 to S2 until a visible submission button is recognized. If it is visible, trigger the submission operation;

[0035] Step S6, if it jumps to the recharge page after submission, extract the payment account information through a regular expression, and associate and store it with the virtual identity information in the database.

[0036] In the embodiment of the present invention, the full process is automatically executed by simulating manual operations through a headless browser, and the page elements are recognized in combination with an object detection model, realizing the full process automation from accessing the website, filling in information to submitting the form, reducing the manual intervention cost, and is especially suitable for scenarios such as batch registration and multi-account management. The intelligent sorting of task priorities sorts the UI elements according to the priority based on a preset rule (such as required items first), ensuring that the interaction tasks are executed in a logical order, avoiding operation errors caused by the disordered element loading order, and improving the process stability.Element coordinate conversion within a frame, for nested frames (such as iframes), converts element coordinates to frame-relative coordinates and switches context, resolving technical difficulties in cross-frame interactions and ensuring precise targeting of the target element. It automatically triggers clicks on button-like elements and injects virtual identity information into input fields, achieving standardized data entry. For CAPTCHAs, it leverages specialized recognition models (such as OCR and machine learning models), overcoming the limitations of traditional automated tools in CAPTCHA scenarios and improving the processing capabilities of complex pages. Dynamic page scroll detection automatically checks submit button visibility, refreshing the visible area and re-identifying elements by scrolling the page. This adapts to scenarios with long pages or dynamically loaded content, avoiding workflow interruptions caused by hidden elements. For multi-page redirects and data associations, if the page is redirected to a top-up page after submission, regular expressions are used to extract payment account information and bind it to the virtual identity for storage, enabling automated cross-page data collection and structured storage. Model-driven, scalable object detection and CAPTCHA recognition models can be continuously optimized through training to adapt to UI changes across different websites, reducing the risk of tool failure due to website redesigns and improving cross-platform adaptability. Data security and compliance are simulated by injecting virtual identity information (rather than real data), reducing the risk of real data leakage, while supporting business needs such as automated testing and data collection in compliance scenarios. In a preferred embodiment of the present invention, step S1, accessing the target website domain name through a headless browser, taking a screenshot of the current visible area after loading is completed, and generating a page image, includes:

[0038] Step S11: After the page is loaded, detect whether there is a pop-up window element blocking the key area in the current visible area; if so, automatically calculate the coordinate position of the pop-up window close button and trigger a simulated click operation to expose the underlying page elements;

[0039] Step S12: Based on the page status of the cleared pop-up window, identify the location of the registration button in the page; calculate the coordinates of the center point of the registration button, and execute the coordinate positioning click through the browser driver to trigger the page to jump to the registration page;

[0040] Step S13: After the registration page is loaded, the browser is controlled to perform a vertical scrolling operation: the initial scrolling position is the top of the page, the scrolling step size is set according to the screen height, and the browser scrolls down step by step until it reaches the bottom of the page; after scrolling to the bottom, a screenshot operation is performed to capture the complete image including all form input boxes and the submit button at the bottom of the page;

[0041] Step S14, after completing the registration form submission operation, monitor the new page URL that the browser jumps to. When the page URL contains characteristic keywords, it is determined to be a recharge page. After the page is loaded, directly capture the current visible area image.In an embodiment of the present invention , the above steps can be implemented by the following specific schemes when specifically applied, for example:

[0043] In the above step S11, the conventional position of the core interactive elements such as the registration button is preset (such as the lower middle part of the page), which is defined as the key area coordinate range (such as the page height 30%~70%, width 20%~80%); the coordinate range of the pop-up element in the screenshot is analyzed to determine whether it overlaps with the key area coordinates. If it overlaps, it is determined to be blocking the pop-up window; based on the coordinates (left, top, width, height) of the pop-up element, its upper right corner area (usually the close button position) is calculated, and the coordinates of the upper left corner of the pop-up window + (width-20 pixels, height-20 pixels) are taken as the coordinates of the center point of the close button. A click event is sent to the coordinates through the browser driver to close the pop-up window. The present invention automatically clears interfering pop-ups, ensures that key elements such as the underlying registration button are visible, avoids the time-consuming operation of manually closing the pop-up window, and improves the degree of process automation.

[0044] In the above step S12, the bounding box coordinates (left, top, width, and height) of the registration button are output by the target detection model. Based on the bounding box coordinates, the coordinates of the center point of the button are calculated as (left + width / 2, top + height / 2); the center point coordinates are mapped to the browser viewport coordinate system, and the mouse is driven to move to the coordinates and trigger a click event to trigger a page jump. The present invention accurately locates the registration entry, avoids manual errors caused by changes in the page layout, ensures that the process is executed according to the preset path, and improves the accuracy of interaction.

[0045] In step S13, the vertical height (H, in pixels) of the browser viewport is obtained, the total page height (which can be obtained through document.body.scrollHeight) is divided equally according to the viewport height, and the step size is set to H. That is, each time the page is scrolled, starting from the top of the page (scroll position 0), the page is scrolled down by H pixels each time until the scroll position reaches or exceeds the total page height, ensuring that all content is loaded. After scrolling to the bottom of the page, the screenshot interface is called to capture the current complete page image, including all form input boxes and submit buttons. The present invention is suitable for long form pages or dynamically loaded content, ensuring that all input fields and submit buttons required for registration are captured, and avoiding information omissions caused by incomplete display of page elements.

[0046] In the above step S14, characteristic keywords (such as "recharge", "pay", "fund", etc.) of the recharge page are preset, and the path or parameter part of the new page URL is parsed to determine whether any keyword is contained. If the URL contains the characteristic keyword, it is determined to be a recharge page and a screenshot operation is triggered; otherwise, monitoring is continued and the image in the current browser viewport is directly captured for subsequent account information extraction. The present invention automatically identifies the jump stage of the registration process, accurately locates the recharge page, and timely captures the key information page, laying the foundation for the subsequent extraction of payment account information through regular expressions, thereby improving the efficiency of cross-page data collection. In a preferred embodiment of the present invention, step S2 inputs the page image into a pre-trained object detection model to identify the category, location coordinates, and confidence level of UI elements in the page; sorts the recognition results based on a preset priority rule to generate an interactive task queue arranged in descending order of priority, including:

[0048] Step S21: Input the page image generated in steps S13 and S14 into a pre-trained object detection model, and output a set of recognition results for each UI element; each recognition result includes a category label, location coordinates, and confidence level;

[0049] Step S22: Prioritize each recognition result outputted in step S21 based on the Euclidean distance between the center point of the element and the center of the page to obtain weights corresponding to the category label, location coordinates, and confidence level; obtain priority scores based on the weights corresponding to the category label, location coordinates, and confidence level, and divide the elements into five levels from high to low according to the scores to obtain a set of elements with priority labels;

[0050] Step S23, arranging the set of elements with priority labels output in step S22 in descending order of priority, and arranging elements of the same priority in descending order of confidence, to generate a structured task queue;

[0051] Step S24, traverse the structured task queue of step S23, calculate the normalized coordinates of each element, that is, calculate its absolute pixel coordinates in the browser window; and convert the absolute pixel coordinates into an XPath path through a DOM position inversion algorithm.In the embodiment of the present invention, when the above steps are applied in a specific manner , they can be implemented by the following specific schemes, for example:

[0053] In the above step S21, the complete page image or the visible area image generated in steps S13 and S14 is scaled to the model input size (e.g., 640×480 pixels), maintaining the pixel ratio to avoid distortion. A pre-trained object detection model (e.g., YOLO, Faster R-CNN) performs convolution calculations on the image to identify the bounding box coordinates (left, top, width, and height) of all visible UI elements (e.g., input boxes, buttons, and labels), and matches them with predefined category labels (e.g., "input," "button," and "captcha"). The model generates a confidence score (between 0 and 1) for each recognition result, indicating the probability that the element is correctly identified (e.g., a confidence score ≥ 0.8 is considered a reliable identification). The present invention automatically extracts key information of all UI elements on the page, replacing manual labeling and positioning, thereby improving element recognition efficiency; and outputs structured data containing coordinates, categories, and confidence scores, providing an accurate basis for subsequent task sorting and interactive operations.

[0054] In step S22, the page center coordinates are preset to (page width / 2, page height / 2); the Euclidean distance between the center point of each element's bounding box (left + width / 2, top + height / 2) and the page center is calculated, with the closer the distance, the higher the priority base value (e.g., elements with a distance ≤ 1 / 4 of the page height are considered "high priority areas"); element category priority weights are preset (e.g., input box weight +30, button weight +20, prompt text weight +10), with key operation elements (e.g., registration button, submit button) having higher weights; elements with a confidence level ≥ 0.9 are given an additional priority score (e.g., +10 points), and elements with a confidence level < 0.5 are marked as "pending" and have their score reduced (e.g., -20 points); the distance, category, and confidence scores are accumulated to obtain a total priority score, which is divided into five levels (level 1 is the highest and level 5 is the lowest), such as: Level 1 (score ≥ 80): Required input box and submit button; Level 3 (score 40-60): optional items, general prompt labels; Level 5 (score < 20): advertising elements, non-critical decorative patterns.

[0058] The present invention combines element position, functional importance and recognition reliability to ensure that core interactive elements (such as registration form input boxes) are processed first to prevent secondary elements from interfering with the process; it adapts to different page layouts (such as the difference between mobile and PC terminals), automatically adjusts element priorities, and improves process flexibility.In step S23 , the first-level sorting (in descending priority) is performed by sorting the elements from 1 to 5 according to the priority levels in step S22, ensuring that high-level elements (such as required items) are placed first. In the second-level sorting (in descending confidence), elements of the same level are arranged from high to low by confidence score (for example, if two input boxes are both level 2, the element with a confidence score of 0.9 is placed before the element with a confidence score of 0.8). The sorted elements are converted into JSON-formatted task items containing "operation type (click / input / verify), element coordinates, priority label, and confidence level," generating an executable queue structure (such as [Task 1, Task 2, ..., Task N]).

[0060] The present invention uses a dual sorting of "priority + confidence" to ensure that the automated process is executed according to the principle of "importance first, reliability first", reducing logical errors caused by the disorder of element order; the structured queue facilitates debugging and logging, and supports advanced functions such as mid-transmission pause and breakpoint resumption. In step S24 , the normalized coordinates are converted to absolute pixels: If the element coordinates are normalized values (between 0 and 1, e.g. , the center point coordinate x = 0.5 represents the horizontal centerline of the page), then multiply by the actual size of the browser window (e.g., if the window width is 1920 pixels, x = 0.5 × 1920 = 960 pixels) to obtain the absolute pixel coordinates (XY). The browser driver retrieves the page DOM tree, starting from the root node (html), and traverses the tree to find the element according to the following logic:

[0063] Based on the absolute pixel coordinates (XY), the elementFromPoint(XY) method is used to locate the topmost visible element. The parent node is recursively traced upwards, and the tag name and attributes (such as / / input[@id='username']) are concatenated to generate a unique XPath path. Redundant levels are removed (such as skipping the div container layer) to retain the shortest valid path (such as / / form[@class='register-form'] / input[1]).

[0064] The present invention solves the matching problem between "visually visible elements" and "underlying code elements" through bidirectional mapping between pixel coordinates and DOM structures, ensuring that simulated operations accurately act on target controls (such as avoiding clicking on hidden buttons that are blocked); the absolute pixel coordinate conversion mechanism adapts to different screen sizes (such as mobile phones, tablets, and desktops), and the XPath path supports dynamic page structure changes, reducing the risk of positioning failure caused by page revisions.In a preferred embodiment of the present invention, the pre -trained target detection model includes:

[0066] Extract target website domain names from the abused domain name database, access the domain names through automated scripts, and perform page loading status verification; for websites that pass verification, automatically trigger pop-up window closing, registration button clicks, and page scrolling operations, and capture full page screenshots containing key interactive elements; Based on the full page screenshots , a hierarchical clustering algorithm is used to group images with similar visual features to obtain duplicate-free images;

[0068] For the deduplicated image, the category labels and location information of 17 types of UI elements are annotated to generate an annotation file that conforms to the target detection format;

[0069] Integrate the annotation files and divide them into training set, verification set and test set according to the preset ratio; Based on the training set, validation set and test set, perform the following automatic optimization process: Initialize a population of parameters, where each set of parameters includes a learning rate, anchor box size, and network structure configuration; perform model training on each parameter combination, and calculate fitness using validation set accuracy and loss values; iteratively generate a new population through selection, crossover, and mutation operations until fitness converges to obtain optimized parameters; and use the optimized parameters to fine-tune the entire network and integrate incremental sample training to obtain a pre-trained object detection model. In an embodiment of the present invention , the above steps can be implemented by the following specific schemes when specifically applied, for example:

[0073] Active domain names (e.g., DNS resolution records within the past 30 days) are selected from the abused domain name database, and domain names marked as inaccessible are excluded. An automated script (e.g., Python's requests library) is used to send HTTP requests and detect the response status code (e.g., 200 indicates success). The response content is parsed to verify whether it contains a complete HTML structure (e.g., a complete HTML structure is included).<!DOCTYPEhtml> statement,<html> Domain names that time out (e.g., no response for more than 5 seconds) or return an error status code (e.g., 404, 500) are marked as invalid and removed from the candidate set. This method ensures that the page data input into model training are valid and accessible samples, avoiding wasting computing resources on invalid domain names. Through structured validation rules, abnormal pages are automatically filtered out, improving the reliability of data collection.

[0074] Preset common pop-up window close button features (such as the "×" icon in the upper right corner, with coordinates located in the area 10% to 20% from the top and 10% to 20% from the right of the page), locate and trigger clicks through image matching technology (such as template matching); locate the button based on page text features (such as "Register" or "SignUp") or visual features (such as orange buttons and larger fonts), calculate the center point coordinates and simulate clicks to trigger the registration process; gradually scroll the page (scrolling 1 / 2 of the window height each time) until the content at the bottom of the page is no longer updated (such as the scrollHeight remains unchanged after two consecutive scrolls), ensuring that the complete image including the footer submit button is captured. The present invention removes interfering elements (pop-up windows) and triggers key interactions (registration) by simulating manual operations to ensure that the screenshot contains the complete registration form structure; the scrolling loading mechanism can capture long pages or asynchronously loaded form elements to avoid data loss.

[0075] Visual features (such as color histogram, HOG histogram of gradients, and CSS style features) are extracted from each screenshot to generate a fixed-dimensional feature vector (e.g., 512 dimensions). Cosine similarity between images is calculated (similarity > 0.8 is considered similar) and a tree-like clustering structure is constructed. The central image of each cluster is selected as the representative sample, and other similar images within the cluster are removed, retaining approximately 30% to 50% of the original images to ensure that only one to two representative samples of the same page are retained. Duplicate or highly similar page images (e.g., registration pages with different domain names but the same template) are eliminated through clustering, reducing the amount of training data while maintaining feature diversity. Improve training efficiency: The amount of data after deduplication is reduced, which can shorten the model training time and avoid repeated learning of redundant information.

[0077] Use annotation tools (such as LabelMe and RectLabel) to manually annotate the deduplicated images with 17 categories of UI elements (such as input boxes, radio buttons, drop-down menus, and verification code boxes), draw bounding boxes, and associate category labels; convert the annotated data into the format required by the target detection model (such as YOLO's .txt format, where each line contains a category index and normalized coordinates). Data set division : The training set, validation set, and test set were randomly divided into a ratio of 7:2:1 . The distribution of elements of each category in each set was ensured to be balanced (e.g., the proportion of input boxes in the training set was consistent with that in the whole set). The manual annotation of the present invention ensured the element positioning accuracy (pixel-level bounding box) and category accuracy, providing a reliable supervision signal for the model. The staged data set supported performance verification during the model training process (parameter adjustment on the validation set) and the final generalization ability test (on the test set).Model parameter optimization process based on genetic algorithm : Parameter population initialization:

[0082] Generate 100 sets of initial parameter combinations, set the learning rate range to 0.001-0.1, and use the anchor box size preset according to the COCO dataset (e.g., (10, 13), (16, 30), etc.). Select YOLOv3-spp or FasterR-CNN variant as the network structure. Train the model for 50 rounds for each parameter set, and record the mean average precision (mAP) and loss value of the validation set. The fitness formula is: fitness = mAP - 0.5 × loss value, where a higher value indicates better performance. Genetic manipulation: Select and retain the parameter combinations with the top 20 % of fitness as the parent generation;

[0085] Crossover, where the parent parameters randomly exchange some dimensions (e.g., the learning rate and the anchor box size are crossed);

[0086] Mutation, adding random perturbations to the parameters after crossover (such as ±10% fluctuations in the learning rate). The iteration is terminated when the fitness improvement is less than 1% for 5 consecutive generations , and the optimal parameter combination is selected.

[0088] Model fine-tuning: Use the optimized parameters to train the model and gradually add newly collected incremental samples (such as 1,000 new screenshots per week) for online learning.

[0089] The present invention can replace manual trial-and-error parameter adjustment, quickly search for the optimal parameter combination through a bionic algorithm, and improve model training efficiency (parameter adjustment time is shortened by more than 70%); the incremental sample training mechanism enables the model to continuously learn new page design patterns (such as new verification code styles), delaying the model failure cycle caused by website revisions.In a preferred embodiment of the present invention, fitness is calculated using verification set accuracy and loss value, including: Based on the model training output of the current parameter combination, load its prediction results on the verification set;

[0092] Count the matching between the predicted box and the real box, that is, for each image in the validation set, calculate the overlap between the target detection model's predicted box and the annotated box; The ratio of the number of correctly detected UI elements to the total number of corresponding category annotations was counted for each of the 17 categories, and the arithmetic average of the accuracy values of all categories was taken to obtain the average accuracy index of the validation set.

[0094] Extract the CIoU loss values of all samples from the object detection model verification log, and take the arithmetic average of the CIoU loss values of all samples to obtain the mean CIoU loss;

[0095] Define the accuracy weight coefficient and the loss weight coefficient, weight the average accuracy index according to the positive correlation, and weight the CIoU loss mean according to the negative correlation to generate the fitness score. In an embodiment of the present invention , the above steps can be implemented by the following specific schemes when applied, for example:

[0097] Using the object detection model trained with the current parameter combination, forward propagation is performed on each image in the validation set, outputting the coordinates of the prediction box (x1, y1, x2, y2), the category label, and the confidence score. The prediction results are stored in a structured format (such as JSON) by image ID, with each prediction box containing [category, confidence, coordinates] information. The actual annotation box information of the corresponding image is read from the validation set annotation file to ensure a one-to-one correspondence between the prediction result and the actual label by image ID. The present invention stores the prediction and actual results in a unified format, providing a basis for subsequent matching calculations, avoiding evaluation errors caused by inconsistent data structures, and saving intermediate results to support subsequent detailed analysis (such as backtracking of false positives / missed positives).

[0098] For each image in the validation set, traverse all its predicted boxes and true annotation boxes.

[0099] IoU calculation, for each predicted box, calculate its intersection over union (IoU) with all real boxes: Intersection area: the number of pixels in the overlapping area between the predicted box and the true box; Union area: predicted box area + true box area - intersection area; IoU = Intersection over Union (IoU) / Area over Union (AUI).

[0103] Matching judgment: if the IoU between a predicted box and a real box is ≥ 0.5 (the threshold is adjustable) and they are of the same category, it is considered a valid match.

[0104] The IoU of the present invention is used as a standard metric to intuitively reflect the degree of spatial overlap between the predicted box and the real target, avoiding the limitation of relying solely on coordinate errors. By traversing all possible box pair combinations, it ensures that each real target has the opportunity to be correctly matched, thereby improving the evaluation accuracy in multi-target scenarios. The calculation process of calculating classification accuracy is as follows: Category -level statistics: Statistics are collected by 17 categories of UI elements (such as input boxes, buttons, and verification codes):

[0107] Number of correct detections: The number of predicted boxes and ground-truth boxes with an IoU of ≥ 0.5 and the same category; Total number of annotations : the total number of true annotation boxes of this category in the validation set.

[0109] Class accuracy calculation: Each class accuracy = number of correct detections / total number of annotations × 100%.

[0110] Mean Average Precision (mAP) calculation: The mean precision of the 17 categories is calculated as the validation set mean average precision (mAP@0.5). This method distinguishes the detection performance of different UI elements, identifies the model's strengths and weaknesses (e.g., persistent omissions of certain categories), avoids evaluation bias caused by uneven category distribution (e.g., a dominant category dominates the results), and ensures that all categories contribute equally to the final metric.

[0111] The CIoU (CompleteIoU) loss value of each sample is extracted from the model validation log. The CIoU loss comprehensively considers the overlap rate, center point distance and aspect ratio of the predicted box and the true box, eliminates extreme loss values (such as values exceeding 3 times the standard deviation) to avoid outliers affecting the overall mean. The CIoU loss values of all valid samples are summed and averaged to obtain the mean CIoU loss.The CIoU loss proposed in this paper simultaneously optimizes the overlap rate, position accuracy and shape consistency, and reflects the quality of the predicted box more comprehensively than the traditional IoU loss. By filtering outliers and reducing noise interference, the loss mean can better represent the true performance of the model and avoid misjudgment caused by fluctuations in individual samples. The calculation process of weighted generation fitness score is as follows: Coefficient setting: Precision weight coefficient (e.g., 0.7): reflects the importance of average precision;

[0115] Loss weight coefficient (e.g. 0.3): reflects the importance of CIoU loss. Normalization process:

[0117] Scale the average precision index to the interval [0, 1] (e.g. if the original precision is 85%, it will be 0.85 after normalization);

[0118] The CIoU loss mean is mapped to the [0, 1] interval by 1 / (1 + loss mean) (the smaller the loss, the larger the mapping value).

[0119] Weighted sum: fitness score = accuracy weight coefficient × normalized accuracy + loss weight coefficient × normalized loss mapping value.

[0120] The present invention simultaneously considers the accuracy (precision) and positioning quality (loss) of the model, avoiding one-sided optimization caused by a single indicator (such as only pursuing a high recall rate at the expense of positioning accuracy). By adjusting the weight coefficient, it can flexibly adapt to different business needs (for example, the loss weight can be increased in scenarios with extremely high positioning accuracy requirements).In a preferred embodiment of the present invention, step S3, for an element in the interactive task queue that is located within a frame, converts its coordinates to coordinates relative to the frame, and controls the browser to switch to the corresponding frame, includes:

[0122] Step S31: Analyze the XPath path features generated in S24. If the path contains an HTML tag or a frame hierarchy, determine that the corresponding element is located within a frame, and extract the frame identifier of the frame in the parent page, i.e., the ID or Name attribute.

[0123] Step S32, relative coordinate conversion calculation, read the pixel coordinates (X_frame, Y_frame) of the upper left corner of the frame in the parent page through the browser interface, and read the frame's own width W_frame and height H_frame; Calculate the relative position of an element within the frame: Relative abscissa=(absolute abscissa of element - X_frame) / W_frame; Relative ordinate = (element absolute ordinate - Y_frame ) / H_frame; Output normalized relative coordinates;

[0128] Step S33, based on the frame identifier extracted in S31, locates the DOM node of the corresponding frame in the browser, and switches the WebDriver operation context to the inside of the frame.

[0129] In an embodiment of the present invention, when applied in a specific application, the above steps can be implemented by the following specific schemes, for example: In step S31 , XPath path parsing is performed to traverse the XPath path of each element in the task queue (e.g., / / html / body / iframe[@id='register'] / div / input) and check whether it contains an iframe or frame tag. If an iframe tag exists in the path, the id or name attribute value of the tag (e.g., register) is extracted as the unique identifier of the frame. If the path contains nested frames (e.g., parent frame → child frame), the identifier of each level is recursively extracted to generate a frame hierarchy chain (e.g., ["parent_frame", "child_frame"]).

[0131] The present invention does not require manual labeling and automatically discovers the frame hierarchy in the page through XPath syntax features, solving the cross-frame positioning problem that is difficult to handle with traditional automation tools; extracting id / name as the positioning basis is more stable than coordinates, reducing the risk of positioning failure caused by changes in page layout. In step S32 , the browser driver obtains the coordinates of the upper left corner of the target frame in the parent page (X_frame, Y_frame), obtains the width (W_frame) and height (H_frame) of the frame, reads the absolute pixel coordinates (X_abs, Y_abs) of the element from the task queue, and calculates the offset of the element relative to the upper left corner of the frame: Horizontal offset = X_abs - X_frame;

[0134] Vertical offset = Y_abs - Y_frame. Normalization process: Divide the offset by the frame size to obtain the normalized relative coordinates: Relative abscissa =horizontal offset / W_frame;

[0138] Relative vertical coordinate=vertical offset / H_frame.

[0139] The normalized coordinates of the present invention are independent of the specific screen size. The same set of coordinates can be reused at different resolutions, improving the compatibility of automation scripts.

[0140] Dynamic frame adaptation: Even if the frame position changes due to page responsiveness, as long as the internal structure of the frame remains unchanged, the relative coordinates can still accurately locate the elements, reducing maintenance costs.

[0141] In the above step S33, the frame identifier (such as id="register") extracted in S31 is used to locate the frame DOM node through the switch_to.frame() method of the browser driver (such as Selenium); a context switch operation is performed to limit the scope of all subsequent operations (such as clicks and inputs) to the target frame; if there are multiple layers of nested frames, the frames are switched in sequence according to the hierarchical chain (such as switching to the parent frame first, and then switching to the child frame); after the switch, an element query (such as find_element_by_xpath()) is executed by the driver to verify whether the elements in the frame can be correctly located to ensure that the switch is successful.

[0142] The present invention enables automation tools to operate elements within the framework like operating ordinary pages through context switching, breaking through the limitation that traditional tools can only operate top-level pages, ensuring that all operations on elements within the framework are performed in the correct context, and avoiding operation failures or erroneous operations caused by context confusion.

[0143] In a preferred embodiment of the present invention, step S4 traverses the interactive task queue and performs the following operations in sequence: performing a simulated click operation on a button-type element; injecting virtual identity information into an input box-type element; and calling a corresponding recognition model to complete verification of a verification code-type element, including:

[0144] Step S41, sequentially reading task items starting from the head of the structured task queue generated in S23, and determining the execution order according to the priority tags in the task items; Step S42 , for the current task item: If the element is located in the frame, the relative coordinates calculated in step S32 are used; if it is located in the main document, the absolute coordinates converted in step S24 are used; the operating coordinates in the browser viewport are dynamically calculated: Main document: Use absolute pixel coordinates directly;

[0148] Within the frame: relative coordinates × actual frame size + frame offset;

[0149] Step S43: Execute corresponding operations according to element categories, including button click operations, waiting 500ms for page response time after click, form input operations, branching processing according to verification code type, and sliding verification code.

[0150] Step S44, after each task item operation is completed, the current page visible area is re-captured, and the target detection model is called to verify the operation effect; if the operation fails, the current element positioning information is recorded and the task is reinserted into the tail of the queue.In an embodiment of the present invention , the above steps can be implemented by the following specific schemes when applied, for example:

[0152] Step S41 obtains the sorted structured task queue from step S23 (arranged in descending order of priority, with confidence levels decreasing for tasks of the same priority). Task items are extracted one by one in the queue order. Each task item includes an element type (button / input box / verification code), coordinate information, and a priority tag. If a high-priority task (such as a required input box) exists in the queue, it is processed first; low-priority tasks (such as optional items) are executed after the high-priority task is completed. The present invention ensures that core operations (such as clicking the register button) are executed first, avoiding process interruptions caused by processing secondary elements. If a high-priority task fails, the process can be quickly terminated, reducing resources consumed by ineffective operations.

[0153] In the above step S42, the XPath path of the task item is checked to see if it contains an iframe tag, and whether the element is located in the frame. If it is in the frame, the relative coordinates of S32 are used; if it is in the main document, the absolute coordinates of S24 are used. Coordinate mapping:

[0155] The main document element directly uses the absolute pixel coordinates (X_abs, Y_abs) as the operation point; Elements within the frame : Get the real-time size ( W_frame , H_frame) and top-left corner offset (X_frame, Y_frame) of the current frame; Calculate the operation coordinates: X = relative horizontal coordinate × W_frame + X_frame, Y = relative vertical coordinate × H_frame + Y_frame; if the calculated coordinates exceed the current viewport, scroll the page to bring the element into view. This invention uniformly handles the positioning of elements within the main document and the frame, eliminating the need to write special logic for different contexts. Even if the page size changes due to interaction during runtime, elements can still be accurately manipulated through real-time calculations.

[0159] In the above step S43, a browser driver (such as Selenium) is used to send a mouse click event to the calculated coordinates, and a 500ms wait is made after the click to allow the page to respond (such as loading new content and displaying a prompt box), locate the input box element, clear the original content, inject virtual identity information (such as a randomly generated name, mobile phone number, and email address) according to preset rules, call the OCR model to recognize the characters in the image, input them into the corresponding input box, identify the position of the slider and the gap, calculate the sliding distance, simulate the human operation trajectory (accelerate first and then decelerate) to drag the slider, recognize the prompt text, and click the corresponding image area.

[0160] The present invention simplifies complex UI interactions into a unified operation interface through classification processing, thereby reducing development costs; and customizes solutions for different types of verification codes, so that the automated process can handle more than 80% of common verification mechanisms.

[0161] In step S44, after completing the current task item, the visible area of the page is captured to generate a new image. The new image is input into the target detection model to identify the state change of the operation element (such as whether the button has changed to the "clicked" style, whether the input box is filled with content), and compare the attribute changes of the element before and after the operation (such as the disabled state of the button, the value attribute of the input box). If the verification fails (such as the input box is still empty), the XPath, coordinates and other positioning information of the element are recorded, and the task is reinserted into the end of the queue and marked as requiring retry (up to 3 retries). The present invention verifies the operation effect in real time through model recognition, avoids the hidden error of "operation successful but not effective", automatically retries failed tasks, and improves the success rate of the process (tests have shown that the overall pass rate can be increased by more than 25%).

[0162] In a preferred embodiment of the present invention, step S5, page scrolling and submission detection, after completing the interactive task queue operation, detects whether there is a visible submit button on the current page. If not, scroll the page and re-execute steps S1 to S2 until a visible submit button is identified. If visible, triggering the submit operation includes:

[0163] Step S51: Based on the latest page screenshot after the completion of the S44 operation, call the target detection model to identify whether there is a "submit button" type element in the current visible area; Step S52 : When the detection result of step S51 is not visible, execute: Get the total height of the current page and the browser window height, set the single scroll amount to 80% of the window height; perform the scrolling operation:

[0166] Record the current scroll bar position as the initial position, trigger the browser to scroll down one scroll unit, and wait for 500ms for the page to redraw;

[0167] State update and loop:

[0168] Step S1 is re-executed to capture a new visible area image; step S2 is re-executed to generate a new interactive task queue; step S51 is returned to perform a submit button check, and the process is repeated until any of the following conditions is met:

[0169] A visible submit button is detected and the cumulative scroll amount exceeds the total height of the page;

[0170] Step S53: During each rolling re-identification process, a tag library of operated elements is established to filter the interactive task queue newly generated in S2;

[0171] Step S54, when the visible submit button is detected in step S51, the button coordinates are located.

[0172] In an embodiment of the present invention, the above steps can be implemented in the following manner. For example, the latest page screenshot generated by S44 is input into the target detection model. The model classifies all elements in the screenshot, filters out elements labeled "submit button" (such as text buttons such as "Register" and "Submit Now"), and checks whether the button coordinates are within the current window range; excludes obscured elements (such as pop-up windows or floating layers above the button). The present invention automatically identifies the submit button through the model, avoids manual hard-coding of the button position, adapts to the UI design differences of different websites, and combines the coordinate and obscuration relationship judgment to ensure that only truly operable submit buttons are identified, reducing the risk of false triggering.

[0173] The above step S51 can be implemented in the following ways, for example, obtaining the total height of the page (such as document.body.scrollHeight) and the window height; setting the single scrolling amount to 80% of the window height (to avoid missing elements due to excessive scrolling).

[0174] Rolling execution:

[0175] Record the current scroll bar position as the initial position, trigger the browser to scroll down one scroll unit (such as current position + window height × 80%), wait 500ms to ensure that the page is redrawn and dynamically loaded, and after each scroll, re-execute step S1 (screenshot) and step S2 (generate task queue), and call step S51 again to detect the submit button until the termination condition is met and the cumulative scroll amount exceeds the total height of the page (indicating that the entire content has been traversed).

[0176] The present invention uses an adaptive scrolling strategy to ensure that even if the submit button is located at the bottom of a long page or in a dynamic loading area, it can be detected. Reasonable scrolling step size and waiting time balance the detection efficiency and page loading integrity, and the average detection time is shortened by 30%.

[0177] In step S53, when step S4 is initially executed, an empty tag library of operated elements (such as the collection processed_elements) is established. Each time a task item is completed, the XPath or unique identifier of the element is added to the tag library. When a new task queue is generated after each scroll, each element in the queue is traversed. If the element identifier exists in the tag library, the element is removed from the queue. The present invention prevents repeated operations on filled input boxes or clicked buttons, reduces invalid interactions, and avoids page status confusion caused by repeated operations (such as clicking the submit button multiple times to trigger repeated requests).

[0178] In the above step S54, the bounding box coordinates (left, top, width, height) of the submit button are extracted from the target detection model output, and the coordinates of the center point of the button are calculated as (left + width / 2, top + height / 2). If the button is located within the frame, the relative coordinate conversion method of S32 is used to map the center point coordinates to the browser window. If it is in the main document, the absolute coordinates are directly used. The present invention calculates the center point through the bounding box output by the model, which is more accurate than traditional positioning based on text or CSS selectors, with a success rate of up to 98%. Even if the website modifies the class or ID of the button, as long as the visual style remains unchanged, it can still be accurately positioned, reducing maintenance costs by 50%.

[0179] In a preferred embodiment of the present invention, step S6, after submission, if the page jumps to the recharge page, the payment account information is extracted by regular expression and associated with the virtual identity information and stored in the database, including:

[0180] Step S61: Based on the page jump result triggered by step S54, monitor whether the new page URL contains the preset keyword, intercept the complete HTML source code of the current page, and perform a double verification mechanism to obtain a recharge page that has passed the verification;

[0181] Step S62, perform hierarchical analysis on the recharge page that has passed the verification in step S61, standardize the data and store it in an associated manner, and when the recharge page fails to pass the verification in step S61, package it and transmit it to the data construction module. In an embodiment of the present invention , the above steps can be implemented in the following manner, for example,

[0183] In the above step S61, the characteristic keywords of the recharge page are preset (such as "recharge", "payaccount", "fund-transfer"), the path or parameter part of the new page URL is parsed to determine whether any keyword is contained, the target detection model is called to identify whether there are UI elements related to recharge in the page (such as "payment account", "opening bank" text labels or account number input box), and whether the HTML source code contains form structures related to payment (such as<formaction=" / pay"> ) or a specific JS file reference (such as a payment SDK script). If the double verification passes, the complete HTML source code is intercepted and redundant content such as comments and blank lines are removed. If it fails (such as jumping to an error page or an advertisement page), the error URL is recorded and subsequent operations are terminated. The present invention combines triple-dimensional verification of URL, visual elements, and source code structure to avoid mistakenly identifying similar pages as recharge pages (such as misidentifying the "Account Settings" page). Only pages that pass verification are subsequently processed, reducing invalid data interference and improving the accuracy of account information extraction.In step S62 , the text content in the HTML tag is matched by a regular expression (e.g.<spanid="account"> 123456789), extract account information related fields, combine contextual semantics (such as "bank account" and "payment account" keywords) to locate target data, and exclude interference information (such as fake accounts in advertisements); and clean the format of the extracted account information:

[0185] The bank account number is stripped of spaces and special symbols and unified into pure numbers. The name of the bank where the account is located is mapped to a standard name (e.g., "Industrial and Commercial Bank of China" is uniformly abbreviated as "ICBC"). The virtual identity information of the current operation (e.g., a generated random name, mobile phone number) is obtained from the task queue. A one-to-one association is established between the account information and the identity information through the task ID. The associated data is written into a structured database (e.g., MySQL), with fields including: virtual name, mobile phone number, payment account, bank where the account is located, and an associated timestamp. If the page fails the recharge page verification, the original URL, screenshot, and source code are packaged and transmitted to the data construction module for expanding the training set or optimizing the verification rules.

[0186] The present invention converts unstructured page data into directly usable business data through layered analysis and standardization processing, adapts to the data format requirements of downstream systems, and data that has not passed verification can be fed back into model training to form a "recognition-verification-optimization" data closed loop, continuously improving the system's ability to recognize new recharge pages; the strong association between virtual identity and account information provides complete context data for subsequent batch recharge, fund management and other businesses, reducing the cost of manual data matching.

[0187] A business carrier automated registration and login and fund account acquisition system, comprising:

[0188] An acquisition module is used to access the target website domain name through a headless browser, and after loading, a screenshot of the current visible area is taken to generate a page image;

[0189] A generation module, configured to input the page image into a pre-trained object detection model to identify the category, location coordinates, and confidence level of UI elements in the page; sort the recognition results based on a preset priority rule to generate an interactive task queue sorted in descending order of priority;

[0190] A conversion module, for converting the coordinates of elements in the interactive task queue that are located within a frame into coordinates relative to the frame, and controlling the browser to switch to the corresponding frame;

[0191] A verification module is used to traverse the interactive task queue and perform the following operations in sequence: simulate click operations on button-type elements; inject virtual identity information into input box-type elements; call corresponding recognition models to complete verification of verification code-type elements;

[0192] A processing module is used for page scrolling and submission detection. After completing the interactive task queue operation, it detects whether there is a visible submit button on the current page until a visible submit button is identified. If it is visible, the submission operation is triggered; if it jumps to the recharge page after submission, the payment account information is extracted through a regular expression, and is associated with the virtual identity information and stored in the database. Data set preparation and model training:

[0194] In this embodiment, a structured image dataset covering 17 types of UI elements was constructed by screenshotting, screening, clustering and deduplication of target domain website pages and manually annotating them, providing high-quality training samples for the target detection model. This example uses the YOLOv series model as the UI recognition framework. During the training process, the batch incremental training strategy, CIoU bounding box loss optimization, and multi-scale feature fusion structure are combined, and a genetic algorithm is introduced to automatically tune the hyperparameters and model structure to improve detection accuracy, training efficiency, and model generalization ability. The specific steps are as follows:

[0195] Step 1: Target Website Collection, Screening, and Access Verification: Approximately 20,000 suspicious business-related website domain names were randomly selected from the domain name database. The system used an automated script to access each domain name and perform a preliminary screening based on page characteristics, removing the following types of sites: 1) those that failed to load or received a 404 error; 2) those with empty content and no business relevance; and 3) those that used non-standard structures and could not be captured.

[0196] Step 2, the automatic screenshot process automatically executes the screenshot process for the screened websites in the following order to ensure that the key registration path pages are captured: 1) Close the homepage pop-up window (if any); 2) Click the registration button after the homepage is loaded; 3) After the registration page is fully loaded, scroll to the bottom of the page and capture the page containing all form fields and the "Submit" button; 4) After registration, jump to the business operation page and capture the page containing operation method information.

[0197] Step 3: Image deduplication and clustering to reduce redundancy. To improve training efficiency, the screenshot images are deduplicated using the Ward-based hierarchical clustering method, duplicate samples are removed, and the cluster center images are retained to ensure data diversity and representativeness. Step 4 : UI element annotation: Use the Yolo_mark tool to manually annotate the 17 types of UI elements in the screenshot image. The annotation format conforms to the txt format required by the YOLOv series. Each file contains the element category number and its normalized coordinates in the image.

[0199] Step 5: Data organization and division After constructing the data set, it is divided into a training set, a validation set, and a test set in a ratio of 8:1:1 to ensure that the samples of the three types of pages (close pop-up window, home page, and registration page) are evenly distributed.

[0200] Step 6, parameter optimization training of integrated genetic algorithm Before the formal training of the model, the system introduces the genetic algorithm to perform global search and automatic optimization on the key training parameters of the YOLOv series model to improve the performance and generalization ability of the model in the web UI element recognition task. Specifically, the following steps are included: (1) Genetic algorithm initialization: define the fitness function, use the performance index of the model on the validation set as the evaluation standard, and the fitness is used to measure the quality of the individual, which is the core basis for the genetic algorithm to perform "selection-evolution". The optimized parameter space includes: learning rate, batch size, anchor box size, multi-scale feature fusion structure configuration (FPN level), attention mechanism insertion position, etc. (2) Population generation and evolution: the system initializes to generate multiple individuals (each individual is a set of parameter combinations), and in each generation, each individual is trained for a small number of rounds (such as 10 to 20 rounds) and its fitness value is calculated. Then, standard genetic operations (selection, crossover, mutation) are performed. (3) Iterative optimization: The algorithm iterates generation by generation according to the change in fitness until the stopping condition is met (such as the fitness improvement is lower than the threshold or the maximum number of generations is reached), and finally retains the individual parameters with the highest fitness for formal training.

[0201] Step 7: Perform three-stage model training based on the optimized parameters. Based on the genetic algorithm optimization results, perform model training in a three-stage manner: Freeze the backbone network and train only the detection head to stabilize basic recognition capabilities; Unfreeze some backbone layers and introduce a multi- scale training mechanism to improve the detection accuracy of UI components of different sizes;

[0204] Full network fine-tuning, combined with strategies such as incremental samples and hard example improvement, further optimizes model performance. Automatic registration process:

[0206] In this embodiment, through deep linkage between model recognition and browser execution, the system automatically analyzes the web page structure and simulates user behavior based on the priority mechanism, breaking through the limitations of traditional script fixed paths. The specific steps are as follows:

[0207] Step 1: Initialization access and page screenshot The system calls Selenium Web Driver through the automation control module, starts a headless browser and accesses the target business class website domain name. After the page is loaded, the current visual area is automatically screenshot and the target detection model trained in Example 1 is passed in for processing.

[0208] Step 2: The UI element identification and prioritization model identifies 17 categories of UI elements on the page and outputs category labels, normalized location coordinates, and confidence values. The system classifies elements into five priority levels (immediate, high, moderate, low, and last) based on a preset priority mechanism:

[0209] immediate priority: "Close pop-up button" and cover key areas;

[0210] High priority: Jump components such as "Register button" and "Login button"; Moderate priority: form input components (user name, password input boxes, etc.);

[0212] Low priority: verification code component (graphic verification code, sliding verification code); Last priority : non-immediately interactive components such as "Submit Button" and "Contact Customer Service". After priority division, an interactive task queue is generated in descending order, containing information such as element category, coordinates, confidence level, and XPath path.

[0214] Step 3, coordinate conversion of elements within the frame. If the element is located in or , the system converts the coordinates to relative positions, switches WebDriver to the corresponding frame, and uses a custom script to convert the coordinates to XPath paths to ensure operation accuracy.

[0215] Step 4: simulate the operation and perform the traversal task queue, and perform the corresponding operation according to the category:

[0216] Click operation: simulate clicking buttons such as "Register", "Close", and "Submit";

[0217] Input operation: input the compliance identity information generated by the virtual information pool into the form field; Verification code processing: Graphic verification code: Use OCR model to identify and input;

[0220] Sliding verification code: Calculate the sliding distance through image processing and simulate dragging. After the operation is completed, automatically take a screenshot and repeat the recognition to enter the next step of the process.

[0221] Step 5: After the page scrolling and submission detection complete the interactive operation, the system analyzes the page status: If a "Submit button" is found in the visible area, submit directly.

[0223] If the button is not recognized or is outside the window, the page is automatically scrolled and re-recognized until the button is detected;

[0224] Establish an interactive locking mechanism to mark the operated elements to avoid repeated triggering.

[0225] Step 6: After the registration status is submitted, determine whether to jump to the business operation page:

[0226] Unsuccessful: record the status and error screenshot, mark the failure and transfer it to the data module;

[0227] Success: Enter the information extraction process.

[0228] Business information extraction:

[0229] After successfully entering the business operation page, the system parses the content by matching key fields with regular expressions, extracts information and stores it in a standardized manner.The specific steps are as follows:

[0230] Step 1: Page parsing and information extraction: Identify operation methods and account information by matching regular expressions with keywords on the page screenshot or HTML source code, including:

[0231] Payment account and account holder information; third-party payment identifier; electronic payment account; other compliance business fields.

[0232] Step 2: Information classification and standardization: The extracted information is formatted (such as removing spaces and mapping standard names) and classified by tags such as type, source, and time.

[0233] Step 3: Data association and storage: The virtual identity information used for registration is associated with the extracted business operation information and stored in the back-end database to provide support for subsequent business analysis. Example 4 : Abnormality and Feedback Mechanism

[0235] Through task status monitoring and failed sample feedback, anomaly identification and data reflow are achieved, supporting model optimization and system iteration. The specific steps are as follows:

[0236] Step 1: Exception handling If the following exceptions are encountered during the registration process, the operation will be interrupted and the cause will be recorded:

[0237] Element recognition failure or misjudgment; page jump abnormality; verification code processing failure; repeated operations leading to logical interruption.

[0238] Step 2: The failed sample feedback and failed task are transmitted back to the data module, and the screenshots and failure classifications are automatically stored for incremental model training and system optimization.

[0239] Step 3, locking mechanism and maximum number of attempts To prevent infinite loops, a maximum number of operations is set for each type of UI element (no more than 2 times by default, except for the close button, which allows up to 5 times).

[0240] The above embodiment uses the business automation process as a scenario and presents the technical solution through compliance description, focusing on:

[0241] The model generalization capability is improved through clustering deduplication, hierarchical labeling, and genetic algorithm optimization. The whole process is automated through priority-based task scheduling, cross-framework operations, and dynamic rolling detection. The system stability and adaptability are continuously improved through the return of failed samples. The solution can be widely used in compliance business scenarios that require automated page interaction and data collection (such as enterprise services, digital operations, etc.).

[0242] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.< / iframe>

Claims

1. A method for automated business carrier registration and login and fund account acquisition, characterized in that: The method comprises: Step S1: access the target website domain name through a headless browser, take a screenshot of the current visible area after loading, and generate a page image; Step S2: Input the page image into a pre-trained object detection model to identify the category, location coordinates, and confidence level of UI elements in the page; sort the recognition results based on preset priority rules to generate an interactive task queue sorted in descending order of priority; the pre-trained object detection model includes: Target website domain names are extracted from a database of abused domain names. An automated script accesses the domain names and verifies the page loading status. For websites that pass verification, pop-up window closing, registration button clicks, and page scrolling are automatically triggered, capturing full page screenshots containing key interactive elements. Based on the full page screenshots, a hierarchical clustering algorithm is used to group images with similar visual features to obtain deduplicated images. The deduplicated images are annotated with the category labels and location information of 17 types of UI elements, generating annotation files that conform to the target detection format. The annotation files are then integrated and divided into training, validation, and test sets according to a preset ratio. Based on the training, validation, and test sets, the following automated optimization process is performed: Initialize the parameter population, where each parameter set includes the learning rate, anchor box size, and network structure configuration; train the model for each parameter combination, and calculate the fitness using the validation set accuracy and loss value; iteratively generate a new population through selection, crossover, and mutation operations until the fitness converges to obtain optimized parameters; use the optimized parameters to fine-tune the entire network and integrate incremental sample training to obtain a pre-trained object detection model; Step S3: for the elements in the interactive task queue that are within the frame, convert their coordinates into coordinates relative to the frame, and control the browser to switch to the corresponding frame; Step S4, traverse the interactive task queue and perform the following operations in sequence: perform a simulated click operation on the button type element; inject virtual identity information into the input box type element; call the corresponding recognition model to complete the verification of the verification code type element; Step S5, page scrolling and submission detection: After completing the interactive task queue operation, check whether there is a visible submit button on the current page. If not, scroll the page and re-execute steps S1 to S2 until a visible submit button is identified. If visible, trigger the submit operation; Step S6: After submission, if the page jumps to the recharge page, the payment account information is extracted through regular expressions, and is associated with the virtual identity information and stored in the database.

2. The method for automatic business carrier registration and login and fund account acquisition according to claim 1, characterized in that: Step S1: Access the target website domain name through a headless browser, take a screenshot of the current visible area after loading, and generate a page image, including: Step S11: After the page is loaded, detect whether there is a pop-up window element blocking the key area in the current visible area; if so, automatically calculate the coordinate position of the pop-up window close button and trigger a simulated click operation to expose the underlying page elements; Step S12: Based on the page status of the cleared pop-up window, identify the location of the registration button on the page; calculate the coordinates of the center point of the registration button, and execute the coordinate positioning click through the browser driver to trigger the page to jump to the registration page; Step S13: After the registration page is loaded, the browser is controlled to perform a vertical scrolling operation: the initial scrolling position is the top of the page, the scrolling step size is set according to the screen height, and the browser scrolls down step by step until it reaches the bottom of the page; after scrolling to the bottom, a screenshot operation is performed to capture a complete image including all form input boxes and the submit button at the bottom of the page; Step S14, after completing the registration form submission operation, monitor the new page URL that the browser jumps to. When the page URL contains characteristic keywords, it is determined to be a recharge page. After the page is loaded, directly capture the current visible area image.

3. The method for automatic business carrier registration and login and fund account acquisition according to claim 2, characterized in that: Step S2: input the page image into a pre-trained object detection model to identify the category, location coordinates and confidence of UI elements in the page; Sort the recognition results based on the preset priority rules and generate an interactive task queue in descending order of priority, including: Step S21: Input the page image generated in steps S13 and S14 into a pre-trained object detection model, and output a set of recognition results for each UI element; each recognition result includes a category label, location coordinates, and confidence level; Step S22: Prioritize each recognition result output from step S21 based on the Euclidean distance between the center point of the element and the center of the page to obtain weights corresponding to the category label, location coordinates, and confidence level. Priority scores are obtained based on the weights corresponding to the category label, location coordinates, and confidence level, and the elements are divided into five levels from high to low according to the scores to obtain a set of elements with priority labels. Step S23, arranging the set of elements with priority labels output in step S22 in descending order of priority, and arranging elements with the same priority in descending order of confidence, to generate a structured task queue; Step S24, traverse the structured task queue of step S23, calculate the normalized coordinates of each element, that is, calculate its absolute pixel coordinates in the browser window; and convert the absolute pixel coordinates into an XPath path through a DOM position inversion algorithm.

4. The method for automatic business carrier registration and login and fund account acquisition according to claim 3 is characterized in that: Step S3, for the elements in the interactive task queue that are within the frame, converting their coordinates to coordinates relative to the frame, and controlling the browser to switch to the corresponding frame, including: Step S31, analyzing the XPath path features generated in S24, if the path contains HTML tags or a frame hierarchy, determining that the corresponding element is within a frame, and extracting the frame identifier of the frame in the parent page, i.e., the ID or Name attribute; Step S32, relative coordinate conversion calculation, read the pixel coordinates (X_frame, Y_frame) of the upper left corner of the frame in the parent page through the browser interface, and read the frame's own width W_frame and height H_frame; Calculate the relative position of an element within a frame: Relative horizontal coordinate = (element absolute horizontal coordinate - X_frame) ÷ W_frame; Relative ordinate = (element absolute ordinate - Y_frame) ÷ H_frame; Output normalized relative coordinates; Step S33: Based on the frame identifier extracted in S31, locate the DOM node of the corresponding frame in the browser, and switch the WebDriver operation context to the inside of the frame.

5. The method for automatic business carrier registration and login and fund account acquisition according to claim 4 is characterized in that: Step S4, traverse the interactive task queue and perform the following operations in sequence: perform a simulated click operation on a button-type element; inject virtual identity information into an input box-type element; Call the corresponding recognition model to complete the verification of the verification code element, including: Step S41, sequentially reading task items starting from the head of the structured task queue generated in S23, and determining the execution order according to the priority tags in the task items; Step S42, for the current task item: If the element is located in the frame, the relative coordinates calculated in step S32 are used; if it is located in the main document, the absolute coordinates converted in step S24 are used; the operation coordinates in the browser viewport are dynamically calculated: Main document: Use absolute pixel coordinates directly; Inside the frame: relative coordinates × actual frame size + frame offset; Step S43: Execute corresponding operations according to element categories, such as button click operations, wait 500ms for page response time after click, form input operations, branch processing according to verification code type, and slide verification code; Step S44: After completing each task item operation, re-capture the current page visible area and call the target detection model to verify the operation effect; if the operation fails, record the current element positioning information and reinsert the task into the tail of the queue.

6. The method for automatic business carrier registration and login and fund account acquisition according to claim 5, characterized in that: Step S5, page scrolling and submission detection. After completing the interactive task queue operation, check whether there is a visible submit button on the current page. If not, scroll the page and re-execute steps S1 to S2 until a visible submit button is identified. If visible, trigger the submit operation, including: Step S51: Based on the latest page screenshot after the completion of S44, call the object detection model to identify whether there is a "submit button" type element in the current visible area; Step S52, when the detection result of step S51 is invisible, execute: Get the total height of the current page and the browser window height, set the single scroll amount to 80% of the window height; perform the scrolling operation: Record the current scroll bar position as the initial position, trigger the browser to scroll down one scroll unit, and wait 500ms for the page to redraw; State update and loop: Re-execute step S1 to capture the new visible area image; re-execute step S2 to generate a new interactive task queue; return to step S51 to perform submit button detection, and loop until any of the following conditions is met: A visible submit button is detected and the cumulative scrolling exceeds the total height of the page; Step S53: During each rolling re-identification process, a library of operated element tags is established to filter the interactive task queue newly generated in S2; Step S54: when the visible submit button is detected in step S51, the button coordinates are located.

7. The method for automatic business carrier registration and login and fund account acquisition according to claim 6, characterized in that: Step S6: After submission, if the page jumps to the recharge page, the payment account information is extracted through regular expressions and associated with the virtual identity information and stored in the database, including: Step S61: Based on the page jump result triggered by step S54, monitor whether the new page URL contains the preset keyword, intercept the complete HTML source code of the current page, and perform a double verification mechanism to obtain a recharge page that has passed the verification; Step S62: perform hierarchical analysis on the recharge page that has passed the verification in step S61, standardize the data and store it in an associated manner. If the recharge page fails the verification in step S61, it is packaged and transmitted to the data construction module.

8. The method for automatic business carrier registration and login and fund account acquisition according to claim 7, characterized in that: The fitness is calculated using the validation set accuracy and loss value, including: Based on the model training output of the current parameter combination, load its prediction results on the validation set; Count the matching between the predicted box and the real box, that is, for each image in the validation set, calculate the overlap between the target detection model's predicted box and the labeled box; Count the proportion of correctly detected UI elements to the total number of corresponding category annotations for each of the 17 categories, take the arithmetic average of the accuracy values of all categories, and obtain the average accuracy index of the validation set; Extract the CIoU loss values of all samples from the object detection model validation log, and take the arithmetic average of the CIoU loss values of all samples to obtain the mean CIoU loss; The accuracy weight coefficient and the loss weight coefficient are defined, and the average accuracy index is weighted as positive correlation and the CIoU loss mean is weighted as negative correlation to generate a fitness score.

9. A business carrier automated registration and login and fund account acquisition system, characterized in that: The system is used to perform the method according to any one of claims 1 to 8, comprising: The acquisition module is used to access the target website domain name through a headless browser, take a screenshot of the current visible area after loading, and generate a page image; A generation module is configured to input the page image into a pre-trained object detection model to identify the category, location coordinates, and confidence level of UI elements in the page; sort the recognition results based on a preset priority rule to generate an interactive task queue sorted in descending order of priority; The conversion module is used to convert the coordinates of the elements in the interactive task queue that are located in the frame into the coordinates of the relative frame, and control the browser to switch to the corresponding frame; The verification module is used to traverse the interactive task queue and perform the following operations in sequence: simulate click operations on button-type elements; inject virtual identity information into input box-type elements; and call corresponding recognition models to complete verification of verification code-type elements; The processing module is used for page scrolling and submission detection. After completing the interactive task queue operation, it detects whether there is a visible submit button on the current page until a visible submit button is identified. If it is visible, the submission operation is triggered; after submission, if the page jumps to the recharge page, the payment account information is extracted through regular expressions and associated with the virtual identity information and stored in the database.

Images