Web page dynamic element picking method and system based on RPA

Abstract

The present invention provides a method and system for picking dynamic elements on a web page based on RPA, which relates to the field of digital technology. First, a life cycle tracking model of the element is established, covering the initial loading, state change, invalidation recovery stage and inter-stage transfer conditions in the element interaction process. Then, multi-source interaction signals such as page loading events, user operation triggers, and element attribute changes of the target web page are collected. The multi-source interaction signals are analyzed based on the life cycle tracking model, and the state evolution pattern of the element in different life cycle stages is extracted, including loading, change, and recovery trigger conditions. According to the state evolution pattern, a dynamic positioning rule set including basic positioning, change response positioning, and recovery compensation positioning is generated. When executing the web page element picking operation, the current stage of the element is determined by the life cycle tracking model, and the corresponding dynamic positioning rule is called to complete the picking. The result is fed back to optimize the state evolution pattern, thereby improving the accuracy and stability of the dynamic element picking on the web page.

Description

Technical Field

[0001] The present invention relates to the field of digital technology, and in particular to a method and system for picking dynamic elements of a web page based on RPA. Background Art

[0002] In today's digital age, web applications are becoming increasingly complex and dynamic, and the presentation and interaction methods of web elements have become extremely rich and diverse. Traditional web element picking methods are mainly based on static page structures, locating elements through pre-set fixed positioning rules, such as those based on element IDs, class names, tag names, and other attributes. However, with the continuous development of web technology, especially the emergence of a large number of web pages that adopt technologies such as dynamic loading, asynchronous updates, and interactive state changes, traditional methods face many challenges. For example, dynamically loaded elements may not exist when the page is initially loaded and may only appear after specific events are triggered; the state of elements can frequently change with user operations, and their properties may also change accordingly; and some elements are recycled and destroyed after completing specific interactions. Existing web element picking technologies have difficulty accurately tracking and adapting to these dynamic changes, resulting in a significant decrease in the accuracy and stability of element picking, which cannot meet the needs of modern web automation processing. Summary of the Invention

[0003] In view of the above-mentioned problems, in combination with the first aspect of the present invention, an embodiment of the present invention provides a method for picking dynamic elements of a web page based on RPA, the method comprising:

[0004] Establishing an element lifecycle tracking model, wherein the lifecycle tracking model includes definitions of the initial loading phase, state change phase, and failure recovery phase of the element during the interaction process, as well as a set of state transition conditions between phases;

[0005] Collecting multi-source interaction signals of the target web page, wherein the multi-source interaction signals include page loading event signals, user operation trigger signals, and element attribute change signals;

[0006] Analyze the multi-source interaction signals based on the lifecycle tracking model to extract state evolution patterns of elements at different lifecycle stages, wherein the state evolution patterns include loading trigger conditions, change trigger conditions, and recycle trigger conditions;

[0007] Generate a dynamic positioning rule set according to the state evolution pattern, wherein the dynamic positioning rule set includes a basic positioning rule, a change response positioning rule, and a recovery compensation positioning rule;

[0008] When executing the web page element picking operation, the current life cycle stage of the element is determined by the life cycle tracking model, the dynamic positioning rules of the corresponding stage are called to complete the picking, and the picking result is fed back to the life cycle tracking model to optimize the state evolution mode.

[0009] On the other hand, an embodiment of the present invention also provides an RPA-based web page dynamic element picking system, including a processor and a machine-readable storage medium, the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0010] Based on the above aspects, by establishing an element lifecycle tracking model, it comprehensively covers the various stages of the elements in the interaction process and their state transition conditions, collects multi-source interaction signals of the target web page, and can obtain information on element state changes from multiple dimensions, ensuring a comprehensive perception of the dynamic behavior of the elements. Based on the lifecycle tracking model, the multi-source interaction signals are analyzed and the state evolution pattern is extracted, making the understanding of the dynamic characteristics of the elements more in-depth and accurate. According to the state evolution pattern, a dynamic positioning rule set is generated, and the dynamic adjustment of the positioning rules as the element state changes, which effectively solves the problem of fixed positioning rules in traditional methods. When executing the web page element picking operation, the current stage of the element is determined by the lifecycle tracking model and the corresponding positioning rule is called, which greatly improves the accuracy and stability of element picking. At the same time, the picking results are fed back to the lifecycle tracking model to optimize the state evolution pattern, further improving the adaptability to complex dynamic web pages. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a schematic diagram of the execution flow of the RPA-based dynamic element picking method for a web page provided in an embodiment of the present invention.

[0012] Figure 2 Schematic diagram of exemplary hardware and software components of an RPA-based web page dynamic element picking system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0013] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1 This is a flow chart of a method for picking dynamic elements from a web page based on RPA provided by an embodiment of the present invention. The following is a detailed introduction to the method for picking dynamic elements from a web page based on RPA.

[0014] Step S110: Establishing a lifecycle tracking model of the element, wherein the lifecycle tracking model includes definitions of the initial loading phase, state change phase, and failure recovery phase of the element during the interaction process, as well as a set of state transition conditions between phases.

[0015] In the healthcare industry, remote consultation platforms allow patients to upload medical records and imaging data, allowing doctors to review the data online and provide consultations and diagnoses and treatment plans. The remote consultation platform page includes elements such as a patient information panel, an imaging display area, and a consultation record input box. These elements undergo different states and changes during interaction, necessitating the establishment of a lifecycle tracking model for accurate management and tracking.

[0016] Step S111: define the initial loading phase as the time interval when the element first appears in the page DOM tree, and the end condition of the phase is when the element triggers the attribute change event for the first time.

[0017] When a doctor logs into the remote consultation platform and chooses to view a patient's consultation page, the DOM tree for the consultation page begins to be constructed. The patient information section elements are first loaded into the DOM tree. These elements contain information such as the patient's name, age, and symptom description. The initial loading phase is the period from when these elements first appear in the DOM tree to when their first attribute changes occur.

[0018] Taking the patient name element as an example, when loading, it may be displayed on the page with the default font and color. When the system's user preference is set to a specific font, the font properties of the patient name element change, and this event marks the end of the initial loading phase of the patient name element. For the symptom description element, it may be a simplified version when loaded. When the background automatically supplements the detailed symptom information, its textContent property changes, which also means that the initial loading phase of the symptom description element ends. When the image display area element is loading, it may first display a placeholder image. When the image data is loaded and replaces the placeholder image, its src property changes, and the initial loading phase ends.

[0019] Step S112: define the state change phase as the time interval from the first attribute change to the last valid interaction of the element, and the end condition of the phase is that the time length during which the element is not accessed continuously and the attribute does not change reaches a preset threshold.

[0020] After an element's first attribute change occurs, it enters the state change phase. For example, when a doctor begins typing in a consultation note, the textContent property of the input box changes, gradually filling in the doctor's text. During the input process, the doctor may adjust the input box's style, such as changing the font size or color, which will also cause the input box's style property to change.

[0021] The state change phase starts from the first change in the input box attributes to the last time the doctor enters or modifies the consultation record and confirms submission. During this phase, the element state changes dynamically with the doctor's operations. If the input box is not accessed by the doctor and the attributes no longer change within a period of time, the state change phase ends when the preset threshold is reached. For example, after the doctor completes the consultation record input, he does not operate the input box for a period of time, and the content and style of the input box remain unchanged. When the preset time is reached, the state change phase of the consultation record input box element ends. For elements in the image display area, the doctor may perform operations such as scaling and rotating the image, which will cause changes to the element's transform and other attributes. The time interval from the first operation that causes the attribute change to the last time the doctor operates on the image is the state change phase. If there is no operation for a long time and the attributes remain unchanged, the preset threshold is reached and the state change phase ends.

[0022] Step S113: define the invalidation recovery phase as the time interval from the end of the state change phase to the removal of the element from the DOM tree, and the phase end condition is that the element node is deleted from the DOM tree.

[0023] When the consultation ends, or the consultation record is no longer needed for display on the current page due to archiving, data cleanup, etc., the page element enters the invalidation and recycling phase. The invalidation and recycling phase is the period from the end of the state change phase to the deletion of the element node from the DOM tree.

[0024] Taking the patient information section element as an example, after the consultation is over, the element still exists in the DOM tree for a period of time, but it is no longer used. As the data cleanup operation progresses, the element can be removed from the DOM tree. In the period before removal, the patient information section element is in the invalidation and recovery stage. For the consultation record input box element, after the consultation record is saved to the archive database, the input box element is no longer necessary and will go through the invalidation and recovery stage until it is deleted from the DOM tree. After the image data is transferred to long-term storage, the image display area element will also enter the invalidation and recovery stage and will eventually be removed from the DOM tree.

[0025] Step S114: Establish a set of state transition conditions between stages, wherein the state transition condition set includes a property change trigger condition from the initial loading stage to the state change stage, a no-interaction duration trigger condition from the state change stage to the invalidation recovery stage, and a page refresh and reload trigger condition from the invalidation recovery stage to the initial loading stage.

[0026] Step S1141: define the attribute change triggering condition from the initial loading phase to the state change phase as at least one id attribute, class attribute or textContent attribute change event of the element occurring during the initial loading phase.

[0027] On the remote consultation platform page, changes to the id attribute, class attribute, or textContent attribute of an element can trigger a transition from the initial loading stage to the state change stage. Taking the patient age element as an example, it may be displayed in a default style when loaded. When the system adds a specific class attribute to it based on the age range, such as the "elderly patient" class, this attribute change event triggers the patient age element to enter the state change stage. For the consultation prompt information element, when the background updates the prompt content, its textContent attribute changes, causing the element to enter the state change stage from the initial loading stage. If the id attribute of the patient information section element changes due to system architecture adjustments, the element will also enter the state change stage.

[0028] Step S1142: define the trigger condition of the non-interaction duration from the state change phase to the invalidation recovery phase as the time duration during which the element is not triggered by user operations and no attribute change event occurs during the state change phase exceeds a preset threshold.

[0029] During the state change phase, the element state changes with user operations. If an element has not been operated by the user and its attributes have not changed within a period of time, when the preset threshold is exceeded, the element enters the invalidation and recovery phase. Taking the consultation record input box element as an example, after the doctor completes the input, the input box is not operated for a long time, and the textContent and style attributes of the input box remain unchanged. When the preset time is reached, the input box element enters the invalidation and recovery phase. After the doctor views the image, if the image display area element has not been zoomed, rotated, or other operations for a long time, and the element attributes remain unchanged, it exceeds the preset threshold and enters the invalidation and recovery phase. After the consultation, if the patient information section element has not been viewed by the doctor or other personnel for a long time, and the attributes have not changed, and the preset time is reached, it enters the invalidation and recovery phase.

[0030] Step S1143: The page refresh and reload triggering condition from the invalid recovery phase to the initial loading phase is defined as a page triggering refresh event and a new node with the same tag type and initial content as the historical element appears in the refreshed DOM tree.

[0031] When the page triggers a refresh event, and a new node with the same tag type and initial content as the historical element appears in the refreshed DOM tree, the element returns from the invalidation recovery stage to the initial loading stage. For example, after the doctor finishes the consultation, he closes the page and reopens it, and the page is refreshed. At this time, the patient information section element is reloaded into the DOM tree, and its tag type and initial display content are the same as before, and the patient information section element enters the initial loading stage. After the page is refreshed, the consultation record input box element is redisplayed as an empty input box, and the tag type and initial state are the same as before, entering the initial loading stage. After the page is refreshed, the image display area element redisplays the placeholder image and waits for the image data to load. Its tag type and initial display content are the same as the historical element, and enters the initial loading stage.

[0032] Step S1144: configure verification logic for each state transition condition, the verification logic includes event type verification for attribute change trigger condition, timestamp difference verification for no interaction duration trigger condition, and node attribute matching verification for page refresh and reload trigger condition.

[0033] When verifying the event type for attribute change trigger conditions, the type of change event must be verified when determining whether an attribute change triggers a phase transition. For example, for an attribute change event for a patient symptom description element, it is necessary to confirm that the change is to the textContent attribute and not to other unrelated attributes. By examining the event details, the change event type meets the requirements, thereby accurately determining whether the attribute change trigger condition has been met.

[0034] For timestamp difference verification with a no-interaction duration trigger, you can record the timestamp of the element's last access and the current timestamp. You can calculate the difference between these two timestamps and compare it with a preset threshold. For example, for a consultation record input box element, you can record the timestamp of the doctor's last input operation and continuously monitor the current timestamp. When the timestamp difference exceeds the preset threshold and the element attributes remain unchanged, a phase transition is triggered.

[0035] For node attribute matching verification in the page refresh reload trigger, after a page refresh, the attributes of the newly appeared nodes are checked. For elements in the patient information section, the new node's attributes, such as label type and initial content, are compared with those of the historical elements to ensure consistency. Only when these attributes match is the page refresh reload trigger considered met and the element enters the initial loading phase.

[0036] Step S1145: Establish a trigger priority order for the state transition conditions, where the attribute change trigger condition takes precedence over the no-interaction duration trigger condition, and the page refresh and reload trigger condition takes precedence over the trigger conditions in other stages. Construct a state transition condition set based on condition definition, verification logic and priority order.

[0037] In the telemedicine platform, to ensure the accuracy and rationality of element state transitions, a trigger priority order for state transition conditions is established. The attribute change trigger condition takes precedence over the no-interaction duration trigger condition. This means that when an element meets both the attribute change trigger condition and the no-interaction duration trigger condition, the attribute change trigger condition takes precedence for phase transition. For example, if a patient's examination result element meets the no-interaction duration trigger condition and its textContent attribute changes, the element will be prioritized for transition from the initial loading phase to the state change phase based on the attribute change trigger condition.

[0038] The page refresh reload trigger condition takes precedence over trigger conditions in other stages. When the page is refreshed, regardless of the stage an element is in, as long as the page refresh reload trigger condition is met, the element will be prioritized to return to the initial loading stage. For example, a patient's basic information element in the state change stage can be immediately returned to the initial loading stage when the page is refreshed and the page refresh reload trigger condition is met. By setting the above priority order, combined with the previous condition definition and verification logic, a complete set of state transition conditions is constructed.

[0039] Step S115: Configure status monitoring indicators for each life cycle stage, which include the DOM node existence indicator in the initial loading stage, the attribute change frequency indicator in the state change stage, and the node removal mark indicator in the failure recovery stage. Build an element life cycle tracking model based on stage definition, transfer conditions and monitoring indicators.

[0040] In the patient consultation page of the remote consultation platform, configuring corresponding status monitoring indicators for different life cycle stages is an important part of building an element life cycle tracking model.

[0041] During the initial loading phase, configure a DOM node existence indicator to monitor whether an element has been successfully loaded into the DOM tree. This continuously checks whether the element's node exists in the DOM tree. If so, it indicates that the element is in the initial loading phase. For example, when a patient's family medical history element appears in the DOM tree, the DOM node existence indicator can confirm that it is in the initial loading phase. If an error occurs during the loading process and the element is not successfully added to the DOM tree, the DOM node existence indicator is not met, and error handling or reloading may be required.

[0042] For the state change phase, configure the attribute change frequency indicator to record the number and time interval of element attribute changes, and determine the activity level of the element by calculating the attribute change frequency. For example, the attributes of a patient's real-time vital signs element (such as heart rate, blood pressure, and other values) will change frequently. You can confirm that the element is in the state change phase by monitoring the attribute change frequency. If the attribute change frequency is too low, it may mean that the element is about to enter the invalidation recovery phase. For the consultation record input box element, when the doctor enters the consultation record, the textContent attribute of the input box will continue to change. By counting the attribute change frequency, you can accurately determine whether the element is in an active state change phase.

[0043] For the invalidation recovery phase, configure the node removal marking indicator. When the node of an element is removed from the DOM tree, the element can be marked as removed, and the node removal marking indicator can be used to confirm that the element is in the invalidation recovery phase. For example, after the patient's expired examination report element is removed from the DOM tree, the node removal marking indicator can be used to confirm that it has entered the invalidation recovery phase. During the data cleaning process, the node removal marking indicator can be used to manage and clean up elements that are no longer needed to ensure page performance and data accuracy. Based on the previous stage definitions, state transition condition sets, and these state monitoring indicators, a complete element lifecycle tracking model is constructed.

[0044] Step S120: collecting multi-source interaction signals of the target web page, wherein the multi-source interaction signals include page loading event signals, user operation trigger signals, and element attribute change signals.

[0045] In the context of remote consultation platforms in the medical industry, collecting multi-source interaction signals on the target web page is a key step in gaining a deeper understanding of the interactions between web page elements. These multi-source interaction signals can reflect the page loading process, user operation behavior, and changes in element attributes.

[0046] Step S121: Register a page loading event listener in the browser kernel layer, capture page initialization loading events, local content dynamic loading events, page refresh events, and record event type, trigger timestamp, and loaded content range as page loading event signals.

[0047] A page load event listener is registered at the browser kernel layer of the remote consultation platform to capture various page load-related events. When a doctor opens a patient's consultation page, a page initialization load event is triggered. The listener records the event type as a page initialization load event, along with the event trigger timestamp and the content scope of the load.

[0048] The page initialization loading event involves the loading of a large number of elements, such as patient basic information elements, consultation history elements, etc. For the patient basic information elements, sub-elements including name, age, gender, etc. will be loaded at this stage, and the loading content range covers all fields of these basic information. Local content dynamic loading events usually occur when a specific part of the page needs to be updated. For example, when a doctor clicks to view the patient's latest examination report, the page will trigger a local content dynamic loading event, and the listener will record the event type, trigger timestamp, and the specific content range loaded, that is, the detailed data of the latest examination report.

[0049] Page refresh events may be triggered when doctors need to obtain the latest information. When a doctor refreshes the consultation page, the listener records the event type as a page refresh event, the trigger timestamp, and the scope of the content reloaded after the refresh. This may include the latest consultation status, the patient's real-time vital signs, and other elements. By recording these events in detail, page load event signals are generated. These page load event signals contain rich page load information, reflecting the page loading status at different stages.

[0050] Step S122: deploy an operation behavior capture module in the user interaction layer, monitor mouse click events, keyboard input events, and scroll bar sliding events, and record the operation type, action element identifier, operation timestamp, and page status before and after the operation as user operation trigger signals.

[0051] The action capture module is deployed in the user interaction layer of the remote consultation platform to monitor various user actions. When a doctor performs a mouse click on the consultation page, such as clicking to view a patient image thumbnail, the action capture module listens for this event. By intercepting the target attribute of the mouse click event, the module retrieves the identifier of the affected element, namely the patient image thumbnail element.

[0052] The operation type is recorded as a mouse click event, and the timestamp of the operation is recorded, as well as the state changes of the elements and pages before and after the click. Before clicking, the image thumbnail may be in the default display state. After clicking, a detailed image viewing window may pop up, and the page layout will be adjusted accordingly. These state changes will be recorded. When a doctor performs a keyboard input event, such as entering a consultation opinion in the consultation record input box, the operation behavior capture module will listen to the event. By intercepting the target attribute of the keyboard input event, the identifier of the affected element, that is, the identifier of the consultation record input box element, is obtained.

[0053] The operation type is recorded as keyboard input event, operation timestamp, and the change of textContent attribute of the element before and after input. Before input, the input box may be empty. After input, the content of the consultation opinion entered by the doctor can be displayed. When the doctor performs a scroll bar sliding event, such as sliding the scroll bar when viewing a long consultation history, the operation behavior capture module will listen to the event. By intercepting the target attribute of the scroll bar sliding event, the identifier of the affected element is obtained, that is, the identifier of the consultation history display area element.

[0054] The recorded operation type is the scroll bar sliding event, the operation timestamp, and the changes in the element's position-related properties before and after the scroll, such as the change in the element's scrollTop property. Before the scroll, the element displays the top content of the consultation history. After the scroll, the displayed content moves downward accordingly. All these position changes are recorded.

[0055] Step S1221: Inject an operation monitoring script into the event processing layer of the browser page, intercept the target attribute of the mouse click event to obtain the affected element identifier, and record the click coordinates, number of clicks, and class attribute changes of the element before and after the click.

[0056] An action listener script is injected into the browser page's event processing layer to specifically handle mouse click events. When a doctor clicks a mouse on the remote consultation platform's consultation page, the action listener script intercepts the target attribute of the mouse click event to retrieve the ID of the clicked element. For example, if a doctor clicks the Edit button in the patient information section, the script intercepts the event and retrieves the ID of the Edit button element.

[0057] At the same time, the script will record the coordinate information of the click, that is, the specific location of the mouse click on the page. The click coordinates can reflect the doctor's operation location preference, which is of great significance for the subsequent analysis of user behavior. The script will also record the number of clicks. For example, the doctor may click the edit button multiple times to perform different editing operations. The number of clicks can help analyze the frequency of operations. In addition, the script will record the changes in the class attributes of the elements before and after the click. Before clicking, the edit button may have a default style class. After clicking, a style class indicating the clicked state may be added. By recording these class attribute changes, you can understand the changes in the style state of the element.

[0058] Step S1222: intercept the target attribute of the keyboard input event to obtain the affected element identifier, and record the input content, input length, and changes in the textContent attributes of the elements before and after the input.

[0059] The operation monitoring script also handles keyboard input events. When a doctor performs keyboard input in an element such as the consultation record input box, the script intercepts the target attribute of the keyboard input event and obtains the ID of the affected element. For example, if a doctor enters a consultation opinion in the consultation record input box, the script obtains the ID of the consultation record input box element.

[0060] Next, the script records the specific content entered by the doctor, which includes important information such as the doctor's analysis and diagnosis of the patient's condition. It also records the length of the input, which reflects the level of detail provided by the doctor. Furthermore, the script records changes in the element's textContent attribute before and after input. Before input, the textContent attribute of the consultation record input box may be empty. After input, the doctor's input will be displayed. By comparing the textContent attributes before and after input, you can clearly see the changes in the element's content.

[0061] Step S1223: intercept the target attribute of the scroll bar sliding event to obtain the affected element identifier, and record the scroll direction, scroll distance, and the offsetTop attribute changes of the element before and after scrolling.

[0062] The operation monitoring script also intercepts scroll bar sliding events. When a doctor slides the scroll bar on an element such as the consultation history display area, the script intercepts the target attribute of the scroll bar sliding event and obtains the ID of the affected element. For example, when a doctor slides the scroll bar while viewing consultation history, the script obtains the ID of the element in the consultation history display area.

[0063] The script records the scrolling direction, whether it's up or down, which can reflect the order in which the doctor views content. It also records the scrolling distance, which can indicate the extent of the content the doctor views. Furthermore, the script records changes in the element's offsetTop property before and after scrolling. Before scrolling, the element's offset relative to the top of its parent element is a specific value. After scrolling, this offset changes accordingly. By recording changes in the offsetTop property, we can understand the element's positional movement within the page.

[0064] Step S1224: De-noising is performed on the intercepted user operation events to filter out false touch operations with a duration less than a preset threshold and invalid input operations with unchanged content.

[0065] After the operation behavior capture module obtains user operation events, it needs to perform noise removal on these events. In actual operation, there may be some false touch operations and invalid input operations, which will interfere with subsequent analysis and processing, so filtering is required.

[0066] False touches that last less than a preset threshold, such as a doctor accidentally tapping a blank area on a page, are filtered out. Invalid inputs where the content remains unchanged, such as a doctor entering some information in a consultation record input box and then deleting it, leaving the content unchanged, are also filtered out. Denoising can improve the quality of user action trigger signals, making subsequent analysis more accurate and effective.

[0067] Step S1225: Convert the processed user operation event into a structured data format, wherein the structured data format includes an operation type field, an effect element identification field, an operation timestamp field, and a state comparison field before and after the operation, and generate a user operation trigger signal set.

[0068] After denoising, user operation events are converted into a structured data format. The above structured data format contains several important fields. The operation type field is used to record the specific type of user operation, such as mouse click, keyboard input, scroll bar sliding, etc. The affected element identification field is used to uniquely identify the element being operated. The corresponding element can be accurately found through this affected element identification field. The operation timestamp field records the specific time when the operation occurs, which facilitates the analysis of the time sequence and frequency of the operation. The state comparison field before and after the operation is used to compare the state changes of the element before and after the operation, such as the style change of the element before and after the mouse click, the content change of the element before and after the keyboard input, etc.

[0069] For example, for a mouse click on the edit button in the patient information section, the structured data would record the action type as a mouse click, the action element identifier as the edit button element identifier, the action timestamp as the time the click occurred, and a pre- and post-click state comparison field to record the change in button style before and after the click. By converting the processed user action events into the structured data format described above, a collection of user action trigger signals is generated.

[0070] Step S123: Mount the attribute change observer at the DOM node layer to monitor the element's id attribute change event, class attribute change event, style attribute change event, and textContent attribute change event, and record the changed attribute name, value before change, value after change, and change timestamp as the element attribute change signal.

[0071] An attribute change observer is mounted on the DOM node layer of the remote consultation platform. Its main purpose is to monitor various attribute change events of elements. When an ID attribute change event occurs on an element on the page, the observer records the changed attribute name as id, along with the ID value before and after the change, and the timestamp of the change. For example, the ID attribute of an element in the patient information section may change due to system architecture adjustments, and the observer will record this information in detail.

[0072] For class attribute change events, for example, when a doctor clicks a button, the class attribute of the button element changes from the default style class to the activated style class. The observer will record the name of the changed attribute as class, the style class before and after the change, and the change timestamp. Style attribute change events are also monitored. When a doctor adjusts the font color, size, and other styles of the consultation record input box, the observer will record the name of the changed attribute as style, the style attribute value before and after the change, and the change timestamp. TextContent attribute change events are equally important. When the content of the patient symptom description element is updated, the observer will record the name of the changed attribute as textContent, the text content before and after the change, and the change timestamp. By recording these attribute change events in detail, element attribute change signals are generated. These signals can reflect the dynamic changes in element attributes.

[0073] Step S124: performing timestamp alignment processing on the page loading event signal, the user operation trigger signal, and the element attribute change signal to generate a multi-source interaction signal sequence with a temporal correlation relationship.

[0074] After collecting page load event signals, user action trigger signals, and element attribute change signals, these signals need to be timestamp aligned. Since these signals may be collected at different times, timestamp alignment can be used to arrange them in chronological order, generating a multi-source interaction signal sequence with a temporally correlated relationship.

[0075] First, the timestamps in each signal are extracted and organized. For page loading event signals, the recorded event trigger timestamp is the key information. For user operation trigger signals, the operation timestamp records the time when the user operation occurred. For element attribute change signals, the change timestamp records the time when the attribute changed. Then, these signals are sorted according to the order of timestamps. For example, the page initialization load event occurs first, then the doctor performs a mouse click operation, and then the element attribute changes. Timestamp alignment can accurately reflect the order of these events. The generated multi-source interaction signal sequence with a temporal correlation relationship can more comprehensively display the interactive process of the page.

[0076] Step S125: performing deduplication filtering on the multi-source interaction signal sequence, deleting the same type of event signals that are repeatedly recorded at the same timestamp, retaining valid signal entries with state changes, and generating a structured multi-source interaction signal set.

[0077] Deduplication filtering is performed on multi-source interactive signal sequences with temporal correlations to improve signal quality and validity. In actual acquisition, similar event signals may be recorded repeatedly at the same timestamp. For example, due to temporary system instability, the same mouse click event may be recorded multiple times. Deduplication filtering removes these duplicate signals, retaining only valid signal entries with state changes.

[0078] Signals with actual state changes, such as element attribute change signals, are retained. Signals recorded without actual state changes are deleted. Through deduplication and filtering, a structured multi-source interaction signal set is generated. This multi-source interaction signal set is more accurate and effective.

[0079] Step S130: Analyze the multi-source interaction signal based on the lifecycle tracking model to extract state evolution patterns of elements at different lifecycle stages, where the state evolution patterns include loading trigger conditions, change trigger conditions, and recycle trigger conditions.

[0080] In the medical industry's remote consultation platform, an established lifecycle tracking model is used to conduct in-depth analysis of collected multi-source interaction signals, aiming to extract the state evolution patterns of elements at different lifecycle stages. These state evolution patterns can reveal the changing patterns of elements at different stages.

[0081] Step S131: Extract the page loading event signal corresponding to the initial loading stage from the multi-source interaction signal, count the page loading event type and loading content range associated with the element when it first appears, and generate a set of loading trigger conditions, which includes page initialization loading trigger conditions and local content dynamic loading trigger conditions.

[0082] The page load event signal corresponding to the initial loading phase is extracted from the structured multi-source interaction signal set. For the elements on the consultation page of the remote consultation platform, the page load event type and loaded content range associated with them when they first appear need to be counted.

[0083] For example, the patient's basic information element first appears when the page is initially loaded. Its associated page load event type is the page initialization load event, and the loaded content covers various fields of basic information such as the patient's name, age, and gender. The patient's latest examination report element may first appear when the doctor clicks the View Report button, triggering a local content dynamic load event. Its associated load event type is the local content dynamic load event, and the loaded content covers the detailed data of the latest examination report.

[0084] Through the above statistical analysis of a large number of elements, a set of loading trigger conditions is generated. This set of loading trigger conditions includes the page initial loading trigger condition and the local content dynamic loading trigger condition. When the page initial loading trigger condition is met, the element enters the initial loading phase during the page initial loading; when the local content dynamic loading trigger condition is met, the element enters the initial loading phase during the local content dynamic loading.

[0085] Step S132: Extract the user operation trigger signal and element attribute change signal corresponding to the state change stage, analyze the correlation between the user operation type and the element attribute change, count the attribute change frequency and change attribute combination of the element under different operation types, and generate a change trigger condition set, which includes the style change condition triggered by the click operation, the content change condition triggered by the input operation, and the position change condition triggered by the scrolling operation.

[0086] After extracting the user operation trigger signal and element attribute change signal corresponding to the state change stage, it is necessary to analyze the correlation between the user operation type and the element attribute change. In the remote consultation platform, different user operation types will lead to different element attribute changes.

[0087] For example, when a doctor inputs a consultation record, the textContent property of the input box changes. By counting the number of input operations and the frequency of textContent property changes, we can understand the relationship between input operations and content changes. For the edit button element in the patient information section, when a doctor clicks it, the button's class property may change. By analyzing the frequency of click operations and class property changes, we can summarize the relationship between click operations and style changes.

[0088] When doctors scroll elements in the image display area, the element's position-related attributes (such as scrollTop) change. By counting scrolling operations and the frequency of attribute changes, we can derive the correlation between scrolling operations and position changes. By counting and analyzing attribute changes for a large number of elements under different operation types, we generate a set of change trigger conditions. This set of change trigger conditions includes style change conditions triggered by click operations, content change conditions triggered by input operations, and position change conditions triggered by scrolling operations.

[0089] Step S133: Extract the page loading event signal and element attribute change signal corresponding to the invalid recovery stage, count the attribute stability time of the element during the period without user operation, and the impact of the page refresh event on the existence of the element, and generate a recovery trigger condition set, which includes the natural recovery condition triggered by the stable time without operation and the forced recovery condition triggered by the page refresh.

[0090] After extracting the page loading event signal and element attribute change signal corresponding to the invalidation recovery phase, it is necessary to count the attribute stability duration of the element during the period without user operation and the impact of the page refresh event on the element existence.

[0091] In a remote consultation platform, after a patient's consultation concludes, some elements may enter the invalidation and recycling phase. For example, if a patient's historical consultation record element has not been viewed by a doctor for a period of time, its attributes remain stable. By measuring the duration of the element's stable attributes during this period of no user activity, we can determine the natural recycling condition that triggers the stable inactivity period. When this period reaches a certain threshold, the element will naturally enter the invalidation and recycling phase.

[0092] Page refresh events also affect the existence of elements. When a doctor refreshes the page, some expired or no longer needed elements may be forcibly removed. For example, a patient's expired examination report element may no longer be displayed after a page refresh. By analyzing the relationship between page refresh events and element removal, a forced recycling condition triggered by a page refresh is generated. Combining these statistics and analyses, a recycling trigger condition set is generated. This recycling trigger condition set includes natural recycling conditions triggered by a stable period of no operation and forced recycling conditions triggered by a page refresh.

[0093] Step S134: performing pattern induction processing on the loading trigger condition set, the changing trigger condition set, and the recycling trigger condition set, and extracting common trigger conditions of different element types in the same interaction scenario as a general evolution pattern.

[0094] The purpose of pattern induction on the generated load trigger condition set, change trigger condition set, and recycle trigger condition set is to extract common evolution patterns. In the remote consultation platform, different types of elements may have common trigger conditions in the same interaction scenario.

[0095] First, the trigger condition set is categorized by element type. Element types include patient basic information elements, consultation record elements, and image display elements. Frequency statistics are then collected for trigger conditions within the same element type. For example, for the patient basic information element, the frequency of trigger conditions appearing in different interaction scenarios is counted. Trigger conditions whose frequency exceeds a preset threshold are selected as candidate common conditions.

[0096] Then, analyze the applicability of candidate common conditions in different interaction scenarios. For example, for the style change conditions triggered by click operations, analyze their applicability in the patient basic information elements and consultation record elements. Retain the trigger conditions that are valid in at least two different interaction scenarios as the basic conditions of the general evolution pattern. Perform logical merging on the basic conditions, and merge the conditions with the same trigger logic into compound trigger conditions. For example, merge the style change conditions triggered by clicking the patient basic information modification button and clicking the consultation record edit button into a compound trigger condition. Perform scenario generalization on the merged compound trigger conditions, remove the restrictive conditions that are strongly related to specific scenarios, and generate a set of general evolution patterns applicable to multiple scenarios.

[0097] Step S135: extracting individual trigger conditions of the elements in the target business scenario as customized evolution patterns, and constructing an element state evolution pattern set based on the general evolution pattern and the customized evolution pattern.

[0098] In the target business scenarios of the remote consultation platform, different elements may have unique individual trigger conditions. For example, a patient's genetic test report element may have a specific trigger condition, such as automatically updating the report content at a specific time point. This is the individual trigger condition of this element.

[0099] Through detailed analysis of each element in the target business scenario, these individual trigger conditions are extracted as customized evolution patterns. Based on the previously generated general and customized evolution patterns, a set of element state evolution patterns is constructed. This set includes both general and customized state evolution patterns for different elements at different lifecycle stages, providing a more comprehensive description of element state changes. General evolution patterns are applicable to most elements and scenarios, while customized evolution patterns address the unique needs of specific elements. The combination of these two makes the set of element state evolution patterns more comprehensive and accurate.

[0100] Step S140: generating a dynamic positioning rule set according to the state evolution pattern, wherein the dynamic positioning rule set includes a basic positioning rule, a change response positioning rule, and a recovery compensation positioning rule.

[0101] In a remote medical consultation platform, a dynamic positioning rule set is generated based on the previously extracted element state evolution patterns to accurately locate elements at different lifecycle stages. These rules can adapt to the dynamic changes of elements and improve the accuracy and efficiency of web element detection.

[0102] Step S141: generating basic positioning rules for the loading triggering conditions in the initial loading phase, wherein the basic positioning rules include static positioning rules based on the DOM node hierarchy path and visual positioning rules based on the initial visible area of ​​the element.

[0103] Generate basic positioning rules based on the loading trigger conditions of the initial loading phase. The static positioning rule based on the DOM node hierarchical path locates elements by using the hierarchical relationship of elements in the DOM tree. In the consultation page of the remote consultation platform, each element has its own specific hierarchical path in the DOM tree. For example, the hierarchical path of the patient's basic information element in the DOM tree can be used as the basis for static positioning. By determining the hierarchical relationship of the element in the DOM tree, such as the parent node and child node, the element can be accurately found.

[0104] Visual positioning rules based on an element's initial visible area consider information such as the element's initial display position and size on the page. For example, the initial display area of ​​a patient symptom description element on the page, including information such as its top-left corner coordinates, width, and height, can be used for visual positioning. When an element is initially loaded, these two basic positioning rules allow for accurate positioning from different angles.

[0105] Step S142: Generate change response positioning rules for the change trigger conditions in the state change phase. The change response positioning rules include dynamic attribute matching rules when attributes change, relative position offset compensation rules when positions change, and semantic content matching rules when contents change.

[0106] Generate change response location rules based on the change trigger conditions during the state change phase. Dynamic attribute matching rules for attribute changes are used to locate elements when their attributes change. For example, when the textContent attribute of a consultation record input box element changes, the element is located by matching the changed textContent attribute. If the doctor enters a new consultation opinion, the input box element can be accurately found based on the new content.

[0107] The relative offset compensation rule for position changes applies to situations where the position of an element changes. In a remote consultation platform, when a doctor zooms or pans the image display area, the position of elements in the image display area will change. By calculating the relative offset of the element before and after the operation, such as the horizontal and vertical movement distance of the element, the new position of the element can be accurately determined.

[0108] Semantic content matching rules for content changes are used to locate elements when their content changes. For example, when the content of a patient symptom description element is updated, the element is located by matching the new semantic content of the symptom description. If the symptom description changes from "cough" to "cough with fever," the symptom description element can be accurately found based on the new semantic content.

[0109] Step S143: generating a recovery compensation positioning rule for the recovery triggering condition in the invalid recovery phase, wherein the recovery compensation positioning rule includes a historical state backtracking rule during natural recovery and a page refresh relocation rule during forced recovery.

[0110] Recycling compensation positioning rules are generated based on the recycling trigger conditions during the invalid recycling phase. The historical state backtracking rules for natural recycling are used to locate elements when they enter the invalid recycling phase due to natural reasons. For example, if a patient's old medical record element naturally enters the invalid recycling phase after a period of inactivity and stable attributes, it can be located by backtracking the element's historical state. The historical state includes information such as the element's hierarchical path in the DOM tree and attribute values. Using this historical information, it is possible to find the element again.

[0111] The page refresh relocation rules during forced recycling apply to situations where elements enter the invalidation recycling phase due to forced reasons such as page refreshes. When a doctor refreshes the page, some elements may be forcibly removed or reloaded. Elements can be relocated based on the new DOM tree structure after the page refresh and the initial characteristics of the elements. For example, after a page refresh, the patient's basic information element can be found again based on the refreshed DOM tree and the element's initial content.

[0112] Step S144: configuring a priority weight for each dynamic positioning rule. The priority weight is dynamically adjusted according to the success rate of the dynamic positioning rule in historical picking tasks. The initial priority weight is set based on the number of trigger conditions covered by the rule.

[0113] Configuring a priority weight for each dynamic positioning rule allows for more effective selection of different rules during the actual element picking process. First, the number of successful picks and total number of attempts for each dynamic positioning rule in historical picking tasks is counted to calculate the rule's historical success rate. This historical success rate reflects the accuracy of the rule's past element locating performance.

[0114] Configuring a time decay factor for historical success rates gives more weight to the success rate of recent pickup tasks than the success rate of more recent tasks. This is because recent tasks better reflect the current performance of the rule. For example, recent tasks may use new page layouts or element styles, and the rule may behave differently than in more recent tasks. Using the time decay factor to generate a weighted historical success rate better reflects the real-time performance of the rule.

[0115] The initial priority weight is set to the product of the number of trigger conditions covered by the dynamic positioning rule and the preset weight coefficient. The preset weight coefficient is set to different values ​​depending on the type of dynamic positioning rule. For example, static positioning rules based on DOM node hierarchical paths may have a higher preset weight coefficient due to their higher stability; while visual positioning rules based on the initial visible area of ​​the element may be more affected by changes in the page layout and have a relatively lower preset weight coefficient.

[0116] Establish a priority weight adjustment mechanism to adjust the initial priority weight positively or negatively based on the weighted historical success rate. If the weighted historical success rate is high, it indicates that the rule has performed well recently, and the initial priority weight will be adjusted positively to increase its priority. If the weighted historical success rate is low, it indicates that the rule has performed poorly recently, and the initial priority weight will be adjusted negatively to decrease its priority. Regularly update the priority weight based on the latest historical pickup data to ensure that the priority weight reflects the actual performance of the rule in a timely manner.

[0117] Step S145: Establish a mapping relationship table between dynamic positioning rules and life cycle stages. The mapping relationship table records the rule type, rule content, and priority weight corresponding to each life cycle stage, and constructs a dynamic positioning rule set based on rule generation and mapping processing.

[0118] The purpose of establishing a mapping table between dynamic positioning rules and lifecycle phases is to clearly record the dynamic positioning rules corresponding to each lifecycle phase. The mapping table records the rule type corresponding to each lifecycle phase, such as the basic positioning rule for the initial loading phase, the change response positioning rule for the state change phase, and the recovery compensation positioning rule for the failure recovery phase.

[0119] At the same time, the specific content of the rules is recorded, such as the specific hierarchical path information of the static positioning rule based on the DOM node hierarchical path, the matching logic of the dynamic attribute matching rule when the attribute changes, etc. The priority weight of each rule is also recorded, and the record is made according to the priority weight configured previously. Through rule generation and mapping processing, a dynamic positioning rule set is constructed. This dynamic positioning rule set not only includes various dynamic positioning rules, but also clarifies the correspondence between the rules and life cycle stages through a mapping relationship table, so that appropriate rules can be accurately selected for element positioning at different life cycle stages.

[0120] Step S1441: Count the number of successful picks and the total number of attempts for each dynamic positioning rule in the historical pick tasks, and calculate the historical success rate of the rule.

[0121] In the actual use of the remote consultation platform, there will be a large number of historical picking tasks. For each dynamic positioning rule, it is necessary to count the number of successful picking and the total number of attempts in these historical tasks. For example, if a static positioning rule based on the DOM node hierarchy path successfully locates an element 80 times in the past 100 element picking tasks, then the number of successful picking is 80 and the total number of attempts is 100.

[0122] The rule's historical success rate is calculated by dividing the number of successful pickups by the total number of attempts. In the example above, the rule's historical success rate is 80%. The historical success rate reflects how well the rule has performed in the past.

[0123] Step S1442: configuring a time decay factor for the historical success rate so that the success rate weight of the recent picking task is higher than the success rate weight of the long-term task, and generating a weighted historical success rate.

[0124] To more accurately reflect the current performance of the rule, configure a time decay factor for the historical success rate. This factor weights the success rate of recent pickups more heavily when calculating the weighted historical success rate. For example, the success rate of recent pickups will have a greater weight in the weighted calculation, while the success rate of more distant pickups will have a smaller weight.

[0125] You can configure the time decay factor using exponential decay. Assuming the time decay factor is a constant less than 1, the success rate of more distant tasks will gradually decrease over time. This approach generates a weighted historical success rate that better reflects the actual performance of the rule in the near term, preventing the results of distant tasks from significantly influencing the rule evaluation.

[0126] Step S1443: setting the initial priority weight to the product of the number of trigger conditions covered by the dynamic positioning rule and a preset weight coefficient, wherein the preset weight coefficient is set to different values ​​according to the type of the dynamic positioning rule.

[0127] The initial priority weight is set to the product of the number of trigger conditions covered by the dynamic targeting rule and the preset weight coefficient. The preset weight coefficient varies depending on the type of dynamic targeting rule. Different types of dynamic targeting rules have different characteristics and applicable scenarios, so different preset weight coefficients are required based on their type.

[0128] For example, static positioning rules based on DOM node hierarchical paths are able to accurately locate elements in many cases due to their high stability, and the preset weight coefficient can be set higher. However, visual positioning rules based on the initial visible area of ​​the element may be greatly affected by factors such as changes in page layout and element display style, and the preset weight coefficient is relatively low. For dynamic attribute matching rules when attributes change, their accuracy depends on the frequency and complexity of element attribute changes, and the preset weight coefficient can be reasonably set according to actual conditions. The initial priority weight is obtained by multiplying the number of trigger conditions covered by the rule by the corresponding preset weight coefficient.

[0129] Step S1444: establishing a priority weight adjustment mechanism, wherein the priority weight adjustment mechanism adjusts the initial priority weight positively or negatively according to the weighted historical success rate.

[0130] The priority weight adjustment mechanism is designed to dynamically adjust the priority weight of a rule based on its actual performance. The initial priority weight is adjusted positively or negatively based on the weighted historical success rate. If the weighted historical success rate is high, indicating that the rule has performed well recently, the initial priority weight is adjusted positively to increase its priority. For example, if a static positioning rule based on a DOM node hierarchy path has a high weighted historical success rate, its priority weight is appropriately increased, making it a more preferred choice during element picking.

[0131] If the weighted historical success rate is low, indicating that the rule has performed poorly recently, the initial priority weight will be adjusted negatively to lower its priority. For example, if the weighted historical success rate of a visual positioning rule based on the initial visible area of ​​an element is low, its priority weight will be appropriately lowered to reduce the likelihood of using this rule during element picking. This adjustment mechanism ensures that the priority weight can promptly reflect the actual performance of the rule.

[0132] Step S1445: regularly updating the priority weights according to the latest historical pickup data to complete the priority weight configuration of the dynamic positioning rule.

[0133] Priority weights are regularly updated based on the latest historical pickup data to ensure accuracy and timeliness. As time passes and more pickup tasks are completed, historical pickup data is continuously updated. Regularly updating priority weights ensures that the priorities of dynamic positioning rules reflect their actual performance.

[0134] For example, historical pick data is collected and analyzed monthly, and priority weights are updated based on the new statistical results. During this update, the weighted historical success rate of each rule is recalculated, and the initial priority weights are adjusted according to the priority weight adjustment mechanism. After the priority weight update is completed, the priority weight configuration of dynamic positioning rules is more reasonable and accurate, which can better guide element picking operations.

[0135] Step S150: When executing the web page element picking operation, the current life cycle stage of the element is determined by the life cycle tracking model, the dynamic positioning rules of the corresponding stage are called to complete the picking, and the picking result is fed back to the life cycle tracking model to optimize the state evolution mode.

[0136] In the remote consultation platform of the medical industry, when performing the web page element picking operation, it is necessary to use the lifecycle tracking model to determine the current lifecycle stage of the element, then call the corresponding dynamic positioning rules to pick the element, and finally feed the picking results back to the lifecycle tracking model to optimize the state evolution pattern.

[0137] Step S151: When the picking operation starts, the current DOM node state and the most recent interaction record of the target element are obtained and input into the stage determination module of the life cycle tracking model.

[0138] When the picking operation begins, the current DOM node state and recent interaction history of the target element must be retrieved. This information includes information such as the element's hierarchical relationship in the DOM tree and attribute values. For example, for a patient diagnosis element, its parent and child nodes in the DOM tree, as well as its current textContent and style attribute values, must be retrieved.

[0139] Recent interaction records record the most recent interaction between a doctor or other user and the element, including the time, type (e.g., click, input), and specific content. For example, the time and text of the doctor's most recent input into the patient's diagnosis element. This information is fed into the stage determination module of the lifecycle tracking model, which uses it to determine the element's current lifecycle stage.

[0140] Step S152: Verify the status monitoring indicator of the stage determination module to determine whether the element is currently in the initial loading stage, the status change stage or the failure recovery stage.

[0141] The phase determination module uses the previously configured status monitoring indicators for verification. For the initial loading phase, it checks whether the element meets the DOM node existence criterion, that is, whether the element exists in the DOM tree and its attributes have not changed. For example, if the patient's family medical history element exists in the DOM tree and its attributes (such as textContent) have not changed, it is determined to be in the initial loading phase.

[0142] For the state change phase, check whether the element meets the attribute change frequency index, that is, whether the attribute of the element has changed within a certain period of time. For example, if the textContent attribute of the consultation record input box element has changed multiple times recently, it is determined to be in the state change phase. For the invalidation recovery phase, check whether the element meets the node removal mark index, that is, whether the element has been removed from the DOM tree. For example, if the patient's expired examination report element has been removed from the DOM tree, it is determined to be in the invalidation recovery phase. Through the verification of the status monitoring indicators, it is accurately determined whether the element is currently in the initial loading phase, the state change phase or the invalidation recovery phase.

[0143] Step S153: According to the mapping relationship table between the dynamic positioning rule set and the life cycle stage, the dynamic positioning rule of the corresponding stage is called to perform the picking operation.

[0144] According to the mapping relationship table between the dynamic positioning rule set and the life cycle stage, when it is determined that the element is in a certain life cycle stage, the dynamic positioning rule corresponding to the stage is called to pick up the element. If the element is in the initial loading stage, the basic positioning rules are called, such as the static positioning rules based on the DOM node hierarchical path or the visual positioning rules based on the initial visible area of ​​the element to locate the element. For example, for the patient basic information element, if it is in the initial loading stage, the element can be accurately found through its hierarchical path in the DOM tree.

[0145] If the element is in the state change stage, call the change response positioning rules, such as the dynamic attribute matching rules when the attributes change, the relative position offset compensation rules when the position changes, and the semantic content matching rules when the content changes, to locate the element. For example, when the textContent attribute of the consultation record input box element changes, the element is located by matching the changed textContent attribute. If the element is in the invalidation recovery stage, call the recovery compensation positioning rules, such as the historical state backtracking rules during natural recovery and the page refresh repositioning rules during forced recovery, to locate the element. For example, for the patient's old medical record element, if it is in the invalidation recovery stage, you can try to reposition the element by backtracking its historical state.

[0146] Step S154: recording the success or failure of the picking operation, the positioning rules used, and the real-time state changes of the elements during the picking process as picking result data.

[0147] After performing a pick operation, you need to record relevant information about the pick operation as the pick result data. This records whether the pick operation was successful, that is, whether the target element was accurately located. If the element was successfully located, the location rule used is recorded, such as whether a static location rule based on the DOM node hierarchy path or a dynamic attribute matching rule for attribute changes was used.

[0148] At the same time, the real-time state changes of elements during the picking process are recorded. For example, when using the relative position offset compensation rule to locate elements in the image display area during position change, the element's position movement during the positioning process is recorded, including the direction and distance of movement. If the element's attributes change during the picking process, such as the textContent attribute of the consultation record input box being modified during the positioning process, these attribute changes must also be recorded. By recording this information, detailed picking result data is generated.

[0149] Step S155: Input the picking result data into the pattern optimization module of the lifecycle tracking model. By comparing the actual picking result with the predicted state evolution pattern, the trigger condition probability distribution and stage transition condition threshold in the state evolution pattern are adjusted to complete the optimization update of the lifecycle tracking model.

[0150] The pickup results are fed into the lifecycle tracking model's pattern optimization module. This module compares the actual pickup results with the predicted state evolution pattern. If there are discrepancies between the actual pickup results and the predicted state evolution pattern, the state evolution pattern may need adjustment.

[0151] For example, if the frequency of element attribute changes during actual pickup differs from the predicted frequency during the state change phase, the probability distribution of the change trigger conditions in the state evolution model is adjusted. If the element's attributes change more frequently in reality, the probability of the corresponding change trigger conditions is increased. Furthermore, the thresholds for phase transition conditions are adjusted based on actual conditions. For example, if an element enters the invalidation and recycling phase earlier than predicted from the state change phase, the preset threshold for the no-interaction duration trigger condition is appropriately lowered. These adjustments enable the optimization and update of the lifecycle tracking model, enabling it to more accurately reflect the state change patterns of the elements.

[0152] Figure 2 This diagram illustrates exemplary hardware and software components of an RPA-based dynamic webpage element picking system 100, which can implement the concepts of this application, as provided in some embodiments of this application. For example, a processor 120 can be used in the RPA-based dynamic webpage element picking system 100 to perform the functions described in this application.

[0153] The RPA-based web page dynamic element picking system 100 can be a general-purpose server or a special-purpose server, both of which can be used to implement the RPA-based web page dynamic element picking method of this application. Although this application only shows a single server, for convenience, the functions described in this application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0154] For example, the RPA-based web page dynamic element picking system 100 may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and different forms of storage media 140, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the RPA-based web page dynamic element picking system 100 may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application can be implemented according to these program instructions. The RPA-based web page dynamic element picking system 100 also includes an I / O interface 150 between the computer and other input and output devices.

[0155] For ease of explanation, only one processor is described in the RPA-based web page dynamic element picking system 100. However, it should be noted that the RPA-based web page dynamic element picking system 100 in the present application may also include multiple processors, so the steps performed by one processor described in the present application may also be performed jointly or individually by multiple processors. For example, if the processor of the RPA-based web page dynamic element picking system 100 executes step A and step B, it should be understood that step A and step B may also be executed jointly by two different processors or individually in one processor. For example, the first processor executes step A, the second processor executes step B, or the first processor and the second processor execute steps A and B together.

[0156] In addition, an embodiment of the present invention further provides a readable storage medium, in which computer-executable instructions are preset. When the processor executes the computer-executable instructions, the above-mentioned RPA-based dynamic element picking method for web pages is implemented.

[0157] It should be noted that in order to simplify the description of the present invention and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, multiple features are sometimes combined into one embodiment, figure or description thereof.

Claims

1. A method for picking dynamic elements of a web page based on RPA, characterized in that: The method comprises: Establishing an element lifecycle tracking model, wherein the lifecycle tracking model includes definitions of the initial loading phase, state change phase, and failure recovery phase of the element during the interaction process, as well as a set of state transition conditions between phases; Collecting multi-source interaction signals of the target web page, wherein the multi-source interaction signals include page loading event signals, user operation trigger signals, and element attribute change signals; Analyze the multi-source interaction signals based on the lifecycle tracking model to extract state evolution patterns of elements at different lifecycle stages, wherein the state evolution patterns include loading trigger conditions, change trigger conditions, and recycle trigger conditions; Generate a dynamic positioning rule set according to the state evolution pattern, wherein the dynamic positioning rule set includes a basic positioning rule, a change response positioning rule, and a recovery compensation positioning rule; When performing a web page element picking operation, the current life cycle stage of the element is determined by the life cycle tracking model, the dynamic positioning rule of the corresponding stage is called to complete the picking, and the picking result is fed back to the life cycle tracking model to optimize the state evolution mode; Generating a dynamic positioning rule set according to the state evolution pattern includes: Generate basic positioning rules for the loading trigger conditions in the initial loading phase, including static positioning rules based on the DOM node hierarchy path and visual positioning rules based on the initial visible area of ​​the element; Generate change response positioning rules for the change trigger conditions in the state change phase. The change response positioning rules include dynamic attribute matching rules when attributes are changed, relative position offset compensation rules when positions are changed, and semantic content matching rules when contents are changed. Generate recovery compensation positioning rules based on the recovery trigger conditions in the invalid recovery phase. The recovery compensation positioning rules include historical state backtracking rules for natural recovery and page refresh relocation rules for forced recovery. Configure a priority weight for each dynamic positioning rule. The priority weight is dynamically adjusted based on the success rate of the dynamic positioning rule in historical picking tasks. The initial priority weight is set based on the number of trigger conditions covered by the rule. Establish a mapping relationship table between dynamic positioning rules and life cycle stages, the mapping relationship table records the rule type, rule content, and priority weight corresponding to each life cycle stage, and constructs a dynamic positioning rule set based on rule generation and mapping processing; Configuring a priority weight for each dynamic positioning rule includes: Count the number of successful picks and the total number of attempts for each dynamic positioning rule in the historical picking tasks, and calculate the historical success rate of the rule; Configure a time decay factor for the historical success rate so that the success rate weight of recent pickup tasks is higher than the success rate weight of distant tasks, generating a weighted historical success rate; Setting the initial priority weight to the product of the number of trigger conditions covered by the dynamic positioning rule and a preset weight coefficient, wherein the preset weight coefficient is set to different values ​​according to the type of dynamic positioning rule; Establishing a priority weight adjustment mechanism that adjusts the initial priority weight positively or negatively based on the weighted historical success rate; Regularly update the priority weights based on the latest historical pickup data to complete the priority weight configuration of dynamic positioning rules.

2. The method for picking dynamic elements of a web page based on RPA according to claim 1, characterized in that: The life cycle tracking model of the element is established, including: The initial loading phase is defined as the time interval when the element first appears in the page DOM tree, and the end condition of the phase is when the element triggers the first attribute change event; The state change phase is defined as the time interval from the first attribute change to the last valid interaction of an element. The end condition of the phase is that the element has not been accessed continuously and the attribute has not changed for a period of time that reaches a preset threshold. The invalidation recovery phase is defined as the time interval from the end of the state change phase to the removal of the element from the DOM tree. The end condition of the phase is that the element node is deleted from the DOM tree. Establishing a state transition condition set between stages, the state transition condition set includes a property change trigger condition from the initial loading stage to the state change stage, a no-interaction duration trigger condition from the state change stage to the invalidation recovery stage, and a page refresh and reload trigger condition from the invalidation recovery stage to the initial loading stage; Status monitoring indicators are configured for each life cycle stage. The status monitoring indicators include the DOM node existence indicator in the initial loading stage, the attribute change frequency indicator in the state change stage, and the node removal mark indicator in the failure recovery stage. An element life cycle tracking model is constructed based on stage definitions, transfer conditions and monitoring indicators.

3. The method for picking dynamic elements of a web page based on RPA according to claim 2, characterized in that: The establishment of a set of inter-stage state transition conditions includes: The trigger condition for attribute changes from the initial loading phase to the state change phase is defined as at least one id attribute, class attribute, or textContent attribute change event occurring during the initial loading phase. The trigger condition for the no-interaction duration from the state change phase to the invalidation recovery phase is defined as the time during which the element is not triggered by user operations and no attribute change events occur during the state change phase, exceeding the preset threshold; The triggering condition for page refresh and reload from the invalid recovery phase to the initial loading phase is defined as the page triggering a refresh event and the appearance of a new node with the same tag type and initial content as the historical element in the refreshed DOM tree; Configure verification logic for each state transition condition, including event type verification for attribute change trigger conditions, timestamp difference verification for no-interaction duration trigger conditions, and node attribute matching verification for page refresh and reload trigger conditions; Establish a trigger priority order for state transition conditions, where the attribute change trigger condition takes precedence over the no-interaction duration trigger condition, and the page refresh and reload trigger condition takes precedence over the trigger conditions in other stages. Build a state transition condition set based on condition definition, verification logic, and priority order.

4. The method for picking dynamic elements of a web page based on RPA according to claim 1, characterized in that: The collecting of multi-source interaction signals of the target webpage includes: Register a page loading event listener in the browser kernel layer to capture page initialization loading events, local content dynamic loading events, and page refresh events, and record the event type, trigger timestamp, and loaded content range as page loading event signals; Deploy an operation behavior capture module in the user interaction layer to monitor mouse click events, keyboard input events, and scroll bar sliding events, and record the operation type, element identifier, operation timestamp, and page status before and after the operation as user operation trigger signals; Mount attribute change observers at the DOM node layer to monitor element id attribute change events, class attribute change events, style attribute change events, and textContent attribute change events, and record the changed attribute name, value before change, value after change, and change timestamp as element attribute change signals; Perform timestamp alignment processing on page loading event signals, user operation trigger signals, and element attribute change signals to generate a multi-source interaction signal sequence with a temporal correlation relationship; The multi-source interaction signal sequence is deduplicated and filtered, the same event signals recorded repeatedly at the same timestamp are deleted, the valid signal entries with state changes are retained, and a structured multi-source interaction signal set is generated.

5. The method for picking dynamic elements of a web page based on RPA according to claim 4 is characterized in that: The operation behavior capture module is deployed in the user interaction layer to monitor mouse click events, keyboard input events, and scroll bar sliding events, and record the operation type, the element identifier, the operation timestamp, and the page status before and after the operation as the user operation trigger signal, including: Inject an operation monitoring script into the event processing layer of the browser page, intercept the target attribute of the mouse click event to obtain the affected element ID, and record the click coordinates, number of clicks, and changes in the class attribute of the element before and after the click; Intercept the target attribute of the keyboard input event to obtain the affected element ID, record the input content, input length, and changes in the textContent attribute of the elements before and after the input; Intercept the target attribute of the scroll bar sliding event to obtain the affected element ID, record the scroll direction, scroll distance, and the offsetTop attribute changes of the element before and after scrolling; De-noising is performed on intercepted user operation events to filter out false touch operations with a duration less than a preset threshold and invalid input operations with unchanged content. The processed user operation event is converted into a structured data format, wherein the structured data format includes an operation type field, an effect element identification field, an operation timestamp field, and a state comparison field before and after the operation, and a user operation trigger signal set is generated.

6. The method for picking dynamic elements of a web page based on RPA according to claim 1, characterized in that: Analyzing the multi-source interaction signals based on the lifecycle tracking model to extract state evolution patterns of elements at different lifecycle stages includes: Extracting the page loading event signal corresponding to the initial loading phase from the multi-source interaction signal, counting the page loading event type and loading content range associated with the first appearance of the element, and generating a set of loading trigger conditions, the loading trigger condition set including the page initialization loading trigger condition and the local content dynamic loading trigger condition; Extract user operation trigger signals and element attribute change signals corresponding to the state change stage, analyze the correlation between user operation types and element attribute changes, count the attribute change frequency and attribute change combinations of elements under different operation types, and generate a set of change trigger conditions. The change trigger condition set includes a style change condition triggered by a click operation, a content change condition triggered by an input operation, and a position change condition triggered by a scroll operation. Extract the page loading event signal and element attribute change signal corresponding to the invalid recovery phase, count the attribute stability duration of the element during the period of no user operation and the impact of the page refresh event on the element existence, and generate a recovery trigger condition set. The recovery trigger condition set includes a natural recovery condition triggered by the stable duration of no operation and a forced recovery condition triggered by a page refresh; Perform pattern induction on the loading trigger condition set, change trigger condition set, and recycling trigger condition set, extracting the common trigger conditions of different element types in the same interaction scenario as the general evolution pattern; The individual trigger conditions of the elements in the target business scenario are extracted as customized evolution patterns, and a set of element state evolution patterns is constructed based on the general evolution pattern and the customized evolution pattern.

7. The method for picking dynamic elements of a web page based on RPA according to claim 6, characterized in that: The loading trigger condition set, the changing trigger condition set, and the recycling trigger condition set are subjected to pattern induction processing to extract common trigger conditions of different element types in the same interaction scenario as a general evolution pattern, including: Classifying the trigger condition set by element type, wherein the element type includes button elements, input box elements, and list item elements; Perform frequency statistics on trigger conditions under the same element type and extract trigger conditions with a frequency exceeding a preset threshold as candidate common conditions; Analyze the applicability of candidate common conditions in different interaction scenarios, and retain the trigger conditions that are valid in at least two different interaction scenarios as the basic conditions of the general evolution model; Perform logical merging on basic conditions, merging conditions with the same trigger logic into composite trigger conditions; The merged composite trigger conditions are generalized in scenarios to remove the restrictive conditions that are strongly related to specific scenarios, and generate a set of general evolution patterns applicable to multiple scenarios.

8. The method for picking dynamic elements of a web page based on RPA according to claim 1, characterized in that: When executing the web page element picking operation, the current life cycle stage of the element is determined by the life cycle tracking model, the dynamic positioning rule of the corresponding stage is called to complete the picking, and the picking result is fed back to the life cycle tracking model to optimize the state evolution mode, including: When the picking operation starts, the current DOM node state and recent interaction records of the target element are obtained and input into the stage determination module of the lifecycle tracking model; Verify the status monitoring indicators of the stage determination module to determine whether the element is currently in the initial loading stage, state change stage or failure recovery stage; According to the mapping relationship table between the dynamic positioning rule set and the life cycle stage, the dynamic positioning rule of the corresponding stage is called to perform the picking operation; Record the success or failure of the picking operation, the positioning rules used, and the real-time status changes of the elements during the picking process as picking result data; The picking result data is input into the pattern optimization module of the lifecycle tracking model. By comparing the actual picking results with the predicted state evolution pattern, the trigger condition probability distribution and stage transfer condition threshold in the state evolution pattern are adjusted to complete the optimization and update of the lifecycle tracking model.

9. A web page dynamic element picking system based on RPA, characterized in that: It includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the RPA-based web page dynamic element picking method described in any one of claims 1 to 8.

Images