Page processing methods, devices, electronic devices, storage media, and products

Abstract

This invention discloses a page processing method, apparatus, electronic device, storage medium, and product. The method includes: responding to a page interaction operation performed on a browser page; identifying at least one page interaction operation event triggered on the browser page through a pre-configured browser plugin and determining the voice content associated with the at least one page interaction operation event, wherein the voice content associated with the page interaction operation event is voice material related to the page interaction operation formed by synchronously providing voice explanation of the page interaction operation performed on the browser page; determining the DOM element positioning information corresponding to the at least one page interaction operation event; generating a target document based on the voice content associated with the page interaction operation event; and generating a target script based on the DOM element positioning information corresponding to the at least one page interaction operation event and the triggering order of the at least one page interaction operation event. This application solves the problem of low efficiency due to repetitive configuration during browser page configuration.

Description

Technical Field

[0001] This invention relates to the field of computer software technology, and in particular to a page processing method, apparatus, electronic device, storage medium, and product. Background Technology

[0002] As web systems become increasingly prevalent and crucial in enterprise business processes, browser page configuration often involves repetitive execution of the same page operations to configure information for multiple entities, such as document processing, data entry, and batch form submission. In these scenarios, multiple operations, including text input, file upload, and process confirmation, must be completed step-by-step on a browser page following a fixed workflow. Current solutions primarily rely on manual, sequential browser page configuration. When the number of browser page configurations is large, this not only consumes significant time and manpower but also easily leads to data entry errors and process omissions due to repetitive operations, reducing business processing efficiency and accuracy. Furthermore, the monotony of repetitive operations exacerbates operator fatigue, further impacting work quality and making it difficult to meet the demands of large-scale, high-efficiency page configuration. Summary of the Invention

[0003] This invention provides a page processing method, apparatus, electronic device, storage medium, and product to solve the problems of low efficiency and error-proneness caused by repetitive configuration during browser page configuration.

[0004] According to one aspect of the present invention, a page processing method is provided, the method comprising: In response to page interaction operations performed on a browser page, at least one page interaction operation event triggered on the browser page is identified by a pre-configured browser plugin, and the voice content associated with the at least one page interaction operation event is determined. The voice content associated with the page interaction operation event is voice material related to the page interaction operation formed by synchronously explaining the page interaction operation performed on the browser page. The event type of the at least one page interaction operation event is related to the page configuration task type associated with the browser page. Align the at least one page interaction event with the audio content associated with the at least one page interaction event according to the timestamp, and determine the DOM element positioning information corresponding to the at least one page interaction event; A target document is generated based on the voice content associated with the page interaction operation event. The target document is formed by extracting the operation steps, operation intentions, operation step descriptions, context descriptions, and business logic meanings corresponding to the page interaction operation from the voice content associated with the page interaction operation event. A target script is generated based on the DOM element location information corresponding to the at least one page interaction event and the triggering order of the at least one page interaction event. The target script is used to reproduce the page interaction operation related to the page configuration task type that needs to be executed on the browser page using the DOM element location information.

[0005] According to another aspect of the present invention, a page processing apparatus is provided, the apparatus comprising: The acquisition module is used to respond to page interaction operations performed on the browser page. It identifies at least one page interaction operation event triggered on the browser page through a pre-configured browser plugin and determines the voice content associated with the at least one page interaction operation event. The voice content associated with the page interaction operation event is voice material related to the page interaction operation, which is generated by synchronously explaining the page interaction operation performed on the browser page. The event type of the at least one page interaction operation event is related to the page configuration task type associated with the browser page. The determination module is used to align the at least one page interaction operation event with the voice content associated with the at least one page interaction operation event according to the timestamp, and to determine the DOM element positioning information corresponding to the at least one page interaction operation event. The generation module is used to generate a target document based on the voice content associated with the page interaction operation event. The target document is formed by extracting the operation steps, operation intentions, operation step descriptions, context descriptions and business logic meanings corresponding to the page interaction operation from the voice content associated with the page interaction operation event. The generation module is further configured to generate a target script based on the DOM element positioning information corresponding to the at least one page interaction operation event and the triggering order of the at least one page interaction operation event. The target script is used to reproduce the page interaction operation related to the page configuration task type that needs to be executed on the browser page using the DOM element positioning information.

[0006] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the page processing method according to any embodiment of the present invention.

[0007] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the page processing method described in any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the page processing method described in any embodiment of the present invention.

[0009] The technical solution of this invention uses page interaction operations as trigger conditions on a browser page. It can combine the targeted collection capabilities of a pre-configured browser plugin to capture page interaction operation events that match the page's configured task type. Furthermore, it synchronously provides voice explanations of the page interaction operations performed on the browser page, forming voice material related to the page interaction operations. This breaks through the limitations of traditional solutions that only record page interaction operations and lack business semantics, giving subsequent documents and scripts business logic content and solving the problems of poor readability and low reusability of pure operation records. Through a timestamp alignment mechanism, page interaction operation events and associated voice content are precisely bound in the time dimension, eliminating semantic misalignment caused by device time differences and operation delays, ensuring a one-to-one correspondence between page interaction operations and voice descriptions. Moreover, it relies on browser plugins to directly read DOM elements to obtain structured positioning information, replacing... It replaces unstable visual recognition, avoiding positioning failures caused by page style changes and resolution differences, achieving deterministic and stable element positioning; it automatically extracts structured information such as operation steps and intents from speech, eliminating the need for manual organization and quickly generating target documents that are both operationally instructive and business-explanatory, with document content highly consistent with actual page interactions and voice explanations, improving document accuracy and reusability; based on DOM positioning information and the triggering sequence of page interactions, it automatically generates target scripts that can accurately reproduce the original page interactions, adapting to the core needs of business scenarios, eliminating the need for manual coding, shortening the automation implementation cycle, and solving the problems of inaccurate positioning and high execution failure rates of traditional automation scripts; and it relies on the same multimodal data source to synchronously generate target documents and target scripts, ensuring that the script and document logic are consistent and the steps are from the same source, avoiding the logical deviation problems caused by the separate creation of documents and scripts in traditional solutions.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a page processing method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a browser page configuration provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a page processing device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device that implements a page processing method according to an embodiment of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] In one embodiment, Figure 1This is a flowchart of a page processing method provided by an embodiment of the present invention. This embodiment is applicable to situations where browser page configuration is performed on a browser page according to the page configuration task type associated with the browser page. This method can be executed by a page processing device, which can be implemented in hardware and / or software and can be configured in an electronic device. The electronic device can be a computer device or a server, etc.

[0016] like Figure 1 As shown, the page processing method in this embodiment may include the following process: S110. In response to a page interaction operation performed on a browser page, identify at least one page interaction operation event triggered on the browser page through a pre-configured browser plugin and determine the voice content associated with at least one page interaction operation event. The voice content associated with the page interaction operation event is voice material related to the page interaction operation formed by synchronously explaining the page interaction operation performed on the browser page. The event type of at least one page interaction operation event is related to the page configuration task type associated with the browser page.

[0017] Page interaction operations can be interactive operations that trigger browser page responses to complete page configuration tasks, used to complete business processes related to the page configuration task type. A page configuration task associated with a browser page can be a structured business task that uses the browser page as an operation carrier to complete a series of page interaction operations step-by-step according to business logic to achieve a business goal. Business goals can include information entry, process approval, and batch processing. A page configuration task on a browser page can be at least one of the following configuration tasks: document processing, information entry, and form submission. For example, taking document processing as a page configuration task, a series of page interaction operations step-by-step according to business logic includes filling in identity information, uploading document photos, and confirming submission. The page configuration task type can be a specific business scenario carried by the browser page, such as document processing, form submission, or order inquiry.

[0018] Optionally, page interaction operations may include at least one of the following: click operations triggered on the browser page, input operations triggered on the browser page, file operations triggered on the browser page, and process operations triggered on the browser page. Click operations triggered on the browser page may include button clicks, link navigation, checkbox selection, and radio button selection. Input operations triggered on the browser page may include text box input, dropdown selection, and date picker entry. File operations triggered on the browser page may include file upload and attachment selection. Process operations triggered on the browser page may include form submission, step confirmation, and pop-up closure.

[0019] Page interaction events refer to structured behavior records containing key information about a page interaction, triggered by native browser listeners or plugins when an interaction is performed on a browser page. They represent a technical record and digital representation of page interaction events. Each page interaction performed on a browser page triggers a native browser response or plugin capture mechanism, generating a corresponding page interaction event. A page interaction is an interaction triggered on a browser page, and one page interaction triggered on a browser page corresponds to one or more associated page interaction events.

[0020] Pre-configured browser plugins refer to functional components that are pre-embedded in a browser and configured with data collection rules, event listening strategies, data format standards, and interaction logic. They can be used without real-time parameter configuration. Pre-configured browser plugins are used to capture page interaction events triggered on the browser page, extract DOM element positioning information, generate visual annotations (such as highlight boxes and screenshots), and report structured data in a preset format.

[0021] The audio content associated with page interaction events can be audio material that describes relevant information about the page interaction operation, generated simultaneously with the page interaction operation performed on the browser page. This audio content is neither arbitrary audio independent of the page interaction operation nor supplementary explanation lagging behind it; rather, it is specific audio material directly related to the page interaction operation triggered on the browser page and the simultaneous audio narration. Simultaneous audio narration ensures a high degree of temporal overlap between the audio recording and the page interaction operation triggered on the browser page, guaranteeing a one-to-one correspondence between the audio content and the page interaction operation and avoiding semantic misalignment.

[0022] The above solution enables precise and synchronized data capture of page interactions and voice narration. By directly capturing operation events and associated voice materials through a pre-configured browser plugin, manual data capture is unnecessary, lowering the barrier to entry. Simultaneously, it ensures a strong correlation between voice content and operational behavior, filling the gap left by traditional solutions that only record operations and lack business semantics. Furthermore, relying on the plugin's pre-defined event listening rules, it can accurately identify valid page interaction events related to the task types configured on the page.

[0023] Based on the above embodiments, a modified embodiment is proposed. This embodiment can be combined with various optional solutions in one or more of the above embodiments. The browser plugin has the following functions: JavaScript is injected into the browser page to initialize the DOM element capture and page interaction event capture functions; it collects the location information of DOM elements on the browser page in real time; it listens for and captures page interaction events on the browser page in real time; and it generates a highlighted area for the screenshot on the browser page according to the screenshot requirements and takes a screenshot of the browser page. The browser plugin provides a more deterministic positioning capability than visual recognition.

[0024] After the browser plugin is launched, it automatically injects JavaScript into the browser page to establish a DOM element capture channel. This ensures that the browser plugin can directly read the page element attributes on the browser page and activate the listening mechanism for page interaction events triggered on the browser page, enabling the browser plugin to capture page interaction operations that occur on the browser page.

[0025] After a script is injected into a browser plugin, the plugin can globally listen for page interaction events triggered on the browser page, capture page interaction events related to the page's configured task type, and filter out invalid operations such as page scrolling and mouse hovering. The captured page interaction events can carry timestamps, event types, and the identifier of the triggering DOM element, thus recording page interaction operations performed on the browser page.

[0026] After a browser plugin injects a script, it can scan the DOM element structure on a browser page, extract complete positioning information of the DOM elements (such as id, class, XPath, CSS Selector, etc.), and store it in a structured format. Browser plugins can avoid the limitations of visual recognition by directly reading DOM element attributes to obtain positioning data.

[0027] When the browser plugin responds to a screenshot request (which can be triggered manually or by voice during simultaneous audio explanation), it generates a visual highlight box for the screenshot area on the browser page and completes the screenshot according to preset rules. This achieves visual annotation and accurate screenshot of the screenshot area. The generated screenshot with the highlight box can be directly used as document material without the need for manual screenshotting and annotation, greatly improving document generation efficiency.

[0028] S120. Align the audio content associated with at least one page interaction event with the audio content associated with at least one page interaction event according to the timestamp, and determine the DOM element positioning information corresponding to at least one page interaction event.

[0029] DOM element location information can be structured data (such as id, class, XPath, CSS Selector) used to uniquely identify page elements on a browser page, and it serves as the basis for locating page interaction operations on the browser page.

[0030] Optionally, at least one page interaction event and the associated audio content are aligned according to timestamps. This includes: collecting the trigger timestamp of the page interaction event and the collection timestamp of the associated audio content, and unifying the timestamps at both ends to the UTC system reference time using the NTP network time protocol to eliminate the time difference caused by device local clock deviation and network latency, and establishing a unified timing sequence; using the timestamp of the page interaction event as an anchor point, setting a time deviation threshold to match audio segments falling within the time window in the timestamp sequence of the audio content, thus completing the initial binding of the page interaction event and the associated audio content.

[0031] The above solution addresses the misalignment between page interaction and voice timing by aligning timestamps. Furthermore, the multi-dimensional and highly deterministic nature of DOM element location information replaces traditional visual recognition methods, preventing location failures caused by browser page style changes. The structured location information of DOM elements provides accurate page interaction operations for automated script generation, ensuring script repeatability.

[0032] S130. Generate a target document based on the voice content associated with the page interaction operation event. The target document is formed by extracting the operation steps, operation intent, operation step description, context description and business logic meaning corresponding to the page interaction operation from the voice content associated with the page interaction operation event.

[0033] The target document can be a standardized business document (such as a PDD document or SOP document) that adapts to the page configuration task on the browser page. It is used to indicate the business process of the page configuration task, guide manual operation or internal training, and has structured and visual features.

[0034] The steps corresponding to a page interaction operation can be a description of the actual actions performed on the browser page, directly corresponding to the page interaction event triggered on the browser page, such as "clicking the order query button," "entering 2025 in the year input box," and "selecting a file and clicking upload." The operation intent corresponding to a page interaction operation can be the business objective of a single page interaction operation or a combination of page interaction operations. The description of the operation steps corresponding to a page interaction operation can be a detailed supplement and execution specification of the operation steps corresponding to the page interaction operation, used to ensure that the page interaction operation is completed accurately and compliantly. The context description corresponding to a page interaction operation can be an explanation of the logical connections, linkages, and execution scenarios between the operation steps corresponding to the page interaction operation, used to connect the scattered operation steps corresponding to the page interaction operation to form a complete and coherent business process. The business logic meaning corresponding to a page interaction operation can be the business rules, business requirements, or business logic contained in the operation steps and / or operation intent of the page interaction operation.

[0035] The above solution eliminates the need for manual transcription of audio, manual formatting of screenshots, and streamlining of workflows. By automatically extracting multi-dimensional semantic information from the audio and populating it into document templates, it significantly shortens document creation cycles and reduces labor costs, making it particularly suitable for document accumulation scenarios involving batch business processes. By extracting operational steps, intents, and explanations, it solves the problem of traditional purely step-by-step documents that only reveal the results but not the reasons, logic, and essence. Furthermore, the audio content and page interaction events are precisely aligned using timestamps, ensuring that the extracted semantic information directly corresponds to actual page interaction behaviors. This avoids issues such as missing steps and logical discrepancies that occur when manually writing documents, ensuring a high degree of consistency between document content and actual business processes.

[0036] S140. Generate a target script based on the DOM element positioning information corresponding to at least one page interaction operation event and the triggering order of at least one page interaction operation event. The target script is used to reproduce the page interaction operation related to the page configuration task type that needs to be executed on the browser page using the DOM element positioning information.

[0037] DOM element positioning information can be structured positioning data corresponding to page interaction events on the browser page. It serves as the basis for the target script to simulate page interaction operations to manipulate page elements and complete page configuration tasks. The triggering order of page interaction events can be the order in which these events are arranged by timestamps. This order corresponds to the logical order of at least one page interaction operation performed on the browser page to complete the page configuration task. The target script can be executable code or a set of instructions adapted to automation frameworks (such as RPA, Playwright, Selenium, and UiPath). After execution, the target script simulates the page interaction operations performed on the browser page to complete the page configuration task, automatically triggering page interaction events and completing the business processes related to the page configuration task without manual intervention.

[0038] The above solution eliminates the need for manually writing automation scripts. Based on the collected DOM location information and the order of operation triggers, it automatically generates executable target scripts, significantly lowering the barrier to automation implementation. Scripts can batch reproduce page interaction operations, replacing repetitive manual tasks and significantly improving the processing efficiency of batch form submissions, data entry, and other business processes. Furthermore, relying on precise DOM element location information, scripts are unaffected by changes in page style and resolution compared to visual recognition scripts. They also automatically add logic for element waiting and exception retries, ensuring stable execution even under dynamic page rendering and network fluctuations. Simultaneously, the target script is generated strictly according to the trigger order of page interaction events and originates from the same source as the target document, ensuring complete consistency between the script execution logic and manual operation logic, as well as the document description logic, thus improving the traceability and maintainability of the automation process.

[0039] Based on the above embodiments, modified embodiments are proposed. This embodiment can be combined with various optional solutions in one or more of the above embodiments. The page processing method of this embodiment may further include the following steps 1a-2a: Step 1a: In response to the voice explanation entering a paused state for the page interaction operation performed on the browser page, determine a reference page interaction operation event from at least one page interaction operation event. The reference page interaction operation event is the page interaction operation triggered on the browser page when the screenshot instruction is executed in the voice content associated with the at least one page interaction operation event.

[0040] Step 2a: Locate the DOM element corresponding to the interactive event of the reference page on the browser page, highlight the area of ​​the DOM element corresponding to the interactive event of the reference page on the browser page, and take a screenshot of the browser page for the purpose of creating and generating the target document.

[0041] The paused audio narration state can be defined as the temporary termination of the audio narration action associated with a page interaction on the browser page, during which no new audio content is generated. Reference page interaction events are selected from all captured page interaction events. These events correspond to the page interaction being executed on the browser page when the screenshot command was issued within the associated audio content, and the reference page interaction event triggers a corresponding page interaction event on the browser page. The screenshot command is a voice command within the associated audio content instructing the user to take a screenshot. The timing of the screenshot command corresponds to the page interaction being executed on the browser page and serves as the basis for selecting reference page interaction events.

[0042] The DOM element corresponding to the reference page interaction event can refer to the DOM element in the browser page that was manipulated by the page interaction when the reference page interaction event was triggered. Highlighting the area where the DOM element is located can be achieved through browser plugins that visually mark the area of ​​the DOM element on the page, making the area of ​​the DOM element clearly visually distinct from other areas of the browser page. The creation and generation of the target document can involve using the highlighted screenshot as visual material, combined with the operation steps, operation intent, operation step description, contextual description, and business logic meaning extracted from the audio content associated with the page interaction event, and integrating them to form the target document.

[0043] When the synchronized audio explanation of browser page interactions is paused, a filtering mechanism is activated to select reference page interaction events from all captured page interaction events. These reference page interaction events correspond to the page interaction actions being executed when the screenshot command was issued in the audio content, and these actions have already been triggered on the browser side. The DOM element corresponding to this reference event is located on the browser page, and the area containing the DOM element is then visually highlighted to clearly emphasize it. Immediately afterwards, a screenshot of the browser page is taken, and the generated highlighted screenshot serves as visual material for the subsequent creation and generation of the target document.

[0044] The above solution uses the pause in voice explanation as a trigger, adapting to the actual page interaction operations recorded by the user. Simultaneously, the voice screenshot command precisely locates the reference page interaction events, binding page interaction events with screenshot requirements. This ensures that the selected events are accurately linked to the screenshot requirements and the user's ongoing actions, avoiding the invalid filtering of irrelevant events, reducing redundant processing costs and storage resource consumption, and guaranteeing a high degree of match between subsequent screenshot materials and core operations. Furthermore, precise location of the DOM element corresponding to the reference page interaction event ensures that the area containing the DOM element is highlighted before the screenshot, clearly highlighting the core operation object in the screenshot. This allows for quick identification of key operation locations when viewing the target document later, significantly improving document readability and operational guidance, and solving the problem of unintuitive and difficult-to-understand pure text descriptions of operation scenarios.

[0045] Based on the above embodiments, a modified embodiment is proposed. This embodiment can be combined with various optional solutions in one or more of the above embodiments. The generation of the target document based on the voice content associated with page interaction events may include the following steps 1b-3b: Step 1b: Convert the voice content associated with the page interaction event into text to obtain the text content associated with the page interaction event.

[0046] Step 2b: Use a natural language model to process the text content associated with the page interaction operation event to obtain the text processing result. The text processing result includes the operation steps, operation intent, operation step description, context description and business logic meaning of the page interaction operation corresponding to the text content associated with the page interaction operation event.

[0047] Step 3b: Generate a target document based on the text processing results using a pre-configured browser plugin. The target document includes a screenshot of the browser page after highlighting the DOM element location area, the operation step number of the page interaction operation performed on the browser page, explanatory text for at least one page interaction operation performed on the browser page, and the DOM element location information corresponding to at least one page interaction operation event.

[0048] The above solution relies on natural language models to process text, accurately extracting operation steps, intentions, instructions, context, and business logic meanings. This achieves structured decomposition of fragmented text, fully reconstructing the action flow and core objectives of page interactions while also uncovering the underlying business rules and logical connections. Based on the text results, a target document is generated using a browser plugin, integrating highlighted screenshots, step numbers, explanatory text, and DOM positioning information. This gives the document both a visual context and structured semantics, facilitating quick location of the core operation and understanding of the execution logic. It also binds operations with positioning information, enhancing the document's guidance and practicality. Furthermore, the automated generation via a browser plugin significantly reduces document production costs, ensures standardized formatting, and improves document output efficiency.

[0049] Based on the above embodiments, a modified embodiment is proposed. This embodiment can be combined with various optional solutions in one or more of the above embodiments. Generating the target script based on the DOM element positioning information corresponding to at least one page interaction event and the triggering order of at least one page interaction event may include the following steps: For each page interaction event in at least one page interaction event, the DOM element positioning information corresponding to the page interaction event is converted into standardized page operation instructions; the page operation instructions are sorted and concatenated according to the triggering order of at least one page interaction event to generate an executable automation script, thus obtaining the target script.

[0050] DOM element positioning information refers to structured data bound to page interaction events, used to uniquely identify page elements in a browser page. Page operation instructions can be instruction fragments adapted to the syntax specifications of automation execution frameworks, which can be directly recognized and executed by automation tools. When operations are performed on a browser page, the execution logic of each page interaction event is formed according to its chronological order. The standardized page operation instructions are then arranged according to the triggering order of these events, and the instruction fragments are integrated according to the syntax rules of the automation framework to form a complete executable logic. This process yields an executable target script, which can simulate the execution of page interaction logic on a browser page and complete the browser page configuration tasks.

[0051] The above solution, for each captured page interaction event, converts the DOM element positioning information bound to the event into standardized page operation instructions according to the preset automation framework syntax rules, thus transforming static positioning data into executable instructions. Furthermore, the instructions are ordered according to the triggering order of the page interaction events to ensure that the script execution logic is consistent with the actual page interaction process, avoiding operation failures due to disordered instruction order. Common execution logic is integrated during instruction concatenation, improving the script's stability and robustness, adapting to the execution needs of complex page scenarios. The resulting executable target script replaces repetitive manual operations, significantly improving the execution efficiency of batch page configuration tasks. Simultaneously, the script is generated based on real operation events, ensuring the accuracy of operation reproduction and solving the problem of logical deviations that easily occur in traditional manually written scripts.

[0052] Based on the above embodiments, modified embodiments are proposed. This embodiment can be combined with various optional solutions in one or more of the above embodiments. The page processing method of this embodiment may further include the following steps 1c-2c: Step 1c: Determine the first timestamp information, the second timestamp information, and the third timestamp information. The first timestamp information is used to record the timestamp of the voice content associated with the page interaction operation event. The second timestamp information is used to record the event timestamp of the page interaction operation event. The third timestamp information is used to indicate the timestamp of taking screenshots and marking key operation steps during the execution of page interaction operations in the browser page.

[0053] Step 2c: Based on the first timestamp information, the second timestamp information, and the third timestamp information, generate a timeline related to the page configuration task. The timeline related to the page configuration task is used to ensure that the order of operation steps in the generated target document is consistent with the reference operation order, to ensure that the execution order of page operation instructions in the target script is consistent with the reference operation order, and to ensure that the explanatory text of the operation steps in the target document expresses the intent included in the voice explanation that is synchronized with the page interaction operation and the page interaction operation performed on the browser page.

[0054] The above-mentioned scheme anchors the timing nodes of voice content, page interaction events, screenshots, and key step markers, establishing a unified timing benchmark for the entire process. This comprehensively preserves the temporal correlation traces of page operations, voice explanations, and screenshot markers, resolving the issues of fragmented timing information and lack of a unified reference across multiple stages. Based on the three types of timestamps, a timeline for associated page configuration tasks is generated, achieving multi-dimensional collaborative timing control. This not only accurately anchors the operation timing benchmark, ensuring consistency between the order of operation steps in the target document, the execution order of operation instructions in the target script, and the actual reference operation order, but also, through the associative mapping of the three types of timestamps, allows the operation step descriptions in the target document to accurately correspond to the intent of the synchronized voice explanation. This achieves a deep binding between operation actions and voice semantics, enabling the document to not only present the operation process but also to reveal the core intent behind the operation. Furthermore, the timeline unifies the timing logic of documents and scripts, ensuring they are from the same source and in the same order, reducing subsequent document and script maintenance costs, improving the standardization and consistency of the overall solution, and making the data correlation between each stage clearer.

[0055] The technical solution of this invention uses page interaction operations as trigger conditions on a browser page. It can combine the targeted collection capabilities of a pre-configured browser plugin to capture page interaction operation events that match the page's configured task type. Furthermore, it synchronously provides voice explanations of the page interaction operations performed on the browser page, forming voice material related to the page interaction operations. This breaks through the limitations of traditional solutions that only record page interaction operations and lack business semantics, giving subsequent documents and scripts business logic content and solving the problems of poor readability and low reusability of pure operation records. Through a timestamp alignment mechanism, page interaction operation events and associated voice content are precisely bound in the time dimension, eliminating semantic misalignment caused by device time differences and operation delays, ensuring a one-to-one correspondence between page interaction operations and voice descriptions. Moreover, it relies on browser plugins to directly read DOM elements to obtain structured positioning information, replacing... It replaces unstable visual recognition, avoiding positioning failures caused by page style changes and resolution differences, achieving deterministic and stable element positioning; it automatically extracts structured information such as operation steps and intents from speech, eliminating the need for manual organization and quickly generating target documents that are both operationally instructive and business-explanatory, with document content highly consistent with actual page interactions and voice explanations, improving document accuracy and reusability; based on DOM positioning information and the triggering sequence of page interactions, it automatically generates target scripts that can accurately reproduce the original page interactions, adapting to the core needs of business scenarios, eliminating the need for manual coding, shortening the automation implementation cycle, and solving the problems of inaccurate positioning and high execution failure rates of traditional automation scripts; and it relies on the same multimodal data source to synchronously generate target documents and target scripts, ensuring that the script and document logic are consistent and the steps are from the same source, avoiding the logical deviation problems caused by the separate creation of documents and scripts in traditional solutions.

[0056] In one embodiment, Figure 3 This is a schematic diagram of a page processing device provided in an embodiment of the present invention. This embodiment is applicable to situations where browser page configuration is performed on a browser page according to the page configuration task type associated with the browser page. The page processing device can be implemented in hardware and / or software and can be configured in an electronic device. The electronic device can be a computer device or a server, etc.

[0057] like Figure 3 As shown, the page processing apparatus of this embodiment may include the following: The acquisition module 310 is used to respond to page interaction operations performed on the browser page, identify at least one page interaction operation event triggered on the browser page through a pre-configured browser plugin, and determine the voice content associated with the at least one page interaction operation event. The voice content associated with the page interaction operation event is voice material related to the page interaction operation, which is generated by synchronously explaining the page interaction operation performed on the browser page. The event type of the at least one page interaction operation event is related to the page configuration task type associated with the browser page. The determining module 320 is used to align the at least one page interaction operation event with the voice content associated with the at least one page interaction operation event according to the timestamp, and to determine the DOM element positioning information corresponding to the at least one page interaction operation event. The generation module 330 is used to generate a target document based on the voice content associated with the page interaction operation event. The target document is formed by extracting the operation steps, operation intentions, operation step descriptions, context descriptions and business logic meanings corresponding to the page interaction operation from the voice content associated with the page interaction operation event. The generation module 330 is further configured to generate a target script based on the DOM element positioning information corresponding to the at least one page interaction operation event and the triggering order of the at least one page interaction operation event. The target script is used to reproduce the page interaction operation related to the page configuration task type that needs to be executed on the browser page using the DOM element positioning information.

[0058] Based on the above embodiments, modified embodiments are proposed. These embodiments can be combined with various optional solutions in one or more of the above embodiments. The apparatus further includes: In response to the paused state of the voice explanation for the page interaction operation performed on the browser page, a reference page interaction operation event is determined from at least one page interaction operation event. The reference page interaction operation event is the page interaction operation triggered on the browser page when the screenshot instruction is executed in the voice content associated with the at least one page interaction operation event. Locate the DOM element corresponding to the interactive operation event of the reference page on the browser page, highlight the area of ​​the DOM element corresponding to the interactive operation event on the browser page, and take a screenshot of the browser page for the purpose of creating and generating the target document.

[0059] Based on the above embodiments, a modified embodiment is proposed. This embodiment can be combined with various optional solutions in one or more of the above embodiments, wherein the browser plugin has the following functions: Inject JavaScript into the browser page to initialize the DOM element capture and page interaction event capture functions; Real-time acquisition of DOM element location information from browser pages; Listen for and capture page interaction events on the browser page in real time; Generate a highlighted area for the screenshot on the browser page according to the screenshot requirements and then take a screenshot of the browser page.

[0060] Based on the above embodiments, a modified embodiment is proposed. This embodiment can be combined with various optional solutions in one or more of the above embodiments. The step of generating a target document based on the voice content associated with the page interaction event includes: The speech content associated with the page interaction event is converted to text to obtain the text content associated with the page interaction event. The text content associated with the page interaction operation event is processed using a natural language model to obtain the text processing result. The text processing result includes the operation steps, operation intent, operation step description, context description and business logic meaning corresponding to the page interaction operation extracted from the text content associated with the page interaction operation event. Based on the text processing results, a target document is generated through a pre-configured browser plugin. The target document includes a screenshot of the browser page after highlighting the DOM element location area, the operation step number of the page interaction operation performed on the browser page, explanatory text for at least one page interaction operation performed on the browser page, and DOM element positioning information corresponding to at least one page interaction operation event.

[0061] Based on the above embodiments, a modified embodiment is proposed. This embodiment can be combined with various optional solutions in one or more of the above embodiments. The generation of the target script based on the DOM element positioning information corresponding to the at least one page interaction event and the triggering order of the at least one page interaction event includes: For each of the at least one page interaction event, the DOM element positioning information corresponding to the page interaction event is converted into standardized page operation instructions. The page operation instructions are sorted and concatenated according to the triggering order of the at least one page interaction operation event to generate an executable automated script, thus obtaining the target script.

[0062] Based on the above embodiments, modified embodiments are proposed. These embodiments can be combined with various optional solutions in one or more of the above embodiments. The apparatus further includes: The first timestamp information, the second timestamp information, and the third timestamp information are determined. The first timestamp information is used to record the timestamp of the voice content associated with the page interaction operation event. The second timestamp information is used to record the event timestamp of the page interaction operation event. The third timestamp information is used to indicate the timestamp of taking screenshots and marking key operation steps during the execution of page interaction operations in the browser page. Based on the first timestamp information, the second timestamp information, and the third timestamp information, a timeline related to the page configuration task is generated. The timeline related to the page configuration task is used to ensure that the order of operation steps in the generated target document is consistent with the reference operation order, to ensure that the execution order of page operation instructions in the target script is consistent with the reference operation order, and to ensure that the explanatory text of the operation steps in the target document expresses the intent included in the voice explanation that is synchronized with the page interaction operation and the page interaction operation executed on the browser page.

[0063] The technical solution of this invention uses page interaction operations as trigger conditions on a browser page. It can combine the targeted collection capabilities of a pre-configured browser plugin to capture page interaction operation events that match the page's configured task type. Furthermore, it synchronously provides voice explanations of the page interaction operations performed on the browser page, forming voice material related to the page interaction operations. This breaks through the limitations of traditional solutions that only record page interaction operations and lack business semantics, giving subsequent documents and scripts business logic content and solving the problems of poor readability and low reusability of pure operation records. Through a timestamp alignment mechanism, page interaction operation events and associated voice content are precisely bound in the time dimension, eliminating semantic misalignment caused by device time differences and operation delays, ensuring a one-to-one correspondence between page interaction operations and voice descriptions. Moreover, it relies on browser plugins to directly read DOM elements to obtain structured positioning information, replacing... It replaces unstable visual recognition, avoiding positioning failures caused by page style changes and resolution differences, achieving deterministic and stable element positioning; it automatically extracts structured information such as operation steps and intents from speech, eliminating the need for manual organization and quickly generating target documents that are both operationally instructive and business-explanatory, with document content highly consistent with actual page interactions and voice explanations, improving document accuracy and reusability; based on DOM positioning information and the triggering sequence of page interactions, it automatically generates target scripts that can accurately reproduce the original page interactions, adapting to the core needs of business scenarios, eliminating the need for manual coding, shortening the automation implementation cycle, and solving the problems of inaccurate positioning and high execution failure rates of traditional automation scripts; and it relies on the same multimodal data source to synchronously generate target documents and target scripts, ensuring that the script and document logic are consistent and the steps are from the same source, avoiding the logical deviation problems caused by the separate creation of documents and scripts in traditional solutions.

[0064] The page processing apparatus provided in the embodiments of the present invention can execute the page processing method provided in any of the embodiments of the present invention, and has the corresponding functions and beneficial effects of executing the page processing method. For details, please refer to the relevant operations of the page processing method in the foregoing embodiments.

[0065] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of the present invention.

[0066] In one embodiment, Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present invention, such as... Figure 4The diagram illustrates a schematic representation of an electronic device 10 that can be used to implement embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0067] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor 11, and the computer program is executed by the at least one processor 11 to enable the at least one processor 11 to execute the page processing method provided by the present invention.

[0068] The processor 11 can perform various appropriate actions and processes based on a computer program stored in the read-only memory (ROM) 12 or a computer program loaded from the storage unit 18 into the random access memory (RAM) 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0069] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0070] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the page processing method provided by this invention.

[0071] In some embodiments, the page processing method provided herein may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the page processing method by any other suitable means (e.g., by means of firmware).

[0072] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0073] Computer programs for implementing the page processing method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0074] In the context of this invention, a computer-readable storage medium stores computer instructions that are used to cause a processor to execute and implement the page processing method provided by this invention.

[0075] The present invention also provides a computer program product comprising a computer program that, when executed by a processor, implements the method provided according to embodiments of the present invention. A computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0076] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0077] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0078] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0079] This invention also provides a computer program product, including a computer program that, when executed by a processor, can implement the methods provided in any embodiment of this application.

[0080] In the implementation of the computer program product, computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0081] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0082] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A page processing method, characterized in that, The method includes: In response to page interaction operations performed on a browser page, at least one page interaction operation event triggered on the browser page is identified by a pre-configured browser plugin, and the voice content associated with the at least one page interaction operation event is determined. The voice content associated with the page interaction operation event is voice material related to the page interaction operation formed by synchronously explaining the page interaction operation performed on the browser page. The event type of the at least one page interaction operation event is related to the page configuration task type associated with the browser page. Align the at least one page interaction event with the audio content associated with the at least one page interaction event according to the timestamp, and determine the DOM element positioning information corresponding to the at least one page interaction event; A target document is generated based on the voice content associated with the page interaction operation event. The target document is formed by extracting the operation steps, operation intentions, operation step descriptions, context descriptions, and business logic meanings corresponding to the page interaction operation from the voice content associated with the page interaction operation event. A target script is generated based on the DOM element location information corresponding to the at least one page interaction event and the triggering order of the at least one page interaction event. The target script is used to reproduce the page interaction operation related to the page configuration task type that needs to be executed on the browser page using the DOM element location information.

2. The method according to claim 1, characterized in that, The method further includes: In response to the paused state of the voice explanation for the page interaction operation performed on the browser page, a reference page interaction operation event is determined from at least one page interaction operation event. The reference page interaction operation event is the page interaction operation triggered on the browser page when the screenshot instruction is executed in the voice content associated with the at least one page interaction operation event. Locate the DOM element corresponding to the interactive operation event of the reference page on the browser page, highlight the area of ​​the DOM element corresponding to the interactive operation event on the browser page, and take a screenshot of the browser page for the purpose of creating and generating the target document.

3. The method according to claim 1, characterized in that, The browser plugin has the following functions: Inject JavaScript into the browser page to initialize the DOM element capture and page interaction event capture functions; Real-time acquisition of DOM element location information from browser pages; Listen for and capture page interaction events on the browser page in real time; Generate a highlighted area for the screenshot on the browser page according to the screenshot requirements and then take a screenshot of the browser page.

4. The method according to claim 1, characterized in that, Generate a target document based on the voice content associated with the page interaction events, including: The speech content associated with the page interaction event is converted to text to obtain the text content associated with the page interaction event. The text content associated with the page interaction operation event is processed using a natural language model to obtain the text processing result. The text processing result includes the operation steps, operation intent, operation step description, context description and business logic meaning corresponding to the page interaction operation extracted from the text content associated with the page interaction operation event. Based on the text processing results, a target document is generated through a pre-configured browser plugin. The target document includes a screenshot of the browser page after highlighting the DOM element location area, the operation step number of the page interaction operation performed on the browser page, explanatory text for at least one page interaction operation performed on the browser page, and DOM element positioning information corresponding to at least one page interaction operation event.

5. The method according to claim 1, characterized in that, Generate a target script based on the DOM element positioning information corresponding to the at least one page interaction event and the triggering order of the at least one page interaction event, including: For each of the at least one page interaction event, the DOM element positioning information corresponding to the page interaction event is converted into standardized page operation instructions. The page operation instructions are sorted and concatenated according to the triggering order of the at least one page interaction operation event to generate an executable automated script, thus obtaining the target script.

6. The method according to claim 1, characterized in that, The method further includes: The first timestamp information, the second timestamp information, and the third timestamp information are determined. The first timestamp information is used to record the timestamp of the voice content associated with the page interaction operation event. The second timestamp information is used to record the event timestamp of the page interaction operation event. The third timestamp information is used to indicate the timestamp of taking screenshots and marking key operation steps during the execution of page interaction operations in the browser page. Based on the first timestamp information, the second timestamp information, and the third timestamp information, a timeline related to the page configuration task is generated. The timeline related to the page configuration task is used to ensure that the order of operation steps in the generated target document is consistent with the reference operation order, to ensure that the execution order of page operation instructions in the target script is consistent with the reference operation order, and to ensure that the explanatory text of the operation steps in the target document expresses the intent included in the voice explanation that is synchronized with the page interaction operation and the page interaction operation executed on the browser page.

7. A page processing apparatus, characterized in that, The device includes: The acquisition module is used to respond to page interaction operations performed on the browser page. It identifies at least one page interaction operation event triggered on the browser page through a pre-configured browser plugin and determines the voice content associated with the at least one page interaction operation event. The voice content associated with the page interaction operation event is voice material related to the page interaction operation, which is generated by synchronously explaining the page interaction operation performed on the browser page. The event type of the at least one page interaction operation event is related to the page configuration task type associated with the browser page. The determination module is used to align the at least one page interaction operation event with the voice content associated with the at least one page interaction operation event according to the timestamp, and to determine the DOM element positioning information corresponding to the at least one page interaction operation event. The generation module is used to generate a target document based on the voice content associated with the page interaction operation event. The target document is formed by extracting the operation steps, operation intentions, operation step descriptions, context descriptions and business logic meanings corresponding to the page interaction operation from the voice content associated with the page interaction operation event. The generation module is further configured to generate a target script based on the DOM element positioning information corresponding to the at least one page interaction operation event and the triggering order of the at least one page interaction operation event. The target script is used to reproduce the page interaction operation related to the page configuration task type that needs to be executed on the browser page using the DOM element positioning information.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the page processing method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the page processing method of any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the page processing method according to any one of claims 1-6.

Images