Webpage interaction method based on semantic analysis and related equipment

Abstract

The invention relates to the technical field of webpage interaction, and provides a semantic analysis-based webpage interaction method and related equipment, the method comprises the following steps: capturing a user voice instruction sent by a user, and obtaining a plurality of controls of a target webpage; performing semantic analysis on the user voice instruction, and screening out a plurality of candidate controls from all controls based on a semantic analysis result; for each candidate control, performing multi-modal scoring on the candidate control to obtain a feature score of the candidate control; determining a target control from all the candidate controls according to all the feature scores, and determining a visibility state of the target control; and performing interaction control on the target control based on the visibility state and the user voice instruction to obtain an interaction result. According to the method provided by the invention, the effect and the universality of webpage interaction controlled by voice can be improved.

Description

Technical Field

[0001] This application relates to the field of web page interaction technology, and in particular to a web page interaction method and related equipment based on semantic parsing. Background Technology

[0002] With the widespread adoption of Web technology stacks in terminals such as in-vehicle cockpits and smart TVs, users have an urgent need for "native application-level" web-based voice control, specifically including core interaction scenarios such as clicking, inputting, selecting, scrolling, and multimedia control. However, existing technologies lack a unified semantic abstraction layer, making it impossible to achieve universal web-based voice interaction capabilities that are "modified, cross-device compatible, and securely controllable." Specific shortcomings are as follows: 1. It heavily relies on the semantic annotations of the webpage itself and cannot be applied to third-party webpages (lacking universality and zero modification capability): Mainstream web-based voice control solutions all require web pages to provide structured semantic support. Specific implementation methods include: marking interactive attributes on Document Object Model (DOM) elements, exposing voice-specific JavaScript APIs (such as handleIntent and exposeActions), and integrating specific voice assistant SDKs (such as Alexa WebAPI). This approach has a fundamental flaw: third-party web developers lack the motivation to adapt, mainstream scenarios such as media and e-commerce generally lack semantic annotation, and the annotation difficulty for dynamic single-page applications (SPAs) increases exponentially, making zero-modification voice control impossible to implement and severely limiting its versatility.

[0003] 2. Continuous DOM scanning and continuous listening incur high performance overhead (unsuitable for constrained device scenarios such as in-vehicle systems). Existing solutions generally adopt the implementation method of "full DOM scanning + continuous MutationObserver listening + high-frequency polling". This mode can run on personal computers (PCs), but it will cause a series of problems in devices with limited computing power such as in-vehicle systems and televisions (TVs): the high utilization rate of the central processing unit (CPU) leads to the blocking of JS threads, page rendering drops or jitter, and the exhaustion of WebView resources affects core business functions, which cannot meet the performance requirements of safety-critical scenarios such as in-vehicle systems.

[0004] 3. Static DOM cannot represent the true structure of modern web pages (V-DOM / ShadowDOM / asynchronous rendering): Modern front-end technologies such as the Virtual DOM in frameworks like React and Vue, the Shadow DOM in Web Components, dynamic routing and lazy loading in SPAs, and asynchronous rendering mean that the static DOM structure cannot reflect the true UI state of the page: controls do not exist before the Virtual DOM is rendered, the context isolation of the Shadow DOM prevents external access to internal nodes, dynamic rendering causes frequent changes in element IDs / class names, and asynchronous loading makes the parsing timing uncontrollable. Existing parsing methods based on the static DOM cannot obtain the true semantics, and the stability of control positioning is extremely poor.

[0005] 4. Lack of unified abstraction between vision and speech makes it difficult to achieve "what you see is what you can say": Existing solutions rely solely on DOM text, IDs, or tag attributes, without incorporating a visual layer for understanding. This makes it impossible to determine an element's actual visibility, occlusion status, real-time multimedia status, and visual layout structure. This lack of visual information prevents the system from matching the user's "what they see" with their "voice commands," resulting in a deficiency in multimodal collaboration capabilities. This becomes a core technological barrier to the successful implementation of "what you see is what you can say."

[0006] 5. Static rule matching cannot automatically understand unknown web pages: Existing technologies rely on developers to predefine action mappings, intent schemas, speech dictionaries, and lists of operable elements. This results in new web pages not being able to adapt automatically, and the speech capabilities immediately becoming invalid after web page content is updated. This requires a lot of manual maintenance, is costly and unstable, and cannot support generalized interaction needs.

[0007] 6. Not compatible with all Web kernels: Different devices use different web engines: Android is based on Chromium / WebView, in-vehicle systems mostly use custom Chromium or self-developed kernels, TVs commonly use WebKit / Qt WebEngine, and Internet of Things (IoT) devices use lightweight kernels. These engines differ significantly in JS injection mechanisms, DOM API completeness, rendering timing, and security sandbox strategies. Existing technologies lack a unified adaptation layer, making cross-device compatibility impossible.

[0008] 7. Lack of multimodal extension: It lacks the global fusion capability of visual models, optical character recognition (OCR), and layout understanding. True "what you see is what you can say" requires visual detection, OCR text extraction, visual partition recognition, spatial relationship perception, and multimodal information fusion (DOM + visual + speech) capabilities. However, existing solutions rely only on DOM information, cannot handle textless controls such as icons and buttons, cannot understand card-style layouts, lack multimodal matching capabilities, and cannot enter the realm of next-generation interactive technologies.

[0009] This shows that current web page interaction suffers from poor performance and low versatility in voice-controlled web page interaction. Summary of the Invention

[0010] This application provides a web page interaction method and related equipment based on semantic parsing, which can solve the problems of poor effect and low versatility of voice-controlled web page interaction.

[0011] In a first aspect, embodiments of this application provide a webpage interaction method based on semantic parsing, the webpage interaction method comprising: Capture user-generated voice commands and obtain multiple controls from the target webpage; Perform semantic parsing on user voice commands, and select multiple candidate controls from all controls based on the semantic parsing results; For each candidate control, a multimodal scoring process is performed to obtain a feature score for the candidate control; the feature score is used to identify and filter candidate controls in the target webpage. The target control is determined from all candidate controls based on all feature scores, and the visibility status of the target control is determined; the visibility status describes whether the target control is visible on the target webpage. Based on the visual status and user voice commands, the target control is interactively controlled to obtain the interaction result.

[0012] Optionally, retrieve multiple controls from the target webpage, including: Identify multiple webpage elements of the target webpage; For each webpage element, perform the following steps: Calculate the visibility score, semantic eloquence score, and interactivity score of web page elements, and calculate the comprehensive score of web page elements based on the visibility score, semantic eloquence score, and interactivity score; Determine whether the overall score meets the preset operability conditions. If so, treat the web page element as a control.

[0013] Optionally, multiple candidate controls can be selected from all controls based on the semantic parsing results, including: Based on the semantic parsing results, semantic matching is performed on each control to obtain the semantic matching degree between each control and the user's voice command; For each control, if the semantic matching degree of the control is greater than or equal to the preset semantic matching degree threshold, then the control is considered as a candidate control.

[0014] Optionally, multimodal scoring is performed on the candidate controls to obtain feature scores for the candidate controls, including: Calculate visual feature scores based on the visual characteristics of the candidate controls; Calculate text feature scores based on the text features of the candidate controls; Calculate the structural feature score based on the structural characteristics of the candidate controls; Calculate the intent matching score between candidate controls and user voice commands; The feature scores of candidate controls are calculated based on visual feature scores, text feature scores, structural feature scores, and intent matching scores.

[0015] Optionally, feature scores for candidate controls are calculated based on visual feature scores, text feature scores, structural feature scores, and intent matching scores, including: Through the formula:

[0016] Calculate feature scores ; in, The weights representing the visual feature scores Indicates visual feature scores, The weights representing the text feature scores Indicates text feature scores, The weights representing the structural feature scores Indicates the structural feature score, This indicates the weight of the intent-matching score. Intent matching score.

[0017] Optionally, the visibility status can be either visible or invisible; Determine the visibility state of the target control, including: Determine whether the webpage area corresponding to the target control meets the direct visibility condition; If the direct visibility condition is met, the visibility state of the target control is visible; If the direct visibility condition is not met, scroll the target webpage and determine whether the webpage area corresponding to the target control meets the scroll visibility condition based on the scrolling result. If the scrollable visibility condition is met, the target control's visibility state is visible; If the scroll visibility condition is not met, the target control's visibility state is invisible.

[0018] Optionally, based on the visual status and user voice commands, interactive control of the target control is performed to obtain interactive results, including: When the visibility state is visible, the target control is interactively controlled based on the semantic parsing results of the user's voice command to obtain the interaction result.

[0019] Secondly, embodiments of this application provide a webpage interaction device based on semantic parsing, comprising: The capture module is used to capture user voice commands and obtain multiple controls on the target webpage. The parsing module is used to perform semantic parsing on user voice commands and select multiple candidate controls from all controls based on the semantic parsing results; The scoring module is used to perform multimodal scoring on each candidate control to obtain a feature score for the candidate control; the feature score is used to identify and filter candidate controls in the target webpage. The determination module is used to identify the target control from all candidate controls based on all feature scores, and to determine the visibility status of the target control; the visibility status describes whether the target control is visible on the target webpage. The interaction module is used to interactively control the target control based on the visual status and user voice commands, and obtain the interaction result.

[0020] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned semantic parsing-based webpage interaction method.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned semantic parsing-based webpage interaction method.

[0022] The above-mentioned solution in this application has the following beneficial effects: In the embodiments of this application, user voice commands are captured, and multiple controls of the target webpage are obtained. Then, semantic parsing of the user voice commands is performed. Based on the semantic parsing results, multiple candidate controls are selected from all controls. For each candidate control, multimodal scoring is performed to obtain a feature score. Then, based on all feature scores, the target control is determined from all candidate controls, and its visibility state is determined. Finally, based on the visibility state and the user voice command, interactive control is performed on the target control to obtain an interaction result. Semantic parsing of user voice commands can uncover and analyze the user's interaction intent, improving the correlation between the selected candidate controls and the user's interaction intent. Interactive control based on feature scores and controllability values ​​considers the characteristics of the controls themselves, improving interaction accuracy and thus enhancing the interactive effect of voice-controlled webpages.

[0023] Furthermore, the method described in this application does not require special processing of the target webpage and user voice commands, is compatible with various webpages and Web kernels, and effectively improves the versatility of webpage interaction.

[0024] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart illustrating a semantic parsing-based webpage interaction method provided in an embodiment of this application; Figure 2 A schematic diagram of the structure of a web page interaction device based on semantic parsing provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0027] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0028] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0029] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0030] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0031] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0032] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0033] To address the issues of poor performance and low versatility in existing voice-controlled web page interactions, this application provides a semantic parsing-based web page interaction method. This method captures user voice commands and obtains multiple controls on the target web page. It then performs semantic parsing on the user voice commands, filters out multiple candidate controls based on the parsing results, performs multimodal scoring on each candidate control to obtain a feature score, determines the target control from all candidate controls based on all feature scores, and determines the target control's visibility state. Finally, based on the visibility state and the user voice command, it performs interactive control on the target control to obtain the interaction result. The semantic parsing of user voice commands can uncover and analyze the user's interaction intent, improving the correlation between the filtered candidate controls and the user's interaction intent. Interaction control based on feature scores and controllability values ​​considers the characteristics of the controls themselves, improving interaction accuracy and thus enhancing the performance of voice-controlled web page interactions.

[0034] Furthermore, the method described in this application does not require special processing of the target webpage and user voice commands, is compatible with various webpages and Web kernels, and effectively improves the versatility of webpage interaction.

[0035] The following is an illustrative example of the semantic parsing-based webpage interaction method provided in this application.

[0036] like Figure 1 As shown, the semantic parsing-based webpage interaction method provided in this application includes the following steps: Step 11: Capture the user's voice commands and obtain multiple controls from the target webpage.

[0037] The aforementioned user voice commands are commands generated based on the user's voice, including trigger time, voice intent, voice source information, etc. The controls are controllable web page elements in the target web page, such as window open button, play button, etc.

[0038] In some embodiments of this application, user speech can be captured using tools such as the Web Speech API and converted into user voice commands. The steps described above for obtaining multiple controls of the target webpage include: The first step is to identify multiple web page elements of the target web page.

[0039] For example, tools such as the Document API can be used to identify multiple web page elements of a target web page.

[0040] The second step involves performing the following steps for each webpage element: Calculate the visibility score, semantic eloquence score, and interactivity score of web page elements, and calculate the comprehensive score of web page elements based on the visibility score, semantic eloquence score, and interactivity score.

[0041] Determine whether the overall score meets the preset operability conditions. If so, treat the web page element as a control.

[0042] Otherwise, no processing is performed on the web page elements.

[0043] It should be noted that the aforementioned predictive operability conditions can be, for example, greater than or equal to a preset operability threshold. Visibility score reflects whether a node (i.e., a webpage element) is in a user-perceptible display state on the current page. Semantic describability score reflects whether a node possesses stable, clear, and naturally describable semantic information. Interactivity score reflects whether a node has the ability to trigger actual page actions. Visibility score can be obtained based on visual data analysis such as the boundary coordinates and size of the webpage area; semantic describability score can be obtained based on textual information analysis such as the text information of the webpage area; and interactivity score can be obtained based on information analysis such as whether the webpage area is interactive.

[0044] For example, for visibility scoring, data such as the boundary coordinates and actual occupied size of the webpage area corresponding to the node can be analyzed and normalized to obtain a visibility score. For instance, by combining the current viewport width and height parameters, it can be determined whether the node falls within the displayable area. A node is determined to be invisible when it meets any of the following conditions: the node is entirely outside the viewport; or the effective display area of ​​the node within the viewport is lower than a preset minimum area threshold.

[0045] Based on the geometric position determination, the calculated style state of the node is further read to verify its rendering visibility. The determination conditions include, but are not limited to: the node is not set to an unrenderable state; the node is not set to an invisible state; the node's transparency is not lower than a preset perceptible threshold.

[0046] For nodes with overlapping relationships, the actual visible area ratio of the node within the current viewport is calculated based on the rendering hierarchy and spatial coverage. Specifically, this includes: analyzing the node's hierarchical order in the rendering tree; calculating the coverage of its visible area by elements above it; and deriving the node's occluded ratio.

[0047] When the area of ​​a node covered by other elements exceeds a preset threshold, it is still determined to be an invisible node even if its geometric position and style state meet the visibility conditions.

[0048] The system will only determine a node as a visible node if it meets the following conditions: there is at least a visible display area that meets the threshold requirement; the style state allows it to be perceived by the user; and the occlusion ratio is within an acceptable range.

[0049] If a node is visible, its visibility score is assigned a value of 1; if it is not visible, it is assigned a value of 0.

[0050] For semantic soliability scoring, the semantic soliability score can be obtained by analyzing the semantic information corresponding to the node. For example, the semantic information of the node can be obtained, including: direct text information (the visible text content that the node displays to the user on the current page, which is used to represent its direct semantic meaning), auxiliary semantic description information (when the node has no visible text or the text is insufficient to express the semantics, its auxiliary description attributes are further obtained to supplement the functional semantics of the node), and semantic supplementary information of the significant visual area (for nodes that have no text but occupy an independent display area on the page and have obvious visual identifiers, the text information corresponding to their displayed content can be obtained to supplement the semantic expression of the node).

[0051] To improve the stability of semantic judgment, the system performs unified processing on the acquired semantic text, including but not limited to: removing meaningless characters and noise; controlling the length of the semantic text to avoid redundant descriptions; and standardizing the text format to reduce the impact of synonyms or format differences. The processed semantic text serves as the standard semantic representation of nodes for subsequent judgment and matching.

[0052] Semantic talkability determination rule: A node is considered to have semantic talkability when it meets the following conditions: at least one stable, clear and reusable semantic text can be extracted; the semantic text can form a unique correspondence with the node's display area or functional features.

[0053] If semantic talkability exists, the semantic talkability score is assigned a value of 1; otherwise, it is assigned a value of 0.

[0054] Interactivity scoring can be achieved by analyzing whether nodes are interactive. For example, based on the basic structural characteristics of a node, it can be determined whether it belongs to an element type with interactive semantics, including but not limited to: control elements used to trigger actions; links or form elements used for navigation or state switching.

[0055] This judgment is used to quickly identify standard control nodes with a clear interactive purpose.

[0056] Event response capability test: For non-standard interactive elements or custom components, further testing is conducted to determine whether they have event response capabilities. Specifically, this includes: whether the node directly declares event response logic for user operations; whether the node binds user input-related events through an event registration mechanism; and whether the node has event listener relationships that can be triggered at runtime.

[0057] This method can identify custom interactive areas dynamically generated by the front-end framework, avoiding omissions caused by relying solely on tag types.

[0058] Focusability and Character Attribute Analysis: Based on the node's focusability and role definition, its interactivity is further assessed, including: whether the node supports gaining focus to receive keyboard or remote control input; and whether the node declares role attributes for expressing interactive purposes.

[0059] This step is used to distinguish between nodes that have input or control capabilities and nodes that are only used to display content.

[0060] Interaction behavior style cues to assist in judgment: When structural and event information is insufficient to directly confirm interactivity, the interactive behavior patterns of nodes can be introduced as auxiliary clues to improve the accuracy of judgment.

[0061] When a node exhibits obvious interactive prompts, it can serve as supplementary evidence of interactive capabilities, but not as a separate criterion for judgment.

[0062] Interactivity determination result: A node is considered interactive when it meets any of the following conditions: it has clear interactive structure characteristics; it has an event response mechanism that can be triggered; and it supports user input or focus interaction capabilities.

[0063] If the device is interactive, assign an interactivity score of 1; otherwise, assign a score of 0.

[0064] The formula for calculating the overall score is:

[0065] in, This indicates the overall score. , , For preset weighting coefficients, For visibility scoring, Score the semantic talkability. Rate the interactivity.

[0066] In some embodiments of this application, when the DOM or other elements change, it is also necessary to update the obtained list of controls. Taking a vehicle infotainment system as an example, the implementation process of this step includes: (1) Voice-triggered activation: Limiting the timing and scope of parsing triggers: The vehicle's voice module listens for wake words in real time and generates standardized VoiceCommandJSON data upon detecting a user's wake-up.

[0067] The data includes: trigger time (accurate to milliseconds), voice intent type (such as media control or page jump), and voice source information (such as driver's side microphone).

[0068] The target parsing region is determined based on the type of voice intent. For example, the "media control" intent only parses the page area related to music playback.

[0069] (2) Callback Channel Establishment: Constructing a communication mechanism between the device and the webpage: During the webpage loading phase, a bidirectional communication channel is established with the vehicle system through a pre-injected initialization script. For example, this is achieved by calling the native interface provided by the vehicle system: registerCallback("voiceEventCallback", handleVoiceEvent), where "voiceEventCallback" is the channel identifier agreed upon by both parties, and "handleVoiceEvent" is the callback function for handling voice events on the webpage.

[0070] Once the communication channel is established, the vehicle system immediately transmits environmental parameters to the webpage, including the current page identification information (to confirm whether it is a music-related page), the size of the currently visible area (e.g., the vehicle screen is 1920*1080, excluding invisible controls outside the screen), and the current focus position or user operation context (e.g., if the focus is on the "playlist", the weight of the node in that area can be increased). These parameters provide a "contextual basis" for subsequent tree construction, avoiding meaningless parsing.

[0071] (3) Hot word scene synchronization: "Labeling" semantic nodes to improve matching efficiency: At the same moment of voice invocation (delay ≤ 50ms to ensure synchronization), the web page reports the "hot word scene" of the current page to the vehicle system through the reportHotWordScenes() interface. Specifically, the scene information includes two core contents: predicted semantic category (such as media control hot words such as "play / pause / next song"); possible target area (such as "the play button is located in the lower left corner of the page, size 100*100px"). This information is used to add "voice intent weight" to the nodes when constructing the semantic tree.

[0072] (4) Debounce Initialization Optimization: To avoid repeated semantic tree construction due to multiple DOM changes within a short period, a "voice event-driven + debouncing" mechanism is used to trigger initialization only once after the voice event has stabilized. This significantly reduces computational load and solves the performance limitations of the in-vehicle WebView and the stuttering caused by frequent parsing.

[0073] (5) Multimodal semantic tree construction (Semantic UI Tree), i.e., obtaining multiple controls: Based on a unified DOM abstract node (UDOMNode), a semantic UI tree oriented towards user operations is constructed. This semantic UI tree is not a simple mapping of the original DOM hierarchy of the webpage, but rather an abstract representation of the control objects and their relationships that are perceptible, describable, and actionable by the user on the page. This supports subsequent voice intent matching and secure operation execution, with each node corresponding to a control on the webpage. During the construction process, the system comprehensively integrates the following multi-dimensional information to uniformly model webpage controls: a. DOM structure information: This includes the control's tag type, attribute information, and bound event types, used to identify the control's basic functions and operability.

[0074] b. CSS and layout information: including the visibility status of controls, their hierarchy, occlusion status, and the actual clickable area, used to determine whether a control is within the currently operable range.

[0075] c. Textual semantic information: including the visible text content of the control, accessibility attributes, and page text recognition results when necessary, used to determine the function name of the control and its semantic features that can be described by the user.

[0076] d. Visual feature information: including the control's position coordinates, size, and relative layout relationship on the page, used to determine the control's prominence and operation priority.

[0077] e. Interaction capability information: This includes whether the control supports operations such as clicking, input, or focus acquisition, and is used to filter out non-executable nodes.

[0078] f. Voice Intent Association Information: Based on the defined voice hot word scenarios and action types, the controls are associated with the user operation intents that may be triggered, in order to improve matching efficiency.

[0079] Triggering sources and their relationship with speech: (1) User voice activation event (main trigger source): After a user initiates voice activation, the system enters a voice interaction cycle. During this cycle, it is necessary to ensure that the semantic tree remains consistent with the actual state of the page. This is the core trigger signal for the construction and updating of the semantic tree.

[0080] (2) Changes in DOM structure: By monitoring key areas covered by the semantic tree through MutationObserver, the semantic tree is validated or partially rebuilt when the DOM is rebuilt, the SPA route is switched, or the control is destroyed and rebuilt, thus preventing voice operations from affecting invalid nodes.

[0081] (3) Changes in the state of specific interactive areas (video play button, tab list, etc.): Some webpage interactions do not cause changes to the DOM structure, but they can alter the executable state of controls, for example: a. The play button toggles between "play / pause"; b. The tab is selected or deselected; c. The list item enters the "Loading / Disabled" state.

[0082] When such a state change is detected, the system does not rebuild the semantic tree structure. Instead, it updates the state attributes and matching weights of the corresponding nodes in the semantic tree to ensure that the voice command matches the current page state. This prevents the voice command from being executed incorrectly when there is a "state mismatch".

[0083] (4) Cross-engine result synchronization (WEAL messageHandler / WebChannel): In different web engines or container environments, the voice execution results need to be returned through the WEAL cross-engine adaptation layer to synchronize the execution status of semantic nodes and avoid duplicate execution or state inconsistency.

[0084] Fusion strategy: It integrates and processes various asynchronous signals from voice activation, DOM structure changes, control state changes, and cross-engine feedback.

[0085] (1) By using a global lock, ensure that there is only one semantic tree update process within the same voice interaction cycle. (2) By using a single-instance debouncing mechanism, multiple changes triggered in a short period of time are merged and processed. (3) All trigger signals enter the same scheduling entry point, and the scheduler determines whether it is necessary to: verify the semantic tree, perform a partial update, delay reconstruction, or simply refresh the node status.

[0086] Generate a semantic tree version that is consistent with the current speech context and synchronized with the real-time state of the page, for subsequent speech matching and operation execution.

[0087] Step 12: Perform semantic parsing on the user's voice commands, and select multiple candidate controls from all controls based on the semantic parsing results.

[0088] The semantic parsing results described above are used to describe the user's interactive intent in the user's voice commands, such as playing music or opening navigation.

[0089] In some embodiments of this application, models such as Support Vector Machine (SVM) can be used to perform semantic parsing on user voice commands to obtain semantic parsing results. The step of selecting multiple candidate controls from all controls based on the semantic parsing results includes: The first step is to perform semantic matching on each control based on the semantic parsing results, and obtain the semantic matching degree between each control and the user's voice command.

[0090] For example, operations such as cosine similarity can be used to perform semantic matching calculations on the semantic parsing results and the semantic information (related text, etc.) of the controls, and the calculation results can be used as the semantic matching degree.

[0091] The second step is to consider each control individually. If the semantic matching degree of the control is greater than or equal to the preset semantic matching degree threshold, then the control is considered as a candidate control.

[0092] If the semantic matching degree is less than the preset semantic matching degree threshold, the control will not be processed.

[0093] Step 13: For each candidate control, perform multimodal scoring to obtain the feature score of the candidate control.

[0094] The aforementioned multimodal scoring includes a comprehensive evaluation of candidate controls from visual, textual, and structural perspectives. Feature scores are used to identify and filter candidate controls on the target webpage.

[0095] In some embodiments of this application, the step of performing multimodal scoring on candidate controls to obtain feature scores for candidate controls includes: The first step is to calculate the visual feature score based on the visual features of the candidate controls.

[0096] The aforementioned visual features include: position (the coordinates of the control on the screen or within the container), area (the actual display size of the control and its proportion of the page), occlusion (whether the control is covered or clipped by other elements, and whether it is within the visible area), and focus area (whether the control is in the current user's visible window or area of ​​attention).

[0097] For example, visual features can be scored according to preset scoring criteria to obtain visual feature scores. For instance: Scoring criteria: whether the control is within the visible area, its screen area size, whether it is obscured, and whether it is within the user's attention area.

[0098] Calculation method: a. Position determination: Use getBoundingClientRect() to determine if the control is within the viewport; b. Area calculation: Actual display area of ​​the control / screen area; the larger the area, the higher the score. c. Occlusion ratio: The coverage ratio is calculated based on the hierarchical relationship; the less occlusion, the higher the score. d. Focus weight: Whether the control is in the current focus area or the user-interactive area.

[0099] The second step is to calculate the text feature score based on the text features of the candidate controls.

[0100] The aforementioned text features include: element text (readable text content displayed by controls, such as buttons, labels, titles, and image text recognition: extracting text from controls that present text as images using OCR), normalization and synonym replacement (unifying different expressions, such as "OK" = "Submit", improving matching flexibility), and keyword extraction (identifying functional keywords or action words for matching with user voice intent), etc.

[0101] For example, text features can be scored according to preset scoring criteria to obtain text feature scores. For instance: Scoring criteria: The degree of matching between the text on the control and the user's intent, including direct text, image text (OCR), synonyms, and keywords.

[0102] Calculation method: a. Text extraction: innerText, textContent, aria-label, title, etc.; b. OCR to extract text from images; c. Text normalization: remove stop words, standardize capitalization, and convert between simplified and traditional Chinese characters; d. Matching Score: Keyword matching with the user's voice intent; the higher the match, the better. The higher the score; e. Synonym mapping: such as treating "OK" and "Submit" as a match, improving matching flexibility.

[0103] The third step is to calculate the structural feature score based on the structural features of the candidate controls.

[0104] The aforementioned structural features include: DOM hierarchy position (the nesting relationship of controls in the HTML / DOM tree), ARIA attributes (accessibility tags, such as aria-label and role, to improve the accuracy of semantic understanding), Shadow DOM path (encapsulated component path, used to identify encapsulated controls), and container type (the list, card, or table structure in which the control is located, which helps to determine the semantics of the control).

[0105] For example, structural features can be scored according to preset scoring criteria to obtain structural feature scores. For instance: Scoring criteria: The semantic position of the control in the page structure, reflecting its functional role and contextual relationship.

[0106] Calculation method: a. DOM hierarchy: The nesting depth of controls in the HTML tree and the type of the parent node; b. ARIA attributes: such as role="button" or aria-label; c. Container type: The list, card, or table structure in which the control is located, serving as a semantic hint; d.ShadowDOM path: The hierarchical path of the encapsulated component.

[0107] The fourth step is to calculate the intent matching score between the candidate controls and the user's voice commands.

[0108] For example, a pre-defined scoring criterion can be used to calculate the intent matching score between candidate controls and user voice commands. For instance: Scoring criteria: Whether the control function matches the user's voice operation intent.

[0109] Calculation method: a. Action type matching: Compare the operation type (click, input, scroll, etc.) with the control properties; b. Action Target Slot: Whether the control category specified in the user intent matches the candidate control; c. Control mapping rules: Predefined operation-control matching table.

[0110] In some embodiments of this application, large language models and the like can also be used to calculate visual feature scores, text feature scores, structural feature scores, and intent matching scores based on the different scoring criteria mentioned above.

[0111] The fifth step is to calculate the feature scores of the candidate controls based on visual feature scores, text feature scores, structural feature scores, and intent matching scores.

[0112] Specifically, through the formula:

[0113] Calculate feature scores .

[0114] in, The weights representing the visual feature scores Indicates visual feature scores, The weights representing the text feature scores Indicates text feature scores, The weights representing the structural feature scores Indicates the structural feature score, This indicates the weight of the intent-matching score. Intent matching score.

[0115] For example, taking music playback as an example, the scoring dimensions of the two candidate controls are shown in Table 1: Table 1

[0116] It is worth mentioning that by conducting multimodal scoring of candidate controls from three aspects—visual, textual, and structural—the degree of matching between candidate controls and user intent and page control was fully explored, thus achieving accurate evaluation of candidate controls.

[0117] Step 14: Determine the target control from all candidate controls based on all feature scores, and determine the visibility status of the target control.

[0118] The visibility states described above describe whether the target control is visible on the target webpage. The visibility state is either visible or invisible.

[0119] In some embodiments of this application, the candidate control corresponding to the maximum feature score can be used as the target control. The steps for determining the visibility state of the target control are as follows: Determine whether the webpage area corresponding to the target control meets the direct visibility condition; If the direct visibility condition is met, the visibility state of the target control is visible; If the direct visibility condition is not met, scroll the target webpage and determine whether the webpage area corresponding to the target control meets the scroll visibility condition based on the scrolling result. If the scrollable visibility condition is met, the target control's visibility state is visible; If the scroll visibility condition is not met, the target control's visibility state is invisible.

[0120] For example, the above direct visibility conditions are: none of the following conditions are met: the control area rect does not overlap with the viewport; the control rect is occluded by a proportion exceeding a preset threshold (e.g., 50%); the control rect area is less than the minimum interaction area threshold.

[0121] The page scrolling process described above is as follows: the scrolling interface is called to scroll the webpage to the vicinity of the target control, and the page scrolling result is obtained.

[0122] The scroll visibility condition mentioned above is: in the page scrolling result, the web page area corresponding to the target control meets the direct visibility condition.

[0123] For example, the specific includes: Step 1: Visibility determination: Obtain the current visible state of the target control through the WEAL interface: const rect = WEAL.getBoundingClientRect(nodeId); const viewport = WEAL.getViewportRect(); The control is deemed not to be directly visible when any of the following conditions are met: the control area (rect) does not overlap with the viewport; the control rect is occluded by a proportion exceeding a preset threshold (e.g., 50%); or the control rect area is less than the minimum interaction area threshold.

[0124] Step 2: Scroll to the reachable area: When the target control is not directly visible, the system calls the scrolling interface to scroll the page to the vicinity of the target control: WEAL.scrollToNode(nodeId, { align: 'center', behavior: 'smooth', maxOffset: viewport.height * 0.3 }); align='center': Ensures the control is positioned in the center of the user's visual field; maxOffset: Prevents excessive scrolling and adapts to different scenarios.

[0125] Step 3: Validate the scrolling results After scrolling is complete, the system retrieves the control's position information again to confirm that it has entered the operable area: const rectAfter = WEAL.getBoundingClientRect(nodeId); const isVisible = isRectInViewport(rectAfter, viewport); If it is still not visible, mark it as an unreachable control, abandon the execution and return a failure status.

[0126] Step 4: Control highlighting tooltip Once a control enters the operable area, the target control is highlighted: WEAL.highlight(nodeId, { style: 'outline', color: '#FFD54F', thickness: 3, timeout: 3000, pointerEvents: 'none' }); Highlighting is only a visual cue and does not change the original clickable area of ​​the control; pointerEvents: 'none' ensures that it does not affect user or system operations; The highlight effect will be automatically removed after the timeout period.

[0127] Step 5: Enter the safe execution phase Once the highlighting is complete, the control enters the safe execution phase, where the corresponding click, input, or navigation operations are performed.

[0128] Step 15: Based on the visual status and user voice commands, perform interactive control on the target control to obtain the interaction result.

[0129] The interaction result is the result of controlling the target control according to the user's intent in the user's voice command. For example, if the user's intent is to start navigation and the target control is the navigation URL, then clicking to open the navigation URL will be used as the interaction result, thus realizing interactive control.

[0130] Specifically, when the visibility state is visible, the target control is interactively controlled based on the semantic parsing results of the user's voice command to obtain the interaction result.

[0131] When the visibility status is not visible, it means that the target control cannot meet the interaction requirements. The feature scores can be sorted from largest to smallest, and the candidate control corresponding to the next feature score of the current target control can be taken as the new target control. Then return to the step 14 to determine the visibility status of the target control.

[0132] It should be noted that a risk analysis is required before implementing interactive control, such as: Determine whether the target control belongs to a risky interaction such as a payment button, form submission, or cross-linking. If so, confirm with the user before execution.

[0133] For example: Predefine the risk level for each control or action, such as: a. High-risk: payment buttons, form submissions, cross-domain redirects, file downloads; b. Medium risk: Modify settings; c. Low risk: Playing videos, scrolling pages, switching tabs.

[0134] Calculate the risk score before the action is performed and compare it with a threshold: a. If RiskScore ≥ High Risk Threshold → Trigger Additional Confirmation Step; b. If RiskScore ≥ Medium Risk Threshold → Prompt the user but allow them to choose quick confirmation; c. Otherwise, execute the operation directly.

[0135] Risk scoring factors include: control type, operation category, whether the target URL or interface is on the whitelist, and historical operation records.

[0136] Prompt the user before the action is executed to reduce the risk of accidental operation.

[0137] Pre-execution safety measures (prompting the user before the action is performed and reducing the risk of accidental operation), implemented as follows: (1) Control highlighting: a. Use CSS or overlays on the page to change the border / background color of the target control, making it visually distinct for the user; b. Simultaneously record the control node ID to ensure that the highlighted area matches the execution target.

[0138] (2) TTS prompts: a. The system reads aloud the action to be performed via text-to-speech (TTS), for example: "Payment is about to be submitted, please confirm." b. The prompts and control highlighting are synchronized to enhance user awareness.

[0139] (3) User confirmation or timeout cancellation: a. A confirmation interface will pop up or you will wait for voice / touch confirmation; b. If no confirmation is received within the preset timeout period (e.g., 5 seconds), the operation will be automatically cancelled.

[0140] c. User confirmation results are directly linked to the risk score; operations with a high RiskScore must be confirmed before execution.

[0141] Provides standardized operation interfaces to ensure that operations are controlled and traceable: (1) simulateClick(node): Simulates a click operation, only effective on nodes that have already been evaluated; (2) inputText(node, safeMode): Input text in safe mode, sensitive information is encrypted or masked; (3) scrollToNode(node): Scrolls the page to the target control to avoid accidental operation; (4) navigate(url): redirects to only whitelisted URLs, preventing access to unknown or dangerous pages.

[0142] Execution process: (1) First, determine whether confirmation is needed through risk scoring; (2) If necessary, first execute control highlighting and TTS prompts; (3) After user confirmation, the corresponding security API is invoked to perform the action; (4) If the operation is abnormal or times out, the rollback mechanism is invoked to undo the operation.

[0143] Auditing and rollback (ensuring operations are traceable and recoverable), implementation process: (1) Operation log recording a. The system records the execution time, control node ID, operation type, and user confirmation status; b. Logs can be used for auditing, bug tracking, or security analysis.

[0144] (2) Form / payment action pre-submission and rollback: a. For high-risk operations, generate a transaction snapshot before execution; b. If the operation is canceled or fails to execute, it can be restored to the snapshot state to ensure page and data consistency.

[0145] It is worth mentioning that semantic parsing of user voice commands can uncover and analyze user interaction intentions, improve the correlation between the selected candidate controls and user interaction intentions, and perform interaction control based on feature scores and controllability values, taking into account the characteristics of the controls themselves, thereby improving the accuracy of interaction and thus enhancing the interactive effect of voice-controlled web pages.

[0146] Furthermore, the method described in this application does not require special processing of the target webpage and user voice commands, is compatible with various webpages and Web kernels, and effectively improves the versatility of webpage interaction.

[0147] The method described in this application also has the following advantages: A semantic tree construction mechanism triggered by voice events replaces the traditional continuous scanning mode, reducing performance overhead at the source; a unified scheduling and state locking mechanism that integrates multi-source asynchronous events solves the problem of semantic node failure in dynamic pages; a triple filtering algorithm for talkable nodes that integrates visibility, semantic features, and interactive capabilities improves interaction accuracy; a semantic tree self-recovery mechanism that supports partial / full reconstruction adapts to dynamic page scenarios such as single-page applications; and a cross-kernel consistent WebEngine Abstraction Layer (WEAL) achieves multi-device and multi-engine compatibility.

[0148] Furthermore, this method is based on the Web Kernel Abstraction Layer (WEAL), which does not require modification of the target webpage's source code. It triggers semantic tree construction through voice commands and is compatible with web engines from different manufacturers (including but not limited to Chromium, WebKit, and self-developed in-vehicle infotainment kernels).

[0149] The following is an exemplary description of the web page interaction device based on semantic parsing provided in this application.

[0150] like Figure 2 As shown, this application embodiment provides a web page interaction device based on semantic parsing. The web page interaction device 200 includes: The capture module 201 is used to capture user voice commands and obtain multiple controls of the target webpage; The parsing module 202 is used to perform semantic parsing on the user's voice commands and to select multiple candidate controls from all controls based on the semantic parsing results; The scoring module 203 is used to perform multimodal scoring on each candidate control to obtain the feature score of the candidate control; the feature score is used to identify and filter candidate controls in the target webpage; The determination module 204 is used to determine the target control from all candidate controls based on all feature scores, and to determine the visibility status of the target control; the visibility status is used to describe whether the target control is visible on the target webpage; The interaction module 205 is used to interactively control the target control based on the visual status and user voice commands, and obtain the interaction result.

[0151] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0152] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0153] like Figure 3 As shown, an embodiment of this application provides a terminal device, wherein the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 3 The diagram shows only one processor, a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 executes the computer program D102 to implement the steps in any of the above method embodiments.

[0154] Specifically, when the processor D100 executes the computer program D102, it captures user voice commands and obtains multiple controls on the target webpage. Then, it performs semantic analysis on the user voice commands, filters multiple candidate controls from all controls based on the semantic analysis results, performs multimodal scoring on each candidate control to obtain its feature score, determines the target control from all candidate controls based on all feature scores, and determines the visibility state of the target control. Finally, based on the visibility state and the user voice command, it performs interactive control on the target control to obtain the interaction result. The semantic analysis of the user voice commands can uncover and analyze the user's interactive intent, improving the correlation between the filtered candidate controls and the user's interactive intent. Interactive control based on feature scores and controllability values ​​considers the characteristics of the controls themselves, improving interaction accuracy and thus enhancing the interactive effect of voice-controlled webpages.

[0155] Furthermore, the method described in this application does not require special processing of the target webpage and user voice commands, is compatible with various webpages and Web kernels, and effectively improves the versatility of webpage interaction.

[0156] The processor D100 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0157] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may be an external storage device of the terminal device D10, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device D10. Furthermore, the memory D101 may include both internal and external storage units of the terminal device D10. The memory D101 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory D101 can also be used to temporarily store data that has been output or will be output.

[0158] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0159] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0160] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a semantically parsing-based web page interaction method device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0161] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0162] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0163] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention.

Claims

1. A webpage interaction method based on semantic parsing, characterized in that, include: Capture user-generated voice commands and obtain multiple controls from the target webpage; The user's voice command is semantically parsed, and multiple candidate controls are selected from all controls based on the semantic parsing results; For each candidate control, a multimodal scoring is performed on the candidate control to obtain the feature score of the candidate control; The feature score is used to identify and filter candidate controls in the target webpage; The target control is determined from all candidate controls based on all feature scores, and the visibility status of the target control is determined. The visibility status is used to describe whether the target control is visible on the target webpage; Based on the visibility status and the user's voice command, the target control is interactively controlled to obtain the interaction result.

2. The webpage interaction method according to claim 1, characterized in that, The multiple controls for obtaining the target webpage include: Identify multiple webpage elements of the target webpage; For each of the aforementioned webpage elements, perform the following steps: Calculate the visibility score, semantic eloquence score, and interactivity score of the web page element, and calculate the comprehensive score of the web page element based on the visibility score, semantic eloquence score, and interactivity score; Determine whether the comprehensive score meets the preset operability conditions. If so, treat the web page element as a control.

3. The webpage interaction method according to claim 2, characterized in that, The process of filtering multiple candidate controls from all controls based on semantic parsing results includes: Based on the semantic parsing results, semantic matching is performed on each control to obtain the semantic matching degree between each control and the user's voice command; For each of the aforementioned controls, if the semantic matching degree corresponding to the control is greater than or equal to a preset semantic matching degree threshold, then the control is considered as a candidate control.

4. The webpage interaction method according to claim 1, characterized in that, The process of performing multimodal scoring on the candidate controls to obtain feature scores for the candidate controls includes: Calculate visual feature scores based on the visual features of the candidate controls; Calculate text feature scores based on the text features of the candidate controls; Calculate the structural feature score based on the structural features of the candidate controls; Calculate the intent matching score between the candidate control and the user's voice command; The feature score of the candidate control is calculated based on the visual feature score, the text feature score, the structural feature score, and the intent matching score.

5. The webpage interaction method according to claim 4, characterized in that, The calculation of the feature score of the candidate control based on the visual feature score, the text feature score, the structural feature score, and the intent matching score includes: Through the formula: Calculate feature scores ; in, The weights representing the visual feature scores Indicates visual feature scores, The weights representing the text feature scores Indicates text feature scores, The weights representing the structural feature scores Indicates the structural feature score, This indicates the weight of the intent-matching score. Intent matching score.

6. The webpage interaction method according to claim 1, characterized in that, The visibility status is either visible or invisible; Determining the visibility state of the target control includes: Determine whether the webpage area corresponding to the target control meets the direct visibility condition; If the direct visibility condition is met, then the visibility state of the target control is visible; If the direct visibility condition is not met, the target webpage is scrolled, and the webpage area corresponding to the target control is determined based on the scrolling result to see if the scrolling visibility condition is met. If the scrollable visibility condition is met, then the visibility state of the target control is visible. If the scroll visibility condition is not met, the visibility state of the target control is invisible.

7. The webpage interaction method according to claim 6, characterized in that, The step of interactively controlling the target control based on the visibility state and the user's voice command to obtain an interaction result includes: When the visibility state is visible, the target control is interactively controlled according to the semantic parsing result of the user's voice command to obtain the interaction result.

8. A webpage interaction device based on semantic parsing, characterized in that, include: The capture module is used to capture user voice commands and obtain multiple controls on the target webpage. The parsing module is used to perform semantic parsing on the user's voice commands and select multiple candidate controls from all controls based on the semantic parsing results; The scoring module is used to perform multimodal scoring on each candidate control to obtain the feature score of the candidate control; The feature score is used to identify and filter candidate controls in the target webpage; The determination module is used to determine the target control from all candidate controls based on all feature scores, and to determine the visibility status of the target control; The visibility status is used to describe whether the target control is visible in the target webpage; The interaction module is used to interactively control the target control based on the visibility status and the user's voice command, and obtain the interaction result.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the web page interaction method based on semantic parsing as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the web page interaction method based on semantic parsing as described in any one of claims 1 to 7.

Images