Web page interaction methods, devices, electronic devices and readable storage media
By performing semantic parsing and safe obstacle avoidance segmentation on the interactive request text data, the webpage image segmentation process was optimized, solving the problems of low semantic integrity of webpage image fragments and high computational resource consumption, thus achieving efficient webpage interaction.
Patent Information
- Application Number
- CN202611199974.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-07
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, fixed mesh mechanical cutting has weak boundary perception capabilities, and indiscriminate scanning of the entire image results in low semantic integrity of web page image fragments, low accuracy of large visual model recognition, excessive consumption of computing resources, and slow and inefficient interaction speed.
The interaction request text data is semantically parsed based on the preset layout distribution data to obtain intent label data and target existence probability values. Based on the intent label data and target existence probability values, the preset physical cutting grid coordinates are safely segmented to obtain local image slice data. Based on the local image slice data, the intent label data is priority scanned and scheduled to obtain target element coordinate data and accompanying cache data. Finally, the target element coordinate data and accompanying cache data are indexed and solidified to generate interaction result data.
It improves scanning efficiency and target hit probability, enhances the recognition accuracy of large visual models, improves the semantic integrity and interactive response speed of web page image fragments, and enhances the response speed and resource utilization efficiency of interactive requests.
Smart Images

Figure CN123115548A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of graphical user interface interaction technology, and in particular to a web page interaction method, apparatus, electronic device, and readable storage medium. Background Technology
[0002] In the field of automated web page interaction, a fixed grid is typically used to mechanically segment the web page, and then a Vision Large Model (VLM) is used to scan the entire image of each slice to locate the target element. However, the fixed cutting lines can easily cut off user interface (UI) elements (such as search boxes and buttons) that cross them, causing the segmented image fragments to lose semantic integrity, making them unrecognizable by the VLM. Indiscriminate scanning of all slices in the entire image consumes a lot of computing resources, resulting in slow interaction speed and low efficiency. In existing technologies, finer grids are commonly used for segmentation, but this leads to an exponential increase in the number of slices, further exacerbating the computational burden. Moreover, it does not provide good support for complex or dynamically rendered pages.
[0003] It is evident that existing technologies suffer from several problems: weak boundary perception due to fixed mesh mechanical cutting, indiscriminate scanning of the entire image, low semantic integrity of webpage image fragments, low accuracy of large visual model recognition, excessive consumption of computational resources, and slow and inefficient interaction speed. Summary of the Invention
[0004] In view of this, the present disclosure provides a web page interaction method, apparatus, electronic device, and readable storage medium to solve the problems in the prior art where the fixed mesh mechanical cutting has weak boundary perception capability, indiscriminate scanning of the entire image results in low semantic integrity of web page image fragments, low accuracy of large visual model recognition, excessive consumption of computing resources, and slow and inefficient interaction speed.
[0005] A first aspect of this disclosure provides a webpage interaction method, comprising: performing semantic parsing processing on interaction request text data based on preset layout distribution data to obtain intent tag data and target existence probability values; performing safe avoidance segmentation processing on preset physical cutting grid coordinates based on intent tag data and target existence probability values to obtain local image slice data; performing priority scanning scheduling processing on intent tag data based on local image slice data to obtain target element coordinate data and accompanying cache data; and performing indexing and solidification processing on target element coordinate data and accompanying cache data to obtain interaction result data corresponding to the interaction request text data.
[0006] In some embodiments, priority scanning scheduling processing is performed on intent label data based on local image slice data to obtain target element coordinate data and accompanying cache data, including: performing queue arrangement processing on local image slice data based on target existence probability values to obtain scan queue data; performing target element recognition processing on local image slice data based on scan queue data to obtain candidate element coordinate data; determining target element coordinate data based on candidate element coordinate data; and performing accompanying element extraction processing on target element coordinate data to obtain accompanying cache data.
[0007] In some embodiments, a safe avoidance segmentation process is performed on the preset physical cutting grid coordinates based on intent tag data and target existence probability values to obtain local image slice data. This includes: performing node filtering processing on preset page structure data based on intent tag data to obtain semantic node data; performing boundary coordinate extraction processing on the semantic node data to obtain node boundary data; performing cross-collision detection processing on the node boundary data and the preset physical cutting grid coordinates to obtain collision determination data; performing dynamic offset processing on the preset physical cutting grid coordinates based on the collision determination data to obtain logical safety boundary data; and performing image cropping processing on the logical safety boundary data to obtain local image slice data.
[0008] In some embodiments, semantic parsing processing of interactive request text data based on preset layout distribution data is performed to obtain intent label data and target existence probability values, including: performing intent feature extraction processing on interactive request text data to obtain intent feature data; performing orientation probability mapping processing on intent feature data based on preset layout distribution data to obtain orientation probability values; sorting the orientation probability values to obtain target existence probability values; and performing label solidification processing on intent feature data to obtain intent label data.
[0009] In some embodiments, the target element coordinate data and the accompanying cache data are indexed and solidified to obtain the interaction result data corresponding to the interaction request text data. This includes: performing key-value pair mapping on the accompanying cache data to obtain cache index data; performing structured encapsulation on the target element coordinate data and the cache index data to obtain structured result data; and performing state annotation on the structured result data to obtain the interaction result data.
[0010] In some embodiments, dynamic offset processing is performed on the preset physical cutting grid coordinates based on collision determination data to obtain logical safety boundary data, including: performing collision direction parsing processing on the collision determination data to obtain offset direction data; performing edge distance calculation processing on the preset physical cutting grid coordinates based on the offset direction data to obtain edge distance values; and performing neighbor offset processing on the preset physical cutting grid coordinates based on the edge distance values to obtain logical safety boundary data.
[0011] In some embodiments, the node boundary data and the preset physical cutting grid coordinates are subjected to cross-collision detection processing to obtain collision determination data, including: performing area quantization processing on the node boundary data to obtain node area values; performing threshold comparison processing on the node area values and the preset screen area ratio threshold to obtain valid node data; and performing boundary cross-detection processing on the valid node data and the preset physical cutting grid coordinates to obtain collision determination data.
[0012] A second aspect of this disclosure provides a web page interaction device, comprising: a first processing module, configured to perform semantic parsing processing on interaction request text data based on preset layout distribution data to obtain intent tag data and target existence probability values; a second processing module, configured to perform safe avoidance segmentation processing on preset physical cutting grid coordinates based on intent tag data and target existence probability values to obtain local image slice data; a third processing module, configured to perform priority scanning scheduling processing on intent tag data based on local image slice data to obtain target element coordinate data and accompanying cache data; and a fourth processing module, configured to perform indexing and solidification processing on target element coordinate data and accompanying cache data to obtain interaction result data corresponding to the interaction request text data.
[0013] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.
[0014] A fourth aspect of this disclosure provides a readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.
[0015] The beneficial effects of this disclosed embodiment compared with the prior art are as follows: Semantic parsing of interactive request text data based on preset layout distribution data yields intent label data and target existence probability values; safe avoidance segmentation of preset physical cutting grid coordinates is performed based on intent label data and target existence probability values to obtain local image slice data; priority scanning scheduling is performed on intent label data based on local image slice data to obtain target element coordinate data and accompanying cache data; and the target element coordinate data and accompanying cache data are indexed and solidified to obtain interactive result data corresponding to the interactive request text data, thereby improving scanning efficiency and target hit probability; enhancing the recognition accuracy of the large visual model; improving the semantic integrity of webpage image fragments and increasing interactive response speed; and enhancing the response speed and resource utilization efficiency of interactive requests. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure; Figure 2 This is a flowchart illustrating a web page interaction method provided in an embodiment of this disclosure; Figure 3 This is a flowchart illustrating another web page interaction method provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of the structure of a web page interaction device provided in an embodiment of this disclosure; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation
[0018] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.
[0019] It should be noted that the user information (including but not limited to terminal device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.
[0020] A web page interaction method and apparatus according to embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings.
[0021] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure. The application scenario may include terminal devices 1, 2, and 3, server 4, and network 5.
[0022] Terminal devices 1, 2, and 3 can be hardware or software. When terminal devices 1, 2, and 3 are hardware, they can be various electronic devices with displays and supporting communication with server 4, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 1, 2, and 3 are software, they can be installed in the aforementioned electronic devices. Terminal devices 1, 2, and 3 can be implemented as multiple software programs or software modules, or as a single software program or software module; this disclosure does not limit this. Furthermore, various applications can be installed on terminal devices 1, 2, and 3, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.
[0023] Server 4 can be a server that provides various services, such as a backend server that receives requests sent by terminal devices with which it has established communication connections. This backend server can receive and analyze the requests sent by the terminal devices and generate processing results. Server 4 can be a single server, a server cluster consisting of several servers, or a cloud computing service center. This disclosure embodiment does not limit this.
[0024] It should be noted that server 4 can be either hardware or software. When server 4 is hardware, it can be various electronic devices that provide various services to terminal devices 1, 2, and 3. When server 4 is software, it can be multiple software programs or software modules that provide various services to terminal devices 1, 2, and 3, or it can be a single software program or software module that provides various services to terminal devices 1, 2, and 3. This disclosure does not limit the scope of the embodiments.
[0025] Network 5 can be a wired network using coaxial cable, twisted pair, and fiber optic connection, or it can be a wireless network that enables interconnection of various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), and Infrared. This disclosure does not limit the scope of the network.
[0026] Users can establish a communication connection with server 4 via network 5 through terminal devices 1, 2, and 3 to receive or send information. Specifically, server 4 can obtain interaction request text data through terminal devices 1, 2, and 3, perform semantic parsing processing on the interaction request text data based on preset layout distribution data to obtain intent label data and target existence probability values; perform safe avoidance segmentation processing on preset physical cutting grid coordinates based on intent label data and target existence probability values to obtain local image slice data; perform priority scanning scheduling processing on intent label data based on local image slice data to obtain target element coordinate data and accompanying cache data; and index and solidify the target element coordinate data and accompanying cache data to obtain interaction result data corresponding to the interaction request text data.
[0027] It should be noted that the specific types, quantities, and combinations of terminal devices 1, 2, and 3, server 4, and network 5 can be adjusted according to the actual needs of the application scenario, and this disclosure embodiment does not impose any restrictions on this.
[0028] Figure 2 This is a flowchart illustrating a web page interaction method provided in an embodiment of this disclosure. Figure 2 Web page interaction methods can be provided by Figure 1 The server executes the command. For example... Figure 2 As shown, the webpage interaction method includes: S201, based on the preset layout distribution data, perform semantic parsing processing on the interactive request text data to obtain intent label data and target existence probability values.
[0029] Specifically, the interactive request text data can be the instruction text input by the user in natural language. This interactive request text data can be used to trigger the entire webpage interaction process and serve as the input starting point for semantic parsing processing. The interactive request text data can include the user's original input content such as operation instructions and interaction goals in natural language form.
[0030] The preset layout distribution data can be statistical distribution data that records the probability of association between different intent types and webpage area locations. This preset layout distribution data can be used to provide prior knowledge support for semantic parsing processing, associating and mapping abstract intents with specific page spatial locations. This preset layout distribution data can include related information such as intent type, basic page type, and orientation probability value. This preset layout distribution data can be historical data obtained through statistical analysis of webpage layouts.
[0031] Furthermore, semantic parsing can be a data processing procedure that transforms unstructured user text commands into structured intent representation data. This semantic parsing can be used to parse and extract intent from interactive request text data to identify core semantic elements and calculate the probability of target existence. Based on preset layout distribution data, this semantic parsing can extract intent features and map directional probabilities from interactive request text data to obtain intent label data and target existence probability values.
[0032] The intent tag data can be identification data representing the core intent category extracted from the interaction request text data. This intent tag data can be used to guide the scanning direction of the page area. The intent tag data can contain structured elements such as intent category identifiers and semantic feature descriptions. The intent tag data can be obtained by extracting intent features from the interaction request text data through semantic parsing processing.
[0033] The target existence probability value can be a quantitative data of directional probability calculated by combining intent label data and preset layout distribution data. This target existence probability value can be used to provide a quantitative basis for generating a region scanning priority queue, guiding the priority scanning of the region where the target is most likely to exist. This target existence probability value can contain probability distribution information such as the probability values and sorting relationships of multiple preset directions. This target existence probability value can be obtained by performing probability mapping calculation on intent label data and preset layout distribution data through semantic parsing processing.
[0034] Furthermore, the semantic parsing process may specifically include extracting intent features from the interactive request text data to obtain intent feature data; performing orientation probability mapping on the intent feature data based on preset layout distribution data to obtain orientation probability values; sorting the orientation probability values to obtain target existence probability values; and performing label solidification on the intent feature data to obtain intent label data.
[0035] This application embodiment obtains intent feature data through intent feature extraction, maps and sorts the orientation probabilities based on preset layout distribution data to obtain the target existence probability value, and simultaneously solidifies the labels to obtain intent label data, thereby enhancing the ability to transform natural language commands into structured intent representations; improving the accuracy of the mapping between intent and page spatial location; and providing a quantitative basis for priority ranking for subsequent region scanning.
[0036] For example, in e-commerce search, the interactive request text data entered on the front-end interface can be "Help me search for Bluetooth headphones". This interactive request text data can be processed to extract intent features to identify search intent and product category, etc. Then, based on the preset layout distribution data, the intent feature data can be processed to perform directional probability mapping. Combined with the e-commerce page type, the probability value of the target existence is calculated to be the highest probability value in the top area, the second highest probability value in the middle area, and the lowest probability value in the bottom area. At the same time, the intent feature data is labeled and solidified to obtain intent label data.
[0037] S202, based on the intent label data and the target existence probability value, perform safe avoidance segmentation processing on the preset physical cutting grid coordinates to obtain local image slice data.
[0038] Specifically, the preset physical cutting grid coordinates can be a set of horizontal and vertical cutting line coordinates predefined according to the screen display area ratio. The preset physical cutting grid coordinates can serve as the starting framework for safe avoidance cutting processing, providing a basic boundary reference for subsequent dynamic adjustments. The preset physical cutting grid coordinates can include grid division information such as horizontal cutting line coordinate sets and vertical cutting line coordinate sets. The preset physical cutting grid coordinates can be generated by preset configuration based on screen ratio parameters.
[0039] Furthermore, the safe avoidance segmentation process can be a data processing procedure that corrects the initial mechanical grid by combining page structure features. This safe avoidance segmentation process can be used to dynamically adjust the coordinates of the cutting line to avoid the boundaries of key user interface elements and ensure the semantic integrity of local image slices. This safe avoidance segmentation process can perform node filtering, collision detection and dynamic offset adjustment on the preset physical cutting grid coordinates based on intent label data and target existence probability values to obtain local image slice data.
[0040] The local image slice data can be image region data corresponding to the logical safety boundary obtained by cropping from the original page image through safe avoidance segmentation processing. This local image slice data can be used to ensure the complete semantic features of user interface elements. The local image slice data can contain multiple complete image region blocks containing functional user interface elements. The local image slice data can be obtained by applying the adjusted logical safety boundary to the original page image and performing a cropping operation.
[0041] Furthermore, the safety avoidance segmentation process can specifically include: filtering nodes in the preset page structure data based on intent tag data to obtain semantic node data; extracting boundary coordinates from the semantic node data to obtain node boundary data; performing cross-collision detection on the node boundary data and the preset physical cutting grid coordinates to obtain collision determination data; dynamically offsetting the preset physical cutting grid coordinates based on the collision determination data to obtain logical safety boundary data; and performing image cropping on the logical safety boundary data to obtain local image slice data.
[0042] This application embodiment obtains semantic node data by filtering page structure data based on intent tag data, extracts node boundary coordinates and performs collision detection with the physical cutting mesh, and dynamically offsets the cutting line according to the collision determination to obtain a logical safety boundary. This allows the original page image to be cropped to obtain local image slice data, thereby enhancing the adaptive avoidance capability of the cutting line to the boundaries of key user interface elements and improving the semantic integrity and stability of the image slice.
[0043] For example, in news browsing, the interactive request text data can be "Find the latest news". Semantic parsing can be used to obtain the intent tag data "news browsing" and the probability value of the target's existence (the probability value is highest in the central area). Initial cutting line coordinates can be generated based on the physical cutting grid coordinates preset by the screen size. When performing safe avoidance segmentation, node filtering can be performed on the preset page structure data based on the intent tag data to identify semantic node data such as the news list container. Then, boundary coordinate extraction can be performed on the semantic node data to obtain node boundary data. Cross-collision detection can be performed on the node boundary data and the preset physical cutting grid coordinates to find that the vertical cutting line passes through the middle of the news list container, obtaining collision judgment data. Based on the collision judgment data, the preset physical cutting grid coordinates can be dynamically offset to slightly adjust the cutting line to the right edge of the container to obtain logical safety boundary data. Image cropping can be performed on the logical safety boundary data to generate local image slice data containing the complete news list area.
[0044] S203, based on the local image slice data, perform priority scanning and scheduling processing on the intent label data to obtain the target element coordinate data and accompanying cache data.
[0045] Specifically, priority scanning scheduling can be a data processing process that identifies and controls local image slice data sequentially according to a preset priority order. This priority scanning scheduling can be used to optimize resource allocation, prioritize scanning the areas where the target is most likely to exist, and improve overall processing efficiency by combining an early stop mechanism. This priority scanning scheduling can queue and identify target elements based on local image slice data to obtain target element coordinate data and accompanying cache data.
[0046] Among them, the target element coordinate data can be the physical position data of the interactive element on the screen corresponding to the user's intent, which is identified by the visual large model during the scanning process. The target element coordinate data can be used to provide the location basis for subsequent automated interactive operations. The target element coordinate data can include positioning elements such as boundary rectangle position information and element category identifier. The target element coordinate data can be obtained by reasoning and recognizing the highest priority local image slice data through priority scanning scheduling processing.
[0047] The accompanying cache data can be a set of semantic identifiers and coordinate data of other interactive elements in the same area that are synchronously acquired during the identification of the target element. This accompanying cache data can be used to build a local user interface element space cache pool. This accompanying cache data can contain index information such as the semantic names and coordinate boundaries of multiple interactive elements. This accompanying cache data can be a byproduct that is extracted together when the target element coordinate data is returned through priority scan scheduling.
[0048] Furthermore, the priority scan scheduling process may specifically include: performing queue arrangement processing on local image slice data based on the target existence probability value to obtain scan queue data; performing target element recognition processing on local image slice data based on scan queue data to obtain candidate element coordinate data; determining target element coordinate data based on candidate element coordinate data; and performing accompanying element extraction processing on target element coordinate data to obtain accompanying cache data.
[0049] This application embodiment obtains scan queue data by queuing slices based on the probability value of the target's existence, identifies target elements in the slices according to the queue order to obtain candidate element coordinate data, and then determines the target element coordinate data. At the same time, it extracts accompanying elements to obtain accompanying cache data, thereby enhancing the optimization capability of scan resource allocation based on probability priority; improving the efficiency of target element identification and the effectiveness of the early stopping mechanism; and realizing the synchronous acquisition and caching of target elements and surrounding interactive elements.
[0050] For example, in e-commerce search, local image slice data can be queued based on the probability value of the target's existence, with the top region slice placed at the beginning of the scan queue data. Then, target element recognition processing can be performed on the top region local image slice data based on the scan queue data, identifying the search box and returning candidate element coordinate data. The target element coordinate data can be determined based on the candidate element coordinate data. Simultaneously, accompanying element extraction processing can be performed on the target element coordinate data to obtain accompanying cache data containing elements such as the registration button and language switch.
[0051] S204, index and solidify the target element coordinate data and the accompanying cache data to obtain the interaction result data corresponding to the interaction request text data.
[0052] Specifically, index solidification processing can be a data processing process that organizes and persistently stores the target element coordinate data and the accompanying cache data according to a preset data structure. This index solidification processing can be used to build a structured local element information library. Based on the target element coordinate data and the accompanying cache data, the index solidification processing can perform key-value pair mapping and structured encapsulation to obtain the interaction result data corresponding to the interaction request text data.
[0053] The interaction result data can be a set of data that is associated with the intent expressed by the interaction request text data through indexing and solidification processing and is prepared to drive the automated interaction engine to perform operations. This interaction result data can be used to generate interaction operation instructions and send them to the browser or operating system level for execution. The interaction result data can include execution elements such as the coordinates of the target operation element and the operation type identifier. This interaction result data can be obtained by structuring the target element coordinate data and the accompanying cache data through indexing and solidification processing.
[0054] Furthermore, the index solidification process may specifically include key-value pair mapping of the accompanying cached data to obtain cached index data; structural encapsulation of the target element coordinate data and cached index data to obtain structured result data; and status annotation of the structured result data to obtain the interaction result data corresponding to the interaction request text data.
[0055] This application embodiment obtains cache index data by mapping accompanying cache data to key-value pairs, encapsulates the target element coordinate data and cache index data in a structured manner, and marks the structured results with status to obtain the interaction result data corresponding to the interaction request text data. This enhances the structured storage capability of the association between the target element and the accompanying element and improves the adaptability of the interaction result data to the execution of the automated interaction engine.
[0056] For example, in e-commerce search, priority scanning and scheduling can be used to obtain target element coordinate data (search box coordinates) and accompanying cache data (semantics and coordinates of elements such as registration buttons and language switches). Then, key-value pair mapping can be performed on the accompanying cache data, using element semantic descriptions as keys and coordinate data as values to construct cache index data. The target element coordinate data and cache index data can be structurally encapsulated to obtain structured result data containing target coordinates and cache indexes. Status annotation processing can be performed on the structured result data to obtain the interaction result data corresponding to the interaction request text data "Help me search for Bluetooth headphones". Cache data such as registration buttons can be stored in a local cache pool. When the user enters "click to register" later, the cache can be directly hit and the click can be executed without triggering page segmentation and visual large model recognition processes again.
[0057] According to the technical solution provided in this disclosure, semantic parsing is performed on the interactive request text data. Intent feature extraction, orientation probability mapping and sorting, and label solidification are used to obtain intent label data and target existence probability values. Based on the intent label data and target existence probability values, a safe avoidance segmentation process is performed on the preset physical cutting grid coordinates. Local image slice data is obtained through node filtering, boundary extraction, collision detection, dynamic offset, and image cropping. Priority scanning and scheduling processing is performed on the intent label data based on the local image slice data. Target element coordinate data and accompanying cache data are obtained through queue arrangement, target element recognition, and accompanying element extraction. The target element coordinates are then... The target data and accompanying cached data are indexed and solidified. Through key-value pair mapping, structured encapsulation, and state annotation, the interaction result data corresponding to the interaction request text data is obtained, thereby enhancing the ability to convert natural language instructions into structured intent representations; improving the adaptive avoidance capability of cutting lines to the boundaries of key user interface elements; enhancing the ability to optimize scanning resource allocation based on probability priority; improving the structured storage capability of the association between target elements and accompanying elements, thereby improving scanning efficiency and target hit probability; enhancing the recognition accuracy of large visual models; improving the semantic integrity of web page image fragments, thereby improving the interaction response speed; and enhancing the response speed and resource utilization efficiency of interaction requests.
[0058] In some embodiments, priority scanning scheduling processing is performed on intent label data based on local image slice data to obtain target element coordinate data and accompanying cache data, including: performing queue arrangement processing on local image slice data based on target existence probability values to obtain scan queue data; performing target element recognition processing on local image slice data based on scan queue data to obtain candidate element coordinate data; determining target element coordinate data based on candidate element coordinate data; and performing accompanying element extraction processing on target element coordinate data to obtain accompanying cache data.
[0059] Specifically, queue orchestration processing can be a data processing process that prioritizes and organizes scan sequences of image slice data based on azimuth probability quantization values. This queue orchestration processing can be used to transform disordered local image slice data into an ordered scan execution sequence. This queue orchestration processing can perform probability mapping and sequence orchestration on local image slice data based on target existence probability values to obtain scan queue data.
[0060] The scan queue data can be an ordered list of local image slices and their scan priorities, formed through queue orchestration processing. This scan queue data can be used to provide execution order constraints for target element recognition processing. The scan queue data can contain serialized data such as slice identifiers and priority sorting information. The scan queue data can be obtained by sorting local image slice data according to the probability value of the target presence through queue orchestration processing.
[0061] Furthermore, the queue arrangement process may specifically include sorting the target existence probability values by orientation to obtain priority sequence data; and performing slice mapping processing on the local image slice data based on the priority sequence data to obtain scan queue data.
[0062] Furthermore, the target element recognition process can be a data processing procedure that inputs the highest priority image slice into a large visual model for interactive element parsing and coordinate extraction. This target element recognition process can be used to perform semantic parsing and element localization on the highest priority local image slice in the scanning queue, and output preliminary recognition results. This target element recognition process can also perform visual parsing and coordinate extraction on the highest priority local image slice data based on the scanning queue data to obtain candidate element coordinate data, which serves as the basis for determining the target element coordinate data.
[0063] Among them, the candidate element coordinate data can be the preliminary result set data obtained by the visual large model to identify the current scanning area. The candidate element coordinate data can be used to provide a set of coordinates to be screened and semantic matching materials for determining the target element coordinate data. The candidate element coordinate data can contain the semantic information of multiple interactive elements and the recognition results such as the coordinates of the boundary rectangle. The candidate element coordinate data can be obtained by visually analyzing the highest priority local image slice data through target element recognition processing.
[0064] Furthermore, the target element recognition process may specifically include extracting the highest priority local image slice data from the scan queue data; performing visual analysis processing on the local image slice data to obtain the coordinate data of the candidate elements.
[0065] Furthermore, determining the target element coordinate data based on the candidate element coordinate data can be a data processing procedure that filters the coordinates of specific interactive elements corresponding to the user's intent from the candidate element set based on semantic matching principles. This determination process can be used to locate the target element that best matches the user intent label data from the candidate element coordinate data. This determination process can perform semantic comparison and matching filtering on the candidate element coordinate data based on the intent label data to obtain the target element coordinate data.
[0066] In addition, the accompanying element extraction process can be a data processing process that simultaneously extracts the semantic identifiers and coordinate information of other interactive elements in the same area during the identification of target elements. This accompanying element extraction process can be used to transform the byproducts of a single scan into reusable cache assets. This accompanying element extraction process can perform associated element identification and index construction based on the local image slice data corresponding to the target element coordinate data to obtain accompanying cache data.
[0067] Furthermore, the accompanying element extraction process may specifically include performing associated element identification processing on the local image slice data corresponding to the target element coordinate data to obtain associated element data; and performing key-value pair mapping processing on the associated element data to obtain accompanying cache data.
[0068] For example, in e-commerce search, local image slice data containing the top search box area, the middle product list area, and the bottom navigation bar area can be queued based on the probability value of the target's existence (highest probability value in the top area, followed by the middle area, and lowest probability value in the bottom area), with the top area slice placed at the top of the scan queue data. Then, based on the scan queue data, target element recognition processing can be performed on the top area local image slice data. The visual big data model identifies the search box and returns candidate element coordinate data containing the search box coordinates. The semantic information of the search box in the candidate element coordinate data can be matched with the intent tag data "search" to determine that the search box coordinates are the target element coordinate data. Simultaneously, accompanying element extraction processing can be performed on the target element coordinate data to identify elements such as the registration button and language switch in the same top slice and construct accompanying cache data in key-value pair format.
[0069] According to the technical solution provided in this disclosure, local image slice data is queued based on the probability value of the target's existence. Scan queue data is obtained through orientation sorting and slice mapping. Target element recognition is performed on the local image slice data based on the scan queue data. The highest priority slice is extracted for visual analysis to obtain candidate element coordinate data. Based on the candidate element coordinate data, target element coordinate data is determined through semantic matching. Accompanying element extraction is performed on the target element coordinate data, simultaneously extracting other interactive elements within the same area and constructing key-value pairs to obtain accompanying cache data. This enhances the probability-priority-based scanning resource allocation and queueing capabilities, improves the efficiency of target element recognition and the accuracy of semantic matching, and achieves synchronous acquisition and cache construction of target elements and surrounding interactive elements.
[0070] In some embodiments, a safe avoidance segmentation process is performed on the preset physical cutting grid coordinates based on intent tag data and target existence probability values to obtain local image slice data. This includes: performing node filtering processing on preset page structure data based on intent tag data to obtain semantic node data; performing boundary coordinate extraction processing on the semantic node data to obtain node boundary data; performing cross-collision detection processing on the node boundary data and the preset physical cutting grid coordinates to obtain collision determination data; performing dynamic offset processing on the preset physical cutting grid coordinates based on the collision determination data to obtain logical safety boundary data; and performing image cropping processing on the logical safety boundary data to obtain local image slice data.
[0071] Specifically, the preset page structure data can be the underlying structure description data of the current webpage. This preset page structure data can be used to provide complete page element hierarchy information and structural relationship description for node filtering processing. This preset page structure data can include structured descriptions such as node hierarchy relationships and node attribute information. This preset page structure data can be the document object model tree structure data obtained through the browser interface.
[0072] In addition, node filtering can be a data processing process that identifies and filters nodes with specific interactive or structural semantics from page structure data based on semantic constraints. This node filtering process can be used to locate key interactive elements and structural containers from page structure data. This node filtering process can perform semantic matching and node type filtering on preset page structure data based on intent tag data to obtain semantic node data.
[0073] The semantic node data can be a set of nodes with specific interactive or structural semantics obtained through node filtering. This semantic node data can be used to provide a set of node objects for the boundary extraction process. The semantic node data can include interactive semantic nodes such as input box nodes, button nodes, link nodes, and form container nodes. The semantic node data can be obtained by semantically filtering the preset page structure data through node filtering.
[0074] Furthermore, the node filtering process may specifically include parsing the preset page structure data to obtain node type data; and performing semantic matching on the node type data based on intent tag data to obtain semantic node data.
[0075] In addition, the boundary coordinate extraction process can be a data processing procedure that extracts the physical boundary information of the semantic nodes in the screen coordinate system from the filtered semantic nodes. This boundary coordinate extraction process can be used to transform abstract node objects into quantifiable coordinate data. This boundary coordinate extraction process can extract absolute rectangular coordinates from the semantic node data based on the browser application programming interface to obtain the node boundary data.
[0076] Among them, node boundary data can be a set of coordinates representing the specific physical area occupied by key interactive elements or structural elements on the screen. This node boundary data can be used to provide spatial location information of nodes for cross-collision detection processing, serving as the basis for determining whether a cutting line passes through a node. This node boundary data can include rectangular boundary elements such as left edge coordinates, top edge coordinates, right edge coordinates, and bottom edge coordinates. This node boundary data can be obtained by extracting coordinates from semantic node data through boundary coordinate extraction processing.
[0077] In addition, the cross collision detection process can be a data processing process that compares the node boundary with the initial cutting grid line in space to determine whether the cutting line passes through the node. This cross collision detection process can be used to identify the position of the cutting line that causes the element to break. This cross collision detection process can perform horizontal and vertical breaking judgments on the preset physical cutting grid coordinates based on the node boundary data to obtain collision judgment data.
[0078] The collision determination data can be the determination result data that records the intersection status and positional relationship between the cutting line and the node rectangle. This collision determination data can be used to provide avoidance direction and offset basis for dynamic offset processing, and guide the intelligent adjustment of the cutting line. The collision determination data can include determination elements such as intersection status identifier and intersection position information. The collision determination data can be obtained by spatially comparing the node boundary data and the preset physical cutting mesh coordinates through intersection collision detection processing.
[0079] Furthermore, the cross-collision detection process can specifically include performing horizontal shredding judgment processing on the node boundary data and the preset physical cutting grid coordinates to obtain horizontal collision data; performing vertical shredding judgment processing on the node boundary data and the preset physical cutting grid coordinates to obtain vertical collision data; and performing collision aggregation processing on the horizontal collision data and the vertical collision data to obtain collision determination data.
[0080] In addition, dynamic offset processing can be used to offset the cutting line passing through the node to the outside of the node edge to avoid the node in the data processing process based on the collision determination result. This dynamic offset processing can be used to transform the initially straight grid line into an elastic boundary that fits the edge of the page element, ensuring the semantic integrity of the image after cutting. This dynamic offset processing can calculate and adjust the coordinates of the preset physical cutting grid coordinates based on the collision determination data according to the principle of proximity wrapping, to obtain logical safety boundary data.
[0081] The logical safety boundary data can be a set of cutting line coordinates that fit the edges of user interface elements on the page, obtained through dynamic offset processing. This logical safety boundary data can be used to provide semantically safe region division boundaries for image cropping, ensuring that each cropping region fully contains functional elements. The logical safety boundary data can include flexible grid information such as adjusted horizontal and vertical cutting line coordinates. The logical safety boundary data can be obtained by adjusting the preset physical cutting grid coordinates through dynamic offset processing to avoid them.
[0082] Furthermore, the dynamic offset processing can specifically include performing collision direction parsing on the collision determination data to obtain offset direction data; performing edge distance calculation on the preset physical cutting grid coordinates based on the offset direction data to obtain edge distance values; and performing neighbor offset processing on the preset physical cutting grid coordinates based on the edge distance values to obtain logical safety boundary data.
[0083] In addition, image cropping can be a data processing process that crops corresponding local images from the original page image based on the regions divided according to the logical security boundary. This image cropping can be used to transform a complete page image into multiple semantically complete local image slices. This image cropping can perform region cropping and slice generation on the original page image based on the logical security boundary data to obtain local image slice data.
[0084] For example, in e-commerce search, node filtering can be performed on preset page structure data based on the intent tag data "search" to identify semantic node data such as search box nodes and login button nodes. Then, boundary coordinate extraction processing can be performed on the semantic node data to obtain the boundary data of search box nodes and login button nodes. Cross-collision detection processing can be performed on the node boundary data and preset physical cutting grid coordinates to find that the vertical cutting line passes through the middle of the search box node, obtaining collision judgment data containing the cross-state indicator. Based on the collision judgment data, the preset physical cutting grid coordinates can be dynamically offset to the right to the outside of the right edge of the search box to obtain logically safe boundary data that fits the boundary of the search box. Image cropping processing can be performed on the logically safe boundary data to generate local image slice data containing the complete search box.
[0085] According to the technical solution provided in this disclosure, node filtering processing is performed on preset page structure data based on intent tag data. Semantic node data is obtained through node type parsing and semantic matching. Boundary coordinate extraction processing is performed on the semantic node data. The physical boundary of the node is obtained through the browser interface to obtain node boundary data. Cross-collision detection processing is performed on the node boundary data and preset physical cutting grid coordinates. Collision judgment data is obtained through horizontal and vertical fragmentation judgment and collision aggregation. Dynamic offset processing is performed on the preset physical cutting grid coordinates based on the collision judgment data. Logical safety boundary data is obtained through collision direction parsing, edge distance calculation and neighbor offset. Image cropping processing is performed on the logical safety boundary data to generate local image slice data, thereby enhancing the adaptive avoidance capability of the cutting line to the boundary of key user interface elements and improving the semantic integrity and stability of the image slice.
[0086] In some embodiments, semantic parsing processing of interactive request text data based on preset layout distribution data is performed to obtain intent label data and target existence probability values, including: performing intent feature extraction processing on interactive request text data to obtain intent feature data; performing orientation probability mapping processing on intent feature data based on preset layout distribution data to obtain orientation probability values; sorting the orientation probability values to obtain target existence probability values; and performing label solidification processing on intent feature data to obtain intent label data.
[0087] Specifically, intent feature extraction processing can be a data processing process that transforms unstructured user text commands into abstract feature representations that characterize the core operational purpose. This intent feature extraction processing can be used to perform semantic parsing and feature extraction on interactive request text data to identify the core semantic elements in user commands. This intent feature extraction processing can also extract semantic elements and vectorize features from interactive request text data based on preset parsing rules to obtain intent feature data.
[0088] Among them, intent feature data can be abstract feature representation data that characterizes the user's core operation purpose, obtained through intent feature extraction processing. This intent feature data can be used to provide semantic input for orientation probability mapping processing and to provide basic classification materials for label solidification processing. This intent feature data can contain abstract representations such as semantic feature vectors and structured intent elements. This intent feature data can be obtained by semantic parsing of interactive request text data through intent feature extraction processing.
[0089] Furthermore, the intent feature extraction process may specifically include performing semantic element parsing on the interactive request text data to obtain semantic element data; and performing feature vectorization on the semantic element data to obtain intent feature data.
[0090] Furthermore, the orientation probability mapping process can be a data processing process that queries the prior statistical distribution based on the semantic features of the intent and maps the intent to the orientation probability of the page. This orientation probability mapping process can be used to transform abstract intent features into specific spatial orientation probability predictions. This orientation probability mapping process can perform intent type matching and orientation probability query on intent feature data based on preset layout distribution data to obtain orientation probability values.
[0091] The directional probability value can be probability data representing the likelihood of a target element existing in a specific location on a webpage, obtained through directional probability mapping. This directional probability value can be used to provide probability materials to be sorted for the sorting process. This directional probability value can contain probability distribution information of multiple preset directional values. This directional probability value can be obtained by performing a correlation query on the intent feature data and the preset layout distribution data through directional probability mapping.
[0092] Furthermore, the orientation probability mapping process may specifically include performing intent type determination processing on the intent feature data to obtain intent type data; and performing orientation probability query processing on the intent type data based on preset layout distribution data to obtain orientation probability values.
[0093] Furthermore, the sorting process can be a data processing procedure that arranges multiple directional probability values in descending order based on their probability values. This sorting process can be used to transform unordered directional probability values into an ordered probability sequence. This sorting process can arrange and organize the directional probability values in descending order based on their probability values to obtain the probability value of the target's existence.
[0094] Furthermore, the label solidification process can be a data processing procedure that converts continuous intent feature data into discrete intent category identifiers with clear semantics. This label solidification process can be used to transform abstract feature representations into machine-recognizable category labels. Based on preset category mapping rules, this label solidification process can classify, encode, and determine the labels of intent feature data to obtain intent label data.
[0095] For example, in e-commerce search, the interactive request text data could be "Help me search for sneakers." This interactive request text data can be processed by semantic element parsing to identify semantic element data such as search intent and product category. Then, feature vectorization processing can be performed on the semantic element data to obtain intent feature data representing the search intent. Based on preset layout distribution data, the intent feature data can be processed by directional probability mapping. Through intent type determination processing, the intent type data can be determined to be a search type. Then, based on preset layout distribution data, directional probability query processing can be performed on the intent type data to obtain directional probability values with the highest probability value at the top, followed by the middle, and the lowest probability value at the bottom. These directional probability values can then be sorted to obtain the target existence probability values in descending order. Simultaneously, the intent feature data can be labeled and solidified to generate intent label data "Search."
[0096] According to the technical solution provided in this disclosure, intent feature extraction processing is performed on interactive request text data. Through semantic element parsing and feature vectorization, intent feature data is obtained. Based on preset layout distribution data, directional probability mapping processing is performed on the intent feature data. Through intent type determination and directional probability query, directional probability values are obtained. The directional probability values are sorted to obtain target existence probability values. The intent feature data is then labeled and solidified. Intent label data is obtained through classification coding. This enhances the ability to convert natural language instructions into structured intent representations, improves the accuracy of the mapping between intent and page spatial location, and provides a quantitative basis for priority ranking for subsequent area scanning.
[0097] In some embodiments, the target element coordinate data and the accompanying cache data are indexed and solidified to obtain the interaction result data corresponding to the interaction request text data. This includes: performing key-value pair mapping on the accompanying cache data to obtain cache index data; performing structured encapsulation on the target element coordinate data and the cache index data to obtain structured result data; and performing state annotation on the structured result data to obtain the interaction result data.
[0098] Specifically, key-value pair mapping can be a data processing process that organizes the semantic identifiers of interactive elements in the accompanying cache data and coordinate boundaries into a key-value association form. This key-value pair mapping can be used to transform discrete accompanying cache data into an index structure that is easy to retrieve quickly. This key-value pair mapping can perform key-value association and index construction based on the semantic names and coordinate boundaries in the accompanying cache data to obtain cache index data.
[0099] The cache index data can be a data structure formed by key-value pair mapping, with semantic names as keys and coordinate boundaries as values. This cache index data can be used to provide an indexed cache information set for structured encapsulation processing. The cache index data can contain index elements such as semantic key names and coordinate value pairs of multiple interactive elements. The cache index data can be obtained by extracting semantic names and associating coordinate boundaries with the accompanying cache data through key-value pair mapping.
[0100] Furthermore, the key-value pair mapping process can specifically include extracting semantic names from the accompanying cache data to obtain semantic key name data; extracting coordinate boundaries from the accompanying cache data to obtain coordinate value data; and associating the semantic key name data and coordinate value data to obtain cache index data.
[0101] Furthermore, structured encapsulation processing can be a data processing procedure that integrates target element coordinates and cache index information into a unified data packet. This structured encapsulation processing can be used to completely encapsulate the identification results and cache assets of a single interaction, forming a traceable and reusable state data carrier. This structured encapsulation processing can integrate fields and encapsulate formats based on target element coordinate data and cache index data to obtain structured result data.
[0102] The structured result data can be an integrated data package containing target element coordinates and cache index information, formed through structured encapsulation processing. This structured result data can be used to provide a complete interactive context state record for state annotation processing, carrying all recognition results and cache states of a single interaction. This structured result data can contain structured elements such as target element coordinate fields, cache index fields, and timestamp identifiers. This structured result data can be obtained by integrating the fields of target element coordinate data and cache index data through structured encapsulation processing.
[0103] Furthermore, the structured encapsulation process can specifically include encapsulating the core fields of the target element coordinate data to obtain the target coordinate field data; encapsulating the index fields of the cached index data to obtain the cached index field data; and integrating the target coordinate field data and the cached index field data to obtain the structured result data.
[0104] In addition, status labeling can be a data processing procedure that adds status identifiers to data packets based on the interaction execution status and cache update status. This status labeling can be used to clarify the final processing result and system status of a single interaction request. This status labeling can perform status determination and label addition based on the interaction execution information and cache status information in the structured result data to obtain the interaction result data.
[0105] Furthermore, the status labeling process can specifically include performing interactive execution determination processing on the structured result data to obtain execution status data; performing cache update determination processing on the structured result data to obtain cache status data; and appending status labels to the execution status data and cache status data to obtain interactive result data.
[0106] For example, in e-commerce search, priority scanning and scheduling processes yield target element coordinate data (search box coordinates) and accompanying cached data (semantic names and coordinate boundaries of elements such as the registration button, language switch, and shopping cart icon). The accompanying cached data can then be mapped using key-value pairs. Semantic name extraction can be performed to obtain semantic key name data, followed by coordinate boundary extraction to obtain coordinate value data. Key-value association between the semantic key name data and coordinate value data results in cached index data with "register button," "language switch," and "shopping cart icon" as keys and their corresponding coordinate boundaries as values. Furthermore, the target element coordinate data and cached index data can be structurally encapsulated. Core field encapsulation of the target element coordinate data yields target coordinate field data, which can then be used to index the cached index data. The encapsulation process yields cached index field data. The target coordinate field data and cached index field data can be formatted and integrated to obtain structured result data containing search box coordinates and cached indexes. Status annotation processing can be applied to this structured result data. Interaction execution judgment processing can then be performed to obtain execution status data indicating target location. Furthermore, cache update judgment processing can be performed to obtain cached status data indicating cache update. Status markers can be appended to the execution status data and cached status data to obtain the interaction result data corresponding to the interaction request text data "Help me search for sneakers," labeled "Operation ready and cache updated." This data can be used to drive the automated interaction engine to perform input operations at the search box location. Simultaneously, its embedded cached index data can provide a directly accessible cache retrieval basis for subsequent user input of "Click shopping cart."
[0107] According to the technical solution provided in this disclosure, key-value pair mapping is performed on the accompanying cache data. After semantic name extraction and coordinate boundary extraction, key-value association is performed to obtain cache index data. The target element coordinate data and cache index data are structurally encapsulated. After core field encapsulation and index field encapsulation, the format is integrated to obtain structured result data. The structured result data is status-labeled. After interaction execution determination and cache update determination, a status identifier is attached to obtain interaction result data. This enhances the structured storage capability of the association between the target element and the accompanying element and improves the adaptability of the interaction result data to the execution of the automated interaction engine.
[0108] In some embodiments, dynamic offset processing is performed on the preset physical cutting grid coordinates based on collision determination data to obtain logical safety boundary data, including: performing collision direction parsing processing on the collision determination data to obtain offset direction data; performing edge distance calculation processing on the preset physical cutting grid coordinates based on the offset direction data to obtain edge distance values; and performing neighbor offset processing on the preset physical cutting grid coordinates based on the edge distance values to obtain logical safety boundary data.
[0109] Specifically, collision direction analysis processing can be a data processing process that extracts collision direction information from collision determination data to determine the offset direction of the cutting line. This collision direction analysis processing can be used to transform abstract collision determination results into specific directional guidance. Based on the intersection position information in the collision determination data, this collision direction analysis processing can determine the horizontal and vertical cutting directions to obtain offset direction data.
[0110] Among them, the offset direction data can be the direction information data representing the offset required of the cutting line obtained by collision direction analysis. This offset direction data can be used to provide directional constraints for edge distance calculation. This offset direction data can include directional elements such as horizontal offset direction identifier and vertical offset direction identifier. This offset direction data can be obtained by extracting the intersection direction from the collision determination data through collision direction analysis.
[0111] Furthermore, the collision direction analysis process can specifically include performing horizontal cross-judgment processing on the collision determination data to obtain horizontal collision data; performing vertical cross-judgment processing on the collision determination data to obtain vertical collision data; and performing direction aggregation processing on the horizontal collision data and the vertical collision data to obtain offset direction data.
[0112] In addition, the edge distance calculation process can be a data processing procedure that calculates the distance from the cutting line to the node edge based on the offset direction to determine the offset magnitude. This edge distance calculation process can be used to convert the direction information into a quantifiable distance value. This edge distance calculation process can perform distance quantization and magnitude calculation on the preset physical cutting grid coordinates and node boundary data based on the offset direction data to obtain the edge distance value.
[0113] The edge distance value can be quantized data representing the specific distance from the cutting line to the node edge, obtained through edge distance calculation. This edge distance value can be used to provide the offset basis for neighbor offset processing. The edge distance value can include distance quantization information such as horizontal and vertical distance values. The edge distance value can be obtained by quantizing the distance between the preset physical cutting grid coordinates and node boundary data through edge distance calculation.
[0114] Furthermore, the edge distance calculation process can specifically include performing horizontal distance calculation on the preset physical cutting grid coordinates and node boundary data based on the offset direction data to obtain a horizontal distance value; performing vertical distance calculation on the preset physical cutting grid coordinates and node boundary data based on the offset direction data to obtain a vertical distance value; and performing distance aggregation processing on the horizontal and vertical distance values to obtain an edge distance value.
[0115] In addition, the neighbor offset processing can be a data processing process that offsets the cutting line to the outside of the node edge based on the edge distance value to form a safe boundary. This neighbor offset processing can be used to transform the initially straight grid line into an elastic boundary that fits the edge of the page element, ensuring the semantic integrity of the image after cutting. This neighbor offset processing can calculate and adjust the coordinates of the preset physical cutting grid coordinates based on the edge distance value according to the principle of proximity wrapping, to obtain logical safe boundary data.
[0116] Furthermore, the adjacent offset processing can specifically include performing a safety margin superposition process on the preset physical cutting grid coordinates based on the edge distance value to obtain offset data; and performing coordinate adjustment processing on the preset physical cutting grid coordinates based on the offset data to obtain logical safety boundary data.
[0117] For example, in e-commerce search, collision detection data (vertical cutting lines passing through the middle of the search box node) is obtained. This collision data can then be parsed for collision direction. Horizontal collision detection can determine if there is no horizontal collision, while vertical collision detection can determine if there is. The horizontal and vertical collision data can be aggregated to obtain the vertical offset direction data. Based on this offset direction data, edge distance calculations can be performed on the preset physical cutting grid coordinates and the search box node boundary data. Horizontal distance calculations can be performed to obtain the horizontal distance value, and vertical distance calculations can be performed to obtain the vertical distance value (the cutting line is closer to the right edge of the search box). The horizontal and vertical distance values can be aggregated to obtain the edge distance value. Based on the edge distance value, the preset physical cutting grid coordinates can be offset by proximity processing, and a safety margin overlay process can be performed to obtain offset data including the reserved margin. Based on the offset data, the preset physical cutting grid coordinates can be adjusted to shift the vertical cutting line to the right outside the right edge of the search box, resulting in logical safety boundary data that fits the search box boundary.
[0118] According to the technical solution provided in this disclosure, collision direction analysis is performed on collision determination data. Offset direction data is obtained through horizontal intersection determination, vertical intersection determination, and direction aggregation. Edge distance calculation is performed on preset physical cutting grid coordinates based on the offset direction data. Edge distance values are obtained through horizontal distance calculation, vertical distance calculation, and distance aggregation. Neighbor offset processing is performed on preset physical cutting grid coordinates based on the edge distance values. Logical safety boundary data is obtained through safety margin superposition and coordinate adjustment. This improves the accuracy of the cutting line collision direction recognition and offset strategy; enhances the adaptive avoidance capability of the cutting line to the boundaries of key user interface elements; and ensures the semantic integrity and stability of the cut image slices.
[0119] In some embodiments, the node boundary data and the preset physical cutting grid coordinates are subjected to cross-collision detection processing to obtain collision determination data, including: performing area quantization processing on the node boundary data to obtain node area values; performing threshold comparison processing on the node area values and the preset screen area ratio threshold to obtain valid node data; and performing boundary cross-detection processing on the valid node data and the preset physical cutting grid coordinates to obtain collision determination data.
[0120] Specifically, area quantization can be a quantization data processing process that calculates the physical coverage of a node on the screen based on the node boundary coordinates. This area quantization can be used to convert abstract boundary coordinates into comparable area values. This area quantization can calculate the coverage and generate area values based on the edge coordinates in the node boundary data to obtain the node area value.
[0121] The node area value can be a quantized numerical data representing the physical coverage of a node on the screen, obtained through area quantization processing. This node area value can be used to provide a comparable numerical basis for threshold comparison processing and serve as a quantification standard for determining whether a node is an ultra-large background node. The node area value can include measurement information such as the area value of a single node. The node area value can be obtained by calculating the coverage of the edge coordinates in the node boundary data through area quantization processing.
[0122] Furthermore, the area quantization process can specifically include performing horizontal span calculation on the node boundary data to obtain the horizontal span value; performing vertical span calculation on the node boundary data to obtain the vertical span value; and performing area aggregation on the horizontal span value and the vertical span value to obtain the node area value.
[0123] In addition, the threshold comparison process can be a data processing procedure that compares the node area value with a preset screen area ratio threshold to filter out oversized background nodes. This threshold comparison process can be used to filter out effective nodes with interactive semantics from the node set and exclude invalid avoidance caused by touching oversized background nodes. This threshold comparison process can perform threshold comparison and node filtering on the node area value based on the preset screen area ratio threshold to obtain effective node data.
[0124] The preset screen area ratio threshold can be a pre-set standard data for defining whether a node belongs to an oversized background node. This preset screen area ratio threshold can be used to provide a judgment benchmark for threshold comparison processing, serving as a quantitative boundary to distinguish between valid nodes and background nodes. The preset screen area ratio threshold can include threshold configuration information such as area ratio values and judgment rules. The preset screen area ratio threshold can be obtained by setting it according to page characteristic parameters through preset configuration.
[0125] The effective node data can be a set of node boundary data whose area does not exceed a preset threshold, obtained through threshold comparison processing. This effective node data can be used to provide a set of node objects to be detected for boundary intersection detection processing, ensuring that collision detection is only performed on key elements with interactive semantics. The effective node data can contain node information such as multiple filtered node boundary coordinates and node type identifiers. The effective node data can be obtained by comparing and filtering the node area value with a preset screen area ratio threshold through threshold comparison processing.
[0126] Furthermore, the threshold comparison process may specifically include performing threshold area calculation processing on the preset screen area ratio threshold and the total screen area data to obtain threshold area data; and performing numerical comparison processing on the node area value and the threshold area data to obtain valid node data.
[0127] In addition, the boundary intersection detection process can be a data processing process that compares the effective node boundary with the initial cutting grid line in space to determine whether the cutting line passes through the node. This boundary intersection detection process can be used to identify the cutting line position that may cause the element to be cut, and provide a basis for avoidance triggering for subsequent dynamic offset. This boundary intersection detection process can perform horizontal and vertical cutting judgment on the preset physical cutting grid coordinates based on the effective node data to obtain collision judgment data.
[0128] Furthermore, the boundary intersection detection process can specifically include performing horizontal shredding judgment processing on the valid node data and the preset physical cutting grid coordinates to obtain horizontal collision data; performing vertical shredding judgment processing on the valid node data and the preset physical cutting grid coordinates to obtain vertical collision data; and performing collision aggregation processing on the horizontal collision data and the vertical collision data to obtain collision judgment data.
[0129] For example, in e-commerce search, boundary coordinate extraction is used to obtain the boundary data of the search box node and the full-screen background container node. Then, area quantization processing can be performed on the node boundary data to calculate the horizontal span values of the search box and the background container. Similarly, vertical span calculation can be performed to obtain the vertical span values of the search box and the background container. These horizontal and vertical span values can then be aggregated to obtain the area values of the search box node and the background container node. Finally, the node area values can be compared with a preset screen area ratio threshold to calculate the threshold area. The data is then processed by comparing the area values of the search box nodes and the background container nodes with the threshold area data, filtering out background container nodes whose area exceeds the threshold, and obtaining valid node data (containing only search box nodes). The valid node data and the preset physical cutting grid coordinates can be subjected to boundary cross-detection processing, and horizontal shredding judgment processing can be performed to obtain horizontal collision data without horizontal collision. Then, vertical shredding judgment processing can be performed to obtain vertical collision data where vertical cutting lines pass through the search box nodes. The horizontal collision data and vertical collision data can be subjected to collision aggregation processing to obtain collision judgment data containing the state where vertical cutting lines pass through the search box nodes.
[0130] According to the technical solution provided in this disclosure, the node boundary data is quantized by area calculation. The node area value is obtained through horizontal span calculation, vertical span calculation, and area aggregation. The node area value is then compared with a preset screen area ratio threshold. Valid node data is obtained through threshold area calculation and numerical comparison. The valid node data is then subjected to boundary cross-detection processing with preset physical cutting grid coordinates. Collision detection data is obtained through horizontal slicing judgment, vertical slicing judgment, and collision aggregation. This improves the accuracy of node validity screening, avoids interference from oversized background nodes on collision detection, enhances the accuracy of cross-collision detection of cutting lines on the boundaries of key interactive elements, and provides a precise collision triggering basis for subsequent dynamic offsets.
[0131] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.
[0132] Figure 3 This is a schematic diagram of another web page interaction method provided in an embodiment of this disclosure. For example... Figure 3 As shown, the webpage interaction method includes: 1. Eye contact prediction and priority ranking based on semantic intent: Intent Feature Extraction: Receive the user's natural language instructions (interactive request text data) (e.g., "Help me search for [product name]"). Extract the core intent tag (e.g., Intent="Search") using a lightweight local natural language processing model (NLP).
[0133] Target Existence Probability Calculation: Maintain a prior statistical distribution library of webpage user interface (UI) layout (e.g., search intent is usually located at the top center of the page, login intent is usually located in the upper right corner or central area). Combining intent tags and the basic type of the current page, calculate the "target existence probability (target existence probability value)" for several preset coarse-grained orientations (top, bottom, left, right, center).
[0134] Generate a scan queue: Generate a list of regions to be scanned on the current page, ranked from highest to lowest probability.
[0135] 2. Intelligent segmentation and "safe avoidance" segmentation based on multimodal fusion: Obtain the initial physical mesh and multimodal features: Generate an initial physical cutting mesh (e.g., a 3×3 mesh) according to the screen ratio, obtaining a set of initial horizontal cutting line coordinates Y_cut and vertical cutting line coordinates X_cut. Simultaneously, instead of relying solely on image pixels, the underlying Document Object Model Tree (DOM) (or accessibility auxiliary tree) of the current page is retrieved, and edge detection using computer vision algorithms is combined to extract the page's structural features.
[0136] Extracting the true boundary coordinates of DOM nodes: This involves using front-end scripts to prioritize nodes with interactive semantics (such as...). <input> , <button>, ) and outer container nodes with complete structural semantics (such as <form> , <header>Call the browser's native Application Programming Interface (API) to obtain the absolute physical rectangular coordinates of the node on the screen, i.e., [left edge X_min, top edge Y_min, right edge X_max, bottom edge Y_max].
[0137] Cross-collision detection (determining if shredded): Traverse the key node coordinates obtained above, and compare the initial cutting mesh lines with the node rectangles: Horizontal slicing judgment: If a horizontal cutting line satisfies Y_min < Y_cut < Y_max, it indicates that the element is cut off vertically.
[0138] Vertical slicing judgment: If a vertical cutting line satisfies X_min < X_cut < X_max, it indicates that the element is cut off to the left or right.
[0139] Filtering excessively large background nodes (exception handling): To prevent the dividing line from triggering invalid avoidance or even infinite expansion due to touching a "full-screen background image" or an "excessively large outer container", an area threshold is set. When the area of a DOM node that collides exceeds a preset proportion (e.g., 20%) of the total visible screen area, it is determined to be a background layer or an irrelevant structure, and the system ignores the collision.
[0140] The algorithm for triggering the "dynamic avoidance" adjustment of the grid lines is as follows: When a valid UI element (such as a key input box) is confirmed to cross a grid line, the coordinates of the grid lines are dynamically adjusted according to the "nearest wrapping" principle. Taking the vertical line X_cut passing through the input box as an example, compare the distances of X_cut from the left edge X_min and the right edge X_max of the node.
[0141] If it is closer to the left edge, then shift the cutting line to the left by X_min. Δ (Δ is a reserved pixel safety margin, such as 5px).
[0142] If it is closer to the right edge, the cutting line will be shifted to the right by X_max+Δ.
[0143] Similarly, fine-tune the horizontal cutting line up and down until the cutting line completely avoids all valid DOM nodes on that line.
[0144] Generate logically safe region slices: After the above "collision detection" and "fine-tuning avoidance", the initially straight and rigid grid lines are transformed into "flexible grids" (i.e., logically safe boundaries) that fit the edges of the page UI elements. The system performs local image cropping based on these defined logically safe boundaries, ensuring that the segmented local image slices completely contain the entire Header tag or Form area, without containing any broken interactive elements.
[0145] 3. Dynamic scheduling and early shutdown: On-demand allocation: Extract the top-ranked slice (e.g., the header area at the top of the page) from the generated priority queue and input it into the Visual Model (VLM) for target element (search box) recognition.
[0146] Early stop execution: If the VLM finds the corresponding search box in the current first-order slice and returns the coordinates, it immediately terminates and discards the scanning plan of the remaining blocks, and directly executes the interactive action.
[0147] Dynamic window expansion mechanism: If the absolute target is not found in the current block, but a strongly related element is found.
[0148] Example scenario: The user command could be "Enter password". No password field is found in the current slice, but VLM recognizes the "Forgot password" button or "Account input field".
[0149] Processing logic: At this point, "dynamic window expansion" is triggered. Using the coordinates of the currently discovered "related element" as the center, a surrounding sub-slice (e.g., expanding 200px) is obtained for further verification. This is because in UI design principles, elements with strong semantic relevance are necessarily close in physical location.
[0150] 4. Build a UI element space cache: Byproduct extraction: When VLM searches for "target element X" in scan area A, in addition to returning the coordinates of X, it can also return the names and coordinates of other clearly visible significant interactive elements in area A (such as "register button" and "language switch button"), which adds almost no extra overhead to VLM.
[0151] Dictionary writing: Elements that are not currently needed are stored in the local memory "space cache pool" in the form of Key (semantic / text)-Value (coordinate boundary).
[0152] Cache hit: When the user's next instruction arrives (e.g., "Then click register"), the local cache pool is retrieved. If a cache hit occurs, the coordinates can be retrieved from the cache and the click can be executed.
[0153] According to the technical solution provided in this disclosure, by combining lightweight NLP intent parsing with UI layout statistical probability, the machine is given the visual intuition that humans use when searching for things, ensuring that the target is hit and early stopping is triggered during the first or second slice scan; by obtaining safety boundaries through cross-modal (lower code structure and upper visual rendering) alignment, it is ensured that the local image sent to the large model always has complete semantic features; the local field of view can be dynamically expanded through related elements in the current area, avoiding context breakage caused by area switching and improving the localization fault tolerance rate.
[0154] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.
[0155] Figure 4 This is a schematic diagram of a web page interaction device provided in an embodiment of this disclosure. Figure 4 As shown, the web page interactive device includes: The first processing module 401 is used to perform semantic parsing processing on the interactive request text data based on the preset layout distribution data to obtain intent label data and target existence probability value; The second processing module 402 is used to perform safe avoidance segmentation processing on the preset physical cutting grid coordinates based on the intent label data and the target existence probability value to obtain local image slice data. The third processing module 403 is used to perform priority scanning and scheduling processing on the intent label data based on the local image slice data to obtain the target element coordinate data and the accompanying cache data. The fourth processing module 404 is used to index and solidify the target element coordinate data and the accompanying cache data to obtain the interaction result data corresponding to the interaction request text data.
[0156] According to the technical solution provided in this disclosure, semantic parsing is performed on the interactive request text data. Intent feature extraction, orientation probability mapping and sorting, and label solidification are used to obtain intent label data and target existence probability values. Based on the intent label data and target existence probability values, a safe avoidance segmentation process is performed on the preset physical cutting grid coordinates. Local image slice data is obtained through node filtering, boundary extraction, collision detection, dynamic offset, and image cropping. Priority scanning and scheduling processing is performed on the intent label data based on the local image slice data. Target element coordinate data and accompanying cache data are obtained through queue arrangement, target element recognition, and accompanying element extraction. The target element coordinates are then... The target data and accompanying cached data are indexed and solidified. Through key-value pair mapping, structured encapsulation, and state annotation, the interaction result data corresponding to the interaction request text data is obtained, thereby enhancing the ability to convert natural language instructions into structured intent representations; improving the adaptive avoidance capability of cutting lines to the boundaries of key user interface elements; enhancing the ability to optimize scanning resource allocation based on probability priority; improving the structured storage capability of the association between target elements and accompanying elements, thereby improving scanning efficiency and target hit probability; enhancing the recognition accuracy of large visual models; improving the semantic integrity of web page image fragments, thereby improving the interaction response speed; and enhancing the response speed and resource utilization efficiency of interaction requests.
[0157] In some embodiments, the third processing module 403 is specifically used to: perform queue arrangement processing on local image slice data based on the target existence probability value to obtain scan queue data; perform target element recognition processing on local image slice data based on scan queue data to obtain candidate element coordinate data; determine target element coordinate data based on candidate element coordinate data; and perform accompanying element extraction processing on target element coordinate data to obtain accompanying cache data.
[0158] In some embodiments, the second processing module 402 is specifically used to: perform node filtering processing on preset page structure data based on intent tag data to obtain semantic node data; perform boundary coordinate extraction processing on semantic node data to obtain node boundary data; perform cross-collision detection processing on node boundary data and preset physical cutting grid coordinates to obtain collision determination data; perform dynamic offset processing on preset physical cutting grid coordinates based on collision determination data to obtain logical safety boundary data; and perform image cropping processing on logical safety boundary data to obtain local image slice data.
[0159] In some embodiments, the first processing module 401 is specifically used to: perform intent feature extraction processing on the interactive request text data to obtain intent feature data; perform orientation probability mapping processing on the intent feature data based on preset layout distribution data to obtain orientation probability values; sort the orientation probability values to obtain target existence probability values; and perform label solidification processing on the intent feature data to obtain intent label data.
[0160] In some embodiments, the fourth processing module 404 is specifically used to perform key-value pair mapping processing on the accompanying cached data to obtain cached index data; perform structured encapsulation processing on the target element coordinate data and cached index data to obtain structured result data; and perform state annotation processing on the structured result data to obtain interaction result data.
[0161] In some embodiments, the dynamic offset processing of preset physical cutting grid coordinates based on collision determination data to obtain logical safety boundary data is specifically used for: performing collision direction parsing processing on collision determination data to obtain offset direction data; performing edge distance calculation processing on preset physical cutting grid coordinates based on offset direction data to obtain edge distance values; and performing neighbor offset processing on preset physical cutting grid coordinates based on edge distance values to obtain logical safety boundary data.
[0162] In some embodiments, the collision determination data is obtained by performing cross-collision detection processing on node boundary data and preset physical cutting grid coordinates. Specifically, this involves performing area quantization processing on node boundary data to obtain node area values; performing threshold comparison processing on node area values and preset screen area ratio thresholds to obtain valid node data; and performing boundary cross-detection processing on valid node data and preset physical cutting grid coordinates to obtain collision determination data.
[0163] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.
[0164] Figure 5 This is a schematic diagram of the electronic device 5 provided in an embodiment of this disclosure. Figure 5 As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, it implements the steps in the various method embodiments described above. Alternatively, when the processor 501 executes the computer program 503, it implements the functions of each module / unit in the various device embodiments described above.
[0165] Electronic device 5 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 5 may include, but is not limited to, processor 501 and memory 502. Those skilled in the art will understand that... Figure 5 This is merely an example of electronic device 5 and does not constitute a limitation on electronic device 5. It may include more or fewer components than shown, or different components.
[0166] The processor 501 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0167] The memory 502 can be an internal storage unit of the electronic device 5, such as a hard disk or RAM of the electronic device 5. The memory 502 can also be an external storage device of the electronic device 5, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 5. The memory 502 can also include both internal and external storage units of the electronic device 5. The memory 502 is used to store computer programs and other programs and data required by the electronic device.
[0168] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0169] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.
[0170] The above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit it. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be included within the protection scope of this disclosure.< / header> < / form> < / button>
Claims
1. A webpage interaction method, characterized in that, include: Based on the preset layout distribution data, semantic parsing processing is performed on the interactive request text data to obtain intent label data and target existence probability values; Based on the intent label data and the target existence probability value, a safe avoidance segmentation process is performed on the preset physical cutting grid coordinates to obtain local image slice data; Based on the local image slice data, the intent label data is subjected to priority scanning and scheduling processing to obtain target element coordinate data and accompanying cache data; The target element coordinate data and the accompanying cache data are indexed and solidified to obtain the interaction result data corresponding to the interaction request text data.
2. The webpage interaction method according to claim 1, characterized in that, The priority scanning and scheduling process of the intent label data based on the local image slice data to obtain target element coordinate data and accompanying cache data includes: Based on the probability value of the existence of the target, the local image slice data is queued to obtain scan queue data; Based on the scan queue data, target element recognition processing is performed on the local image slice data to obtain candidate element coordinate data; Based on the candidate element coordinate data, determine the target element coordinate data; The target element coordinate data is subjected to accompanying element extraction processing to obtain the accompanying cache data.
3. The webpage interaction method according to claim 1, characterized in that, The step of performing safe avoidance segmentation processing on the preset physical cutting grid coordinates based on the intent label data and the target presence probability value to obtain local image slice data includes: Based on the intent tag data, the preset page structure data is filtered to obtain semantic node data; The semantic node data is processed by extracting boundary coordinates to obtain node boundary data; The node boundary data and the preset physical cutting mesh coordinates are subjected to cross-collision detection processing to obtain collision determination data; Based on the collision determination data, the preset physical cutting grid coordinates are dynamically offset to obtain logical safety boundary data; The logical security boundary data is processed by image cropping to obtain the local image slice data.
4. The webpage interaction method according to claim 1, characterized in that, The semantic parsing processing of the interactive request text data based on the preset layout distribution data yields intent tag data and target presence probability values, including: The interaction request text data is subjected to intent feature extraction processing to obtain intent feature data; Based on the preset layout distribution data, the intention feature data is subjected to directional probability mapping processing to obtain directional probability values; The location probability values are sorted to obtain the target existence probability values. The intent feature data is subjected to label solidification processing to obtain the intent label data.
5. The webpage interaction method according to claim 1, characterized in that, The step of indexing and solidifying the target element coordinate data and the accompanying cache data to obtain the interaction result data corresponding to the interaction request text data includes: The accompanying cached data is processed by key-value pair mapping to obtain cache index data; The target element coordinate data and the cache index data are encapsulated in a structured manner to obtain structured result data; The structured result data is processed with state annotation to obtain the interaction result data.
6. The webpage interaction method according to claim 3, characterized in that, The step of dynamically offsetting the preset physical cutting mesh coordinates based on the collision determination data to obtain logical safety boundary data includes: The collision determination data is processed by collision direction parsing to obtain offset direction data; Based on the offset direction data, the edge distance is calculated on the preset physical cutting grid coordinates to obtain the edge distance value; The logical safety boundary data is obtained by performing a proximity offset process on the preset physical cutting grid coordinates based on the edge distance value.
7. The webpage interaction method according to claim 3, characterized in that, The step of performing cross-collision detection processing on the node boundary data and the preset physical cutting mesh coordinates to obtain collision determination data includes: The node boundary data is subjected to area quantization processing to obtain the node area value; The node area value is compared with a preset screen area ratio threshold to obtain valid node data. The effective node data and the preset physical cutting grid coordinates are subjected to boundary intersection detection processing to obtain the collision determination data.
8. A webpage interactive device, characterized in that, include: The first processing module is used to perform semantic parsing processing on the interactive request text data based on the preset layout distribution data to obtain intent label data and target existence probability value; The second processing module is used to perform safe avoidance segmentation processing on the preset physical cutting grid coordinates based on the intent label data and the target existence probability value to obtain local image slice data. The third processing module is used to perform priority scanning and scheduling processing on the intent label data based on the local image slice data to obtain target element coordinate data and accompanying cache data. The fourth processing module is used to index and solidify the target element coordinate data and the accompanying cache data to obtain the interaction result data corresponding to the interaction request text data.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.
10. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.
