Webpage automation interaction method, device and electronic equipment

Abstract

Embodiments of the present application provide a webpage automatic interaction method and device and electronic equipment, and relate to the technical field of webpage automation. The webpage automatic interaction method comprises: obtaining first image information corresponding to a target webpage and task description information of an automatic interaction task performed on the target webpage; the first image information comprises webpage image and webpage structure information; inputting the first image information and the task description information into a pre-trained gaze point prediction model to predict position information of a target gaze point on the target webpage; determining a task execution strategy of the automatic interaction task according to the position information of the target gaze point, the webpage structure information and the task description information, and interacting with the target webpage based on the task execution strategy. The present application can simulate the visual attention behavior of a person in a complex webpage, dynamically focus the operation object of the target webpage on a key visual area, and improve the precision and efficiency of webpage automatic interaction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of webpage automation, and in particular to a webpage automation interaction method and device and electronic equipment. BACKGROUND

[0002] At present, webpage autonomous GUI (Graphical User Interface, graphical user interface) agents usually analyze webpage structures through rules or visual models (such as target detection, optical character recognition OCR, etc.), and perform simulated clicks, inputs and other interactions based on task targets. However, when facing content-intensive, complex-structured or high-resolution webpages, these agent methods lack a “human-like” attention mechanism, and it is difficult to accurately determine the key interaction areas on the webpage, and problems such as misoperation and inefficient navigation are prone to occur. Specifically, webpage agents cannot make the following actions like humans: dynamically focusing on key visual areas on the webpage (such as CTA (Call-To-Action, call-to-action) buttons, important texts, etc.), selecting the most reasonable next operation according to the current task semantics and interface layout, maintaining accurate and personalized visual understanding and behavior planning in high-resolution scenarios. Therefore, there is a lack of a method that can accurately predict the task intention of a webpage automation task and guide the task action. SUMMARY

[0003] Embodiments of the present application provide a webpage automation interaction method, device and electronic equipment to alleviate or solve one or more technical problems existing in the prior art.

[0004] In a first aspect, embodiments of the present application provide a webpage automation interaction method, comprising:

[0005] obtaining first image information corresponding to a target webpage and task description information of an automation interaction task performed on the target webpage; the first image information includes webpage images and webpage structure information;

[0006] inputting the first image information and the task description information into a pre-trained fixation point prediction model, and predicting position information of a target fixation point on the target webpage through the fixation point prediction model;

[0007] determining a task execution strategy of the automation interaction task according to the position information of the target fixation point, the webpage structure information and the task description information, and interacting with the target webpage based on the task execution strategy.

[0008] In a second aspect, embodiments of the present application provide a webpage automation interaction device, comprising:

[0009] The acquisition module is configured to acquire first image information corresponding to a target webpage and task description information of an automatic interaction task performed on the target webpage, wherein the first image information comprises webpage image and webpage structure information.

[0010] The prediction module is configured to input the first image information and the task description information into a pre-trained gaze point prediction model, and predict position information of a target gaze point on the target webpage by using the gaze point prediction model.

[0011] The determination and execution module is configured to determine a task execution strategy of the automatic interaction task according to the position information of the target gaze point, the webpage structure information and the task description information, and perform interaction with the target webpage based on the task execution strategy.

[0012] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory. The processor implements any of the methods provided in the embodiments of the present application when executing the computer program.

[0013] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement any of the methods provided in the embodiments of the present application.

[0014] In a fifth aspect, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement any of the methods provided in the embodiments of the present application.

[0015] According to the technical scheme of the embodiments of the present application, the first image information corresponding to the target webpage and the task description information of the automatic interaction task performed on the target webpage are acquired, and the first image information comprises webpage image and webpage structure information. The first image information and the task description information are input into a pre-trained gaze point prediction model, and the position information of the target gaze point on the target webpage is predicted by using the gaze point prediction model. Then, the position information of the target gaze point, the webpage structure information and the task description information are used to determine the task execution strategy of the automatic interaction task, and the target webpage is interacted based on the task execution strategy. It can be seen that the technical scheme can predict the position information of the target gaze point on the target webpage by using the pre-trained gaze point prediction model, realize the simulation of the user's attention, simulate the visual attention behavior of the human in the complex webpage, dynamically focus the operation object of the target webpage on the key visual area, and avoid the webpage misoperation. In addition, the position information of the predicted target gaze point is used to guide the generation of the task execution strategy to guide the webpage automatic interaction, thereby improving the precision and efficiency of the webpage automatic interaction decision.

[0016] The above description is only a summary of the technical solutions of the present application. In order to enable a clearer understanding of the technical means of the present application, the present application can be implemented in accordance with the content of the description, and in order to enable the above and other purposes, features and advantages of the present application to be more apparent and easy to understand, the specific embodiments of the present application are described below. BRIEF DESCRIPTION OF DRAWINGS

[0017] In the drawings, like reference numerals refer to same or similar components throughout the several views. These drawings are not necessarily to scale. It should be understood that these drawings only depict certain embodiments in accordance with the present application and should not be considered limiting of the scope of the present application.

[0018] Figure 1 A flow chart of a webpage automatic interaction method provided by an embodiment of the present application is shown;

[0019] Figure 2 A structure diagram of a gaze point prediction model provided by an embodiment of the present application is shown;

[0020] Figure 3 A block diagram of a webpage automatic interaction system provided by an embodiment of the present application is shown;

[0021] Figure 4 A block diagram of a webpage automatic interaction apparatus provided by an embodiment of the present application is shown;

[0022] Figure 5 A block diagram of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] In the following, only certain exemplary embodiments are simply described. As those skilled in the art can recognize, the described embodiments can be modified in various different ways without departing from the concept or scope of the present application. Therefore, the drawings and the description are considered to be exemplary in nature, rather than limiting.

[0024] In order to facilitate understanding of the technical solutions of the embodiments of the present application, the related technologies of the embodiments of the present application are described below. The following related technologies can be combined with the technical solutions of the embodiments of the present application in any way as optional solutions, which all belong to the protection scope of the embodiments of the present application.

[0025] In the following, the following terms will be used:

[0026] Webpage autonomous GUI agent: an AI (Artificial Intelligence)-based intelligent agent capable of actively understanding webpage content and dynamically making decisions. Webpage GUI refers to a webpage interface that interacts with users through visual elements (such as icons, buttons, menus, windows, etc.), allowing users to interact with the webpage through intuitive operations such as clicking and dragging, without the need to input complex commands.

[0027] The technical solutions of the present application and how the technical solutions of the present application solve the aforementioned technical problems will be described in detail below with specific examples. Several specific examples listed can be combined with each other, and for the same or similar concepts or processes, some examples can not be described in detail. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0028] Figure 1 A flowchart of a webpage automation interaction method provided by an embodiment of the present application is shown, as shown in Figure 1 The method can include the following steps S101, S102 and S103.

[0029] Step S101, obtaining first image information corresponding to a target webpage, and task description information for performing an automation interaction task on the target webpage, the first image information including webpage images and webpage structure information.

[0030] Optionally, the webpage images can be obtained by taking a screenshot of the target webpage. The target webpage refers to the current visible area when performing the webpage automation interaction task, and the webpage image can be obtained by taking a screenshot of the webpage interface image of the current visible area.

[0031] In some embodiments, the taken webpage interface image can be preprocessed to preserve the original resolution of the webpage image as much as possible, avoiding the ambiguity of low-resolution heat maps. The webpage image acquisition method can include the following steps:

[0032] First, take a screenshot of the target webpage to obtain a webpage interface image.

[0033] Second, based on a preset segmentation method, the webpage interface image is segmented into multiple sub-interface images. The preset segmentation method can include at least one of the following: segmentation size, segmentation ratio, and segmentation times. The segmentation size can be represented as the size of each sub-interface image. The segmentation ratio refers to the proportion of the segmented sub-interface image to the webpage interface image. Assuming that the segmentation ratio is 12x1, i.e., the webpage interface image is segmented into 12 rows and 1 column of cells. Taking the segmentation times as an example, assuming that the segmentation times is 5, the webpage interface image is segmented 5 times, and multiple sub-interface images are obtained.

[0034] Again, based on the preset image format requirements, the image processing is performed on each sub-interface image respectively to obtain the webpage image. The preset image format requirements include the requirement for the image size, which can be set according to the reference size compatible with the gaze point prediction model. For example, if the reference size compatible with the gaze point prediction model is 448*448, the multiple sub-interface images are scaled to the reference size of 448*448, and the webpage image is obtained.

[0035] The webpage structure information can be represented as a webpage DOM structure, which includes but is not limited to at least one of the following information: node type (such as document node, element node, attribute node, etc.), element information (such as node tag name, node attribute, style, class name, etc.), hierarchical relationship (such as parent node, child node, etc.), node content (such as text content, form element value, etc.), data attribute (such as custom data), dynamic modification information (such as creating a node, deleting a node, modifying an attribute, updating a style, etc.), and the like.

[0036] In step S102, the first image information and the task description information are input into the pre-trained gaze point prediction model, and the position information of the target gaze point on the target webpage is predicted by the gaze point prediction model.

[0037] The target gaze point includes multiple target gaze points, and according to the information of the multiple target gaze points, the target gaze area on the target webpage can be located. The position information of the target gaze point can be represented as position coordinates.

[0038] In step S103, the task execution strategy of the automated interaction task is determined according to the position information of the target gaze point, the webpage structure information and the task description information, and the target webpage is interacted based on the task execution strategy.

[0039] The task execution strategy can include a webpage interaction element and a webpage interaction action. When the target webpage is interacted based on the task execution strategy, the webpage interaction action can be performed on the webpage interaction element on the target webpage based on the task execution strategy.

[0040] According to the technical scheme of the embodiment of the present application, the first image information corresponding to the target webpage and the task description information of performing the automatic interactive task on the target webpage are acquired, the first image information includes webpage image and webpage structure information; the first image information and the task description information are input into the pre-trained gaze point prediction model, and the position information of the target gaze point on the target webpage is predicted by the gaze point prediction model; then, the task execution strategy of the automatic interactive task is determined according to the position information of the target gaze point, the webpage structure information and the task description information, and the target webpage is interacted based on the task execution strategy. It can be seen that the technical scheme can predict the position information of the target gaze point on the target webpage through the pre-trained gaze point prediction model, realize the human-like simulation of user attention, simulate the visual attention behavior of people in the complex webpage, dynamically focus the operation object of the target webpage on the key visual area, and avoid the situation of webpage misoperation. In addition, the position information of the predicted target gaze point is used to guide the generation of the task execution strategy to guide the webpage automatic interactive, thereby improving the precision and efficiency of the webpage automatic interactive decision.

[0041] In some embodiments, before the first image information corresponding to the target webpage and the task description information of performing the automatic interactive task on the target webpage are acquired (i.e., step S101 is performed), the gaze point prediction model is pre-trained.

[0042] The training process of the gaze point prediction model can include the following steps A1 to A4:

[0043] Step A1, acquiring a sample data set, the sample data set including: sample image information corresponding to a sample webpage, sample task description information and sample gaze point information.

[0044] The sample image information includes sample webpage image and sample webpage structure information. The sample webpage image can be obtained by taking a screenshot of the sample webpage. The sample webpage refers to the current visible area when performing a certain automatic interactive task, and the sample webpage image can be obtained by taking a screenshot of the sample webpage interface of the current visible area. The sample webpage structure information can be represented as a sample webpage DOM structure, and the sample webpage DOM structure includes but is not limited to at least one of the following information: node type (such as document node, element node, attribute node, etc.), element information (such as node tag name, node attribute, style, class name, etc.), hierarchical relationship (such as parent node, child node, etc.), node content (such as text content, form element value, etc.), data attribute (such as custom data), dynamic modification information (such as creating a node, deleting a node, modifying an attribute, updating a style, etc.), etc. The sample gaze point information includes sample position information and sample quantity of the sample gaze point.

[0045] Step A2, inputting the sample data set into the gaze point prediction model to be trained, and predicting the sample gaze point by the gaze point prediction model to be trained to obtain gaze point prediction information.

[0046] The gaze point prediction information includes predicted position information and a predicted number of the sample gaze point.

[0047] Step A3, calculating a total reward value of the gaze point prediction model to be trained according to the gaze point prediction information and the sample gaze point information.

[0048] Step A4, training the gaze point prediction model to be trained based on the total reward value to obtain a trained gaze point prediction model.

[0049] In this embodiment, step A3 can be performed as steps A31 to A33:

[0050] Step A31, calculating a format consistency reward value of the gaze point prediction model to be trained according to the degree of fit between the predicted number and the sample number.

[0051] Optionally, the calculation of the format consistency reward value can be represented by the following formula:

[0052]

[0053] In the above formula, represents the format consistency reward value, represents a basic reward, represents an additional reward, represents the predicted number in the gaze point prediction information, represents the sample number in the sample gaze point information. The above formula indicates that if there is a case of missing special markers in the gaze point prediction information, the format consistency reward value is set to zero. The special markers can be markers that must be included in advance, for example, the output data of the model is pre-set to contain " <ref>Number of gaze points: N< / ref> ", indicating the termination of the predicted number, and " <ref>" indicates the start of the prediction number, in "< / ref> " belongs to the special markers. If any of the special markers is missing in the gaze point prediction information, for example, if " <ref>" and "< / ref> " is missing, it can be determined that the gaze point prediction information format output by the model is incorrect, and the format consistency reward value is determined to be zero. If there is no case of missing special markers in the gaze point prediction information, the basic reward and the additional reward can be obtained, and the additional reward can be adjusted according to the degree of fit between the predicted number of the sample gaze point and the sample number. The values of the basic reward and the additional reward can be set by the user, for example, the value of the basic reward is 0.2, and the value of the additional reward is 0.8.

[0054] Step A32, according to the spatial distance between the predicted position information and the sample position information, calculate the spatial consistency reward value of the gaze point prediction model to be trained.

[0055] Optionally, the calculation of the spatial consistency reward value can be represented as the following formula:

[0056]

[0057] In the above formula, represents the spatial consistency reward value, represents the maximum possible square distance in the normalized coordinate space, represents the predicted number of sample gaze points. represents the predicted position information of the sample gaze point, represents the sample position information of the sample gaze point, j is the true number of sample gaze points (i.e. the sample number), T represents the normalized coordinate space. The above formula indicates that if the spatial positions corresponding to the predicted position information and the sample position information are close, the spatial consistency reward value will be close to 1, and with the increase of the error, the spatial consistency reward value will be exponentially attenuated. It can be understood that the spatial consistency reward mechanism is to minimize the Euclidean distance between the spatial positions corresponding to the predicted position information and the sample position information, and this mechanism encourages accurate spatial positioning by punishing large position differences, thereby improving the accuracy of gaze point prediction.

[0058] Step A33, according to the format consistency reward value and the spatial consistency reward value, calculate the total reward value.

[0059] Optionally, the calculation of the total reward value R can be represented as the following formula:

[0060]

[0061] The calculation method of the total reward value indicates that by combining the format consistency reward and the spatial consistency reward, the compound reward mechanism can guide the model to generate format consistent and spatial accurate output, while ensuring strong generalization ability under different input conditions. Of course, the format consistency reward value and the spatial consistency reward value can also be configured with their respective weights, so as to obtain the total reward value R by calculating the weighted sum of the format consistency reward value and the spatial consistency reward value.

[0062] Figure 2 Fig. 1 shows a structural diagram of a gaze point prediction model provided by an embodiment of the present application, as Figure 2As shown, the gaze point prediction model comprises a visual encoder, a text encoder and a prediction module. The visual encoder is configured to extract visual features of sample image information; the text encoder is configured to learn a semantic mapping relationship between sample task description information and sample gaze point information; the prediction module is configured to explicitly indicate a task type or a task target currently required to be executed by the gaze point prediction model, and predict position information and a number of sample gaze points according to the visual features extracted by the visual encoder and the semantic mapping relationship learned by the text encoder. Optionally, the prediction module is configured to structure the position coordinates and the number of the sample gaze points, and train the gaze point prediction model according to the predicted position coordinates and the number of the sample gaze points.

[0063] The sample gaze points usually comprise a plurality of gaze points, and position coordinates of the plurality of sample gaze points can be expressed in the following set form: [[230, 310], [345, 420],...].

[0064] Optionally, the gaze point prediction information output by the gaze point prediction model can be expressed in the following form: <point> ， <ref>), wherein <point>predicted position coordinates of a sample gaze point, <ref>This indicates the number of predicted fixations in the sample. For example, the fixation prediction model outputs the following data:

[0065] <ref>Number of gaze points: 9< / ref>

[0066] <point>[[x1, y1], [x2, y2],..., [x9, y9]]< / point>

[0067] In the example above, " <ref>" and "< / ref> The content between the quotation marks indicates the predicted number of fixations in the sample. <point>" and "< / point> The content between the quotation marks indicates the predicted location coordinates of the sample gaze point.

[0068] When inputting a sample dataset into a gaze prediction model to be trained, a tokenizer can be used to convert the sample dataset into a token sequence, thereby forming a unified semantic input space as the basis for subsequent model inference.

[0069] The following is a specific example of a sample dataset used to input a gaze prediction model. The sample dataset consists of at least two parts: one part contains sample image information, and the other part contains sample task description information and sample gaze information.

[0070] The sample image information includes sample webpage images and sample webpage structure information. Sample webpage images can be obtained by taking screenshots of the sample webpage. Optionally, first, a screenshot of the sample webpage is taken to obtain the sample webpage interface image. Second, based on a preset segmentation method, the sample webpage interface image is segmented into multiple sample sub-interface images. Third, based on preset image format requirements, image processing is performed on each sample sub-interface image to obtain the sample webpage image.

[0071] Sample task description information and sample fixation information can be represented in the following format:

[0072] {

[0073] "page_id":"page_0001",

[0074] "task": "Enter login information"

[0075] "task_id":"task_0001",

[0076] "frames":[

[0077] {

[0078] "second":0,

[0079] "fixations":[

[0080] {"x":410, "y":265, "user_id":"user_0012"},

[0081] {"x":420, "y":272, "user_id":"user_0021"}, ... ]

[0083] } ]

[0085] }

[0086] In each sample data set, the gaze point coordinates of multiple users of the sample web page under a certain automated interaction task can be included. The ""task":"fill in login information"" is the sample task description information, and the ""user_id":"user_0012"" is the identification information of the user 0012, and the ""user_id":"user_0021"" is the identification information of the user 0021. The gaze point coordinates of the user 0012 of the sample web page under the current automated interaction task are (410, 265), and the gaze point coordinates of the user 0021 are (420, 272).

[0087] Optionally, the sample task description information can also be recorded separately in the sample data set, so as to more clearly guide the behavior of the automated interaction task. For example, in addition to the sample image information and the sample task description information containing the sample gaze point information in the above example, the following sample task description information can also be recorded separately:

[0088] {

[0089] "task":"please find and click the 'contact customer service' button in the web page.",

[0090] "task_id":"task_0001",

[0091] }

[0092] It should be noted that in the above example, the format of the input data and the output data of the gaze point prediction model is only illustrative, and in actual application, the format of the input data and the output data of the gaze point prediction model can be customized according to actual needs.

[0093] In this embodiment, the visual features of the sample image information are extracted, and the semantic mapping relationship between the sample task description information and the sample gaze point information is learned, so as to predict the position information and the number of the sample gaze points by using the visual features of the sample image information and the semantic mapping relationship between the task description information and the sample gaze point information, and train the gaze point prediction model based on the predicted position coordinates and the number of the sample gaze points, so that the trained gaze point prediction model can accurately predict the position information and the number of the target gaze points on the target webpage based on the first image information of the target webpage and the task description information of the automated interactive task, thereby simulating the visual attention behavior of a person in a complex webpage, dynamically focusing the operation object of the target webpage on the key visual area, avoiding the situation of interactive target offset when interacting with the webpage, and improving the precision and efficiency of automated interaction.

[0094] In some embodiments, the task execution strategy includes a webpage interactive element and a webpage interactive action. When the task execution strategy of the automated interactive task is determined according to the position information of the target gaze point, the webpage structure information and the task description information (i.e., step S103 is performed), the following steps B1 and B2 are performed:

[0095] Step B1, spatially mapping the position information of the target gaze point and the webpage structure information, and determining the webpage interactive element according to the spatial mapping result.

[0096] The webpage structure information includes the position information of each point on the target webpage and the webpage DOM structure. When step B1 is performed, the target gaze point can be first mapped to the target webpage according to the position information of the target gaze point and the position information of each point on the target webpage, so that multiple target gaze points can be mapped to a region on the target webpage, which is the key gaze region. Then, the key gaze region and the webpage DOM structure are compared. Since the webpage DOM structure includes the position information of each webpage element, the webpage element corresponding to the key gaze region can be determined by comparison, and the webpage element corresponding to the key gaze region is the webpage interactive element.

[0097] Step B2, determining the webpage interactive action according to the webpage image, the task description information and the webpage interactive element.

[0098] Optionally, the webpage interactive action can also be determined in combination with the position information of the target gaze point. For example, the webpage interactive action is: scrolling to the bottom, inputting text, clicking XX button, etc.

[0099] In some embodiments, after interacting with the target webpage based on the task execution strategy, steps C1 to C3 can also be performed.

[0100] Step C1, obtaining a webpage interaction result of interacting with the target webpage, the webpage interaction result comprising an interface diagram of the interaction result.

[0101] Step C2, determining whether the automated interaction task is executed successfully according to the task description information and the webpage interaction result.

[0102] The webpage interaction result comprises, but is not limited to, the following types: DOM update, loading of a new webpage, appearance of a pop-up window, etc.

[0103] Optionally, when step C2 is executed, the target interaction result of executing the automated interaction task on the target webpage is determined according to the task description information; then it is determined whether the webpage interaction result and the target interaction result match, and in the case where the webpage interaction result and the target interaction result match, it is determined that the automated interaction task is executed successfully.

[0104] Optionally, the webpage interaction result comprises a current webpage actually reached after the automated interaction task is executed, and the target interaction result comprises a final webpage that should be reached after the automated interaction task is executed. When it is determined whether the webpage interaction result and the target interaction result match, the current webpage and the final webpage can be compared to determine whether a similarity thereof reaches a threshold. In the case where the similarity of the current webpage and the final webpage reaches the preset threshold, it is determined that the webpage interaction result and the target interaction result match.

[0105] Optionally, the webpage interaction result comprises webpage elements contained in the current webpage reached after the automated interaction task is executed. The target interaction result comprises key content, such as a key option, a key text, a key interaction portal, etc., that should be contained in the final webpage. When it is determined whether the webpage interaction result and the target interaction result match, the webpage elements contained in the current webpage and the key content that should be contained in the final webpage can be compared to determine whether the key content is contained in the webpage elements contained in the current webpage, and if so, it is determined that the webpage interaction result and the target interaction result match, otherwise, it is determined that the webpage interaction result and the target interaction result do not match.

[0106] Step C3, if the automated interaction task is not executed successfully, second image information of the current webpage is obtained, and the current webpage is interacted based on the second image information.

[0107] If the automated interaction task is not executed successfully, the next automated interaction action is continued to be executed based on the current webpage and the task description information. The implementation manner of each automated interaction action can refer to the method described in any of the above embodiments, and thus is not repeated. The automated interaction task is executed through the iterative mechanism of automation, so that the execution of the webpage automated interaction task is more intelligent and automated, and the user does not need to manually participate, and the multi-round context-aware interaction task can be completed.

[0108] Figure 3 A block diagram of the automated web page interaction system provided in an embodiment of this application is shown, such as... Figure 3 As shown, the automated web page interaction system includes: a web page image acquisition module, a multimodal embedding and task fusion module, a gaze point prediction module, a web page interaction decision module, and a task execution module.

[0109] The webpage image acquisition module is used to acquire the first image information corresponding to the target webpage, including the webpage image and webpage structure information. The webpage image acquisition module preprocesses the captured webpage interface image to ensure that the processed webpage image retains as much of the original resolution as possible.

[0110] The multimodal embedding and task fusion module is used to construct a unified text-visual encoding architecture using large language models (such as InternVL, VideoLLaMA, etc.). The text-visual encoding architecture includes image modal data and text modal data. The image modal data includes standardized first-image information, including webpage images in formats compatible with the large language model. The text modal data includes task description information, such as "login account," "find discounted items," and "close pop-up." The multimodal embedding and task fusion module also converts the image and text modal data into token sequences, forming a unified semantic input space as the basis for subsequent model inference.

[0111] The fixation prediction module is equipped with a fixation prediction model. The model structure of the fixation prediction model can be found in [reference needed]. Figure 2 As shown, this will not be repeated here. During the model training phase, the input data for the gaze prediction model includes: sample image information corresponding to the sample webpage, sample task description information, and sample gaze point information. During the model application phase (or inference phase), the input data for the gaze prediction model includes the first image information and task description information corresponding to the target webpage. The output data for the gaze prediction model includes the location information and number of gaze points.

[0112] The webpage interaction decision module is used to determine the task execution strategy for automated interaction tasks based on the target gaze point's location information, webpage structure information, and task description information. The task execution strategy includes webpage interaction elements and webpage interaction actions. Specifically, firstly, the target gaze point's location information and webpage structure information are spatially mapped, and the webpage interaction elements are determined based on the spatial mapping results; secondly, the webpage interaction actions are determined based on the webpage images, task description information, and webpage interaction elements.

[0113] The task execution module is configured to automatically interact with the target webpage according to the task execution strategy determined by the webpage interaction decision module, obtain webpage interaction results, and determine whether to perform visual perception and gaze point prediction in the next round according to the webpage interaction results and the task description information, thereby ensuring the success of the execution of the automatic interaction task, that is, ensuring that the execution result of the automatic interaction task is consistent with the target interaction result.

[0114] In addition, the webpage automatic interaction system further comprises a task acquisition module configured to acquire task description information of the automatic interaction task.

[0115] Optionally, the webpage automatic interaction system further comprises a user attribute management module configured to manage user attribute information of a plurality of users, which includes but is not limited to the following information: user name, user account, user unique identifier, and the like. In the prediction stage of the gaze point prediction model, the user attribute management module can provide user attribute information of a plurality of users and gaze point position information of each user when the gaze point prediction model executes a certain automatic interaction task, thereby providing label information with a supervised role for model training.

[0116] In the prediction stage of the gaze point prediction model, the webpage image acquisition module, the task acquisition module, and the user attribute management module provide sample data for the gaze point prediction model from different dimensions. In the application stage of the gaze point prediction model, the webpage image acquisition module and the task acquisition module provide input data for the gaze point prediction model, so that the gaze point prediction model predicts the position information of the target gaze point on the target webpage.

[0117] It can be seen that, by using the webpage automatic interaction system provided in the embodiments of the present application, the webpage image acquisition module is used to acquire first image information corresponding to a target webpage, the first image information including webpage images and webpage structure information; the task acquisition module is used to acquire task description information of an automatic interaction task performed on the target webpage; the multi-modal embedding and task fusion module is used to construct formatted first image information and task description information, and the first image information and the task description information are input into the gaze point prediction module. Then, the gaze point prediction module is used to predict position information of a target gaze point on the target webpage; then, the webpage interaction decision module is used to determine a task execution strategy of the automatic interaction task according to the position information of the target gaze point, the webpage structure information and the task description information. Then, the task execution module is used to interact with the target webpage based on the task execution strategy. It can be seen that, the system can predict the position information of the target gaze point on the target webpage by using a pre-trained gaze point prediction model, realizes human-like simulation of user attention, thereby simulating visual attention behavior of a person in a complex webpage, dynamically focusing on an operation object of the target webpage in a key visual area, and avoiding webpage misoperation. In addition, the position information of the predicted target gaze point is used to guide generation of the task execution strategy, so as to guide webpage automatic interaction, thereby improving the precision and efficiency of webpage automatic interaction decision.

[0118] Corresponding to the application scenarios and methods of the method provided in the embodiments of the present application, the embodiments of the present application further provide a webpage automatic interaction device.

[0119] Figure 4 A block diagram of the webpage automatic interaction device provided in the embodiments of the present application is shown as follows, Figure 4 As shown in the figure, the webpage automatic interaction device includes:

[0120] The acquisition module 41 is configured to acquire first image information corresponding to a target webpage and task description information of an automatic interaction task performed on the target webpage, the first image information including webpage images and webpage structure information.

[0121] The prediction module 42 is configured to input the first image information and the task description information into a pre-trained gaze point prediction model, and predict position information of a target gaze point on the target webpage by using the gaze point prediction model.

[0122] The determination and execution module 43 is configured to determine a task execution strategy of the automatic interaction task according to the position information of the target gaze point, the webpage structure information and the task description information, and interact with the target webpage based on the task execution strategy.

[0123] In some embodiments, the device further includes:

[0124] The second acquisition module is configured to acquire a sample data set before the first image information corresponding to the target web page and the task description information for performing the automated interactive task on the target web page are acquired; the sample data set comprises sample image information, sample task description information and sample gaze point information corresponding to a sample web page; the sample image information comprises a sample web page image and sample web page structure information; and the sample gaze point information comprises sample position information and a sample number of sample gaze points.

[0125] The second prediction module is configured to input the sample data set into a gaze point prediction model to be trained, and predict the sample gaze points by using the gaze point prediction model to be trained to obtain gaze point prediction information; the gaze point prediction information comprises predicted position information and a predicted number of the sample gaze points.

[0126] The calculation module is configured to calculate a total reward value of the gaze point prediction model to be trained according to the gaze point prediction information and the sample gaze point information.

[0127] The training module is configured to train the gaze point prediction model to be trained based on the total reward value to obtain a trained gaze point prediction model.

[0128] In some embodiments, when the calculation module calculates the total reward value of the gaze point prediction model to be trained according to the gaze point prediction information and the sample gaze point information, the following steps are performed:

[0129] According to the degree of coincidence between the predicted number and the sample number, a format consistency reward value of the gaze point prediction model to be trained is calculated;

[0130] According to the spatial distance between the predicted position information and the sample position information, a spatial consistency reward value of the gaze point prediction model to be trained is calculated;

[0131] According to the format consistency reward value and the spatial consistency reward value, the total reward value is calculated.

[0132] In some embodiments, when the acquisition module 41 acquires the first image information corresponding to the target web page, the following steps are performed:

[0133] Taking a screenshot of the target web page to obtain a web interface image;

[0134] Based on a preset segmentation manner, the web interface image is segmented into a plurality of sub-interface images;

[0135] Based on a preset image format requirement, the sub-interface images are respectively subjected to image processing to obtain the web page image.

[0136] In some embodiments, the task execution strategy comprises a webpage interaction element and a webpage interaction action.

[0137] The determining and executing module 43 performs the following steps when determining the task execution strategy of the automated interaction task according to the position information of the target gaze point, the webpage structure information, and the task description information:

[0138] spatially mapping the position information of the target gaze point and the webpage structure information, and determining the webpage interaction element according to a spatial mapping result;

[0139] determining the webpage interaction action according to the webpage image, the task description information, and the webpage interaction element.

[0140] In some embodiments, the apparatus further comprises:

[0141] a third acquiring module configured to acquire a webpage interaction result of interacting with the target webpage based on the task execution strategy; the webpage interaction result comprises an interaction result interface image;

[0142] a judging module configured to judge whether the automated interaction task is successfully executed according to the task description information and the webpage interaction result;

[0143] an interaction module configured to acquire second image information of a current webpage and interact with the current webpage based on the second image information if the automated interaction task is not successfully executed.

[0144] In some embodiments, the judging module performs the following steps when judging whether the automated interaction task is successfully executed according to the task description information and the webpage interaction result:

[0145] determining a target interaction result of executing the automated interaction task on the target webpage according to the task description information;

[0146] determining that the automated interaction task is successfully executed if the webpage interaction result matches the target interaction result.

[0147] The apparatus according to embodiments of this application acquires first image information corresponding to a target webpage and task description information for performing automated interaction tasks on the target webpage. The first image information includes a webpage image and webpage structure information. The first image information and task description information are input into a pre-trained gaze prediction model, which predicts the position information of a target gaze point on the target webpage. Then, based on the position information of the target gaze point, the webpage structure information, and the task description information, a task execution strategy for the automated interaction task is determined, and interaction with the target webpage is performed based on the task execution strategy. It is evident that this apparatus can predict the position information of a target gaze point on a target webpage through a pre-trained gaze prediction model, achieving human-like simulation of user attention. This simulates human visual attention behavior on complex webpages, dynamically focusing the target webpage's operational objects on key visual areas and avoiding accidental webpage operations. Furthermore, by using the predicted position information of the target gaze point to guide the generation of a task execution strategy to guide automated webpage interaction, the accuracy and efficiency of automated webpage interaction decision-making are improved.

[0148] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0149] Figure 5 This is a block diagram for implementing the electronic device provided in the embodiments of this application. Figure 5 As shown, the electronic device includes a memory 501 and a processor 502. The memory 501 stores a computer program that can run on the processor 502. When the processor 502 executes the computer program, it implements the method described in the above embodiments. The number of memories 501 and processors 502 can be one or more. In a specific implementation, the electronic device may also include a communication interface 503 for communicating with external devices and exchanging data.

[0150] In practical implementation, if the memory 501, processor 502, and communication interface 503 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bus or only one type of bus can be present. Thus, the dashed line should not be interpreted to infer a requirement that there must be another bus;

[0151] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can complete the communication among each other through an internal interface.

[0152] The embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method provided in the embodiment of the present application.

[0153] The embodiment of the present application provides a computer program product, which comprises a computer program, and the program is executed by a processor to implement the method provided in the embodiment of the present application.

[0154] The embodiment of the present application further provides a chip, which comprises a processor, and the processor is used for calling and running instructions stored in a memory, so that a communication device installed with the chip executes the method provided in the embodiment of the present application.

[0155] The embodiment of the present application further provides a chip, which comprises an input interface, an output interface, a processor and a memory, and the input interface, the output interface, the processor and the memory are connected through internal connection channels, and the processor is used for executing code in the memory, and when the code is executed, the processor is used for executing the method provided in the embodiment of the present application.

[0156] It should be understood that the processor described above can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. It should be noted that the processor can be a processor supporting an advanced RISC machine (ARM) architecture.

[0157] Further, the memory can include a read-only memory and a random access memory, optionally. The memory can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memory. The non-volatile memory can include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), or a flash memory, among others. The volatile memory can include a random access memory (RAM), which is used as an external cache. By way of example, and not limitation, many forms of RAM are available. The RAM can include a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a Sync link DRAM (SLDRAM), and a direct Rambus RAM (DR RAM), among others.

[0158] In the above-described embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the present disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium.

[0159] In the description of the application, reference to "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. The appearances of the phrases "in one embodiment" or "in some embodiments" in various places in the specification are not necessarily all referring to the same embodiment or example, nor are they necessarily referring to some, but not all, embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0160] Furthermore, the terms "first", "second", or the like are used only to describe a particular aspect in a direction and are not intended to denote relative importance or a preference over another aspect. Thus, a feature defined with "first", "second" can explicitly or implicitly include at least one of the feature. In the description of the application, the meaning of "a plurality" is two or more, unless otherwise explicitly specifically limited.

[0161] Any process or method described in flowcharts or otherwise described herein can be understood as a representation of executable instructions, code, or a module, segment, or portion of code for performing specified logic functions or steps in the process. And the scope of preferred embodiments of the application includes additional implementations that can not perform the functions in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order according to the functions involved.

[0162] The logic and / or steps described in flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions, code, or a module, segment, or portion of code for performing specified logic functions, which can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions.

[0163] It should be understood that parts of the application can be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above-described embodiment methods can be instructed by a program to relevant hardware, which can be stored in a computer-readable storage medium, and the program includes one or a combination of the steps of the method embodiments when executed.

[0164] In addition, each of the function units in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software function module. When the integrated module is realized in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium. The storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0165] The above is only exemplary embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.< / ref> < / point> < / ref> < / point>

Claims

1. A method for web page automated interaction, the method comprising: The method comprises: obtaining first image information corresponding to a target webpage and task description information of an automatic interaction task performed on the target webpage; the first image information comprises webpage images and webpage structure information; inputting the first image information and the task description information into a pre-trained gaze point prediction model to predict position information of a target gaze point on the target webpage by using the gaze point prediction model; determining a task execution strategy of the automatic interaction task according to the position information of the target gaze point, the webpage structure information and the task description information, and interacting with the target webpage based on the task execution strategy; the method further comprises: obtaining a sample data set, the sample data set comprising: sample image information corresponding to a sample webpage, sample task description information and sample gaze point information; the sample image information comprises sample webpage images and sample webpage structure information; the sample gaze point information comprises sample position information and a sample number of sample gaze points; inputting the sample data set into a gaze point prediction model to be trained, extracting visual features of the sample image information by using the gaze point prediction model to be trained, learning a semantic mapping relationship between the sample task description information and the sample gaze point information, predicting the position information and the number of the sample gaze points by using the visual features and the semantic mapping relationship, and training the gaze point prediction model based on the predicted position information and number; the training of the gaze point prediction model based on the predicted position information and number comprises: calculating a format consistency reward value of the gaze point prediction model to be trained according to the degree of coincidence between the predicted number and the sample number; calculating a spatial consistency reward value of the gaze point prediction model to be trained according to the spatial distance between the predicted position information and the sample position information; calculating a total reward value according to the format consistency reward value and the spatial consistency reward value; training the gaze point prediction model to be trained based on the total reward value to obtain a trained gaze point prediction model.

2. The method of claim 1, wherein, the method further comprises: taking a screenshot of the target webpage to obtain a webpage interface image; segmenting the webpage interface image into a plurality of sub-interface images based on a preset segmentation method; performing image processing on each of the sub-interface images based on a preset image format requirement to obtain the webpage images.

3. The method of claim 1, wherein, the task execution strategy comprises webpage interaction elements and webpage interaction actions; the method further comprises: spatially mapping the position information of the target gaze point and the webpage structure information to determine the webpage interaction elements according to the spatial mapping result; determining the webpage interaction actions according to the webpage images, the task description information and the webpage interaction elements.

4. The method of claim 1, wherein, after the interaction with the target webpage based on the task execution strategy, the method further comprises: obtaining a webpage interaction result of interaction with the target webpage; the webpage interaction result comprises an interaction result interface image; judging whether the automatic interaction task is successfully executed according to the task description information and the webpage interaction result; if the automatic interaction task is not successfully executed, obtaining second image information of a current webpage, and interacting with the current webpage based on the second image information.

5. The method of claim 4, wherein, The judging whether the automatic interaction task is successfully executed according to the task description information and the webpage interaction result comprises: determining a target interaction result of executing the automatic interaction task on the target webpage according to the task description information; in the case that the webpage interaction result and the target interaction result match, determining that the automatic interaction task is successfully executed.

6. A webpage automated interaction device, characterized in that, comprise: an obtaining module, configured to obtain first image information corresponding to a target webpage and task description information of an automatic interaction task executed on the target webpage; the first image information comprises webpage image and webpage structure information; a prediction module, configured to input the first image information and the task description information into a pre-trained gaze point prediction model, and predict position information of a target gaze point on the target webpage through the gaze point prediction model; a determination and execution module, configured to determine a task execution strategy of the automatic interaction task according to the position information of the target gaze point, the webpage structure information and the task description information, and interact with the target webpage based on the task execution strategy; obtaining a sample data set, the sample data set comprising: sample image information corresponding to a sample webpage, sample task description information and sample gaze point information; the sample image information comprises sample webpage image and sample webpage structure information; the sample gaze point information comprises sample position information and sample quantity of sample gaze points; inputting the sample data set into a gaze point prediction model to be trained, extracting visual features of the sample image information through the gaze point prediction model to be trained, learning semantic mapping relationship between the sample task description information and the sample gaze point information, predicting position information and quantity of the sample gaze points by using the visual features and the semantic mapping relationship, and training the gaze point prediction model based on the predicted position information and quantity; The training of the gaze point prediction model based on the predicted position information and quantity comprises: calculating a format consistency reward value of the gaze point prediction model to be trained according to the degree of coincidence between the predicted quantity and the sample quantity; calculating a spatial consistency reward value of the gaze point prediction model to be trained according to the spatial distance between the predicted position information and the sample position information; calculating a total reward value according to the format consistency reward value and the spatial consistency reward value; training the gaze point prediction model to be trained based on the total reward value to obtain a trained gaze point prediction model.

7. An electronic device, comprising: A computer program product comprising a memory, a processor and a computer program stored on the memory, the processor implementing the method of any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, A computer program product comprising a memory, a processor and a computer program stored on the memory, the processor implementing the method of any one of claims 1 to 5 when executing the computer program.

Images