Webpage task operation method and device, electronic equipment and computer program product

Abstract

The invention discloses a webpage task operation method and device, electronic equipment and a computer program product. The method comprises the steps that a task processing request initiated by a task object is received, and task information is determined according to the task processing request; according to the task information, a group of target webpage controls is extracted from a control knowledge base, the control knowledge base comprises M groups of webpage controls associated with M task webpages, each group of webpage controls is used for describing the mode of operating each task webpage, and M is a positive integer; the operation of the target task webpage is carried out according to N webpage controls in the set of target webpage controls, a task operation result is obtained, and N is a positive integer. Through the method and the device, the technical problems of low operation efficiency and low accuracy when task operation is performed on the webpage in the related technology are solved.

Description

Technical Field

[0001] The present application relates to the field of financial technology, and more specifically, to a method, device, electronic device and computer program product for operating web page tasks. Background Art

[0002] In the financial industry, users need to complete a series of tasks such as account management and risk assessment through business systems. These tasks usually involve the interaction of multiple web pages, including filling out forms, uploading files, clicking confirmation buttons, etc., and each task has its own specific operation process and necessary information. In order to improve the efficiency of task processing, financial institutions use intelligent agents to assist operations. For example, when a user needs to submit a task application in a financial business system, the financial institution receives the task requirements proposed by the user in natural language and obtains a screenshot of the web page currently operated by the user. When the intelligent agent determines that the current page cannot directly meet the user's needs, it will start a loop process, observe the page, identify executable controls (such as click, input, scroll, etc.), and generate descriptions of these controls, and then generate a thought chain, plan the shortest path or optimal path from the current page to the target page, and execute the control operation according to the thought chain to complete the user's task. After each operation, the intelligent agent checks whether the user's requirements have been met. If not, it continues to observe the new page and generate a new thought chain until the task is completed.

[0003] However, this type of agent based on a multimodal large model still has some challenges and limitations. First, due to the complexity and diversity of financial tasks, agents need to make multiple attempts to perform tasks, resulting in resource waste and inefficient operations. For example, agents need to try different ways of inputting information on multiple different application pages to find the most effective application process. Second, when dealing with highly personalized tasks such as risk assessment, agents cannot correctly understand user needs at one time, resulting in multiple attempts and adjustments, which prolongs the task processing time. In addition, existing agent technology makes it difficult to accurately understand and comply with all complex rules while ensuring efficiency.

[0004] With regard to the technical problems of low efficiency and low accuracy in performing task operations on web pages in related technologies, no effective solution has been proposed so far. Summary of the invention

[0005] The main purpose of the present application is to provide a web page task operation method, device, electronic device and computer program product to solve the technical problems of low efficiency and low accuracy in performing task operations on web pages in the related art.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for operating a webpage task is provided. The method comprises: receiving a task processing request initiated by a task object, and determining task information according to the task processing request; extracting a group of target webpage controls from a control knowledge base according to the task information, wherein the control knowledge base comprises M groups of webpage controls associated with M task webpages, each group of webpage controls is used to describe the method of operating each task webpage, and M is a positive integer; operating a target task webpage according to N webpage controls in a group of target webpage controls, and obtaining a task operation result, wherein N is a positive integer.

[0007] Furthermore, the M groups of web page controls in the control knowledge base are determined in the following manner: M task web pages are determined, and a visual language model is obtained, wherein the visual language model is used to extract visual features for each task page; for a task web page, the task web page is processed using the visual language model to obtain a group of control attribute data, wherein a group of control attribute data includes location information and type information of Y web page controls of the task web page, and Y is a positive integer; based on the group of control attribute data, the Y web page controls are processed to obtain a group of web page controls, wherein a group of web page controls is associated with control description information, and the control description information is used to describe the operation path from the initial web page to the task web page.

[0008] Furthermore, the task web page is processed using a visual language model to obtain a set of control attribute data including: for a task web page, the web page code of the task web page is extracted, and the web page code is parsed to obtain Y control initial attributes, wherein each control initial attribute includes a style attribute of a web page control and control operation information; the visual language model is used to extract features of the task web page and the Y control initial attributes to obtain a set of control attribute data.

[0009] Furthermore, the Y web page controls of the task web page are determined in the following manner: the task web page is segmented into modules to obtain K functional modules, where each functional module is associated with a web page function and K is a positive integer; the K functional modules are identified as controls using a visual language model to obtain K candidate controls; the K candidate controls are screened according to the initial properties of the Y controls to obtain Y web page controls.

[0010] Further, based on a set of control attribute data, Y web page controls are processed to obtain a set of web page controls, including: obtaining a web page processing algorithm, and obtaining L user operation data of a historical time period, wherein the web page processing algorithm is used to parse the L user operation data, each user operation data is used to characterize a situation in which a user operates any of the Y web page controls to complete a task, L is greater than Y, and L is a positive integer; using the web page processing algorithm to parse the operation behavior of the L user operation data to obtain Y control execution data, wherein each control execution data characterizes the execution status of each web page control; using a set of control attribute data to adjust parameters of the Y control execution data to obtain a set of web page controls.

[0011] Further, extracting a set of target web page controls from the control knowledge base according to the task information includes: extracting web page information of the task web page associated with the task information to obtain target web page information, wherein the target web page information is associated with a task web page; and extracting a set of target web page controls of the target task web page from the control knowledge base using the target web page information.

[0012] Furthermore, the target task web page is operated according to N web page controls in a group of target web page controls to obtain the task operation result, including: determining the target task web page according to the task information, and extracting the control description information associated with the N web page controls to obtain N control description information; controlling the N web page controls to operate the target task web page according to the N control description information to obtain the task operation result.

[0013] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a webpage task operation device is provided. The device comprises: a receiving unit, used to receive a task processing request initiated by a task object, and determine task information according to the task processing request; an extracting unit, used to extract a group of target webpage controls from a control knowledge base according to the task information, wherein the control knowledge base comprises M groups of webpage controls associated with M task webpages, each group of webpage controls is used to describe the way of operating each task webpage, and M is a positive integer; an operating unit, used to operate the target task webpage according to N webpage controls in a group of target webpage controls, and obtain a task operation result, wherein N is a positive integer.

[0014] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, a method for controlling a device where the computer-readable storage medium is located to perform any of the above-mentioned web page tasks is implemented.

[0015] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory stores an executable program, and the processor is used to run the program, wherein when the one or more programs are executed by one or more processors, the one or more processors implement an operating method for any one of the above-mentioned web page tasks.

[0016] According to another aspect of an embodiment of the present invention, a computer program product is further provided. The computer program product includes a computer program, wherein when the computer program is executed by a processor, the operating method of any one of the above-mentioned web page tasks is implemented.

[0017] In an embodiment of the present application, a web page task operation method is adopted, by receiving a task processing request initiated by a task object, and determining task information according to the task processing request; extracting a group of target web page controls from a control knowledge base according to the task information, wherein the control knowledge base includes M groups of web page controls associated with M task web pages, each group of web page controls is used to describe the method of operating each task web page, and M is a positive integer; operating a target task web page according to N web page controls in a group of target web page controls, and obtaining a task operation result, wherein N is a positive integer, thereby achieving the technical effect of improving the execution efficiency and accuracy of operating task web pages and reducing the task execution cost, and further solving the technical problem of low efficiency and low accuracy when performing task operations on web pages. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 It is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an operation method of a web page task;

[0020] Figure 2 is a flowchart of a method for operating a web page task provided in an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of a web page trajectory of a task web page provided according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a device for operating a web page task according to an embodiment of the present application;

[0023] Figure 5 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0027] It should be noted that the collected information used in this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse to use it.

[0028] Example 1

[0029] According to an embodiment of the present application, a method embodiment for operating a web page task is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 It is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an operation method of a web page task, such as Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more ( Figure 1 102a, 102b, ..., 102n are used to illustrate) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU (Microcontroller Unit) or a programmable logic device FPGA (Field-Programmable Gate Array)), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a keyboard, a cursor control device, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0031] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0032] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for operating the web page task in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the method for operating the web page task described above is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0033] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC) and a network interface, which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0034] The display may be, for example, a touch screen liquid crystal display (LCD), which may enable a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0035] Under the above operating environment, this application provides Figure 2 How to operate the web page task shown. Figure 2 is a flowchart of a method for operating a web page task according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0036] Step S201: receiving a task processing request initiated by a task object, and determining task information according to the task processing request.

[0037] Specifically, in order to improve the processing efficiency of tasks issued by users, after receiving the task processing request issued by the task object through the task webpage, the task information can be extracted from the request. For example, the task object enters the task processing request through the smart dialog box provided by the online service platform of the financial institution, for example, the task processing request is "query account balance". After receiving the above task processing request, the request can be parsed, the description language in the request can be identified, and then the task information can be generated according to the description voice.

[0038] Step S202, extracting a group of target web page controls from a control knowledge base according to the task information, wherein the control knowledge base includes M groups of web page controls associated with M task web pages, each group of web page controls is used to describe the way of operating each task web page, and M is a positive integer.

[0039] Specifically, after determining the task information of the task object, in order to accurately execute the task associated with the task information, the task can be processed through the associated controls in the control knowledge base, where the control knowledge base contains web control information related to various task web pages. These controls can help different web pages interact, thereby simulating users to perform a series of operations, such as filling out forms, clicking buttons, uploading files, etc. For example, when a web page control is an input box for filling in "user name" and "password", its functional description includes "used for user login" or "verifying user identity".

[0040] After receiving the task information, web page controls related to the task can be extracted from the control knowledge base according to the type of the information or the associated web page information. For example, for an account balance query task, controls related to login (such as user name and password input boxes) and controls related to account overview and balance query can be extracted.

[0041] Step S203, performing operations on the target task webpage according to N webpage controls in a group of target webpage controls to obtain a task operation result, wherein N is a positive integer.

[0042] Specifically, after obtaining multiple web page controls corresponding to the above task information, the best operation path to complete the task can be planned first according to the logical relationship between the controls, the operation sequence and possible exception handling strategies, wherein the best operation path lists in detail each step required to complete the task, including which control to perform, what type of operation to perform (such as click, input text, select drop-down option) and the parameters of the operation (such as the input account number, amount), etc. Then, each operation is performed according to the path, that is, the user's behavior, such as mouse clicks, keyboard input, etc., can be simulated to interact with the controls on the web page to complete each step of the task until the task is completed.

[0043] It should be noted that after completing the above operations, you can also verify whether the task results meet user needs by monitoring the status changes of the web page. If the task is successfully completed, the results will be fed back to the user; if an error or exception occurs, you need to adjust subsequent operations, such as re-searching alternative solutions from the control knowledge base for re-verification.

[0044] The operating method of a web page task provided in an embodiment of the present application receives a task processing request initiated by a task object, determines task information according to the task processing request; extracts a group of target web page controls from a control knowledge base according to the task information, wherein the control knowledge base includes M groups of web page controls associated with M task web pages, each group of web page controls is used to describe the method of operating each task web page, and M is a positive integer; operates a target task web page according to N web page controls in a group of target web page controls to obtain a task operation result, wherein N is a positive integer, solves the technical problem of low operation efficiency and low accuracy when performing task operations on web pages in the related art, determines task information according to the task processing request initiated by the task object, extracts a group of target web page controls from the control knowledge base, and uses the group of target web page controls to operate the target task web page, thereby achieving the technical effect of improving the execution efficiency and accuracy of operating task web pages and reducing the task execution cost.

[0045] Optionally, in the operation method of a web page task provided in an embodiment of the present application, M groups of web page controls in a control knowledge base are determined in the following manner: M task web pages are determined, and a visual language model is obtained, wherein the visual language model is used to extract visual features for each task page; for a task web page, the task web page is processed using the visual language model to obtain a set of control attribute data, wherein a set of control attribute data includes location information and type information of Y web page controls of the task web page, and Y is a positive integer; based on the set of control attribute data, Y web page controls are processed to obtain a set of web page controls, wherein a set of web page controls is associated with control description information, and the control description information is used to describe the operation path from the initial web page to the task web page.

[0046] Specifically, in order to improve the efficiency of task processing, the control knowledge base can be used to store controls, and then task operations can be performed based on different web page controls in the control knowledge base. First, multiple task web pages of different task types need to be identified, where tasks may include bill inquiries, online payments, loan applications, etc., and each task may involve one or more web pages. For example, when processing an account balance inquiry task, it may be necessary to access the login page, account overview page, etc.

[0047] Furthermore, after determining multiple task web pages, the content and structure of each task web page can be parsed and mined through the visual language model. For example, the layout of each page, the control type (such as buttons, input boxes, drop-down menus) and the location information of each control can be parsed. Among them, the visual language model (VLM: Vision-Language Model) can be trained by a multimodal large model.

[0048] When the model is used to process web pages, visual features can be extracted based on the web page screenshots and codes of each task web page, and the web page can be deeply analyzed based on visual and language information, so as to accurately identify the type and position of the control, thereby obtaining a set of control attribute data, wherein these data may include the location information (such as coordinates), type information (such as input box, button) and other related properties (such as whether it is interactive, control name) of each control. Finally, description information is generated for each control based on the extracted control attribute data, and then a set of web page controls are obtained based on the description information and web page controls, wherein these description information can characterize the function of each web page control and the operation path from the initial web page to the task web page. This embodiment determines the control properties and operation path by using a visual language model, and then obtains the corresponding web page control, so that the web page control required for the task can be quickly located when the user issues a task, reducing the search and trial and error time, and significantly improving the task processing efficiency.

[0049] Optionally, in the operating method of the web page task provided in the embodiment of the present application, the task web page is processed using a visual language model to obtain a set of control attribute data including: for a task web page, the web page code of the task web page is extracted, and the web page code is parsed to obtain Y control initial attributes, wherein each control initial attribute includes a style attribute of a web page control and control operation information; the visual language model is used to extract features of the task web page and the Y control initial attributes to obtain a set of control attribute data.

[0050] When processing a task web page using a visual language model, the web page code of the target task web page can first be obtained through interaction with the application programming interface of the task web page, and then the web page code is parsed to extract the initial properties of the web page control of the web page, wherein each initial property may include the style property of the control (such as color, size, position, etc.) and operation information (such as whether it is clickable, whether text needs to be entered, etc.), the style property can understand the visual performance of the control in the web page, and the operation information guides how to interact with the control. Then the style property of the web page control is associated with the operation information to form a plurality of initial properties of the control.

[0051] Furthermore, the visual language model can be used to deeply analyze the webpage screenshot of the task webpage and the initial properties of the above-mentioned controls, and the visual features (such as the image and layout information of the control) are merged with the language features (such as the control description and operation guide) to obtain a set of control property data. This embodiment uses the visual language model to analyze the webpage screenshot and the initial properties of the control, so that the webpage control can be more accurately identified and operated, the task failure rate caused by control recognition errors can be reduced, unnecessary page loading and operation attempts can be reduced, and the utilization of computing resources can be optimized.

[0052] Optionally, in the operation method of the web page task provided in the embodiment of the present application, the Y web page controls of the task web page are determined in the following manner: the task web page is divided into modules to obtain K functional modules, wherein each functional module is associated with a web page function, and K is a positive integer; the K functional modules are identified as controls using a visual language model to obtain K candidate controls; the K candidate controls are screened according to the initial properties of the Y controls to obtain Y web page controls.

[0053] Specifically, when determining the web page controls, first, after obtaining the web page screenshots of each task web page, the web page screenshots are segmented according to the module classification, thereby obtaining multiple functional modules, wherein each module represents an independent functional area on the web page, such as a login module, a search module, a shopping cart module, etc. Then, the web page function associated with each segmented functional module is determined, for example, the function associated with the login module is "login to an account", and the function associated with the search module is "search for products".

[0054] Further, the visual language model is used to identify each functional module, and the candidate controls in the module are identified, that is, the visual information (such as the appearance and position of the control) and the language information (such as the label and description of the control) are combined to determine the size and other information of all candidate controls. Finally, the initial properties of the control extracted from the web page code are used to verify the properties of the candidate controls, for example, the type, position, size, state, etc. of the control are verified to ensure that the identified control properties match the control properties in the actual web page, and then multiple web page controls that are crucial to completing the user task and are not repeated are selected from the candidate controls according to the result of the property verification. This embodiment can reduce operation errors and improve the success rate and efficiency of task execution by dividing the task web page into modules, and then more accurately locating and operating the web page controls on the task web page from different functional modules. Even in the face of the update of the web page layout or the control style, by determining the function and position of each web page control, it can adapt to the changes and ensure the continuity and stability of the operation.

[0055] Optionally, in the operation method of the web page task provided in the embodiment of the present application, Y web page controls are processed based on a set of control attribute data to obtain a set of web page controls, including: obtaining a web page processing algorithm, and obtaining L user operation data of a historical time period, wherein the web page processing algorithm is used to parse the L user operation data, and each user operation data is used to characterize a situation in which a user operates any of the Y web page controls to complete a task, L is greater than Y, and L is a positive integer; using the web page processing algorithm to parse the operation behavior of the L user operation data to obtain Y control execution data, wherein each control execution data characterizes the execution status of each web page control; using a set of control attribute data to adjust parameters of the Y control execution data to obtain a set of web page controls.

[0056] Specifically, when processing the web page controls extracted from the functional module, the web page processing algorithm can be used to locate and parse various controls on the web page, such as positioning buttons, input boxes, drop-down menus, scroll bars, etc. Then, the user operation data collected from the historical time period is parsed to determine the specific operations and behavior patterns of users when using the task web page, and to identify common operation paths and strategies. For example, the type of operation (click, input, scroll, etc.), the object of the operation (control), the order of the operation, the timestamp, etc., and the change of the web page state before and after the operation can be parsed. Thus, the control execution data representing the execution status of the web page control is obtained, wherein the control execution data may include the frequency, operation type and order of the control being operated, and may also include the state change of the control before and after the operation, such as the change of the content of the input box, the page jump after the button is clicked, etc.

[0057] It should be noted that the user operation data can represent the shortest operation path of the user from the initial web page w_1 through the web page controls {a_(1, 1), ..., a_(m, n-1)} to the destination web page w_n. After recording the route u_n of the destination web page, the user operation data of the destination web page w_n can be generated: {t_1, ..., t_d} = {[w_1, a_(1, 1), w_2, ..., a_(m, n-1), w_n, u_n, r_1], ..., [w_1, a_(1, 1), w_2, ..., a_(m, n-1), w_n, u_n, r_d]}, wherein a_(m, n-1) refers to the n-1th web page control on the w_m task page, and {r_1, ..., r_d} represents the control attribute data of each task page. Figure 3 is a schematic diagram of a web page trajectory of a task web page provided according to an embodiment of the present application, such as Figure 3As shown, the user operation data can be an operation path of jumping from the initial web page w_1 through the web page controls {a_(1, 1), a_(1, 2), a_(1, 3)} or {a_(2, 1), a_(3, 2)} to the end web page w_4, an operation path of jumping from the initial web page w_1 through the web page controls {a_(1, 1), a_(2, 2)} to the end web page w_6, and an operation path of jumping from the initial web page w_1 through the web page control a_(3, 1) to the end web page w_7.

[0058] Furthermore, a set of control attribute data is used to adjust the parameters in the control execution data to ensure the accuracy and effectiveness of the control operation, thereby obtaining a set of web page controls in the control knowledge base, for example, optimizing the operation sequence, reducing unnecessary operation steps, and enhancing the recognition and operation capabilities of key controls. This embodiment determines the behavior pattern by using the web page processing algorithm and historical user operation data, and then determines multiple web page controls, which can improve the speed and efficiency of subsequent control operations, thereby completing user tasks more quickly.

[0059] Optionally, in the operation method of the web page task provided in the embodiment of the present application, extracting a group of target web page controls from the control knowledge base according to the task information includes: extracting the web page information of the task web page associated with the task information to obtain the target web page information, wherein the target web page information is associated with a task web page; and extracting a group of target web page controls of the target task web page from the control knowledge base using the target web page information.

[0060] Specifically, in order to accurately execute user tasks, it is first necessary to identify the task information and accurately extract the key elements of the task, such as the query object (bill), query conditions (user), etc. Then, based on the key elements of the task information, match the target web page information associated with the task. For example, the target web page information of the task of querying the bill is the bill query page information. Furthermore, once the target task web page is determined based on the target web page information, the target web page control associated with the target task web page can be extracted from the control knowledge base. This embodiment can reduce unnecessary web page loading and control search time and improve the speed and efficiency of task execution by extracting the web page information associated with the task and then extracting the target web page control based on the web page information.

[0061] Optionally, in the web page task operation method provided in the embodiment of the present application, the target task web page is operated according to N web page controls in a group of target web page controls to obtain the task operation result, including: determining the target task web page according to the task information, and extracting the control description information associated with the N web page controls to obtain N control description information; controlling the N web page controls to operate the target task web page according to the N control description information to obtain the task operation result.

[0062] Specifically, when using the extracted multiple web page controls to perform operations, the target task web page to be operated can be first determined according to the task information, and then the associated control description information can be extracted from each web page control, and then the operation strategy can be generated according to the control description information. For example, the operation strategy can include determining the operation sequence of the control, the specific content of the input, the type of operation (such as click, input, selection), etc., to ensure that the operation can be performed in accordance with the user's task requirements. Further, the web page control can be controlled according to the generated operation strategy to perform precise operations on the target task web page, that is, by simulating user behavior to perform operations such as clicking a button, entering information in an input box, selecting a menu item, etc., and then the task operation result can be obtained.

[0063] It should be noted that after completing a series of control operations, the change in the webpage state can also be checked to confirm whether the operation is successful. If the operation is successful, the task operation result is presented to the user; if the operation fails, Ze Mining will try again or notify the developer to debug according to the reason for the failure. This embodiment can use the webpage control to quickly execute tasks by identifying the target webpage and webpage control, reducing unnecessary waiting and searching time, and significantly improving operation efficiency.

[0064] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0065] Example 2

[0066] The embodiment of the present application also provides a web task operation device. It should be noted that the web task operation device of the embodiment of the present application can be used to execute the web task operation method provided by the embodiment of the present application. The web task operation device provided by the embodiment of the present application is introduced below.

[0067] According to an embodiment of the present application, a device for implementing the above-mentioned web page task operation method is also provided. Figure 4 is a schematic diagram of a web page task operation device provided according to an embodiment of the present application, such as Figure 4 As shown, the device includes: a receiving unit 40, an extracting unit 41, and an operating unit 42.

[0068] A receiving unit 40 is used to receive a task processing request initiated by a task object and determine task information according to the task processing request;

[0069] An extraction unit 41 is used to extract a group of target web page controls from a control knowledge base according to the task information, wherein the control knowledge base includes M groups of web page controls associated with M task web pages, each group of web page controls is used to describe a method of operating each task web page, and M is a positive integer;

[0070] The operating unit 42 is used to operate the target task webpage according to N webpage controls in a group of target webpage controls to obtain a task operation result, wherein N is a positive integer.

[0071] The operating device for web page tasks provided in the embodiment of the present application receives a task processing request initiated by a task object through a receiving unit 40, and determines task information according to the task processing request; an extraction unit 41 extracts a group of target web page controls from a control knowledge base according to the task information, wherein the control knowledge base includes M groups of web page controls associated with M task web pages, each group of web page controls is used to describe the method of operating each task web page, and M is a positive integer; an operating unit 42 operates a target task web page according to N web page controls in a group of target web page controls to obtain a task operation result, wherein N is a positive integer, and solves the technical problem of low operation efficiency and low accuracy when performing task operations on web pages in the related art, determines task information according to the task processing request initiated by the task object, extracts a group of target web page controls from the control knowledge base, and uses the group of target web page controls to operate the target task web page, thereby achieving the technical effect of improving the execution efficiency and accuracy of operating task web pages and reducing the task execution cost.

[0072] Optionally, in the operating device of the web page task provided in the embodiment of the present application, the extraction unit 41 includes: a first determination module, used to determine M task web pages and obtain a visual language model, wherein the visual language model is used to extract visual features of each task page; a first processing module, used to process a task web page using the visual language model to obtain a set of control attribute data, wherein a set of control attribute data includes location information and type information of Y web page controls of the task web page, and Y is a positive integer; a second processing module, used to process Y web page controls based on a set of control attribute data to obtain a set of web page controls, wherein a set of web page controls is associated with control description information, and the control description information is used to describe the operation path from the initial web page to the task web page.

[0073] Optionally, in the operating device of the web page task provided in the embodiment of the present application, the extraction unit 41 includes: a first extraction module, used to extract the web page code of a task web page, and parse the web page code to obtain Y control initial attributes, wherein each control initial attribute includes a style attribute of a web page control and control operation information; a second extraction module, used to perform feature extraction on the task web page and Y control initial attributes by a visual language model to obtain a set of control attribute data.

[0074] Optionally, in the operating device for the web page task provided in the embodiment of the present application, the extraction unit 41 includes: a segmentation module, used to perform module segmentation on the task web page to obtain K functional modules, wherein each functional module is associated with a web page function, and K is a positive integer; an identification module, used to perform control identification on the K functional modules using a visual language model to obtain K candidate controls; and a screening module, used to screen the K candidate controls according to Y initial properties of the controls to obtain Y web page controls.

[0075] Optionally, in the operating device of the web page task provided in the embodiment of the present application, the extraction unit 41 includes: an acquisition module, used to acquire a web page processing algorithm, and acquire L user operation data of a historical time period, wherein the web page processing algorithm is used to parse the L user operation data, and each user operation data is used to characterize a situation in which a user operates any one of Y web page controls to complete a task, where L is greater than Y and L is a positive integer; a parsing module, used to use the web page processing algorithm to perform operation behavior analysis on the L user operation data to obtain Y control execution data, wherein each control execution data characterizes the execution status of each web page control; an adjustment module, used to use a set of control attribute data to perform parameter adjustment on the Y control execution data to obtain a set of web page controls.

[0076] Optionally, in the operating device of the web page task provided in the embodiment of the present application, the extraction unit 41 includes: a third extraction module, used to extract the web page information of the task web page associated with the task information to obtain the target web page information, wherein the target web page information is associated with a task web page; a fourth extraction module, used to extract a set of target web page controls of the target task web page from the control knowledge base using the target web page information.

[0077] Optionally, in the operating device for web page tasks provided in an embodiment of the present application, the operating unit 42 includes: a second determination module, used to determine the target task web page according to the task information, and extract the control description information associated with N web page controls to obtain N control description information; a control module, used to control N web page controls to operate the target task web page according to the N control description information to obtain the task operation result.

[0078] It should be noted that the receiving unit 40, the extracting unit 41, and the operating unit 42 correspond to steps S201 to S203 in Example 1, and the examples and application scenarios implemented by the above units and the corresponding steps are the same, but are not limited to the contents disclosed in the above Example 1. It should be noted that the above modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above units can also be part of the device and can be run in the computer terminal 10 provided in Example 1.

[0079] Example 3

[0080] The embodiment of the present application may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal or an electronic device.

[0081] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0082] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the operation method of the web page task: receiving a task processing request initiated by the task object, and determining the task information according to the task processing request; extracting a group of target web page controls from a control knowledge base according to the task information, wherein the control knowledge base includes M groups of web page controls associated with M task web pages, each group of web page controls is used to describe the method of operating each task web page, and M is a positive integer; operating the target task web page according to N web page controls in a group of target web page controls to obtain a task operation result, wherein N is a positive integer.

[0083] Optionally, the computer terminal can execute the following steps of the program code in the method for operating a web page task: M groups of web page controls in a control knowledge base are determined in the following manner: M task web pages are determined, and a visual language model is obtained, wherein the visual language model is used to extract visual features for each task page; for a task web page, the task web page is processed using the visual language model to obtain a set of control attribute data, wherein a set of control attribute data includes location information and type information of Y web page controls of the task web page, and Y is a positive integer; based on the set of control attribute data, Y web page controls are processed to obtain a set of web page controls, wherein a set of web page controls is associated with control description information, and the control description information is used to describe the operation path from the initial web page to the task web page.

[0084] Optionally, the computer terminal may execute the following program code of the method for operating a web page task: using a visual language model to process the task web page to obtain a set of control attribute data including: for a task web page, extracting the web page code of the task web page, and parsing the web page code to obtain Y control initial attributes, wherein each control initial attribute includes a style attribute of a web page control and control operation information; using a visual language model to perform feature extraction on the task web page and Y control initial attributes to obtain a set of control attribute data.

[0085] Optionally, the computer terminal can execute the following program codes in the method for operating a web page task: the Y web page controls of the task web page are determined in the following manner: the task web page is divided into modules to obtain K functional modules, where each functional module is associated with a web page function and K is a positive integer; the K functional modules are identified as controls using a visual language model to obtain K candidate controls; the K candidate controls are screened according to the initial properties of the Y controls to obtain Y web page controls.

[0086] Optionally, the computer terminal can execute the following steps of the program code in the method for operating a web page task: processing Y web page controls based on a set of control attribute data to obtain a set of web page controls including: obtaining a web page processing algorithm and obtaining L user operation data of a historical time period, wherein the web page processing algorithm is used to parse the L user operation data, each user operation data is used to characterize a situation in which a user operates any of the Y web page controls to complete a task, L is greater than Y, and L is a positive integer; using the web page processing algorithm to parse the operation behavior of the L user operation data to obtain Y control execution data, wherein each control execution data characterizes the execution status of each web page control; using a set of control attribute data to adjust the parameters of the Y control execution data to obtain a set of web page controls.

[0087] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the operation method of the web page task: extracting a group of target web page controls from the control knowledge base according to the task information, including: extracting the web page information of the task web page associated with the task information to obtain the target web page information, wherein the target web page information is associated with a task web page; using the target web page information to extract a group of target web page controls of the target task web page from the control knowledge base.

[0088] Optionally, the computer terminal may execute the following program codes of the method for operating a web page task: operating a target task web page according to N web page controls in a group of target web page controls, and obtaining a task operation result, including: determining the target task web page according to the task information, and extracting control description information associated with the N web page controls to obtain N control description information; controlling the N web page controls to operate the target task web page according to the N control description information to obtain a task operation result.

[0089] Optionally, Figure 5 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 5 As shown, the electronic device may include: one or more ( Figure 5 (only one is shown) processor 502, memory 504, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0090] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the operating method and device of the web page task in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the operating method of the above-mentioned web page task. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0091] The processor can call the information and application programs stored in the memory through the transmission device to execute the above steps in the above web page task operation method.

[0092] By adopting the embodiment of the present application, a scheme for operating a webpage task is provided. By receiving a task processing request initiated by a task object, task information is determined according to the task processing request; a group of target webpage controls are extracted from a control knowledge base according to the task information, wherein the control knowledge base includes M groups of webpage controls associated with M task webpages, each group of webpage controls is used to describe the way of operating each task webpage, and M is a positive integer; the target task webpage is operated according to N webpage controls in a group of target webpage controls, and the task operation result is obtained, wherein N is a positive integer, thereby achieving the technical effect of improving the execution efficiency and accuracy of operating the task webpage and reducing the task execution cost, thereby solving the technical problem of low efficiency and low accuracy when performing task operations on webpages.

[0093] It can be understood by those skilled in the art that Figure 5 The structure shown is for illustration only, and the electronic device may also be a smart phone, a tablet computer, a PDA, a mobile Internet device (MID), a PAD or other terminal device. Figure 5 The structure of the electronic device is not limited. Figure 5 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 5 Different configurations are shown.

[0094] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0095] Example 4

[0096] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the method for operating the web page task provided in the first embodiment.

[0097] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0098] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: receiving a task processing request initiated by a task object, and determining task information based on the task processing request; extracting a group of target web page controls from a control knowledge base based on the task information, wherein the control knowledge base includes M groups of web page controls associated with M task web pages, each group of web page controls is used to describe the method of operating each task web page, and M is a positive integer; performing operations on the target task web page according to N web page controls in a group of target web page controls to obtain a task operation result, wherein N is a positive integer.

[0099] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the program of the operation method steps of the web page task.

[0100] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0101] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0103] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0104] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0105] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0106] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for operating a web page task, characterized in that: include: Receive a task processing request initiated by a task object, and determine task information according to the task processing request; Extracting a group of target web page controls from a control knowledge base according to the task information, wherein the control knowledge base includes M groups of web page controls associated with M task web pages, each group of web page controls is used to describe a method of operating each task web page, and M is a positive integer; The target task webpage is operated according to N webpage controls in the group of target webpage controls to obtain a task operation result, wherein N is a positive integer.

2. The method according to claim 1, characterized in that The M groups of web page controls in the control knowledge base are determined in the following manner: Determine the M task web pages, and obtain a visual language model, wherein the visual language model is used to extract visual features from each task page; For a task webpage, the visual language model is used to process the task webpage to obtain a set of control attribute data, wherein the set of control attribute data includes position information and type information of Y webpage controls of the task webpage, where Y is a positive integer; The Y web page controls are processed based on the set of control attribute data to obtain a set of web page controls, wherein the set of web page controls is associated with control description information, and the control description information is used to describe an operation path from an initial web page to the task web page.

3. The method according to claim 2, characterized in that The task web page is processed using the visual language model to obtain a set of control attribute data including: For a task web page, extract the web page code of the task web page, and parse the web page code to obtain Y initial properties of controls, wherein each initial property of the control includes a style property of a web page control and control operation information; The visual language model performs feature extraction on the task webpage and the Y initial attributes of the controls to obtain the set of control attribute data.

4. The method according to claim 2, characterized in that: The Y web page controls of the task web page are determined in the following manner: Performing module division on the task webpage to obtain K functional modules, wherein each functional module is associated with a webpage function, and K is a positive integer; Using the visual language model to perform control recognition on the K functional modules to obtain K candidate controls; The K candidate controls are screened according to the Y initial properties of the controls to obtain the Y web page controls.

5. The method according to claim 2, characterized in that: The Y web page controls are processed based on the set of control attribute data to obtain a set of web page controls including: Obtain a web page processing algorithm, and obtain L user operation data in a historical time period, wherein the web page processing algorithm is used to parse the L user operation data, each user operation data is used to characterize a situation in which a user operates any of the Y web page controls to complete a task, and L is greater than Y, and L is a positive integer; Analyzing the L user operation data using the web page processing algorithm to obtain Y control execution data, wherein each control execution data represents the execution status of each web page control; The set of control attribute data is used to adjust parameters of the Y control execution data to obtain the set of web page controls.

6. The method according to claim 1, characterized in that Extracting a set of target web page controls from a control knowledge base according to the task information includes: Extracting webpage information of a task webpage associated with the task information to obtain target webpage information, wherein the target webpage information is associated with a task webpage; The target web page information is used to extract the set of target web page controls of the target task web page from the control knowledge base.

7. The method according to claim 1, characterized in that The target task webpage is operated according to the N webpage controls in the group of target webpage controls, and the task operation result obtained includes: Determine the target task webpage according to the task information, and extract the control description information associated with the N webpage controls to obtain N control description information; The N web page controls are controlled according to the N control description information to operate the target task web page to obtain the task operation result.

8. A web page task operation device, characterized in that: include: A receiving unit, configured to receive a task processing request initiated by a task object, and determine task information according to the task processing request; An extraction unit, used to extract a group of target web page controls from a control knowledge base according to the task information, wherein the control knowledge base includes M groups of web page controls associated with M task web pages, each group of web page controls is used to describe a method of operating each task web page, and M is a positive integer; The operating unit is used to operate the target task webpage according to N webpage controls in the group of target webpage controls to obtain a task operation result, wherein N is a positive integer.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for operating a web page task according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory storing an executable program; A processor is used to run the program, wherein the program executes the operating method of the web page task described in any one of claims 1 to 7 when running.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method for operating a web page task described in any one of claims 1 to 7 are implemented.

Images