Webpage calendar selection method and device suitable for RPA

Abstract

The invention discloses a webpage calendar selection method and device suitable for RPA, and the method comprises the steps: positioning a calendar element on a webpage, and determining a root node and a calendar layout of the calendar element; acquiring an HTML (Hypertext Markup Language) of a root node of the calendar element, and performing hierarchical processing on the HTML of the root node of the calendar element; according to the webpage screenshot and the HTML of the root node of the processed calendar element, generating a calendar jump code by using a first multi-mode large language model; the calendar jump code is operated, and the calendar jumps to the correct year and month layout; according to the calendar layout, the webpage screenshot and the HTML of the root node of the processed calendar element, a date selection code is generated through a second multi-mode large language model; the date selection code is operated, and the calendar jumps to the correct date. According to the method and the device, cross-year and cross-month buttons can be automatically clicked according to date requirements of users, and correct dates can be selected on different calendar layouts.

Description

Technical Field

[0001] The present application relates to a web calendar selection method and device applicable to RPA, and belongs to the field of software technology. Background Art

[0002] In the process automation design of RPA (Robotic Process Automation), users can plan continuous web page operations according to their own needs. Each operation is an interactive action between the user and the web page UI (User Interface), such as clicking and inputting. After the web page automation process design is completed, the RPA program will automatically execute the page operation according to the user's design steps. Through the process design and automatic execution of RPA, users can greatly reduce repetitive and tedious manual page operations.

[0003] For conventional web automation operations in the RPA process, such as mouse clicks, text input, mouse hovers, or table capture, the underlying logic of the RPA software coding is to obtain a fixed element from the web page, such as "obtain a web page element with the type of input box and the text content of Jiuzhaigou", and perform automated operations based on this fixed element. The user's requirements for obtaining these fixed elements will not change over time, but calendar selection is different. For example, the user's first requirement is "select July 3", and the second requirement is "select July 5". It is difficult for the user's selection requirements to be fixed on the same date, that is, it is impossible to select a fixed element for automated operations.

[0004] The calendar layouts on different web pages are also different. For example, on a travel service website, the calendar will display the date information of two months; while on a private business website, the calendar will display the date information of a single month. This requires the calendar selection operation of RPA to be able to adapt to different calendar layouts, but conventional software coding methods make it difficult to detect different calendar layouts and adapt the date extraction logic of different layouts. In addition, the calendar selection logic involves the calendar jumping across years and months. How to locate the web page elements that cross years and months on different calendar layouts is also a challenge for conventional software coding. Combined with the dynamic changes in date requirements, how to set the number of clicks on the cross-month element according to the current date is also a difficult problem to solve. Summary of the invention

[0005] Multimodal large language models (MLLM), such as GPT4o and Claude3.5, have powerful context comprehension and image recognition capabilities. Use MLLM to develop web automation agents. Users can use prompt words and input images to let the MLLM agent plan page operations that meet their needs and perform web operations on their behalf. MLLM can also handle the difficulties in the above calendar selection to a certain extent, helping RPA to implement a stable and reliable calendar selection tool.

[0006] Purpose: In view of at least one of the above technical problems, the present application provides a web calendar selection method and device suitable for RPA, which can automatically click the cross-year and cross-month buttons according to the user's date requirements and select the correct date on different calendar pages.

[0007] Technical solution: To solve the above technical problems, the technical solution adopted in this application is:

[0008] In a first aspect, the present application provides a web calendar selection method applicable to RPA, the method comprising:

[0009] S1, locate the calendar element on the web page, determine the root node of the calendar element and the calendar layout;

[0010] S2. Obtain the HTML of the root node of the calendar element, and perform hierarchical processing on the HTML of the root node of the calendar element;

[0011] S3, according to the webpage screenshot and the HTML of the root node of the calendar element processed by S2, using the first multimodal large language model MLLM to generate a calendar jump code; the calendar jump code can automatically jump to the calendar month and year;

[0012] S4, running the calendar jump code, and the calendar jumps to the correct year and month layout;

[0013] S5, generating a date selection code using a second multimodal large language model MLLM according to the calendar layout obtained in S1, the webpage screenshot, and the HTML of the root node of the calendar element processed by S2;

[0014] S6. Run the date selection code and the calendar jumps to the correct date.

[0015] In a second aspect, the present application provides a web calendar selection device applicable to RPA, the device comprising:

[0016] The acquisition module is used to: locate the calendar element on the web page, determine the root node of the calendar element and the calendar layout;

[0017] The HTML processing module is used to: obtain the HTML of the root node of the calendar element, and perform hierarchical processing on the HTML of the root node of the calendar element;

[0018] A first code generation module is used to: generate a calendar jump code using a first multimodal large language model MLLM according to the webpage screenshot and the HTML of the root node of the calendar element processed by S2; the calendar jump code can automatically jump to the calendar month and year;

[0019] The first code running module is used to: run the calendar jump code so that the calendar jumps to the correct year and month layout;

[0020] A second code generation module is used to generate a date selection code using a second multimodal large language model MLLM according to the calendar layout and webpage screenshot obtained by S1 and the HTML of the root node of the calendar element processed by S2;

[0021] The second code running module is used to: run the date selection code so that the calendar jumps to the correct date.

[0022] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.

[0023] In a fourth aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.

[0024] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0025] Beneficial effects: The web calendar selection method and device suitable for RPA provided in this application have the following advantages: MLLM is used to develop an intelligent agent for automated web operations. Users can let the MLLM agent plan page operations that meet the needs and perform web operations on behalf of the user based on the prompt words and the input web page screenshots. MLLM can also handle the difficulties in the above-mentioned calendar selection to a certain extent, helping RPA to implement a stable and reliable calendar selection tool; the generated calendar jump code and date selection code constitute the calendar selection function, and the calendars of different web pages correspond to different code structures. According to the user's date requirements, it can automatically click the cross-year and cross-month buttons, and select the correct date on different calendar pages. This also enables the calendar selection function developed based on MLLM to adapt to different web page environments, with extremely high stability and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of an application scenario of a method according to an embodiment of the present application;

[0027] Figure 2 This is a flow chart of a web calendar selection method applicable to RPA according to an embodiment of the present application;

[0028] Figure 3 A schematic diagram of a web calendar selection device applicable to RPA provided according to an embodiment of the present application;

[0029] Figure 4 A schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] The present application is further described below in conjunction with the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and cannot be used to limit the protection scope of the present application.

[0031] In this application, the professional terms are explained as follows:

[0032] Multimodal Large Language Model (MLLM): Using a powerful large language model (LLM) as the "brain", various multimodal tasks can be performed. The large language model LLM is a deep learning model trained with a large amount of text data, with billions of parameters, capable of understanding and processing natural language. It is a general natural language processing (NLP) model.

[0033] GPT-4o is a language model released by OpenAI for the chatbot ChatGPT, which "can reason about audio, vision, and text in real time.

[0034] Prompt words: text content used to prompt MLLM to provide targeted answers, usually including system prompts used to prompt MLLM in what role, in what scenario, and in what format to generate answers in the current context window; user prompts generally refer to the conversation content entered by the user, used to prompt MLLM to complete tasks, including but not limited to daily conversations, answering questions, translation, writing, and coding.

[0035] Robotic process automation (RPA) is a program that simulates the manual operations of users on the computer desktop through RPA process code to achieve process automation.

[0036] Before describing the embodiments of the present application in detail, a specific scenario example is used to reveal the scenario in which the technical solution of the present application can be used. In a possible scenario example, specifically, Figure 1As shown, the user performs specific operations on the client 110, and the client 110 is deployed with a data collection device and a multimodal large language model MLLM, the data collection device is used to collect web page screenshots and user needs on the client 110, and locate the calendar element on the web page, and determine the root node of the calendar element and the calendar layout; the collected web page screenshots, user date requirements, the root node of the calendar element, and the calendar layout are transmitted to the server through the network, and the server generates a calendar jump code using the first multimodal large language model MLLM according to the web page screenshot and the processed HTML of the root node of the calendar element through a preset program; the calendar jump code can automatically jump to the calendar month and year; according to the calendar layout and the web page screenshot and the processed HTML of the root node of the calendar element, a date selection code is generated using the second multimodal large language model MLLM; the RPA program first runs the calendar jump code according to the user's date requirements, and the calendar jumps to the correct year and month layout; then runs the date selection code, and the calendar jumps to the correct date.

[0037] The multimodal large language model MLLM can run on the client terminal side or the server side. The method of the present application can be configured in the multimodal large language model MLLM on the client terminal / or server side, and the client terminal and the server are connected through network transmission to realize multi-terminal interaction.

[0038] The server may be an electronic device with certain computing and processing capabilities. For example, the server may be a server of a distributed system, or a system with multiple processors, memories, network communication modules, etc. operating in collaboration. The server may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The server may also be a server cluster formed by several servers. Alternatively, with the development of science and technology, the server may also be a new technical means capable of realizing the corresponding functions of the embodiments of the specification. For example, it may be a new form of "server" based on quantum computing.

[0039] The client terminal may be an electronic device with network access capability. Specifically, for example, the terminal may be a desktop computer, a tablet computer, a laptop computer, a smart phone, etc. Alternatively, the terminal may also be software that can run in the electronic device.

[0040] The network may be any type of network that can support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. The one or more networks may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0041] Example 1: Figure 2 As shown, this embodiment provides a web calendar selection method applicable to RPA, which can be applied to a multimodal large language model MLLM, which can run in a client terminal. The web calendar selection method applicable to RPA includes:

[0042] Step S1, locating the calendar element on the web page, determining the root node of the calendar element and the calendar layout;

[0043] In some embodiments, step S1 may include:

[0044] S11, filtering out all calendar elements from the web page HTML according to keywords; wherein the web page HTML records information of all web page elements;

[0045] It should be noted that this application selects calendar-related elements from the HTML of a web page based on some specific keywords. Conventional commercial websites rarely use standard calendar controls, and they all write their own calendar elements with special functions to form a calendar structure. The class names of these calendar elements often contain fields related to "calendar".

[0046] Therefore, in some embodiments, the keyword is selected from at least one of "calendar" or "datetime"; and the element whose web page element class name contains the keyword is matched in the HTML of the web page as a calendar element.

[0047] The calendar elements include a calendar header element, a month element, and a jump element.

[0048] It should be noted that S11 will return multiple calendar elements, such as the calendar header element, month element, and jump element. These calendar elements are located under the same root node, which represents the complete calendar structure. The root node of the calendar element is located, and these scattered calendar elements are integrated to obtain a complete calendar.

[0049] S12. Obtain the XPath of each calendar element. For example, the month element is " / html / body / div / div[1] / li[3] / div[2]", and the date element is " / html / body / div / div[1] / li[3] / div[2] / span[5]"; count the common parent node of all calendar element XPaths, such as " / html / body / div / div[1]", as the root node of the calendar element; there are one or more root nodes of the calendar element;

[0050] In most cases, S12 will return the root node of a unique calendar element. However, sometimes there are some interfering elements in the web page HTML, resulting in S12 returning multiple root nodes. At this time, use MLLM to determine which root node of the calendar element corresponds to the real web page calendar. The MLLM used in this solution is GPT-4o, which supports various modalities of input such as text, pictures, and audio.

[0051] S13. If there are multiple root nodes of the calendar element, use the multi-modal large language model MLLM to determine the correct root node of the calendar element and the calendar layout from multiple root nodes;

[0052] S131. Obtain the HTML of the root node of each calendar element, and convert the HTML into a DOM tree with a simpler expression form;

[0053] The element expression form in the DOM tree is like "id:23,tag:span,text:October 2024", which is composed of the element tag and text content in the original HTML.

[0054] S132. Use the multi-modal large language model to judge whether the corresponding root node represents a calendar according to the DOM tree, so as to determine the correct root node of the calendar element and mark the calendar layout;

[0055] In this step S132, using the DOM tree of the root node of the calendar element as part of the MLLM prompt can save the input amount of the MLLM prompt and reduce the cost of the MLLM. The prompt also includes two parts: task description and output format. The task description requires judging whether the current root node represents a calendar according to the DOM tree and analyzing whether the calendar layout shows the date information of a single month or multiple months. According to the text prompt, the MLLM can distinguish the correct root node of the calendar element from multiple root nodes and mark the calendar layout.

[0056] S2. Obtain the HTML of the root node of the calendar element, and perform hierarchical processing on the HTML of the root node of the calendar element;

[0057] According to the root node of the calendar element obtained in S1, the HTML content of the root node is obtained. In order to allow the MLLM in the subsequent steps to better understand the HTML content, some special processing needs to be done on the HTML of the root node of the calendar element. The HTML of the root node of the calendar element contains some redundant element information, including web page elements such as div and group that are irrelevant to web page interaction and page visibility. If these elements have no text content, they are deleted from the HTML of the root node of the calendar element.

[0058] In addition, when the HTML of the root node of the calendar element contains multiple months, the date element is likely to be a child node of the month element; the HTML of the root node of the calendar element needs to clearly display the hierarchical relationship of these elements. However, the HTML content obtained through third-party Python libraries such as playwright cannot intuitively represent the hierarchical relationship, so this method represents the hierarchical relationship of HTML by adding space placeholders (Table 1). Through subsequent steps, it is verified that this processed hierarchical representation can greatly improve MLLM's understanding of the hierarchical relationship of elements.

[0059] Table 1. Raw HTML and hierarchical HTML

[0060]

[0061] Therefore, in some embodiments, step S2 includes: obtaining the HTML of the root node of the calendar element, deleting redundant element information; if the calendar HTML contains multiple months, adding space placeholders to represent the hierarchical relationship of the HTML.

[0062] S3, according to the webpage screenshot and the HTML of the root node of the calendar element processed by S2, using the first multimodal large language model MLLM to generate a calendar jump code; wherein the calendar jump code is a python function code; the calendar jump code can automatically jump to the calendar month and year;

[0063] It should be noted that, according to the HTML of the root node of the calendar element processed by S2, MLLM is used to generate the python code that can click on the cross-year and cross-month elements based on any input date. The text prompt word consists of the task description, the HTML of the root node of the calendar element, the picture description, the first python function sample, the first task rule and the output format. In addition to the text prompt word, the screenshot of the web page corresponding to the calendar will also be sent to MLLM.

[0064] The task description of the prompt word requires MLLM to first observe the HTML of the root node of the calendar element, and then generate code that can automatically jump to the calendar month and year. The output format requires MLLM to only output Python function code, not introduce Python third-party function libraries, and not display HTML specific content. Step S3 specifically introduces the Python function samples and task rules in the prompt word, which play an important guiding role for MLLM.

[0065] S31 first python function example

[0066] The python function example illustrates the form of MLLM programming, but does not limit the generated code to the format of the sample code. This method uses Playwright as the web automation execution library, so the function sample will display the corresponding sample code for Playwright. The text prompt words of the multimodal large language model include python function examples and task rules; wherein the python function example splits the cross-year and cross-month codes into a main function and multiple calling functions;

[0067] The input parameters of the main function include the Playwright page information and the user's specified date. The source, type and data format of the input parameters are introduced through comments at the beginning of the main function; the data format of the specified date, such as "2024 / 11 / 12", allows MLLM to correctly process the time information in HTML, such as processing the "2024" HTML text into date data, such as "2024". The main function obtains the specific year and month information based on the specified date, and then continuously calls the check page function and the click cross-year and cross-month functions in a loop; the Playwright page information is refreshed at the end of each loop.

[0068] The calling function is divided into the year check function, month check function, and month-crossing element click function; the input parameters of the year check function include Playwright page information and year, and the year check function includes: 1) parameter introduction; 2) using incomplete code prompts to imply that MLLM should obtain the current year based on the page information, such as "current_year=..." implying that MLLM should obtain the current year based on the page information and correctly process the year field; 3) return a Boolean value, True or False. If True is returned, it means that the current calendar needs to cross the year to display the date content required by the user.

[0069] The input parameters of the check month function include Playwright page information and month. The check month function includes: 1) parameter introduction; 2) using incomplete code prompts to imply that MLLM should obtain the current month based on the page information and correctly process the month field; 3) return a Boolean value, True or False. If True is returned, it means that the current calendar needs to cross months to display the date content required by the user. The composition of the check month year function is similar to the check year function, and the acquisition of the current month is slightly different. Because the calendar may contain information for multiple months, an additional comment is used to imply that the month obtained by MLLM may be a list rather than a single value.

[0070] The input parameter of the click cross-month element function only contains the Playwright page information. The click cross-month element function includes: 1) parameter introduction; 2) cross-month element acquisition; 3) filter and click cross-month elements. A calendar with multiple months sometimes contains multiple cross-month elements. Only the last one of these cross-month elements can be effectively interacted with, so the function selects the last one by default. The click cross-year element function has the same form as the click cross-month element function, so it is not shown in the function sample.

[0071] S32. The first task rule and the first python function example cooperate with each other to guide MLLM's programming. The task rules include: Rule 1, requiring MLLM to observe the HTML of the root node of the calendar element, refer to the first python function example and use Playwright coding; Rule 2, if there is a cross-year element in the HTML of the root node of the calendar element, generate a check and click cross-year function; this rule allows MLLM to generate variable code according to the actual calendar layout, rather than sticking to fixed functions; Rule 3, the Playwright page must be refreshed after clicking the cross-year or cross-month; the code generated by MLLM is forced to be able to re-acquire the page content after the click operation to ensure the normal operation of the code. Rule 4, if there are backtracking year and month buttons in the HTML of the root node of the calendar element, refer to the first python function example to generate code.

[0072] Through text prompts, MLLM generates calendar jump codes. These codes can locate the existence of cross-year and cross-month elements, check the year and month information of the current calendar information, and finally click the cross-year and cross-month elements with the correct number of times according to the user-specified date. Regardless of any date or calendar layout, MLLM can generate codes that meet the current scenario. These codes can maintain normal operation during the automated execution phase without being affected by the dynamic changes in date requirements.

[0073] S4, run the calendar jump code, the calendar jumps to the correct year and month layout. MLLM is also required to generate another code to obtain the correct date.

[0074] S5, generating a date selection code using a second multimodal large language model MLLM according to the calendar layout obtained in S1, the webpage screenshot, and the HTML of the root node of the calendar element processed by S2; wherein the date selection code is a python function code;

[0075] In this step, the composition of the prompt word and the S3 type, here we still focus on the role of the python function sample and the task rule; the text prompt word of the second multimodal large language model includes the second python function sample and the second task rule;

[0076] The second python function example is divided into two, one is a single-month function example and the other is a multi-month function example. If the calendar page displays the date information of a single month, select the single-month function example; if the calendar page displays the date information of multiple months, select the multi-month function example.

[0077] There are two function samples, one for a single month and one for multiple months. According to the calendar layout obtained in S1, the corresponding sample will be selected here. The single-month function sample has only one function, and the input parameters are the Playwright page information and the specified date. The function includes four parts: 1) parameter introduction; 2) get the specific month and date according to the specified date; 3) locate the date element; 4) click the date element. Among them, the sample code for locating the date element needs to guide MLLM to process the non-interactive elements in the HTML of the root node of the calendar element.

[0078] The calendars on commercial websites contain some date information that cannot be interacted with. For example, on ticket booking websites, dates are arranged in weeks. If the 1st of the month is Saturday, the first five days of the week will be displayed as empty on the calendar. The HTML information of these empty elements does not contain valid text content, and these elements cannot be clicked. These elements cannot interact normally on the page, and non-interactive elements will cause subsequent Playwright interfaces to report errors. Therefore, the sample code for locating date elements instructs MLLM to check whether the HTML field of the date element contains "disable" and other similar fields, which indicate that the element cannot be interacted with. By checking and blocking these non-interactive elements, the Playwright code can run normally.

[0079] The input parameters of the multi-month function sample are Playwright page information and the specified date. The function includes four parts: 1) Introduction to the input parameters; 2) Obtaining the specific month and date based on the specified date; 3) Locating the month element, and then locating the date element based on the month element; 4) Clicking the date element. Among them, the sample code for locating the date element needs to guide MLLM to process the non-interactive elements in the HTML of the root node of the calendar element. It should be noted that the code of the multi-month function sample has more prompts for locating the month layout and selecting the date based on the layout than the single-month function sample, suggesting that when facing a calendar with multiple months, the code generated by MLLM can select the month first and then the date. Without these prompts, the generated code may select the same date element on multiple months. If the unique element is not specified, the subsequent Playwright operation interface will report an error.

[0080] The rules for the second task include: Rule 1, requiring MLLM to observe the HTML of the root node of the calendar element, refer to the second python function example and use Playwright coding; Rule 2, when facing a multi-month calendar layout, select the month first and then the date; Rule 3, the selected month element must be the parent node of the date element; Rule 4, avoid all elements containing the "disable" field.

[0081] Through the text prompt, MLLM generates a date selection code. Similar to step S3, the code generated by MLLM can meet the scenario of any date and any calendar layout, and maintain normal operation during the automated execution stage.

[0082] S6. Run the date selection code and the calendar jumps to the correct date.

[0083] In summary, the web calendar selection method applicable to RPA provided in the above implementation method constitutes a calendar selection function according to the calendar jump code generated by S3 and the date selection code generated by S5. Calendars of different web pages correspond to different code structures, which also enables the calendar selection function developed based on MLLM to adapt to different web page environments and has extremely high stability and robustness.

[0084] Example 2: Based on Example 1, Figure 3 As shown, the present application provides a web calendar selection device applicable to RPA, the device comprising:

[0085] The acquisition module is used to: locate the calendar element on the web page, determine the root node of the calendar element and the calendar layout;

[0086] The HTML processing module is used to: obtain the HTML of the root node of the calendar element, and perform hierarchical processing on the HTML of the root node of the calendar element;

[0087] A first code generation module is used to: generate a calendar jump code using a first multimodal large language model MLLM according to the webpage screenshot and the HTML of the root node of the calendar element processed by S2; the calendar jump code can automatically jump to the calendar month and year;

[0088] The first code running module is used to: run the calendar jump code so that the calendar jumps to the correct year and month layout;

[0089] A second code generation module is used to generate a date selection code using a second multimodal large language model MLLM according to the calendar layout and webpage screenshot obtained by S1 and the HTML of the root node of the calendar element processed by S2;

[0090] The second code running module is used to: run the date selection code so that the calendar jumps to the correct date.

[0091] The specific functional implementation of each of the above modules can be found in the relevant contents of the method in Example 1 and will not be elaborated here.

[0092] Embodiment 3: Based on Embodiment 1, this embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in Embodiment 1 are implemented.

[0093] Embodiment 4: Based on embodiment 1, as Figure 4 As shown, this embodiment provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in Embodiment 1 when executing the computer program.

[0094] Embodiment 5: Based on Embodiment 1, this embodiment proposes a hardware system that applies a web calendar selection method applicable to RPA in Embodiment 1. The method is applied on the server side.

[0095] The terminal may be an electronic device with network access capability, for example, a desktop computer, a tablet computer, a laptop computer, a smart phone, a digital assistant, a shopping guide terminal, a television, etc.

[0096] The server can be an electronic device with certain computing and processing capabilities. For example, the server can be a server of a distributed system, or a system with multiple processors, memories, network communication modules, etc. operating in collaboration. The server can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The server can also be a server cluster formed by several servers. Alternatively, with the development of science and technology, the server can also be a new technical means that can realize the corresponding functions of the implementation method of the specification. For example, it can be a new form of "server" based on quantum computing.

[0097] Example 6: Based on Example 1, this example further provides a computer program product comprising instructions, which, when executed by a computer, enables the computer to execute a web calendar selection method applicable to RPA in Example 1.

[0098] It should be understood that the specific examples in this article are only intended to help those skilled in the art better understand the implementation methods of this specification, rather than limiting the scope of this application.

[0099] It can be understood that in the various implementations of this specification, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the implementation methods of this specification.

[0100] It can be understood that the various embodiments described in this specification can be implemented individually or in combination, and the embodiments of this specification are not limited to this.

[0101] Unless otherwise specified, all technical and scientific terms used in the embodiments of this specification have the same meaning as those generally understood by those skilled in the art of the technical field of this specification. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the scope of this specification. The term "and / or" used in this specification includes any and all combinations of one or more related listed items. The singular forms of "a", "above", and "the" used in the embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0102] It can be understood that the processor of the embodiment of this specification can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method implementation can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiment of this specification can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of this specification can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0103] It is understood that the memory in the embodiments of this specification may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (programmable ROM, PROM), an erasable programmable read-only memory (erasablePROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0104] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0105] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method implementation methods and will not be repeated here.

[0106] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0107] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0108] In addition, each functional unit in each embodiment of the present specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0109] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0110] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A web calendar selection method suitable for RPA, characterized in that: The method comprises: S1, locate the calendar element on the web page, determine the root node of the calendar element and the calendar layout; S2. Obtain the HTML of the root node of the calendar element, and perform hierarchical processing on the HTML of the root node of the calendar element; S3, according to the webpage screenshot and the HTML of the root node of the calendar element processed by S2, using the first multimodal large language model MLLM to generate a calendar jump code; the calendar jump code can automatically jump to the calendar month and year; S4, running the calendar jump code, and the calendar jumps to the correct year and month layout; S5, generating a date selection code using a second multimodal large language model MLLM according to the calendar layout obtained in S1, the webpage screenshot, and the HTML of the root node of the calendar element processed by S2; S6. Run the date selection code and the calendar jumps to the correct date.

2. The web calendar selection method applicable to RPA according to claim 1, characterized in that: Step S1 specifically includes: S11, filtering out all calendar elements from the web page HTML according to keywords; wherein the web page HTML records information of all web page elements; S12, obtaining the XPath of each calendar element, counting the common parent node of the XPath of all calendar elements as the root node of the calendar element; the root node of the calendar element is one or more; S13. If there are multiple root nodes of the calendar element, use a multimodal large language model MLLM to determine the correct root node of the calendar element and the calendar layout from the multiple root nodes.

3. The web calendar selection method applicable to RPA according to claim 2, characterized in that: In step S11, the keyword is selected from at least one of "calendar" or "datetime"; an element whose web page element class name contains the keyword is matched in the web page HTML as a calendar element; the calendar element includes a calendar header element, a month element, and a jump element.

4. The web calendar selection method applicable to RPA according to claim 2, characterized in that: In step S11 and step S13, a multimodal large language model MLLM is used to determine the correct root node of the calendar element and the calendar layout from multiple root nodes, including: S131, obtaining the HTML of the root node of each calendar element, and converting the HTML into a DOM tree; wherein the DOM tree is composed of element tags and text content in the original HTML; S132. Using the multimodal large language model, judging whether the corresponding root node represents a calendar according to the DOM tree, so as to determine the correct root node of the calendar element and mark the calendar layout.

5. The web calendar selection method applicable to RPA according to claim 1, characterized in that: Step S2 specifically includes: Get the HTML of the root node of the calendar element and delete redundant element information; if the calendar HTML contains multiple months, add space placeholders to represent the hierarchical relationship of the HTML.

6. The web calendar selection method applicable to RPA according to claim 1, characterized in that: The text prompt words of the first multimodal large language model include a first python function example and a first task rule. Step S3 specifically includes: according to the webpage screenshot and the HTML of the root node of the calendar element processed by S2, using the first multimodal large language model MLLM to refer to the first python function example and generate a calendar jump code according to the first task rule; wherein the calendar jump code is a python function code.

7. The web calendar selection method applicable to RPA according to claim 6, characterized in that: The first python function example splits the cross-year and cross-month codes into a main function and multiple calling functions; The input parameters of the main function include the Playwright page information and the user's specified date. The source, type and data format of the input parameters are introduced through comments at the beginning of the main function. The main function obtains the specific year and month information according to the specified date, and then continuously calls the check page function and the click cross-year and cross-month functions in a loop. At the end of each loop, the Playwright page information will be refreshed. The calling function is divided into the year check function, month check function, and month-crossing element click function. The input parameters of the year check function include the Playwright page information and the year. The year check function includes: 1) parameter introduction; 2) using incomplete code prompts to imply that MLLM should obtain the current year based on the page information and correctly process the year field; 3) return a Boolean value, True or False. If True is returned, it means that the current calendar needs to cross the year to display the date content required by the user; The input parameters of the month check function include Playwright page information and month. The month check function includes: 1) parameter introduction; 2) using incomplete code prompts to hint MLLM to obtain the current month based on page information and correctly process the month field; 3) return a Boolean value, True or False. If True is returned, it means that the current calendar needs to cross months to display the date content required by the user; The input parameter of the click cross-month element function only contains the Playwright page information. The click cross-month element function includes: 1) input parameter introduction; 2) cross-month element acquisition; 3) select cross-month elements and click; And / or, the first task rules include: Rule 1, requiring MLLM to observe the HTML of the root node of the calendar element, refer to the first python function example and use Playwright encoding; Rule 2, if there is a cross-year element in the HTML of the root node of the calendar element, generate a check and click cross-year function; Rule 3, the Playwright page must be refreshed after clicking the cross-year or cross-month; Rule 4, if there is a backtrack year or month button in the HTML of the root node of the calendar element, refer to the first python function example to generate code.

8. The web calendar selection method applicable to RPA according to claim 1, characterized in that: The text prompt words of the second multimodal large language model include a second python function example and a second task rule. Step S5 specifically includes: according to the calendar layout obtained by S1, the web page screenshot and the HTML of the root node of the calendar element processed by S2, using the second multimodal large language model MLLM to refer to the second python function example and generate a date selection code according to the second task rule; wherein the date selection code is a python function code.

9. The web calendar selection method applicable to RPA according to claim 8, characterized in that: The second python function example is divided into two, one is a single-month function example and the other is a multi-month function example. If the calendar page displays the date information of a single month, select the single-month function example; if the calendar page displays the date information of multiple months, select the multi-month function example. The input parameters of the single-month function example are the Playwright page information and the specified date. The function includes four parts: 1) Introduction to the input parameters; 2) Obtaining the specific month and date based on the specified date; 3) Locating the date element; 4) Clicking the date element; The input parameters of the multi-month function example are the Playwright page information and the specified date. The function consists of four parts: 1) Introduction to the input parameters; 2) Obtaining the specific month and date based on the specified date; 3) Locating the month element, and then locating the date element based on the month element; 4) Click the date element; and / or, the second task rules include: Rule 1, requiring the MLLM to observe the HTML of the root node of the calendar element, refer to the second python function sample and use Playwright coding; Rule 2: When faced with a multi-month calendar page, select the month first and then the date. Rule 3: The selected month element must be the parent node of the date element. Rule 4: Avoid all elements containing the "disable" field.

10. A web calendar selection device suitable for RPA, characterized in that: The device comprises: The acquisition module is used to: locate the calendar element on the web page, determine the root node of the calendar element and the calendar layout; The HTML processing module is used to: obtain the HTML of the root node of the calendar element, and perform hierarchical processing on the HTML of the root node of the calendar element; A first code generation module is used to: generate a calendar jump code using a first multimodal large language model MLLM according to the webpage screenshot and the HTML of the root node of the calendar element processed by S2; the calendar jump code can automatically jump to the calendar month and year; The first code running module is used to: run the calendar jump code so that the calendar jumps to the correct year and month layout; A second code generation module is used to generate a date selection code using a second multimodal large language model MLLM according to the calendar layout and webpage screenshot obtained by S1 and the HTML of the root node of the calendar element processed by S2; The second code running module is used to: run the date selection code so that the calendar jumps to the correct date.

Images