Learning method, estimation method, learning estimation system, and program

Abstract

In order to learn / estimate correspondence relationships between input fields and elements corresponding to those input fields on the basis of a visual configuration of a screen, this learning estimation system 200 comprises a learning device 210 and an estimation device 220. The learning device comprises: a time-of-learning disposition means 231 that disposes input fields on a screen and elements corresponding with those input fields in a planar coordinate system; a time-of-learning acquisition means 232 that acquires the relative positional relationships in the planar coordinate system of the input fields and the elements corresponding to those input fields; and a machine learning model creation means 233 that creates a machine learning model by carrying out machine learning using teaching data that uses the relative positional relationships as correct-answer labels as input. The estimation device comprises: a time-of-estimation disposition means 241 that disposes input fields on a screen and elements in a planar coordinate system; a time-of-estimation acquisition means 242 that acquires the relative positional relationship in the planar coordinate system of a certain input field and element; and an estimation means 243 that inputs the relative positional relationship into the learned model, and estimates that the elements in output relative positions are the elements corresponding to the input fields.

Description

[Technical Field]

[0001] The present invention relates to a learning method, an estimation method, a learning estimation system, and a program. [Background technology]

[0002] With the advancement of the Internet and cloud computing, the number of applications that run in local environments is decreasing, and instead, applications that run on the Internet or intranet and are used via a web browser are increasing. For these applications, there is a demand for the ability to automatically enter information into input fields on web pages. For example, to ensure that an application is created according to specifications, there is a need to automatically enter various information into input fields during application development, thereby efficiently testing the application's behavior.

[0003] Additionally, when introducing RPA (Robotic Process Automation) to existing applications to perform operations previously performed by humans, there is a demand for automatically entering information into input fields on web pages. In such cases, it is necessary to identify the input field and enter the information required by that field appropriately, but currently, in order to identify the input field, it is necessary to manually obtain the XPath of the input field.

[0004] Furthermore, on a web page, input fields are usually displayed together with words, phrases, or sentences (hereinafter referred to as "elements") that explain the input fields. In order to identify an input field, it is necessary to associate it with the element that corresponds to it. However, because web pages are written in a structured document called HTML (Hypertext Markup Language), conventionally, input fields are identified by associating the two on a rule-based basis using the hierarchical structure of HTML.

[0005] For example, Patent Document 1 discloses a webpage analysis device that can efficiently extract desired information from many webpages on Internet websites. This webpage analysis device analyzes the hierarchical structure of tags in structured documents that make up a webpage, assigns depth points to each line according to its depth from the root, and adds keyword points according to the keyword to the depth points for lines containing keywords. Based on the depth points and keyword points, the device then extracts a predetermined range, including the line containing the keyword and several lines before and after it, as a target block. By extracting information that meets predetermined conditions from within the extracted target block, the device can automatically extract desired information that is often posted close to the keyword from the webpage.

[0006] Patent Document 2 discloses a user interface recognition device that can correctly recognize a user interface. This user interface recognition device includes a search rule generation unit that generates search rules for searching for search target parts based on user interface information that indicates a group of parts that make up the user interface and the positional relationships between the group of parts, search target part information that indicates parts to be searched for among the group of parts, and structure definition information that defines the structure of the user interface, a part search unit that, when new user interface information is input, searches for the search target parts from the group of parts indicated in the new user interface information using the search rules generated by the search rule generation unit, and an output unit that outputs the search results by the part search unit. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Republished Publication No. 2013-38519 [Patent Document 2] Republished Publication No. 2009-66762 Summary of the Invention [Problem to be solved by the invention]

[0008] However, HTML coding methods vary widely depending on the application, and elements and input fields do not necessarily have the same hierarchical structure, making it difficult to support all web pages based on rules.On the other hand, input fields on web pages and the elements that correspond to them are usually arranged so that humans can visually determine the correspondence. Therefore, the present invention provides a method, device, system, and program for learning and estimating the correspondence between input fields on a web page and the elements corresponding to the input fields, based on the visual structure of the web page rendered by a web browser, rather than on the HTML that describes the web page.Furthermore, taking into account the growth of new devices such as smartphones and the continued widespread use of applications on Windows (registered trademark), the present invention provides a method, device, system, and program for learning and estimating the correspondence between input fields on a screen and the elements corresponding to the input fields, based on the visual structure of the screen displayed on the display by application programs executed on various platforms, not just web browsers. [Means for solving the problem]

[0009] In order to solve the above problem, a learning method executed by a computer is provided, which includes the steps of: arranging one or more input fields and elements corresponding to the one or more input fields in a screen displayed by a computer program on a display in a planar coordinate system; acquiring the relative positional relationship between one of the one or more input fields and the element corresponding to the input field in the planar coordinate system; and performing machine learning using training data, which uses the acquired relative positional relationship as a correct label, as input, to create a machine learning model that outputs the relative positional relationship between the one input field and the element corresponding to the input field in response to the relative positional relationship between the one input field and the one or more elements being input. According to this, by placing the input fields displayed on the display and the elements corresponding to those input fields in a planar coordinate system using a computer program and using the relative positional relationship between the two in the planar coordinate system as training data for machine learning, it is possible to provide a learning method for learning the correspondence between the input fields on the screen and the elements corresponding to those input fields based on the visual configuration of the screen displayed on the display.

[0010] Furthermore, the method may include a step of creating a partial plane in the plane coordinate system that includes an element corresponding to the one input field, other input fields, and other elements corresponding to the other input fields that are within a predetermined range around the one input field, and the obtaining step may be characterized by obtaining a relative positional relationship between the one input field and the element corresponding to the input field that are included in the partial plane. According to this, by using machine learning to learn the relative positional relationship between an input field in a planar coordinate system and the element corresponding to that input field as training data for each partial plane included in a specified range around the input field that is part of the display screen, it is possible to learn patterns of relative positional relationships that are common across partial planes even if the layout of the entire display screen is different.

[0011] Furthermore, the plane coordinate system may be characterized as a plane divided into a grid of a predetermined size. According to this, since the plane coordinate system is a grid of a predetermined size, learning can be performed with high learning accuracy even if the size of the displayed font changes.

[0012] Furthermore, the method may include a step of deleting blank rows and blank columns in a plane divided into a grid, in which no input field or element is included in a row or column, and the obtaining step may be characterized by obtaining the relative positional relationship between the input field and the element corresponding to that input field after the blank rows and blank columns have been deleted. According to this, by learning based on the relative positional relationship in which blank rows and columns of the grid are deleted, learning can be performed with high learning accuracy even if the margins change.

[0013] In order to solve the above problem, there is provided an estimation method that is executed by a computer that references a machine learning model learned by the above learning method, and that estimates the correspondence between input fields and elements in a screen that a computer program displays on a display, the estimation method including the steps of: arranging one or more input fields and one or more elements in a screen that the computer program displays on a display in a planar coordinate system; acquiring the relative positional relationship between one of the one or more input fields and the one or more elements in the planar coordinate system; inputting the acquired relative positional relationship into a machine learning model, acquiring the relative positional relationship in which an element corresponding to the one input field exists that is output, and estimating that the element existing in that relative position is the element corresponding to the one input field. According to this, the input field and elements displayed on the display are placed in a planar coordinate system by a computer program, the relative positional relationship between the two in the planar coordinate system is input into a machine learning model learned by the above-mentioned learning method, and the element corresponding to the input field is estimated based on the relative positional relationship in which the element corresponding to the output input field exists, thereby providing an estimation method for estimating the correspondence between the input field on the screen and the element corresponding to that input field based on the visual configuration of the screen displayed on the display.

[0014] Furthermore, the method may include a step of creating a partial plane in a planar coordinate system that includes one or more elements within a predetermined range around the one input field, and the acquiring step may be characterized by acquiring the relative positional relationship between the one input field and the one or more elements included in the partial plane. According to this, by estimating the elements corresponding to the input field in units of partial planes included in a specified range around the input field that is part of the display screen, it is possible to estimate the pattern of relative positional relationships that is common to the partial planes even if the layout of the entire display screen is different.

[0015] Furthermore, the plane coordinate system may be characterized as a plane divided into a grid of a predetermined size. According to this, since the plane coordinate system is a grid of a predetermined size, estimation can be performed with high accuracy even if the size of the displayed font changes.

[0016] Furthermore, the method may include a step of deleting blank rows and blank columns in a plane divided into a grid, in which no input field or element is included in a row or column, and the obtaining step may be characterized by obtaining the relative positional relationship between the input field and the one or more elements thereof after the blank rows and blank columns have been deleted. According to this, by making an estimation based on the relative positional relationship in which blank rows and columns of the grid have been deleted, it is possible to make an estimation with high accuracy even if the margins change.

[0017] In order to solve the above problems, a learning and estimation system is provided, which includes a learning device that learns the correspondence between input fields and elements in a screen that a computer program displays on a display, and an estimation device that estimates the correspondence. The learning device includes: a learning-time arrangement means that arranges one or more input fields in a screen that the computer program displays on a display and the elements corresponding to the one or more input fields in a plane coordinate system; a learning-time acquisition means that acquires the relative positional relationship between one of the one or more input fields and the element corresponding to the input field in the plane coordinate system; and a learning-time acquisition means that performs machine learning using training data that uses the acquired relative positional relationship as an input and that labels the one or more elements correctly, thereby estimating the relative positional relationship between the one input field and the one or more elements in response to the input of the relative positional relationship between the one input field and the one or more elements. and a machine learning model creation means for creating a machine learning model that outputs the relative positional relationship between the input field and an element corresponding to that input field, wherein the estimation device comprises: an estimation-time placement means for placing one or more input fields and one or more elements in a screen displayed on a display by a computer program in a plane coordinate system; an estimation-time acquisition means for acquiring the relative positional relationship between one of the one or more input fields and the one or more elements in the plane coordinate system; and an estimation means for inputting the acquired relative positional relationship into a trained machine learning model created by the learning device, acquiring the relative positional relationship in which an element corresponding to the one input field is output, and estimating that the element present at that relative position is the element corresponding to the one input field. According to this, by using a computer program to place the input field displayed on the display and the element corresponding to that input field in a planar coordinate system, and by using the relative positional relationship between the two in the planar coordinate system as training data for machine learning, it is possible to learn the correspondence between the input field on the screen and the element corresponding to that input field based on the visual configuration of the screen displayed on the display; by placing the input field and element displayed on the display in a planar coordinate system, inputting the relative positional relationship between the two in the planar coordinate system into the above-mentioned trained machine learning model, and estimating the element corresponding to the input field based on the relative positional relationship in which the element corresponding to the input field is located, it is possible to provide a learning estimation system that can estimate the correspondence between the input field on the screen and the element corresponding to that input field based on the visual configuration of the screen displayed on the display.

[0018] In order to solve the above problem, a program is provided to cause a computer to function as: a learning means that arranges one or more input fields and elements corresponding to the one or more input fields in a screen displayed by a computer program in a planar coordinate system, obtains the relative positional relationship between one of the one or more input fields and the elements corresponding to the input field in the planar coordinate system, and performs machine learning using training data with the obtained relative positional relationship as a correct label as input, thereby creating a machine learning model that outputs the relative positional relationship between one input field and the elements corresponding to the input field in response to the relative positional relationship between the one input field and the one or more elements being input; and an estimation means that arranges one or more input fields and one or more elements in a screen displayed by a computer program in a planar coordinate system, obtains the relative positional relationship between one of the one or more input fields and the one or more elements in the planar coordinate system, inputs the obtained relative positional relationship into the trained machine learning model created by the learning means, obtains the relative positional relationship in which an element corresponding to the one input field is present and estimates that the element present in that relative position is the element corresponding to the one input field. According to this, by using a computer program to place the input field displayed on the display and the element corresponding to that input field in a planar coordinate system, and by using the relative positional relationship between the two in the planar coordinate system as training data for machine learning, it is possible to learn the correspondence between the input field on the screen and the element corresponding to that input field based on the visual configuration of the screen displayed on the display; by placing the input field and element displayed on the display in a planar coordinate system, inputting the relative positional relationship between the two in the planar coordinate system into the above-mentioned trained machine learning model, and estimating the element corresponding to the input field based on the relative positional relationship in which the element corresponding to the input field is located, it is possible to provide a program that causes a computer to function to estimate the correspondence between the input field on the screen and the element corresponding to that input field based on the visual configuration of the screen displayed on the display. [Effects of the Invention]

[0019] As described above, the present invention provides a method, device, system, and program for learning and estimating the correspondence between input fields in a web page and the elements corresponding to the input fields, based on the visual configuration of the web page rendered by a web browser, rather than on the HTML that describes the web page.Furthermore, the present invention provides a method, device, system, and program for learning and estimating the correspondence between input fields in a screen and the elements corresponding to the input fields, based on the visual configuration of the screen displayed on a display by an application program executed on various platforms. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a diagram illustrating a computer system for implementing an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of a learning estimation system according to an embodiment of the present invention. [Figure 3] FIG. 3 is a diagram illustrating an example of functional blocks of the learning estimation system according to the embodiment of the present invention. [Figure 4]FIG. 4 is a diagram showing an example of a screen displayed on the display. [Figure 5] FIG. 5 is a diagram showing a processing step I of the learning phase in the learning estimation system according to the embodiment of the present invention. [Figure 6] FIG. 6 is a diagram showing a processing step II of the learning phase in the learning estimation system according to the embodiment of the present invention. [Figure 7] FIG. 7 is a diagram showing a processing step III of the learning phase in the learning estimation system according to the embodiment of the present invention. [Figure 8] FIG. 8 is a diagram showing processing step IV of the learning phase in the learning estimation system according to the embodiment of the present invention. [Figure 9] FIG. 9 is a diagram showing a processing step V of the learning phase in the learning estimation system according to the embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing processing step I of the estimation phase in the learning estimation system according to the embodiment of the present invention. [Figure 11] FIG. 11 is a diagram showing a processing step II of the estimation phase in the learning estimation system according to the embodiment of the present invention. [Figure 12] FIG. 12 is a diagram showing processing step III of the estimation phase in the learning estimation system according to the embodiment of the present invention. [Figure 13] FIG. 13 is a diagram showing processing step IV of the estimation phase in the learning estimation system according to the embodiment of the present invention. [Figure 14] FIG. 14 is a diagram showing a processing step V of the estimation phase in the learning estimation system according to the embodiment of the present invention. [Figure 15] FIG. 15 is a diagram showing a processing step VI of the estimation phase in the learning estimation system according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. First, referring to Figure 1, a computer system 300 for implementing embodiments of the present disclosure will be described. The mechanisms and devices of various embodiments disclosed herein may be applied to any suitable computing system. The main components of the computer system 300 include one or more processors 302, memory 304, a terminal interface 312, a storage interface 314, an I / O (input / output) device interface 316, and a network interface 318. These components may be interconnected via a memory bus 306, an I / O bus 308, a bus interface unit 309, and an I / O bus interface unit 310.

[0022] Computer system 300 may include one or more general-purpose programmable central processing units (CPUs) 302A and 302B, collectively referred to as processors 302. In some embodiments, computer system 300 may include multiple processors, while in other embodiments, computer system 300 may be a single CPU system. Each processor 302 executes instructions stored in memory 304 and may include an on-board cache.

[0023] In one embodiment, memory 304 may include random-access semiconductor memory, storage devices, or storage media (either volatile or non-volatile) for storing data and programs. Memory 304 may store all or part of the programs, software modules, and data structures that implement the functions described herein. For example, memory 304 may store a learning estimation application 350. In one embodiment, learning estimation application 350 may include instructions or descriptions that execute the functions described below on processor 302.

[0024] In some embodiments, learning estimation application 350 may be implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and / or other physical hardware devices instead of or in addition to a processor-based system. In some embodiments, learning estimation application 350 may include data other than instructions or descriptions. In some embodiments, cameras, sensors, or other data input devices (not shown) may be provided to communicate directly with bus interface unit 309, processor 302, or other hardware of computer system 300.

[0025] Computer system 300 may include a bus interface unit 309 that provides communication between processor 302, memory 304, display system 324, and I / O bus interface unit 310. I / O bus interface unit 310 may couple to an I / O bus 308 for transferring data to and from various I / O units. I / O bus interface unit 310 may communicate via I / O bus 308 with multiple I / O interface units 312, 314, 316, and 318, also known as I / O processors (IOPs) or I / O adapters (IOAs).

[0026] Display system 324 may include a display controller, a display memory, or both. The display controller may provide video, audio, or both data to display device 326. Computer system 300 may also include one or more sensors or other devices configured to collect data and provide the data to processor 302. Display system 324 may be connected to display device 326, such as a standalone display screen, a television, a tablet, or a handheld device.

[0027] The I / O interface unit provides functionality for communicating with various storage or I / O devices. For example, the terminal interface unit 312 may be attached to user I / O devices 320, such as user output devices such as a video display, a television with speakers, and user input devices such as a keyboard, a mouse, a keypad, a touchpad, a trackball, buttons, a light pen, or other pointing device. A user may use a user interface to enter input data or instructions into the user I / O devices 320 and the computer system 300 and receive output data from the computer system 300 by operating the user input devices. The user interface may be displayed on a display, played through speakers, or printed via a printer via the user I / O devices 320, for example.

[0028] Storage interface 314 may accept one or more disk drives or direct access storage devices 322 (typically magnetic disk drive storage devices, but may also be an array of disk drives or other storage devices configured to appear as a single disk drive). In some embodiments, storage device 322 may be implemented as any secondary storage device. Contents of memory 304 may be stored in storage device 322 and retrieved as needed from storage device 322. I / O device interface 316 may provide an interface to other I / O devices, such as printers, fax machines, etc. Network interface 318 may provide a communications path that allows computer system 300 and other devices to communicate with each other. This communications path may be, for example, network 330.

[0029] In some embodiments, computer system 300 may be a device that receives requests from other computer systems (clients) without a direct user interface, such as a multi-user mainframe computer system, a single-user system, or a server computer. In other embodiments, computer system 300 may be a desktop computer, a portable computer, a laptop, a tablet computer, a pocket computer, a telephone, a smartphone, or any other suitable electronic device.

[0030] Next, a learning estimation system 200 according to the present invention will be described with reference to Fig. 2. The learning estimation system 200 according to the present invention includes a learning device 210, an estimation device 220, a mobile terminal 250, and an external system 260, which are connected via a communication network 265. The communication network 265 may include, for example, a local area network (LAN), a wide area network (WAN), a satellite network, a cable network, a Wi-Fi network, or any combination thereof.

[0031] The learning device 210 and the estimation device 220 are computing devices that respectively execute a learning method for learning a correspondence between an input field on a screen and an element corresponding to the input field and an estimation method for estimating a correspondence between an input field on a screen and an element corresponding to the input field according to the present invention. The learning device 210 and the estimation device 220 may be, for example, a desktop computer, a server computer, a laptop computer, a tablet computer, a workstation, a mobile terminal, or any other type of computing device, and are not particularly limited in the present invention.

[0032] The learning device 210 includes a processor 211 for executing instructions stored in memory 230, an I / O interface 212 for controlling communication between internal and external devices of the learning device 210, a network interface 213 for controlling communication via a communication network 265, a user I / O interface 214 for accepting input from a user, memory 230 storing functional units for performing each function of the learning method according to an embodiment of the present invention, and a bus 216 for controlling communication between these components.

[0033] The estimation device 220 includes a processor 221 for executing instructions stored in a memory 240, an I / O interface 222 for controlling communication between internal and external devices of the estimation device 220, a network interface 223 for controlling communication via a communication network 265, a user I / O interface 224 for receiving input from a user, the memory 240 storing functional units for performing each function of the estimation method according to an embodiment of the present invention, and a bus 226 for controlling communication between these components.

[0034] The memory 230 of the learning device 210 includes, for example, a learning screen information input means 234 that accepts input of screen information to be displayed on a display by a computer program for learning; a learning element acquisition means 235 that creates a list of elements present on the displayed screen; a learning arrangement means 231 that arranges one or more input fields on the screen and the elements corresponding to the one or more input fields in a planar coordinate system; a learning acquisition means 232 that acquires the relative positional relationship between one of the one or more input fields and the element corresponding to the input field in the planar coordinate system; and a machine learning model creation means 233 that creates a machine learning model that outputs the relative positional relationship between the one input field and the element corresponding to the input field in response to the input of the relative positional relationship between the one input field and one or more elements by machine learning using training data that uses the acquired relative positional relationship as an input and labels the acquired relative positional relationship.

[0035] The memory 240 of the estimation device 220 includes, for example, an estimation-time screen information input means 244 that accepts input of screen information to be displayed on a display by a computer program for estimation; an estimation-time element acquisition means 245 that creates a list of elements present on the displayed screen; an estimation-time arrangement means 241 that arranges one or more input fields on the screen and elements corresponding to the one or more input fields in a plane coordinate system; an estimation-time acquisition means 242 that acquires a relative positional relationship between one of the one or more input fields and the one or more elements thereof in the plane coordinate system; an estimation means 243 that inputs the acquired relative positional relationship into a trained machine learning model created by the learning device, acquires a relative positional relationship in which an element corresponding to the one or more input fields is present, and estimates that an element present at that relative position is the element corresponding to the one or more input fields; and an output means 246 that outputs the estimated element.

[0036] The functional units included in the learning device 210 and the estimation device 220 may be software modules constituting the learning estimation application 350 in the computer system 300 shown in FIG. 1, or may be independent dedicated hardware devices. The functional units may be implemented in the same computing environment or in a distributed computing environment. While the learning device 210 and the estimation device 220 are executed on different computing devices in this embodiment, they may also be integrated and executed on a single computing device.

[0037] The external system 260 may be a system that executes a computer program that generates a screen to be displayed on a display. This external system 260 may be, for example, a system configured from a remote personal computer, a server, a storage device, a database, cloud-based storage, or a combination of these. For convenience of explanation, one external system 260 is shown in FIG. 2 , but in reality, the learning device 210 and the estimation device 220 are connected to multiple external systems and access the necessary external system depending on the content and purpose of the screen to be learned and estimated.

[0038] The computer program that generates the screen to be displayed on the display of external system 260 is not particularly limited and may be an operating system such as Windows (registered trademark) or iOS (registered trademark), middleware or an application that runs on the operating system, a web server that outputs HTML, etc. External system 260 may be configured to access a server connected to the Internet, crawl and collect a wide variety of data on the web, and generate a screen to be displayed on the display of learning device 210 or estimation device 220.

[0039] The mobile terminal 250 may access the external system 260 and display a screen, or may access the learning device 210 or the estimation device 220 and be used as an operation terminal for these devices. The mobile terminal 250 may be, for example, a personal computer, a mobile terminal such as a smartphone, a tablet computer, or the like.

[0040] 3 to 15, the logical configuration of each functional unit included in the learning device 210 and the estimation device 220 will be described. As described in FIG. 2, the functional units included in the learning device 210 are a learning-time screen information input means 234, a learning-time element acquisition means 235, a learning-time arrangement means 231, a learning-time acquisition means 232, and a machine learning model creation means 233. As described in FIG. 2, the functional units included in the estimation device 220 are an estimation-time screen information input means 244, an estimation-time element acquisition means 245, an estimation-time arrangement means 241, an estimation-time acquisition means 242, an estimation means 243, and an output means 246.

[0041] First, the learning device 210 will be described. The learning screen information input means 234 has a function of accepting input of screen information to be displayed on the display by a computer program. A computer program is not limited to a native application that runs on an operating system such as Windows (registered trademark), or a web browser that renders HTML output from a remote web server. A display is a device that displays what humans perceive through their eyes, including not only a display function but also a touch panel function and a projector function that projects onto a screen.

[0042] The screen information displayed on a display may be the entire or part of a web page composed of HTML in the case of a web browser, or in the case of a native application, it may be the relative position from the base point to the pixel when drawing on the display, or the relative position from the base point of the window frame to the pixel. The screen information displayed on a display is information that represents what humans perceive through their vision.

[0043] FIG. 4 shows an example of a screen displayed on a display, an example of an inquiry form commonly found on the Internet. This inquiry form has the words "Name" below the words "Inquiry Form," the words "Last Name" to the right of that, an input field 1 to the right, the words "First Name" to the right of that, an input field 2 to the right, the words "Phone Number" below the words "Name," an input field 3 to the right of that, the words "Email Address" below the words "Phone Number," an input field 4 to the right of that, and the words "Content" below the words "Email Address," an input field 5 below that. The learning screen information input means 234 may accept input of such an inquiry form as HTML information, or may accept input as relative positional information in terms of the number of horizontal and vertical pixels from the upper left corner of the "Name" characters to each character and input field. Furthermore, when the learning screen information input means 234 accepts input as HTML information, any tags may be used for the input.

[0044] The learning element acquisition means 235 creates a list of elements present on the displayed screen. Elements corresponding to input fields are characters or images arranged so that a human can visually recognize their correspondence with the input fields. For example, in the example of FIG. 4, the elements corresponding to input field 1 are the characters "first name" and "last name," the elements corresponding to input field 2 are the characters "first name" and "first name," the elements corresponding to input field 3 are the characters "phone number," the elements corresponding to input field 4 are the characters "email address," and the elements corresponding to input field 5 are the characters "content." Such correspondences are easily recognized by humans, but HTML description methods vary widely depending on the application, and elements and input fields do not necessarily have the same hierarchical structure. Furthermore, it is difficult to recognize elements on all screens using pixel information or rules.

[0045] The learning-time element acquisition means 235 first creates a list of what elements are present on the displayed screen. For example, in the example of Fig. 4, although the correspondence with the input fields is not clear, this screen contains a mixture of characters such as "name," "telephone number," "email address," and "contents" along with the input fields, so it is easy to recognize that there are elements on the displayed screen that may correspond to input fields 1 to 5. Therefore, the learning-time element acquisition means 235 creates a list of these elements and stores it in the storage unit 215.

[0046] The learning placement means 231, as shown in FIG. 3, includes a grid division process and a blank row / column deletion process, and places one or more input fields on the screen and elements corresponding to the one or more input fields on a plane coordinate system. The grid division process places the input fields and elements drawn on the screen, based on the screen information received by the learning screen information input means 234, on a plane divided into grids of a predetermined size. The predetermined size of the grid is preferably equal to the font size of the drawn characters, etc., particularly in the vertical direction, and is preferably large enough to accommodate one or several characters of that font size in the horizontal direction. Furthermore, if there are multiple font sizes on the screen, a grid that matches the smallest font size is preferred to avoid placing multiple elements in one grid. Furthermore, if there are input fields smaller than the characters, a grid that matches the size of the input fields is preferred.

[0047] The grid division process divides the screen shown in Figure 4 into grids, as shown in Figure 5, for example. On this screen, characters such as "Inquiry Form" and "Name" are rendered in the same font size. Therefore, when the vertical size is determined using these font sizes, one line separates "Inquiry Form" and "Name," two lines separate "Name" and "Phone Number," two lines separate "Phone Number" and "Email Address," and two lines separate "Email Address" and "Content." When characters are separated by two lines, blank lines are formed between them. The input field 5 below the "Content" characters is positioned so that it is included in multiple grids vertically. Because the horizontal grid size is large enough to accommodate approximately two to three characters, the cell surrounded by the vertical and horizontal lines forms a rectangle that is longer in the horizontal direction.

[0048] When the learning-time placement means 231 places input fields and elements on a plane divided into grids, the cells where the upper left corners of the input fields and elements are located are the positions where the input fields and elements are placed. For example, the "inquiry form" element is placed in cell (r1, c1) where the upper left corner of the characters "inquiry form" is located. The "name" element is placed in cell (r2, c1) where the upper left corner of the characters "name" is located, and the "last name" element is placed in cell (r2, c3) where the upper left corner of the characters "last name" is located. Explanations of other elements will be omitted, but the cells in which the elements are placed are indicated by a dot pattern. Input field 1 is placed in cell (r2, c4), and input field 2 is placed in cell (r2, c8). Explanations of other input fields will be omitted, but the cells in which the input fields are placed are indicated in gray.

[0049] Because text displayed on a computer program screen is typically written horizontally and aligned left to right, cells containing elements and input fields (the dotted or gray cells in this figure) are often aligned not only horizontally but also vertically. This can result in not only blank rows but also blank columns, where cells with no input fields or elements are aligned vertically. In this example, the planar coordinate system is a grid represented by rows and columns, but is not limited to this. For example, the planar coordinate system may be an XY plane coordinate system with the upper left coordinate of the inquiry form as the origin, or a polar coordinate system, as long as it can represent the relative positions of one or more input fields and elements on the screen. However, by dividing the planar coordinate system into a grid of a predetermined size, high-accuracy learning can be achieved even when the displayed font size changes.

[0050] When a blank row or column that does not contain any input field or element occurs in a row or column on a plane divided into grids, the blank row / column deletion process of the learning-time arrangement means 231 deletes the blank row and column as shown in Fig. 6. That is, in the left diagram of Fig. 6 (same as the right diagram of Fig. 5), rows r3, r5, r7, and r10 are blank rows, and columns c2, c5, c6, c9, and c10 are blank columns, so the blank row / column deletion process deletes these rows and columns. This results in a grid like the one shown on the right diagram of the figure, and the cells (dot patterns) in which character elements are placed are lined up vertically in column C1 with R1 to R5, becoming cells (R2, C2) and cells (R2, C4).

[0051] On the other hand, the cells (gray) in which input fields 1 to 4 are placed are placed to the right of the cells (dot pattern) in which character elements are placed in rows R2 to R4, and only input field 5 is placed below the cell (dot pattern) in which character elements are placed. The learning acquisition means 232, which will be described later, acquires the relative positional relationship between the input fields and the elements corresponding to those input fields after deleting blank rows and columns in this way, and thus performs learning based on the relative positional relationship in which blank rows and columns in the grid have been deleted, making it possible to perform learning with high learning accuracy even if the margins change.

[0052] Next, the learning acquisition means 232, which includes a partial plane process and a relative positional relationship extraction process as shown in FIG. 3, acquires the relative positional relationship between a certain input field among one or more input fields in a plane coordinate system and the element corresponding to that input field. The partial plane process creates a partial plane that includes the element corresponding to that input field, other input fields, and other elements corresponding to the other input fields within a predetermined range around that input field in a grid plane coordinate system. For example, as shown in FIG. 7, the partial plane process creates a partial plane within a range of ±1 row and ±2 columns around input field 1 in cell (R2, C3), i.e., a 5x3 cell. This partial plane includes multiple elements including the characters "Inquiry Form," "Name," "Last Name," "First Name," and "Phone Number," as well as input field 2 and input field 3.

[0053] The partial plane processing can also create a partial plane within a similar 5x3 cell around input field 3 in cell (R3, C2). This partial plane includes multiple elements, including the characters for "Name," "Last Name," "First Name," "Phone Number," and "Email Address," as well as input field 1 and input field 4. The partial plane processing may also create a partial plane centered on input field 2, input field 4, or input field 5. While the partial plane processing is optional, it is preferable to perform this processing because it can learn patterns of relative positional relationships common to partial planes even when the overall layout of the display screen is different. The partial plane processing may also be performed using XY plane coordinates or polar coordinates, rather than just a grid coordinate system. For example, a partial plane may be created by setting a predetermined XY range or radius range with input field 1 as the origin.

[0054] Next, the relative positional relationship extraction process of the learning-time acquisition means 232 acquires the relative positional relationship between a certain input field and the element corresponding to that input field, which is included in the entire screen or in each partial plane. For example, as shown in the upper right diagram of FIG. 7, with respect to input field 1 of cell (R2, C3), cell (R2, C1) containing the characters "Name", which is the element corresponding to this input field 1, may be expressed as cell (0, -2) or column direction (-2). Also, with respect to input field 1 of cell (R2, C3), cell (R1, C1) containing the characters "Inquiry Form", which is an element not corresponding to this input field 1, may be expressed as cell (-1, -2) or column direction (-7).

[0055] Furthermore, the relative positional relationship extraction process may, for example, as shown in the lower right diagram of Figure 7, determine that the characters "phone number" in cell (R3, C1) have a relative positional relationship with the input field 3 in cell (R3, C2) expressed as cell (0, -1) or in the column direction (-1). Conversely, the input field 3 in cell (R3, C2) may have a relative positional relationship with the characters "phone number" in cell (R3, C1) expressed as cell (0, +1) or in the column direction (+1). In other words, the relative positional relationship extraction process obtains the relative positional relationship between a certain input field and an element corresponding to that input field.

[0056] Next, the machine learning model creation means 233, as shown in FIG. 3, includes a feature extraction process and a machine learning process. By performing machine learning using training data with the acquired relative positional relationships as correct labels as input, the machine learning model creates a model that outputs the relative positional relationship between an input field and one or more elements in response to the input of the relative positional relationship between the input field and the elements corresponding to the input field. For example, the upper center diagram of FIG. 8 (the upper right diagram of FIG. 7) shows a case where, in a partial plane of a 5x3 grid centered on input field 1 in cell (R2, C3), the numbers 0 to 14 are assigned to 15 cells (R1, C1) to (R3, C5) in order by the relative positional relationship extraction process, and relative positional relationships are extracted. In this case, input field 1 is assigned the number 7, and as possible corresponding elements (dot patterns), 0 is assigned to the characters "inquiry form," 5 to the characters "name," 6 to the characters "last name," 8 to the characters "last name," and 10 to the characters "phone number."

[0057] When there is such a relative positional relationship, the feature extraction process extracts a 15-dimensional vector (1 0 0 0 0 1 1 2 1 2 1 2 0 0 0) as a feature. In the vector, 1 indicates a cell where an element is placed, 2 indicates a cell where an input field is placed, and 0 indicates a cell where nothing is placed. The element corresponding to input field 1 is known to be the correct answer, which is the character for "Name" in cell (R2, C1) assigned the number 5, so the correct answer label can be set to 5.

[0058] The lower center diagram of FIG. 8 (lower right diagram of FIG. 7) shows a case where relative positional relationships are extracted by similarly assigning numbers 0 to 14 to a 5x3 grid subplane centered on input field 3 in cell (R3, C2). In this case, input field 3 is assigned the number 7, and potentially corresponding elements (dot patterns) are assigned the following: 1 to the characters in "Name," 2 to the characters in "Last Name," 4 to the characters in "Last Name," 6 to the characters in "Phone Number," and 11 to the characters in "Email Address." In the case of such relative positional relationships, the feature extraction process extracts a 15-dimensional vector (0 1 1 2 1 0 1 2 0 0 0 1 2 0 0) as a feature. Since the element corresponding to input field 3 is known to be the characters in "Phone Number" in cell (R3, C1) assigned the number 6, the correct answer can be assigned a correct answer label of 6. The feature extraction process stores training data with such relative positional relationships as correct answer labels in the storage unit 215.

[0059] The machine learning process of the machine learning model creation means 233 performs machine learning using the training data created by the feature extraction process as input. For example, as shown in FIG. 9, the machine learning process inputs a pair of a 15-dimensional vector as a feature and a correct label, and executes learning to build a model. While a decision tree or random forest is preferred as the machine learning algorithm, there is no particular limitation, and neural networks or other algorithms may also be used. When the machine learning process determines that it has learned with a sufficient amount of training data, it stores the trained model in the storage unit 215. In response to input of the relative positional relationship between a certain input field and one or more elements, this trained model outputs the relative positional relationship between the input field and the elements corresponding to the input field.

[0060] In this way, by having learning-time arrangement means 231 that arranges input fields displayed on a display and elements corresponding to the input fields in a planar coordinate system using a computer program, learning-time acquisition means 232 that acquires the relative positional relationship between them in the planar coordinate system, and machine-learning model creation means 233 that performs machine learning using this relative positional relationship as training data, it is possible to provide a learning device 210 that learns the correspondence between input fields on a screen and the elements corresponding to the input fields based on the visual configuration of the screen displayed on the display. Also, by performing partial plane processing using the learning-time acquisition means 232 to perform machine learning using the relative positional relationship between input fields and elements corresponding to the input fields in the planar coordinate system as training data in units of partial planes included in a predetermined range around the input fields that are part of the display screen, it is possible to learn patterns of relative positional relationships that are common to partial planes even if the arrangement of the entire display screen is different.

[0061] Furthermore, what has been described above is a learning method in which input fields displayed on a display and elements corresponding to those input fields are placed in a planar coordinate system, and the relative positional relationship between the two in the planar coordinate system is used as training data for machine learning.This makes it possible to provide a learning method for learning the correspondence between input fields on a screen and elements corresponding to those input fields, based on the visual configuration of the screen displayed on a display.

[0062] Learning the correspondence between input fields on a screen and the elements corresponding to those input fields based on the visual configuration of the screen displayed on a display means that it is not dependent on the display method on the screen. In other words, taking into account the growth of new devices such as smartphones and the fact that applications on Windows (registered trademark) are still widely used, the device and method according to the present invention can learn the correspondence between input fields on a screen and the elements corresponding to those input fields based on the visual configuration of screens displayed on a display by application programs executed on various platforms, not just web browsers.

[0063] Next, the estimation device 220 will be described. The estimation time screen information input means 244 has a function of accepting input of screen information to be displayed on a display by a computer program. As described above, the computer program and the display are not particularly limited. As described above, the screen information displayed on the display is also not particularly limited, and is information that expresses what humans perceive through their eyesight.

[0064] The left diagram in FIG. 10 shows an example of a screen displayed on a display. This is an example of an inquiry form commonly seen on the Internet, but has a different configuration from the example screen during learning shown in FIG. 4. This inquiry form has the characters "Name" below the word "Inquiry," the characters "Last Name" to the right of that, input field 1 further to the right, the characters "First Name" to the right of that, input field 2 further to the right, the characters "Age" below the characters "Name," input field 3 to the right of that, the characters "Years" to the right of that, the characters "Address" below the characters "Age," input field 4 to the right of that, the characters "Inquiry Content" below the characters "Address," and input field 5 to the lower right of that. The estimation time screen information input means 244 may accept input of such an inquiry form as HTML information, or may accept input as information on relative positional relationships in terms of the number of pixels in the horizontal and vertical directions from the upper left corner of the characters "Name" as the origin to each character and input field.

[0065] The estimation-time element acquisition means 245 creates a list of elements present on the displayed screen, similar to the learning-time element acquisition means 235. For example, in the example on the left side of FIG. 10 , this screen contains input fields as well as characters such as "Name," "Age," "Address," and "Inquiry details," so it is easy to recognize that elements that may correspond to input fields 1 to 5 are present on the displayed screen. Therefore, the estimation-time element acquisition means 245 creates a list of these elements and stores it in the storage unit 225.

[0066] The estimation placement means 241, as shown in FIG. 3, includes a grid division process and a blank row / column deletion process, and places one or more input fields and one or more elements on the screen in a plane coordinate system. The grid division process places the input fields and elements drawn on the screen on a plane divided into grids of a predetermined size based on the screen information received by the estimation screen information input means 244. The predetermined size of the grid is preferably equal to the font size of the drawn characters, etc., particularly in the vertical direction, and is preferably large enough to accommodate one or several characters of that font size in the horizontal direction. Furthermore, if there are multiple font sizes on the screen, a grid that matches the smallest font size is preferred to avoid placing multiple elements on one grid. Furthermore, if there are input fields smaller than the characters, a grid that matches the size of the input fields is preferred.

[0067] The grid division process divides the screen shown in the left diagram of this figure into grids, as shown in the right diagram of Figure 10, for example. In this screen, characters such as "Inquiry" and "Name" are drawn using the same font size. Therefore, when the vertical size is determined using these font sizes, one line separates "Inquiry" and "Name," two lines separate "Name" and "Age," two lines separate "Age" and "Address," and two lines separate "Address" and "Inquiry Details." When characters are separated by two lines, blank lines are created between them. The input field 5 to the lower right of the "Inquiry Details" characters is positioned so that it is included in multiple grids vertically. Because the horizontal grid size is large enough to accommodate approximately two to three characters, the cell enclosed by the vertical and horizontal lines forms a rectangle that is longer in the horizontal direction.

[0068] When the estimation time placement means 241 places input fields and elements on a plane divided into grids, the cells where the upper left corners of the input fields and elements are located are regarded as the positions where the input fields and elements are placed. For example, the "inquiry" element is placed in cell (r1, c1) where the upper left corner of the characters "inquiry" is located. Furthermore, the "name" element is placed in cell (r2, c1) where the upper left corner of the characters "name" is located, and the "last name" element is placed in cell (r2, c3) where the upper left corner of the characters "last name" is located. Explanation of other elements will be omitted, but the cells where the elements are placed are indicated by a dot pattern. Furthermore, input field 1 is placed in cell (r2, c4), and input field 2 is placed in cell (r2, c8). Explanation of other input fields will be omitted, but the cells where the input fields are placed are indicated in gray.

[0069] In this example, a grid represented by rows and columns is used as a planar coordinate system, but if grid division processing is performed in the learning-time placement means 231 of the learning device 210, it is preferable that grid division processing is also performed in the estimation-time placement means 241 of the estimating device 220. In the estimating device 220, since the planar coordinate system is a grid of a predetermined size, estimation can be performed with high accuracy even if the size of the displayed font changes.

[0070] As described above, cells (dot patterns or gray cells in this figure) in which elements or input fields are placed are often arranged not only horizontally but also vertically, which can result in blank rows and blank columns. When a blank row or column that does not include any input fields or elements occurs within a row or column on a plane divided into grids, the blank row and column deletion process of the estimation-time placement means 241 deletes the blank row and column as shown in FIG. 11. That is, in the left diagram of FIG. 11 (same as the right diagram of FIG. 10), rows r3, r5, r7, and r10 are blank rows, and columns c2, c5, c6, c9, and c10 are blank columns, so the blank row and column deletion process deletes these rows and columns. This results in a grid like the one shown on the right diagram of this figure, where cells (dot patterns) in which character elements are placed are arranged vertically in column C1, with R1 to R5 lined up, resulting in cells (R2, C2), (R2, C4), and (R3, C4).

[0071] On the other hand, the cells (gray) in which input fields 1 to 4 are placed are placed to the right of the cells (dot pattern) in which character elements are placed in rows R2 to R4, and only input field 5 is placed to the lower right of the cell (dot pattern) in which character elements are placed. The estimation time acquisition means 242, which will be described later, acquires the relative positional relationship between the input fields and the elements corresponding to those input fields after deleting blank rows and columns in this way, thereby making an estimation based on the relative positional relationship after deleting blank rows and columns in the grid, and can provide high estimation accuracy even if the margins change.

[0072] If grid division processing is not performed in the placement means 231 during learning, it is preferable that grid division processing is not performed in the placement means 241 during estimation. Furthermore, if XY plane coordinates or polar coordinates are adopted in the placement means 231 during learning, it is preferable that the placement means 241 during estimation also adopts a similar coordinate system.

[0073] Next, the estimation time acquisition means 242, which includes partial plane processing and relative positional relationship extraction processing as shown in FIG. 3, acquires the relative positional relationship between a certain input field among one or more input fields and one or more elements in a plane coordinate system. The partial plane processing creates a partial plane including one or more elements and other input fields within a predetermined range around a certain input field in a grid plane coordinate system. For example, as shown in FIG. 12, the partial plane processing creates a partial plane with a size of 5x3 cells within a range of ±1 row and ±2 columns around input field 1 in cell (R2, C3). This partial plane includes multiple elements including the characters "inquiry", "name", "last name", "first name", "age", and "years old", as well as input field 2 and input field 3.

[0074] The partial plane processing can also create a similar 5x3 partial plane centered around input field 2 in cell (R3, C2). This partial plane includes multiple elements, including the characters for "Name," "Last Name," "First Name," "Age," and "Address," as well as input field 1 and input field 4. The partial plane processing may also create partial planes centered around input field 2, input field 4, or input field 5. While the partial plane processing is optional, it is preferable to perform this processing because it can estimate a common relative positional relationship pattern across partial planes even when the overall layout of the display screen is different. The partial plane processing may also be performed in XY plane coordinates or polar coordinates, rather than just the grid coordinate system. For example, a partial plane may be created by setting a predetermined XY range or radius range with input field 1 as the origin.

[0075] Next, the relative positional relationship extraction process of the estimation time acquisition means 242 acquires the relative positional relationship between a certain input field and one or more elements contained in the entire screen or each partial plane. For example, as shown in the upper right diagram of FIG. 12, the characters "Inquiry" in cell (R1, C1) may be expressed as cell (-1, -2) or column (-7) relative to input field 1 in cell (R2, C3). Also, the characters "Name" in cell (R2, C1) may be expressed as cell (0, -2) or column (-2) relative to input field 1 in cell (R2, C3).

[0076] Furthermore, in the relative positional relationship extraction process, for example, as shown in the lower right diagram of FIG. 12, the characters "age" in cell (R3, C1) may be expressed as cell (0, -1) or as column (-1) relative to the input field 3 in cell (R3, C2). Also, the characters "address" in cell (R4, C1) may be expressed as cell (1, -1) or as column (+4) relative to the input field 3 in cell (R3, C2). In other words, the relative positional relationship extraction process obtains the relative positional relationship between a certain input field and one or more elements.

[0077] In this way, the estimation time acquisition means 242 may create a partial plane including one or more elements within a predetermined range around one input field in the planar coordinate system, and acquire the relative positional relationship between the one input field included in the partial plane and one or more surrounding elements. By estimating the elements corresponding to the input field in units of partial planes included in a predetermined range around the input field that is part of the display screen, it is possible to estimate a pattern of relative positional relationship common to the partial planes even if the layout of the entire display screen is different.

[0078] Next, the estimation means 243 includes a feature extraction process and an estimation process as shown in Figure 3, inputs the relative positional relationship acquired by the estimation acquisition means 242 into the trained model created by the learning device 210, acquires the relative positional relationship in which an element corresponding to a certain input field to be output exists, and estimates that the element existing at that relative position is the element corresponding to that certain input field.

[0079] For example, the upper center diagram of FIG. 13 (the upper right diagram of FIG. 12) shows a case where, in a partial plane of a 5x3 grid centered on input field 1 in cell (R2, C3), the relative positional relationship extraction process assigns numbers 0 to 14 to 15 cells (R1, C1) to (R3, C5) in that order, extracting relative positional relationships. In this case, input field 1 is assigned the number 7, and by referring to the list of element names, the following possible corresponding elements (dot patterns) are assigned: 0 to the character "inquiry," 5 to the character "name," 6 to the character "surname," 8 to the character "family name," 10 to the character "age," and 13 to the character "years old." In the case of such relative positional relationships, the feature extraction process extracts a 15-dimensional vector (1 0 0 0 0 1 1 2 1 2 1 2 0 1 0) as a feature and stores it in storage unit 225 as an input feature. In the vector, 1 indicates a cell where an element is placed, 2 indicates a cell where an input field is placed, and 0 indicates a cell where nothing is placed.

[0080] 13 (the lower right diagram in FIG. 12) shows a case where numbers 0 to 14 are similarly assigned to a 5x3 grid partial plane centered on input field 3 of cell (R3, C2) and relative positional relationships are extracted. In this case, input field 3 is assigned 7, and potentially corresponding elements (dot patterns) are assigned 1 to the characters in "Name", 2 to the characters in "Last Name", 4 to the characters in "Last Name", 6 to the characters in "Age", and 11 to the characters in "Address". In the case of such a relative positional relationship, the feature extraction process extracts a 15-dimensional vector (0 1 1 2 1 0 1 2 0 0 0 1 2 0 0) as a feature and stores it in storage unit 225 as an input feature.

[0081] The estimation process inputs a vector (input feature) indicating the relative positional relationship between an input field and an element extracted in the feature extraction process into a trained model trained by the learning device 210. The trained model is configured to output the relative positional relationship between the input field and the elements corresponding to the input field when the relative positional relationship between the input field and one or more elements is input. Therefore, when the estimation process inputs a vector indicating the relative positional relationship between the input field and the elements into the trained model, the trained model outputs the relative positional relationship of the elements corresponding to the input field. The estimation process acquires the relative positional relationship that is output, and estimates that the element present at that relative position is the element corresponding to the input field.

[0082] For example, as shown in FIG. 14, when the vector (1 0 0 0 0 1 1 2 1 2 1 2 0 1 0) indicating the relative positional relationship obtained from the partial plane in the upper center of FIG. 13 is input to the trained model, the estimation result is obtained as a predicted label of 5. Referring to FIG. 13, the predicted label of 5 is the character element "Name" in cell (R2, C1). Note that this example is based on the premise that the element corresponding to input field 1, which is assigned the number 7 and is located in the center of the 15 cells, is to be found. Therefore, the estimation process estimated that the element corresponding to input field 1 is "Name".

[0083] Furthermore, when the vector (0 1 1 2 1 0 1 2 0 0 0 1 2 0 0) indicating the relative positional relationship obtained from the partial plane in the lower center diagram of FIG. 13 is input into the trained model, the estimation process obtains a predicted label of 6 as an estimation result. Referring to FIG. 13, the predicted label of 6 is the character element "age" in cell (R3, C1). Similarly, since the premise is to find the element corresponding to input field 3, which is assigned the number 7 and is located in the center of the 15 cells, the estimation process estimates that the element corresponding to input field 3 is "age". These estimation results are stored in the storage unit 225.

[0084] In this way, by arranging the input fields and elements displayed on the display in a planar coordinate system, inputting the relative positional relationship between the input fields and the elements in the planar coordinate system into a machine learning model learned by the learning device 210, and estimating the elements corresponding to the input fields based on the relative positional relationship in which the elements corresponding to the output input fields exist, it is possible to provide an estimation device 220 that estimates the correspondence between the input fields on the screen and the elements corresponding to those input fields based on the visual configuration of the screen displayed on the display.

[0085] Furthermore, what has been described above is an estimation method that arranges one or more input fields and one or more elements on a screen to be displayed on a display in a planar coordinate system, obtains the relative positional relationship between one of the one or more input fields and one or more elements in the planar coordinate system, inputs the obtained relative positional relationship into a trained machine learning model, obtains the relative positional relationship in which an element corresponding to the input field is output, and estimates that the element present at that relative position is the element corresponding to the input field. This makes it possible to estimate the correspondence between input fields on the screen and the elements corresponding to the input field based on the visual configuration of the screen displayed on the display.

[0086] Furthermore, estimating the correspondence between input fields on a screen and the elements corresponding to those input fields based on the visual configuration of the screen displayed on a display means that it is independent of the method of displaying the screen. In other words, the device and method according to the present invention can estimate the correspondence between input fields on a screen and the elements corresponding to those input fields based on the visual configuration of the screen displayed on a display by an application program executed on various platforms.

[0087] The learning and estimation system 200, which includes the learning device 210 and the estimation device 220, can learn and estimate the correspondence between input fields on a screen and the elements corresponding to those input fields, based on the visual configuration of the screen displayed on a display. The learning method and estimation method described above may also be provided as a program for causing a computer to function as a learning means or an estimation means. The program may be configured to be usable by downloading it from a server on which it is stored, or may be stored in a computer-readable recording medium and provided by the recording medium.

[0088] If the computer program displaying the screen is a web browser, the screen information displayed on the display may be written in HTML. Therefore, the estimation device 220 may include a receiving unit (not shown) that receives input of the URL of a web page written in HTML, and an HTML acquisition unit (not shown) that accesses the URL and acquires the HTML describing the web page. In this case, the estimation-time placement means 241 functions as a planar coordinate system placement unit that renders the HTML and places input fields and elements in a planar coordinate system, and the estimation-time acquisition means 242 functions as a relative positional relationship acquisition unit that acquires the relative positional relationship between an input field and one or more elements among one or more input fields in the planar coordinate system. The estimation means 243 then inputs the acquired relative positional relationship into a trained machine learning model, acquires the relative positional relationship in which an element corresponding to a certain input field is present, and estimates that the element present at that relative position is the element corresponding to the input field.

[0089] The output means 246 (output unit) may output the XPath of the input field based on HTML and the estimated element corresponding to the input field. For example, as shown in the upper part of Fig. 15, if the estimation means 243 estimates the predicted label as 5 for the position of the element corresponding to input field 1, the cell corresponds to cell (R2, C1), and cell (R2, C1) contains an element with the characters "Name". In this case, the output means 246 associates the XPath called XPath1 of input field 1 with the "Name" element and outputs them to the storage unit 225.

[0090] 15, when the estimation means 243 estimates that the predicted label for the position of the element corresponding to input field 3 is 6, the cell corresponds to cell (R3, C1), and cell (R3, C1) contains an element with the characters "age." In this case, the output means 246 associates the XPath called XPath3 in input field 3 with the "age" element and outputs them to the storage unit 225.

[0091] According to this, an estimation device can be provided that automatically outputs an element in a desired web page and the XPath of its corresponding input field by arranging the input field and elements of the web page displayed by the web browser in a planar coordinate system, inputting the relative positional relationship between them in the planar coordinate system into a machine learning model trained by the above-mentioned learning method, estimating the element corresponding to the input field based on the relative positional relationship in which the element corresponding to the output input field exists, and outputting the element corresponding to the XPath of the input field.

[0092] It should be noted that the present invention is not limited to the illustrated examples, and can be implemented in configurations that do not deviate from the scope of the claims. That is, although the present invention has been particularly shown and described primarily with reference to specific embodiments, those skilled in the art can make various modifications to the above-described embodiments in terms of the number and other details without departing from the scope of the technical idea and purpose of the present invention. [Explanation of symbols]

[0093] 200: Learning estimation system, 210: Learning device, 231: Arrangement means during learning, 232: Acquisition means during learning, 233: Machine learning model creation means, 234: Screen information input means during learning, 235: Element acquisition means during learning, 220: Estimation device, 241: Arrangement means during estimation, 242: Acquisition means during estimation, 243: Estimation means, 244: Screen information input means during estimation, 245: Element acquisition means during estimation, 246: Output means

Claims

1. A step of arranging one or more input fields in a screen displayed on a display by a computer program and elements corresponding to the one or more input fields in a plane coordinate system; acquiring a relative positional relationship between one of the one or more input fields and an element corresponding to the input field in the plane coordinate system; a step of performing machine learning using training data with the acquired relative positional relationship as a correct label as an input, to create a machine learning model that outputs the relative positional relationship between the one input field and one or more elements in response to the relative positional relationship between the one input field and the elements corresponding to that input field; A computer-implemented learning method comprising:

2. creating a partial plane in the plane coordinate system that includes an element corresponding to the one input field, other input fields, and other elements corresponding to the other input fields, the partial plane being within a predetermined range around the one input field; 2. The learning method according to claim 1, wherein the acquiring step acquires a relative positional relationship between the one input field and an element corresponding to the input field, the relative positional relationship being included in the partial plane.

3. 3. The learning method according to claim 1, wherein the plane coordinate system is a plane divided into grids of a predetermined size.

4. a step of deleting blank rows and blank columns in which no input field or element is included in a row or a column in the plane divided into the grid, 4. The learning method according to claim 3, wherein the acquiring step acquires the relative positional relationship between the one input field and the element corresponding to that input field after deleting blank lines and blank columns.

5. An estimation method that is executed by a computer that references a machine learning model trained by the learning method according to any one of claims 1 to 4, and that estimates correspondences between input fields and elements on a screen displayed by a computer program on a display, A step of arranging one or more input fields and one or more elements in a screen displayed on a display by a computer program in a plane coordinate system; acquiring a relative positional relationship between one of the one or more input fields and the one or more elements in the plane coordinate system; a step of inputting the obtained relative positional relationship into the machine learning model, obtaining an output relative positional relationship in which an element corresponding to the one input field exists, and estimating that the element existing in that relative position is the element corresponding to the one input field; Estimation methods including:

6. creating a partial plane including one or more elements within a predetermined range around the one input field in the plane coordinate system; 6. The estimation method according to claim 5, wherein the obtaining step obtains a relative positional relationship between the one input field and the one or more elements included in the partial plane.

7. 7. The estimation method according to claim 5, wherein the plane coordinate system is a plane divided into grids of a predetermined size.

8. a step of deleting blank rows and blank columns in which no input field or element is included in a row or a column in the plane divided into the grid, 8. The estimation method according to claim 7, wherein the acquiring step acquires the relative positional relationship between the one input field and the one or more elements after deleting blank lines and blank columns.

9. A learning and estimation system including a learning device that learns a correspondence between an input field and an element on a screen displayed by a computer program on a display, and an estimation device that estimates the correspondence, The learning device a learning-time arrangement means for arranging one or more input fields in a screen displayed on a display by a computer program and elements corresponding to the one or more input fields in a plane coordinate system; a learning time acquisition means for acquiring a relative positional relationship between one of the one or more input fields and an element corresponding to the input field in the plane coordinate system; a machine learning model creation means for creating a machine learning model that outputs the relative positional relationship between the one input field and one or more elements in response to the input of the relative positional relationship between the one input field and one or more elements, by performing machine learning using training data that has the acquired relative positional relationship as a correct label as an input; and Equipped with The estimation device includes: an estimation time placement means for placing one or more input fields and one or more elements in a screen displayed on a display by a computer program on a plane coordinate system; an estimation time acquisition means for acquiring a relative positional relationship between one of the one or more input fields and the one or more elements in the plane coordinate system; an estimation means for inputting the acquired relative positional relationship into a trained machine learning model created by the learning device, acquiring the relative positional relationship in which an element corresponding to the one input field exists, and estimating that the element existing in that relative position is the element corresponding to the one input field; A learning estimation system comprising:

10. Computer, a learning means for creating a machine learning model that outputs the relative positional relationship between one input field and the element corresponding to the one input field in response to the relative positional relationship between the one input field and the one or more elements input by arranging one or more input fields and the elements corresponding to the one or more input fields in a plane coordinate system, acquiring the relative positional relationship between one of the one or more input fields and the element corresponding to the input field in the plane coordinate system, and performing machine learning using training data that has the acquired relative positional relationship as a correct answer label as an input; and an estimation means for arranging one or more input fields and one or more elements in a screen displayed by a computer program on a display in a plane coordinate system, obtaining a relative positional relationship between one of the one or more input fields and the one or more elements in the plane coordinate system, inputting the obtained relative positional relationship into the machine learning model that has been trained and created by the learning means, obtaining a relative positional relationship in which an element corresponding to the one input field exists that is output, and estimating that the element existing in that relative position is the element corresponding to the one input field; A program to function as a

Images