Information processing method and information processing apparatus

Abstract

This invention provides an information processing method that recognizes objects contained in a GUI screen and selectively displays the recognized objects. [Solution] The method includes: a screen acquisition step of acquiring a GUI screen 200 at predetermined intervals; a difference detection step of detecting changes in the GUI screen 200; a recognition step of executing recognition processing for graphic objects and character objects included in the GUI screen 200 if the change is above a predetermined threshold; an information acquisition step of acquiring coordinate information of a first frame surrounding each graphic object, character information of each character object, and coordinate information of a second frame surrounding each character object obtained by executing the recognition processing; and a display processing step of displaying the first frame when a cursor operated by the user enters the area of ​​any of the first frames on the GUI screen 200, and displaying the character information of the character objects in the second frame when the cursor enters the area of ​​any of the second frames.

Description

Technical Field

[0001] The present invention relates to an information processing method and an information processing apparatus, and particularly to an information processing method and an information processing apparatus for an object included in a GUI (Graphical User Interface) screen.

Background Art

[0002] For example, Patent Document 1 discloses that in a selection unit, any one of a plurality of recognition artificial intelligences is selected, and in a recognition unit, the selected recognition artificial intelligence is used to recognize characters included in image data, so that high-precision character recognition is performed by an artificial intelligence suitable for recognition.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the technology disclosed in Patent Document 1, although conversion from image data such as handwritten paper or facsimile to text data (character data) can be performed, recognition of objects such as graphic icons and buttons included in the GUI screen is not mentioned.

[0005] Also, for example, in the operation verification of a target program having a GUI screen, when selecting and operating the target program, it is necessary to select and operate objects such as graphic icons and buttons for operating the target program within the GUI screen. Selection of an object within the GUI screen requires inputting the coordinates of the object within the GUI screen to select the object, which is troublesome.

[0006] Therefore, the inventors of the present invention conceived of a technology to recognize graphic objects such as graphic icons included in a GUI screen and characters (character objects) included in a GUI screen. They then discovered that by simply pointing at objects such as graphic objects and character objects with a cursor and clicking, it is possible to specify an object and input the coordinates of the specified object.

[0007] In one aspect, this disclosure aims to provide an information processing method, etc., that can recognize objects included in a GUI screen and selectively display the recognized objects. [Means for solving the problem]

[0008] (1) The information processing method is characterized by including: a screen acquisition step of acquiring a GUI screen at predetermined intervals; a difference detection step of comparing a first GUI screen with a second GUI screen obtained at a predetermined time interval from the time the first GUI screen was acquired, and detecting a change in the GUI screen; a recognition step of executing recognition processing for one or more graphic objects and one or more character objects included in the second GUI screen if the change is greater than or equal to a predetermined threshold, and not executing recognition processing for graphic objects and character objects included in the second GUI screen if the change is less than a predetermined threshold; an information acquisition step of acquiring coordinate information of a first frame surrounding each of the one or more graphic objects in the second GUI screen obtained by executing the recognition processing, character information of each of the one or more character objects, and coordinate information of a second frame surrounding each of the one or more character objects; and a display processing step of displaying the first frame when a cursor operated by a user enters the area of ​​the first frame of any of the one or more graphic objects in the GUI screen, and displaying the character information of the character object in the second frame when the cursor enters the area of ​​the second frame of any of the one or more character objects. • "First GUI screen, second GUI screen": Of two GUI screens acquired at predetermined intervals, the one acquired earlier in time is referred to as the first GUI screen, and the one acquired later in time is referred to as the second GUI screen.

[0009] (2) In such an information processing method, it is preferable that in the information acquisition step, coordinate information of a third frame surrounding a graphic object and a character object attached to the graphic object is further acquired, and in the display processing step, if the cursor enters the area of ​​the third frame and the area of ​​the first frame and the first frame and the third frame overlap, the first frame is displayed preferentially, and if the cursor enters the area of ​​the third frame and the area of ​​the second frame and the second frame and the third frame overlap, the second frame is displayed preferentially.

[0010] (3) In such an information processing method, each of the one or more graphic objects may be a graphic icon, and each of the one or more character objects may be an image representing the character assigned to the graphic icon.

[0011] (4) The information processing method further includes a notification step of notifying the user terminal of coordinate information of a first frame surrounding each of the one or more graphic objects, character information of each of the one or more character objects, and coordinate information of a second frame surrounding each of the one or more character objects, wherein the screen acquisition step, the difference detection step, and the notification step are performed by the server, the recognition step is performed using the server's AI, and the information acquisition step and the display processing step are performed by the user terminal.

[0012] (5) In addition, in such an information processing method, the GUI screen is a screen created by the server using an emulator to verify the operation of the programs associated with each of the one or more graphic objects and the one or more character objects, and may be displayed on the user terminal.

[0013] (6) Preferably, the screen acquisition step, the difference detection step, the information acquisition step, and the display processing step are performed by a user terminal, and the recognition step is performed using the AI ​​of the user terminal.

[0014] (7) In such information processing methods, the user operation is the user's mouse operation or touch operation.

[0015] (8) In addition, in such information processing methods, the user operation may be an operation performed by the user's gaze using the gaze detection function of the user terminal.

[0016] (9) Another embodiment of the information processing device is characterized by comprising: a screen acquisition unit that acquires a GUI screen at predetermined intervals; a difference detection unit that compares a first GUI screen with a second GUI screen obtained at a predetermined time interval from the time the first GUI screen was acquired, and detects changes in the GUI screen; a recognition unit that, if the change is greater than or equal to a predetermined threshold, performs recognition processing for one or more graphic objects and one or more character objects included in the second GUI screen, and does not perform recognition processing for graphic objects and character objects included in the second GUI screen if the change is less than a predetermined threshold; an information acquisition unit that acquires coordinate information of a first frame surrounding each of the one or more graphic objects in the second GUI screen obtained by performing the recognition processing, character information of each of the one or more character objects, and coordinate information of a second frame surrounding each of the one or more character objects; and a display processing unit that displays the first frame when a cursor operated by a user enters the area of ​​the first frame of any of the one or more graphic objects in the GUI screen, and displays the character information of the character object in the second frame when the cursor enters the area of ​​the second frame of any of the one or more character objects. [Effects of the Invention]

[0017] From one perspective, it is possible to recognize objects contained in the GUI screen and selectively display the recognized objects.

Brief Description of Drawings

[0018] [Figure 1] It is a schematic diagram showing an overview of the information processing system of the first embodiment. [Figure 2] It is a functional block diagram showing the information processing apparatus and the user terminal of the first embodiment. [Figure 3] It is a schematic diagram showing an example of a GUI screen. [Figure 4] It is a schematic diagram showing an example of recognition processing for a GUI screen. [Figure 5] It is a schematic diagram showing an example of the data structure of a recognition processing result database. [Figure 6] It is a schematic diagram showing an example of a frame display including a graphic icon and characters. [Figure 7] It is a schematic diagram showing an example of a frame display including a graphic icon. [Figure 8] It is a schematic diagram showing an example of the hardware configuration of an information processing apparatus. [Figure 9] It is a schematic diagram showing an example of the hardware configuration of a user terminal. [Figure 10] It is a schematic flow showing an example of the operations of the information processing apparatus and the user terminal of the first embodiment. [Figure 11] It is a functional block diagram showing the information processing apparatus of the second embodiment. [Figure 12] It is a schematic flow showing an example of the operation of the information processing apparatus of the second embodiment.

Modes for Carrying Out the Invention

[0019] Hereinafter, a program and verification device according to one aspect of the present invention will be specifically described with reference to the drawings. Note that the embodiments described below are all specific examples of this disclosure. The numerical values, shapes, components, and their positions shown in the following embodiments are examples only and are not intended to limit this disclosure. Furthermore, among the components in the following embodiments, those not described in the independent claim representing the highest-level concept will be described as optional components. Also, in all embodiments, the contents of each can be combined.

[0020] Furthermore, the following drawings are schematic, and some components may be omitted for illustrative purposes. Also, common parts in one or more embodiments may be denoted by the same reference numeral, and their descriptions may be omitted.

[0021] [1. Overview] <First Embodiment> (Introduction) Automated testing tools are commonly used to verify the operation of programs that display multiple screens in a transitional manner. These automated testing tools execute the program being verified, automatically perform the operations to transition between the screens displayed by the program, and generate a screen transition diagram showing the relationships between each screen. The screen transition diagram is generated for the verifier to confirm whether the screens transition according to the specifications.

[0022] The information processing device 1 according to this embodiment can recognize objects included in the GUI screen and selectively display the recognized objects. Furthermore, the information processing device 1 according to this embodiment can recognize objects even if they move within the GUI screen, and by selectively displaying a frame surrounding the object, it is possible to intuitively understand whether an object has been recognized. In addition, the user can specify an object within the GUI screen by simply selecting an object included in the GUI screen with the cursor and clicking, input the coordinate information of the specified object, and run the target program, thus enabling efficient operation verification.

[0023] (Information Processing System) Figure 1 is a schematic diagram showing the outline of the information processing system of the first embodiment.

[0024] First, an overview of the information processing system of the first embodiment will be explained using Figure 1. The information processing system shown in Figure 1 assists users in verifying the operation of a target program using a user terminal 2 and a GUI screen provided by emulator 4.

[0025] (Information Processing Device 1) The information processing device 1, for example, captures the GUI screen provided by the emulator 4 at predetermined intervals, and uses the image recognition AI 51 and character recognition AI 52 to recognize objects such as graphic icons and characters that have changed to some extent within the GUI screen, and obtains the coordinate information of the objects. The information processing device 1 notifies the user terminal 2 of the obtained coordinate information. When the user moves the cursor around an object included in the GUI screen provided by the emulator 4, the user terminal 2 displays a frame around the object included in the GUI screen, indicating that the object has been recognized. When the frame is displayed, the user can click to specify the object and input the coordinate information of the specified object, and then run the target program.

[0026] (User) A user is someone who is using user terminal 2 and is the person to whom information processing device 1 provides support.

[0027] (User terminal 2) User terminal 2 is the terminal that the user is using.

[0028] (Communication Network 3) The information processing device 1, user terminal 2, emulator 4, image recognition AI 51, and character recognition AI 52 are connected to communicate via the Internet or an intranet (communication network 3). Communication network 3 is, for example, the Internet. Communication network 3 may also be an intranet, a wired or wireless communication network. Wireless communication may be, for example, Bluetooth®, Wi-Fi®, LPWA (Low Power Wide Area), etc. As LPWA, LTE® or a frequency band that does not require a license may be used.

[0029] (Emulator 4) An emulator is software that can mimic the behavior of a device, software, or system and be used as a substitute. Emulator 4 mimics the behavior of the target program. Emulator 4 is used to verify the operation of the target program.

[0030] (AI51 for image recognition) The AI51 for image recognition is trained using training data and can be used to recognize geometric objects such as geometric icons.

[0031] (AI52 for character recognition) The AI ​​52 for character recognition is trained using training data and can be used to recognize character objects such as images that represent characters. The AI ​​51 for image recognition may recognize character objects using OCR (Optical Character Reader) recognition.

[0032] [2. Details of each configuration] Figure 2 is a functional block diagram showing the information processing device and user terminal of the first embodiment. Figure 3 is a schematic diagram showing an example of a GUI screen. Figure 4 is a schematic diagram showing an example of recognition processing for the GUI screen. Note that in Figure 4, the reference numerals used in Figure 3 are omitted, but the same reference numerals are used for elements similar to those in Figure 3. Figure 5 is a schematic diagram showing an example of the data structure of the recognition processing result database. Figure 6 is a schematic diagram showing an example of a frame display including graphic icons and text. Figure 7 is a schematic diagram showing an example of a frame display including graphic icons. Note that the same reference numerals are used for elements similar to those in Figure 3, and detailed explanations are omitted.

[0033] (Information Processing System) The information processing system of the first embodiment comprises an information processing device 1 and a plurality of user terminals 2 that are communicated with each other. The information processing device 1 and the plurality of user terminals 2 are communicated with each other via a communication network 3.

[0034] (Information Processing Device 1) The information processing device 1 is, for example, a server such as a personal computer. In this embodiment, as shown in Figure 2, the information processing device 1 mainly comprises a screen acquisition unit 11, a storage unit 12, a difference detection unit 13, a determination unit 14, a recognition unit 15, an information acquisition unit 16, and a notification unit 17. The information processing device 1 may further include an image recognition AI 51a and a character recognition AI 52b, and may utilize these, or it may utilize an image recognition AI 51 and a character recognition AI 52 provided outside the information processing device 1. Note that the information processing device 1 does not necessarily have a storage unit 12. The storage unit 12 can be provided outside the information processing device 1 and should be accessible by the information processing device 1.

[0035] (Screen acquisition unit 11) The screen acquisition unit 11 acquires the GUI screen at predetermined intervals. In this embodiment, the screen acquisition unit 11 acquires the GUI screen 200, for example, as shown in Figure 5, at predetermined intervals.

[0036] (GUI screen 200) GUI screen 200 is a screen (image) created by emulator 4 to verify the operation of programs associated with one or more graphic objects and one or more text objects. GUI screen 200 is also displayed on the display of user terminal 2. Each of the one or more graphic objects is, for example, a graphic icon, and each of the one or more text objects is an image representing the text attached to the graphic icon. Note that GUI screen 200 is not limited to those created by emulator 4; it may also be a GUI screen displayed by user terminal 2.

[0037] The GUI screen 200 includes, for example, a graphic icon 201a, a text icon 202a representing the text attached to the graphic icon 201a, a graphic icon 201b, and a text icon 202b representing the text attached to the graphic icon 201b, as shown in Figure 3. The graphic icon 201a is an icon composed of a graphic image representing, for example, a house. The text icon 202a is an icon composed of a text image representing Home. Similarly, the graphic icon 201b is an icon composed of a graphic image representing, for example, a circular arrow indicating back. The text icon 202b is an icon composed of a text image representing Back.

[0038] Furthermore, the GUI screen 200 includes, for example, a graphic icon 201c, a text icon 202c indicating the text attached to the graphic icon 201c, and a dotted line frame 203c indicating the area where the graphic icon 201c and the text icon 202c are combined, as shown in Figure 3. The graphic icon 201c is, for example, an icon composed of a graphic image combining two curved arrows. The text icon 202c is an icon composed of a text image indicating "Update".

[0039] Furthermore, the GUI screen 200 includes, for example, a graphic icon 201d, a text icon 202d indicating the text attached to the graphic icon 201d, and a dotted line frame 203d indicating the area where the graphic icon 201d and the text icon 202d are combined, as shown in Figure 3. The graphic icon 201d is an icon composed of a graphic image representing, for example, a WiFi spot. The text icon 202d is an icon composed of a text image indicating a WiFi spot.

[0040] Furthermore, the same applies to other graphic icons, text icons, and dotted line frames in GUI screen 200, so their explanations will be omitted.

[0041] (Storage unit 12) The storage unit 12 is composed of an HDD (Hard Disk Drive), an SSD (Solid State Drive), or memory, etc. In this embodiment, for example, the storage unit 12 stores a predetermined threshold value 1714 that is set in advance by the user or the like and used in the difference detection unit 13 described later. The storage unit 12 also stores the recognition processing results of the recognition unit 15 described later. The storage unit 12 also stores GUI screens 200 etc. that are acquired by the screen acquisition unit 11.

[0042] (Difference detection unit 13) The difference detection unit 13 detects changes in the GUI screen by comparing the first GUI screen with the second GUI screen acquired a predetermined time after the time the first GUI screen was acquired. In this embodiment, the difference detection unit 13 detects changes in the GUI screen 200 acquired by the screen acquisition unit 11 at predetermined time intervals. Of the two GUI screens 200 acquired at predetermined time intervals, the one acquired earlier in time is referred to as the first GUI screen, and the one acquired later in time is referred to as the second GUI screen. In other words, the difference detection unit 13 detects changes in the GUI screen 200 by comparing the GUI screen 200 acquired by the screen acquisition unit 11 (the second GUI screen) with the GUI screen 200 acquired by the screen acquisition unit 11 a predetermined time earlier (the first GUI screen). Changes in the GUI screen 200 include, for example, changes in the representation of objects such as graphic icons and text icons included in the GUI screen 200, changes in the position and content of objects, etc.

[0043] (Judgment section 14) The determination unit 14 determines whether the change in the GUI screen detected by the difference detection unit 13 is above a predetermined threshold. The predetermined threshold should be set to be greater than the value indicating a change in the representation of objects such as graphic icons and text icons, such as color and brightness, and less than or equal to the value indicating a change in the position or content of an object.

[0044] (Recognition part 15) The recognition unit 15 performs recognition processing for one or more graphic objects and one or more character objects included in the second GUI screen if the change in the GUI screen is greater than or equal to a predetermined threshold. The recognition unit 15 does not perform recognition processing for graphic objects and character objects included in the second GUI screen if the change in the GUI screen is less than the predetermined threshold.

[0045] In this embodiment, the recognition unit 15 performs recognition processing of objects included in the GUI screen 200 acquired by the screen acquisition unit 11, according to the determination result of the determination unit 14, using the image recognition AI 51a and the character recognition AI 52b. For example, if the change in the GUI screen 200 is greater than or equal to a predetermined threshold, the recognition unit 15 performs recognition processing of graphic icons 201a, 201b, 201c, and 201d included in the GUI screen 200 shown in Figure 3, using the image recognition AI 51a. Also, for example, if the change in the GUI screen 200 is greater than or equal to a predetermined threshold, the recognition unit 15 performs recognition processing of character icons 202a, 202b, 202c, and 202d included in the GUI screen 200 shown in Figure 3, using the character recognition AI 52b. The recognition unit 15 does not perform recognition processing if the change in the GUI screen 200 is less than a predetermined threshold.

[0046] (AI51a for image recognition, AI52b for character recognition) Image recognition AI51a may be a copy of all or part of image recognition AI51 and can be used to recognize graphic objects such as graphic icons. Character recognition AI52b may be a copy of all or part of character recognition AI52 and can be used to recognize character objects such as images that represent characters.

[0047] The image recognition AI 51a performs a full search of the GUI screen 200 shown in Figure 3 and recognizes (image recognizes) graphic icons 201a, 201b, 201c, and 201d contained within the GUI screen 200. Furthermore, by performing the recognition process (image recognition), the image recognition AI 51a adds frames 251a, 251b, 251c, and 251d surrounding the graphic icons 201a, 201b, 201c, and 201d, as shown in Figure 4. Similarly, the character recognition AI 52b performs a full search of the GUI screen 200 shown in Figure 3 and recognizes (character recognizes) character icons 202a, 202b, 202c, and 202d contained within the GUI screen 200, acquiring their character information. Furthermore, the AI52b for character recognition performs recognition processing (character recognition) and, for example, as shown in Figure 4, adds frames 252a, 252b, 252c, and 252d surrounding the character icons 202a, 202b, 202c, and 202d.

[0048] Furthermore, as shown in Figure 4, the AI ​​51a for image recognition has frames 253a surrounding the graphic icon 201a and the text icon 202a, and frames 253b surrounding the graphic icon 201b and the text icon 202b. Also, as shown in Figure 4, the AI ​​51a for image recognition has frames 253c surrounding the graphic icon 201c and the text icon 202c, and frames 253d surrounding the graphic icon 201d and the text icon 202d.

[0049] (Information acquisition unit 16) The information acquisition unit 16 acquires the coordinate information of the first frame surrounding each of the one or more graphic objects on the second GUI screen obtained by the recognition unit 15 through the recognition process. The information acquisition unit 16 also acquires the character information (text information) of each of the one or more character objects obtained by the recognition unit 15 through the recognition process, and the coordinate information of the second frame surrounding each of the one or more character objects on the second GUI screen. The information acquisition unit 16 also acquires the coordinate information of the third frame surrounding the graphic objects and the character objects attached to the graphic objects on the second GUI screen obtained by the recognition unit 15 through the recognition process.

[0050] In this embodiment, the information acquisition unit 16 acquires coordinate information for frames 251a, 251b, 251c, and 251d, for example, as shown in Figure 4. The information acquisition unit 16 acquires character information for character icons 202a, 202b, 202c, and 202d included in the GUI screen 200 recognized by the recognition unit 15, and coordinate information for frames 252a, 252b, 252c, and 252d, as shown in Figure 4. The information acquisition unit 16 also acquires coordinate information for frames 253a, 253b, 253c, and 253d, for example, as shown in Figure 4, included in the GUI screen 200 recognized by the recognition unit 15. The information acquisition unit 16 may also acquire the above coordinate information and character information using the image recognition AI 51a and the character recognition AI 52b.

[0051] Furthermore, the recognition unit 15 stores the position information and character information of objects including such graphic icons and character icons in the recognition result DB 1715, for example, shown in Figure 5. The recognition result DB 1715 shown in Figure 5 includes GUI screen information 1715a, object ID 1715b, position information 1515c, character 1515d, and character information 1515e.

[0052] GUI screen information 1715a is information for identifying the GUI screen 200. GUI screen information 1715a is indicated, for example, by the time t101 when the screen acquisition unit 11 acquired the GUI screen 200. Note that GUI screen information 1715a may also be a number indicating the order in which the screen acquisition unit 11 acquired the GUI screen 200.

[0053] Object ID 1715b is an identification number used to identify recognized objects such as graphic icons and text icons. For example, the object IDs included in GUI screen 200 are shown as o101, o102, o103, o104, ... in the order they were recognized. In the example shown in Figure 5, the object ID indicated by o101 is graphic icon 201a shown in Figure 3, etc., and the object ID indicated by o102 is text icon 202a shown in Figure 3, etc. Also, the object ID indicated by o103 is graphic icon 201b shown in Figure 3, etc., and the object ID indicated by o104 is text icon 202b shown in Figure 3, etc.

[0054] Location information 1715c represents the coordinates of an object on the GUI screen. In the example shown in Figure 5, location information 1715c for the object ID indicated by o101 corresponds to the coordinates (x1, y1) to (x2, y2) of the area of ​​frame 251a shown in Figure 4. Location information 1715c for the object ID indicated by o102 corresponds to the coordinates (x3, y3) to (x4, y4) of the area of ​​frame 252a shown in Figure 4. Location information 1715c for the object ID indicated by o103 corresponds to the coordinates (x5, y5) to (x6, y6) of the area of ​​frame 253a shown in Figure 4. Also, location information 1715c for the object ID indicated by o104 corresponds to the coordinates (x7, y7) to (x8, y8) of the area of ​​frame 251b shown in Figure 4. The location information 1715c of the object ID indicated by o105 corresponds to the coordinates (x9, y9) to (x10, y10) of the area of ​​frame 252b shown in Figure 4.

[0055] The character 1515d indicates whether or not the object identified by the object ID contains a character. In the example shown in Figure 5, the object identified by the object ID o102 contains a character and is represented as "〇".

[0056] Character information 1515e indicates the character (text) information represented by the character icon when the object identified by the object ID contains characters (i.e., a character icon). In the example shown in Figure 5, the object identified by the object ID o102 contains characters (character icon 202a), and it is indicated that the character is Home.

[0057] (Notification section 17) The notification unit 17 notifies the user terminal 2 of the coordinate information of the first frame surrounding each of the one or more graphic objects, the character information (text information) of each of the one or more character objects, and the coordinate information of the second frame surrounding each of the one or more character objects.

[0058] In this embodiment, the notification unit 17 notifies the user terminal 2 of the coordinate information and character information 17a acquired by the information acquisition unit 16.

[0059] (User terminal 2) User terminal 2 is a device capable of communicating with information processing device 1, and is, for example, a personal computer (PC). Alternatively, a tablet PC, smartphone, or the like may be used as user terminal 2.

[0060] In this embodiment, the user terminal 2, as shown in Figure 2, includes an operation unit 21, a terminal acquisition unit 22, a display processing unit 23, an execution unit 24, and a gaze detection unit 25. Note that the information processing device 1 does not necessarily need to include the gaze detection unit 25.

[0061] (Terminal acquisition unit 22) The terminal acquisition unit 22 acquires information notified by the information processing device 1. In this embodiment, the terminal acquisition unit 22 acquires coordinate information and character information 17a.

[0062] (Operation unit 21) The operation unit 21 moves the cursor on the GUI screen displayed on a display terminal such as a display, according to user operation. User operation is not limited to, but is performed by the user using the mouse or touch on the user terminal 2. User operation may also be performed by the user's gaze using the gaze detection function of the user terminal 2.

[0063] (Display processing unit 23) The display processing unit 23 displays information acquired by the user terminal 2 on the display terminal. The display terminal can be, for example, a device that displays images, such as a liquid crystal display (LCD), a plasma display panel (PDP), or an organic electroluminescent (EL) display. Preferably, the display terminal has a touch panel function.

[0064] The display processing unit 23 displays the first frame of one or more graphic objects on the GUI screen when the cursor, operated by the user, enters the area of ​​the first frame of one or more graphic objects. The display processing unit 23 also displays the character information (text information) of the character object in the second frame when the cursor, operated by the user, enters the area of ​​the second frame of one or more character objects.

[0065] Furthermore, if the frames to be displayed overlap, the display processing unit 23 may prioritize displaying the frame with the smaller area, or if the frames to be displayed include the second frame (text frame), it may prioritize displaying the second frame. For example, if the cursor operated by the user enters the area of ​​the third frame and the area of ​​the first frame, and the first and third frames overlap, the display processing unit 23 may prioritize displaying the first frame. Similarly, if the cursor operated by the user enters the area of ​​the third frame and the area of ​​the second frame, and the second and third frames overlap, the display processing unit 23 may prioritize displaying the second frame. Moreover, when the display processing unit 23 displays the second frame, it may also display the character information (text information) of the character object in the second frame.

[0066] In this embodiment, the display processing unit 23 displays the GUI screen 200 created by the emulator 4 on the display terminal. The display processing unit 23 also displays information on the display terminal that corresponds to user operations, based on the coordinate information and character information 17a acquired by the user terminal 2.

[0067] For example, as shown in Figure 6, when the cursor 221a, operated by the user, enters the area of ​​the dotted frame 203c on the GUI screen 200, the display processing unit 23 displays the frame 253c on the GUI screen 200. Also, for example, as shown in Figure 7, when the cursor 221b, operated by the user, enters the area surrounding the graphic icon 201c on the GUI screen 200 (the area of ​​frame 251c), the display processing unit 23 displays the frame 251c on the GUI screen 200.

[0068] (Execution Unit 24) The execution unit 24 can execute the operation of programs associated with one or more graphic objects and one or more text objects included in the GUI screen. The execution unit 24 can execute the program of a graphic object or text object specified by the user, or execute the associated programs in the order of graphic objects and text objects specified by the user. In addition, if an object included in the GUI screen is selected and clicked by the user, the execution unit 24 may specify the object in the GUI screen and input the coordinate information of the specified object to the program being executed.

[0069] (Eye-gaze detection unit 25) The gaze detection unit 25 detects the user's gaze using the gaze detection function of the user terminal 2. For example, the gaze detection unit 25 detects the user's gaze from video footage acquired by a camera installed in or connected to the user terminal 2. The gaze detection function includes eye line detection and blink detection, and is a function that enables operation by eye line (gaze) and blink.

[0070] [3. Hardware Configuration] Figure 8 is a schematic diagram showing an example of the hardware configuration of the information processing device. Figure 9 is a schematic diagram showing an example of the hardware configuration of the user terminal. The hardware configurations of the information processing device 1 and the user terminal 2 will be described below using Figures 8 and 9.

[0071] (Hardware configuration of information processing device 1) The hardware configuration of the information processing device 1 is described below. As shown in Figure 8, the information processing device 1 of this embodiment uses, for example, a computer. The information processing device 1 is equipped with a CPU 170. The CPU 170 is connected to a memory 171, a connection port 173 for connecting / reading storage devices 172, etc., and a communication circuit 174 for communicating with the outside via a network, all via a bus line 175. The memory 171 stores a program 1713 for processing the information processing device 1. It may also store a browser program 1712 and an OS 1711 (operating system). Furthermore, the memory 171 stores a predetermined threshold 1714 and a recognition processing result database (hereinafter referred to as the recognition processing result DB) 1715. The memory 171 may be a storage unit 12, or it may be a different memory unit from the storage unit 12. If the memory 171 and the storage unit 12 are different, the predetermined threshold 1714 and the recognition processing result database DB 1317 are stored in the storage unit 12.

[0072] (Hardware configuration of user terminal 2) The hardware configuration of user terminal 2 is almost the same as that of the information processing device 1 described above, so the same parts are denoted by the same reference numerals, and their explanation is omitted. In this embodiment, user terminal 2 uses, for example, a personal computer.

[0073] The computer's memory 171 stores a program 1716 for processing user terminal 2. It may also store a browser program 1712 and an OS 1711 (operating system).

[0074] In this embodiment, program 1713 and program 1316 work together by utilizing the functions of OS 1711 and browser program 1712, respectively. Alternatively, program 1713 and program 1716 may operate independently without utilizing browser program 1712 and OS 1711, respectively.

[0075] In the hardware configuration of the information processing device 1 and user terminal 2 described above, the functions shown in Figure 2 are implemented, for example, using the CPU 170 and programs 1713 and 1716. However, some or all of these functions may be sequence-controlled using logic circuits such as a microcontroller or a PLC (programmable logic controller).

[0076] (Recognition processing result DB1317) The recognition processing result DB1317 stores the recognition processing results, including the position information and character information of objects included in the GUI screen that have been recognized using AI. As mentioned above, the recognition processing results include, but are not limited to, the position information of each graphic object, the position information of each character object, and the character information of each character object recognized by AI when the changes in the GUI screen exceed a predetermined threshold. The recognition processing results only need to be information used to select and control the display of frames for graphic icons, etc., on the GUI screen displayed on the user terminal 2.

[0077] [4. Operation of the information processing device and user terminal of the first embodiment] (Operation of Information Processing Device 1 and User Terminal 2) Next, the operation of the information processing device and user terminal of the first embodiment will be explained using Figure 10. Figure 10 is a schematic flow showing an example of the operation of the information processing device 1 and user terminal 2 of the first embodiment. Figure 10 shows a flowchart illustrating one embodiment of the processing of program 1713 used in the information processing device 1 and program 1716 used in the user terminal 2.

[0078] (S11, S12) When a predetermined time has elapsed (Yes in S11), the CPU 170 of the information processing device 1 acquires the GUI screen. In this embodiment, the information processing device 1 acquires the GUI screen 200, for example, shown in Figure 3, which is displayed on the user terminal 2 and created by the emulator 4, at predetermined intervals. If the predetermined time has not elapsed (No in S11), the CPU 170 of the information processing device 1 returns to S11.

[0079] (S13) The CPU 170 of the information processing device 1 detects a change in the GUI screen. In this embodiment, the information processing device 1 compares the GUI screen 200 (second GUI screen) acquired in S12 with the GUI screen 200 (first GUI screen) acquired a predetermined time earlier to detect a change in the GUI screen 200.

[0080] (S14, S15) If the change in the GUI screen is greater than or equal to a predetermined threshold (Yes in S14), the CPU 170 of the information processing device 1 performs recognition processing for one or more graphic objects and one or more character objects included in the GUI screen. In this embodiment, if the change between the GUI screen 200 (second GUI screen) acquired in S12 and the GUI screen 200 (first GUI screen) acquired a predetermined time earlier is greater than or equal to a predetermined threshold, the information processing device 1 performs recognition processing for graphic objects and character objects included in the GUI screen 200 (second GUI screen). If the change in the GUI screen is less than a predetermined threshold (No in S14), the CPU 170 of the information processing device 1 returns to S11.

[0081] (S16) The CPU 170 of the information processing device 1 acquires coordinate information of the first frame surrounding each of the one or more graphic objects, the character information (text information) of each of the one or more character objects, and the coordinate information of the second frame surrounding each of the one or more character objects. In this embodiment, as shown in Figures 4 and 5, for example, the information processing device 1 acquires coordinate information of the frame surrounding each of the one or more graphic objects included in the GUI screen 200 (second GUI screen), the character information of each of the one or more character objects, the coordinate information of the frame surrounding each of the one or more character objects, and the coordinate information of the frame surrounding the graphic icons and character icons.

[0082] The CPU 170 of the information processing device 1 notifies the user terminal 2 of the coordinate information and character information 17a acquired in S16.

[0083] (S17) When the user's cursor enters the area of ​​the first frame of one or more graphic objects on the GUI screen, the CPU 170 of user terminal 2 displays the first frame. Also, when the user's cursor enters the area of ​​the second frame of one or more character objects, the CPU 170 of user terminal 2 displays the character information of the character object in the second frame.

[0084] In this embodiment, as shown in Figure 6, for example, when the cursor 221a operated by the user enters the area of ​​the dotted line frame 203c on the GUI screen 200, the user terminal 2 displays frame 253c on the GUI screen 200. Also, as shown in Figure 7, for example, when the cursor 221b operated by the user enters the area surrounding the graphic icon 201c on the GUI screen 200 (the area of ​​frame 251c), the user terminal 2 displays frame 251c on the GUI screen 200.

[0085] <Second Embodiment> (Configuration of Information Processing Device 1A) In the first embodiment, the information processing device 1 and the user terminal 2 were configured as separate components, but this is not limited to that configuration. An information processing device 1A may be configured in which the user terminal 2 and the information processing device 1 are integrated. In this case, the information processing device 1A is, for example, a personal computer, and the user performs operations (user operations) on a GUI screen 200 displayed on a display device connected to the information processing device 1A.

[0086] Figure 11 is a functional block diagram showing an information processing device of the second embodiment. The same reference numerals are used for elements similar to those in Figure 2, etc., and detailed explanations are omitted. In this embodiment, as shown in Figure 11, the information processing device 1A includes a screen acquisition unit 11, a storage unit 12, a difference detection unit 13, a determination unit 14, an information acquisition unit 16, an operation unit 21, a display processing unit 23, an execution unit 24, and a gaze detection unit 25. Note that the information processing device 1A does not necessarily have to include the storage unit 12 and the gaze detection unit 25.

[0087] (Operation of Information Processing Device 1A) Next, the operation of the information processing device 1A will be explained using Figure 12. Figure 12 is a schematic flow showing an example of the operation of the information processing device of the second embodiment. Figure 12 shows a flowchart illustrating one embodiment of the processing of program 1713 used in the information processing device 1A.

[0088] (S21, S22) When a predetermined time has elapsed (Yes in S21), the CPU 170 of the information processing device 1A retrieves the GUI screen. If the predetermined time has not elapsed (No in S21), the CPU 170 of the information processing device 1A returns to S21.

[0089] (S23) The CPU 170 of the information processing device 1A detects changes in the GUI screen.

[0090] (S24, S25) If the change in the GUI screen is greater than or equal to a predetermined threshold (Yes in S24), the CPU 170 of the information processing device 1A performs recognition processing for one or more graphic objects and one or more character objects included in the GUI screen. If the change in the GUI screen is less than a predetermined threshold (No in S24), the CPU 170 of the information processing device 1A returns to S21.

[0091] (S26) The CPU 170 of the information processing device 1A acquires coordinate information of the first frame surrounding each of the one or more graphic objects, character information (text information) of each of the one or more character objects, and coordinate information of the second frame surrounding each of the one or more character objects.

[0092] (S27) The CPU 170 of the information processing device 1A displays the first frame when the cursor, operated by the user, enters the area of ​​the first frame of one or more graphic objects on the GUI screen. The CPU 170 of the user terminal 2 also displays the character information of the character object in the second frame when the cursor, operated by the user, enters the area of ​​the second frame of one or more character objects.

[0093] [5. Others] (1) The embodiments described above can be used in combination as appropriate. (2) Some of the functions of the information processing device 1 may be executed on an external server. (3) In the embodiments described above, the recognition of geometric objects was performed by image recognition using AI, but the recognition of geometric objects may also be performed by object recognition using AI. (4) In the embodiments described above, display processing control was performed using a cursor, but this is not limited to this. Display processing control may also be performed using a pointer, or any other method that has the same functionality as a cursor on the GUI screen is acceptable.

[0094] (5) In the embodiments described above, a frame is displayed when the cursor enters the area surrounding a graphic object, a frame surrounding a text object, or a frame surrounding both a graphic object and a text object. However, the invention is not limited to this. A frame may also be displayed when the cursor approaches a graphic object or a text object. Furthermore, the invention is not limited to such frame displays; any kind of highlighting of a graphic object or a text object is acceptable. For example, the text information of a text object may be enlarged.

[0095] (6) In the embodiments described above, the present invention has been explained as being usable for specifying icons when verifying the operation of a target program, but it is not limited to this. For example, by combining it with an eye-tracking function, it can also be used for support displays when operating the screen touchlessly. Since the position of the gaze can be displayed as a cursor (pointer), the position of the cursor becomes clearer, and selection display control becomes more convenient.

[0096] (7) The information processing devices 1 and 1A are, for example, personal computers and are used for specifying icons when verifying the operation of the target program, but are not limited to this. The information processing devices 1 and 1A may be provided in a device connected to a drive recorder. In this case, instead of the GUI screen 200, images (video) taken by the drive recorder at predetermined intervals can be acquired, and road sign images, billboard images, etc. included in the acquired images can be recognized as text objects. This makes it possible to tag the images in the drive recorder.

[0097] (8) The information processing devices 1 and 1A may also be installed in a device connected to a camera that photographs small quantities of various products displayed in a convenience store or the like. In this case, instead of the GUI screen 200, images can be acquired at predetermined intervals, and product images included in the acquired images can be recognized as graphic objects or text objects. This makes it possible to track the movement of products as well as tag product images in the images.

[0098] [6. Summary] (1) The information processing method is characterized by including a screen acquisition step of acquiring a GUI screen 200 at predetermined time intervals; a difference detection step of comparing a first GUI screen with a second GUI screen acquired at a predetermined time interval from the time the first GUI screen was acquired to detect changes in the GUI screen 200; a recognition step of executing recognition processing for one or more graphic objects and one or more character objects included in the second GUI screen if the change is greater than or equal to a predetermined threshold, and not executing recognition processing for graphic objects and character objects included in the second GUI screen if the change is less than a predetermined threshold; an information acquisition step of acquiring coordinate information of the first frame surrounding each of the one or more graphic objects in the second GUI screen obtained by executing the recognition processing, character information of each of the one or more character objects, and coordinate information of the second frame surrounding each of the one or more character objects; and a display processing step of displaying the first frame when a cursor operated by a user enters the area of ​​the first frame of any of the one or more graphic objects in the GUI screen 200, and displaying the character information of the character object in the second frame when the cursor enters the area of ​​the second frame of any of the one or more character objects.

[0099] In this way, the system can perform image recognition on graphic objects included in the GUI screen 200, image recognition and character recognition on text objects included in the GUI screen 200, obtain coordinate information of the frames surrounding the graphic objects and text objects, and obtain character information of the text objects. As a result, even if the graphic objects or text objects move within the GUI screen 200, the moved graphic objects and text objects can be recognized with high accuracy. In addition, the system can display either the first or second frame depending on whether the cursor is present or not, allowing for an intuitive understanding of whether a graphic object or text object has been recognized by selectively displaying the frame.

[0100] Furthermore, users can specify objects within the GUI screen and obtain or input their coordinate information simply by selecting them with the cursor and clicking. This eliminates the need to manually input the coordinate information of objects within the GUI screen, thus saving time and effort.

[0101] Furthermore, for example, by simply moving the cursor over an object included in the GUI screen 200 created by emulator 4, the user can input the object's specifications and coordinate information without manually entering them. This reduces the time and effort required to run the program associated with that object, allowing for efficient operational verification.

[0102] (2) In such an information processing method, in the information acquisition step, coordinate information of the third frame surrounding the graphic object and the text object attached to the graphic object is further acquired, and in the display processing step, if the cursor enters the area of ​​the third frame and the area of ​​the first frame and the first frame and the third frame overlap, the first frame is preferably displayed first, and if the cursor enters the area of ​​the third frame and the area of ​​the second frame and the second frame and the third frame overlap, the second frame is preferably displayed first. As a result, when the displayed frames overlap, the frame with the smaller area is displayed first, making the selectively displayed frame easier to see and making it easier to specify the object.

[0103] (3) In this information processing method, each of the one or more graphic objects is a graphic icon, and each of the one or more character objects is an image representing the character attached to the graphic icon. Therefore, even if the graphic icons and the characters attached to the graphic icons included in the GUI screen 200 move within the GUI screen 200, the moved graphic icons and the characters attached to the graphic icons can be recognized with high accuracy.

[0104] (4) The information processing method further includes a notification step of notifying the user terminal 2 of the coordinate information of the first frame surrounding each of one or more graphic objects, the character information of each of one or more character objects, and the coordinate information of the second frame surrounding each of one or more character objects. The screen acquisition step, the difference detection step, and the notification step are performed by the information processing device 1, the recognition step is performed using the AI ​​of the information processing device 1, and the information acquisition step and the display processing step may be performed by the user terminal 2. This allows for accurate recognition of graphic objects and character objects even if they move within the GUI screen 200 provided to the user terminal 2.

[0105] (5) In addition, in such an information processing method, the GUI screen 200 is a screen created by the information processing device 1 using the emulator 4 to verify the operation of programs associated with one or more graphic objects and one or more character objects, and may be displayed on the user terminal 2. Even if the GUI screen provided to the user terminal 2 is a screen created using the emulator 4, the graphic objects and character objects that have been moved on the GUI screen can be recognized with high accuracy.

[0106] (6) Furthermore, the screen acquisition step, difference detection step, information acquisition step, and display processing step may be performed by the user terminal 2, and the recognition step may be performed using the AI ​​of the user terminal 2. This allows the user terminal 2 to accurately recognize graphic objects and text objects that have been moved on the GUI screen. In addition, since the user terminal 2 can be installed in a device that is connected to a device such as a drive recorder, convenience is improved.

[0107] (7) In this information processing method, user operation is performed by the user using a mouse or touch. This allows the user to place the cursor over the area of ​​the graphic object or text object they wish to select using a mouse or touch, thereby selectively displaying the outline of the graphic object or text object they wish to select.

[0108] (8) In addition, in such information processing methods, user operation may be performed by the user's gaze using the gaze detection function of the user terminal. This allows the user to place the cursor over the area of ​​the graphic object or text object they wish to select by using their gaze, thereby selectively displaying the outline of the graphic object or text object they wish to select.

[0109] (9) Another embodiment of the information processing device is characterized by comprising: a screen acquisition unit 11 that acquires a GUI screen 200 at predetermined intervals; a difference detection unit 13 that compares a first GUI screen with a second GUI screen acquired a predetermined time after the time the first GUI screen was acquired and detects changes in the GUI screen 200; a recognition unit 15 that, if the change is greater than or equal to a predetermined threshold, performs recognition processing for one or more graphic objects and one or more character objects included in the second GUI screen, and does not perform recognition processing for graphic objects and character objects included in the second GUI screen if the change is less than a predetermined threshold; an information acquisition unit 16 that acquires coordinate information of a first frame surrounding each of the one or more graphic objects in the second GUI screen obtained by performing the recognition processing, character information of each of the one or more character objects, and coordinate information of a second frame surrounding each of the one or more character objects; and a display processing unit 23 that, when a cursor operated by a user enters the area of ​​the first frame of any of the one or more graphic objects in the GUI screen, displays the first frame, and when the cursor enters the area of ​​the second frame of any of the one or more character objects, displays the character information of the character object in the second frame.

[0110] The GUI screen 200 can perform image recognition on graphic objects and image and character recognition on text objects, obtain coordinate information of the frames surrounding the graphic and character objects, and obtain character information for the text objects. This allows for accurate recognition of the moved graphic and character objects even if they move within the GUI screen 200. Furthermore, the first or second frame can be displayed depending on whether the cursor is present or not, allowing for an intuitive understanding of whether a graphic or character object has been recognized by selectively displaying the frame.

[0111] Furthermore, users can obtain or input coordinate information for objects within the GUI screen simply by selecting and clicking on them with the cursor. This eliminates the need to manually input the object's specifications and coordinate information, thus saving time and effort. [Explanation of symbols]

[0112] 1. 1A Information Processing Device 2 User terminals 3. Communication Network 4 Emulators 11 Screen acquisition section 12 Storage section 13 Difference detection unit 14 Judgment section 15 Recognition part 16 Information acquisition department 17 Notification Department 17a Coordinate information, text information 21 Control section 22 Terminal acquisition section 23 Display Processing Unit 24 Execution Department 25 Eye-line detection unit 51, 51a AI for image recognition 52, 52a AI for character recognition 201a, 201b, 201c, 201d Shape Icons 202a, 202b, 202c, 202d Text Icons 203c, 203d Dotted line frame 221a, 221b cursor Frames 251a, 251b, 251c, 251d, 252a, 252b, 252c, 252d, 253c, 253d 170 CPU 171 memory 172 Storage Devices 173 connection ports 174 Communication Circuit 175 Bus Line 1711 OS 1712 Browser Program 1713 Program 1714 Predetermined threshold 1715 Recognition Processing Result DB 1715a GUI screen information 1715b Object ID 1715c Location information 1715d characters 1715e Text Information

Claims

1. A screen acquisition step to acquire the GUI screen at predetermined intervals, A difference detection step involves comparing a first GUI screen with a second GUI screen obtained a predetermined time after the time the first GUI screen was acquired, in order to detect a change in the GUI screen. If the value indicating the change is greater than or equal to a predetermined threshold, the recognition process for one or more graphic objects and one or more character objects included in the second GUI screen is executed; if the value indicating the change is less than the predetermined threshold, the recognition process for the graphic objects and character objects included in the second GUI screen is not executed. An information acquisition step is to acquire the coordinate information of the first frame surrounding each of the one or more graphic objects in the second GUI screen obtained by performing the recognition process, the character information of each of the one or more character objects, and the coordinate information of the second frame surrounding each of the one or more character objects. The display processing step includes: displaying the first frame when the cursor, operated by the user, enters the area of ​​the first frame of any of the one or more graphic objects on the GUI screen; and displaying the character information of the character object in the second frame when the cursor enters the area of ​​the second frame of any of the one or more character objects. Information processing methods.

2. In the information acquisition step described above, the coordinate information of the third frame surrounding the graphic object and the text object attached to the graphic object is further acquired. In the aforementioned display processing step, If the cursor enters the area of ​​the third frame and the area of ​​the first frame, and the first frame and the third frame overlap, the first frame will be displayed preferentially. If the cursor enters the area of ​​the third frame and the area of ​​the second frame, and the second frame and the third frame overlap, the second frame will be displayed preferentially. The information processing method according to claim 1.

3. Each of the one or more graphic objects mentioned above is a graphic icon, Each of the one or more character objects mentioned above is an image representing the character to be attached to the graphic icon. The information processing method according to claim 1.

4. The aforementioned information processing method is Furthermore, the process includes a notification step of notifying the user terminal of the coordinate information of the first frame surrounding each of the one or more graphic objects, the character information of each of the one or more character objects, and the coordinate information of the second frame surrounding each of the one or more character objects. The screen acquisition step, the difference detection step, and the notification step are performed by the server. The aforementioned recognition step is performed using the AI ​​of the server. The information acquisition step and the display processing step are performed by the user terminal. The information processing method according to claim 3.

5. The GUI screen is a screen created by the server using an emulator to verify the operation of the programs associated with each of the one or more graphic objects and the one or more character objects, and is displayed on the user terminal. The information processing method according to claim 4.

6. The screen acquisition step, the difference detection step, and the information acquisition step and the display processing step are performed by the user terminal. The recognition step is performed using the AI ​​of the user terminal. The information processing method according to claim 3.

7. The user operation is a mouse operation or touch operation by the user. The information processing method according to any one of claims 1 to 6.

8. The aforementioned user operation is an operation performed by the user's gaze using the eye-tracking function of the user terminal. The information processing method according to any one of claims 1 to 6.

9. A screen acquisition unit that acquires GUI screens at predetermined intervals, A difference detection unit that compares a first GUI screen with a second GUI screen obtained a predetermined time after the time the first GUI screen was acquired, and detects changes in the GUI screen. If the value indicating the change is greater than or equal to a predetermined threshold, the recognition unit performs recognition processing for one or more graphic objects and one or more character objects included in the second GUI screen, and if the value indicating the change is less than the predetermined threshold, the recognition unit does not perform recognition processing for the graphic objects and character objects included in the second GUI screen. An information acquisition unit that acquires coordinate information of the first frame surrounding each of the one or more graphic objects in the second GUI screen obtained by performing the recognition process, character information of each of the one or more character objects, and coordinate information of the second frame surrounding each of the one or more character objects, The system includes a display processing unit that, when a cursor operated by a user enters the area of ​​the first frame of any of the one or more graphic objects on the GUI screen, displays the first frame, and when the cursor enters the area of ​​the second frame of any of the one or more character objects, displays the character information of the character object in the second frame. Information processing device.

Images