Information processing device, information processing method, program

Abstract

To support the efficiency of input processing for user operations. [Solution] An information processing device comprising: a first acquisition means for acquiring screen data of a first screen displayed on a display device; a detection means for detecting a specific object from the acquired screen data; an assignment means for assigning identification information to each of the detected objects; a display means for causing the display device to display a second screen in which identifiers corresponding to the identification information assigned to each object are superimposed on the first screen at the positions on the first screen where the detected objects are displayed; a second acquisition means for acquiring voice instructions from a user; an extraction means for extracting identifiers and operation content included in the voice instructions; and an execution means for causing the object corresponding to the extracted identifier to execute input processing based on the extracted operation content.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Conventionally, there has been a demand for a system that supports the improvement of the efficiency of operation input processing.

[0003] For example, Patent Document 1 discloses a technique for performing RPA robot control by voice input.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the technique of Patent Document 1 is premised on operation control that saves a series of information processing in advance and triggers a predetermined action, which is also known as so-called RPA (Robotic Process Automation). Therefore, control for operations such as when the order of operation inputs is not necessarily determined or when the values related to operation inputs are not necessarily complete has not been considered.

[0006] An object of the present invention is to support the improvement of the efficiency of operation input processing.

Means for Solving the Problems

[0007] An information processing apparatus according to one aspect of the present invention comprises: a first acquisition means for acquiring screen data displayed on a display device; a detection means for detecting a specific object from the acquired screen data; an assignment means for assigning identification information to each of the detected objects; a display means for causing the display device to display a second screen in which the identification information assigned to each object is superimposed on the first screen at the position where the detected objects are displayed on the first screen; a second acquisition means for acquiring voice instructions from a user; an extraction means for extracting identification information and operation content included in the voice instructions; and an execution means for causing the object to which the extracted identification information is assigned to execute input processing based on the extracted operation content. This embodiment can help improve the efficiency of input processing for operations.

[0008] An information processing method according to one aspect of the present invention involves one or more information processing devices performing: a first acquisition step of acquiring screen data displayed on a display device; a detection step of detecting a specific object from the acquired screen data; an assignment step of assigning identification information to each of the detected objects; a display step of causing the display device to display a second screen in which the identification information assigned to each object is superimposed on the first screen at the position where the detected objects are displayed on the first screen; a second acquisition step of acquiring voice instructions from a user; an extraction step of extracting identification information and operation content included in the voice instructions; and an execution step of causing the object to which the extracted identification information has been assigned to perform input processing based on the extracted operation content. This embodiment can help improve the efficiency of input processing for operations.

[0009] A program according to one aspect of the present invention causes one or more information processing devices to perform a first acquisition step of acquiring screen data displayed on a display device; a detection step of detecting a specific object from the acquired screen data; an assignment step of assigning identification information to each of the detected objects; a display step of causing the display device to display a second screen in which the identification information assigned to each object is superimposed on the first screen at the position where the detected objects are displayed on the first screen; a second acquisition step of acquiring voice instructions from a user; an extraction step of extracting identification information and operation content included in the voice instructions; and an execution step of causing the object to which the extracted identification information has been assigned to perform input processing based on the extracted operation content. This embodiment can help improve the efficiency of input processing for operations. [Effects of the Invention]

[0010] According to the present invention, it is possible to support the efficiency of input processing for operations. [Brief explanation of the drawing]

[0011] [Figure 1] The configuration of an information processing system relating to one aspect of this disclosure is shown. [Figure 2] This is a diagram illustrating the operation of an information processing system relating to one aspect of this disclosure. [Figure 3] This shows the hardware configuration of an information processing device relating to one aspect of this disclosure. [Figure 4] This shows the functional configuration of an information processing device according to one aspect of this disclosure. [Figure 5] An example screen related to one aspect of this disclosure is shown. [Figure 6] An example screen related to one aspect of this disclosure is shown. [Figure 7] An example screen related to one aspect of this disclosure is shown. [Figure 8] An example screen related to one aspect of this disclosure is shown. [Figure 9]It is a sequence diagram showing an operation example of an information processing system according to an aspect of the present disclosure.

Mode for Carrying Out the Invention

[0012] Hereinafter, a first aspect of the present invention will be described with appropriate reference to the drawings.

[0013] (First Aspect) 1. Configuration of Information Processing System FIG. 1 illustrates the configuration of an information processing system 1 according to an aspect of the present disclosure. The information processing system 1 includes an information processing apparatus 10, a display system 3 including a display apparatus 30, and an audio input apparatus 50. Each component included in the information processing system 1 is communicably connected via a communication network N.

[0014] The communication network N is a wired or wireless network. For example, the Internet, LAN (Local Area Network), dedicated line, telephone line, enterprise internal network, mobile communication network, Bluetooth (registered trademark), WiFi (Wireless Fidelity), other communication lines, combinations thereof, and the like can be mentioned. Each component of the information processing system 1 may be within the same facility or within different facilities.

[0015] The information processing apparatus 10 has a functional configuration described later and is an apparatus that executes various information processes based on various information input from the display apparatus 30 and the audio input apparatus 50. The information processing apparatus 10 is not particularly limited, and examples thereof include a desktop, a laptop, and other dedicated or general-purpose computers. The information processing apparatus 10 may be composed of a single computer or may be composed of a plurality of computers on the communication network N.

[0016] The display apparatus 30 is an apparatus that displays a screen including various information. For example, a monitor, a liquid crystal display, and displays such as a smartwatch and AR glasses can be mentioned.

[0017] In the present disclosure, the display device 30 is described as having a communication function with the information processing device 10 and other devices. However, the display device 30 may be configured to communicate with the information processing device 10 and further other devices via any connection device or any facility such as other devices. In this case, an information processing system including the display device 30 and any facility providing a communication function is referred to as a display system 3 in the present disclosure.

[0018] The voice input device 50 is a device used when a user inputs voice. For example, a microphone, and wearable devices such as a head-mounted display, smart glasses, and a smart watch having a microphone can be mentioned.

[0019] FIG. 2 is a diagram for explaining a usage mode of the information processing system 1. The user inputs voice using the voice input device 50. The information processing device 10 executes input processing of an operation on the screen displayed on the display device 30 based on the voice.

[0020] In the present disclosure, the information processing system 1 is described as including the display device 30 and the voice input device 50, but it is not necessarily limited to this configuration. For example, the functions of the display device 30 and the voice input device 50 may be realized by appropriately using the hardware configuration of the information processing device 10.

[0021] 2. Hardware Configuration FIG. 3 illustrates the hardware configuration of the information processing device 10. The information processing device 10 includes a processor 11, a storage device 12, a communication IF (Interface) 13, an input device 14, and an output device 15.

[0022] The processor 11 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0023] The storage device 12 is a memory, HDD (Hard Disk Drive), and / or SSD (Solid State Drive), etc. The memory is, for example, RAM (Random Access Memory) and / or ROM (Read Only Memory).

[0024] Communication IF13 is a device that communicates data with other devices via a communication network N, either by wire or wirelessly.

[0025] The input device 14 accepts information input. For example, it may be a keyboard, touch panel, mouse, camera, and / or microphone.

[0026] The output device 15 outputs information. For example, it may be a display, a touch panel, and / or a speaker.

[0027] The hardware configuration of the display device 30 and the audio input device 50 can be the same as that of the information processing device 10, so a description is omitted.

[0028] 3. Functional Configuration Figure 4 illustrates the functional configuration of the information processing device 10. The information processing device 10 includes a storage unit 100 and a control unit 200. The control unit 200 includes a first acquisition means 210, a detection means 220, an assignment means 230, a display means 240, a second acquisition means 250, an extraction means 260, an execution means 270, and a learning means 280. The storage unit 110 can be realized by a storage device 12 provided by the information processing device 10. The control unit 120 can be realized by the processor 11 provided by the information processing device 10 executing a program stored in the storage device 12. This program can be stored in a storage medium. The storage medium on which the program is stored may be a non-transitory computer-readable medium. The non-transitory storage medium may be, for example, a USB (Universal Serial Bus) memory or a CD-ROM (Compact Disc Read-Only Memory).

[0029] (Storage part) The memory unit 100 stores various types of information necessary for the information processing device 10 to operate.

[0030] The storage unit 100 may store an identification information DB 110. The identification information DB 110 may store assigned identifiers. The identification information DB 110 may further store the correspondence between each identifier and an object. The identification information DB 110 may be referenced by the extraction means 260 and execution means 270, which will be described later.

[0031] The storage unit 100 may store an operation content DB 120. The operation content DB 120 may store rules that define input processing to be performed in response to the user's voice instructions. The rules for input processing are not particularly limited, but for example, they may be: "When the user says 'Select (identifier)1' (while a screen with an identifier is displayed), input a click operation to the object corresponding to identifier 1," or "When the user says 'Enter OK (text to be entered)' (while a click operation has been entered to the object corresponding to identifier 1), perform a text input operation of 'OK (text to be entered)' to the object corresponding to identifier 1." Another example of a rule is: "When the user says 'Enter OK (text to be entered) for (identifier)1', input a click operation to the object corresponding to identifier 2, and perform a text input operation of 'OK (text to be entered)'." The operation content DB 120 may be referenced by the extraction means 260 and execution means 270 described later.

[0032] The operation content DB120 may store a variety of utterances that instruct a single operation. For example, the operation "input a click operation" may be stored as being instructed by either the utterance "select" or "click". This allows the system to execute "input a click operation" even if there is variation in the operation extracted from the voice instruction, such as "select" or "click," when referenced by the extraction means 260 and execution means 270 described later. The operation content DB120 may also store utterances registered by the user. For example, the user may be able to pre-register words that are likely to be used or words that are less likely to be misrecognized.

[0033] The memory unit 100 may store the learning DB 130. The learning DB 130 may store the mathematical model M used by the learning means 280, which will be described later. The learning DB 130 may store image information acquired by the first acquisition means 210 and image information displaying objects detected by the detection means 220 as learning data for the mathematical model M.

[0034] (1st acquisition method) The first acquisition means 210 acquires screen data displayed on the display device 30.

[0035] In this disclosure, the acquisition of information (data) may include receiving information (data) via output signals from other devices or equipment, etc., via a communication network N or any connected device, and reading information (data) stored in the storage unit 100.

[0036] Screen data is information used to display a screen. Screen data may include image information for displaying visual elements displayed on the screen (e.g., application windows, object icons, mouse cursors, and backgrounds), information about the display position (coordinate information) for reproducing the display environment, and information about the date and time of display. Image information may include information for displaying images of characters, symbols, or numbers of visual elements, as well as information for displaying icons and backgrounds other than text of visual elements.

[0037] The screen data is, for example, screen data captured from the screen (first screen) displayed on the display device 30.

[0038] Screen capture can be either a screenshot, which temporarily records the screen, or a video capture, which sequentially records the screen as a video.

[0039] Screen capture may be performed using the recording function provided by the display device 30. For example, the displayed screen may be captured using the screen recording function provided by the display device 30. For example, the displayed screen may be captured using the screen recording function provided by a software program executed by the display device 30. The information processing device 10 can acquire screen data by receiving an output signal from the display device 30.

[0040] Screen capture may be performed using a device connected to the display device 30 or the display system 3. For example, the displayed screen may be captured using a capture device that uses HDMI (High-Definition Multimedia Interface) (registered trademark). For example, the displayed screen may be captured using a capture device that uses USB (Universal Serial Bus). The information processing device 10 can acquire screen data by receiving output signals from these devices or by receiving output signals from these devices via the display device 30 or the display system 3 connected to these devices.

[0041] The acquisition of screen data may be performed in response to the detection of a change in the screen data displayed on the display device 30. For example, it may be performed in response to a change in the information displayed on the screen due to a scroll operation or a click operation. The acquisition of screen data may also be performed continuously.

[0042] (Detection means) The detection means 220 detects specific objects from the visual elements included in the screen data acquired by the first acquisition means 210. The objects to be detected may be objects that can accept user input. Examples of objects that can accept user input include clickable objects and text fields with text or numerical input fields.

[0043] Object detection may be performed using well-known detection methods. For example, the detection means 220 may detect objects by image recognition. For example, objects may be detected based on the feature quantities of image information contained in the screen data. Specifically, for example, objects may be detected based on the image feature quantities corresponding to the shape, edges, color, and design of the object. Template matching may also be performed, and the template images used for matching may be stored in advance in the storage unit 110 and referenced by the detection means 220. The template images used for matching may also be used for training by the learning means 280, which will be described later. Furthermore, the detection means 220 may detect objects using OCR (Optical Character Recognition). Specifically, for example, it may calculate the coordinates of the position where the word "OK" is displayed and detect the object. Internal structure data such as XML may be used to calculate the coordinates. Furthermore, the detection means 220 may detect objects using software or the like that provides detection functions for other well-known visual elements. Object detection can be achieved by combining the methods described above as appropriate.

[0044] Object detection may be configured to detect objects specified in advance by the user. For example, image information containing objects to be detected, as specified in advance by the user, may be used as training data, and a mathematical model M trained by the training means 280 described later may be used to detect the objects.

[0045] Figure 5 is an illustrative diagram of the operation in which the detection means 220 detects a specific object from screen data. In addition, as will be described later in a second embodiment, the information processing device 10 may further display the image diagram D50 (third screen) illustrated in Figure 5. In the example of image diagram D50, multiple specific objects included in the screen data are detected, and a detection frame indicating detection is displayed at the position of each detected object. For example, a clickable object d51 for expanding the display of the item "Prepare part A" is detected. By clicking object d51, the display of the item "Prepare part A" can be expanded and displayed in a pull-down format. Also, in the example of image diagram D50, the display of the item "1. Result Input" is expanded and displayed, and clickable objects (for example, object d52) for selecting each of the four options displayed in selection format: "1. Work AJU", "2. Work DHJ", "3. Work KHS", and "4. Work KMO". For example, by clicking object d52, the option "3. Work KHS" can be selected. In addition, in the example shown in image D50, a clickable object d53 for selecting "OK" is detected. By clicking object d53, "OK" can be selected.

[0046] Object detection from screen data may be performed each time screen data is acquired.

[0047] (Means of granting) The assignment means 230 assigns identification information to each object detected by the detection means 220.

[0048] Identification information is information used to display an identifier that makes each object identifiable. Identifiers can include, for example, numbers, letters or symbols such as alphabets or hiragana, or combinations thereof.

[0049] Identifiers may be assigned according to the position in which objects are contained on the screen. For example, suppose the screen contains an object for selecting the first option, an object for selecting the second option, and an object for selecting the third option, starting from the top to the bottom. In this case, the assignment means 230 may associate identifier 1 with the first object, identifier 2 with the second object, and identifier 3 with the third object, in accordance with the sequence of positions in which objects are contained on the screen.

[0050] In this disclosure, when we refer to assigning identification information to an object, this may include associating information for displaying an identifier with information for displaying the object. The object may be displayed together with the corresponding identifier by the display means 240 described later.

[0051] Figure 6 is an illustrative diagram of the operation in which identification information is assigned to each object by the assignment means 230. In the example of illustrative diagram D60, identification information is assigned to each detected object, and the corresponding identifier is displayed at the location of each detected object. For example, the identifier d61 "1" is displayed at the location of clickable object d51 for expanding the display of the item "Prepare part A". Also, the identifier d62 "5" is displayed at the location of clickable object d52 for selecting the option "4. Work KMO". Furthermore, the identifier d63 "7" is displayed at the location of clickable object d53 for selecting "OK". As will be described later, the information processing device 10 can also display the image diagram D60 (second screen) illustrated in Figure 6 using the display means 240.

[0052] (Display means) The display means 240 causes the display device 30 to display a second screen containing an identifier at a predetermined position on the first screen.

[0053] In this disclosure, when the information processing device 10 causes the display device 30 to display a screen, this may include the information processing device 10 transmitting to the display device 30 information necessary for the display device 30 to display the screen.

[0054] In one embodiment, the second screen includes a display of an identifier corresponding to the object at the position where the object of the first screen is displayed. The display of the identifier may be superimposed on part or all of the corresponding object. The display of the identifier may be around the corresponding object. The display of the identifier may be around the corresponding object.

[0055] (Second acquisition method) The second acquisition means 250 acquires voice instructions from the user. For example, it acquires voice instructions by acquiring voice data entered by the user. Voice instructions are voice data in which the user instructs the input process. Preferably, voice instructions include at least an identifier and the content of the operation.

[0056] The operations may include, for example, the operation of input devices such as a mouse, keyboard, touchscreen, and trackpad. For example, they may include clicking (double-clicking is also acceptable), tapping, and scrolling.

[0057] The operation may be, for example, entering or deleting text. For example, the operation may be "Enter OK", "Enter 50.2mL", "Delete", etc. "OK", "50.2mL", etc., which may be included in the operation are the text to be entered or deleted.

[0058] The operation description may further include information regarding the direction of the operation. For example, the operation description may be dragging identifier 1 to the right and swiping identifier 2 to the left.

[0059] Voice commands may be acquired using the recording function of the voice input device 50. For example, the information processing device 10 can acquire voice commands by receiving voice data recorded using the recording function of the voice input device 50.

[0060] Furthermore, the information processing device 10 can acquire voice instructions by receiving voice data input to the information processing device 10.

[0061] (extraction means) The extraction means 260 extracts identifiers and operation details included in the voice instructions.

[0062] For example, if a user inputs the voice command "Select number 5" via the voice input device 50, the extraction means 260 extracts the identifier "5" from the voice command and extracts the operation content "Select".

[0063] Identifiers and operation details can be extracted using well-known speech recognition techniques. For example, speech data can be converted into text data and a predetermined keyword can be searched for. For example, the location of a predetermined keyword in the speech data can be searched for.

[0064] (Means of execution) The execution means 270 causes the object to which the identification information of the identifier extracted by the extraction means 260 has been assigned to perform input processing based on the extracted operation content.

[0065] For example, using an identifier extracted from a voice command as an indicator, the system refers to the identification information database and the operation content database to identify the object corresponding to that identifier. Furthermore, it refers to the operation content database to execute the operation content extracted from the voice command.

[0066] Figure 7 is an illustrative diagram of the operation in which the execution means 270 performs input processing. For example, when a user inputs the voice command "Select number 5" via the voice input device 50, a click operation (process a7 shown in the illustrative diagram D70) is performed on the object d52 corresponding to the identifier "5".

[0067] (Learning methods) The learning method 280 learns images containing objects using a mathematical model M. The mathematical model M is, for example, a model that learns image features using images of objects as training data. Alternatively, the mathematical model M may be a pre-trained model that takes screen data as input data and outputs information about the positions of objects contained in that screen as output data.

[0068] 4.Operation Referring to Figure 9, an example of the operation of the information processing system 1 will be explained. For the sake of explanation, the screen examples in Figures 5 to 7 will be referenced as appropriate.

[0069] First, the display device 30 displays the first screen (S100). The information processing device 10 captures the first screen using the first acquisition means 210 (S110) and acquires the screen data displayed on the display device 30 (S120).

[0070] The information processing device 10 detects a specific object from the screen data using the detection means 220 (S130).

[0071] The information processing device 10 assigns identification information to each object using the assignment means 230 (S140).

[0072] The information processing device 10 causes the display device to display the second screen using the display means 240 (S150). Here, the second screen includes identifiers of identification information assigned to each object at the positions where objects are displayed on the first screen captured in S110.

[0073] The display device 30 receives information from the information processing device 10 for displaying the second screen and displays the second screen (S160).

[0074] Next, when the user inputs a voice command using the voice input device 50 (S170), the information processing device 10 acquires the voice command using the second acquisition means 250 (S180).

[0075] The information processing device 10 extracts the identifier and operation content contained in the voice instruction using the extraction means 260 (S190).

[0076] The information processing device 10, using the execution means 270, causes the object corresponding to the identifier extracted from the voice instruction to perform input processing based on the operation content extracted from the voice instruction (S200).

[0077] 5. Other aspects The first aspect has been described above; other aspects will be described below.

[0078] (Second aspect) In one embodiment, the information processing device 10 may further display a third screen on the display device 30 after the detection processing by the detection means 220. An example of operation in this case will be described. The acquisition processing by the first acquisition means 210 and the detection processing by the detection means 220 are the same as in the first embodiment described above, so their explanation will be omitted. The information processing device 10 assigns identification supplementary information to each of the detected objects using the assignment means 230. Then, the information processing device 10 displays the third screen on the display device 30 using the display means 240. Here, the third screen includes identification supplementary information at the position where the object is displayed on the captured first screen. Identification supplementary information is information that makes the location of each detected object verifiable. Examples of identification supplementary information include a frame surrounding each object, a point marker targeting the center or characteristic points of each object, a highlight superimposed on the range of each object, effects that add shadows or blurring effects around each object, a line that surrounds the shape of each object according to its shape, or a combination of these. The display method of the identification supplement information may be selected according to the type of identification supplement information. For example, the identification supplement information may be displayed superimposed on part or all of the corresponding object. The identification supplement information may also be displayed around the corresponding object. The display on the third screen display device 30 may occur after the display of the first screen. Conversely, the display of the third screen may occur before the display of the second screen. In parallel with the process for displaying the third screen on the display device 30, the process for displaying the second screen on the display device 30 may be executed. Furthermore, the second screen may be displayed superimposed on the identification supplementary information of the third screen. In this embodiment, the third screen after the detection processing by the detection means 220 is displayed on the display device 30, allowing the user to check the progress of the processing. For example, the progress of the processing can be checked on the third screen, which has relatively less visual interference compared to the second screen on which identification information is attached. This makes it possible to detect defects in the detection process. In one embodiment, the information processing device 10 may also display, on the screen after the detection processing by the detection means 220, an identifier corresponding to the identification information assigned to each object, along with identification supplementary information to allow confirmation of the position of each object, at the position where the detected object is displayed. For example, in the screen D80 illustrated in Figure 8, the identifier d83 is displayed along with a frame d82 surrounding object d81 at the position of object d81 for displaying the item "Prepare part A". Identification supplementary information and identifiers corresponding to the same object may be displayed in a predetermined positional relationship. For example, they may be displayed within a predetermined distance range that is defined in advance. With this embodiment, it becomes easier for the user to recognize the object that each identifier is targeting.

[0079] (Third aspect) In one embodiment, the information processing device 10 may acquire screen data, detect objects, and assign identification information in response to the detection of changes to the screen data. An example of this operation will be described below. The information processing device 10 acquires screen data each time it detects a change to the screen data displayed on the display device 30. For example, it acquires screen data in response to changes in the information displayed on the display device 30 due to user input such as scrolling or clicking. The information processing device 10 detects objects from the acquired screen data and assigns identification information to the detected objects. The assignment of identifiers to objects may be performed each time screen data is acquired. For example, suppose a given screen displays three objects from top to bottom for selecting "Option 1," "Option 2," and "Option 3." Suppose that, according to the position of these three objects, identifier 1 is assigned to the "Option 1" object, identifier 2 to the "Option 2" object, and identifier 3 to the "Option 3" object. Now, suppose a scroll operation is input, causing the objects for selecting "Option 1" and "Option 2" to move out of frame, resulting in the display of objects for selecting "Option 3," "Option 4," and "Option 5" from top to bottom. In this case, the assignment means 230 may reassign identifiers, such as assigning identifier 1 to the "Option 3" object, identifier 2 to the "Option 4" object, and identifier 3 to the "Option 5" object. In this embodiment, each time the content displayed on the screen changes, an identifier is assigned (for example, sequentially from the top to the bottom of the screen, depending on the position where the objects are displayed). Therefore, even if there are many objects displayed on the screen, it becomes unnecessary to use a vast number of different identifiers. For example, in a screen where 10 objects are displayed on one screen, but considering the area reachable by scrolling, 2000 objects may be displayed, instead of associating one identifier with each object and fixing the correspondence, identification information can be assigned to all objects displayed on one screen. Therefore, this is useful in that it reduces the burden on data storage. Furthermore, for the user, it is useful in that the visibility is good because there are fewer types of identifiers involved.

[0080] (Fourth aspect) In one embodiment, the screen may include an object for expanding a display in a pull-down format. In this case, the assignment means 230 may assign identification information each time the display is expanded. For example, in the image diagram D50 of Figure 5, an object d51 for expanding the display of the item "Prepare part A" is displayed. When the user selects the object d51, the display of the item "Prepare part A" is expanded and displayed in a pull-down format. The first acquisition means 210 then acquires the displayed screen data, the detection means 220 detects the object, and the assignment means 230 can assign identification information to the object displayed by the expansion. The identifiers for newly displayed objects after expansion may be assigned while maintaining the order and assignment of identifiers to the objects that were displayed before the expansion. For example, suppose that before the expansion, the objects displayed were assigned identifier 1 to the first object, identifier 2 to the second object, and identifier 3 to the third object, according to their positions. Now, suppose that after the display expansion, a fourth and fifth object are newly displayed between the first and second objects. In this case, the assignment means 230 may assign identifier 4 to the fourth object and identifier 5 to the fifth object while maintaining the assignment of identifiers 1 to 3 to the first to third objects. In this embodiment, even if additional displays are added by expanding the display in a pull-down format, the process of assigning identification information to the additional displays can be executed without having to restart the process of assigning identification information from the beginning. Therefore, it is useful in that it improves the efficiency of data processing, and is also useful for the user in that it provides good visibility.

[0081] (Fifth aspect) In one embodiment, the display means 240 may display identifiers in different ways depending on the operations that the object can accept. For example, the display of identifiers for objects that include a text box may be different from the display of identifiers for objects that do not include a text box. This embodiment can improve the visibility of identifiers. In this embodiment, the information processing device 10 may include detecting the operation content that each object can accept using the detection means 220. In this case, the detection means 220 may estimate the operation content that each object can accept using a mathematical model M learned using training data.

[0082] (Sixth aspect) In one embodiment, the execution means 270 may further perform processing to control the cursor display position according to the content of the input operation to be performed. For example, as illustrated in Figure 7, when the user inputs the voice command "click number 5", a click operation is input to the object d52 corresponding to the identifier "5". When a click operation is input, the cursor display position may be controlled so that the cursor display position is closer to the position of object d52. This embodiment is useful because it improves the visibility of the data operation control status.

[0083] (7th aspect) In one embodiment, the screen data may be internal structure data of an application or website, which further includes, for example, structural data of visual elements. The internal structure data may include the hierarchical structure of HTML elements, or the attributes, values, or states (enabled or disabled, unselected or selected, etc.) of visual elements. The information processing device 10 acquires the internal structure data as screen data using the first acquisition means 210 and detects objects using the detection means 220. The detection means 220 may use a well-known detection function module (for example, a detection function module using CSS selectors or XPath, etc.) that can detect objects from the internal structure data. The internal structure data may be used in combination with screen data captured from the screen. This embodiment is useful because it allows for smoother detection even when the screen layout changes, compared to an embodiment that uses only screen capture data. It is also useful because it allows for highly accurate detection.

[0084] As described above, this disclosure can help improve the efficiency of input processing for operations. Users will be able to manipulate data using their voice. Specifically, they can add (text input, etc.) or delete (text deletion, etc.) information displayed on the screen using their voice. Therefore, when users input instructions related to data manipulation, they will no longer need to use their hands, and will be able to control operations on the screen without physical contact. As a result, users will no longer have to stop their current work to input operations, but will be able to perform data input operations while continuing their current work, which is useful from the standpoint of time efficiency. Furthermore, even in cases where two people were required—one to handle operation input and another to handle the current work—this technology eliminates the need to have a person in charge of operation input, which is useful from the standpoint of labor saving. In addition, compared to operation control modules known as RPA, which store a series of information processing and enable the execution of that series of information processing triggered by a predetermined action, this technology enables the execution of information processing that performs desired operation input on a desired object. Therefore, the technology of this disclosure is applicable to data input operations that have characteristics such as the order of operations not being fixed or the values ​​related to the operations not being consistent, as well as operations such as recording various data and progress associated with experiments.

[0085] It should be noted that the present invention is not limited to the embodiments described above, and can be implemented in various other forms without departing from the spirit of the invention. For this reason, the above embodiments are merely illustrative in all respects and should not be interpreted restrictively. For example, the order of each processing step described above can be arbitrarily changed or executed in parallel, as long as there is no inconsistency in the processing content. [Explanation of Symbols]

[0086] 1... Information processing system, 3... Display system, 10... Information processing device, 11... Processor, 12... Memory device, 13... Communication interface, 14... Input device, 15... Output device, 210... First acquisition means, 220... Detection means, 230... Assignment means, 240... Display means, 250... Second acquisition means, 260... Extraction means, 270... Execution means, 280... Learning means

Claims

1. A first acquisition means for acquiring screen data of a first screen displayed on a display device, A detection means for detecting a specific object from the acquired screen data, A means for assigning identification information to each of the detected objects, Display means for displaying on the display device a second screen in which identifiers corresponding to the identification information assigned to each object are superimposed on the first screen at the position where the detected object is displayed on the first screen, A second acquisition means for acquiring voice commands from the user, Extraction means for extracting identifiers and operation details included in the aforementioned voice instructions, An execution means for causing the extracted identifier to perform input processing based on the operation content on the corresponding object, An information processing device equipped with the following features.

2. The aforementioned screen data is screen data captured from the first screen displayed on the display device. The information processing apparatus according to claim 1.

3. The aforementioned operation is an operation related to at least one of the following: mouse, keyboard, touchscreen, and trackpad. The information processing apparatus according to claim 1.

4. The aforementioned operation is at least one of the following: clicking, scrolling, entering text, and deleting text. The information processing apparatus according to claim 1.

5. The display means causes the display mode of the identifier to differ depending on the operation content that the object can accept. The information processing apparatus according to claim 1.

6. The system further includes a learning means for learning images containing the object to be detected, which is specified in advance by the user. The detection means detects the object using a trained mathematical model that takes screen data as input data and outputs information about the position of the object contained in the screen data, based on the training data from the learning means. The information processing apparatus according to claim 1.

7. One or more information processing devices A first acquisition step involves acquiring screen data of the first screen displayed on the display device, A detection step to detect a specific object from the acquired screen data, The steps include assigning identification information to each of the detected objects, A display step in which the display device displays a second screen in which the identifier corresponding to the identification information assigned to each object is superimposed on the first screen at the position where the detected object is displayed on the first screen, The second acquisition step involves obtaining voice commands from the user, An extraction step for extracting identifiers and operation details included in the aforementioned voice instruction, An execution step which causes the extracted identifier to perform input processing based on the extracted operation content on the object corresponding to the extracted identifier, An information processing method that performs [this action].

8. One or more information processing devices, A first acquisition step involves acquiring screen data of the first screen displayed on the display device, A detection step to detect a specific object from the acquired screen data, The steps include assigning identification information to each of the detected objects, A display step in which the display device displays a second screen in which an identifier corresponding to the identification information assigned to each object is superimposed on the first screen at the position where the detected object is displayed on the first screen, The second acquisition step involves obtaining voice commands from the user, An extraction step for extracting identifiers and operation details included in the aforementioned voice instruction, An execution step which causes the extracted identifier to perform input processing based on the extracted operation content on the object corresponding to the extracted identifier, A program that executes something.

Images