Work support system, smart glasses, work support method and program

Abstract

To provide a work support system, smart glasses, a work support method and a program, capable of appropriately performing a work support.SOLUTION: A work support system uses smart glasses having an imaging unit and a gaze detection unit. The work support system includes: a storage part configured to store list information showing a plurality of objects to be confirmed and a state of confirmation of each of the objects; a selection part configured to select an object of a confirmation target from images taken by the imaging unit according to a visual line detected by the gaze detection unit; a determination part configured to determine whether or not the selected object matches any one of the objects shown in the list information according to the list information and an image recognition result to the selected object; and an update part configured to update a status of confirmation of the objects included in the list information when matched.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present disclosure relates to a work assistance system, smart glasses, a work assistance method, and a program. [Background technology]

[0002] Patent Document 1 describes a system that uses smart glasses to assist in the picking of items. In this system, picking is performed, for example, as follows: First, a picking instruction for the item to be picked is sent from a server to the smart glasses. Upon receiving the picking instruction, the smart glasses display a picking instruction screen on the display, superimposed on the real landscape, instructing the picking based on the information contained in the received picking instruction. The worker refers to the location information displayed on the picking instruction screen and moves to the shelf indicated by that location. After moving to the designated shelf, the smart glasses capture images of markers affixed to the items via a camera in response to changes in the worker's line of sight. The smart glasses then identify the item to which the marker is affixed based on the captured image of the marker. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 146959 Summary of the Invention [Problem to be solved by the invention]

[0004] In the system described in Patent Document 1, work assistance is provided by smart glasses based on work instructions that specify the work target received from a server. Therefore, there is a problem that, for example, when there are multiple work targets and the worker wants to work on the work targets in a desired order, the work assistance may not be provided appropriately.

[0005] The present disclosure aims to provide a work assistance system, smart glasses, a work assistance method, and a program that solve the above-mentioned problems. [Means for solving the problem]

[0006] The work assistance system of the present disclosure is a work assistance system using smart glasses having an image capturing unit and a line of sight detection unit, and includes: a memory unit that stores list information indicating multiple objects to be confirmed and the confirmation status of each of the objects; a selection unit that selects an object to be confirmed from an image captured by the image capturing unit based on the line of sight detected by the line of sight detection unit; a determination unit that determines whether the selected object matches any of the objects indicated in the list information based on the list information and the image recognition result for the selected object; and an update unit that updates the confirmation status of the object included in the list information if there is a match.

[0007] The smart glasses of the present disclosure include an image capturing unit, a line of sight detection unit, a memory unit that stores list information indicating multiple objects to be confirmed and the confirmation status of each of the objects, a selection unit that selects an object to be confirmed from an image captured by the image capturing unit based on the line of sight detected by the line of sight detection unit, a determination unit that determines whether the selected object matches any of the objects indicated in the list information based on the list information and image recognition results for the selected object, and an update unit that updates the confirmation status of the object included in the list information if there is a match.

[0008] The work assistance method disclosed herein includes the steps of using smart glasses having an image capturing unit and a line of sight detection unit to store list information indicating multiple objects to be confirmed and the confirmation status of each of the objects; selecting an object to be confirmed from an image captured by the image capturing unit based on the line of sight detected by the line of sight detection unit; determining whether the selected object matches any of the objects indicated in the list information based on the list information and the image recognition result for the selected object; and updating the confirmation status of the object included in the list information if there is a match.

[0009] The program disclosed herein uses smart glasses having an image capturing unit, a line of sight detection unit, and a computer to cause the computer to execute the following steps: storing list information indicating multiple objects to be confirmed and the confirmation status of each of the objects; selecting an object to be confirmed from an image captured by the image capturing unit based on the line of sight detected by the line of sight detection unit; determining, based on the list information and the image recognition result for the selected object, whether the selected object matches any of the objects indicated in the list information; and, if there is a match, updating the confirmation status of the object included in the list information. [Effects of the Invention]

[0010] The work assistance system, smart glasses, work assistance method, and program disclosed herein can provide appropriate work assistance. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram illustrating a configuration example of a work assistance system according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a schematic diagram illustrating an example of the configuration of a database according to an embodiment of the present disclosure. [Figure 3] 1 is a flowchart illustrating an example of work assistance using a work assistance system according to an embodiment of the present disclosure. [Figure 4] 10 is a flowchart illustrating an example of the operation of an information input controller according to an embodiment of the present disclosure. [Figure 5] FIG. 10 is a schematic diagram illustrating an example of a registration screen in an information input controller according to an embodiment of the present disclosure. [Figure 6] 10 is a flowchart illustrating an example of the operation of smart glasses according to an embodiment of the present disclosure. [Figure 7] 10A and 10B are schematic diagrams for explaining an example of the operation of smart glasses according to an embodiment of the present disclosure. [Figure 8]10 is a flowchart illustrating an example of the operation of the smart glasses and the information management server according to an embodiment of the present disclosure. [Figure 9] FIG. 10 is a schematic diagram illustrating an example of the configuration of a checklist according to an embodiment of the present disclosure. [Figure 10] 10A and 10B are schematic diagrams for explaining an example of the operation of smart glasses according to an embodiment of the present disclosure. [Figure 11] 10A and 10B are schematic diagrams for explaining an example of the operation of smart glasses according to an embodiment of the present disclosure. [Figure 12] FIG. 2 is a schematic diagram illustrating an example of registered content in a database according to an embodiment of the present disclosure. [Figure 13] FIG. 2 is a schematic diagram illustrating an example of registered content in a database according to an embodiment of the present disclosure. [Figure 14] 1 is a block diagram illustrating a configuration example of a work assistance system according to an embodiment of the present disclosure. [Figure 15] 10 is a flowchart illustrating an example of operation of the work assistance system according to an embodiment of the present disclosure. [Figure 16] FIG. 1 is a block diagram illustrating an example configuration of smart glasses according to an embodiment of the present disclosure. [Figure 17] FIG. 1 is a block diagram illustrating an example of a basic configuration of a computer according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In each drawing, the same or corresponding components are designated by the same reference numerals, and the description thereof will be omitted as appropriate.

[0013] First Embodiment Fig. 1 is a block diagram showing an example configuration of a work assistance system according to an embodiment of the present disclosure. The work assistance system 10 shown in Fig. 1 includes smart glasses 1, an information management server 2, and an information input controller 3. The smart glasses 1 are glasses-type wearable terminals (terminals worn on the body) that are worn on the head. The information management server 2 and the information input controller 3 are computers such as personal computers, tablet terminals, smartphones, etc.

[0014] In this embodiment, the work support system 10 is a system that supports a worker in performing a "check operation" on multiple "objects," such as materials, components, and articles used in a "predetermined task," before or after (or both before and after) the "predetermined task." In this embodiment, "objects," such as materials, components, or articles, include raw materials, parts, semi-finished products, finished products, products, jigs, tools, instruments, machinery, and equipment. The "check operation" is a process of using a checklist to confirm that each object listed on the checklist matches the object being checked. The check status includes whether the check operation has been completed or whether further check operation is required. While there are no limitations on the "predetermined task," in this embodiment, the "predetermined task" may, for example, be a component replacement for a server or an investigation into the cause of a failure. The location of the "predetermined task" is, for example, a facility, a server room, or an office. In this embodiment, examples of the "objects" to be checked include a personal computer, a memory, an HDD (hard disk drive), a motherboard, a Phillips head screwdriver, and the like.

[0015] The smart glasses 1 shown in FIG. 1 include a control device 11, an image capturing unit 12, an image display unit 13, and a gaze detection unit 14. The control device 11 can be configured using a computer such as a microcontroller, and includes the following functional blocks configured by a combination of hardware such as a computer and software such as a program executed by the computer. Specifically, the control device 11 includes, as functional blocks, a checklist acquisition unit 111, an image acquisition unit 112, a gaze acquisition unit 113, an image recognition unit 114, a selection unit 115, a determination unit 116, a checklist update unit 117, a highlight display unit 118, a display control unit 119, an image registration unit 120, a communication unit 121, and a memory unit 122. The memory unit 122 stores a checklist 123 and the like.

[0016] The image capturing unit 12 is an imaging device (camera) that captures still images or videos in the same direction as the line of sight of the worker wearing the smart glasses 1. The image display unit 13 is a display device (display) that uses the eyeglass-type lenses of the smart glasses 1 as a screen to display predetermined images such as still images and videos superimposed on an actual scene. The gaze detection unit 14 measures the gaze movement of the wearer of the smart glasses 1 using one or more cameras and LEDs (light-emitting diodes). The gaze detection unit 14 outputs the measured gaze movement, for example, as time-series data of coordinates in a predetermined coordinate system. The gaze detection unit 14 may also detect eyelid movement (for example, by detecting the occurrence of a blink) and output data indicating the detection result.

[0017] In the control device 11, the checklist acquisition unit 111 acquires a checklist, which is data (or a file) indicating multiple objects to be checked and the check status of each object, directly from the information management server 2 or via the information input controller 3. Here, "acquire" means, for example, receiving the checklist via a communication line such as mobile communication, local area wireless communication, or short-range wireless communication, or inputting the checklist via a communication cable, a removable recording medium, or the like. The checklist acquisition unit 111 stores the acquired checklist in the storage unit 122 as a checklist 123. The checklist 123 includes information indicating multiple objects to be checked and the check status of each object. The checklist 123 includes, for example, the object name, serial number, image information of the object, information encoded in a one-dimensional or two-dimensional barcode attached or printed on the object, the model of the object (item), information indicating a specified date and time related to the manufacture of the object, and other identification information, as well as information indicating the check status. The check status indicates the work status of the check work (such as checked, waiting for check, or no check is required). In addition, in the checklist 123, information indicating the confirmation status, such as confirmed or waiting for confirmation, is information that can be updated. Also, the checklist 123 is an example of "list information" according to the present disclosure. A detailed example of the checklist 123 will be described later.

[0018] The image acquisition unit 112 acquires still images and moving images (hereinafter, still images and moving images are also collectively referred to as images) captured by the image capturing unit 12, and stores them in the storage unit 122.

[0019] The gaze acquisition unit 113, for example, receives input of coordinates of the gaze detected by the gaze detection unit 14, converts them into pixel positions on the image captured by the image capture unit 12, and outputs the converted coordinates. The gaze acquisition unit 113 performs a conversion process using, for example, information for calibrating the relationship between the gaze coordinates output by the gaze detection unit 14 and the image captured by the image capture unit 12. The gaze acquisition unit 113 may also output information indicating that an operation on an icon (hereinafter also referred to as an operation by gaze input) has been performed in accordance with the movement of the gaze on an icon for input operation displayed on the image display unit 13, in cooperation with, for example, the display control unit 119. Note that a part or all of the gaze acquisition unit 113 may be included in the gaze detection unit 14 (or may be configured as a fixed entity together with the gaze detection unit 14).

[0020] The image recognition unit 114 performs a predetermined image recognition process on the image acquired by the image acquisition unit 112 and outputs the processing result as an image recognition result. The image recognition unit 114 performs image recognition processes, such as extracting image regions similar to a reference image or figure using pattern matching and converting characters, symbols, etc. included in the image into character data. For example, the image recognition unit 114 recognizes one or more objects included in the image and outputs image regions corresponding to each object as an image recognition result. Alternatively, the image recognition unit 114 compares the extracted image regions corresponding to the objects with image information included in the checklist 123, calculates the similarity (matching degree), and outputs the calculated result as an image recognition result. Alternatively, the image recognition unit 114 recognizes, for example, a serial number included in the image, converts it into character data, and outputs the calculated result as an image recognition result. Alternatively, the image recognition unit 114 recognizes information encoded in a one-dimensional or two-dimensional barcode, the model and name of an item, character information indicating a specified date and time, etc. included in the image acquired by the image acquisition unit 112, and outputs the recognized information as an image recognition result. Alternatively, the image recognition unit 114 outputs, as the image recognition result, for example, pixel positions corresponding to part or all of the outline of the area of ​​the image corresponding to the recognized object.

[0021] The selection unit 115 selects an object to be confirmed from the image captured by the image capturing unit 12 based on a pixel position corresponding to the line of sight output by the line of sight acquisition unit 113 based on the line of sight detected by the line of sight detection unit 14. In this case, the selection unit 115 selects an object to be confirmed from the image captured by the image capturing unit 12 based on the line of sight detected by the line of sight detection unit 14. For example, when the line of sight detected by the line of sight detection unit 14 stays on an object recognized by the image recognition unit 114 for a predetermined period of time or more, the selection unit 115 determines that the object is being gazed at (stared at), and selects the object as the object to be confirmed. Note that selection based on the line of sight is not limited to gaze, and may be based on, for example, a specific line of sight movement or a combination of the line of sight and the blinking state. The selection unit 115 outputs, for example, information indicating a position or area on the image corresponding to the selected object.

[0022] The determination unit 116 determines whether the selected object matches any of the objects to be checked indicated by the checklist 123, based on the information included in the checklist 123 and the image recognition results of the image recognition unit 114 for the object selected by the selection unit 115. The determination unit 116 determines whether there is a match based on, for example, a result of comparing the serial numbers of each object to be checked indicated by the checklist 123 with the serial numbers of the selected objects obtained as the image recognition results. Alternatively, the determination unit 116 determines whether there is a match based on, for example, a result of comparing image information of each object to be checked indicated by the checklist 123 with an image of an area corresponding to the selected object obtained as the image recognition results. Alternatively, the determination unit 116 determines whether there is a match based on, for example, a result of comparing information encoded in a one-dimensional or two-dimensional barcode, character information indicating the model, name, location, specified date and time, or other identification information obtained as the image recognition results, with the corresponding information regarding each object to be checked indicated by the checklist 123.

[0023] The checklist update unit 117 updates information indicating the confirmation status in the checklist 123 for an object that the determination unit 116 has determined to be a match. For example, the checklist update unit 117 updates the confirmation status of the object in the checklist 123 from "waiting for confirmation" to "confirmed" ("confirmed"). The checklist update unit 117 is an example of an "update unit" according to the present disclosure.

[0024] If the object selected by the selection unit 115 does not match any of the objects to be confirmed indicated in the checklist 123, that is, if the object selected by the selection unit 115 is not registered in the checklist 123, the highlight display unit 118 highlights (also referred to as highlighting) the selected object using the image display unit 13. Hereinafter, an object that is not registered will also be referred to as a non-target object. Highlighting is, for example, a display in which the display color of all or part of the outline of the selected object is changed to another color, a line surrounding the outline of the selected object is displayed in a predetermined color, these displays are blinked, or one or more figures are displayed on the outline. Alternatively, in addition to the display by the highlight display unit 118, the image display unit 13 may display text information or the like indicating that the object is not a target object for confirmation.

[0025] The display control unit 119 controls the display of predetermined characters and figures on the image display unit 13. For example, when the object selected by the selection unit 115 does not match any of the objects to be checked indicated in the checklist 123, the display control unit 119 displays some or all of the information registered in the checklist 123 on the image display unit 13. For example, the display control unit 119 displays some or all of the names of the objects included in the checklist 123 on the image display unit 13, or displays some or all of the names of the objects included in the checklist 123 whose check status information is, for example, "waiting for check," on the image display unit 13. Furthermore, the display control unit 119 may hide the information of the checklist 123 that was being displayed, for example, when the object selected by the selection unit 115 matches any of the objects to be checked indicated in the checklist 123.

[0026] Furthermore, when there is an object to be confirmed indicated in the checklist 123 that has not been determined to match the selected object and no selection has been made by the selection unit 115 for a predetermined time or longer, the display control unit 119 displays predetermined information on the image display unit 13. That is, when the checklist 123 includes an object to be confirmed that has not been determined to match the object selected by the selection unit 115 and no object to be confirmed is selected by the selection unit 115 for a predetermined time or longer, the display control unit 119 displays predetermined information on the image display unit 13. For example, when an object that has not been "confirmed" remains in the checklist 123 and no new object to be confirmed is selected by the selection unit 115 for a predetermined time or longer, the display control unit 119 displays text information such as "There is an unconfirmed object" on the image display unit 13.

[0027] Furthermore, the display control unit 119 can display on the image display unit 13 an image such as a figure corresponding to the line of sight output by the line of sight acquisition unit 113.

[0028] The image registration unit 120 captures an image of an object to be registered in the checklist 123 using the image capturing unit 12 in response to an operation by the worker using a predetermined line-of-sight input, and stores the image data representing the captured image and the results of image recognition on the captured image as image information in the storage unit 122, the storage unit 33 in the information input controller 3, etc. The results of image recognition are, for example, information that can be used as a comparison standard in pattern matching of image features obtained when the captured image is recognized by the image recognition unit 114, etc.

[0029] The communication unit 121 transmits and receives predetermined information to and from the information input controller 3 and the information management server 2 via wireless communication via a wide area communication network, a mobile communication network, an in-house communication network, etc., or wireless communication via short-range wireless communication.

[0030] The storage unit 122 stores information such as a checklist 123 and image information based on the image captured by the image capturing unit 12 .

[0031] The information management server 2 also includes an information management unit 21, a database 22, and a communication unit 23 as functional blocks configured from a combination of hardware such as a computer and software such as a program executed by the computer. The information management unit 21 registers information in the database 22 in response to requests from the smart glasses 1 or the information input controller 3, updates the information registered in the database 22, and creates the above-mentioned checklist based on the information registered in the database 22 and transmits it to the smart glasses 1, etc. In this embodiment, the process performed by the information management unit 21 when performing a confirmation task on the smart glasses 1 is referred to as a confirmation process (or a carry-in confirmation process or a take-out confirmation process).

[0032] The database 22 stores, for example, a plurality of tables 221 as shown in FIG. 2 for each "predetermined task." FIG. 2 is a schematic diagram showing an example configuration of the database 22 according to an embodiment of the present disclosure. The table 221 shown in FIG. 2 includes a table 222 and a table 223. The table 222 is a table that stores basic information about the "predetermined tasks" set in the table 221, and includes the worker ID (worker identification information), the work date and time, the work location, and the update date and time of the table 221. Note that, for example, there may be multiple worker IDs. Furthermore, the work date and time may be only the work day without the hour and minute, or the work period may include multiple work days.

[0033] Table 223 includes multiple records R corresponding to objects (hereinafter also referred to as components or equipment) used in the work defined in table 222. Each record R includes multiple fields F indicating an item number, equipment name, serial number, image information, work name, work ID, bring-in status, and take-out status. The item number is identification information for each record R. The equipment name is the name of the component. The serial number is a number (number or code) that uniquely identifies each component. The image information includes image data representing one or more images of the component, information indicating the storage location of each image data, information indicating the image recognition results for each image, etc. The work name indicates the content of the work for each component. The work ID is identification information for distinguishing the relationship between a component and a work from the relationship between other components and a work. For example, even if the work name is the same, if the work ID is different, the components used for that work name will be different. In the example shown in FIG. 2 , for example, the memory with item number A006 is the component to be removed from the device, and the memory with item number A007 is the component to be attached to the device in exchange for the memory with item number A006. The carry-in status indicates whether or not confirmation work is required before entering a work location, for example, before entering a building or room, and if so, whether or not the confirmation work has been completed. In the example shown in FIG. 2 , “waiting for confirmation” indicates a state in which confirmation work is required but has not been completed. “Not brought in” indicates a state in which confirmation work is not required. Note that in this embodiment, once confirmation work is completed, the carry-in status becomes “bring-in confirmation OK.” The carry-out status indicates whether or not confirmation work is required after leaving a work location, for example, after leaving a building or room, and if so, whether or not the confirmation work has been completed. In the example shown in FIG. 2 , “waiting for confirmation” indicates a state in which confirmation work is required but has not been completed. “Not brought out” indicates a state in which confirmation work is not required. Note that in this embodiment, once confirmation work is completed, the carry-out status becomes “bring-in confirmation OK.”

[0034] The communication unit 23 transmits and receives predetermined information to and from the smart glasses 1 and the information input controller 3 by wired or wireless communication via a wide area communication network, a mobile communication network, an in-house communication network, or the like.

[0035] The information input controller 3 also includes an information registration unit 31, an input / output unit 32, a storage unit 33, and a communication unit 34 as functional blocks configured by a combination of hardware such as a computer and software such as a program executed by the computer.

[0036] The information registration unit 31 uses the input / output unit 32 to input information to be registered in the database 22 based on an input operation by the worker.

[0037] The input / output unit 32 inputs and outputs information using input devices such as a keyboard, a mouse, and a touch sensor, and output devices such as a display device.

[0038] The storage unit 33 stores, for example, information input from the input / output unit 32, information received from the smart glasses 1, and the like.

[0039] The communication unit 34 transmits and receives predetermined information to and from the smart glasses 1 and the information management server 2 via wired or wireless communication via a wide area communication network, a mobile communication network, an in-house communication network, etc., or via short-range wireless communication, etc.

[0040] Next, a basic flow of work support according to this embodiment will be described with reference to FIG. 3. FIG. 3 is a flowchart showing an example of work support using the work support system 10 according to the embodiment of the present disclosure. As shown in FIG. 3, in work support using the work support system 10, first, materials to be brought into a work location where a target "predetermined work" is to be performed and materials to be taken out from the work location are registered in the database 22 (step S1). Next, the brought-in materials are confirmed using the smart glasses 1 (step S2). Next, the worker performs the "predetermined work" (step S3). Next, the taken-out materials are confirmed using the smart glasses 1 (step S4). Note that, for example, the worker who performs the registration work in step S1 may or may not be the same person as the worker who performs the confirmation work in step S2 or step S3. Also, one of step S2 and step S4 may be omitted.

[0041] Next, with reference to FIGS. 4 to 7, an example of operation when the brought-in members and the taken-out members are registered in the database 22 in step S1 of FIG. 3 in the work support system 10 will be described. FIG. 4 is a flowchart showing an example of operation of the information input controller according to an embodiment of the present disclosure. FIG. 5 is a schematic diagram showing an example of a registration screen in the information input controller according to an embodiment of the present disclosure. FIG. 6 is a flowchart showing an example of operation of smart glasses according to an embodiment of the present disclosure. FIG. 7 is a schematic diagram for explaining an example of operation of smart glasses according to an embodiment of the present disclosure.

[0042] In this operation example, as shown in FIG. 4, each piece of data is input by the information input controller 3 on the registration screen D1 shown in FIG. 5 (step S11). Note that in step S11, image information to be registered in the database 22 is acquired using the smart glasses 1, and the image information is stored in a predetermined storage destination of, for example, the storage unit 33 of the information input controller 3. Next, the storage destination (attached image) (input area D109) of the image information is input by the information input controller 3 on the registration screen D1 shown in FIG. 5. Then, each piece of data input by the information input controller 3 is registered in the database 22 (step S12). Then, the processes of steps S11 and S12 are repeated until all components used in the "predetermined work" are registered (until step S13: YES). Note that the acquisition and input of image information using the smart glasses 1 may be performed for each component collectively, for example, after information other than image information has been input for all components. Furthermore, when inputting the serial number using the registration screen D1, the serial number recognized from the image captured using the smart glasses 1 may be automatically input when the image information is input. Furthermore, when a completion button or the like (not shown) is pressed (clicked), the registration process is completed (step S13: YES).

[0043] 5 is displayed on the input / output unit 32 after a predetermined application software is started by the information input controller 3 and a predetermined input operation is performed on the input / output unit 32. The registration screen D1 includes an input area D101 for the work date and time, an input area D102 for the work location, an input area D103 for the worker ID, a display area D104 for the item number, an input area D105 for the device name, an input area D106 for the serial number, an input area D107 for the work name, an input area D108 for the work ID, an input area D109 for the storage location (folder name, file name, etc.) of the attached image, a setting area D110 for whether to perform a confirmation operation for either or both of take-out and take-in, a data input button D111, a data registration button D112, and a display area D113 for registration decision. The item number display area D104 displays the item number automatically assigned to the next component when the input button D111 is pressed. In the registration decision display area D113, when the registration button D112 is pressed, text information or the like indicating whether or not the data has been successfully registered in the database 22 is displayed.

[0044] Next, an example of the operation of the smart glasses 1 when using the smart glasses 1 to acquire image information from the database 22 will be described with reference to Figures 6 and 7. The process shown in Figure 6 starts when a predetermined application software is launched on the smart glasses 1 and an input operation is performed, for example, by a predetermined eye-gaze input.

[0045] In the process shown in Fig. 6, the image registration unit 120 shown in Fig. 1 determines whether or not the operator has performed a predetermined operation by eye-gaze input to end the registration (step S21). If the operation to end the registration has been performed (step S21: YES), the image registration unit 120 ends the process shown in Fig. 6.

[0046] If an operation to terminate registration has not been performed (step S21: NO), the image registration unit 120 determines whether or not the object recognized by the image recognition unit 114 has been gazed at, based on the pixel position corresponding to the gaze output by the gaze acquisition unit 113 based on the gaze detected by the gaze detection unit 14 (step S22). If the object has not been gazed at (step S22: YES), the image registration unit 120 executes the processes from step S21 onwards again. If the object has been gazed at (step S22: YES), the image registration unit 120 acquires the image captured by the image capturing unit 12 using the image acquisition unit 112 (step S23). Next, the image registration unit 120 causes the image recognition unit 114 to execute image recognition processing to recognize the serial number from the image acquired by the image acquisition unit 112 (step S24). Next, the image registration unit 120 determines whether or not the serial number has been recognized (step S25). If the serial number is recognized (step S25: YES), the image registration unit 120 links the image data representing the captured image to the serial number and stores it in the memory unit 122 or the memory unit 33 in the information input controller 3 (step S26), and then executes the processing from step S21 onwards again.

[0047] On the other hand, if the serial number has not been recognized (step S25: NO), the image registration unit 120 determines whether or not recognition of the serial number is necessary (step S27). In step S27, the image registration unit 120, for example, requests an input operation regarding whether recognition is necessary, and determines whether or not recognition of the serial number is necessary based on the input operation. If recognition of the serial number is not necessary (step S27: NO), the image registration unit 120 stores image data representing the captured image in the storage unit 122, the storage unit 33 in the information input controller 3, or the like (step S28), and executes the processes from step S21 onwards again. If recognition of the serial number is necessary (step S27: YES), the image registration unit 120 executes the processes from step S21 onwards again.

[0048] FIG. 7 schematically illustrates an example of the state of the field of view of the worker P1 in the process illustrated in FIG. 6. In the real space RS1, there is a worker P1 wearing smart glasses 1, and objects OBJ1, OBJ2, and OBJ3. The line of sight LS of the worker P1 is directed toward the object OBJ1. The field of view ES1 of the worker P1 includes objects OBJ1 to OBJ3 and images D21, D22, and D23 displayed on the image display unit 13. In this example, image D21 represents text information indicating that component registration is in progress. Image D22 represents a list including the equipment names and serial numbers of components whose images have been saved. Image D23 represents a dashed circle indicating the line of sight of the worker P1. The serial number is information recognized in step S24, and the equipment name is information input by the information input controller 3 in association with the recognized serial number.

[0049] Next, with reference to FIGS. 8 to 13, an example of operation when checking brought-in or taken-out materials in step S2 or step S4 of FIG. 3 in the work support system 10 will be described. FIG. 8 is a flowchart showing an example of operation of smart glasses and an information management server according to an embodiment of the present disclosure. FIG. 9 is a schematic diagram showing an example of the configuration of a checklist according to an embodiment of the present disclosure. FIGS. 10 and 11 are schematic diagrams for explaining an example of operation of smart glasses according to an embodiment of the present disclosure. FIGS. 12 and 13 are schematic diagrams showing examples of registered contents of a database according to an embodiment of the present disclosure.

[0050] The process shown in FIG. 8 is started when a predetermined application software is launched on the smart glasses 1 and an input operation is performed, for example, by a predetermined eye-gaze input. In the process shown in FIG. 8, when the smart glasses 1 receive an input operation from the worker instructing the start of the check work for brought-in materials or the check work for brought-out materials, the checklist acquisition unit 111, for example, notifies the information management server 2 of the start of the check work (step S101). Note that in step S101, the check work is started by specifying whether it is the check work for brought-out materials or the check work for brought-out materials. In the information management server 2, the information management unit 21 starts the check process (step S201), identifies the "predetermined work" based on the worker ID, work date and time, work location, etc., creates a checklist corresponding to the checklist 123 shown in FIG. 9 based on the registered information in the database 22, and transmits it to the smart glasses 1 (step S202). In the smart glasses 1, the checklist acquisition unit 111 receives the transmitted checklist and stores it in the storage unit 122 as the checklist 123 (step S102).

[0051] Here, an example of the configuration of checklist 123 will be described with reference to FIG. 9. FIG. 9 shows an example of the configuration of checklist 123 as initially stored in memory unit 122 when carrying out the confirmation work of brought-in materials. Checklist 123 shown in FIG. 9 includes table 124 and table 125. Checklist 123 has the same configuration as table 221 shown in FIG. 2, with some exceptions. Table 124 shown in FIG. 9 has the same configuration as table 222 shown in FIG. 2. Table 125 shown in FIG. 9 has a configuration obtained by omitting field F for the job ID from table 223 shown in FIG. 2. However, table 125 shown in FIG. 9 may also have the same configuration as table 223 shown in FIG. 2. Note that in this embodiment, a common checklist 123 is used for carry-in confirmation and carry-out confirmation, but separate checklists for carry-in confirmation and carry-out confirmation may also be prepared.

[0052] 8, next, the display control unit 119 determines whether or not a no-operation state has continued for a predetermined time or more (step S103). If the no-operation state has not continued for the predetermined time or more (step S103: NO), the display control unit 119 determines whether or not a confirmation work completion button displayed by the image display unit 13 has been pressed, based on the output of the line-of-sight acquisition unit 113 (step S104). If the confirmation work completion button has not been pressed (step S104: NO), the selection unit 115 determines whether or not the object has been gazed at (step S105).

[0053] If the object is gazed at (step S105: YES), the image acquisition unit 112 acquires an image (step S106), and the image recognition unit 114 performs image recognition processing on an area in the image corresponding to the selected object (step S107). Next, the determination unit 116 determines whether the selected object is on the checklist 123 (step S108). If the selected object is included in the checklist 123 (step S108: YES), the checklist update unit 117 updates the carry-in status or carry-out status from "waiting for confirmation" to "bring-in confirmation OK" or "bring-out confirmation OK" (step S109). Note that if the carry-in status or carry-out status has already been updated, the content is not changed. On the other hand, if the selected object is not included in the checklist 123 (step S108: NO), the highlight display unit 118 highlights the non-target object (step S110).

[0054] After step S109 or step S110, or if the object is not being gazed at (step S105: NO), the processes from step S103 onwards are executed again.

[0055] On the other hand, if the no-operation state continues for a predetermined time or longer in step S103 (step S103: YES), or if the confirmation work completion button is pressed in step S104 (step S104: YES), the display control unit 119 determines whether or not there are any unconfirmed objects remaining in the checklist 123 (step S111), and if there are (step S111: YES), displays some or all of the contents of the checklist 123 on the image display unit 13 (step S112). After step S112, the processes from step S103 onwards are executed again.

[0056] Also, if there are no items remaining (step S111: NO), the communication unit 121 notifies the user to update the database (notifies the user to update the carry-in status or carry-out status) (step S113), and then notifies the user that the confirmation work has been completed (step S114), and ends the processing in the smart glasses 1.

[0057] When the information management server 2 receives the database update notification, it updates the corresponding information in the database 22 (step S203), and when it receives the confirmation work completion notification, it ends the confirmation process (step S204).

[0058] FIG. 10 schematically illustrates an example of the state of the field of view of worker P1 in the process shown in FIG. 8. In real space RS1, there is worker P1 wearing smart glasses 1, and objects OBJ1, OBJ2, and OBJ3. The line of sight LS of worker P1 is directed toward object OBJ1. In the field of view ES2 of worker P1, there are objects OBJ1 to OBJ3, and images D31, D32, and D33 displayed on image display unit 13. In this case, image D31 displays text information indicating that carry-in confirmation is in progress. Image D32 displays a confirmation work completion button. Image D33 displays a dashed circle indicating the line of sight of worker P1.

[0059] FIG. 11 schematically illustrates another example of the state of the field of view of the worker P1 in the process illustrated in FIG. 8. In the real space RS1, there are a worker P1 wearing smart glasses 1, an object OBJ1, an object OBJ2, and a non-target object OBJ4. The line of sight LS of the worker P1 is directed toward the non-target object OBJ4. In the field of view ES3 of the worker P1, there are objects OBJ1 and OBJ2, the non-target object OBJ4, and images D31, D32, D34, and D35 displayed on the image display unit 13. In this case, image D31 displays text information indicating that carry-in confirmation is in progress. Image D32 displays a confirmation work completion button. Image D34 is an image highlighted in step S110. Image D35 is an image showing part of the contents of the checklist 123 displayed in step S111.

[0060] Fig. 12 shows an example of the contents of database 22 updated in step S203 in the bring-in confirmation process. In Fig. 12, in table 223a corresponding to table 223 shown in Fig. 7, the bring-in status "waiting confirmation" has been updated to "bring-in confirmation OK". Also, Fig. 13 shows an example of the contents of database 22 updated in step S203 in the take-out confirmation process. In table 223b shown in Fig. 13 corresponding to table 223a shown in Fig. 12, the bring-out status "waiting confirmation" has been updated to "bring-out confirmation OK".

[0061] As described above, the work assistance system 10 of this embodiment is a work assistance system using smart glasses 1 having an image capturing unit 12 and a line-of-sight detection unit 14, and includes a memory unit 122, a selection unit 115, a determination unit 116, and a checklist update unit 117. The memory unit 122 stores a checklist 123 indicating multiple objects to be confirmed and the confirmation status of each object. The selection unit 115 selects an object to be confirmed from an image captured by the image capturing unit 12 based on the line of sight detected by the line-of-sight detection unit 14. The determination unit 116 determines whether the selected object matches any of the objects indicated in the checklist 123 based on information included in the checklist 123 and the image recognition result of the image recognition unit 114 for the selected object. If there is a match, the checklist update unit 117 updates the confirmation status of the object included in the checklist 123. This configuration allows for appropriate support of the confirmation work, even when there are multiple confirmation targets and the worker performs the confirmation work on the confirmation targets in any order. That is, according to this embodiment, work support can be performed appropriately.

[0062] In this embodiment, the smart glasses 1 also have an image display unit 13 that displays a predetermined image superimposed on an actual scene. The work assistance system 10 also has a highlight display unit that highlights the selected object on the image display unit 13 when the object selected by the selection unit 115 does not match the object indicated by the checklist 123. This configuration makes it possible to clearly indicate that the selected object is not a target object.

[0063] Furthermore, the work assistance system 10 of this embodiment includes a display control unit 119 that displays part or all of the checklist 123 on the image display unit 13 when the object selected by the selection unit 115 does not match the object indicated by the checklist 123. With this configuration, when a non-target object is selected, for example, information about an unconfirmed object (or a confirmed object) can be presented.

[0064] Furthermore, when there is an object indicated by the checklist 123 that has not been determined to match the object selected by the selection unit 115, and no selection by the selection unit 115 has been made for a predetermined time or longer, the display control unit 119 displays predetermined information on the image display unit 13. According to this configuration, when no object is selected for a predetermined time, it is possible to present information indicating, for example, that there is an unidentified object and what the unidentified object is.

[0065] If the checklist 123 includes information indicating the serial number of the object to be checked, the determination unit 116 can determine whether or not there is a match based on the result of comparing the serial number of the object indicated by the checklist 123 with the serial number of the selected object obtained as the image recognition result. With this configuration, it is possible to determine whether or not the objects match through a relatively simple image recognition process.

[0066] In this embodiment, the checklist 123 includes at least one of information indicating each object that should be checked before a predetermined task and information indicating each object that should be checked after the predetermined task. With this configuration, it is possible to check brought-in or taken-out materials before, after, or both before and after the predetermined task.

[0067] Furthermore, the checklist 123 can include image information showing each image of each object. With this configuration, for example, even objects that do not have serial numbers can be checked.

[0068] In this embodiment, the smart glasses 1 are provided with all of the components such as the image capturing unit 12, the line of sight detection unit 14, the memory unit 122, the selection unit 115, the determination unit 116, the checklist update unit 117, and the image recognition unit 114, but some of the functional blocks provided in the smart glasses 1, for example, may be provided in the information input controller 3, the information management server 2, etc., or may be distributed across the information input controller 3, the information management server 2, etc. However, by providing the image capturing unit 12, the line of sight detection unit 14, the memory unit 122 storing the checklist 123, the selection unit 115, the determination unit 116, and the checklist update unit 117 in the smart glasses 1, the checklist 123 can be updated by the smart glasses 1 alone, for example, based on the confirmation results.

[0069] Second Embodiment Next, a second embodiment of the present disclosure will be described with reference to FIGS. 14 and 15. FIG. 14 is a block diagram illustrating a configuration example of a work support system according to an embodiment of the present disclosure. FIG. 15 is a flowchart illustrating an operation example of the work support system according to an embodiment of the present disclosure. The work support system 100 according to an embodiment of the present disclosure illustrated in FIG. 14 is a work support system 100 using smart glasses having an image capturing unit and a line of sight detection unit, and includes a storage unit 101, a selection unit 102, a determination unit 103, and an update unit 104. The storage unit 101 stores list information indicating multiple objects to be confirmed and the confirmation status of each object. The selection unit 102 selects an object to be confirmed from an image captured by the image capturing unit based on the line of sight detected by the line of sight detection unit. The determination unit 103 determines whether the selected object matches any of the objects indicated in the list information based on the list information and the image recognition result for the selected object. If there is a match, the update unit 104 updates the confirmation status of the object included in the list information.

[0070] 15, the work assistance method according to the embodiment of the present disclosure includes steps S301 to S304, which are executed using smart glasses having an image capturing unit and a line-of-sight detection unit. In step S301, list information indicating multiple objects to be confirmed and the confirmation status of each object is stored. In step S302, an object to be confirmed is selected from an image captured by the image capturing unit based on the line of sight detected by the line-of-sight detection unit. In step S303, based on the list information and the image recognition result for the selected object, it is determined whether the selected object matches any of the objects indicated in the list information. In step S304, if there is a match, the confirmation status of the object included in the list information is updated.

[0071] <Third embodiment> Next, a third embodiment of the present disclosure will be described with reference to FIG. 16 . FIG. 16 is a block diagram illustrating an example configuration of smart glasses according to an embodiment of the present disclosure. The smart glasses 200 illustrated in FIG. 16 include an image capturing unit 201, a line-of-sight detection unit 202, a storage unit 203, a selection unit 204, a determination unit 205, and an update unit 206. The storage unit 203 stores list information indicating multiple objects to be confirmed and the confirmation status of each object. The selection unit 204 selects an object to be confirmed from the image captured by the image capturing unit 201 based on the line of sight detected by the line-of-sight detection unit 202. The determination unit 103 determines whether the selected object matches any of the objects indicated in the list information based on the list information and the image recognition result for the selected object. If there is a match, the update unit 104 updates the confirmation status of the object included in the list information.

[0072] <Computer configuration> FIG. 17 is a block diagram showing an example of a basic configuration of a computer according to an embodiment of the present disclosure. The computer 700 includes a CPU (Central Processing Unit) 710, a main memory device 720, an auxiliary memory device 730, and an interface 740. For example, a non-volatile memory medium 750 is connected to the interface 740. The smart glasses 1, the control device 11, the information management server 2, and the information input controller 3 described above are implemented in the computer 700. The operations of each of the processing units described above are stored in the auxiliary memory device 730 in the form of a program. The CPU 710 reads the program from the auxiliary memory device 730, loads it into the main memory device 720, and executes the above-described processing in accordance with the program. The CPU 710 also allocates storage areas in the main memory device 720 corresponding to each of the storage units described above in accordance with the program.

[0073] The program may be for realizing some of the functions to be performed by the computer 700. For example, the program may be combined with other programs already stored in the auxiliary storage device 730 or other programs implemented in other devices to perform the functions. In other embodiments, the computer may include a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to or instead of the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, some or all of the functions realized by the CPU 710 may be realized by the integrated circuit.

[0074] Examples of auxiliary storage device 730 include a hard disk drive (HDD), a solid state drive (SSD), a magnetic disk, a magneto-optical disk, a compact disc read-only memory (CD-ROM), a digital versatile disc read-only memory (DVD-ROM), and semiconductor memory. Auxiliary storage device 730 may be internal media directly connected to the bus of computer 700, or may be external media connected to computer 700 via interface 740 or a communication line. Furthermore, if this program is distributed to computer 700 via a communication line, computer 700 that receives the program may load the program into main storage device 720 and execute the above-described processing. In at least one embodiment, auxiliary storage device 730 is a non-transitory tangible storage medium.

[0075] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications within the scope of the present disclosure that would be understood by those skilled in the art are possible in the configuration and details of the present disclosure. For example, in the above-described embodiments, an example was given in which the work assistance system according to the present disclosure is applied to the task of checking objects brought into or taken out of a server room before or after work performed in the server room. However, the work assistance system according to the present disclosure may also be applied to, for example, the task of checking packages involved in the packing of home delivery items, the task of checking deliveries involved in delivery work, or the task of checking items involved in food service work at restaurants. Each embodiment can be combined with other embodiments as appropriate. Furthermore, part or all of the program executed by the computer in the above-described embodiments can be distributed via a computer-readable recording medium or a communication line.

[0076] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.

[0077] (Appendix 1) A work assistance system using smart glasses having an image capturing unit and a line of sight detection unit, a storage unit that stores list information indicating a plurality of objects to be confirmed and the confirmation status of each of the objects; a selection unit that selects an object to be confirmed from the image captured by the image capturing unit based on the line of sight detected by the line of sight detection unit; a determination unit that determines whether the selected object matches any of the objects indicated in the list information based on the list information and an image recognition result for the selected object; an update unit that updates the confirmation status of the object included in the list information when the confirmation status matches; A work support system comprising:

[0078] (Appendix 2) The smart glasses have an image display unit that displays a predetermined image superimposed on an actual scene, a highlight display unit that displays the selected object in an emphasized manner on the image display unit when the selected object does not match the object indicated by the list information; The work support system according to claim 1 further comprises:

[0079] (Appendix 3) In Supplementary Note 1, the smart glasses have an image display unit that displays a predetermined image superimposed on an actual scene, a display control unit that displays a part or all of the list information on the image display unit when the selected object does not match the object indicated by the list information; The work support system according to claim 1 or 2 further comprises:

[0080] (Appendix 4) In Supplementary Note 1 and 2, the display control unit may further include a display control unit that displays a part or all of the list information on the image display unit when the selected object does not match the object indicated by the list information, When the objects indicated by the list information include objects that have not been determined to match the selected object, and when the selection by the selection unit has not been made for a predetermined time or longer, the display control unit displays predetermined information on the image display unit. 4. A work support system according to any one of appendices 1 to 3.

[0081] (Appendix 5) the list information includes information indicating the serial number of the object to be confirmed; The determination unit determines whether or not the serial numbers of the objects indicated by the list information match based on a result of comparing the serial numbers of the selected objects obtained as a result of the image recognition. 5. A work support system according to any one of appendices 1 to 4.

[0082] (Appendix 6) The list information includes at least one of information indicating each of the objects to be checked before a predetermined task and information indicating each of the objects to be checked after the task. 6. A work support system according to any one of appendices 1 to 5.

[0083] (Appendix 7) The list information includes image information indicating each image obtained by capturing each of the objects. 7. A work support system according to any one of appendices 1 to 6.

[0084] (Appendix 8) an image capturing unit; A gaze detection unit; a storage unit that stores list information indicating a plurality of objects to be confirmed and the confirmation status of each of the objects; a selection unit that selects an object to be confirmed from the image captured by the image capturing unit based on the line of sight detected by the line of sight detection unit; a determination unit that determines whether the selected object matches any of the objects indicated in the list information based on the list information and an image recognition result for the selected object; an update unit that updates the confirmation status of the object included in the list information when the confirmation status matches; Smart glasses equipped with

[0085] (Appendix 9) an image display unit that displays a predetermined image superimposed on an actual scene; a highlight display unit that, when the selected object does not match the object indicated by the list information, highlights the selected object using the image display unit; Further comprising the smart glasses according to appendix 8.

[0086] (Appendix 10) In Supplementary Note 8, an image display unit is further provided that displays a predetermined image superimposed on an actual scene, a display control unit that displays a part or all of the list information on the image display unit when the selected object does not match the object indicated by the list information; The smart glasses according to claim 8 or 9, further comprising:

[0087] (Appendix 11) In Supplementary Note 8 and Supplementary Note 9, a display control unit is provided that, when the selected object does not match the object indicated by the list information, displays part or all of the list information on the image display unit; When the objects indicated by the list information include objects that have not been determined to match the selected object, and when the selection by the selection unit has not been made for a predetermined time or longer, the display control unit displays predetermined information on the image display unit. Attachment 8. The smart glasses according to any one of attachments 8 to 10.

[0088] (Appendix 12) the list information includes information indicating the serial number of the object to be confirmed; The determination unit determines whether or not the serial numbers of the objects indicated by the list information match based on a result of comparing the serial numbers of the selected objects obtained as a result of the image recognition. 12. The smart glasses according to any one of appendices 8 to 11.

[0089] (Appendix 13) The list information includes at least one of information indicating each of the objects to be checked before a predetermined task and information indicating each of the objects to be checked after the task. Attachment 8-12. The smart glasses according to any one of claims 8 to 12.

[0090] (Appendix 14) The list information includes image information indicating each image obtained by capturing each of the objects. Attachment 8-13: The smart glasses according to any one of claims 8 to 13.

[0091] (Appendix 15) Using smart glasses having an image capturing unit and a line of sight detection unit, storing a list of objects to be identified and the status of identification of each object; selecting an object to be confirmed from the image captured by the image capturing unit based on the line of sight detected by the line of sight detection unit; determining whether the selected object matches any of the objects indicated in the list information based on the list information and an image recognition result for the selected object; If there is a match, updating the confirmation status of the object contained in the inventory information; A work support method including:

[0092] (Appendix 16) The smart glasses have an image display unit that displays a predetermined image superimposed on an actual scene, a step of highlighting the selected object by the image display unit when the selected object does not match the object indicated by the list information; The work assistance method according to claim 15, further comprising:

[0093] (Appendix 17) In Supplementary Note 15, the smart glasses have an image display unit that displays a predetermined image superimposed on an actual scene, a step of displaying a part or all of the list information on the image display unit when the selected object does not match the object indicated by the list information; The work assistance method according to claim 15 or 16, further comprising:

[0094] (Appendix 18) a step of displaying predetermined information on the image display unit when the selection has not been made for a predetermined time or longer in the case where the objects indicated by the list information include those that have not been determined to match the selected object. The work support method according to any one of appendices 15 to 17, further comprising:

[0095] (Appendix 19) the list information includes information indicating the serial number of the object to be confirmed; In the determining step, the serial numbers of the objects indicated by the list information are compared with the serial numbers of the selected objects obtained as a result of the image recognition, and the determination as to whether or not they match is made based on the result. A work assistance method according to any one of appendices 15 to 18.

[0096] (Appendix 20) The list information includes at least one of information indicating each of the objects to be checked before a predetermined task and information indicating each of the objects to be checked after the task. 20. A work assistance method according to any one of appendices 15 to 19.

[0097] (Appendix 21) The list information includes image information indicating each image obtained by capturing each of the objects. A work assistance method according to any one of appendices 15 to 20.

[0098] (Appendix 22) Using smart glasses having an image capturing unit, a line of sight detection unit, and a computer, storing a list of objects to be identified and the status of identification of each object; selecting an object to be confirmed from the image captured by the image capturing unit based on the line of sight detected by the line of sight detection unit; determining whether the selected object matches any of the objects indicated in the list information based on the list information and an image recognition result for the selected object; If there is a match, updating the confirmation status of the object contained in the inventory information; A program that causes a computer to execute the following.

[0099] (Appendix 23) The smart glasses have an image display unit that displays a predetermined image superimposed on an actual scene, a step of highlighting the selected object by the image display unit when the selected object does not match the object indicated by the list information; 23. The program according to claim 22, further executed.

[0100] (Appendix 24) In Supplementary Note 15, the smart glasses have an image display unit that displays a predetermined image superimposed on an actual scene, a step of displaying a part or all of the list information on the image display unit when the selected object does not match the object indicated by the list information; 24. The program according to claim 22 or 23, further executed.

[0101] (Appendix 25) a step of displaying predetermined information on the image display unit when the selection has not been made for a predetermined time or longer in the case where the objects indicated by the list information include those that have not been determined to match the selected object. The program according to any one of appendices 22 to 24, which is further executed.

[0102] (Appendix 26) the list information includes information indicating the serial number of the object to be confirmed; In the determining step, the serial numbers of the objects indicated by the list information are compared with the serial numbers of the selected objects obtained as a result of the image recognition, and the determination as to whether or not they match is made based on the result. A program according to any one of appendices 22 to 25.

[0103] (Appendix 27) The list information includes at least one of information indicating each of the objects to be checked before a predetermined task and information indicating each of the objects to be checked after the task. A program according to any one of Appendices 22 to 26.

[0104] (Appendix 28) The list information includes image information indicating each image obtained by capturing each of the objects. A program according to any one of appendices 22 to 27. [Explanation of symbols]

[0105] 10, 100 Work support system 1,200 Smart Glasses 12, 201 Image capture unit 13 Image display section 14, 202 Line of sight detection unit 101, 122, 203 Storage section 102, 115, 204 Selection section 103, 116, 205 Judgment section 104, 206 Update section 117 Checklist Update Department 118 Highlight display area 119 Display control unit 123 Checklist

Claims

1. A work assistance system using smart glasses having an image capturing unit and a line of sight detection unit, a storage unit that stores list information indicating a plurality of objects to be confirmed and the confirmation status of each of the objects; a selection unit that selects an object to be confirmed from the image captured by the image capturing unit based on the line of sight detected by the line of sight detection unit; a determination unit that determines whether the selected object matches any of the objects indicated in the list information based on the list information and an image recognition result for the selected object; an update unit that updates the confirmation status of the object included in the list information when the confirmation status matches; A work support system comprising:

2. The smart glasses have an image display unit that displays a predetermined image superimposed on an actual scene, a highlight display unit that displays the selected object in an emphasized manner on the image display unit when the selected object does not match the object indicated by the list information; The work support system according to claim 1 further comprising:

3. a display control unit that displays a part or all of the list information on the image display unit when the selected object does not match the object indicated by the list information; The work support system according to claim 2 further comprising:

4. When the objects indicated by the list information include objects that have not been determined to match the selected object, and when the selection by the selection unit has not been made for a predetermined time or longer, the display control unit displays predetermined information on the image display unit. The work support system according to claim 3 .

5. the list information includes information indicating the serial number of the object to be confirmed; The determination unit determines whether or not the serial numbers of the objects indicated by the list information match based on a result of comparing the serial numbers of the selected objects obtained as a result of the image recognition. The work support system according to claim 4 .

6. The list information includes at least one of information indicating each of the objects to be checked before a predetermined task and information indicating each of the objects to be checked after the task. The work support system according to claim 5 .

7. The list information includes image information indicating each image obtained by capturing each of the objects. The work support system according to claim 6 .

8. an image capturing unit; A gaze detection unit; a storage unit that stores list information indicating a plurality of objects to be confirmed and the confirmation status of each of the objects; a selection unit that selects an object to be confirmed from the image captured by the image capturing unit based on the line of sight detected by the line of sight detection unit; a determination unit that determines whether the selected object matches any of the objects indicated in the list information based on the list information and an image recognition result for the selected object; an update unit that updates the confirmation status of the object included in the list information when the confirmation status matches; Smart glasses equipped with

9. Using smart glasses having an image capturing unit and a line of sight detection unit, storing a list of objects to be identified and the status of identification of each object; selecting an object to be confirmed from the image captured by the image capturing unit based on the line of sight detected by the line of sight detection unit; determining whether the selected object matches any of the objects indicated in the list information based on the list information and an image recognition result for the selected object; If there is a match, updating the confirmation status of the object contained in the inventory information; A work support method including:

10. Using smart glasses having an image capturing unit and a line of sight detection unit, storing a list of objects to be identified and the status of identification of each object; selecting an object to be confirmed from the image captured by the image capturing unit based on the line of sight detected by the line of sight detection unit; determining whether the selected object matches any of the objects indicated in the list information based on the list information and an image recognition result for the selected object; If there is a match, updating the confirmation status of the object contained in the inventory information; A program that causes a computer to execute the following.

Images