Work support system, work support method, and work support program

Abstract

To surely record contents of executed work.SOLUTION: A work support system 1 includes a display device 14 for displaying a screen representing a situation of work by an operator to the operator, a photographing device 16 for photographing an image of the situation of the work by the operator, a storage device 12 for storing working information obtained by associating image information being information representing characteristics of an image of an object used for the work with work contents information being information showing contents of the work, and a control device for displaying the screen in the display device 14, makes the photographing device 16 photograph the image, determines whether the operator executes the work on the basis of the photographed image and the image information, and outputs the information showing the contents of the completed work on the basis of the work contents information in the case that the worker executes the work.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a work assistance system, a work assistance method, and a work assistance program. [Background technology]

[0002] In field services, such as manufacturing, inspection, or repair of products, work is carried out through various work processes. After completing the work, the worker may be required to report the details of the work carried out. In such cases, the worker must keep a record of the work carried out.

[0003] In relation to this point, for example, Patent Document 1 discloses a technology for recording the manufacturing process of a product, which relates to a manufacturing process management system that includes a tablet terminal placed at each manufacturing process and a personal computer that communicates with the tablet terminal, in which the tablet terminal identifies information (work information) relating to work in the manufacturing process (its own process) in which the tablet terminal is placed and transmits the identified work information to the personal computer, which receives the work information from the tablet terminal and records the received work information in association with a lot number uniquely assigned to each planned production lot of the product. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2013-156717 A Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1, workers use a tablet terminal to read barcodes or input various data into the tablet terminal to register the details of the work. However, in the case of a job that involves complicated processes, these registration tasks can be cumbersome, and there is a risk of data being omitted or data being over or under input. As a result, there is a risk that reports with appropriate content cannot be created.

[0006] The present invention has been made in consideration of the above background, and its purpose is to provide a work support system, a work support method, and a work support program that can reliably record the content of the work performed. [Means for solving the problem]

[0007] One aspect of the present invention for solving the above-mentioned problems is a work support system comprising: a display device that displays to a worker a screen showing the status of work being performed by the worker; a photographing device that photographs an image of the status of the work being performed by the worker; a storage device that stores work information that corresponds image information, which is information that represents image characteristics of an object used in the work, with work content information, which is information that indicates the content of the work; and a control device that causes the display device to display the screen and causes the photographing device to photograph the image, determines whether the worker has performed the work based on the photographed image and the image information, and if it is determined that the worker has performed the work, outputs information showing the content of the completed work based on the work content information. Effect of the Invention

[0008] According to the present invention, the content of the performed work can be reliably recorded. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of a configuration of a work support system according to an embodiment of the present invention. [Diagram 2] FIG. 11 is a diagram showing an example of a material information DB. [Diagram 3]FIG. 11 is a diagram showing an example of a work process DB. [Figure 4] FIG. 13 is a diagram illustrating an example of a parts DB. [Diagram 5] FIG. 11 is a flow diagram illustrating an outline of a report creation support process. [Figure 6] FIG. 4 is a diagram showing an example of a work screen displayed on a display device. [Figure 7] FIG. 11 is a flow diagram illustrating details of a target device identification process. [Figure 8] FIG. 11 is a flow diagram illustrating details of a work process registration process. [Figure 9] FIG. 4 is a diagram showing an example of a screen displayed on a display device. [Figure 10] FIG. 4 is a diagram showing an example of a screen displayed on a display device. [Figure 11] FIG. 11 is a diagram showing an example of a work information DB to be created. [Figure 12] FIG. 13 is a diagram showing an example of a created used parts DB. [Figure 13] FIG. 13 is a diagram showing an example of a work content report DB to be created. [Figure 14] FIG. 11 is a flow diagram illustrating details of a synchronization process. [Figure 15] FIG. 13 is a diagram showing an example of the content of a generated work content report. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The work support system of this embodiment is an information processing system used when a worker wears a so-called Augmented Reality (AR) device to perform a specific task. The work support system identifies the content of the task performed by the worker through the AR device and creates a report on the task.

[0011] 1 is a diagram showing an example of the configuration of a task support system 1 according to this embodiment. The task support system 1 includes an AR device 10 and a server 20. The AR device 10 and the server 20 are connected to each other via a wired or wireless communication network 5, such as the Internet, a local area network (LAN), a wide area network (WAN), or a dedicated line.

[0012] The AR device 10 is configured as an information processing device (computer) having a shape similar to glasses, such as smart glasses. The AR device 10 is an information processing device worn by a worker. Specifically, the AR device 10 includes a control device 11, a storage device 12, an input device 13, a display device 14, a communication device 15, and an image capturing device 16. Although not shown, the AR device 10 may include a position sensor, an attitude sensor, an acceleration sensor, or the like for identifying its own position (or coordinates).

[0013] The control device 11 is a processor such as a central processing unit (CPU). The storage device 12 is a random access memory (RAM) or a read only memory (ROM). The AR device 10 may include a non-volatile memory such as a solid state drive (SSD).

[0014] The input device 13 is made up of a touch panel or buttons, and realizes input operations to the photographing device 16 and execution of processing by the control device 11.

[0015] The display device 14 is composed of a liquid crystal panel having a predetermined display area (screen), etc. The display device 14 displays an image generated by the control device 11, etc.

[0016] Specifically, for example, the display device 14 displays an image (image in real space) corresponding to the field of view of the worker wearing the AR device 10. In addition, the display device 14 displays images of characters, figures, etc. generated by each process of the control device 11 by superimposing them on the image in real space.

[0017] 1, the images of real space displayed on the display device 14 are, for example, images of a work target equipment 31 which is an object to be worked on, an object or body 32 used in the work, and a part 33 used in the work. The object or body 32 used in the work is, for example, the body of a worker or equipment or tools used by the worker. In this embodiment, the object or body 32 used in the work is assumed to be the hand of the worker.

[0018] The photographing device 16 photographs an image corresponding to the field of view (the screen displayed on the display device 14) of a worker wearing the AR device 10. For example, the photographing device 16 photographs an image of the status of work being performed by the worker.

[0019] The communication device 15 is composed of a wireless communication module and the like, and communicates with other information processing devices.

[0020] Next, the AR device 10 stores the work information 121 in the storage device 12. The work information 121 includes a material information DB 200, a work process DB 300, a part DB 400, and an object discrimination model 500.

[0021] The equipment information DB 200 stores information on objects (hereinafter referred to as equipment) that are the targets of work performed by workers. The work process DB 300 stores information on the contents of each work process to be performed in work on the equipment. The parts DB 400 stores information on parts to be used in work on the equipment.

[0022] Object discrimination model 500 is a trained model that discriminates the type of object contained in image data (specifically, equipment, parts, parts of a worker's body (e.g., the worker's hands), or tools used by a worker in work, etc.) The object discrimination model 500 is generated based on image information in parts DB 400, as described below.

[0023] In this embodiment, the object discrimination model 500 is a neural network model (for example, a model using a convolution neural network (CNN) or a recurrent neural network (RNN)) having an input layer to which image data as an input value is input, one or more intermediate layers (hidden layers) that extract and output feature amounts of the image data from the input value, and an output layer that outputs the presence or absence of an object type from each feature amount. Note that the object discrimination model 500 may employ a model that applies a support vector machine (SVM), a Bayesian network, or a regression tree, in addition to a neural network, or may be configured as a database, a template, or a program. In other words, the object discrimination model 500 may be anything that outputs information that represents the features of an image of an object used in a task.

[0024] The AR device 10 also stores work result information 122, which is information related to the performed work, in the storage device 12. The work result information 122 includes a work information DB 600 that stores information about each performed work process, a used parts DB 700 that stores information about parts used in each work process, and a work content statement DB 800 that stores details of each performed work process. The details of these pieces of information will be described later.

[0025] The AR device 10 may acquire the work information 121 and the work result information 122 from another information processing device (such as a data server) via the communication network 5 or the like.

[0026] The authentication unit 123 performs authentication to allow the AR device 10 to start executing a process.

[0027] The device identifying unit 124 identifies the device that is the target of the work.

[0028] The work screen display unit 125 causes the display device 14 to display a screen showing the work in progress and a screen related to the work.

[0029] The determination unit 126 determines whether or not the worker has completed each work step in the work. For example, the determination unit 126 causes the display device 14 to display a screen showing the status of the work performed by the worker, and causes the photographing device 16 to photograph an image of the status of the work performed by the worker.

[0030] Then, the judgment unit 126 judges whether or not the worker has performed each work step of the work based on the captured image, the parts DB 400, and the object discrimination model 500, and if it is determined that the worker has performed each work step of the work, it outputs information indicating the content of the completed work to the work content report DB 800 based on the work process DB 300.

[0031] Specifically, the determination unit 126 determines whether or not a part is captured in the captured image and whether or not the captured image captures the worker's body or a tool used by the worker (in this embodiment, the worker's hand), and determines that the worker has started work if a part is captured in the captured image and whether or not the captured image captures the worker's body or a tool used by the worker.

[0032] In addition, the determination unit 126 determines whether or not a part is captured in the captured image and the worker's body or a tool used by the worker (in this embodiment, the worker's hand) is not captured in the captured image, and determines that the worker has completed the work if a part is captured in the captured image and the worker's body or a tool used by the worker is not captured in the captured image.

[0033] (Equipment information DB) 2 is a diagram showing an example of equipment information DB 200. Equipment information DB 200 has data items of equipment ID 201 in which the ID of the equipment is set, equipment name 202 in which the name of the equipment is set, and equipment overview 203 in which a sentence (text, etc.) explaining the overview of the equipment is set. Equipment information DB 200 is created in advance based on input from, for example, an operator or a manager.

[0034] (Work process DB) 3 is a diagram showing an example of a work process DB 300. The work process DB 300 has the following data items: work process ID 301 in which the ID of the work process is set; work process overview 302 in which text explaining the work process or related images is set; equipment ID 303 in which the ID of the equipment related to the work process is set; part ID 304 in which the ID of the part used in the work process is set; work location coordinates 305 in which the coordinates of the reference position in the area (work area) where the work by the part is performed are set; work box size information 306 in which coordinate information for identifying the outline of the work area is set; and work content template 307 in which a text (text, etc.) for a report explaining the contents of the work process is set. The work process DB 300 is created in advance, for example, by input from a worker or a manager.

[0035] In this embodiment, coordinate information in a coordinate system of a three-dimensional space is set in the working location coordinates 305 and the working box size information 306. The AR device 10 converts the coordinates of each object in the three-dimensional space into coordinates in a predetermined coordinate system set for the screen of the display device 14, thereby displaying an image of each object on the screen of the display device 14.

[0036] The method of setting and calculating the coordinates of each equipment and part in this embodiment is an example. For example, coordinates based on a predetermined position in real space may be set for the reference position of the equipment and the position of the part, and the AR device 10 may display images of the equipment and parts on the screen of the display device 14 based on those coordinates.

[0037] Furthermore, a work process title 3071 and a sentence 3072 explaining the details of the work process can be set in the work content template 307. A special character string 3073 for specifying information on parts and equipment used in the work process can be set in the title 3071 and the sentence 3072.

[0038] (Parts DB) 4 is a diagram showing an example of part DB 400. Part DB 400 has data items of part ID 401 in which the ID of a part is set, part name 402 in which the name (or type) of the part is set, and object recognition information 403 in which information indicating image data of the part is set. Data related to object recognition information 403 constitutes teacher data for object discrimination model 500. Although not shown, part DB 400 (specifically, object recognition information 403) also stores image data of each part (hands, etc.) of the worker's body. Part DB 400 is created in advance, for example, based on an input from a worker or a manager.

[0039] In this embodiment, it is assumed that the object discrimination model 500 is generated in advance before the execution of the report creation support process described later. For example, a predetermined information processing device performs machine learning using image data of each object (data of object recognition information 403) as teacher data and names of each object registered in the parts DB 400 (names corresponding to part names 402, worker's hands, etc.) as correct answer data to generate the object discrimination model 500 using image data as input values ​​and information on the presence or absence of objects (names of parts, worker's hands, etc.) included in the image data as output values. Note that the object discrimination model 500 may be composed of multiple models, such as a model for parts and a model for the body.

[0040] Next, the server 20 is an information processing device including a CRM (Customer Relationship Management) system 21. The server 20 stores and accumulates the work result information 122 generated by the AR device .

[0041] The above-described functions of the AR device 10 are realized by the hardware of the AR device 10, or by the control device 11 of each information processing device reading and executing each program stored in the storage device 12. In addition, these programs are stored in, for example, a storage device such as a secondary storage device, a non-volatile semiconductor memory, a hard disk drive, or an SSD, or a recording medium readable by the information processing device, such as an IC card, an SD card, or a DVD. Note that all or a part of the functions of each information processing device in the work support system 1 including the AR device 10 may be realized using virtual information processing resources provided using a virtualization technology, a process space separation technology, or the like, such as a virtual server provided by a cloud system. In addition, all or a part of the functions provided by these devices may be realized by, for example, a service provided by a cloud system via an API (Application Programming Interface), or the like. Next, the processing performed in the work support system 1 will be described.

[0042] <Report creation support processing> 5 is a flow diagram illustrating an outline of the report creation support process. The report creation support process is started, for example, when a predetermined input is made by the worker to the input unit 13 of the AR device 10.

[0043] The authentication unit 123 accepts a login from a worker (s11). For example, the authentication unit 123 accepts input of a user ID and password from the worker, and checks the input user ID and password against a predetermined database. If the check is successful, the authentication unit 123 completes the login.

[0044] The equipment identification unit 124 executes a target equipment identification process s12 for identifying the target equipment in the work to be performed by the worker (hereinafter, referred to as the target work) by using an identification code attached to the target equipment of the work (hereinafter, referred to as the target equipment). The target equipment identification process s12 will be described in detail later.

[0045] Then, the work screen display unit 125 displays information on the work to be performed and the work process (hereinafter, referred to as work content display content) on the AR device 10 based on the coordinates identified in the target equipment identification process s12 (s13).

[0046] For example, the work screen display unit 125 receives a predetermined start input from the worker to the input device 13. Upon receiving the start input, the work screen display unit 125 displays a work screen (described in detail later) showing work content display content.

[0047] Thereafter, the work screen display unit 125 identifies the work process to be started (s14). The work screen display unit 125 stores the identified work process (hereinafter, referred to as the target work process). For example, the work screen display unit 125 identifies the target work process by receiving an input from the worker to the input device 13 specifying one of the work processes displayed on the work screen.

[0048] Thereafter, the work screen display unit 125 displays a figure (hereinafter referred to as box-type content for displaying the work location) representing the target equipment and its surrounding spatial area (hereinafter referred to as the work space) where the target work process is to be performed, in an area on the screen of the display device 14 corresponding to the real space where the target work process is to be performed (s15).

[0049] For example, the work screen display unit 125 refers to the work process DB 300 and acquires the contents of the work location coordinates 305 and work box size information 306 of the record related to the target work process specified in s14. Based on the acquired coordinate information and information on the reference position of the target equipment identified in the target equipment identification process s12 (described later), the work screen display unit 125 calculates the coordinates (coordinates in the coordinate system of the screen of the display device 14) of a three-dimensional figure representing the space where the target work process is performed, and draws the three-dimensional figure on the screen of the display device 14 based on the calculated coordinates.

[0050] 6 is a diagram showing an example of a work screen displayed on the display device 14 in the process of s15. As shown in the figure, this work screen displays a target device 611, a rectangular parallelepiped box-type content 612 for displaying a work location, and a content 613 for displaying work details.

[0051] The work location display box type content 612 is a graphic that indicates the target equipment related to the target work and the space around the target equipment. The work location display box type content 612 is a graphic that indicates a space including, for example, a space where a target part should be attached to the target equipment and a space where the hand or tool of a worker who attaches the part should be positioned.

[0052] The work content display content 613 displays the name 614 of the target equipment identified in the target equipment identification process s12, the name 615 of each work process, and detailed information 616 of each work process (for example, data of the work process overview 302 of the work process DB 300). Also, a predetermined highlighting or the like is applied to a display portion 617 of information related to the target work process that the worker is about to perform. Note that the work content display content 613 may display only information about the target work process that the worker is about to perform.

[0053] In addition, in s14, the work screen display unit 125 may automatically identify the target work process according to the order of the work processes displayed in the work content display content 613 (the order of the actual work).

[0054] Using this type of work screen, the worker can check the content of the work to be performed and the location on the target device where the work should be performed.

[0055] 5, the determination unit 126 executes a work process registration process s16 to determine whether or not the worker has completed the target work process, and if the worker has completed the target work process, to generate information indicating that fact and register it in the work content sentence DB 800. In the work process registration process s16, the determination unit 126 registers information regarding the completed target work process in the work information DB 600, the used parts DB 700, and the work content sentence DB 800. Details of the work process registration process s16 will be described later.

[0056] The determination unit 126 determines whether or not all work steps in the target work have been completed by referring to the work step DB 300 and the work information DB 600 (s17).

[0057] If the work steps in the target work have not been completed (s17: NO), the determination unit 126 executes s18. If all work steps in the target work have been completed (s17: YES), the determination unit 126 stores the current time as the end time of the work and executes s19.

[0058] In s18, the determination unit 126 identifies one of the incomplete work processes as a new target work process. For example, the determination unit 126 identifies the next work process of the current target work process from the work process DB 300, or receives a designation of the work process from the worker via the input device 13. After that, the process of s15 is repeated.

[0059] On the other hand, in s19, the determination unit 126 executes a synchronization process s19 for synchronizing the contents of the work information with the server 20. The synchronization process s19 will be described in detail later.

[0060] Thereafter, the CRM system 21 displays the synchronized work information (for example, the work content text DB 800) on the screen (s20). By checking the displayed screen, the manager or the like can confirm the work content performed by the worker in the form of a report. Each process will be described in detail below.

[0061] <Target equipment specific processing> FIG. 7 is a flow diagram illustrating the target device identification process s12 in detail.

[0062] The device identification unit 124 causes the display device 14 to display the target device and the identification code attached to the target device (s31). For example, first, the equipment identification unit 124 displays content (such as a frame graphic) for setting a shooting area for an identification code at a predetermined position on the screen of the display device 14. The worker changes his / her line of sight (the direction of the display device 14) so ​​that the target equipment is located inside the content displayed on the display device 14. The identification code is data that includes attribute information of the target equipment (for example, the name of the equipment, the name of the manufacturer, and the date of manufacture), and is, for example, a QR code (registered trademark) or a barcode.

[0063] The equipment identification unit 124 causes the photographing device 16 to photograph the identification code displayed in s31 (s32). For example, the equipment identification unit 124 accepts an input of a photographing instruction to the input device 13 from the operator.

[0064] The equipment identification unit 124 analyzes the identification code photographed in s32 to acquire attribute information of the target equipment (s33). In addition, the equipment identification unit 124 calculates the coordinates (coordinates in a three-dimensional space coordinate system or a coordinate system related to the display device 14) of the target equipment displayed on the display device 14 in s32 based on the attitude or position information of the AR device 10, and stores the calculated coordinates as information on the reference position of the target equipment.

[0065] The equipment identification unit 124 determines whether or not the attribute information acquired in s33 is registered in the equipment information DB 200 (s34). For example, the equipment identification unit 124 refers to the equipment information DB 200, determines whether or not there is a record in which the attribute information acquired in s33 is set in the equipment name 202, and if there is such a record, acquires the data of the record.

[0066] If the attribute information acquired in s33 is registered in the equipment information DB 200 (s34: YES), the equipment identification unit 124 stores the current time as the start time of the work, and the target equipment identification process s12 ends. On the other hand, if the attribute information acquired in s33 is not registered in the equipment information DB 200 (s34: NO), the equipment identification unit 124 repeats the processes from s32 onwards.

[0067] <Work process registration process> FIG. 8 is a flow diagram illustrating the details of the work process registration process s16.

[0068] The determination unit 126 takes an image of a part to be used in a target work process (s51).

[0069] For example, the determination unit 126 accepts an input of a photographing instruction from a worker to the input device 13. The worker holds a part to be used in the target work process in his / her hand and changes his / her line of sight (the direction of the display device 14) so ​​that the part is displayed on the screen of the display device 14. The worker then inputs a photographing instruction to the input device 13. The determination unit 126 then causes the photographing device 16 to photograph an image of the part.

[0070] The determination unit 126 determines whether or not the part indicated by the image captured in s51 is registered in the part DB 400 (s52).

[0071] For example, the determination unit 126 acquires the name of the part corresponding to the image data captured in s51 by inputting the image data to the object discrimination model 500. The determination unit 126 determines whether or not the acquired part name is registered in any record of the part DB 400, and if registered, acquires the contents of that record.

[0072] If the part indicated by the image taken in s51 is registered in the part DB 400 (s52: YES), the judgment unit 126 executes the process of s53, and if the part indicated by the image taken in s51 is not registered in the part DB 400 (s52: NO), the judgment unit 126 repeats the process of s51.

[0073] In s53, the determination unit 126 determines whether or not the part photographed in s52 is a part to be used in the target work process (that is, whether or not it is a target part).

[0074] For example, the judgment unit 126 refers to the work process DB 300, acquires the contents of the part ID 304 for which the ID of the target work process is set in the work process ID 301, and judges whether the acquired contents are the same as the part ID of the record acquired in s52.

[0075] If the part photographed in s52 is not the part to be used in the target work process (s53: NO), the determination unit 126 executes the process of s51. On the other hand, if the part photographed in s52 is the part to be used in the target work process (s53: YES), the determination unit 126 executes the process of s54. In this case, the determination unit 126 may display information on the display device 14 indicating whether the worker has picked up the correct part or not, to urge the worker to continue the work.

[0076] 9 is a diagram showing an example of a screen displayed on the display device 14 in s51-s53. As shown in the diagram, the screen of the display device 14 displays a target device 901, a work location display box-type content 902, work content display content 904 in which target work process information 903 is highlighted, a worker's hand 905, a target part 906 held by the worker, and a name 907 of the target part.

[0077] After that, the worker moves the part he / she is holding (i.e., the target part) to the location (work space) of the target equipment where the target part should be attached. At this time, the display device 14 of the AR device 10 worn by the worker faces the direction of the work space. In addition, the display device 14 of the AR device 10 worn by the worker continues to display box-type content for displaying the work location.

[0078] 8, the determination unit 126 determines whether or not the target part and the worker's hands are displayed in the area within the box-type content for displaying a work location on the display device 14. In other words, the determination unit 126 determines whether or not the worker has started the target work process.

[0079] For example, the determination unit 126 acquires an image of an area of ​​a box-type content for displaying a work location on the display device 14. The determination unit 126 inputs the acquired image to the object discrimination model 500, thereby determining whether or not the image includes a target part and the worker's hand.

[0080] If the target part and the worker's hands are displayed in the area within the box-type content for displaying the work location (s54: YES), the judgment unit 126 stores the current time as the start time of the target work process and executes the processing of s55, and if the target part or the worker's hands are not displayed in the area within the box-type content for displaying the work location (s54: NO), the judgment unit 126 repeats the processing of s54.

[0081] 10 is a diagram showing an example of a screen displayed on the display device 14 in s54-s55. As shown in the figure, a target device 1001, a work location display box-type content 1002, and a work content display content 1004 in which target work process information 1003 is highlighted are displayed on the screen of the display device 14. In addition, a worker's hand 1005 during work and a target part 1006 that the worker is holding for work are displayed in the work location display box-type content 1002.

[0082] 8, in s55, the determination unit 126 registers in the work information DB 600 that the target work process has been carried out, and registers in the used parts DB 700 that the target part has been used. For example, the determination unit 126 registers the target equipment, the order of the target work process (the order in which the process was carried out), and the ID of the target work process in the work information DB 600. The determination unit 126 also registers in the used parts DB 700 the ID of the target part and information indicating that the target part has been used.

[0083] The determination unit 126 also registers information indicating that the target work process has been performed using the target part in the work content report DB 800 (s56). For example, the determination unit 126 refers to the work process DB 300, acquires the content of the work content template 307 of a record in which the ID of the target equipment, the ID of the target work process, and the ID of the target part are registered in the equipment ID 303, the work process ID 301, and the part ID 304, respectively, and registers the data obtained by performing predetermined processing on the acquired content in a newly created record of the work content report DB 800.

[0084] After that, the worker performs the target work process using the target part. For example, the worker performs the work of attaching the target part to the target equipment.

[0085] In this case, the determination unit 126 determines whether or not the target part is displayed and the worker's hands are not displayed in the area of ​​the work location display box-type content on the display device 14 (s57). In other words, the determination unit 126 determines whether or not the worker has finished the part installation work, removed his / her hands from the part (target equipment), and completed the target work process.

[0086] For example, the determination unit 126 acquires an image of an area of ​​a box-type content for displaying a work location on the display device 14. The determination unit 126 inputs the acquired image to an object discrimination model, thereby determining whether or not the image includes a target part and does not include a worker's hand.

[0087] If the target part is displayed but the worker's hands are not displayed in the area within the box-type content for displaying the work location (s57: YES), the judgment unit 126 stores the current time as the end time of the target work process and terminates the work process registration process s16; otherwise (s57: NO), the judgment unit 126 repeats the process of s57.

[0088] The determination unit 126 may execute the processes of s55 and s56 at the timing of s57: YES.

[0089] (Work information DB) 11 is a diagram showing an example of the created work information DB 600. The work information DB 600 has data items of a work ID 601 in which the ID of the work is set, a material ID 602 ​​in which the ID of the material used in the work is set, a work order 603 in which the order in which work processes related to a work process ID 604 described below were performed in the work related to the work ID 601 is set, and a work process ID 604 in which the ID of the work process that was performed is set.

[0090] (Used parts DB) 12 is a diagram showing an example of the created used parts DB 700. The used parts DB 700 has data items of a job ID 701 in which the ID of the job is set, a part ID 702 in which the ID of the part used in the job is set, and a number of parts 703 in which the number of parts associated with the part ID 702 used in the job associated with the job ID is set.

[0091] (Work content report DB) 13 is a diagram showing an example of a created work content report DB 800. The work content report DB 800 has data items of a work ID 801 in which the ID of the work is set, and a work content report 802 in which data of the report related to the work (text, etc.) is set. In the work content report 802, data based on the data of the work content fixed phrase 307 in the work process DB 300 is set.

[0092] <Synchronization process> FIG. 14 is a flow diagram illustrating the details of the synchronization process s19.

[0093] First, the determination unit 126 receives an instruction to synchronize the work information generated in the processes up to now with the server 20 (s71). For example, the determination unit 126 receives an input of a synchronization instruction to the input device 13 from the worker.

[0094] When the determination unit 126 receives the instruction, it transmits the work result information 122 generated in the processes up to this point to the server 20 (s72). For example, the determination unit 126 transmits the work information DB 600, the used parts DB 700, and the work content sentence DB 800 to the CRM system 21 of the server 20. This ends the synchronization process s19.

[0095] (Work content report) 15 is a diagram showing an example of the contents of a generated work content report 802. As shown in the figure, this work content report 1500 has the following items: report title 1510, title of each work process 1520, implementation details of each work process 1530, information after work is completed 1540, work time 1550, worker 1560, and parts used 1570.

[0096] The report title 1510 is automatically embedded with the target equipment name 1511 set in the work information DB 600 and information on all used parts 1512 set in the used parts DB 700. Note that the report title 1510 may be embedded with information corresponding to the equipment summary 203 in the equipment information DB 200.

[0097] Information corresponding to the work content template 307 in the work process DB 300 is embedded in the work process title 1520 and the work process implementation content 1530. In this case, information 1521, 1531 on the equipment and parts defined in the work content template 307 is embedded.

[0098] Task time 1550 includes time from start to finish of task 1551 and actual time when task was performed 1552. The start time and end time of task measured by task support system 1 in task support processing are embedded in task time 1551. The total time from start to finish of task measured by task support system 1 in task support processing is embedded in actual time when task was performed 1552.

[0099] In the used part 1570, information based on the contents of each record in the used part DB 700 (the name of the used part 1571 and the number of uses of the used part 1572) is embedded.

[0100] In generating the work content report 802, a predetermined natural language processing may be performed to make the sentence sound natural.

[0101] As described above, the work support system 1 of this embodiment causes the display device 14 to display a screen showing the status of the work being performed by the worker, and also causes the photographing device 16 to take an image of the status of the work being performed by the worker, and determines whether or not the worker has performed the work based on the taken image and image information (parts DB 400, etc.), and if it is determined that the worker has performed the work, outputs information (work content report DB 800) showing the content of the completed work based on the work content information (work process DB 300).

[0102] That is, the work support system 1 displays a screen showing the status of the work of the worker, judges the completion of the work based on the image, and outputs information on the content of the completed work when it judges that the work is completed. This allows the worker to record the content of the work he or she has performed without having to perform cumbersome tasks such as recording the content of the work he or she has performed one by one.

[0103] As described above, according to the work assistance system 1 of this embodiment, the content of the performed work can be reliably recorded.

[0104] Furthermore, the work support system 1 of this embodiment determines that work has started when a part and the worker's body or tool (e.g., a hand) are photographed in an image captured by the photographing device 16 and information (parts DB 400, object discrimination model 500) representing the characteristics of the part image and the image of the worker's body or tool (e.g., a hand) is photographed in the image.

[0105] This makes it possible to reliably detect when a worker has started work using a part.

[0106] Furthermore, the work support system 1 of this embodiment determines that a task is completed based on an image captured by the photographing device 16 and information (parts DB 400, object discrimination model 500) representing the characteristics of the part image and the image of the worker's body or tool (e.g., hand), if the image captures a part and does not capture the worker's body or tool.

[0107] This makes it possible to reliably detect when a worker has completed a task using a part.

[0108] Furthermore, the work support system 1 of this embodiment calculates a position on the screen of the display device 14 that corresponds to the space in which the work is performed, and displays a predetermined figure (box-type content 612 for displaying a work location) at the calculated position on the screen of the display device 14.

[0109] This allows the worker to identify the location where work should be performed and perform the appropriate work.

[0110] Furthermore, the work support system 1 of this embodiment causes the display device 14 to display a screen showing information (work content display content) indicating the process of the work to be performed by the worker, together with the status of the work performed by the worker.

[0111] This allows the worker to understand the content of the work to be performed and perform the appropriate work.

[0112] Furthermore, the work support system 1 of this embodiment causes the photographing device 16 to photograph an image of an identification code attached to the equipment, acquires information about the equipment based on the image, and calculates the position of the equipment. If the information about the equipment is registered in the equipment information DB 200, the work support system 1 causes a screen including the equipment to be displayed at a position (coordinates) of the display device 14 calculated based on the calculated position of the equipment.

[0113] In this way, by photographing the identification code attached to the equipment with the photographing device 16, it is possible to confirm whether the equipment is the equipment to be worked on, and also to calculate the position of the equipment and identify its display position on the screen of the display device 14, thereby enabling the processing to be performed on the equipment (identifying the equipment and identifying the display position of the equipment) to be performed efficiently.

[0114] Furthermore, the work support system 1 of this embodiment identifies the parts used by the worker based on the image captured by the photographing device 16 and the image information (parts DB 400, etc.), and when it is determined that the worker has performed the work based on the identified parts, it outputs information indicating the content of the completed work (work content report DB 800) based on the work content information (work process DB 300) and the information on the identified object (parts DB 400, etc.).

[0115] In this way, by incorporating information about the parts used that are photographed during work by the photographing device 16 (such as parts used DB700) into the work content report DB800 and outputting it, the work content performed by the worker can be recorded in detail.

[0116] The above description of the embodiment is for the purpose of facilitating understanding of the present invention, and is not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit thereof, and the present invention includes equivalents thereof.

[0117] For example, part of the hardware included in each device of this embodiment may be provided in another device.

[0118] Furthermore, each program of each device may be provided in another device, a program may consist of multiple programs, or multiple programs may be integrated into one program.

[0119] Furthermore, the data in the work content report DB 800 may include not only text but also images or videos (for example, photos or videos of the work situation captured by the AR device 10).

[0120] The method of determining the start and end of a task described in the present embodiment is merely an example. For example, the AR device 10 may determine the start and end of a task by recognizing the movements of a worker, equipment, and parts on the screen displayed on the display device 14.

[0121] The method of acquiring and calculating the position information (coordinate information) of the equipment and parts described in the present embodiment is merely an example. For example, the AR device 10 may accept the input of the position information of the equipment from the worker in advance, or may calculate the position information (coordinate information) of the equipment and parts by capturing images of the equipment and parts. [Explanation of symbols]

[0122] 1 Work support system, 10 AR device, 11 control device, 12 storage device, 13 input device, 14 display device, 16 imaging device

Claims

1. a display device that displays a screen showing the status of work performed by the worker to the worker; A photographing device that photographs an image of a work situation of the worker; a storage device that stores work information in which image information, which is information that represents image characteristics of an object used in the work, and work content information, which is information that represents the content of the work, are associated with each other; a control device that causes the display device to display the screen, causes the photographing device to photograph the image, and determines whether or not the worker has performed the work based on the photographed image and the image information, and when it is determined that the worker has performed the work, outputs information indicating the content of the completed work based on the work content information; and A work support system comprising:

2. The storage device stores, as the image information, an image of an object used in the work and information representing characteristics of an image of the body of the worker or an image of a tool used by the worker; The control device determines, based on the captured image and information representing features of the image, whether or not the captured image includes an object used in the work and also includes the body of the worker or a tool used by the worker, and determines that the worker has started the work if the captured image includes an object used in the work and also includes the body of the worker or a tool used by the worker. The work support system according to claim 1 .

3. The storage device stores, as the image information, an image of an object used in the work and information representing characteristics of an image of the body of the worker or an image of a tool used by the worker; The control device determines, based on the captured image and information representing features of the image, whether or not the captured image includes an object used in the work and does not include the worker's body or a tool used by the worker, and determines that the worker has completed the work when an object used in the work is included and the captured image does not include the worker's body or a tool used by the worker. The work support system according to claim 1 .

4. the control device calculates a position on a screen of the display device corresponding to a space in which the work is to be performed, and causes a predetermined figure to be displayed at the calculated position on the screen of the display device; The work support system according to claim 2 or 3.

5. The storage device stores information indicating a process of a task to be performed by the worker, The control device causes the display device to display a screen showing information indicating a process of the work together with a status of the work performed by the worker. The work support system according to claim 1 .

6. The control device includes: causing the photographing device to photograph an image of an identification code attached to a predetermined position of an object that is a target of the work; acquiring information about the object based on the captured image and calculating a position of the object; determining whether the acquired information of the object matches information of a previously registered object, and if the acquired information of the object matches information of a previously registered object, displaying a screen including the object at a position on the screen of the display device calculated based on the calculated position; The work support system according to claim 1 .

7. The control device identifies an object used by the worker based on the captured image and the image information, determines whether or not the worker performed the work based on the identified object, and when it is determined that the worker performed the work, outputs information indicating the content of the completed work based on the work content information and the information of the identified object. The work support system according to claim 1 .

8. a display device that displays a screen showing the status of work performed by the worker to the worker; A photographing device that photographs an image of a work situation of the worker; An information processing device including a storage device that stores work information in which image information, which is information that represents image features of an object used in the work, and work content information, which is information that represents the content of the work, are associated with each other, and a control device; causing the display device to display the screen and causing the photographing device to photograph the image, determining whether or not the worker has performed the work based on the photographed image and the image information, and when it is determined that the worker has performed the work, outputting information indicating the content of the completed work based on the work content information. Work support method.

9. a display device that displays a screen showing the status of work performed by the worker to the worker; A photographing device that photographs an image of a work situation of the worker; An information processing device including a storage device and a control device, storing work information in which image information, which is information expressing image characteristics of an object used in the work, and work content information, which is information expressing the content of the work, are associated with each other; causing the display device to display the screen, and causing the photographing device to photograph the image, determining whether or not the worker has performed the work based on the photographed image and the image information, and when it is determined that the worker has performed the work, outputting information indicating the content of the completed work based on the work content information. Occupational support programs.

Images