Work analysis apparatus, work analysis method, program and system

Abstract

This technology enables easy determination of when a task has been completed, even when object detection and non-detection are repeated within a given work area. [Solution] A work analysis device for determining whether work has been performed in a predetermined work process, comprising: area setting means for setting a plurality of work areas for work analysis from an image of the work situation; order setting means for setting the order in which work is performed for the plurality of work areas; object detection means for detecting an object from the image; and determination means for determining that work has started in a predetermined work area when the object is detected in the predetermined work area, and determining that work has finished in the predetermined work area when the object is detected in the next work area in the sequence following the predetermined work area.

Description

Technical Field

[0001] The present invention relates to a system for determining whether work has been performed in a predetermined work process.

Background Art

[0002] Conventionally, in a manufacturing site or the like, there are work processes such as performing an assembly operation in a predetermined order and with predetermined tools. In such a work process, the overall work time and the work time for each work are often determined, and it is determined whether the work has been completed within the work time, and the determination result is notified to the worker.

[0003] However, it is a heavy burden for a person to constantly check whether a work process is being performed within a predetermined work time in a predetermined work order. For this reason, by analyzing an image obtained by imaging the work situation, it is determined whether the work process is being performed within a predetermined work time.

[0004] In Patent Document 1, a method is described in which a worker is detected from an image obtained by imaging a plurality of work areas, it is determined in which work area the detected worker is located, and the work time of each work area is measured from the time when a worker is present in each work area.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] According to Patent Document 1, in a plurality of work areas, it is determined that the work has started according to the detection of a worker for each work area, and it is determined that the work has ended according to the disappearance of the detection of the worker in the same work area.

[0007] However, in Patent Document 1, if the detection and non-detection of workers are repeated in a certain work area, it is difficult to determine that work has been completed in that work area.

[0008] This invention has been made in view of the above problems, and its purpose is to realize a technology that can easily determine when a task has been completed, even when the detection and non-detection of an object are repeated in a certain work area. [Means for solving the problem]

[0009] To solve the above problems and achieve the objective, the present invention provides a work analysis device for determining whether work has been performed in a predetermined work process, comprising: area setting means for setting a plurality of work areas for which work analysis is to be performed from an image of the work situation; order setting means for setting the order in which work is to be performed for the plurality of work areas; object detection means for detecting an object from the image; and determination means for determining that work has started in a predetermined work area when the object is detected in the predetermined work area, and determining that work has been completed in the predetermined work area when the object is detected in the next work area in the sequence following the predetermined work area. [Effects of the Invention]

[0010] According to the present invention, even when the detection and non-detection of an object are repeated in a certain work area, it is possible to easily determine when the work has been completed. [Brief explanation of the drawing]

[0011] [Figure 1] A block diagram illustrating the hardware configuration of the system according to the first embodiment. [Figure 2] A block diagram illustrating the configuration of a work analysis device that realizes the functions of the system according to the first embodiment. [Figure 3] A diagram illustrating a use case according to the first embodiment. [Figure 4] A diagram illustrating the application screen of the system according to the first embodiment. [Figure 5] Figure illustrating the setting screen of the system according to the first embodiment. [Figure 6] Flowchart illustrating the control process of the system according to the first embodiment. [Figure 7] Flowchart illustrating the control process of the system according to the first embodiment. [Figure 7] Flowchart illustrating the deviation determination process of the work order according to the first embodiment. [Figure 8] Flowchart illustrating the work end determination process according to the first embodiment. [Figure 9] Flowchart illustrating the work start determination process according to the first embodiment. [Figure 10] Figure illustrating the determination result information by the control process of the system according to the first embodiment. [Figure 11] Block diagram illustrating the functional configuration of the system according to the second embodiment. [Figure 12] Figure illustrating the setting screen of the system according to the second embodiment. [Figure 13] Flowchart illustrating the control process of the system according to the second embodiment. [Figure 14] Block diagram illustrating the functional configuration of the system according to the third embodiment. [Figure 15] Flowchart illustrating the control process of the system according to the third embodiment.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0013] In this embodiment, a work analysis system and a work analysis apparatus that capture the work situation at a manufacturing site or the like and determine the work situation by image analysis processing will be described.

[0014] In this embodiment, as the work analysis system and the work analysis apparatus, a computer device performs image analysis processing and determines whether the work time and work process in the work area are being complied with.

[0015] Note that the processing of the work analysis system and the work analysis apparatus of this embodiment may be realized by a single computer device, or each function may be distributed and realized by a plurality of computer devices as necessary. The plurality of computer devices are connected to be communicable with each other.

[0016] Also, the computer device can take the form of a personal computer (desktop PC, notebook PC), tablet PC, smartphone, cloud computer, or the like.

[0017] <Hardware Configuration> FIG. 1 is a block diagram illustrating the hardware configuration of the work analysis system according to this embodiment.

[0018] The work analysis system (hereinafter, the system) 100 of this embodiment includes a control device 101, a non-volatile memory 102, a volatile memory 103, a storage 104, an input device 105, an output device 106, a communication interface (I / F) 107, an imaging device 108, and a system bus 109.

[0019] The control device 101 includes an arithmetic processing processor such as a CPU or MPU that comprehensively controls the entire system 100. The non-volatile memory 102 is a program memory such as a ROM that stores programs and parameters executed by the processor of the control device 101. Here, the program is a program for executing the control processing described later in FIGS. 6 to 9, FIG. 13, and FIG. 15.

[0020] The volatile memory 103 is a data memory such as RAM that temporarily stores programs read from the non-volatile memory 102, as well as constants and variables necessary for executing the program. The volatile memory 103 includes the program work area of ​​the control device 101, a backup area for error handling, and a program load area.

[0021] In addition, program memory may be realized by loading a program into the volatile memory 103 from an external storage device connected to the system 100 of this embodiment.

[0022] Storage 104 is a storage device such as a hard disk drive or solid state drive that is built into or can be connected to system 100. Alternatively, storage 104 can be implemented as an external storage device consisting of, for example, media (recording medium) and a media drive capable of accessing that media. Such media include, for example, flexible disks (FD), CD-ROMs, DVDs, USB memory, MOs, and flash memory. The external storage device may also be a server device connected via a network.

[0023] Storage 104 stores programs that implement the control processing described later in Figures 6 to 9, 13 and 15, judgment result information created in the control processing, learning models and parameters used in the control processing, and image data captured by the imaging device 108. The learning models and parameters may also be configured to be acquired from an external device via the communication interface 107.

[0024] The input device 105 is an operating component such as a mouse, touch panel, or other pointing device, joystick, or keyboard that accepts user input and outputs operation instructions to the control device 101.

[0025] The output device 106 is a display device such as an LCD or organic EL display or monitor, which displays the application's GUI (Graphical User Interface), menu screen, captured image, work area determination result, various operation buttons, etc.

[0026] The communication interface 107 communicates with external devices via a wired or wireless method over a network such as the Internet or a LAN (Local Area Network). In this embodiment, system 100 transmits images of the work situation captured by the imaging device 108 to an external device via the communication interface 107, receives the image analysis results performed by the external device, and manages the image analysis results on the external device.

[0027] The imaging device 108 is built into or connectable to the system 100 and captures images of one or more work areas to generate image data for determining the start and end of work in each work area. The imaging device 108 includes, for example, a camera built into the system 100, a connected camera, a network camera, an industrial camera, etc.

[0028] The system bus 109 includes an address bus, a data bus, and a control bus that connect the components 101 to 108 of the system 100 in a manner that enables the exchange of data.

[0029] The non-volatile memory 102 stores the operating system (OS), which is the basic software executed by the control device 101, and programs including applications that work in cooperation with the OS to realize advanced functions.

[0030] The functions of system 100 in this embodiment are realized by software provided by an application. The application is assumed to have software for utilizing the basic functions of the OS installed on system 100. The OS of system 100 may also have software for implementing the processing in this embodiment.

[0031] <Functional Configuration> Figure 2 is a block diagram illustrating the configuration of a work analysis device that realizes the functions of system 100 according to this embodiment.

[0032] The work analysis device 200 of this embodiment includes a control unit 201, an image acquisition unit 202, a display unit 203, a work area setting unit 204, a work sequence setting unit 205, a work time setting unit 206, an object detection unit 207, a region determination unit 208, a work start determination unit 209, a work end determination unit 210, a work sequence deviation determination unit 211, a work time measurement unit 212, a work time deviation determination unit 213, a determination result information creation unit 214, and a storage unit 215.

[0033] Each functional unit of the work analysis device 200 in this embodiment is realized by the hardware shown in Figure 1 and / or the application program executed by the control device 101.

[0034] The control unit 201 causes the processor of the control device 101 in Figure 1 to function as the respective functional units 202 to 215 in Figure 2.

[0035] The image acquisition unit 202 acquires images of the work situation captured from the imaging device 108.

[0036] The display unit 203 controls the display of the output device 106 shown in Figure 1. The display unit 203 controls the display of the main screen 400 (described later in Figure 4) and the settings screen 500 (described later in Figure 5) shown in Figure 5 on the display of the output device 106.

[0037] The work area setting unit 204 sets one or more work areas to be analyzed from the images acquired by the image acquisition unit 202.

[0038] The work sequence setting unit 205 sets the order of operations for the work areas set by the work area setting unit 204.

[0039] The work time setting unit 206 sets the work time for each work area in the order set by the work area setting unit 204 and the work sequence setting unit 205.

[0040] The object detection unit 207 detects objects in the image and obtains object information such as the type and confidence level of the detected object, as well as coordinate information such as the vertex coordinates and center coordinates of a rectangle circumscribing the object in the image. Furthermore, the object detection method in this embodiment targets tools that can be detected using an object detection model that has been pre-trained to detect tools from an image, but is not limited to this, and any object detection method may be used. For example, people or people's hands may be detected as objects using a detection model that has been pre-trained to detect people or people's hands from an image. Objects may also be detected using a method that uses feature point matching. Furthermore, two-dimensional codes may be detected as objects using a method that uses two-dimensional barcode detection. Furthermore, objects may be detected using a method that uses semantic segmentation to detect the regions of each object that occupy an image.

[0041] The area determination unit 208 determines whether or not an object has been detected within the work area set by the work area setting unit 204, based on the object detection result of the object detection unit 207.

[0042] The work start determination unit 209 determines whether or not to start the next work based on the determination result of the area determination unit 208.

[0043] The work completion determination unit 210 determines, based on the determination result of the area determination unit 208, whether the current work has been completed or whether all work has been completed.

[0044] The work sequence deviation determination unit 211 determines whether or not the work sequence has been deviated from based on the determination result of the area determination unit 208.

[0045] The work time measurement unit 212 measures the current work time (elapsed time from the start of work) for each work area set by the work area setting unit 204.

[0046] The work time deviation determination unit 213 determines whether the work time measured by the work time measurement unit 212 is within the time limit set by the work time setting unit 206.

[0047] The judgment result information creation unit 214 creates judgment result information based on various information stored in the storage unit 215 and the results determined by the area determination unit 208, the work start determination unit 209, the work end determination unit 210, the work sequence deviation determination unit 211, and the work time deviation determination unit 213. The judgment result information creation unit 214 also superimposes the judgment result information onto the image.

[0048] The storage unit 215 controls access to at least one of the non-volatile memory 102, volatile memory 103, and storage 104 shown in Figure 1. The storage unit 215 stores images, setting information set by the work area setting unit 204, work order setting unit 205, and work time setting unit 206, and the determination results of the area determination unit 208, work start determination unit 209, work end determination unit 210, work order deviation determination unit 211, and work time deviation determination unit 213.

[0049] <Screen layout> Figure 3 illustrates a use case according to this embodiment.

[0050] In this embodiment, as shown in Figure 3, the worker 301 holds the tool 302 in his hand and performs screw tightening work on the screw holes 304, 305, 306, and 307 of the workpiece 303.

[0051] Figure 4 illustrates the main screen 400 provided by the application of the system 100 according to this embodiment.

[0052] The main screen 400 is displayed by the control unit 201 controlling the display unit 203. The main screen 400 displays the captured image 401, three pieces of information indicating the work judgment result, and two buttons for operating the system 100. The display of the main screen 400 is also updated according to the control process 600 described later.

[0053] The main screen 400 shown in Figure 4 displays a scene in progress where worker 301 is holding a tool 302 and performing screw tightening operations on screw holes 304, 305, 306, and 307 of the work object 303. Worker 301 has completed the first task, tightening screw hole 304, and is currently working on the second task, tightening screw hole 305. Furthermore, while work on screw hole 304 was started within the time limit, work on screw hole 305 was not started within the time limit.

[0054] On the main screen 400, the captured image 401 displays the work areas 502, 503, 504, 505, and 506 superimposed based on the setting information 507 stored in the storage unit 215, which will be described later. The captured image 401 also displays the work area determination results 402, 403, 404, 405, and 406 superimposed based on the determination result information stored in the storage unit 215. Furthermore, the captured image 401 displays a rectangular frame 407 superimposed around the tool 302, based on the coordinate information of the tool 302 detected by the object detection unit 207.

[0055] Work area determination results 402, 403, 404, 405, and 406 correspond to work areas 502, 503, 504, 505, and 506, respectively. The main screen 400 displays the determination results of the work start determination unit 209, the work end determination unit 210, the work sequence deviation determination unit 211, and the work time deviation determination unit 213 for each work area.

[0056] Work area determination result 402 displays information indicating that the work in the first work area 502 has been completed and that neither the work order nor the work time has deviated. Work area determination result 403 displays information indicating that the work in the second work area 502 has been started, that the work order has not deviated, but the work time has deviated. Work area determination result 404 displays information indicating that the work in the third work area 504 has neither been started nor finished, and that neither the work order nor the work time has deviated. Work area determination result 405 displays information indicating that the work in the fourth work area 505 has neither been started nor finished, and that neither the work order nor the work time has deviated. Work area determination result 406 displays information indicating that the work in the fifth work area 506 has neither been started nor finished, and that neither the work order nor the work time has deviated. The fifth work area 506 is an area provided to determine the completion of the overall work, which will be described later. The fifth work area 506 will be determined to be completed immediately after the start of work is determined.

[0057] The rectangular frame 407 indicates that the tool 302 detected by the object detection unit 207 has been detected. The next work area 408 indicates the number of the next work to be started, and displays information indicating that the third work will be performed next. The work sequence 409 shows the result of determining whether the work sequence has been deviated up to the current work, and displays information indicating that the work sequence has not been deviated. The work time 410 shows the result of determining whether the work time has been deviated up to the current work, and displays information indicating that the work time has been deviated.

[0058] The settings button 411 is an operation button that, when operated by the user, displays a settings screen 500 for setting and saving the work area information, work order information, and work time information for the tasks described later.

[0059] The execution button 412 is an operation button that, when operated by the user, executes a process to analyze whether work has been completed in a given work area, in accordance with the control process 600 described later.

[0060] Figure 5 illustrates the settings screen 500 that appears when the settings button 411 in Figure 4 is operated.

[0061] The settings screen 500 is displayed by the control unit 201 controlling the display unit 203. The settings screen 500 exemplifies the settings screens for the work area setting unit 204, work sequence setting unit 205, and work time setting unit 206 shown in Figure 2. The settings screen 500 displays the captured image 501 captured by the imaging device 108 and stored in the storage unit 215. The settings screen 500 also includes a setting information 507, which is a list of set information, and an add button 513, a region setting button 514, a sequence setting button 515, a time setting button 516, a sequence change button 517, an OK button 518, and a save button 519.

[0062] In this embodiment, based on user operation, work area information, work sequence information, and work time information are set, and the set information list, which is the setting information 507, is stored in the storage unit 215. The captured image 501 includes screw holes 304, 305, 306, and 307 of the work target object 303. The captured image 501 also displays a set work area 502 corresponding to screw hole 304. The captured image 501 also displays a set work area 503 corresponding to screw hole 305. The captured image 501 also displays a set work area 504 corresponding to screw hole 306. The captured image 501 also displays a set work area 505 corresponding to screw hole 307. The captured image 501 also displays a set work area 506 corresponding to the work completion position. Work areas 502, 503, 504, 505, and 506 are created based on the settings in setting information 507: work order 508, center coordinates X 509, center coordinates Y 510, and radius 511. In addition, only records where center coordinates X 509, center coordinates Y 510, radius 511, work order 508, and work start time limit (ms) 512 are set will be displayed.

[0063] The configuration information 507 includes work area information, which consists of center coordinates X509, center coordinates Y510, and radius 511; work sequence information, which consists of work sequence 508; and work time information, which consists of work time limit (ms) 512.

[0064] The add button 513, when operated by the user, creates a new record in the configuration information 507. Each field in the newly created record is initially empty.

[0065] The area setting button 514, when operated by the user, sets the center coordinates X509, Y510, and radius 511 of the record pre-selected in the setting information 507. In this case, the center coordinates X509, Y510, and radius 511 can be set by directly entering values ​​into the corresponding fields, or they can be set using a GUI operation such as a mouse on the captured image 501. The area setting is completed when the user operates the OK button 518.

[0066] The sequence setting button 515, when operated by the user, sets the work order 508 of the records pre-selected in the setting information 507. In this case, the sequential number in the work order 508 already set in the setting information 507 is set in the field of the work order 508. Also, the last record in the work order 508 is set as the work end area. The sequence setting is completed when the user operates the OK button 518.

[0067] The time setting button 516 allows the user to set the work start time limit (ms) 512 for the record pre-selected in the setting information 507. In this case, the work start time limit (ms) 512 is set by directly entering a value in the field. The set work start time limit (ms) 512 is the time limit until the corresponding work in the work sequence 508 is determined to have started. For example, in the setting information 507 of Figure 5, the work start time limit (ms) entered for work sequence '1' is the time limit from when the process 600 described later starts until the first work starts. The work start time limit (ms) entered for work sequence '2' is the time limit from when the first work starts until the second work starts. The user can finish setting the time by operating the OK button 518.

[0068] The order swap button 517 is an operation button that, when operated by the user, swaps the work order of two records pre-selected in the setting information 507, and displays the setting information 507 sorted in ascending order of the work order 508.

[0069] The OK button 518 is an operation button that the user can use to terminate the process of setting the area, order, and time.

[0070] The save button 519 is an operation button that, when operated by the user, saves the setting information 507 to the storage unit 215 and terminates the processing of the setting screen 500.

[0071] <Control Processing> Figure 6 is a flowchart illustrating the control process of system 100 according to this embodiment.

[0072] The process 600 in Figure 6 is realized by the control unit 201 shown in Figure 2 controlling each functional unit in Figure 2 according to the application program. Furthermore, the process 600 in Figure 6 is started when the user operates the execution button 412 on the main screen 400 in Figure 4. The same applies to Figures 13 and 15, which will be described later.

[0073] In step S601, the control unit 201 controls the image acquisition unit 202 to acquire an image from the imaging device 108.

[0074] In step S602, the control unit 201 controls the work time measurement unit 212 to measure the current time, and by referring to the judgment result information stored in the storage unit 215, calculates the difference (work time) between this time and the start time of work in the work area corresponding to the current work sequence.

[0075] In step S603, the control unit 201 controls the work time deviation determination unit 213 to compare the setting information stored in the storage unit 215 with the work time calculated in step S602 and determine if there is a deviation in work time. The control unit 201 compares the work time calculated in step S602 with the work start limit time (ms) in the setting information corresponding to the current work order. The control unit 201 then determines that there is no deviation in work time if the work time is less than the work start limit time, and determines that there is a deviation in work time if the work time is equal to or greater than the work start limit time.

[0076] In step S604, the control unit 201 controls the object detection unit 207 to detect objects in the image acquired in step S601, and acquires information about the detected object and the coordinate information of the object in the image.

[0077] In step S605, the control unit 201 determines whether or not an object was detected based on the object detection result obtained in step S604. If an object was detected in step S604, the control unit 201 executes the process in step S606; otherwise, it executes the process in step S612.

[0078] In step S606, the control unit 201 controls the area determination unit 208 to determine whether or not an object exists within the work area corresponding to the current work order, based on the setting information stored in the storage unit 215 and the object coordinate information acquired in step S604. The control unit 201 then acquires the determination result and the order value of the work area to be determined. In this embodiment, the determination of whether or not an object exists within the work area is performed by calculating and acquiring the center coordinates from the object coordinate information acquired in step S604, and determining whether or not the center coordinate value is within a threshold for each work area set in the setting information. The center coordinates are calculated, for example, as shown in Equation 1 below. (Formula 1) In Equation 1 of TIFF2026060576000002.tif13151, X corresponds to the X coordinate of the center coordinate obtained from the object's coordinate information acquired in step S604, and Y corresponds to the Y coordinate of the center coordinate. Cxi corresponds to the center coordinate X of the i-th work order set in the configuration information, Cyi corresponds to the center coordinate Y of the i-th work order, and ri corresponds to the radius of the i-th work order. As in Equation 1, if the distance between the object's coordinate information and the center coordinate of each work area set in the configuration information is less than or equal to the radius of each work area, it is determined that the object exists within the work area; if it is greater than the radius, it is determined that the object does not exist. In addition, the order value is obtained as 'i' if it is determined that the object exists in the i-th work area, which is the work order, and as '0' if it is determined that the object does not exist in any work area. These values ​​may be set to any values ​​other than those defined. For example, the distance between an arbitrary coordinate value and the center coordinate of each work area may be calculated from the coordinate information of the detected object, and it may be determined whether or not the object exists within the work area. Furthermore, when determining which objects to judge, it is also possible to select any objects to determine whether they exist within the work area, such as selecting only the detection result with the highest reliability among the detected objects. Alternatively, multiple objects may be selected, and each may be determined to exist within the work area.

[0079] In step S607, the control unit 201 determines whether or not an object exists within the work area based on the work area determination result obtained in step S606. If the control unit 201 determines in step S606 that an object exists within the work area, it executes the process in step S608. If it does not determine that an object exists within the work area, it executes the process in step S612.

[0080] In step S608, the control unit 201 controls the work order deviation determination unit 211 to determine whether or not the work order has been deviated from, based on the determination result information stored in the storage unit 215 and the work area determination result obtained in step S606.

[0081] In step S609, the control unit 201 determines whether or not the work order has been deviated from based on the work order deviation determination result obtained in step S608. If the control unit 201 determines in step S608 that the work order has not been deviated from, it executes the process in step S610. If it does not determine that the work order has been deviated from, it executes the process in step S612.

[0082] In step S610, the control unit 201 controls the work completion determination unit 210 to determine the completion of work based on the setting information stored in the storage unit 215 and the work area determination result determined in step S606.

[0083] In step S611, the control unit 201 controls the work start determination unit 209 to determine whether to start work based on the setting information stored in the storage unit 215 and the work area determination result determined in step S606.

[0084] In step S612, the control unit 201 controls the judgment result information creation unit 214 to create judgment result information.

[0085] In step S613, the control unit 201 controls the display unit 203 to update the display on the main screen 400 based on the judgment result information created in step S612.

[0086] In step S614, the control unit 201 controls the storage unit 215 to save the determination result information created in step S612.

[0087] In step S615, the control unit 201 determines whether all tasks have been completed based on the task completion determination result obtained in step S610. If it is determined in step S610 that all tasks have been completed, process 600 is terminated; otherwise, the process in step S601 is executed.

[0088] Figure 7 is a flowchart illustrating the work sequence deviation determination process in step S608 of Figure 6.

[0089] In step S701, the work order deviation determination unit 211 obtains the order value of the current work area where the work is 'started' from the determination result information stored in the storage unit 215.

[0090] In step S702, the work order deviation determination unit 211 obtains the order value of the work area in which an object is determined to exist among the area determination results obtained in step S606.

[0091] In step S703, the work order deviation determination unit 211 determines whether the order value of the work area obtained in step S702 is equal to the order value of the current work area obtained in step S701. If the work order deviation determination unit 211 determines that the order value of the work area obtained in step S702 is equal to the order value of the current work area obtained in step S701, it executes the process in step S705; otherwise, it executes the process in step S704.

[0092] In step S704, the work order deviation determination unit 211 determines whether the order value of the work area obtained in step S702 is the order value of the next work area to be worked on. For example, the work order deviation determination unit 211 determines whether the order value of the work area obtained in step S702 is equal to the current work area order value obtained in step S701 plus 1. Then, if the order value of the work area obtained in step S702 is the order value of the next work area to be worked on, the work order deviation determination unit 211 executes the process in step S705, otherwise it executes the process in step S706.

[0093] In step S705, the work sequence deviation determination unit 211 determines that there is no deviation from the work sequence.

[0094] In step S706, the work sequence deviation determination unit 211 determines that the work sequence has been deviated from.

[0095] Figure 8 is a flowchart illustrating the process for determining the completion of work in step S610 of Figure 6.

[0096] In step S801, the work completion determination unit 210 obtains the sequence value of the current work area where the work is 'started' from the determination result information stored in the storage unit 215.

[0097] In step S802, the work completion determination unit 210 obtains the sequential value of the work area in which an object is determined to exist, from the area determination results obtained in step S606.

[0098] In step S803, the work completion determination unit 210 determines whether the sequence value of the work area obtained in step S802 is equal to the sequence value of the current work area obtained in step S801. If the sequence value of the work area obtained in step S802 is equal to the sequence value of the current work area obtained in step S801, the work completion determination unit 210 determines that work is in progress in the current work area and terminates process 800. If the sequence value of the work area obtained in step S802 is not equal to the sequence value of the current work area obtained in step S801, the work completion determination unit 210 executes the process in step S804.

[0099] In step S804, the work completion determination unit 210 determines that the work in the current work area, which was acquired in step S801, has been completed. In this case, since an object has been detected in the work area that is in the next work sequence after the current work area, it is determined that the work in the current work area has been completed.

[0100] In step S805, the work completion determination unit 210 obtains the sequence value of the last work area, which is the work completion area, from the setting information stored in the storage unit 215.

[0101] In step S806, the work completion determination unit 210 determines whether the sequence value of the work area in which an object is determined to exist, obtained in step S802, is the last work area. The work completion determination unit 210 determines whether the sequence value of the work area in which an object is determined to exist, obtained in step S802, is equal to the sequence value of the work area that is the work completion area, obtained in step S805. If the work completion determination unit 210 determines that the sequence value of the work area in which an object is determined to exist, obtained in step S802, is the last work area, it executes the process in step S807; otherwise, it terminates process 800.

[0102] In step S807, the work completion determination unit 210 determines that all work has been completed.

[0103] Figure 9 is a flowchart illustrating the work start determination process in step S611 of Figure 6.

[0104] In step S901, the work start determination unit 209 obtains the sequence value of the current work area where the work is 'started' from the determination result information stored in the storage unit 215.

[0105] In step S902, the work start determination unit 209 obtains the sequential value of the work area in which an object is determined to exist, from the area determination results determined in step S606.

[0106] In step S903, the work start determination unit 209 determines whether the sequence value of the work area obtained in step S902 is equal to the sequence value of the current work area obtained in step S901. If the sequence value of the work area obtained in step S902 is equal to the sequence value of the current work area obtained in step S901, the work start determination unit 209 determines that work is in progress in the current work area and terminates process 900. If the sequence value of the work area obtained in step S902 is not equal to the sequence value of the current work area obtained in step S901, the work start determination unit 209 executes the process in step S904.

[0107] In step S904, the work start determination unit 209 obtains the sequence value of the last work area, which is the work end area, from the setting information stored in the storage unit 215.

[0108] In step S905, the work start determination unit 209 determines whether the sequence value of the work area in which an object is determined to exist, obtained in step S902, is the same as the sequence value of the last work area obtained in step S904. If the sequence value of the work area in which an object is determined to exist, obtained in step S902, is the same as the sequence value of the last work area obtained in step S904, the work start determination unit 209 terminates process 900; otherwise, it executes the process in step S906.

[0109] In step S906, the work start determination unit 209 determines that it has started work in the work area corresponding to the sequence value of the work area in which an object was determined to exist in the work area obtained in step S902, and terminates process 900.

[0110] Figure 10 illustrates the judgment result information created in step S612 of Figure 6.

[0111] The judgment result information 1000 includes the work order 1001, the work judgment 1002, the work sequence judgment 1003, the work time judgment 1004, and the work time (ms) 1005.

[0112] The work order 1001 stores a numerical value corresponding to the work order 508 stored in the storage unit 215. The work determination 1002 stores the results determined by the work start determination unit 209 and the work end determination unit 210. For example, in the work area corresponding to each work order 1001, if the work end determination unit 210 determines that the work has ended, the work determination 1002 stores "end". If the work start determination unit 209 determines that the work has started, the work determination 1002 stores "start". If it is not determined to be either end or start, "-" is stored. The work sequence determination 1003 stores the results determined by the work sequence deviation determination unit 211. For example, in the work area corresponding to each work order 1001, if the work sequence deviation determination unit 211 determines that the work sequence has deviated, the work sequence determination 1003 stores "NG". If the work order deviation determination unit 211 determines that the work order has not deviated, the work order determination 1003 stores "OK". The work time determination 1004 stores the result determined by the work time deviation determination unit 213. For example, if the work time deviation determination unit 213 determines that the work time has deviated in the work area corresponding to each work order 1001, the work time determination 1004 stores "NG". If the work time deviation determination unit 213 determines that the work time has not deviated, the work time determination 1004 stores "OK". The work time (ms) 1005 stores the elapsed time from the start of the work, as measured by the work time measurement unit 212.

[0113] As described above, according to the first embodiment, the sequential value of a work area in which an object exists is compared with the sequential value of a predetermined work area where the work has 'started'. If there is no deviation from the work order and the sequential values ​​of the two work areas are not equal, it is determined that an object has been detected in the work area in the next work order after the predetermined work area, and therefore the work in the predetermined work area is determined to be finished.

[0114] Furthermore, if the sequence value of the second work area in which an object exists is equal to the sequence value of the last work area in the sequence, it is determined that all operations have been completed.

[0115] This makes it easy to determine when an operation has ended, even when objects are repeatedly detected and not detected within the work area.

[0116] [Second Embodiment] Next, a second embodiment will be described.

[0117] In the following sections, explanations common to the first embodiment will be omitted, and the differences from the first embodiment will be described.

[0118] In the first embodiment, the work area setting unit 204 described a method for setting a work area so that it is circular. In the first embodiment, the work area can only be set to be circular, and it is not possible to set arbitrary polygons such as elongated work areas, and it is necessary to specify the values ​​again each time the camera's field of view changes. Therefore, in the second embodiment, an example is described in which various work areas such as rectangles as well as circles can be set, and furthermore, in addition to setting by numerical specification, numerical values ​​obtained from object detection or the like (such as coordinate values) can be specified.

[0119] The hardware configuration of the system 100 and the configuration of the main screen 400 according to the second embodiment are the same as those in Figures 1 and 4 of the first embodiment.

[0120] Figure 11 is a block diagram illustrating the configuration of a work analysis device that realizes the functions of system 100 according to the second embodiment.

[0121] The work analysis device 1100 of this embodiment has a region setting object detection unit 1101, and the functions of the work region setting unit 1102 and the region determination unit 1103 are different.

[0122] The region-defined object detection unit 1101 detects objects in the captured image and acquires the detected object information and coordinate information such as the vertex coordinates and center coordinates of a rectangle circumscribing the object in the image. Similar to the object detection unit 207, the region-defined object detection unit 1101 detects arbitrary objects using various object detection methods. The object detection unit 1101 detects objects set in the region-defined object text box 1221, which will be described later. Note that the detected objects and object detection methods in this embodiment are the same as those of the object detection unit 207 in the first embodiment.

[0123] The work area setting unit 1102 sets one or more work areas for work analysis based on the object detection results of the area setting object detection unit 1101.

[0124] The area determination unit 1103 determines whether or not an object has been detected within the work area set by the work area setting unit 1102, based on the work area information set by the work area setting unit 1102 and the object detection result obtained by the object detection unit 207.

[0125] Figure 12 illustrates the settings screen 1200, which is displayed by operating the settings button 411 on the main screen 400 in Figure 4. The settings screen 1200 is displayed by the control unit 201 controlling the display unit 203. The settings screen 1200 displays the captured image 1201 that has been previously captured by the imaging device 108 and stored in the storage unit 215. The settings screen 1200 also includes a setting information list 1211, an add button 513, a region setting object text box 1221, a region setting button 1222, a sequence setting button 515, a time setting button 516, a sequence change button 517, an OK button 518, and a save button 1223.

[0126] Image 1201 shows the screw holes 304, 305, 306, and 307 of the object to be worked on 303. Image 1201 also displays a work area 1203 corresponding to screw hole 304, a work area 1205 corresponding to screw hole 305, a work area 1207 corresponding to screw hole 306, a work area 1209 corresponding to screw hole 307, a work area 1210 corresponding to the end position of the work, and detection rectangles 1202, 1204, 1206, and 1208 of the screw holes detected by the area setting object detection unit 1101.

[0127] Work areas 1203, 1205, 1207, 1209, and 1210 are created based on the settings of setting information 1211: work order 508, area type 1212, top-left coordinate X / center coordinate X1213, top-left coordinate Y / center coordinate Y1214, bottom-right coordinate X1215, bottom-right coordinate Y1216, radius / long side / major axis 1217, short side / minor axis 1218, rotation angle 1219, and judgment area ratio 1220.

[0128] The setting information 1211 includes work area information such as work order 508, area type 1212, top-left coordinate X / center coordinate X 1213, top-left coordinate Y / center coordinate Y 1214, bottom-right coordinate X 1215, bottom-right coordinate Y 1216, radius / long side / major axis 1217, short side / minor axis 1218, rotation angle 1219, judgment area ratio 1220, and work time information such as work limit time (ms) 512.

[0129] The region setting object text box 1221 allows the user to directly enter text describing the type of object they want to detect, thereby setting the region setting object information.

[0130] The area setting button 1222 allows the user to set the area type 1212, top-left coordinate X / center coordinate X1213, top-left coordinate Y / center coordinate Y1214, bottom-right coordinate X1215, bottom-right coordinate Y1216, radius / long side / major axis 1217, short side / minor axis 1218, rotation angle 1219, and judgment area ratio 1220 of the record pre-selected in the setting information 1211.

[0131] The region type 1212 indicates the type of region that has been set. In this embodiment, the region type can be set to any region type from "rectangle", "rotated rectangle", "circle", and "ellipse". In addition to the above example, it may be possible to set any shape as a region by operating the GUI using a mouse or the like. Furthermore, it may be possible to set the region of any object that can be acquired by a method such as semantic segmentation using the region setting object detection unit 1101.

[0132] The top-left coordinate X / center coordinate X1213, top-left coordinate Y / center coordinate Y1214, bottom-right coordinate X1215, bottom-right coordinate Y1216, radius / long side / major axis 1217, short side / minor axis 1218, and rotation angle 1219 are set based on the type of region set in region type 1212.

[0133] If the area type 1212 is "rectangle," the top-left coordinate X / center coordinate X1213 and the top-left coordinate Y / center coordinate Y1214 are set as the top-left coordinate values ​​of the rectangle. Also, the bottom-right coordinate X1215 and bottom-right coordinate Y1216 are set as the bottom-right coordinate values ​​of the rectangle. Then, the work area is created based on these settings.

[0134] If the area type 1212 is "rotating rectangle," the top-left coordinate X / center coordinate X1213 and the top-left coordinate Y / center coordinate Y1214 are set as the center coordinate values ​​of the rectangle. Additionally, the radius / long side / major axis 1217 is set as the length of the long side of the rectangle. Furthermore, the short side / minor axis 1218 is set as the length of the short side of the rectangle. Finally, the rotation angle 1219 is set as the numerical value of the rotation angle of the rectangle. Based on these settings, the work area is created.

[0135] If the area type 1212 is "circular", the top-left coordinate X / center coordinate X1213 and the top-left coordinate Y / center coordinate Y1214 are set as the center coordinate values ​​of the circle, and the radius / longest side / major axis 1217 is set as the radius of the circle, and the work area is created based on these settings.

[0136] If the region type 1212 is "Ellipse," the top-left coordinate X / center coordinate X1213 and the top-left coordinate Y / center coordinate Y1214 are set as the center coordinate values ​​of the ellipse. Additionally, the radius / long side / major axis 1217 is set as the major axis of the ellipse, and the short side / minor axis 1218 is set as the minor axis of the ellipse. A work area is then created based on these settings.

[0137] Additionally, the top-left coordinate X / center coordinate X1213, top-left coordinate Y / center coordinate Y1214, bottom-right coordinate X1215, bottom-right coordinate Y1216, major side / major axis 1217, minor side / minor axis 1218, and rotation angle 1219 may be set by directly entering values ​​into the corresponding fields. Alternatively, these settings may be configured using a GUI operation such as a mouse on the captured image 1201, or each setting value may be configured using the object detection results. In the example in Figure 12, for example, in the record set as "1" in the work sequence 508, the region type 1212 is selected as rectangle. The top-left X coordinate / center coordinate X1213, top-left Y coordinate / center coordinate Y1214, bottom-right X coordinate X1215, and bottom-right Y coordinate Y1216 are set to the top-left X coordinate, top-left Y coordinate, top-right X coordinate, and top-right Y coordinate of the first detection result detected by the area setting object detection unit 1101, based on the type of object entered in the area setting object text box 1221. In addition to the above example, any coordinate values ​​of the object detection result, such as the X coordinate of the center of the detected object and the Y coordinate of the center, may also be set.

[0138] The judgment area ratio 1220 is set as a threshold for determining whether or not an object is present within the work area, and the user can set it by directly entering a value into the corresponding field.

[0139] The OK button 518 is an operation button that the user can use to finish the area setting.

[0140] The save button 1223 is an operation button that, when operated by the user, saves the setting information 1211 and the area setting object information entered in the area setting object text box 1221 to the save unit 215, and terminates the processing of the setting screen 500.

[0141] Figure 13 is a flowchart illustrating the control process of system 100 according to this embodiment.

[0142] In Figure 13, the same steps as in Figure 6 are given the same step number and their explanations are omitted.

[0143] In step S1301, the control unit 201 controls the region setting object detection unit 1101 to detect an object from the captured image acquired in step S601 based on the region setting object information stored in the storage unit 215. The control unit 201 also acquires the object information detected by the region setting object detection unit 1101 and the vertex coordinates of a rectangle circumscribing the object in the captured image.

[0144] In step S1302, the control unit 201 controls the work area setting unit 1102 to set the work area based on the setting information stored in the storage unit 215 and the object detection results acquired in step S1301.

[0145] In step S1303, the control unit 201 controls the area determination unit 1103 to determine whether or not an object exists within the work area based on the setting information stored in the storage unit 215, the work area information set in step S1302, and the object coordinate information acquired in step S604. The control unit 201 then acquires the determination result and the order value of the work area to be determined. The determination of whether or not an object exists within the work area is made by determining whether the area obtained from the object coordinate information acquired in step S604 and the respective work areas set in step S1302 overlap, and whether or not the area is greater than or equal to the determination area ratio set for each work order, as shown in Equation 2 below. (Formula 2) In Equation 2 of TIFF2026060576000003.tif20151, Thi corresponds to the determination area ratio for the i-th work step set in the configuration information. Areai_ob indicates the area of ​​the overlapping region between the configured work area for the i-th work step and the region obtained from the object's coordinate information acquired in step S604. Areai indicates the area of ​​the configured work area for the i-th work step. As shown in Equation 2, if the area of ​​the overlapping region between the configured work area for the i-th work step and the region obtained from the object's coordinate information acquired in step S604 is greater than or equal to the determination area ratio, it is determined that the object is within the work area. If it is less than the determination area ratio, it is determined that the object is not within the work area. Note that the determination of whether or not an object is within the work area is not limited to the above example; for example, the determination of whether or not an object is within the area may be made by checking whether the area overlapping between the detected object and the work area is greater than or equal to a threshold relative to the region obtained from the detected object's coordinate information.

[0146] As described above, according to the second embodiment, in addition to the effects of the first embodiment, the shape of the work area can be set to various shapes such as rectangles as well as circles, and furthermore, it can be set not only by numerical specification but also by numerical values ​​(such as coordinate values) obtained from object detection.

[0147] [Third Embodiment] Next, a third embodiment will be described.

[0148] In the following sections, explanations common to the first embodiment will be omitted, and the differences from the first embodiment will be described.

[0149] In the first embodiment, a method was described in which the area determination unit 208 calculates the center coordinates from the object coordinate information acquired by the object detection unit 207 and determines whether or not the object's center coordinates are included within each work area. For example, in detailed work using hands, it is necessary to determine whether or not an object exists within the work area using not only the object's coordinate information but also detailed coordinate information such as the worker's fingertips. Therefore, in this embodiment, an example is described in which the posture estimation unit 1401 can acquire the posture information of an object in the captured image for an object detected by the object detection unit 207.

[0150] The hardware configuration of the system 100 and the configuration of the main screen 400 according to the third embodiment are the same as those in Figures 1 and 4 of the first embodiment.

[0151] Figure 14 is a block diagram illustrating the configuration of a work analysis device that realizes the functions of system 100 according to the third embodiment.

[0152] The work analysis device 1400 of this embodiment has a posture estimation unit 1401 and a different function of the region determination unit 1402.

[0153] The posture estimation unit 1401 acquires coordinate information of the joint points of an object in an image for an object detected by the object detection unit 207. For example, in this embodiment, the posture estimation unit 1401 acquires coordinate information of the joint points of a tool using a posture estimation model that has been trained in advance to estimate the positions of multiple joint points such as the tip point, center point, and end point of a tool from an image.

[0154] The area determination unit 1402 determines whether or not an object has been detected within the work area set by the work area setting unit 204, based on the attitude estimation result obtained by the attitude estimation unit 1401.

[0155] Figure 15 is a flowchart illustrating the control process of system 100 according to this embodiment.

[0156] In Figure 15, the same steps as in Figure 6 are given the same step number and their explanations are omitted.

[0157] In step S1501, the control unit 201 controls the attitude estimation unit 1401 to obtain the joint point coordinates of the object detected in step S604 from the captured image acquired in step S601.

[0158] In step S1502, the control unit 201 controls the area determination unit 1402 to compare the setting information stored in the storage unit 215 with the joint point coordinates of the object obtained in step S1501, and determines whether or not an object exists within the work area. The control unit 201 then obtains the determination result and the order value of the work area to be determined. In this embodiment, the determination of whether or not an object exists within the work area is made by calculating the distance between the joint point coordinates of the object obtained in step S1501 and the center coordinates of each area, as shown in Equation 3 below, and determining whether or not an object is included in each area. (Formula 3) In Equation 3 of TIFF2026060576000004.tif13151, KX corresponds to the X coordinate of the joint point coordinate of the object obtained in step S1501. KY corresponds to the Y coordinate of the joint point coordinate of the object. Cxi corresponds to the X coordinate of the center in the i-th work order set in the configuration information. Cyi corresponds to the Y coordinate of the center in the i-th work order. ri corresponds to the radius in the i-th work order. As in Equation 3, if the distance between the joint point coordinate of the object and the center coordinate of each work area set in the configuration information is less than or equal to the radius of each work area, it is determined that the object is within the work area; if it is greater than the radius, it is determined that the object is not within the work area. The order value obtained is 'i' if it is determined that the object is in the i-th work area, and '0' if it is determined that the object is not in any work area. These values ​​can also be set to arbitrary values; for example, any joint point coordinate from the object's joint point coordinates can be selected to determine whether or not it is within the area. Alternatively, multiple joint point coordinates may be selected from the joint point coordinates of an object, and it may be determined whether or not each of them is within the specified area.

[0159] As described above, according to the third embodiment, in addition to the effects of the first embodiment, it is possible to determine whether or not an object exists within the work area by using not only the coordinate information of the object but also detailed coordinate information such as the joint points of the object.

[0160] [Other embodiments] This embodiment can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiment to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0161] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention.

[0162] The disclosures herein include the following work analysis apparatus, work analysis methods, programs, and systems. [Item 1] A work analysis device that determines whether work was performed according to a predetermined work process, A region setting means for setting multiple work areas for work analysis from images captured of the work situation, A sequence setting means for setting the order in which to perform tasks for the aforementioned multiple work areas, Object detection means for detecting an object from the aforementioned image, When the object is detected in the predetermined work area, it is determined that work has started in the predetermined work area. A work analysis device characterized by having determination means for determining that work in the predetermined work area has been completed in response to the detection of the object in the work area in the order following the predetermined work area. [Item 2] The work analysis apparatus according to item 1, further comprising an area determination means that determines whether or not an object has been detected within the work area set by the area setting means, based on the detection result of the object detection means. [Item 3] The work analysis apparatus according to item 2, further comprising a work order deviation determination means that determines whether or not the work order set by the order setting means has been deviated from, based on the work area in which the area determination means has determined that an object exists and the work area in which the work has been determined to have started. [Item 4] The work analysis apparatus according to item 3, characterized in that the determination means determines that work has started in the predetermined work area, and if the work has not deviated from the order of work set by the order setting means, determines that work in the predetermined work area has been completed in response to the detection of the object in the work area that follows the predetermined work area. [Item 5] The work analysis apparatus according to item 3, characterized in that the determination means determines that work has been completed in all work areas if the order of work in the work area in which an object is determined to exist by the area determination means is the same as the order of work in the last work area. [Item 6] The work analysis apparatus according to item 2, characterized in that the determination means determines that work has started in the work area in which an object is determined to exist when the order of work in the work area in which an object is determined to exist differs from the order of work in the work area in which work has been determined to have started. [Item 7] A work time setting means for setting the work time for each work area set by the area setting means, A work time measuring means for measuring the work time for each of the aforementioned work areas, The work analysis apparatus according to item 6, further comprising: a work time deviation determination means for determining whether the work time measured by the work time measurement means is within the time limit set by the work time setting means. [Item 8] A means for creating and displaying determination result information for each work area, including the order of the work, the determination result of the start or end of the work, the determination result of deviation from the order of the work, the determination result of deviation from the work time, and the work time. The work analysis device according to item 7, further comprising storage means for storing the image, the coordinate information of the object, the coordinate information of the work area, the order of the work, the determination result of the start or end of the work, the determination result of deviation from the order of the work, and the determination result of deviation from the work time. [Item 9] The work analysis apparatus according to any one of items 1 to 8, characterized in that the object detection means detects an object using a learning model that has been pre-trained to acquire coordinate information of an object in the image. [Item 10] The work analysis apparatus according to any one of items 1 to 8, characterized in that the object detection means detects objects by using a learning model that has been pre-trained to detect the features of objects in the image, or by using matching the features of objects in the image, or by detecting a two-dimensional code as an object, or by detecting the region of each object that occupies the image. [Item 11] The work analysis apparatus according to any one of items 1 to 10, characterized in that the area setting means can set the shape of the work area. [Item 12] The work analysis apparatus according to any one of items 1 to 10, characterized in that the area setting means can set the shape of the work area using coordinate information of an object detected from the image. [Item 13] The work analysis apparatus according to any one of items 2 to 8, characterized in that the area determination means determines that an object exists in the work area when the distance between the coordinate information of the object detected by the object detection means and the center of the work area set by the area setting means is within a threshold. [Item 14] The work analysis apparatus according to any one of items 2 to 8, characterized in that the area determination means determines that an object exists in the work area when the area ratio in which the area obtained from the coordinate information of the object detected by the object detection means and the work area set by the area setting means overlap is greater than or equal to a threshold. [Item 15] The work analysis device according to any one of items 2 to 8, characterized in that the area determination means determines that an object exists in the work area when the distance between the coordinate information of one or more joint points among the posture information of the object detected by the object detection means and the center of the work area set by the area setting means is within a threshold. [Item 16] A work analysis method for determining whether work was performed according to a predetermined work process, The area setting means includes the step of setting multiple work areas for work analysis from images captured of the work situation, The sequence setting means includes the step of setting the order in which to perform work on the plurality of work areas, The object detection means includes the step of detecting an object from the image, The determination means determines that work has started in the predetermined work area in response to the detection of the object in the predetermined work area, A work analysis method characterized by comprising the step of determining that work in the predetermined work area has been completed in response to the detection of the object in the work area in the next order of the predetermined work area. [Item 17] A program for causing a computer to function as one of the means of the work analysis apparatus described in any of items 1 through 15. [Item 18] A work analyzer as described in any of items 1 to 15, An imaging device for capturing the aforementioned image, A system including an output device that superimposes judgment result information from the work analysis device onto the aforementioned image and outputs it. [Explanation of Symbols]

[0163] 201...Control unit, 204, 1102...Work area setting unit, 207...Object detection unit, 208, 1103, 1402...Area determination unit, 209...Work start determination unit, 210...Work end determination unit, 211...Work sequence deviation determination unit, 213...Work time deviation determination unit

Claims

1. A work analysis device that determines whether work was performed according to a predetermined work process, A region setting means for setting multiple work areas for work analysis from images captured of the work situation, A sequence setting means for setting the order in which to perform tasks for the aforementioned multiple work areas, Object detection means for detecting an object from the aforementioned image, When the object is detected in the predetermined work area, it is determined that work has started in the predetermined work area. A work analysis device characterized by having determination means for determining that work in the predetermined work area has been completed when the object is detected in the work area in the order following the predetermined work area.

2. The work analysis apparatus according to claim 1, further comprising a region determination means that determines whether or not an object has been detected within a work area set by the region setting means, based on the detection result of the object detection means.

3. The work analysis apparatus according to claim 2, further comprising a work order deviation determination means that determines whether or not the work order set by the order setting means has been deviated from, based on the work area in which the area determination means has determined that an object exists and the work area in which the work has been determined to have started.

4. The work analysis apparatus according to claim 3, characterized in that the determination means determines that work has started in the predetermined work area, and if the order of work set by the sequence setting means has not been deviated from, then determines that work in the predetermined work area has been completed in response to the detection of the object in the work area that follows the predetermined work area.

5. The work analysis apparatus according to claim 3, characterized in that the determination means determines that work has been completed in all work areas if the order of work in the work area in which the area determination means has determined that an object exists is the same as the order of work in the last work area.

6. The work analysis apparatus according to claim 2, characterized in that the determination means determines that work has started in the work area in which an object is determined to exist when the order of work in the work area in which an object is determined to exist differs from the order of work in the work area in which work has been determined to have started.

7. A work time setting means for setting the work time for each work area set by the area setting means, A work time measuring means for measuring the work time for each of the aforementioned work areas, The work analysis apparatus according to claim 6, further comprising: a work time deviation determination means for determining whether the work time measured by the work time measurement means is within the time limit set by the work time setting means.

8. A means for creating and displaying determination result information for each work area, including the order of the work, the determination result of the start or end of the work, the determination result of deviation from the order of the work, the determination result of deviation from the work time, and the work time. The work analysis apparatus according to claim 7, further comprising storage means for storing the image, the coordinate information of the object, the coordinate information of the work area, the order of the work, the determination result of the start or end of the work, the determination result of deviation from the order of the work, and the determination result of deviation from the work time.

9. The work analysis apparatus according to claim 1, characterized in that the object detection means detects objects using a learning model that has been pre-trained to acquire coordinate information of objects in the image.

10. The work analysis apparatus according to claim 1, characterized in that the object detection means detects objects by using a learning model that has been pre-trained to detect the features of objects in the image, or by using matching the features of objects in the image, or by detecting a two-dimensional code as an object, or by detecting the region of each object that occupies the image.

11. The work analysis apparatus according to claim 1, characterized in that the area setting means can set the shape of the work area.

12. The work analysis apparatus according to claim 1, characterized in that the area setting means can set the shape of the work area using coordinate information of an object detected from the image.

13. The work analysis apparatus according to claim 2, characterized in that the area determination means determines that an object exists in the work area when the distance between the coordinate information of the object detected by the object detection means and the center of the work area set by the area setting means is within a threshold.

14. The work analysis apparatus according to claim 2, characterized in that the area determination means determines that an object exists in the work area when the area ratio in which the area obtained from the coordinate information of the object detected by the object detection means and the work area set by the area setting means overlap is greater than or equal to a threshold.

15. The work analysis apparatus according to claim 2, characterized in that the area determination means determines that an object exists in the work area when the distance between the coordinate information of one or more joint points among the posture information of the object detected by the object detection means and the center of the work area set by the area setting means is within a threshold.

16. A work analysis method for determining whether work was performed according to a predetermined work process, The area setting means includes the step of setting multiple work areas for work analysis from images captured of the work situation, The sequence setting means includes the step of setting the order in which to perform work on the plurality of work areas, The object detection means includes the step of detecting an object from the image, The determination means determines that work has started in the predetermined work area in response to the detection of the object in the predetermined work area, A work analysis method characterized by comprising the step of determining that work in the predetermined work area has been completed in response to the detection of the object in the work area in the next order of the predetermined work area.

17. A program for causing a computer to function as one of the means of the work analysis apparatus described in any one of claims 1 to 15.

18. A work analysis apparatus according to any one of claims 1 to 15, An imaging device for capturing the aforementioned image, A system including an output device that superimposes judgment result information from the work analysis device onto the aforementioned image and outputs it.

Images