Work management system and work management method

Abstract

To provide a work management system and a work management method that accurately determine start and completion of a process.SOLUTION: A work management system is configured by a computer having an arithmetic unit that executes predetermined arithmetic processing and a storage unit that is accessible by the arithmetic unit. The arithmetic unit includes a process start-completion condition creation unit that creates a process start / completion condition for determining whether a process is started or completed. The process start / completion condition creation unit receives a designation of a start / completion determination target area for determining whether a process is started or completed, detects a change in an object at timing of starting or completing the process as a process start / completion condition candidate by analyzing a sample video, presents the detected process start / completion condition candidate, combines the process start / completion condition candidates selected by an operator, and records them as the process start / completion condition.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a work management system. [Background technology]

[0002] In the manufacturing industry, there are systems that determine the process and instruct workers on the next task.

[0003] The following prior art is included as background technology in this technical field. Patent Document 1 (JP 2021-082204 A) describes a work instruction system that includes an imaging device that captures images including a worker performing multiple elemental tasks on an object, an information processing device that includes a completion determination unit that determines the completion of each elemental task and a task identification unit that identifies the elemental task that the worker should perform next, and a display device that displays the elemental task that the worker should perform next. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-082204 Summary of the Invention [Problem to be solved by the invention]

[0005] In the above-mentioned work instruction system, it is necessary to determine the start and completion conditions of a process in order to determine the start and completion of the process. In general, the start and completion conditions of a process are determined by an operator watching a video of the process and extracting events (e.g., the movement of an object) that are the conditions for starting and completing the process.

[0006] In recent years, the manufacturing industry has often constructed production lines for a wide variety of products, making it difficult to mechanically determine the division of work processes. For example, when there are multiple work locations or multiple work objects scattered within a process, it is difficult to determine whether a process has been completed under one condition. Also, when moving to the next process without changing the positions of people or objects, even if an image of the work location is analyzed, it appears that work within the same process is continuing, making it difficult to determine whether the next process has started. Furthermore, when a work object is moving on the production line, the coordinates of the object are shifted from the angle of view when the image is taken, making it difficult to determine whether a process has been completed.

[0007] In the background art described above, task completion is determined based on the position of hands or objects in the video, and completion determination for tasks with a large range of motion is not anticipated. [Means for solving the problem]

[0008] A representative example of the invention disclosed in the present application is as follows: That is, a work management system is configured by a computer having an arithmetic device that executes a predetermined arithmetic process and a storage device accessible by the arithmetic device, the arithmetic device is provided with a process completion condition creation unit that creates a process completion condition for determining the start or completion of a process, the process completion condition creation unit receives a designation of a completion determination target area in an image for determining the start or completion of a process, detects a change in an object at the timing of the start or completion of a process as a process completion condition candidate by analyzing a sample video, presents the detected process completion condition candidates, and records a combination of the process completion condition candidates selected by an operator as a process completion condition. Effect of the Invention

[0009] According to one aspect of the present invention, a process start / finish condition for accurately determining the start and completion of a process can be created. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a work management system according to a first embodiment of the present invention. [Diagram 2] 2 is a flowchart of a process executed by the work management system of the first embodiment. [Diagram 3] 11 is a flowchart of an area definition process according to the first embodiment. [Figure 4] 11 is a flowchart of an analysis information accumulation process according to the first embodiment. [Diagram 5] 1 is a flowchart of a process completion condition setting process in the first embodiment. [Figure 6] 1 is a flowchart of a process completion determination process in the first embodiment. [Figure 7] FIG. 11 is a diagram illustrating an example of a configuration of area definition information according to the first embodiment. [Figure 8] FIG. 13 is a diagram showing an example of a process completion condition in the first embodiment. [Figure 9] FIG. 4 is a diagram showing a configuration example of a coordinate detection result in the first embodiment. [Figure 10] FIG. 13 is a diagram showing an example of an area definition screen in the case where a moving object is present in the first embodiment. [Figure 11] FIG. 13 is a diagram showing an example of a person area definition screen according to the first embodiment. [Figure 12] FIG. 13 is a diagram showing an example of an area definition screen using a lattice frame in the first embodiment. [Figure 13] FIG. 13 is a diagram showing an example of a process completion condition setting screen in the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] <Example 1> FIG. 1 is a diagram showing an example of the configuration of a work management system according to an embodiment of the present invention.

[0012] The work management system of this embodiment includes a server 1 , a database 2 , an image display device 3 , and a camera 4 .

[0013] The server 1 is configured by a computer having a processor (CPU) 10, a memory 11, a storage device 12, a communication interface 13, and an input / output interface 14.

[0014] The processor 10 is a calculation device that executes programs stored in the memory 11. The processor 10 executes various programs to realize the functions of the various functional units of the server 1 (such as the image analysis unit 15, the process completion condition creation unit 16, the process completion judgment execution unit 17, the process completion condition display unit 18, and the process completion judgment result display unit 19). Note that some of the processes performed by the processor 10 executing the programs may be executed by other calculation devices (for example, hardware such as ASIC and FPGA).

[0015] The memory 11 includes a ROM, which is a non-volatile storage element, and a RAM, which is a volatile storage element. The ROM stores unchanging programs (e.g., BIOS) and the like. The RAM is a high-speed, volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the processor 10 and data used when the programs are executed.

[0016] The storage device 12 is, for example, a large-capacity non-volatile storage device such as a magnetic storage device (HDD) or a flash memory (SSD). The storage device 12 also stores data used by the processor 10 when executing a program, and the program executed by the processor 10. That is, the program is read from the storage device 12, loaded into the memory 11, and executed by the processor 10 to realize each function of the server 1.

[0017] The communication interface 13 is a network interface device that controls communications with other devices (such as the database 2, the image display device 3, the camera 4, and the control device 5) in accordance with a predetermined protocol.

[0018] The input / output interface 14 is an interface to which input devices such as a keyboard and a mouse are connected to receive input from an operator, and which is connected to output devices such as a display device and a printer (not shown) to output the results of program execution in a format that can be viewed by the operator. Note that the image display device 3 connected to the server 1 may provide the input device and the output device. In this case, the server 1 may have a web server function, and the user terminal may access the server 1 using a predetermined protocol (e.g., http).

[0019] The programs executed by the processor 10 are provided to the server 1 via a removable medium (such as a CD-ROM or a flash memory) or a network, and are stored in a non-volatile storage device 12, which is a non-transitory storage medium. For this reason, the server 1 should preferably have an interface for reading data from the removable medium.

[0020] Server 1 is a computer system configured on one physical computer, or on multiple logically or physically configured computers, and may run on a virtual computer constructed on multiple physical computer resources. For example, multiple programs that realize the functions of server 1 may each run on a separate physical or logical computer, or multiple programs may be combined to run on a single physical or logical computer.

[0021] The image analysis unit 15 analyzes the video data. The process completion condition creation unit 16 sets the process completion conditions using the sample video. The process completion judgment execution unit 17 judges whether a process has started or been completed using the real-time video. The process completion condition display unit 18 generates display data for displaying the set process completion conditions. The process completion judgment result display unit 19 generates display data for displaying the set process completion judgment result (see FIG. 13).

[0022] The database 2 is configured as a storage system having a control device, a storage device, and a communication interface. The server 1 and the database 2 may be configured as separate devices or as a single device. The database 2 has an area definition storage unit 21 that stores data used by the server 1 when executing a program, a detection coordinate storage unit 22, and a process completion condition storage unit 23. The area definition storage unit 21 stores coordinate information. The detection coordinate storage unit 22 stores image analysis results. The process completion condition storage unit 23 stores process completion conditions 610 (see FIG. 8).

[0023] The image display device 3 is a display device that outputs the results of calculations performed by the server 1. The image display device 3 may have the above-mentioned input interface (for example, a touch panel superimposed on a display screen). The image display device 3 and the server 1 may be connected directly (for example, via a serial interface) or via a network.

[0024] The camera 4 may be an imaging device that captures two-dimensional images, or an imaging device that can obtain the distance to a subject (for example, a distance imaging camera). The camera 4 has an imaging element configured with a CMOS sensor or a CCD, and an image processing unit that processes the image data captured by the imaging element, and captures images of people and objects at a work site managed by the work management system. The camera 4 and the server 1 may be connected directly (for example, via a serial interface) or via a network.

[0025] FIG. 2 is a flowchart of the process executed by the work management system.

[0026] The process executed by the server 1 is roughly divided into a process (S10) in which the process completion condition creation unit 16 sets the process completion condition using a sample video, and a process (S20) in which the process completion judgment execution unit 17 judges the start or completion of the process using a real-time video. The process completion judgment process (S20) will be described later with reference to FIG. 6.

[0027] In the process using the sample video (S10), first, the process completion condition creation unit 16 executes an area definition process (S11) to receive designation of an area to be determined as a target for completion of arrival, which is a range within the frame to be determined as a target for completion of arrival. Details of the area definition process will be described later with reference to FIG. 3. Then, the process completion condition creation unit 16 executes an analysis information accumulation process to accumulate analysis information of the designated area (S12). Details of the analysis information accumulation process will be described later with reference to FIG. 4. Then, the process completion condition creation unit 16 executes a process completion condition setting process to set the process completion condition (S13). Details of the process completion condition setting process will be described later with reference to FIG. 5.

[0028] FIG. 3 is a flowchart of the area definition process (S11).

[0029] In the area definition process, the process completion condition creation unit 16 judges whether a moving object is included in the sample video according to the operator's designation (S110). If the moving object is included in the sample video, the moving object is recognized using an object recognition model, and a predetermined area within the area (e.g., bounding box) in which the moving object is recognized is specified as the analysis target range (S111). For example, in the area definition screen 1100 (see FIG. 10), the operator designates a candidate for the arrival completion determination target area 1121 in the bounding box 1120 in which the moving object is recognized in the frame image. At this time, it is preferable that the candidate for the arrival completion determination target area 1121 is displayed in coordinates with a predetermined point (e.g., the upper left vertex) of the bounding box 1120 as the origin. Even if the location of the moving object changes in the frame image, the position of the moving object in the bounding box 1120 does not change substantially. Therefore, by designating the relative position in the bounding box 1120, rather than the position in the frame image, the desired position of the moving object can be accurately designated as the candidate for the arrival completion determination target area 1121.

[0030] The object recognition model used in step S111 can be a machine learning model trained on images of objects used by the work management system to determine process completion, and an appropriate model can be selected for each object on the area definition screen 1100.

[0031] Thereafter, the process completion condition creating unit 16 acquires coordinate information of objects from all frame images of the sample video, and stores the coordinate information of the objects for each frame image in the coordinate detection result 620 (S112). In step S112, the coordinate information of all objects such as people, objects, colors, and object number counts is recorded in the coordinate detection result 620.

[0032] Thereafter, the process completion condition creating unit 16 sets an effective time period, which is a time period for analyzing images in the sample moving image (S113). The effective time period is input by the operator, for example, on the process completion condition setting screen shown in FIG.

[0033] The process after step S113 branches depending on whether the method of selecting area definition candidates for image analysis is a measurement system or a classification system. The operator specifies whether the method is a classification system or a measurement system. The classification system is a process of extracting areas where many objects are detected as a result of object clustering as candidates for areas for completion determination, and is specified when selecting the number of groups in the area definition support areas 1270, 1370 of the area definition specification screen (FIGS. 11, 12). The measurement system is a process other than the classification system, and is a process of extracting areas as candidates for completion determination based on the magnitude of change in pixels of the grid, and is specified when selecting the minimum number of stay frames in the area definition support areas 1270, 1370 of the area definition specification screen (FIGS. 11, 12).

[0034] When the method for selecting area definition candidates for the image analysis target is a measurement system, the process completion condition creation unit 16 divides each frame into grids of a predetermined size and calculates for each grid (S114). A grid in which the rate of change is equal to or greater than a predetermined threshold is extracted as an area definition candidate (S115). For example, it is preferable to compare the brightness and color changes of pixels in the grid with predetermined thresholds and extract a grid in which the change of pixels exceeds the predetermined threshold.

[0035] On the other hand, when the method for selecting area definition candidates for the image analysis target is a classification system, the process completion condition creation unit 16 classifies a collection of objects into n groups (S116), and extracts the areas of the collection classified into the groups as area definition candidates (S117).

[0036] After the area definition candidates are extracted, the process completion condition creation unit 16 displays the area definition candidates on the area definition screen (S118). Then, the process completion condition creation unit 16 accepts the area definition by the operator and stores the position information of the specified area in the area definition information 600 (S119). For example, if the object to be grouped is a person, the area extracted by the classification processing is displayed on the person area definition screen 1200 (see FIG. 11), and the position information of the process completion determination area specified by the operator among the area candidates is stored in the area definition information 600. The area extracted by the measurement processing is displayed on the lattice frame area definition screen 1300 (see FIG. 12), and the position information of the process completion determination area specified by the operator with reference to the area candidates is stored in the area definition information 600.

[0037] FIG. 10 is a diagram showing an example of an area definition screen 1100 when a moving object is present.

[0038] The area definition screen 1100 shown in FIG. 10 includes a frame image display area 1110, a bounding box display area 1120, a moving object detection model designation area 1130, and an area size designation area 1140. The frame image display area 1110 displays a frame image used for area definition. The bounding box display area 1120 displays an enlarged bounding box in which an object is detected in the frame image displayed in the frame image display area 1110. The bounding box display area 1120 displays a specified arrival completion determination target area 1121. The moving object detection model designation area 1130 is an area in which options for machine learning models used for process arrival completion determination are displayed. The area size designation area 1140 displays the size and position of the specified arrival completion determination target area. The operator can change the size and position of the arrival completion determination target area by modifying the size and position values ​​of the arrival completion determination target area.

[0039] The area definition screen 1100 may include a recognition object designation area for selecting an object to be recognized, and a linked PLC designation area for designating a control device that controls a moving object to be detected.

[0040] FIG. 11 is a diagram showing an example of a person area definition screen 1200. As shown in FIG.

[0041] The person area definition screen 1200 includes an actual image display area 1210, candidates for the arrival / completion determination target area 1220, a property setting area 1230, a "Save definition and analyze" button 1240, a "Save definition" button 1250, a "Back" button 1260, an area definition support area 1270, and a "Display candidate areas" button 1280. The actual image display area 1210 displays a frame image used for area definition. The arrival / completion determination target area 1220 is an area where the start or completion of a process is determined by detecting an object. The property setting area 1230 is an area for setting attributes of the specified process arrival / completion determination area.

[0042] An area is defined by specifying a rectangle in the actual image display area 1210 using a mouse or touch panel. For each defined area, the area size and area position in the property setting area 1230 are automatically displayed. Also, properties such as the object to be analyzed in the defined area are specified. In the area definition support area 1270, conditions are set for analyzing a sample video and extracting candidates for the arrival completion determination target area 1220. In the area definition support area 1270, when the minimum number of stay frames is selected, the selection method for the measurement-based area definition designation candidate is set, and when the number of groups is selected, the selection method for the classification-based area definition designation candidate is set. Using the candidates displayed in the "Show candidate areas" button 1280 as a reference, an area can be defined by specifying a rectangle using a mouse or touch panel.

[0043] The “Save definition” button 1250 is operated to record the position of the arrival / completion determination target area 1220 selected by the operator in the process arrival / completion condition 610 .

[0044] The “Save definition and analyze” button 1240, in addition to the function of 1250, is operated to analyze the presence or absence and coordinates of the object specified in the property setting area 1230 for the specified area, detect whether an object exists in the area, and record the presence or absence in the coordinate detection result 620.

[0045] FIG. 12 is a diagram showing an example of an area definition screen 1300 using a lattice frame.

[0046] The area definition screen 1300 using a lattice frame includes an actual image display area 1310, an area for determining whether or not arrival has been completed 1320, a property setting area 1330, a “Save definition and analyze” button 1340, a “Save definition” button 1350, a “Back” button 1360, an area definition support area 1370, and a “Display candidate areas” button 1380.

[0047] The actual image display area 1310 displays a frame image used for area definition. Images are omitted from the illustration. The completion / completion determination target area 1320 is an area where the start or completion of a process is determined by detecting an object. The property setting area 1330 is an area for setting attributes of the specified process completion / completion determination area.

[0048] Areas are defined by dragging and dropping the actual image display area 1310 with the mouse. For each defined area, the area size and area position are automatically displayed in the property setting area 1330. In addition, properties such as the object to be analyzed in the defined area are specified. In the area definition support area 1370, conditions are set for analyzing a sample video and extracting candidates for the arrival completion determination target area 1320. Areas can be defined by dragging and dropping, referring to the candidates displayed with the "Show candidate areas" button 1380.

[0049] The “save definition” button 1350 is operated to record the position of the arrival / completion determination target area 1320 selected by the operator in the process arrival / completion condition 610 .

[0050] The “Save definition and analyze” button 1340, in addition to the function of 1350, is operated to analyze the presence or absence and coordinates of the object specified in the property setting area 1330 for the specified area, detect whether an object exists in the area, and record the presence or absence in the coordinate detection result 620.

[0051] FIG. 4 is a flowchart of the analysis information accumulation process (S12).

[0052] In the analysis information accumulation process, the image analysis unit 15 analyzes the image of the specified area recorded in the area definition information 600 (S120), and stores the analysis result in the detection coordinate storage unit 22.

[0053] Thereafter, when the object associated with the area is a person and the area is a human area, the image analysis unit 15 executes a person recognition process and records the recognition result in the human area detection presence / absence column of the coordinate detection result 620 (S121). In the person recognition process of step S121, for example, a person in the frame image is recognized by a person recognition model that has been machine-learned from a person image, and if the center coordinates of a bounding box in which the person is recognized are within the arrival completion determination target area, it is determined that a person has been detected within the arrival completion determination target area.

[0054] Furthermore, when the object associated with the area is an object and the area is an object area, the image analysis unit 15 executes an object detection process and records the detection result in the object area detection presence / absence column of the coordinate detection result 620 (S122). In the object detection process of step S122, for example, a predetermined object in a frame image is recognized by an object recognition model that is machine-learned from an image of a predetermined object used by the work management system for process completion judgment, and it is determined that an object has been detected in the completion judgment area if the central coordinates of a bounding box in which the object is recognized are within the completion judgment area or if the similarity between the color of a pixel in the process completion judgment area and the color of the predetermined object is determined to recognize the presence or absence of an object.

[0055] Furthermore, if the object associated with an area is a color and the area is a color area, the image analysis unit 15 executes a color recognition process and outputs the detection result as a coordinate detection result 620. The color recognition process in step S123 records the color object detected in the color area detection column (S123). For example, if the number of pixels in a predetermined color range in the area for determining whether or not the item has been delivered is equal to or greater than a predetermined threshold, it is determined that a color object has been detected in the area for determining whether or not the item has been delivered.

[0056] Furthermore, when the object associated with the area is an object to be counted and the area is an object count area, the image analysis unit 15 executes an object count recognition process and records the object count result in the object count detection presence / absence column of the coordinate detection result 620 (S124). In the object count recognition process of step S124, for example, the work management system recognizes the number of predetermined objects in the arrival completion determination target area by an object recognition model that machine-learned images of the predetermined objects to be counted, and if the number of objects having the center coordinates of the bounding box in which the object is recognized in the arrival completion determination target area is greater than a predetermined threshold, it is determined that the predetermined number of objects has been detected. Note that the number of objects that have passed through the arrival completion determination target area may be recognized to determine that the predetermined number of objects has been detected.

[0057] FIG. 5 is a flowchart of the process completion condition setting process (S13).

[0058] In the process completion condition setting process, the process completion condition creating unit 16 acquires the object detection result within the completion determination target area for each frame from the coordinate detection result 620 (S130).

[0059] The operator determines whether or not the process completion condition creation unit 16 wants to detect multiple types of objects (S131). When detecting multiple objects, the process completion condition creation unit 16 determines the process type, process sequence, and arrival completion conditions for each object type (for example, product type). For this reason, the operator specifies the work object for which the process completion conditions are to be created (S132). Since a work process is set for each work object, it is necessary to specify the work object first when multiple objects are detected in a mixed flow.

[0060] Thereafter, the process start / completion condition creation unit 16 sets a reference frame for the start / completion of the process. For example, an assumed start frame and end frame may be provisionally set (S133). Since the frame where the presence or absence of a person or object changes marks the start or end of the process, the presence or absence of candidates for people or objects is scanned and displayed as a list. The operator can easily find the dividing frame by looking at the displayed list. For example, if it can be determined that a certain frame is a specific process, the frame can be designated as the reference frame to select a candidate frame for the start of that process or a candidate frame for the end of the previous process.

[0061] Then, the process completion condition creating unit 16 extracts the object detection results of a predetermined number (n) of frames before and after the reference frame (S134).

[0062] Then, the process completion condition creating unit 16 identifies frames in which there is a change in each of the areas subject to the completion judgment within the range of frames before and after the reference frame (S135). For example, it is preferable to compare the change in brightness and color of pixels in the areas subject to the completion judgment with a predetermined threshold value and identify frames including the areas subject to the completion judgment in which the change in pixels exceeds the predetermined threshold value, or frames in which a person moves into or leaves a specific area.

[0063] Then, the process completion condition creating unit 16 displays the identified frames as process start frame candidates or process completion frame candidates on the process completion condition setting screen (FIG. 13), and prompts the operator to select the process start timing or process completion timing (S136).

[0064] Then, the process arrival completion judgment result display unit 19 displays the process arrival completion condition setting screen 1000 (see FIG. 13). The process arrival completion condition creation unit 16 sets the arrival completion judgment target area and the detected object of the frame at the selected timing as the process arrival completion condition, and writes it to the process arrival completion condition 610 (S137). The operator can also check the status of various areas for each frame and manually specify the process arrival completion condition. For example, the start condition for process 1 (1023) can be specified as "when there is a person in the worker 2 area and an object exists in the object area 1."

[0065] Then, the process completion condition that has been set and the judgment result based on the process completion condition are displayed on a process completion condition setting screen 1000 shown in FIG. 13 (S138).

[0066] FIG. 13 is a diagram showing an example of a process completion condition setting screen 1000.

[0067] FIG. 13 shows an example of a process completion condition setting screen 1000 for setting the process completion condition for work object A. The process completion condition setting screen 1000 displays a frame display area 1010, a process completion condition setting area 1020, and a process display area 1030, with the horizontal axis representing frames (time). In the frame display area 1010, frame images of the video to be analyzed are displayed along the time axis. The displayed frame images may be a portion of frame images thinned out at a predetermined time interval. In the process completion condition setting area 1020, a non-effective time period 1021, a valid time period 1022, and a detection result 1024 are displayed. The non-effective time period 1021 is a time period during which the process completion condition is not judged. The valid time period or non-effective time period is defined as the elapsed time from the start of the process in area units in the area definition screen 1200 in FIG. 11 (omitted in FIG. 11). For valid time period 1022, information on whether or not a person is present in the worker 2 area during this time period becomes valid as a process completion condition. Conversely, for non-valid time period 1021, information on whether or not a person is present in the worker 1 area during this time period becomes invalid as a process completion condition. In other words, even if a person is present, the non-valid time period is determined to be "not present." The process display area 1030 displays the results of dividing the process into frame images according to the process completion condition. The detection result 1024 displays the time period in which an object was detected under the process completion condition.

[0068] FIG. 6 is a flowchart of the process of determining whether a process has been completed.

[0069] The process completion determination execution unit 17 acquires the on-site video to be analyzed (S210). The on-site video acquired in step S210 may be real-time video data of the work site captured by the camera 4 or recorded on-site video data.

[0070] Then, the process arrival completion judgment execution unit 17 identifies the work object for which the process arrival completion is to be judged, and selects a necessary condition from the process arrival completion conditions 610 (S211). For example, the operator may select the work object by viewing the on-site video data acquired in step S210, and select the process arrival completion condition for the selected work object. When multiple types of objects are targeted, such as in a mixed production line, the system automatically determines the type of object and selects the process arrival completion condition. For example, the work management system recognizes a specified object in a frame image using an object recognition model that has been machine-learned to learn an image of a specified object used for process arrival completion judgment.

[0071] Then, the process completion determination executing unit 17 analyzes the on-site video data acquired in step S210 (S212). The video data analysis in step S212 may perform person recognition, object recognition, etc., and execute the same process as the analysis information accumulation process (FIG. 4).

[0072] Then, the process completion / start determination execution unit 17 compares the selected process completion / start conditions with the image data analysis results to determine the start and completion of the process (S213).

[0073] The time when the frame when the start of the process was judged was photographed and the time when the frame when the completion of the process was judged was photographed are measured as the working time of the process, and the measured working time is displayed (S214). The frame number when the process started, the frame number when the process was completed, the time when the process started, the time when the process was completed, and the process number may be displayed together with the measured working time.

[0074] FIG. 7 is a diagram showing an example of the configuration of the area definition information 600. As shown in FIG.

[0075] The area definition information 600 is data stored in the database 2, and stores coordinates designated as the process completion area. The area definition information 600 includes the position of the process completion area (top left X coordinate, top left Y coordinate, bottom right X coordinate, bottom right Y coordinate, front Z coordinate, depth Z coordinate), a valid time period, the type of the process completion area (person area, object area, color area, object count area), and an area number. The position of the designated process completion area is determined, for example, by the top left X coordinate, top left Y coordinate, bottom right X coordinate, bottom right Y coordinate, front Z coordinate, and depth Z coordinate, but may be determined by other coordinates. The valid time period is the time when the information obtained by analyzing the image in the sample video becomes valid as a process completion condition. The type of the process completion area is the type of object to be detected in the process completion area, and is either a person, an object, a color, or an object count.

[0076] FIG. 8 is a diagram showing an example of the configuration of the process completion condition 610.

[0077] The process completion condition 610 is data stored in the process completion condition storage unit 23 of the database 2, and stores the process completion condition. The process completion condition 610 includes a product type, a process number, a completion determination target area, an AND condition, and an OR condition. The process completion condition storage unit 23 may include an object type and a process completion condition.

[0078] The process number is the number of the process identified by the process completion condition. The product type is the type of product manufactured in the process identified by the process completion condition. The arrival completion judgment target area is identification information of the arrival completion judgment target area used in the process completion condition. The AND condition and the OR condition are operators that combine multiple conditions to judge the process. The AND condition indicates that both the condition in the column and the condition in the next column must be met, and the OR condition indicates that either the condition in the column or the condition in the next column must be met.

[0079] FIG. 9 is a diagram showing an example of the configuration of the coordinate detection result 620. As shown in FIG.

[0080] The coordinate detection result 620 is data stored in the database 2, and stores the coordinates of an object detected by image analysis. The coordinate detection result 620 includes each item of a video frame number, a position of a detected object (upper left X coordinate, upper left Y coordinate, lower right X coordinate, lower right Y coordinate, front Z coordinate, rear Z coordinate, center X coordinate, center Y coordinate), elapsed time, whether or not a person area 1 is detected, whether or not a person area 2 is detected, and whether or not an object area is detected. Although not shown in the figure, the coordinate detection result 620 may include items of whether or not a color is detected and the number of objects counted, depending on the type of the detected object.

[0081] The video frame number is a number for uniquely identifying a video frame in which an object is detected. The position of the detected object is determined, for example, by the upper left X coordinate, the upper left Y coordinate, the lower right X coordinate, the lower right Y coordinate, the front side Z coordinate, the back side Z coordinate, the center X coordinate, and the center Y coordinate, but may be determined by other coordinates. These coordinates may use coordinates in a real three-dimensional space or coordinates based on the shooting point. The elapsed time is the elapsed time from the start time of the video. The presence or absence of human area 1 detection and the presence or absence of human area 2 detection are flags indicating that a person has been detected. The presence or absence of object area detection is a flag indicating that an object has been detected. The presence or absence of color detection is a flag indicating that a predetermined color has been detected in the process completion area. The object count number is the number of predetermined objects counted in the process completion area.

[0082] Although the embodiment of the present invention has been described above, and the detection of a specific object (person, thing), the detection of an object of a specific color, and the counting result of a specific object (thing) are used as the conditions for the completion of a process, sounds detected at the work site may also be used as the conditions for the completion of a process. For example, the end of a process may be detected by the voice of a worker saying "Done" or "Good".

[0083] In this way, in the work management system according to the embodiment of the present invention, any detectable phenomenon such as a person, an object, a color, a quantity of an object, or a sound can be adopted as a process completion condition.

[0084] In the above-described embodiment, one process completion condition is constituted by any one of the detection of a person, the detection of a specified object, the detection of a specified color, and counting a specified number of specified objects. However, one process completion condition may be constituted by a combination of the detection of a person, the detection of a specified object, the detection of a specified color, and counting a specified number of specified objects.

[0085] As described above, the work management system according to the embodiment of the present invention sets a process completion condition that combines the conditions of work in a process, so that it is possible to determine the start and completion of a process in which multiple work elements are intertwined. In particular, it is possible to determine the start and completion of a process even if there are multiple work locations and multiple work objects scattered within the process. In addition, by setting a non-valid time period, it is possible to properly determine the completion of a process even if the position of a person or object does not change and the process moves to the next process. Furthermore, by setting the completion determination target area using relative coordinates based on a bounding box that recognizes a moving object, it is possible to properly determine the completion of a process even if the work object is moving on the production line.

[0086] The present invention is not limited to the above-described embodiments, and includes various modified examples and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. Furthermore, a part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, a part of the configuration of each embodiment may be added, deleted, or replaced with another configuration.

[0087] In addition, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in hardware, for example by designing some or all of them as an integrated circuit, or may be realized in software by a processor interpreting and executing a program that realizes each function.

[0088] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0089] In addition, the control lines and information lines shown are those considered necessary for the explanation, and do not necessarily show all the control lines and information lines necessary for implementation. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0090] 1 Server 2 Database 3 Image display device 4. Camera 5. Control device 10 Processors 11. Memory 12 Storage device 13 Communication Interface 14 Input / Output Interface 15 Image Analysis Unit 16 Process completion condition creation department 17 Process Completion Judgment Execution Unit 18 Process completion condition display section 19 Process completion judgment result display section 21 Area definition storage section 22 Detection coordinate storage unit 23 Process completion condition storage unit 600 Area definition information 610 Process completion conditions 620 Coordinate detection result

Claims

1. A work management system, The computer is configured by a computing device that executes predetermined computational processing and a storage device that can be accessed by the computing device, the computing device comprises a process completion condition creation unit that creates a process completion condition for determining whether a process is started or completed due to a change in an object related to the process, The process completion condition creation unit Accept the designation of the arrival / completion judgment target area for judging the start or completion of the process in the image, By analyzing the sample video, changes in objects at the start or completion of a process are detected as candidate conditions for the process completion. Present the detected candidate process completion conditions; A work management system characterized in that a combination of the candidate process completion conditions selected by an operator is recorded as a process completion condition.

2. The work management system according to claim 1, A work management system that provides an interface that allows the user to select at least one of the following as the process completion conditions: detection of a person in the area subject to completion determination, detection of a specified object, detection of a specified color, and counting a specified number of specified objects.

3. The work management system according to claim 1, The work management system is characterized in that the process completion condition creation unit is capable of setting an ineffective time for the process completion condition candidate.

4. The work management system according to claim 3, The work management system is characterized in that the process completion condition creation unit detects process completion condition candidates during a time period other than the non-effective time.

5. The work management system according to claim 1, The process completion condition creation unit Displaying an area showing the detected object in the sample video; Accept the designation of an area for determining completion within the displayed area; A work management system characterized in that the specified area for determining completion of delivery is recorded using relative coordinates within the displayed area.

6. The work management system according to claim 1, The process completion condition creation unit Identifying a frame in which there is a change in the object in the arrival completion determination target area within a range of at least a predetermined number of frames before and after the set reference frame; A work management system characterized in that a change in an object in the specified frame is detected as a candidate for a process start / completion condition.

7. A work management system according to claim 1, The process completion condition creation unit analyzing the sample video to detect changes in the sample video; A work management system characterized in that the area in which the change is detected is presented as a candidate area for completion determination.

8. A work management system according to claim 1, The process completion condition creation unit generating a set of detected objects by analyzing the sample video; A work management system characterized in that an area including the generated set is presented as a candidate area for completion determination.

9. A work management system according to claim 1, A work management system characterized by comprising a process completion determination execution unit that compares video data captured by a camera with the recorded process completion conditions and determines the start and completion of the process in the video data.

10. A work management system according to claim 9, The process completion determination execution unit The sample video is compared with the generated process completion conditions to determine the start and completion of the process in the sample video; A work management system that generates display data for displaying frame images of the sample video in association with the results of the judgment.

11. A work management method executed by a work management system, the work management system is configured by a computer having a computing device that executes predetermined computational processing and a storage device that can be accessed by the computing device, The work management method includes a step of creating a process completion condition in which a computing device creates a process completion condition for determining whether a process is started or completed, In the procedure for creating the process completion conditions, The calculation device receives designation of a completion / completion determination target area in the image for determining whether a process has started or completed; The calculation device analyzes the sample video to detect changes in the object at the timing of starting or completing the process as candidate conditions for the process completion; The calculation device presents the detected candidate process completion conditions, A work management method characterized in that a calculation device records the combination of the candidate process completion conditions selected by the operator as the process completion conditions.

Images