Monitoring system and monitoring method

Abstract

This monitoring system monitors, using at least one camera, a work process in which a plurality of steps are performed on a target. The monitoring system comprises: an image acquisition unit that acquires image data indicating each of a plurality of monitoring images obtained by at least one camera capturing a plurality of steps; and a control unit that executes image recognition processing for recognizing the state of each step in the work process on the basis of the image data. On the basis of sensor information indicating the results of detecting the target in the work process, the control unit selects, from among the plurality of monitoring images, a monitoring image that will serve as input for the image recognition processing, and the control unit executes the image recognition processing on the selected monitoring image, without executing the image recognition processing on monitoring images not selected from among the plurality of monitoring images.

Description

Monitoring system and monitoring method

[0001] The present disclosure relates to a monitoring system and a monitoring method for monitoring a work process in which a plurality of processes are performed on an object using at least one camera.

[0002] Patent Literature 1 discloses an image processing device that aims to reduce the overall amount of calculation while suppressing a decrease in the processing accuracy of a specific object among multiple objects in image processing using images from multiple image capture units. In image processing to generate a 3D model of the object to be processed, this image processing device uses images captured by more image capture units for a specific object that is set in response to a user operation among the multiple objects, compared to images captured by other objects. In this way, the image processing device suppresses a decrease in the accuracy of the 3D model for the specific object that the user is focusing on.

[0003] Japanese Patent Application Laid-Open No. 2019-016161

[0004] The present disclosure provides a monitoring system and a monitoring method that can reduce the processing load of monitoring a work process using at least one camera.

[0005] A monitoring system according to one aspect of the present disclosure monitors a work process in which multiple processes are performed on an object using at least one camera. The monitoring system includes an image acquisition unit and a control unit. The image acquisition unit acquires image data representing multiple monitoring images captured by the at least one camera of the multiple processes. The control unit performs image recognition processing to recognize the state of each process in the work process based on the image data. The control unit selects a monitoring image to be used as input for the image recognition processing from among the multiple monitoring images based on sensor information indicating the detection result of the object in the work process. The control unit performs image recognition processing on the selected monitoring image without performing image recognition processing on monitoring images that were not selected from the multiple monitoring images.

[0006] These general and specific aspects may be realized by a system, a method, and a computer program, as well as combinations thereof.

[0007] According to the monitoring system and monitoring method of the present disclosure, it is possible to reduce the processing load of monitoring a work process using at least one camera.

[0008] FIG. 1 is a diagram showing an overview of a site monitoring system according to the first embodiment; FIG. 2 is a block diagram illustrating the configuration of a site monitoring PC in the site monitoring system; FIG. 3 is a block diagram illustrating the configuration of an analysis PC in the site monitoring system; FIG. 4 is a diagram illustrating the arrangement of cameras in the site monitoring system; FIG. 5 is a diagram illustrating the arrangement of sensors in the site monitoring system; FIG. 6 is a diagram illustrating a bottleneck in the site monitoring system;

[0009] Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed description of already well-known matters or redundant description of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the inventor(s) provide the accompanying drawings and the following description to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0010] (First embodiment) 1. Configuration A site monitoring system according to a first embodiment will be described as an example of a monitoring system in the present disclosure with reference to Fig. 1. Fig. 1 is a diagram showing an overview of a site monitoring system 1 according to this embodiment.

[0011] 1-1. System Overview As shown in FIG. 1, a site monitoring system 1 according to this embodiment includes multiple cameras 2-1, 2-2, and 2-3, a site monitoring PC 5, and a sensor 85. This system 1 is used to monitor the work processes performed by multiple workers W1, W2, and W3 at a site 6, such as a factory, and analyze the work efficiency of each worker W1 to W3. This system 1 also includes an analysis PC 4 that provides an analysis screen or the like used for analysis to a user 3, such as a manager or analyst at the site 6. In this system 1, sensor data output from a sensor 85 installed at the site 6 is transmitted to the site monitoring PC 5.

[0012] 1, a plurality of cells 8-1, 8-2, 8-3, etc. are arranged in succession as work areas in which workers W1, W2, and W3 work, respectively, and work such as manufacturing a product 9 is performed. For example, in each of the cells 8-1 to 8-3, a plurality of steps in the work for each product 9 are performed sequentially, and upon completion of one step, the product 9 is sent to the cell corresponding to the next step. In this embodiment, the product 9 is an example of an object to which a plurality of steps in the work is performed, which is an example of a work process.

[0013] In this embodiment, cameras 2-1, 2-2, and 2-3 correspond to cells 8-1, 8-2, and 8-3, respectively, and are positioned so as to capture images of the range in which processes are performed in each cell at the site 6. Hereinafter, cameras 2-1 to 2-3 will also be collectively referred to as cameras 2. Cells 8-1 to 8-3 will also be collectively referred to as cells 8, and workers W1 to W3 will also be collectively referred to as workers W. Although not shown in FIG. 1 , there may be four or more cameras 2 and four or more cells 8. The images of each cell 8 taken by camera 2 are an example of multiple monitoring images in this embodiment.

[0014] The camera 2 of the system 1 repeats an image capturing operation at a predetermined cycle, for example, at a site 6, and generates image data representing the captured image obtained at each time of the image capturing cycle. The camera 2 is connected to the site monitoring PC 5, for example, so that the image data is transmitted to the site monitoring PC 5.

[0015] The site monitoring PC 5 of this system 1 uses image data from each camera 2 to recognize the progress of each process in each cell 8, and outputs various information to the analysis PC 4 to be used for analyzing the work efficiency of the worker W in each process. The progress of each process is an example of the state of each process in this embodiment. For example, from the information output by the analysis PC 4, the user 3 can analyze a bottleneck process that is the starting point of work delays, such as when the progress of a process performed by the worker W in a specific cell 8 delays subsequent processes. Furthermore, in this system 1, the user 3 can analyze the images of each cell 8 captured by each camera 2 to consider the causes of the bottleneck.

[0016] The site monitoring PC 5 and the analysis PC 4 of the system 1 are each configured as a personal computer (PC) as an example of an information processing device. The configurations of the site monitoring PC 5 and the analysis PC 4 will be described below with reference to FIGS. 2 and 3, respectively.

[0017] 1-2. Configuration of the site monitoring PC Fig. 2 is a block diagram illustrating the configuration of the site monitoring PC 5. The site monitoring PC 5 in Fig. 2 includes a control unit 50, a storage unit 52, an operation unit 53, a device interface 54, and an output interface 55. Hereinafter, the interface will be abbreviated as "I / F."

[0018] The control unit 50 includes, for example, a CPU or MPU that cooperates with software to realize predetermined functions, and controls the overall operation of the site monitoring PC 5. The control unit 50 reads data and programs stored in the memory unit 52, performs various arithmetic processing, and realizes various functions.

[0019] For example, the control unit 50 executes a program including a set of instructions for realizing the functions of the site monitoring PC 5. The program may be provided via a communication network such as the Internet, or may be stored on a portable recording medium. The control unit 50 may also include an internal memory as a temporary storage area for storing various data and programs.

[0020] The control unit 50 of this embodiment includes an image recognition unit 51 as a functional component that functions in cooperation with software. Based on image data, the image recognition unit 51 recognizes the progress of a work process (e.g., the behavioral state of a worker W performing the work) in the area depicted in the image represented by the image data. For example, the image recognition unit 51 uses image recognition using various machine learning techniques to output a category corresponding to the progress of the process captured in the input image, out of multiple categories that classify work states. For example, the multiple categories are set to classify the progress states into "Working" (in which the worker W is working) and "Waiting" (in which the worker W is waiting for the product 9 to be worked on), as well as "Absent" (in which the worker W is away from the cell 8).

[0021] The control unit 50 may be a dedicated electronic circuit designed to realize a predetermined function or a hardware circuit such as a reconfigurable electronic circuit, etc. The control unit 50 may be configured with various semiconductor integrated circuits such as a CPU, an MPU, a GPU, a GPGPU, a TPU, a microcomputer, a DSP, an FPGA, and an ASIC.

[0022] The storage unit 52 is a storage medium that stores programs and data necessary to realize the functions of the site monitoring PC 5. For example, the storage unit 52 is configured with an HDD, an SSD, or the like, and stores the above-mentioned programs, image data acquired from the cameras 2, and map data D0 that indicates the locations of the cells 8 and the cameras 2 at the site 6. For example, the map data D0 manages each cell 8 and the camera 2 that captures images of the process in each cell 8 in association with each other.

[0023] The storage unit 52 includes an area configured by a RAM such as a DRAM or an SRAM for temporarily storing (i.e., holding) data. The storage unit 52 may function as a work area for the control unit 50, or may include a storage area in the internal memory of the control unit 50.

[0024] The operation unit 53 is a general term for operation members that accept operations by the user 3. For example, the operation unit 53 is configured by any one or a combination of a keyboard, a mouse, a trackpad, a touchpad, buttons, switches, etc. The operation unit 53 acquires various information input by operations by the user 3.

[0025] The device I / F 54 is a circuit for connecting external devices such as the camera 2 and the sensor 85 to the site monitoring PC 5. The device I / F 54 performs data communication in accordance with a predetermined communication standard. Examples of the predetermined communication standard include USB, HDMI (registered trademark), IEEE 802.11, Wi-Fi (registered trademark), and Bluetooth (registered trademark). The device I / F 54 is an example of an input interface that the site monitoring PC 5 uses to acquire or receive various information from external devices. For example, the site monitoring PC 5 receives image data representing an image captured by the camera 2 and sensor data output from the sensor 85 via the device I / F 54. The device I / F 54 is an example of an image acquisition unit in the site monitoring PC 5 of this embodiment.

[0026] The output I / F 55 is a circuit for outputting information. For example, the output I / F 55 outputs various types of information to an external device such as the analysis PC 4 in accordance with a predetermined communication standard. For example, the predetermined communication standard includes USB, HDMI, IEEE802.11, Wi-Fi, Bluetooth, etc.

[0027] The above-described configuration of the site monitoring PC 5 is merely an example, and the configuration of the site monitoring PC 5 is not limited thereto. The site monitoring PC 5 may be configured with various computers including a server, and may perform data communication with the camera 2 and the sensor 85 at the site 6 via a communication network, for example, via the device I / F 54. The site monitoring PC 5 may include, for example, a display unit configured as a built-in display device, such as a liquid crystal display or an organic EL display, in addition to or instead of the output I / F 55. The monitoring method of this embodiment may also be executed in distributed computing.

[0028] In addition to or instead of the above configuration, the site monitoring PC 5 may have a configuration for communicating with external devices via a communication network. For example, the operation unit 53 may be configured to accept operations from external devices connected via the communication network. The output I / F 55 may be connected to the external devices via the communication network to transmit various information.

[0029] The input interface in the site monitoring PC 5 may be realized in cooperation with various software in the control unit 50, etc. The input interface in the site monitoring PC 5 may acquire various pieces of information by reading out the information stored in various storage media (e.g., the storage unit 52) ​​into the work area of ​​the control unit 50.

[0030] 3 is a block diagram illustrating the configuration of the analysis PC 4. The analysis PC 4 in FIG. 3 includes a control unit 40, a storage unit 42, an operation unit 43, a communication I / F 44, and a display unit 45.

[0031] The control unit 40 includes, for example, a CPU or MPU that cooperates with software to realize predetermined functions, and controls the overall operation of the analysis PC 4. The control unit 40 reads data and programs stored in the storage unit 42, performs various arithmetic processing, and realizes various functions.

[0032] For example, the control unit 40 executes a program including a set of instructions for realizing the functions of the analysis PC 4. The program may be provided via a communication network such as the Internet, or may be stored on a portable recording medium. The control unit 40 may also include an internal memory as a temporary storage area for storing various data and programs.

[0033] The control unit 40 may be a dedicated electronic circuit designed to realize a predetermined function or a hardware circuit such as a reconfigurable electronic circuit, etc. The control unit 40 may be configured with various semiconductor integrated circuits such as a CPU, an MPU, a GPU, a GPGPU, a TPU, a microcomputer, a DSP, an FPGA, and an ASIC.

[0034] The storage unit 42 is a storage medium that stores programs and data necessary to realize the functions of the analysis PC 4. The storage unit 42 is configured with RAM such as DRAM or SRAM and includes an area for temporarily storing (i.e., holding) data. The storage unit 42 may function as a work area for the control unit 40, or may include a storage area in the internal memory of the control unit 40.

[0035] The operation unit 43 is a general term for operation members that accept operations by the user 3. For example, the operation unit 43 is configured by any one or a combination of a keyboard, a mouse, a trackpad, a touchpad, buttons, switches, etc. The operation unit 43 acquires various information input by operations by the user 3.

[0036] The communication I / F 44 is a circuit for connecting the analysis PC 4 to a communication network via a wireless or wired communication line in accordance with a predetermined communication standard. Examples of predetermined communication standards include USB, HDMI, IEEE 802.11, Wi-Fi, Bluetooth, etc. For example, the communication I / F 44 connects to the site monitoring PC 5 via the communication network and receives information output by the output I / F 55 of the site monitoring PC 5. The communication I / F 44 is an example of an input interface that allows the analysis PC 4 to acquire or receive various information from external devices.

[0037] The display unit 45 is a display device configured with a liquid crystal display, an organic EL display, etc. For example, the display unit 45 displays information received from the site monitoring PC 5 via the communication I / F 44. The display unit 45 may also display various icons for operating the operation unit 43, information input from the operation unit 43, etc.

[0038] The above configuration of the analysis PC 4 is an example and is not limited thereto. For example, the analysis PC 4 may be configured as various computers such as a tablet terminal including a display unit or a smartphone. Furthermore, in addition to or instead of the above configuration, the analysis PC 4 may have a configuration that allows it to connect to an external information processing device without going through a communication network.

[0039] The input interface in the analysis PC 4 may be realized in cooperation with various software in the control unit 40, etc. The input interface in the analysis PC 4 may acquire various pieces of information by reading out the information stored in various storage media (e.g., the storage unit 42) into the working area of ​​the control unit 40.

[0040] 1-4. Arrangement of Cameras and Sensors The arrangement of the cameras 2 and sensors 85 connected to the site monitoring PC 5 described above in the site monitoring system 1 will be described with reference to FIGS.

[0041] Figure 4 is a diagram for explaining the arrangement of the cameras 2 in the site monitoring system 1. Figure 5 is a diagram for explaining the arrangement of the sensors 85 in the site monitoring system 1. Hereinafter, the vertical direction at the site 6 illustrated in Figure 1 will be referred to as the Z direction. Furthermore, two directions perpendicular to each other on a horizontal plane orthogonal to the Z direction will be referred to as the X direction and the Y direction, respectively. Furthermore, the +Z direction may be referred to as upward, and the -Z direction may be referred to as downward.

[0042] 4 shows an elevation view of a cell 8 at the work site 6 as viewed from the X direction. For example, each cell 8 is provided with a cell stand 83 on which a camera 2 is installed, a workbench 80 on which a worker W works on a product 9, and a buffer 81 on which work-in-progress items are temporarily placed before or after work is performed on the product 9 at the workbench 80. In the present system 1, each camera 2 is positioned so as to capture an image from above of the worker W working on the product 9 on the workbench 80 of the corresponding cell 8. The camera 2 may be positioned so as to capture an image including the buffer 81.

[0043] Fig. 5 shows a plan view of a cell 8 at a work site 6 as seen from above, corresponding to the viewpoint from a camera 2 arranged as shown in Fig. 4, for example. Each cell 8 at the work site 6 illustrated in Fig. 1 is provided with two buffers 81a and 81b, as shown in Fig. 5. The buffers 81a and 81b of each cell 8 are provided as storage areas for work-in-progress before and after work is performed on a product 9 at a workbench 80, with the cell 8 as the reference point.

[0044] In the work site 6 of this example, work in each process is carried out in each cell 8 while the product 9 is fed in the +X direction, one product at a time. In each cell 8, a worker W works on a product 9 picked up from a buffer 81a on a workbench 80, and places the product 9 after the work in a buffer 81b, thereby progressing the work in each process. For example, the buffer 81a of each cell 8 is shared with the cell 8 in which the process before the process in that cell 8 is carried out, and the buffer 81b is shared with the cell 8 in which the process after the process in that cell 8 is carried out.

[0045] In this system 1, the sensor 85 is installed on the buffer 81 (i.e., each buffer 81a, 81b) of the cell 8. For example, the sensor 85 is a proximity sensor that detects the proximity of an object such as a product 9 to the sensor 85 without contact, and is composed of a distance sensor, a magnetic field sensor, or the like. In this system 1, the sensor 85 repeats detection at a predetermined cycle and outputs sensor data indicating the detection result. For example, the sensor data indicates the presence or absence of an object detected in the proximity of the sensor 85 at each detection cycle.

[0046] 2. Operation The operation of the site monitoring system 1 and the site monitoring PC 5 configured as above will be described below.

[0047] In this system 1, as shown in Fig. 1, for example, the site monitoring PC 5 acquires image data of each cell 8 at the site 6 captured by the corresponding camera 2, and sensor data obtained by detecting the presence or absence of products 9 in the buffer 81 using the sensor 85 of the cell 8. Based on the acquired image data and sensor data, the site monitoring PC 5 outputs information regarding the progress of the process performed by the worker W in each cell 8, using image recognition processing or the like to recognize the progress of the process performed on a cell 8 basis. The site monitoring system 1 visualizes the information output from the site monitoring PC 5 using the analysis PC 4 so as to present it to the user 3.

[0048] For example, the site monitoring system 1 identifies a bottleneck process that causes a delay in work at the site 6 based on the progress status of each process recognized in the image of each cell 8 by the site monitoring PC 5, and displays a warning or the like for the identified process on the analysis PC 4. For example, in response to a user operation in response to such a warning or the like, the analysis PC 4 displays an image of the cell 8 corresponding to the process, captured by the camera 2 at the time the bottleneck process was performed.

[0049] 2-1. Bottleneck Determination and Processing Load Issues The following describes issues that arise when identifying a bottleneck cell 8 using image recognition processing in the site monitoring system 1, with reference to FIG.

[0050] Fig. 6 is a diagram illustrating a bottleneck in the site monitoring system 1. Fig. 6 shows cameras 2-1 to 2-3 at the site 6 illustrated in Fig. 1 and cells 8-1 to 8-3 imaged from above by the cameras 2-1 to 2-3, respectively. In the example of Fig. 6, the products 9 that have undergone the process in each cell 8 are placed in a buffer 81 between the cells 8 so that workers W1 to W3 sequentially perform the work process for each product 9 in the cells 8-1 to 8-3.

[0051] For example, in the scenario shown in Figure 6, workers W1 and W2 are working in cells 8-1 and 8-2, respectively. However, a product 9 is already placed in the buffer 81 between cells 8-1 and 8-2, so the product 9 to be worked on in cell 8-1 cannot be placed next. Also, for example, due to a situation in which worker W2's work in cell 8-2 is delayed, there are no products 9 in the buffer 81 between cells 8-2 and 8-3, and worker W3 is waiting in cell 8-3. In such a scenario, cell 8-2 is considered to be a bottleneck causing work delays.

[0052] The site monitoring PC 5 of this system 1 performs image recognition processing on the images from each of the cameras 2-1 to 2-3 to recognize the progress of the processes in cells 8-1 to 8-3. The site monitoring PC 5 identifies the bottleneck process in cell 8-2 as described above based on the combination of the recognized progress states. The site monitoring PC 5 of this embodiment uses relatively advanced image recognition processing using machine learning, etc., to accurately recognize the progress of the processes in each cell 8. However, if this image recognition processing occurs frequently, there is a concern that the processing load will increase.

[0053] For example, if the site monitoring PC 5 performs image recognition processing on images of all cells 8 at the site 6, there is a concern that the computational cost of the image recognition processing will be excessive. Furthermore, if there are a large number of cells 8 and cameras 2 at the site 6, the processing load may become enormous. This may result in an increase in the introduction cost of the present system 1, such as by introducing a high-performance PC as the site monitoring PC 5 or by implementing distributed processing using multiple PCs.

[0054] Therefore, the on-site monitoring PC 5 of the present system 1, based on the sensor data, limits the images of cells 8 taken by each camera 2 to be used as input for the image recognition process in order to omit image recognition processing for images of cells 8 that are deemed unrelated to the bottleneck. For example, in the scene shown in Figure 6, the sensor data shows that product 9 is detected in the buffer 81 on the cell 8-1 side, but not in the buffer 81 on the cell 8-3 side, which suggests that product 9 is stagnating in the process of cell 8-2. The on-site monitoring PC 5 of this embodiment identifies images of cells 8 that are deemed related to the bottleneck by detecting the process where such product 9 is stagnating based on the center data.

[0055] The process in cell 8-2 where the accumulation of product 9 was detected is considered to be relatively likely to be a bottleneck. In this case, the site monitoring PC 5 of this embodiment selects, from the images of each cell 8, the image of cell 8-2 corresponding to the process where the accumulation was detected, and the images of cells 8-1 and 8-3 where processes before and after the process in cell 8-2 are performed, respectively, as inputs for image recognition processing. Then, the site monitoring PC 5 performs image recognition processing on the selected three images from each cell 8, but does not perform image recognition processing on images that were not selected.

[0056] As described above, the site monitoring PC 5 of this embodiment selectively executes image recognition processing from images of multiple cells 8 in response to detection of a product 9 being stuck. Such a site monitoring PC 5 can accurately recognize the progress of processes in cells 8 while reducing the processing load of the image recognition processing, and can perform bottleneck determination to identify bottleneck processes by determining whether each process is a bottleneck. This allows the site monitoring PC 5 of this system 1 to reduce the processing load of image recognition processing, such as monitoring work in which multiple processes are performed on products 9, from images of the processes in each cell 8 taken by each camera 2, and can monitor the work efficiently.

[0057] 2-2. Operation of the Site Monitoring PC The operation of the site monitoring PC 5 in the site monitoring system 1 will be described with reference to FIGS.

[0058] 7 is a flowchart illustrating the operation of the site monitoring PC 5 in embodiment 1. For example, the processing of this flowchart is executed as processing for one cell 8, for each target cell 8 that is the processing target, at a predetermined processing cycle while work is being performed at the site 6. For example, the processing cycle is set to be equal to or longer than the cycle at which sensor data is output from the sensor 85. Each processing of this flowchart is executed by the control unit 50 of the site monitoring PC 5.

[0059] The on-site monitoring PC 5 of this embodiment detects the process in which the product 9 is stagnating for each cell 8 based on the period during which the product 9 is detected by the sensor 85 on the buffer 81 used in the process of each cell 8 (S1 to S3). For example, in the scene of Figure 6, the stagnation of the product 9 in the process of cell 8-2 is detected from the period during which the product 9 is detected on the buffer 81 between cells 8-1 and 8-2 and the period during which the product 9 is detected on the buffer 81 between cells 8-2 and 8-3.

[0060] First, the control unit 50 acquires sensor data corresponding to the detection results of the products 9 by the sensors 85 during a predetermined calculation period, for example, from each sensor 85 via the device I / F 54 (S1). For example, the calculation period is set to twice the processing cycle of this flowchart. The control unit 50 may acquire only the sensor data from the sensor 85 corresponding to the target cell among the sensors 85.

[0061] The control unit 50 calculates a delay index corresponding to the retention of the product 9 in the target cell during the calculation period based on the acquired sensor data and the period during which the product 9 was detected (S2). The delay index indicates the degree to which the product 9 is retained upstream of the process in each cell 8, and is set as an index that increases as the product 9 is retained. The process of calculating the delay index will be described in detail later.

[0062] The control unit 50 determines whether the delay index calculated for the target cell is greater than a retention threshold set as a delay index magnitude that may cause a delay in the work (S3). The control unit 50 detects a process in which the product 9 is retained by comparing the delay index with the retention threshold. The retention threshold is stored in advance in the memory unit 52, for example.

[0063] If the delay index is greater than the retention threshold (YES in S3), the control unit 50 selects three images, an image of the target cell and images of cells 8 corresponding to processes before and after the target cell, from the images of each cell 8 taken by each camera 2, as inputs for image recognition processing (S4). For example, the control unit 50 may store a flag or the like in the storage unit 52 indicating whether an image of each cell 8 has been selected in order to manage the selected images.

[0064] The control unit 50 acquires image data for the three images selected as input for the image recognition process, for example, from the camera 2 that captured each image via the device I / F 54 (S5). For example, the control unit 50 may identify the camera 2 associated with the cell 8 corresponding to each of the three images by referring to the map data D0 in the storage unit 52.

[0065] The control unit 50 performs image recognition processing on the selected three images based on the acquired image data, and recognizes the progress status of the three processes shown in each image (S6). For example, the control unit 50 inputs image data to the image recognition unit 51, and causes it to output one of the categories of "working," "waiting," and "not present" as the progress status of the process shown in the input image indicated by the image data.

[0066] 8 is a diagram for explaining the image recognition process (S6) in the site monitoring PC 5. Figures 8A to 8C illustrate images Im1 to Im3 captured from above by the camera 2 corresponding to one cell 8 at the site 6 shown in Figure 1. For example, when images Im1, Im2, and Im3 are input, the image recognition unit 51 recognizes the progress of the process by the worker W in each cell 8 and outputs each category: working, waiting, and absent.

[0067] The control unit 50 outputs information for visualizing the image selected in step S4 to the user 3, for example, in association with the images captured by each camera 2, to the analysis PC 4 via the output I / F 55 (S7). The information output in step S7 includes information for identifying the image selected from the images captured by each camera 2. Details of such visualization will be described later.

[0068] The control unit 50 determines whether the target cell is a bottleneck based on the progress of the three processes recognized by the image recognition process for the selected image (S8). In this embodiment, information that defines a bottleneck process determination rule by combining the progress of each process is stored in advance in the storage unit 52 of the site monitoring PC 5. The control unit 50 determines whether the target cell is a bottleneck by referring to this bottleneck determination information.

[0069] 9 is a diagram for explaining the bottleneck determination information D2 in the site monitoring PC 5. The bottleneck determination information D2 manages the progress status of three processes in association with the determination result of whether the process of the target cell corresponds to a bottleneck. The three processes correspond to the three images selected as input for the image recognition process, and are the process of the target cell and the pre-process and post-process that are carried out before and after the process, respectively.

[0070] 9, when the progress status of the preceding process is "waiting" or "working," the progress status of the target cell is "working," and the progress status of the subsequent process is "waiting," the target cell is determined to be a bottleneck. On the other hand, when the progress status of the target cell is "not present," the process of the target cell is not determined to be a bottleneck, regardless of the progress status of the preceding and subsequent processes.

[0071] For example, in the scene shown in Figure 6 above, when cell 8-2 is the target cell, cell 8-2 can be determined to be a bottleneck using bottleneck determination information D2 based on the progress status recognized when images of cells 8-1 to 8-3 are used as input for the image recognition process.

[0072] If the control unit 50 determines that the target cell is a bottleneck (YES in S8), it outputs information for visualizing the target cell as a bottleneck to the analysis PC 4 via the output I / F 55 (S9). Details of such visualization will be described later together with the visualization in step S7.

[0073] The control unit 50 waits until the next processing cycle (S10) and then repeats the processing from step S1 onwards. If the control unit 50 determines that the target cell is not a bottleneck (NO in S8), the control unit 50 proceeds to step S10 without visualizing the bottleneck (S9).

[0074] For example, even if a backlog of product 9 is detected in the process of the target cell (YES in S3), if the subsequent process is in the "in progress" state, the process of the target cell may still only be a sign of a bottleneck. In such a case, the bottleneck determination information D2 in FIG. 9 does not determine the process of the target cell as a bottleneck (NO in S8). Also, even if a backlog is detected (YES in S3), if the process of the target cell is in the "absent" state, the delay index may be high due to factors such as worker W being on break or an irregular process in which only the buffer 81 of the target cell is being used and no work is being performed. In such a case, the bottleneck determination information D2 does not determine the target cell as a bottleneck (NO in S8).

[0075] On the other hand, if the delay index of the target cell is equal to or less than the retention threshold (NO in S3), the control unit 50 skips the execution of the image recognition process such as step S6 (S11) without performing the processes of steps S4 to S9 described above. The control unit 50 then waits until the next processing cycle (S10) and repeats the processes from step S1 onward. In this way, using the delay index calculated based on the sensor data, image recognition process is not performed on images of cells 8 that are not selected as input for the image recognition process from among the images of multiple cells 8 at the site 6. This reduces the processing load compared to, for example, performing image recognition process on all cells 8.

[0076] According to the above-described operation of the on-site monitoring PC 5, if the delay index of the target cell calculated from the sensor data is greater than the congestion threshold (YES in S3), images of the cells 8 corresponding to the process of the target cell as well as the preceding and succeeding processes are selected as inputs for image recognition processing (S4). Then, image recognition processing is performed on the selected images (S6), and bottlenecks are identified based on the progress of the processes recognized in each image (S8). On the other hand, if the delay index is equal to or less than the congestion threshold (NO in S3), execution of this image recognition processing is skipped (S11).

[0077] As described above, image recognition processing can be selectively performed on images of a process in each cell 8 of the site 6 where a bottleneck is relatively likely due to the detection of a product 9 being stuck, as well as the processes before and after that process, enabling accurate bottleneck detection while reducing the processing load. For example, a false detection of a product 9 being stuck may occur due to factors such as the sensor 85 detecting an item other than the product 9, but even in this case, the accuracy of bottleneck detection can be improved by combining the progress states of the three processes recognized by image recognition processing.

[0078] When the above processes are executed synchronously for each cell 8, the selection results in each step S4 may be shared between two processes that have cells 8 in two consecutive steps as target cells. For example, when images of overlapping cells 8 are selected as inputs for image recognition processes between two processes, a common image recognition process may be executed for the overlapping selected cells 8, and the recognition results may be shared.

[0079] Furthermore, for example, if the target cell is cell 8 where the first process in work at site 6 is performed, the same processing as above can be performed in steps S4 to S7, using images of the process and the subsequent process of the target cell as inputs to the image recognition processing. Furthermore, if the target cell is cell 8 where the last process in work is performed, the same processing as above can be performed in steps S4 to S7, using images of the process and the previous process of the target cell as inputs to the image recognition processing.

[0080] 2-2-1 Calculation of Delay Indicator The calculation process for calculating the delay indicator of the target cell in step S2 of FIG. 7 will be described with reference to FIG.

[0081] FIG. 10 is a diagram illustrating the calculation process of the delay index in the site monitoring PC 5. In this example, the sensor data D1 indicates the presence or absence of a product 9 in the buffer 81 based on whether the product 9 is detected or not by the sensor 85. FIG. 10 illustrates four calculation periods F1, F2, F3, and F4 for each processing cycle of the process shown in FIG. 7. In the example of FIG. 10, the processing cycle is 2 / T, and each of the calculation periods F1 to F4 is set to a period twice the processing cycle.

[0082] The control unit 50 calculates the ratio of the period during which the product 9 was detected in the sensor data D1 during the calculation period as the detection rate of the product 9 in the buffer 81 using the following calculation formula 1. In the example of FIG. 9, the detection rate is "1" during calculation period F1, "1" during calculation period F2, "0.5" during calculation period F3, and "0" during calculation period F4. (Calculation formula 1) Detection rate of the product 9 in the buffer 81 = Period during which the product 9 was detected in the buffer 81 during the calculation period / Calculation period

[0083] In step S2, the control unit 50 calculates the detection rate for the buffer 81 related to the target cell using the above-described calculation formula 1. For example, in each cell 8 as shown in FIG. 5, the control unit 50 calculates the detection rate for each of the buffers 81a and 81b used before and after work is performed on the product 9 on the workbench 80.

[0084] The control unit 50 calculates the delay index of the cell 8 by subtracting the detection rate in the buffer 81b on the downstream side from the detection rate in the buffer 81a on the upstream side of the process in the cell 8 using the following calculation formula 2: Delay index of the cell 8 = Detection rate in the buffer 81a - Detection rate in the buffer 81b (Calculation formula 2)

[0085] According to the above calculation process, the delay index is calculated as an index whose value increases as the detection rate of products 9 in buffer 81a upstream of the process in cell 8 and as the detection rate of products 9 in buffer 81b downstream of the process increases. In this way, the delay index can be calculated so as to indicate the degree to which products 9 are stagnating upstream of the process in each cell 8.

[0086] 2-2-2 Visualization The visualization in steps S7 and S9 in FIG. 7 will be described with reference to FIG.

[0087] 11 is a diagram illustrating an analysis screen in the site monitoring system 1 of this embodiment. For example, the analysis PC 4 of this system 1 displays an analysis screen on the display unit 45 for the user 3 to analyze the work efficiency and the like at the site 6 based on the information output from the site monitoring PC 5 in steps S7 and S9.

[0088] The control unit 40 of the analysis PC 4 first displays, as an analysis screen, an operation timeline 70 on the display unit 45, which indicates whether or not a task is being performed at each time for the worker W in each cell 8. For example, the operation timeline 70 shown in FIG. 11 is generated based on the recognized progress status for each process in cells 8-1 to 8-3 when a stagnation is detected in the process in cell 8-2 where worker W2 is working (YES in S3). For example, in step S9, the site monitoring PC 5 outputs, via the output I / F 55 to the analysis PC 4, in addition to the generated operation timeline 70, information indicating the cell 8 (or the corresponding worker W) of the process determined to be a bottleneck in step S8 and the time.

[0089] The analysis PC 4 displays a warning icon 75, for example, superimposed on the work timeline 70, to warn of the occurrence of a bottleneck. In the example of FIG. 10 , the warning icon 75 is displayed for worker W2 in cell 8-2. The analysis PC 4 accepts an operation by user 3 to specify the warning icon 75 by clicking or the like on the operation unit 43, and displays a video including an image captured by the corresponding camera 2 at the time the bottleneck occurred for cell 8 or the like in the bottleneck process. The analysis PC 4 may obtain image data representing such a video directly from the camera 2 via the communication I / F 44, or may obtain the image data via the site monitoring PC 5.

[0090] In response to a user operation to designate a warning icon 75, the analysis PC 4 displays a viewer 71 for the user 3 to view videos of the bottleneck process, etc. The viewer 71 shown in Fig. 10 displays a layout diagram 72 showing the layout of the cells 8 as seen from above the site 6, a camera list 73 displaying thumbnails of images captured by each camera 2, and a player 74 for playing the videos.

[0091] 11 , for example, the display unit 45 of the analysis PC 4 displays a bottleneck display C1 indicating a bottleneck cell 8 and a selection display C2 indicating a cell 8 in an image selected as input for the image recognition process in different modes, superimposed on the layout diagram 72. The display unit 45 also highlights, in the camera list 73, the images of the bottleneck process among the images taken by each camera 2 using the bottleneck display C1, and the images selected as input for the image recognition process using the selection display C2. In the example of FIG. 11 , in response to the selection of three images in step S4 of FIG. 7 , the three cells 8 are highlighted in the layout diagram 72 using the selection display C2, and the three images are highlighted in the camera list 73.

[0092] For example, the analysis PC 4 displays a video of the bottleneck cell 8 on the player 74 in response to a user operation specifying the bottleneck display C1 using the operation unit 43. This makes it easier for the user 3 to view the video of the bottleneck cell 8 and to analyze the causes of the bottleneck, even when a large number of cells 8 are installed at the site 6, for example.

[0093] Furthermore, for example, in this embodiment, the analysis PC 4 displays a video corresponding to the specified image on the player 74 in response to a user operation specifying one of the three images highlighted by the selection display C2. The analysis PC 4 also displays a video of the cell 8 corresponding to the user operation specifying one of the three cells 8 highlighted by the selection display C2. The selection display C2 can alert the user 3 or the like to the image selected as input for the image recognition process related to the process in which the accumulation of products 9 was detected, and the cell 8 in that image. This makes it easier to analyze, by viewing a video, the cells 8 in the process related to the accumulation of products 9, which may be a sign of a bottleneck.

[0094] In the above example, the analysis PC 4 displays the bottleneck display C1 and the selection display C2 on both the layout plan 72 and the camera list 73, but they may be displayed on either the layout plan 72 or the camera list 73. Furthermore, such a display that visualizes the image selection results resulting from the detection of congestion may be displayed on the viewer 71 or the like even when the target cell is not determined to be a bottleneck in step S8 of FIG.

[0095] Furthermore, although an example of visualization on the analysis PC 4 has been described above, for example, a lamp or the like may be installed for each cell 8 at the site 6, and the bottleneck cell 8 and / or the cell 8 corresponding to the image selected upon detection of congestion may be visualized by lighting the lamp. Also, for example, the occurrence of a bottleneck or congestion of products 9 may be notified to a worker W in the cell using a speaker or the like.

[0096] (Modification of First Embodiment) In the above-described first embodiment, an example has been described in which the site monitoring system 1 visualizes images of bottleneck processes and images selected as inputs for image recognition processing. The site monitoring system 1 may further perform visualization in accordance with the processing capacity required for the image recognition processing. Such a modification of the first embodiment will be described with reference to FIGS. 12 and 13 .

[0097] 12 is a flowchart illustrating the operation of the site monitoring PC 5 in this modified example. In addition to the same processes (S1 to S11) as in the first embodiment, this flowchart includes processes (S41, S42) related to visualization according to the processing capacity of the site monitoring PC 5. For example, in the site monitoring PC 5 in this modified example, an upper limit on the number of cells 8, which corresponds to the number of images to which image recognition processing can be applied at one time, is preset as the upper limit of processing capacity and stored in the storage unit 52.

[0098] For example, the control unit 50 of the site monitoring PC 5 selects input for image recognition processing from images of each cell 8 (S4), and then determines whether the upper limit of processing capacity has been reached by the image recognition processing for the selected input images (S41). For example, when the control unit 50 applies image recognition processing to the images of the three cells 8 selected in step S4, the control unit 50 determines that the upper limit of processing capacity has been reached if the total number of cells 8 corresponding to the number of selected images in the processing of FIG. 12 executed for each cell 8 becomes greater than the upper limit. Whether an image for each cell 8 has been selected as input for image recognition processing may be managed by a common flag or the like across multiple processing processes that target each cell 8. For example, the control unit 50 may store the flag in the storage unit 52.

[0099] If the processing capacity has not reached the upper limit (NO in S41), the control unit 50 acquires the image data of the image selected in step S4 (S5), and performs image recognition processing (S6).

[0100] If the processing capacity reaches its upper limit (YES in S41), the control unit 50 outputs information to the analysis PC 4 for visualizing the cells 8 in the image selected in step S4 as unprocessed cells to which image recognition processing has not been applied to the selected image. For example, information identifying images of unprocessed cells among the images captured by each camera 2 is output. In this case, the control unit 50 skips execution of the image recognition processing (S11) and proceeds to step S10.

[0101] 13 is a diagram illustrating an analysis screen in the site monitoring system 1 of this modified example. In this modified example, the analysis PC 4 displays, on the analysis screen, in addition to the bottleneck display C1 and selection display C2 similar to those in the first embodiment, an unprocessed display C3 indicating an unprocessed cell in a manner different from the displays C1 and C2. For example, as shown in FIG. 13, the display unit 45 of the analysis PC 4 highlights the unprocessed cell and the image from the corresponding camera 2 using the unprocessed display C3 in each of the layout diagram 72 and the camera list 73.

[0102] As described above, according to the site monitoring system 1 of this modified example, if applying image recognition processing to the image selected in step S4 would reach the upper limit of processing capacity (YES in S41), the cell 8 in that image is visualized as an unprocessed cell (S42). In this case, the image recognition processing is not performed on the selected image (S11). As a result, even if image recognition processing cannot be applied due to processing capacity constraints, for example, it is possible to alert the user 3 or the like to the image of the unprocessed cell selected as input for image recognition processing in relation to a process in which a stagnation of product 9 has been detected and which may be a bottleneck.

[0103] 3. Effects, etc. As described above, in this embodiment, the site monitoring system 1 monitors a task (an example of a work process) in which multiple processes are performed on a product 9 (an example of an object) using at least one example of a camera 2. The site monitoring system 1 includes a device I / F 54 (an example of an image acquisition unit) and a control unit 50 in the site monitoring PC 5. The device I / F 54 acquires image data representing multiple images (an example of multiple monitoring images) captured by the camera 2 during the multiple processes (S5). The control unit 50 performs image recognition processing to recognize a progress status as an example of the status of each process in the task based on the image data (S4-S6). The control unit 50 selects an image to be input for the image recognition processing from among the multiple images based on sensor data as an example of sensor information indicating the detection results of the product 9 during the task (S4). The control unit 50 does not perform image recognition processing on the images not selected from the multiple images (S11), but performs image recognition processing on the selected image (S6).

[0104] According to the above-described on-site monitoring PC 5, it is possible to selectively execute image recognition processing by inputting an image selected based on sensor data from among images of a plurality of processes captured by the camera 2, thereby reducing the processing load. This reduces the processing load of monitoring the work by the camera 2 when the work involves performing a plurality of processes on the product 9.

[0105] In this embodiment, the control unit 50 detects a process in which the product 9 is stuck based on the sensor data (S2, S3), and selects, from the multiple images, a monitoring image corresponding to the process in which the stuck state was detected, as input for image recognition processing (S4). This makes it possible to select an image to be used as input for image recognition processing, for example, based on the fact that the process in which the product 9 is stuck is relatively likely to become a bottleneck.

[0106] In this embodiment, the control unit 50 acquires the detection result of the product 9 by the sensor 85, which is different from the camera 2, as sensor data, and detects the process where the product 9 is stuck based on the sensor data (S1 to S3). This makes it possible to detect the process where the product 9 is stuck, for example, based on the detection result of the product 9 acquired from the sensor 85, depending on the period during which the product 9 is detected, etc.

[0107] In this embodiment, the sensor 85 includes a proximity sensor that detects the proximity of the product 9 to the sensor 85. For example, the sensor 85 can detect the presence or absence of the product 9 based on whether or not proximity is detected.

[0108] In this embodiment, the control unit 50 executes image recognition processing on images among the multiple images corresponding to a process in which congestion was detected (YES in S3, S6), but does not execute image recognition processing on images corresponding to a process in which congestion was not detected (NO in S3, S11). This allows selective image recognition processing to be executed on images of a process related to a bottleneck, for example, in response to the relatively high possibility that a process in which congestion was detected is a bottleneck.

[0109] In this embodiment, the control unit 50 selects, from the multiple images, an image corresponding to the process where a congestion was detected and images corresponding to the processes before and after the process where the congestion was detected, as inputs for the image recognition process (S4). For example, in the work at the site 6 shown in Figure 1, if a product 9 is detected in the process of cell 8-2 as in the scene of Figure 6, three images are selected: an image of the detected process of cell 8-2 taken by camera 2-2, and images of the processes before and after the process taken by cameras 2-1 and 2-3. In this way, as images of processes related to a bottleneck, for example, image recognition processing can be performed on the image of the process where a congestion was detected, as well as images of the processes before and after the process, and the progress status recognized in each image can be obtained.

[0110] In this embodiment, the control unit 50 identifies a bottleneck process as an example of a process that causes a delay in work in multiple processes, based on the progress status of each process recognized by the image recognition process (S8). As a result, even if a congestion of the product 9 is erroneously detected based on the sensor data due to, for example, an erroneous reaction by the sensor 85 to an item other than the product 9, the bottleneck process can be accurately identified based on the progress status recognized by the image recognition process.

[0111] In this embodiment, the control unit 50 identifies a bottleneck process based on a combination of the progress status of the process where the congestion was detected and the progress status of each process before and after the process where the congestion was detected (S8, see FIG. 9). This allows for improved accuracy in bottleneck determination by combining the progress statuses recognized in the images of processes that are thought to be related to the bottleneck.

[0112] In this embodiment, the site monitoring PC 5 of the site monitoring system 1 further includes an output I / F 55 as an example of an output unit that outputs information. The control unit 50 causes the output I / F 55 to output information identifying a selected image from the multiple images (S7). This allows, for example, a PC 4 or the like that receives the output information to display multiple images of multiple processes based on the output information and to perform visualization by changing the display mode of the image selected as input for image recognition processing. This visualization can alert the user 3 or the like to, for example, an image of a process selected as being related to a bottleneck and / or a cell 8 in the image.

[0113] In this embodiment, the control unit 50 outputs information identifying an image in which a bottleneck process is captured among the multiple images (S9). This allows, for example, a PC 4 that receives the information to display multiple images in which multiple processes are captured, similar to the visualization based on the information in step S7, and to perform visualization in which the display mode of the image in which the bottleneck process is captured changes. This visualization allows, for example, a warning to the user 3 about the image of the bottleneck process and / or the cell 8 in the image.

[0114] In this embodiment, the at least one camera 2 includes a plurality of cameras 2-1 to 2-3, etc., and a plurality of images are captured by the plurality of cameras 2-1 to 2-3, etc. In this manner, in this embodiment, an image to be used as input for image recognition processing is selected from a plurality of images captured by each camera 2 as a plurality of monitoring images.

[0115] In this embodiment, the image recognition process recognizes the behavior of the worker W in the work as the progress status of each process (S6). This makes it possible to recognize the progress status of each process by the worker W in accordance with the behavior of the worker W, such as working, waiting, and being absent as described above.

[0116] In this embodiment, a monitoring method is provided for monitoring a task (an example of a task process) in which a plurality of steps are performed on a product 9 (an example of an object) using at least one example of a camera 2. This method includes a step (S5) in which a site monitoring PC 5 (an example of a computer) acquires image data representing a plurality of images (an example of a plurality of monitoring images) of the plurality of steps captured by the at least one camera 2, and steps (S4 to S6) in which a control unit 50 of the site monitoring PC 5 executes image recognition processing to recognize a progress state (an example of a state) of each task process based on the image data. In the step of executing the image recognition processing, the control unit 50 selects an image to be input for the image recognition processing from among the plurality of images based on sensor data as an example of sensor information indicating a detection result of the product 9 during the task (S4), and executes the image recognition processing on the selected image (S6) without executing the image recognition processing on the images not selected from the plurality of images (S11).

[0117] In this embodiment, a program is provided for causing the control unit 50 (an example of a computer control unit) of the site monitoring PC 5 to execute the above-described monitoring method. The monitoring method and program of this embodiment reduce the processing load of the image recognition process, thereby reducing the processing load of monitoring work using the camera 2.

[0118] (Other Embodiments) As described above, embodiment 1 has been described as an example of the technology disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made as appropriate. Furthermore, it is also possible to combine the components described in each of the above embodiments to create a new embodiment. Therefore, other embodiments will be described below as examples.

[0119] In the above-described first embodiment, an example has been described in which there is one cell 8, one camera 2, and one worker W. In the present embodiment, the relationship is not limited to one-to-one correspondence, and for example, images of multiple cells 8 may be captured by a common camera 2, or multiple workers W may work in one cell 8. While three cells 8 are illustrated in FIG. 1 , the number of cells 8 in the work site 6 may be two, and the number of cameras 2 may be two or less. Furthermore, the relationship between processes and cells 8 is not limited to the above-described example, and for example, multiple consecutive processes may be performed in one cell 8.

[0120] In the above-described first embodiment, an example was described in which the site monitoring system 1 includes multiple cameras 2. In this embodiment, the site monitoring system may include only one camera. The camera in this embodiment may be configured, for example, as an omnidirectional camera, and is positioned to capture an overhead image of each cell 8 at the site 6. In this case, too, processing similar to that in the first embodiment can be performed by, for example, selecting an input area for image recognition processing for each area of ​​each cell 8 captured in an image by a single camera, instead of the images from each camera 2 in the first embodiment.

[0121] In the above-described first embodiment, an example has been described in which the site monitoring system 1 displays the work timeline 70 indicating whether or not work is being performed at each time by the worker W in each cell 8 associated with the process in which a stagnation of the product 9 has been detected. In this embodiment, in addition to or instead of the work timeline 70 of the first embodiment, the site monitoring system may display, for example, a delay index calculated for each time for each cell 8 regardless of whether a stagnation has been detected.

[0122] In the first embodiment described above, the sensor 85 of the site monitoring system 1 is installed in the buffer 81 of the cell 8. In the site monitoring system of this embodiment, the sensor 85 may be installed in the workbench 80 of the cell 8. For example, the site monitoring PC 5 of this embodiment may calculate the detection rate of the products 9 at each workbench 80, similar to the detection rate of the products 9 at each buffer 81, and use this as a delay index for each cell 8.

[0123] In the first embodiment described above, the sensor 85 of the site monitoring system 1 is configured as a proximity sensor that detects the proximity of the product 9 in a non-contact manner. In this embodiment, the sensor of the site monitoring system may be a contact-type physical sensor that detects contact with the product 9. Alternatively, the sensor may be configured as a camera, and may detect the presence or absence of the product 9 in the cell 8 based on a temporal change in brightness in a video captured of the cell 8 in which the cell 8 is placed during work. The camera may be a camera separate from the camera 2 that captures the image used as input for image recognition processing.

[0124] As described above, in each of the above embodiments, the sensors of the site monitoring system 1 include at least one of a proximity sensor such as the sensor 85 and a camera other than the camera 2. Furthermore, the camera from which the sensor data is acquired is not limited to the above examples and may be the same as the camera 2. That is, the control unit 50 may acquire image data from the camera 2 as sensor data and, based on the sensor data, detect the process in which the product 9 is stagnating.

[0125] In the above-described first embodiment, an example has been described in which the site monitoring system 1 includes the camera 2 and the sensor 85. The site monitoring system of this embodiment may not include the camera 2 and / or may not include the sensor 85. Even in this case, the site monitoring system can receive image data from an external camera and / or receive sensor data from an external sensor, and perform operations similar to those of the first embodiment. For example, the monitoring system of the present disclosure may be realized by a site monitoring PC 5 as a site monitoring system.

[0126] In the first embodiment described above, the site monitoring system 1 visualized the image selected as input for the image recognition process using the selection display C2. For example, if the site monitoring system of this embodiment determines that the target cell is a bottleneck (YES in S8), it may identify cells 8 adjacent to the bottleneck cell 8 in the site 6 using the map data D0. In this embodiment, the analysis PC 4 may visualize the adjacent cells 8 and their images instead of the selection display C2. In this case, the processing of step S7 in FIG. 7 may not be executed. This visualization also allows the user 3 and others to be presented with cells 8 that may be related to the bottleneck.

[0127] In the first embodiment described above, in the site monitoring system 1, the site monitoring PC 5 acquires sensor data from the sensor 85 and image data from the camera 2. In this embodiment, the sensor data and / or image data may be stored in a database external to the site monitoring PC 5, and the site monitoring PC 5 may acquire each piece of data from the database. Alternatively, the analysis PC 4 may acquire image data from the database and play back the video.

[0128] In the above-described first embodiment, an example has been described in which the site monitoring system 1 operates in real time while work is being performed at the site 6. In this embodiment, the operation of the site monitoring system may be applied to image data recorded on a recording medium and collected sensor data during the period in which work is being performed.

[0129] In the above-described first embodiment, an example has been described in which the image recognition processing is executed in the site monitoring PC 5 at the site 6. In the present embodiment, the image recognition processing may be executed in a server or the like external to the site monitoring PC 5. In this case, the site monitoring PC 5 may cause the server to acquire image data in step S5 of FIG. 7 , and may receive the result of the image recognition processing from the server in step S6.

[0130] In each of the above embodiments, an example has been described in which the site monitoring system 1 is applied to a site 6 such as a factory. In this embodiment, the site to which the site monitoring system 1 and the site monitoring PC 5 are applied is not limited to the above-described site 6, but may be various other sites such as a logistics warehouse or a store sales floor. Furthermore, the work determined by the site monitoring system 1 is not limited to the above-described example, but may be various work suited to various sites. For example, at a logistics warehouse site, the site monitoring system 1 may be applied to the analysis of work including processes such as collection and packing of items such as luggage instead of products 9.

[0131] In each of the above embodiments, an example has been described in which the site monitoring system 1 is applied to a person such as a worker W. In this embodiment, the worker to be analyzed by the site monitoring system 1 is not limited to a person, but may be a mobile object capable of performing various tasks, or may be production equipment or the like provided for each task. For example, the mobile object may be a robot, or various manned or unmanned vehicles.

[0132] As described above, the embodiments have been described as examples of the technology in the present disclosure, and for that purpose, the accompanying drawings and detailed description have been provided.

[0133] Therefore, the components shown in the accompanying drawings and detailed description may include not only essential components for solving the problem, but also components that are not essential for solving the problem in order to illustrate the above technology. Therefore, the fact that these non-essential components are shown in the accompanying drawings or detailed description should not be interpreted as immediately indicating that these non-essential components are essential.

[0134] (Aspects of the Present Disclosure) The following describes examples of aspects of the present disclosure.

[0135] A first aspect of the present disclosure is a monitoring system that monitors a work process in which multiple processes are performed on an object using at least one camera. The monitoring system includes an image acquisition unit that acquires image data representing multiple monitoring images captured by the at least one camera of the multiple processes, and a control unit that executes image recognition processing to recognize a state of each process in the work process based on the image data. The control unit selects a monitoring image from the multiple monitoring images to be used as input for the image recognition processing based on sensor information that indicates a detection result of an object in the work process, and executes the image recognition processing on the selected monitoring image without executing the image recognition processing on the monitoring images that were not selected from the multiple monitoring images.

[0136] In a second aspect, in the monitoring system of the first aspect, the control unit detects, based on sensor information, a process among multiple processes in which an object is stagnant, and selects, from the multiple monitoring images, a monitoring image corresponding to the process in which the stagnant object is detected, as input for image recognition processing.

[0137] In a third aspect, in the monitoring system of the second aspect, the control unit acquires the detection results of the object by a sensor other than the at least one camera as sensor information, and detects the process in which the object is stagnant based on the sensor information.

[0138] In a fourth aspect, in the surveillance system of the third aspect, the sensor includes at least one of a proximity sensor and another camera different from the at least one camera.

[0139] In a fifth aspect, in the monitoring system of the second aspect, the control unit acquires image data from the at least one camera as sensor information, and detects a process in which the object is staying based on the sensor information.

[0140] In a sixth aspect, in the monitoring system of any of the second to fifth aspects, the control unit performs image recognition processing on monitoring images among multiple monitoring images corresponding to a process in which stagnation is detected, and does not perform image recognition processing on monitoring images corresponding to a process in which stagnation is not detected.

[0141] In a seventh aspect, in the monitoring system of any of the second to sixth aspects, the control unit selects, from among the multiple monitoring images, a monitoring image corresponding to the process in which the congestion was detected and monitoring images corresponding to the processes before and after the process in which the congestion was detected as inputs for the image recognition processing.

[0142] In an eighth aspect, in the monitoring system of any of the first to seventh aspects, the control unit identifies a process among multiple processes that is the cause of a delay in the work process based on the state of each process recognized by image recognition processing.

[0143] In a ninth aspect, in the monitoring system of the seventh aspect, the control unit identifies a process among multiple processes that is the cause of the delay in the work process based on a combination of the state of the process in which the congestion was detected and the state of each of the processes before and after the process in which the congestion was detected.

[0144] In a tenth aspect, in the monitoring system of any of the first to ninth aspects, the monitoring system further includes an output unit that outputs information, and the control unit causes the output unit to output information that identifies a monitoring image selected from the plurality of monitoring images.

[0145] In an eleventh aspect, in the surveillance system of any one of the first to tenth aspects, the at least one camera includes a plurality of cameras, and the plurality of surveillance images are captured by the plurality of cameras.

[0146] In a twelfth aspect, in the monitoring system of any one of the first to eleventh aspects, the image recognition processing recognizes the behavior of a worker in a work process as the state of each process.

[0147] A thirteenth aspect of the present disclosure is a monitoring method for monitoring a work process in which a plurality of processes are performed on an object using at least one camera. The monitoring method includes a step of a computer acquiring image data representing a plurality of monitoring images captured by the at least one camera of the plurality of processes, and a step of a control unit of the computer executing an image recognition process to recognize a state of each process in the work process based on the image data. In the step of executing the image recognition process, the control unit selects a monitoring image to be input for the image recognition process from the plurality of monitoring images based on sensor information representing a detection result of the object in the work process, and executes the image recognition process on the selected monitoring image without executing the image recognition process on the monitoring images not selected from the plurality of monitoring images.

[0148] A fourteenth aspect of the present disclosure is a program for causing a control unit of a computer to execute the monitoring method according to the thirteenth aspect.

[0149] The present disclosure is applicable to data analysis applications for analyzing the efficiency of work performed in various environments such as factories or logistics sites.

Claims

1. A monitoring system that uses at least one camera to monitor a work process in which multiple processes are performed on an object, comprising: an image acquisition unit that acquires image data showing multiple monitoring images captured by the at least one camera of the multiple processes; and a control unit that executes image recognition processing to recognize the state of each process in the work process based on the image data, wherein the control unit selects a monitoring image to be used as input for the image recognition processing from among the multiple monitoring images based on sensor information that indicates the detection result of the object in the work process, and executes the image recognition processing on the selected monitoring image without executing the image recognition processing on monitoring images that were not selected from the multiple monitoring images.

2. The monitoring system according to claim 1, wherein the control unit detects a process among the plurality of processes in which the object is stuck based on the sensor information, and selects, from the plurality of monitoring images, a monitoring image corresponding to the process in which the sticking was detected as input for the image recognition processing.

3. The monitoring system according to claim 2, wherein the control unit acquires the detection result of the object by a sensor other than the at least one camera as the sensor information, and detects the process in which the object is retained based on the sensor information.

4. The surveillance system according to claim 3, wherein the sensor includes at least one of a proximity sensor and another camera different from the at least one camera.

5. The monitoring system according to claim 2, wherein the control unit acquires image data from the at least one camera as the sensor information, and detects the process in which the object is retained based on the sensor information.

6. The monitoring system of claim 2, wherein the control unit executes the image recognition process on a monitoring image among the plurality of monitoring images corresponding to a process in which the stagnation is detected, and does not execute the image recognition process on a monitoring image corresponding to a process in which the stagnation is not detected.

7. The monitoring system according to claim 2, wherein the control unit selects, from among the plurality of monitoring images, as input for the image recognition processing, a monitoring image corresponding to the process in which the stagnation was detected and monitoring images corresponding to processes before and after the process in which the stagnation was detected.

8. The monitoring system according to claim 1, wherein the control unit identifies a process among the plurality of processes that is a cause of a delay in the work process based on the state of each process recognized by the image recognition processing.

9. The monitoring system of claim 7, wherein the control unit identifies a process among the plurality of processes that is causing a delay in the work process based on a combination of the state of the process in which the congestion was detected and the states of each of the processes before and after the process in which the congestion was detected.

10. The surveillance system according to claim 1, further comprising an output unit that outputs information, wherein the control unit causes the output unit to output information identifying the selected surveillance image from among the plurality of surveillance images.

11. The surveillance system according to claim 1, wherein the at least one camera includes a plurality of cameras, and the plurality of surveillance images are captured by the plurality of cameras.

12. The monitoring system according to claim 1, wherein the image recognition process recognizes the actions of a worker in the work process as the state of each process.

13. A monitoring method for monitoring a work process in which a plurality of processes are performed on an object using at least one camera, comprising: a step in which a computer acquires image data showing each of a plurality of monitoring images captured by the at least one camera of the plurality of processes; and a step in which a control unit of the computer executes an image recognition process to recognize a state of each process in the work process based on the image data, wherein in the step of executing the image recognition process, the control unit selects a monitoring image to be used as input for the image recognition process from among the plurality of monitoring images based on sensor information showing the detection result of the object in the work process, and executes the image recognition process on the selected monitoring image without executing the image recognition process on monitoring images not selected from the plurality of monitoring images.

14. A program for causing a control unit of a computer to execute the monitoring method according to claim 13.

Images