Judgment system, judgment device, judgment method, and program

Abstract

The present disclosure provides a determination system, a determination device, and a determination method which are capable of efficiently analyzing an image. A determination system according to the present disclosure comprises an acquisition unit and a determination unit. The acquisition unit acquires a feature amount of each region in an image. For example, the acquisition unit acquires a feature amount of each region in an image captured by a camera. The determination unit determines the necessity of analyzing each region in the image on the basis of the feature amount of each region acquired by the acquisition unit. For example, the determination unit outputs a determination result of the necessity of analyzing each region in the image.

Description

Determination system, determination device, and determination method

[0001] The present disclosure relates to a determination system, a determination device, and a determination method.

[0002] There are known techniques for monitoring work performed by workers. For example, Patent Literature 1 describes a technique for acquiring photographed image data of a monitoring area including a work area, identifying a work area in the monitoring area where the worker is present based on the photographed image data, and starting to photograph the work based on the identified work area.

[0003] Japanese Patent Application Laid-Open No. 2020-166493

[0004] However, in Patent Document 1, only the start or stop of capturing images of a worker's work is considered, so that in Patent Document 1, it is difficult to efficiently analyze the captured images, for example, when performing analysis processing on the captured images.

[0005] In view of the above problems, one of the objects of the present disclosure is to provide a determination system, a determination device, and a determination method that are capable of efficiently analyzing images.

[0006] A determination system according to one aspect of the present disclosure includes an acquisition means for acquiring features of each region in an image, and a determination means for determining whether or not analysis is required for each region based on the acquired features of each region.

[0007] A determination device according to one aspect of the present disclosure includes an acquisition means for acquiring features of each region in an image, and a determination means for determining whether or not analysis is required for each region based on the acquired features of each region.

[0008] A determination method according to one aspect of the present disclosure acquires features of each region in an image, and determines whether or not analysis is required for each region based on the acquired features of each region.

[0009] According to the present disclosure, images can be analyzed efficiently.

[0010] 1 is a diagram showing an example of an image for explaining a related hand movement analysis method. FIG. 2 is a diagram showing an example of a cycle waveform for explaining a related hand movement analysis method. FIG. 3 is a diagram showing an example of an image for explaining a related finger movement analysis method. FIG. 4 is a diagram showing an example of an analysis result for explaining a related finger movement analysis method. FIG. 5 is a configuration diagram showing an example of a configuration of a determination system according to an embodiment. FIG. 6 is a configuration diagram showing an example of a configuration of a determination device according to an embodiment. FIG. 7 is a flowchart showing an example of a determination method according to an embodiment. FIG. 8 is a configuration diagram showing an example of a configuration of a monitoring system according to an embodiment. FIG. 9 is a configuration diagram showing an example of a configuration of each device in the monitoring system according to an embodiment. FIG. 10 is a configuration diagram showing an example of a configuration of a hand movement analysis unit according to an embodiment. FIG. 11 is a configuration diagram showing an example of a configuration of a finger movement learning unit according to an embodiment. FIG. 12 is a configuration diagram showing an example of a configuration of a finger movement analysis unit according to an embodiment. FIG. 13 is a flowchart showing an example of operation of the monitoring system according to an embodiment. FIG. 14 is a flowchart showing an example of a detailed analysis area identification process according to an embodiment. FIG. 15 is a diagram showing an example of a cycle waveform for explaining the detailed analysis area identification process according to an embodiment. FIG. 16 is a diagram for explaining a specific example of the detailed analysis area identification process according to an embodiment. FIG. 17 is a flowchart showing another example of the detailed analysis area identification process according to an embodiment. FIG. 1 is a diagram showing another example of a table used in the detailed analysis area identification process according to the embodiment. FIG. 2 is a flowchart showing another example of the detailed analysis area identification process according to the embodiment. FIG. 3 is a diagram showing an example of a table used in the detailed analysis area identification process according to the embodiment. FIG. 4 is a diagram showing an example of an analysis result of a hand movement of the monitoring system according to the embodiment. FIG. 5 is a diagram showing an example of an analysis result of a finger movement of the monitoring system according to the embodiment. FIG. 6 is a configuration diagram showing an example of the configuration of each device in the monitoring system according to the embodiment. FIG. 7 is a flowchart showing an example of the operation of the monitoring system according to the embodiment. FIG. 8 is a flowchart showing an example of the sharpening area identification process according to the embodiment. FIG. 9 is a flowchart showing another example of the sharpening area identification process according to the embodiment. FIG. 10 is a configuration diagram showing an example of the configuration of hardware of a computer according to the embodiment.

[0011] Hereinafter, embodiments will be described with reference to the drawings. In the drawings, the same elements are denoted by the same reference numerals, and redundant description will be omitted as necessary.

[0012] (Discussion Leading to Embodiments) There is a technology for monitoring the work of workers by analyzing video footage of the work at a manufacturing site. For example, a related analysis method is a hand movement analysis method that simply analyzes the hand movements of a worker from video footage of the worker's hands. FIG. 1 shows an example of image analysis using the hand movement analysis method. As shown in FIG. 1, the hand movement analysis method identifies, for example, the work area of ​​each work process from video footage of each work process and detects the worker's hands within each work area. Furthermore, the hand movement analysis method analyzes the worker's work status based on the detection results of the worker's hands in each work area. For example, if hands are detected in the work area, it can be determined that the worker is currently working, and if hands are not detected in the work area, it can be determined that the worker's work has stopped.

[0013] FIG. 2 shows an example of a cycle waveform representing the analysis results of the hand movement analysis method. A cycle is a unit of a series of tasks repeatedly performed in a work area. The cycle waveform is a waveform that shows hand detection results in a time series and represents the work status of each cycle. In FIG. 2, the horizontal axis represents time, and the vertical axis represents hand detection results. For example, the hand detection results may indicate whether a hand is detected in the work area or the hand coordinates within the camera's field of view. For example, hand position information between the start and end of a task, or the time when the hand is within the work area, may be converted to 1 and the rest to 0, and the time averaged over time may be used to create a cycle waveform. As shown in FIG. 2, the cycle waveform can be used to understand the start and stop of a task and the cycle time from the start to the end of a task. The hand movement analysis method allows for easy analysis of a worker's work status based on the detection results of the worker's hand movements.

[0014] Another related analysis method is a finger movement analysis method that analyzes the finger movements (finger movements) of a worker in detail from video footage of the worker's hands. FIG. 3 shows an example of image analysis using the finger movement analysis method. As shown in FIG. 3, the finger movement analysis method detects the skeletal structure of the worker's fingers from video footage of the work performed in each work process and detects the parts the worker is holding. Furthermore, the finger movement analysis method analyzes the detailed work content of the worker based on the detection results of the skeletal shape of the worker's fingers and the parts the worker is holding. FIG. 4 shows an example of the analysis results of the finger movement analysis. As shown in FIG. 4, the finger movement analysis method can identify the work content (e.g., tasks A1 to A5) performed by the worker at each time point. The finger movement analysis method allows the detailed analysis of the work content of the worker from the detection results of the worker's finger movements. Furthermore, the finger movement analysis results can be used to analyze the work content, determine the progress and causes of delays in the work process, and provide advice on personnel allocation on the production line.

[0015] The inventors have considered the issues involved in introducing a finger movement analysis method into a system for monitoring a production line. By introducing the finger movement analysis method, it is possible to automatically analyze the work of workers in detail 24 hours a day. However, there is a problem with the finger movement analysis method: it is costly to introduce the finger movement analysis method to images from multiple cameras installed on the production line.

[0016] For example, in a system including an edge device connected to a camera and a cloud server capable of communicating with the edge device, when processing a finger motion analysis method on the cloud server, the edge device may only be able to accommodate one camera. "Accommodating" refers to connecting a camera to the edge device and collecting and processing data from the camera to provide a specific function (finger motion analysis). In this case, an edge device is required for each camera, which poses a problem of increased costs depending on the number of cameras. Furthermore, when introducing a new finger motion analysis method, it is difficult to efficiently analyze the image or the area within the image that needs to be analyzed by the finger motion analysis method.

[0017] (Outline of the embodiment) Next, an outline of the embodiment will be described. Fig. 5 shows an example of the configuration of a determination system 10 according to some embodiments. The determination system 10 can be applied to a system that analyzes images when monitoring the work of workers on a production line, for example.

[0018] In the example of FIG. 5 , the determination system 10 includes an acquisition unit 11 and a determination unit 12. The acquisition unit 11 acquires feature quantities for each region in an image. For example, the acquisition unit 11 acquires an image captured by a camera installed on a production line and acquires feature quantities for each region in the acquired image. The region in the image may be, for example, a work area within a work process, an area within a work area, or may include multiple work areas. The feature quantities are indices related to the image features used to determine whether analysis is necessary. For example, the feature quantities may be based on at least one of the work time of the work performed in the image, the object captured in the image, and the status of the person captured in the image. The work time may be a cycle time based on a cycle waveform. The feature of the object captured may be a specific part of the worker (e.g., right hand, left hand) or a specific part of the work area. The person's status may be the person's level of proficiency in the work, etc. The person's status may be an indicator for distinguishing between a new employee and a veteran, etc.

[0019] The determination unit 12 determines whether or not analysis of each region of the image is necessary based on the feature amount of each region acquired by the acquisition unit 11. The determination unit 12 may determine whether or not analysis is necessary based on signs of abnormality detected based on the feature amount of each region. Signs of abnormality are situations in which there is a high possibility of an abnormality occurring. For example, the determination unit 12 may detect signs of abnormality based on work time. Furthermore, the determination unit 12 may determine whether or not detailed analysis of each region is necessary based on the subject captured in the image, the level of proficiency in the work of the person captured in the image, etc.

[0020] For example, the acquisition unit 11 may acquire a first analysis result from a first analysis for each region in an image, and the determination unit 12 may determine, based on the first analysis result for each region, whether a second analysis, which is more detailed than the first analysis, is necessary for each region. The first analysis is an analysis with a lighter processing load, and the second analysis is an analysis with a heavier processing load than the first analysis. For example, the first analysis may be hand movement analysis, person detection, skeleton estimation, object detection, object anomaly detection, etc. For example, the second analysis may be finger movement analysis, action recognition, skeleton estimation, segmentation, scene segmentation, video summarization, etc.

[0021] For example, the determination system 10 may include a first device including a first analysis processor that performs a first analysis, and a second device including a second analysis processor that performs a second analysis. The first device may include a transmitter that transmits an image including an area determined to require a second analysis from among the areas where the first analysis processor performed the first analysis to the second analysis processor. For example, the first device may be an edge device located at an edge, and the second device may be a cloud server (management server) located in a cloud. Note that, depending on the processing load of the edge and the cloud, the first analysis may be performed on the cloud side. Furthermore, if there are heavy processes and light processes within the same process, the light and heavy processes may be distributed between the edge and the cloud.

[0022] The determination system 10 may be configured by one device or by multiple devices. Fig. 6 shows an example configuration of a determination device 20 according to some embodiments. In the example of Fig. 6, the determination device 20 includes the acquisition unit 11 and the determination unit 12 shown in Fig. 5. For example, part or all of the determination system 10 and the determination device 20 may be arranged in an edge device or a cloud server.

[0023] 7 shows an example of a determination method according to some embodiments. For example, the determination method according to some embodiments may be performed by the determination system 10 of FIG. 5 or the determination device 20 of FIG.

[0024] 7, the acquisition unit 11 acquires feature amounts for each region in an image (S11). For example, based on an image captured by a camera installed on a production line, the acquisition unit 11 acquires feature amounts based on the work time of the work performed in the image, the subject captured in the image, the level of proficiency in the work of the person captured in the image, etc.

[0025] Next, the determination unit 12 determines whether or not detailed analysis of each region is necessary based on the acquired feature amount of each region (S12). For example, the determination unit 12 may determine whether or not analysis of each region is necessary based on the work time of the work performed in the image, the subject photographed in the image, the proficiency level of the person photographed in the image at the work, etc., and output the determination result.

[0026] In this manner, in the embodiment, the necessity of analysis for each region in the image is determined based on the feature quantities of each region. For example, by determining the necessity of analysis based on the work time of the task performed in the image, the subject photographed in the image, the level of proficiency of the person photographed in the image, etc., it is possible to perform analysis only on the necessary regions, thereby enabling efficient analysis. For example, when a simple first analysis such as hand movement analysis is performed and a detailed second analysis such as finger movement analysis is performed, the second analysis is performed only on the images that are required based on the results of the first analysis, thereby reducing processing load and costs.

[0027] (Embodiment 1) Next, a description will be given of embodiment 1. This embodiment is an example in which an area is narrowed down by light hand movement analysis processing, and heavy finger movement analysis processing is applied to the narrowed down area.

[0028] FIG. 8 shows an example configuration of a monitoring system 1 according to some embodiments. The monitoring system 1 is, for example, a system that monitors the work of workers using video captured by cameras installed on a production line. The production line is a line that manufactures target products through multiple work processes. For example, the target products may be, but are not limited to, home appliances such as televisions, electronic devices such as personal computers, electronic components such as semiconductor devices, mobile devices such as automobiles, processed foods, etc. Note that video includes multiple images (frames) in a time series, so the terms video and image are interchangeable. In other words, the monitoring system 1 can be considered both as a video processing system that processes video captured by cameras and as an image processing system that processes images captured by cameras.

[0029] 8, the monitoring system 1 includes multiple cameras 101, an edge device 100, a control server 200, a management server 300, and a display device 400. For example, the edge device 100 is located at a manufacturing site (edge) close to the camera 101. The control server 200 and the management server 300 are located at a remote location (cloud) away from the manufacturing site. Note that either or both of the control server 200 and the management server 300 may be located at the same manufacturing site as the edge device 100.

[0030] The cameras 101 and the edge device 100 are communicably connected via a network NW1. The network NW1 may be a wired network such as a local area network (LAN) or a wireless network such as a wireless LAN.

[0031] The edge device 100 and the management server 300 are communicatively connected via a network NW2. Like the network NW1, the network NW2 may be a wired network or a wireless network. Examples of wired networks include the Internet, a wide area network (WAN), and a LAN. Examples of wireless networks include 4G, local 5G / 5G, long term evolution (LTE), and a wireless LAN. Furthermore, the networks NW1 and NW2 may be the same network.

[0032] The edge device 100 and the control server 200 are communicatively connected via any network. The edge device 100 and the control server 200 may be connected via the network NW2 or the network NW1, or via another network.

[0033] Multiple cameras 101 are installed on the production line. Each camera 101 is installed so as to be able to capture images of each work process on the production line. If there are multiple production lines, a camera 101 is installed at each work process on each production line. A camera 101 may be installed not only for each work process but also for each production line. In this example, the camera 101 captures the work area of ​​each work process. For example, the camera 101 can capture an image of a workbench (work table) from above, capturing images of a worker working, the worker's hands, the parts being worked on, the tools used in the work, etc. The camera 101 may be a fixed camera fixed at a predetermined position, a wearable camera worn by a worker, or a camera equipped on a robot having an autonomous mobile mechanism. The camera 101 may capture an image of a work area of ​​one work process or of work areas of multiple work processes. In other words, the image captured by the camera 101 may include one work area or multiple work areas. The image captured by the camera 101 may be an image of a production line including a plurality of work areas. The camera 101 transmits the captured image including the work areas to the edge device 100 via the network NW1.

[0034] The edge device 100 is a simple analysis device that houses multiple cameras 101 and easily analyzes and processes images from the multiple cameras 101. For example, the edge device 100 is a hand movement analysis device. The edge device 100 is also a video transmission device that transmits images from the cameras 101 to the management server 300. The edge device 100 is an edge processing device arranged on the edge side, such as MEC (Multi-access Edge Computing). For example, if a base station is arranged between the network NW1 and the network NW2, the edge device 100 may be connected to the base station.

[0035] The edge device 100 receives video including the work area captured by the camera 101, performs a simple analysis process on the received video, and transmits the analysis results to the control server 200. For example, as the simple analysis process, the edge device 100 performs a hand movement analysis process for a hand movement analysis method such as that shown in FIG. 1 . The edge device 100 also receives a video instruction indicating the video to be transmitted, and transmits the corresponding video to the management server 300 in accordance with the received video instruction. The video instruction may indicate a video captured by a specific camera 101 out of the multiple cameras 101, or may indicate a specific area within the video captured by the specific camera 101.

[0036] The control server 200 is a determination device that determines whether a detailed analysis of the video is necessary based on the results of a simple analysis of the video by the edge device 100. The control server 200 is also a control device that controls the transmission of the video from the edge device 100 based on the determination results. The control server 200 may be one or more physical servers, or may be a virtualized server built on a virtualization platform. The control server 200 may be a cloud server on a cloud, or an on-premise server installed at a work site. The control server 200 and the management server 300 may be the same server.

[0037] The control server 200 receives the analysis result from the edge device 100, determines whether a detailed analysis of the video including the work area is necessary based on the received analysis result, and transmits the determination result to the edge device 100. The detailed analysis is, for example, finger motion analysis. For example, the control server 200 transmits to the edge device 100 the video from the camera 101 that is determined to require detailed analysis, or a video instruction indicating an area within the video from the camera 101.

[0038] The management server 300 is a monitoring device that monitors the work of each work process on a production line. Like the control server 200, the management server 300 may be one or more physical servers, or may be a virtualized server built on a virtualization platform. The management server 300 may be a cloud server on the cloud, or an on-premise server installed at a work site. For example, the management server 300 is a GPU (Graphics Processing Unit) server that processes images.

[0039] The management server 300 is a detailed analysis device that performs detailed analysis processing of the video captured by the camera 101. For example, the management server 300 is a finger movement analysis device. It is also a video receiving device that receives the video captured by the camera 101 from the edge device 100. The management server 300 receives the video including the work area captured by the camera 101 from the edge device 100, performs detailed analysis processing on the received video, and outputs the analysis results to the display device 400. For example, as the detailed analysis processing, the management server 300 performs finger movement analysis processing for the finger movement analysis method shown in FIG. 3. The management server 300 may output the video received from the edge device 100 to the display device 400. When the management server 300 receives the hand movement analysis results from the edge device 100, it may output the hand movement analysis results to the display device 400.

[0040] The display device 400 displays information output from the management server 300 on a display screen so that an administrator can monitor the work of workers. For example, the display device 400 displays the finger motion analysis results output from the management server 300 on a GUI (Graphical User Interface) that is a display screen. The display device 400 may display an image including the work area and the finger motion analysis results. The display device 400 may also display the hand motion analysis results. The display device 400 may display the information output from the management server 300 as is, or may output the information in another manner. For example, the display device 400 may output an alert according to the finger motion or hand motion analysis results, or may output an identification number for the work location of the alert. The management server 300 and the display device 400 may be configured as a single device.

[0041] 9 shows an example configuration of each device in the monitoring system 1 according to some embodiments. Note that the configuration of each device is an example, and other configurations may be used as long as they are capable of executing the operation examples described below. Some of the functions of the control server 200 may be arranged in the edge device 100 or the management server 300. Some of the functions of the edge device 100 may be arranged in the control server 200 or the management server 300. Some of the functions of the management server 300 may be arranged in the edge device 100 or the control server 200.

[0042] In the example of FIG. 9 , the edge device 100 includes an image acquisition unit 110 , a hand movement analysis unit 120 , an image selection unit 130 , and an image transmission unit 140 .

[0043] The video acquisition unit 110 acquires video captured by each of the multiple cameras 101 via the network NW1. The video captured by the cameras 101 is also referred to as work video. For example, the work video includes video of a work area for one work process, or video of work areas for multiple work processes. The video acquisition unit 110 also functions as an image acquisition unit that acquires multiple images, i.e., frames, in time series.

[0044] The hand movement analysis unit 120 analyzes the hand movements of the worker from multiple work videos acquired from multiple cameras 101. The hand movement analysis unit 120 is a simplified analysis unit that performs hand movement analysis processing to simply analyze hand movements. The hand movement analysis processing is processing for the hand movement analysis method shown in FIG. 1 . That is, the hand movement analysis unit 120 detects the worker's hands in the work video and analyzes the worker's work status in the work video. When the work video includes multiple work areas, the hand movement analysis unit 120 detects the worker's hands in each work area and analyzes the worker's work status in each work area. The hand movement analysis unit 120 transmits the analysis results obtained by performing hand movement analysis on the multiple work videos to the control server 200. The hand movement analysis unit 120 includes a notification unit that notifies the control server 200 of the analysis results. The hand movement analysis unit 120 may notify the control server 200 of the analysis results via the network NW2 or via another communication path.

[0045] 10 shows an example of the configuration of the hand movement analysis unit 120 according to some embodiments. In the example of Fig. 10, the hand movement analysis unit 120 includes an object detection unit 121 and a work situation analysis unit 122.

[0046] The object detection unit 121 detects objects in the acquired work video. For example, the object detection unit 121 detects objects in a work area in the work video. Detecting an object includes identifying the type of object and detecting the position of the object. The object detection unit 121 extracts an object region, such as a rectangle containing an object, from each image included in the work video and recognizes the object type of the object in the extracted object region. The object detection unit 121 calculates image features of the object included in the object region and recognizes the object based on the calculated features. For example, the object detection unit 121 recognizes objects in an image using an object recognition engine that uses machine learning such as deep learning. Objects can be recognized by machine learning the image features and object type of the object. The object detection result includes the object type, position information of the object region containing the object, an object type score, etc. The object position information is, for example, the coordinates of each vertex of the object region, but may also be the center position of the object region or the position of any point on the object. The object type score is the likelihood of the detected object type, i.e., reliability or confidence.

[0047] The object detection unit 121 determines whether the object types of objects detected in the work area include a person's hand, and if a hand is included, outputs a detection result of the hand. The object detection unit 121 may output a detection result of not only a person's hand, but also parts, tools, etc. used in work. Furthermore, the object detection unit 121 may detect a person (worker) performing work.

[0048] The work situation analysis unit 122 analyzes the work situation of the worker based on the hand detection result of the object detection unit 121. For example, the work situation analysis unit 122 generates a cycle waveform as shown in FIG. 2 based on the detection result of the worker's hand in the work area of ​​the work video, and outputs the generated cycle waveform as the analysis result. The work situation analysis unit 122 may calculate a cycle time from the start to the end of the work based on the cycle waveform, and output the calculated cycle time of the work as the analysis result. Note that the analysis result may include information detected by the object detection unit 121, such as the detection result of the hand, the detection result of parts or tools, and the detection result of the worker.

[0049] Returning to FIG. 9 , the video selection unit 130 selects video to be transmitted to the management server 300 from among the multiple work videos acquired from the multiple cameras 101. The selected and transmitted video is video to be analyzed in detail by the management server 300. The video selection unit 130 receives a video instruction instructing the video to be transmitted from the control server 200, and selects the video to be transmitted in accordance with the received video instruction. The video selection unit 130 includes an acquisition unit that acquires the video instruction from the control server 200. The video selection unit 130 may select, as the video to be transmitted, a work video captured by a specific camera 101 from among the multiple work videos, or may select a video of a specific work area within the work video captured by the specific camera 101. Note that the video selection unit 130 may determine whether detailed analysis is necessary and select the video to be transmitted in accordance with the analysis results of the hand movement analysis unit 120.

[0050] The video transmission unit 140 transmits the video selected by the video selection unit 130 to the management server 300 via the network NW2. The video transmission unit 140 is a communication interface capable of communication via the network NW2, and may be, for example, a wired interface such as a LAN or a WAN, or a wireless interface such as 4G, local 5G / 5G, LTE, or a wireless LAN. The video transmission unit 140 may transmit the analysis results of the hand movement analysis unit 120 to the management server 300.

[0051] In the example of FIG. 9 , the control server 200 includes an analysis result acquisition unit 210 , a detailed analysis necessity determination unit 220 , and a video instruction unit 230 .

[0052] The analysis result acquisition unit 210 acquires the analysis results of the hand movement analysis for the multiple work videos from the edge device 100. The analysis result acquisition unit 210 may acquire the analysis results from the edge device 100 via the network NW2, or may acquire the analysis results from the edge device 100 via another communication path.

[0053] The detailed analysis necessity determination unit 220 determines whether detailed analysis is necessary for each work video based on the analysis results of the acquired multiple work videos. The detailed analysis necessity determination unit 220 may determine whether detailed analysis is necessary for each work video, or may determine whether detailed analysis is necessary for each work area within the work video. Determining whether detailed analysis is necessary for each work video or work area also identifies the work video or work area to be analyzed in detail.

[0054] For example, the detailed analysis necessity determination unit 220 may detect signs of abnormality in the work video or the work area based on the analysis results of the work video including the work area, and identify the work video or work area to be analyzed in detail according to the detection results of the signs of abnormality. The signs of abnormality (possibility of abnormality) may be determined based on the cycle waveform, cycle time, specific hand movement patterns, etc.

[0055] The detailed analysis necessity determination unit 220 may also identify a work video or work area to be analyzed in detail based on the characteristics of the captured object included in the work video. The characteristics of the captured object may be set for each work process or for each worker. For example, the characteristics of the captured object may be the worker's right hand, left hand, dominant hand, etc. The characteristics of the captured object may also be a specific location in the work video or work area. For example, it may be the location where the hand is placed in the work area or the location of a component. The detailed analysis necessity determination unit 220 may identify a location that shows the characteristics of the captured object as an area to be analyzed in detail, or may identify an area to be analyzed in detail based on the feature amounts of the image of the captured object. For example, it may identify an area to be analyzed in detail that is similar to the feature amounts of the image of the right hand or left hand.

[0056] The detailed analysis necessity determination unit 220 may also identify an image or area to be analyzed in detail based on the status of the subject included in the work video. For example, the status of the subject is an attribute of the worker. The status of the subject (worker) may be information that identifies the work they are responsible for, or information that indicates their work experience, years of service, and level of proficiency, such as whether they are a newcomer or a veteran.

[0057] The video instruction unit 230 notifies the edge device 100 of the determination result as to whether detailed analysis is necessary. The video instruction unit 230 may notify the edge device 100 of the determination result via the network NW2, or may notify the edge device 100 of the determination result via another communication path. As the determination result as to whether detailed analysis is necessary, the video instruction unit 230 indicates the work video or the work area within the work video to be analyzed in detail.

[0058] In the example of FIG. 9 , the management server 300 includes a video receiving unit 310 , a storage unit 320 , a finger motion learning unit 330 , a finger motion analysis unit 340 , and an output unit 350 .

[0059] The video receiving unit 310 receives the video transmitted from the edge device 100 via the network NW2. The video receiving unit 310 receives a task video determined to require detailed analysis or a video including a task area within the task video. The video receiving unit 310 may receive the analysis results of the hand gesture from the edge device 100. Like the video transmitting unit 140, the video receiving unit 310 is a communication interface capable of communication via the network NW2, and may be a wired interface or a wireless interface. Note that when the control server 200 receives a video from the edge device 100, the video receiving unit 310 may receive the video from the control server 200.

[0060] The storage unit 320 stores a finger action recognition model 321 that recognizes the finger actions of a worker from video. The finger action recognition model 321 is a learning model that can learn and infer the work content according to the finger actions of the worker included in the video. The finger action recognition model 321 may be a convolutional neural network (CNN) or a recurrent neural network (RNN), or may be any other neural network. Furthermore, the learning model is not limited to a neural network and may be any other learning model.

[0061] During learning, the finger movement learning unit 330 uses learning videos to learn the work content corresponding to the worker's finger movements. The videos used for learning may be learning videos prepared in advance, or may be videos captured by the camera 101 and received from the edge device 100. For example, cycle waveforms of multiple videos obtained from the analysis results of hand movements may be compared, and videos having similar cycle waveform patterns may be extracted and used as learning data. For example, the finger movement learning unit 330 inputs the learning videos and labels indicating the work content in the videos into the finger movement recognition model 321, and trains the finger movement recognition model 321 to predict the work content corresponding to the finger movements in the videos.

[0062] 11 shows an example of the configuration of a finger motion learning unit 330 according to some embodiments. In the example of Fig. 11, the finger motion learning unit 330 includes a skeleton estimation unit 331, an object detection unit 332, and a learning unit 333.

[0063] The skeleton estimation unit 331 estimates the finger skeletons of a person in the input video. For example, the skeleton estimation unit 331 estimates the finger skeletons from each image included in the video using a skeleton estimation engine that uses machine learning such as deep learning. The skeleton estimation unit 331 detects feature points such as the joints of each finger in the image and estimates the skeleton of each finger connecting the feature points. The skeleton estimation unit 331 calculates feature amounts of the estimated finger skeletons and outputs the calculated feature amounts of the finger skeletons. The feature amounts of the finger skeletons are feature amounts corresponding to the shape and movement of the finger skeletons.

[0064] The object detection unit 332 detects objects in the input video. For example, similar to the object detection unit 121 of the edge device 100, the object detection unit 332 recognizes objects using an object recognition engine that uses machine learning such as deep learning. The object detection unit 332 extracts an object area, such as a rectangle, that includes an object from each image of the video and recognizes the object type of the object in the extracted object area. The object detection unit 332 determines whether the object type of the object detected around the person's finger includes a part or tool used for work, and if a specific part or tool is included, calculates feature amounts of the image including the part or tool and outputs the calculated feature amounts of the image.

[0065] The learning unit 333 learns the work content of the worker based on the skeleton estimation result of the skeleton estimation unit 331 and the object detection result of the object detection unit 332. For example, the learning unit 333 inputs the finger skeleton feature amounts estimated by the skeleton estimation unit 331 and the feature amounts of the image of the part or tool detected by the object detection unit 332 to the finger action recognition model 321, learns the co-occurrence relationship between the finger skeleton feature amounts and the image feature amounts, and generates the learned finger action recognition model 321.

[0066] Returning to FIG. 9 , after the finger movement recognition model 321 has completed learning, the finger movement analysis unit 340 analyzes the finger movements in the video based on the received video. The finger movement analysis unit 340 inputs the received work video or a video including a work area in the work video into the learned finger movement recognition model 321, and predicts the work content according to the finger movements in the video. The finger movement analysis unit 340 outputs the predicted work content as the result of the finger movement analysis. For example, the finger movement analysis unit 340 outputs the predicted work content for each time period, as shown in FIG. 4 .

[0067] 12 shows an example of the configuration of a finger motion analysis unit 340 according to some embodiments. In the example of Fig. 12, the finger motion analysis unit 340 includes a skeleton estimation unit 341, an object detection unit 342, and an inference unit 343.

[0068] The skeleton estimation unit 341 has the same configuration as the skeleton estimation unit 331 of the finger movement learning unit 330. The skeleton estimation unit 341 estimates the finger skeletons of a person in the received video. The skeleton estimation unit 331 of the finger movement learning unit 330 and the skeleton estimation unit 341 of the finger movement analysis unit 340 may be combined into a single block. The skeleton estimation unit 341 outputs feature amounts of the finger skeletons detected in the received video.

[0069] The object detection unit 342 has the same configuration as the object detection unit 332 of the finger motion learning unit 330. The object detection unit 342 detects objects in the received video. The object detection unit 332 of the finger motion learning unit 330 and the object detection unit 342 of the finger motion analysis unit 340 may be combined into a single block. The object detection unit 342 outputs feature amounts of an image including a part or tool detected in the received video.

[0070] The inference unit 343 infers the work content of the worker based on the skeleton estimation result of the skeleton estimation unit 341 and the object detection result of the object detection unit 342. For example, the inference unit 343 inputs the finger skeleton feature amount estimated by the skeleton estimation unit 341 and the feature amount of the image of the part or tool detected by the object detection unit 342 to the trained finger movement recognition model 321, and infers the work content corresponding to the finger movement based on the co-occurrence relationship between the finger skeleton feature amount and the image feature amount.

[0071] 9 , the output unit 350 outputs the analysis result of the finger movement analysis unit 340 to the display device 400. For example, the output unit 350 may output the received video and the work content obtained by analyzing the video to the display device 400. For example, the output unit 350 may output the estimation result of the finger skeleton and the detection result of parts, etc. When the output unit 350 receives the hand movement analysis result from the edge device 100, it may output the hand movement analysis result to the display device 400.

[0072] 13 shows an example of the operation of the monitoring system 1 according to some embodiments. For example, the following description will be given assuming that the edge device 100 executes S101 to S103 and S106, the control server 200 executes S104 to S105, and the management server 300 executes S107 to S109, but this is not limiting and any of the devices may execute each process.

[0073] 13 , the edge device 100 acquires images from multiple cameras 101 (S101). The multiple cameras 101 capture images of the work area of ​​each work process and transmit the captured images to the edge device 100. The image acquisition unit 110 acquires the images transmitted from the multiple cameras 101, i.e., the work images, via the network NW1. For example, the work images include images of the hands of a worker performing work in the work area, parts and tools to be worked on, etc.

[0074] Next, the edge device 100 analyzes the worker's hand movements based on the multiple acquired work videos (S102). The object detection unit 121 of the hand movement analysis unit 120 uses an object recognition engine to detect object regions, such as rectangles, within images included in the work videos and recognize the object types of objects within the detected object regions. The object detection unit 121 outputs detection results for objects detected in the work area of ​​the work videos whose object types include a person's hand. The work situation analysis unit 122 of the hand movement analysis unit 120 generates cycle waveforms, cycle times, etc. based on the detection results of the worker's hands in the work area of ​​the work videos and outputs the generated cycle waveforms, cycle times, etc. as analysis results.

[0075] Next, the edge device 100 notifies the control server 200 of the analysis result of the hand movement (S103). The hand movement analysis unit 120 transmits the analysis result of the worker's hand movement, which is analyzed from multiple work videos, to the control server 200 via the network NW2 or another communication path.

[0076] Next, the control server 200 identifies an area to be analyzed in detail (S104). The analysis result acquisition unit 210 acquires the analysis results of the hand movements for the multiple work videos from the edge device 100 via the network NW2 or another communication path. The detailed analysis necessity determination unit 220 identifies work videos or work areas to be analyzed in detail from the multiple work videos based on the analysis results of the hand movements for the multiple work videos acquired. Here, an example of identifying a work area (area) to be analyzed in detail will be described.

[0077] 14 shows an example of the detailed analysis area identification process (S104) according to some embodiments. In the example of Fig. 14, the detailed analysis necessity determination unit 220 detects signs of abnormality based on the acquired analysis results (S201), and identifies the area of ​​the video in which the signs of abnormality were detected as an area to be analyzed in detail (S202). When the detailed analysis necessity determination unit 220 detects signs of abnormality in a work area, it identifies the detected work area as an area to be analyzed in detail.

[0078] For example, if a cycle waveform of a work area is acquired as a result of hand movement analysis, the detailed analysis necessity determination unit 220 may detect signs of abnormality in the work area based on the shape (waveform pattern) of the acquired cycle waveform. FIG. 15A shows an example of a normal cycle waveform, and FIG. 15B shows an example of an abnormal cycle waveform. In FIG. 15A , work is performed at a regular cycle, while in FIG. 15B , the cycle period of the work is disrupted. For example, the detailed analysis necessity determination unit 220 may compare the shape of the acquired cycle waveform with that of a normal cycle waveform and determine whether or not there is a sign of abnormality based on whether or not the acquired cycle waveform and the normal cycle waveform are identical or similar. For example, if a cycle waveform like that shown in FIG. 15B is acquired, the acquired cycle waveform and the normal cycle waveform shown in FIG. 15A do not match (are similar), so it is determined that there is a sign of abnormality. If the acquired cycle waveform is distorted (for example, if it is not of a regular period), as shown in Figure 15B, the detailed analysis necessity determination unit 220 may determine that there is a sign of an abnormality, as there may have been a problem with the work.

[0079] Furthermore, when the cycle time of a work area is acquired as a result of the analysis of the manual movements, the detailed analysis necessity determination unit 220 may detect signs of abnormality in the work area based on the acquired cycle time. For example, the acquired cycle time may be compared with a predetermined work time, and the presence or absence of signs of abnormality may be determined based on the comparison result. The reference predetermined work time may be, for example, the time designed when the production line was designed. For example, the detailed analysis necessity determination unit 220 may determine that there are signs of abnormality when the difference between the acquired cycle time and the predetermined work time is greater than a predetermined value, when the difference between the average of the acquired cycle times and the predetermined work time is greater than a predetermined value, or when the variance of the acquired cycle times is greater than a predetermined value.

[0080] FIG. 16 shows an example of detecting signs of an abnormality based on cycle time. FIG. 16 illustrates an example of manufacturing televisions using work processes W1 to W8. In this example, work processes W1 and W3 are performed in parallel, followed by work processes W4 to W8. Furthermore, work process W2 must be performed before work process W6. For example, analysis of manual movements reveals that the cycle times for each process are 23 seconds for work process W1, 63 seconds for work process W2, 40 seconds for work process W4, 6 seconds for work process W5, and 43 seconds for work process W6. In this example, the cycle time for work process W2 is longer than the specified work time, which may result in delays to work process W6. Therefore, it is determined that there is a sign of an abnormality in work process W2.

[0081] Furthermore, when a detection result including hand position information is acquired as a result of the hand movement analysis, the detailed analysis necessity determination unit 220 may detect signs of abnormality based on the hand movement pattern. The detailed analysis necessity determination unit 220 may detect the absence or presence of a specific task from the hand movement pattern (movement history) in the work area, and detect signs of abnormality. For example, this task requires the hand to move to the area of ​​a specific plate, but if the hand does not reach the specific plate, a sign of abnormality may be detected.

[0082] 17 shows another example of the detailed analysis area identification process (S104) according to some embodiments. In the example of FIG. 17, the detailed analysis necessity determination unit 220 identifies features of the subject included in the work video (S211) and identifies an area including the identified features of the subject as an area to be analyzed in detail (S212). When the detailed analysis necessity determination unit 220 identifies the features of the subject in the work area, the detailed analysis necessity determination unit 220 identifies the work area or an area within the work area including the identified features of the subject as an area to be analyzed in detail.

[0083] For example, the characteristics of the object to be photographed may be set in advance in a table or the like for each work process or worker. The example is not limited to using a table; the characteristics of the object to be photographed may also be identified using a learning model that has machine-learned the characteristics of the object to be photographed according to the work process or worker. FIG. 18 shows an example of a feature table for setting the characteristics of the object to be photographed. In the example of FIG. 18, the characteristics of the object to be photographed are associated with each work process. For example, when a detection result including hand position information is obtained as a result of hand movement analysis, the detailed analysis necessity determination unit 220 refers to the feature table of FIG. 18 and identifies, as the area to be analyzed in detail, an area including the characteristics of the object to be photographed that corresponds to the work process in the work area where the hand movement was analyzed. For example, if the work process is W1, the area including the right hand is identified as the area to be analyzed in detail; if the work process is W2, the area including the left hand is identified as the area to be analyzed in detail; and if the work process is W3, the area where the worker's hand is placed is identified as the area to be analyzed in detail. The area where the hand is placed may be a location where a specific part or tool is located.

[0084] FIG. 19 shows another example of a feature table for setting the features of the subject. In the example of FIG. 19, dominant hand is associated with each worker as a feature of the subject. For example, when a detection result including hand position information is acquired as a result of hand movement analysis, the detailed analysis necessity determination unit 220 references the feature table of FIG. 19 and identifies the area including the feature (dominant hand) of the subject corresponding to the worker included in the video as the area to be analyzed in detail. For example, if the worker is P1 to P3, the area including the right hand is identified as the area to be analyzed in detail, and if the worker is P4, the area including the left hand is identified as the area to be analyzed in detail. For example, the worker may be identified from an image of a person detected from the work video, and the area of ​​the dominant hand associated with the worker may be identified.

[0085] 20 shows another example of the detailed analysis area identification process (S104) according to some embodiments. In the example of Fig. 20, the detailed analysis necessity determination unit 220 acquires the status of the subject included in the work video (S221), and identifies an area to be analyzed in detail based on the acquired status of the subject (S222). When the detailed analysis necessity determination unit 220 acquires the status of the subject in the work area, it designates the work area or an area within the work area that includes characteristics of the subject whose status has been acquired as the area to be analyzed in detail.

[0086] For example, the detailed analysis necessity determination unit 220 identifies a worker from an image of a person detected from the work video, and acquires status information associated with the worker from a database or the like in which information about the worker is registered. For example, the necessity of detailed analysis for each worker's status may be set in advance in a table or the like. Note that the example is not limited to using a table, and the necessity of detailed analysis may also be determined using a learning model that has been machine-learned to determine the necessity of detailed analysis according to the worker's status.

[0087] FIG. 21 shows an example of a status table for setting whether detailed analysis is required. In the example of FIG. 21, whether attention is required is associated with each worker's status. "Attention required" indicates that detailed analysis is required. For example, when the detailed analysis necessity determination unit 220 acquires the status of a worker included in a work video, it refers to the status table of FIG. 21 and determines whether detailed analysis is required according to the worker's status. The detailed analysis necessity determination unit 220 identifies an area including a worker requiring attention as an area to be analyzed in detail. For example, if the worker is a newcomer, it determines that attention is required, and identifies the area including the newcomer as an area to be analyzed in detail. If the worker is a veteran, it determines that attention is not required, and does not identify the area including the veteran as an area to be analyzed in detail.

[0088] 13 , following S104, the control server 200 instructs the edge device 100 on the video to be analyzed in detail (S105). When the detailed analysis necessity determination unit 220 identifies the area to be analyzed in detail, the video instruction unit 230 instructs the edge device 100 on the area to be analyzed in detail via the network NW2 or another communication path. For example, the video instruction unit 230 transmits information that identifies the work video to be analyzed in detail or the work area within the work video.

[0089] Next, the edge device 100 transmits the instructed video to the management server 300 (S106). The video selection unit 130 receives an instruction for video to be analyzed in detail from the control server 200 via the network NW2 or another communication path. The video selection unit 130 selects the instructed work video or video of the work area as the video to be transmitted from the multiple work videos acquired from the camera 101. The video transmission unit 140 transmits the selected video to the management server 300 via the network NW2.

[0090] Next, the management server 300 receives the video from the edge device 100 (S107). The video receiving unit 310 receives the video from the edge device 100 via the network NW2.

[0091] Next, the management server 300 analyzes the finger movements based on the received video (S108). The skeleton estimation unit 341 of the finger movement analysis unit 340 uses a skeleton estimation engine to estimate the finger skeletons in the received video and outputs feature quantities of the estimated finger skeletons. The object detection unit 342 of the finger movement analysis unit 340 uses an object recognition engine to recognize objects in the received video and output feature quantities of images including parts, tools, etc. used for work. The inference unit 343 of the finger movement analysis unit 340 uses the trained finger movement recognition model 321 to infer the work content corresponding to the finger movements based on the feature quantities of the finger skeletons and the feature quantities of the image.

[0092] Next, the management server 300 outputs the analysis results of the finger movements to the display device 400 (S109). The output unit 350 outputs the video received from the edge device 100 and the analysis results of the finger movements analyzed by the finger movement analysis unit 340 to the display device 400. The display device 400 displays the video transmitted from the edge device 100 on the display screen, as well as the analysis results of the finger movements in the video (task content).

[0093] FIG. 22 shows an example of the results of hand gesture analysis by the edge device 100, and FIG. 23 shows an example of the results of finger gesture analysis by the management server 300. In the example of FIG. 22, the edge device 100 analyzes hand gestures in work video of work processes W1 to W3, and the cycle times of work processes W1 to W3 are obtained from the hand gesture analysis results. For example, the control server 200 identifies the video of work process W2 as the video to be analyzed in detail because the cycle time of work process W2 is longer than that of work processes W1 and W3. In the example of FIG. 23, the management server 300 analyzes finger gestures in the video of work process W2, and the finger gesture analysis results indicate that tasks A1 to A4 are being performed in work process W2. For example, the finger gesture analysis results shown in FIG. 23 may be displayed on the display device 400.

[0094] As described above, in this embodiment, hand movement analysis is performed on video captured by a camera in an edge device, and the control server determines whether detailed analysis is necessary based on the results of the hand movement analysis. Furthermore, finger movement analysis is performed on video of an area determined by the management server to require detailed analysis. This allows finger movement analysis to be performed only on video determined to require detailed analysis based on the results of the hand movement analysis, thereby enabling more efficient analysis processing than when finger movement analysis is performed on all video, thereby reducing the load and cost of the finger movement analysis processing. For example, when finger movement analysis is performed on all video, only one camera can be connected to the edge device. However, in this embodiment, multiple cameras can be connected to the edge device, thereby reducing the cost of introducing finger movement analysis.

[0095] (Embodiment 2) Next, embodiment 2 will be described. This embodiment is an example of identifying an area to be sharpened in a video to be analyzed in detail. Note that this embodiment can be implemented in combination with embodiment 1, and each configuration shown in embodiment 1 may be used as appropriate.

[0096] 24 shows an example of the configuration of each device in a monitoring system 1 according to some embodiments. In the example of Fig. 24, the edge device 100 includes an image quality control unit 150 in addition to the configuration of Fig. 9.

[0097] The image quality control unit 150 controls the image quality of the image selected by the image selection unit 130. The image quality control unit 150 receives instructions from the control server 200 indicating areas to be sharpened in the image to be transmitted, and sharpens the specified areas. The image quality control unit 150 makes the sharpened areas higher quality than the non-sharpened areas, and makes the non-sharpened areas lower quality than the sharpened areas. For example, by changing the compression rates of the high-image-quality area and the low-image-quality area, the high-image-quality area may be made higher quality than the low-image-quality area, and the low-image-quality area may be made lower quality than the high-image-quality area.

[0098] In the example of FIG. 24, the control server 200 includes a sharpening region specifying unit 240 in addition to the configuration of FIG.

[0099] The sharpening region specifying unit 240 specifies a sharpening region in the video to be analyzed in detail specified by the detailed analysis necessity determination unit 220. For example, the sharpening region specifying unit 240 may specify a sharpening region based on the characteristics of the subject included in the work video. The detailed analysis necessity determination unit 220 may specify a video to be analyzed in detail based on the detection result of a sign of abnormality, and the sharpening region specifying unit 240 may specify a sharpening region based on the characteristics of the subject included in the work video.

[0100] Furthermore, the sharpening region specifying unit 240 may specify the sharpening region based on the status of the subject included in the work video. The detailed analysis necessity determining unit 220 may specify the video to be analyzed in detail based on the detection result of the sign of abnormality, and the sharpening region specifying unit 240 may specify the sharpening region based on the status of the subject included in the work video. The other configurations are the same as those in the example described in the first embodiment.

[0101] Fig. 25 shows an example of the operation of the monitoring system 1 according to some embodiments. In the example of Fig. 25, similar to Fig. 13, the edge device 100 acquires images from multiple cameras 101 (S101), analyzes hand movements (S102), and notifies the control server 200 of the analysis results (S103).

[0102] Next, the control server 200 identifies an area to be analyzed in detail (S104), and further identifies an area to be sharpened (S110). For example, as in the example of Fig. 14, the detailed analysis necessity determination unit 220 detects signs of abnormality based on the acquired analysis results (S201), and identifies the area where the signs of abnormality are detected as an area to be analyzed in detail (S202). The sharpening area identification unit 240 identifies an area to be sharpened from the identified work video or work area to be analyzed in detail.

[0103] 26 shows an example of the sharpening region identification process (S110) according to some embodiments. In the example of FIG. 26, the sharpening region identification unit 240 identifies features of the subject included in the work video (S301) and identifies an area including the identified features of the subject as a sharpening region (S302). When the sharpening region identification unit 240 identifies the features of the subject in the work area, the area within the work area including the identified features of the subject is designated as a sharpening region.

[0104] The sharpening region specifying unit 240 may specify the features of the subject to be filmed, similarly to S211 in Fig. 17 . For example, when a detection result including hand position information is acquired as a result of analyzing a hand movement, the sharpening region specifying unit 240 may refer to the feature table in Fig. 18 and specify, as a sharpening region, a region including the features of the subject to be filmed (right hand, left hand, etc.) corresponding to the work process in the work area in which the hand movement was analyzed. Furthermore, for example, when a detection result including hand position information is acquired as a result of analyzing a hand movement, the sharpening region specifying unit 240 may refer to the feature table in Fig. 19 and specify, as a sharpening region, a region including the features of the subject to be filmed (dominant hand) corresponding to the worker included in the video.

[0105] 27 shows another example of the sharpening area identification process (S110) according to some embodiments. In the example of FIG. 27, the sharpening area identification unit 240 acquires the status of a subject included in the work video (S311), and identifies a sharpening area based on the acquired status of the subject (S312). When the sharpening area identification unit 240 acquires the status of the subject in the work area, it designates the work area or an area within the work area that includes features of the subject whose status has been acquired as the sharpening area.

[0106] The sharpening region specifying unit 240 may specify the sharpening region based on the status of the subject to be filmed, similar to S221 and S222 in Fig. 20. For example, when the sharpening region specifying unit 240 acquires the status of a worker included in the work video, the sharpening region specifying unit 240 refers to the status table in Fig. 21 and determines the sharpening region according to the worker's status. For example, the sharpening region specifying unit 240 specifies an area requiring attention in Fig. 21 (such as an area including a new employee) as the sharpening region.

[0107] 25 , following S110, the control server 200 instructs the edge device 100 on the video to be analyzed in detail and the sharpening region (S111). After the detailed analysis necessity determination unit 220 identifies the region to be analyzed in detail and the sharpening region identification unit 240 identifies the region to be sharpened, the video instruction unit 230 instructs the edge device 100 on the video to be analyzed in detail and the sharpening region.

[0108] Next, the edge device 100 sharpens the specified area of ​​the image (S112). The edge device 100 receives instructions on the image to be analyzed in detail and the area to be sharpened from the control server 200. The image selection unit 130 selects the specified work image or image of the work area from the multiple work images acquired from the camera 101 as the image to be transmitted.

[0109] The image quality control unit 150 sharpens the designated area in the selected image. For example, by lowering the compression rate of the sharpened area compared to the compression rate of other areas, the image quality of the sharpened area within the work image or work area is increased and the image quality of the other areas is reduced.

[0110] Next, the edge device 100 transmits the sharpened image to the management server 300 via the network NW2. Thereafter, similar to Fig. 13 , the management server 300 receives the image (S107), analyzes the finger movement (S108), and outputs the analysis result (S109).

[0111] As described above, in this embodiment, the image to be analyzed in detail is identified based on the results of the hand movement analysis, and the sharpening area in the image is identified. This sharpens the area of ​​the image to be transmitted from the edge device to the management server that performs the finger movement analysis, thereby improving the accuracy of the finger movement analysis in the management server.

[0112] The present disclosure is not limited to the above-described embodiments and may be modified as appropriate without departing from the spirit and scope of the present disclosure. For example, in the above-described embodiments, detailed analysis processing is described as being performed by a management server on the cloud side, but this is not limited to this. That is, detailed analysis processing does not necessarily have to be performed. For example, lightweight processing may be performed on an edge device, and areas requiring more detailed analysis (heavy processing) may be presented to an operator. This allows the operator to select areas requiring more detailed processing. Furthermore, a heavy analysis system may be proposed for implementation in a manufacturing line where areas requiring more detailed analysis frequently occur.

[0113] Each component in the above-described embodiments may be configured with hardware or software, or both, and may be configured with a single piece of hardware or software, or may be configured with multiple pieces of hardware or software. Each device and each function (processing) such as the edge device, control server, and management server may be realized by a computer 30 having a processor 31 such as a CPU (Central Processing Unit) and a memory 32 serving as a storage device, as shown in FIG. 28. For example, a program for performing the method (determination method) in the embodiment may be stored in the memory 32, and each function may be realized by the processor 31 executing the program stored in the memory 32.

[0114] These programs include instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The programs may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The programs may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0115] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0116] Each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0117] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0118] (Supplementary Note 1) A determination system comprising: an acquisition means for acquiring a feature amount of each region in an image; and a determination means for determining whether or not analysis of each region is necessary based on the acquired feature amount of each region. (Supplementary Note 2) The determination system described in Supplementary Note 1, wherein the determination means determines whether or not analysis is necessary based on signs of abnormality in each region detected according to the feature amount of each region. (Supplementary Note 3) The determination system described in Supplementary Note 1 or 2, wherein the feature amount is based on at least one of a work time for an operation performed in the image, a subject photographed in the image, and a level of proficiency in the operation of a person photographed in the image. (Supplementary Note 4) The determination system described in any one of Supplements 1 to 3, wherein the acquisition means acquires a first analysis result by a first analysis for each region in the image, and the determination means determines whether or not a second analysis, which is more detailed than the first analysis, is necessary for each region based on the first analysis result of the each region. (Supplementary Note 5) The judgment system according to Supplementary Note 4, comprising: a first device including a first analysis processing means that performs the first analysis; and a second device including a second analysis processing means that performs the second analysis, wherein the first device includes a transmission means that transmits to the second analysis processing means an image including an area that has been determined to require the second analysis from among the areas where the first analysis processing means has performed the first analysis. (Supplementary Note 6) The judgment system according to Supplementary Note 5, wherein the determination means identifies an image quality control area that controls image quality in the area determined to require the second analysis, and the transmission means transmits to the second analysis processing means an image in which the image quality of the identified image quality control area has been controlled. (Supplementary Note 7) A judgment device comprising: acquisition means that acquires feature amounts of each area in an image; and judgment means that judges whether or not analysis is required for each of the areas based on the acquired feature amounts of each area. (Supplementary Note 8) The determination device according to Supplementary Note 7, wherein the determination means determines whether or not the analysis is necessary based on signs of abnormality in each of the regions detected according to feature amounts of each of the regions. (Supplementary Note 9) The determination device according to Supplementary Note 7 or 8, wherein the feature amounts are based on at least one of a work time for the work performed in the image, a subject photographed in the image, and a level of proficiency in the work of a person photographed in the image.(Supplementary Note 10) The determination device according to any one of Supplementary Notes 7 to 9, wherein the acquisition means acquires a first analysis result from a first analysis for each region in the image, and the determination means determines, based on the first analysis result for each region, whether or not a second analysis is necessary that is more detailed than the first analysis for each region. (Supplementary Note 11) A determination method that acquires a feature amount for each region in the image and determines, based on the acquired feature amount for each region, whether or not analysis is necessary for each region. (Supplementary Note 12) The determination method according to Supplementary Note 11, wherein the determination is made based on signs of abnormality in each region detected according to the feature amount for each region. (Supplementary Note 13) The determination method according to Supplementary Note 11 or 12, wherein the feature amount is based on at least one of a work time for a task performed in the image, a subject photographed in the image, and a level of proficiency in the task of a person photographed in the image. (Supplementary Note 14) The determination method according to any one of Supplementary Notes 11 to 13, comprising: obtaining a first analysis result from a first analysis for each region within the image; and determining whether or not a second analysis that is more detailed than the first analysis is necessary for each region based on the first analysis result for each region. (Supplementary Note 15) The determination method according to Supplementary Note 14, comprising: performing the first analysis by a first device; performing the second analysis by a second device; and transmitting an image including a region determined to require the second analysis from the first device to the second device, among the regions subjected to the first analysis. (Supplementary Note 16) The determination method according to Supplementary Note 15, comprising: identifying an image quality control region that controls image quality in the region determined to require the second analysis; and transmitting an image in which the image quality of the identified image quality control region has been controlled from the first device to the second device.

[0119] REFERENCE SIGNS LIST 1 Monitoring system 10 Determination system 11 Acquisition unit 12 Determination unit 20 Determination device 30 Computer 31 Processor 32 Memory 100 Edge device 101 Camera 110 Video acquisition unit 120 Hand movement analysis unit 121 Object detection unit 122 Work situation analysis unit 130 Video selection unit 140 Video transmission unit 150 Image quality control unit 200 Control server 210 Analysis result acquisition unit 220 Detailed analysis necessity determination unit 230 Video instruction unit 240 Sharpening area identification unit 300 Management server 310 Video reception unit 320 Memory unit 321 Finger movement recognition model 330 Finger movement learning unit 331 Skeleton estimation unit 332 Object detection unit 333 Learning unit 340 Finger movement analysis unit 341 Skeleton estimation unit 342 Object detection unit 343 Inference unit 350 Output unit 400 Display device

Claims

1. A means for obtaining feature quantities for each region within an image, A determination means for determining whether analysis is necessary in each of the aforementioned regions based on the acquired feature quantities of each region, A judgment system equipped with the following features.

2. The determination means determines whether the analysis is necessary based on signs of abnormality in each region detected according to the characteristic quantities of each region. The determination system according to claim 1.

3. The aforementioned feature is based on at least one of the following: the time taken for the work performed in the image, the subject being photographed in the image, and the level of proficiency of the person being photographed in the image in performing the work. The determination system according to claim 1 or 2.

4. The acquisition means acquires the first analysis results obtained by performing a first analysis on each region within the image. The determination means determines, based on the first analysis results of each region, whether a second analysis more detailed than the first analysis is necessary in each region. The determination system according to claim 1 or 2.

5. A first apparatus including a first analytical processing means for performing the first analysis, The apparatus comprises a second device including a second analytical processing means for performing the second analysis, The first apparatus includes a transmission means for transmitting an image to the second analysis processing means that includes an area from the area in which the first analysis processing means performed the first analysis, in which the area has been determined to require the second analysis. The determination system according to claim 4.

6. The determination means identifies an image quality control region in which image quality is controlled in the region where the second analysis is determined to be necessary. The transmission means transmits the image with controlled image quality of the identified image quality control region to the second analysis processing means. The determination system according to claim 5.

7. A means for obtaining feature quantities for each region within an image, A determination means for determining whether analysis is necessary in each of the aforementioned regions based on the acquired feature quantities of each region, A determination device equipped with the following features.

8. The determination means determines whether the analysis is necessary based on signs of abnormality in each region detected according to the characteristic quantities of each region. The determination device according to claim 7.

9. By obtaining the feature quantities of each region in the image, Based on the acquired feature quantities for each region, the necessity of analysis in each region is determined. Judgment method.

10. Obtain the feature quantities of each region in the image, Based on the acquired feature quantities for each region, the necessity of analysis in each region is determined. A program that causes a computer to perform a process.