Image processing apparatus, image processing method, and storage medium

Abstract

An image processing apparatus has an image acquisition unit that acquires a dynamic image obtained by an image sensor that captures a work area, an image generation unit that generates an image of a face portion of a worker who works in the work area from the dynamic image, a statistical information generation unit that generates statistical information of work of the worker from the dynamic image, a data generation unit that generates data that associates the generated image of the face portion of the worker with the generated statistical information of work of the worker, and an output unit that outputs the generated data.

Description

Technical Field

[0001] This invention relates to a technique for analyzing the actions of objects using image sensors. Background Technology

[0002] In the factory automation market, the following technologies are actually used: image sensors installed on cell lines to monitor the operators, the number of machines working, and the work positions, and to analyze the content of each work step, thereby helping to improve productivity.

[0003] For example, Patent Document 1 proposes a technique that reads ID tags installed on workpieces operating on a production line via online entry and exit points to determine the start and end times of workpiece operations and to measure the operation time. Additionally, Patent Document 2 proposes a technique that displays time charts and dynamic images related to the work content of the operator being tracked during a series of operations within a work process.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2006-202255

[0007] Patent Document 2: Japanese Patent Application Publication No. 2018-073176 Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] However, in existing technologies, it is necessary to install ID tags on workpieces or designate personnel to track operations to determine the analysis objects of the work content. Furthermore, when using ID tags, the accuracy of measurement time is affected by the detection precision of sensors that read the ID tags, requiring ID tags to be read at each step. Additionally, when users designate operators as tracking objects in product line images, they do not identify each operator individually; therefore, even if the work content of each operator can be understood, it is impossible to identify the individual and grasp the work content.

[0010] The present invention is proposed in view of the above-mentioned actual situation, and provides a technology that can identify individual workers and analyze the work content based on images obtained by image sensors.

[0011] Methods for solving problems

[0012] To achieve the above objectives, the present invention adopts the following structure.

[0013] The first aspect of the present invention relates to an image processing apparatus comprising: an image acquisition unit that acquires a dynamic image of a work area obtained by an image sensor; an image generation unit that generates an image of a face portion of a worker performing work within the work area from the dynamic image; a statistical information generation unit that generates statistical information about the worker's work based on the dynamic image; a data generation unit that generates data that establishes a correlation between the generated image of the worker's face portion and the generated statistical information about the worker's work; and an output unit that outputs the generated data. Thus, since the worker's individual information and statistical information are correlated based on the dynamic image from the image sensor, it is possible to identify the worker and analyze the work.

[0014] Alternatively, the data generation unit can generate data that associates images of the faces of multiple workers generated by the image generation unit with statistical information about the work of multiple workers generated by the statistical information generation unit. The data generation unit then generates new data by merging at least two images from the facial images of the multiple workers and the statistical information associated with those images, based on the data output by the output unit. Furthermore, the at least two images can be images selected based on user input. Thus, when multiple facial images are generated for the same worker, it is possible to combine multiple images into one image while simultaneously merging the statistical information associated with those multiple images.

[0015] Furthermore, the data generation unit may generate new data based on the data output by the output unit, changing the image of the worker's face to another image of the worker's face generated by the image generation unit. Alternatively, the data generation unit may generate data that associates an image of the worker's face selected based on at least one of the following criteria: size, orientation, and facial similarity, with the generated statistical information of the worker's work. Furthermore, the data generation unit may generate data that associates an image of the worker's face selected from the pre-obtained images with the generated statistical information of the worker's work, based on matching pre-obtained images of multiple worker faces with the generated image of the worker's face. Thus, the image of the worker's face can be changed to an image that is easier to identify, thereby facilitating the determination of the worker's statistical information.

[0016] Alternatively, the statistical information generation unit can generate summary and detailed statistical information about the worker's work based on the dynamic image, and the data generation unit can generate data that correlates the image of the worker's face, the generated summary statistical information about the worker's work, and the generated detailed statistical information about the worker's work. Thus, by correlating the image of the worker's face with various information helpful for analyzing the worker's work, more efficient work analysis is possible.

[0017] Alternatively, the data generation unit can generate a dynamic image associated with the statistical information of the worker's work based on the acquired dynamic image. The data generation unit generates data that correlates the image of the worker's face, the statistical information of the worker's work, and the generated dynamic image. Alternatively, the statistical information generation unit can generate statistical information of the worker's work for each of the more than one work steps included in the work area, based on the dynamic image. The data generation unit generates a dynamic image associated with the statistical information of the worker's work for each of the more than one work steps, based on the acquired dynamic image. The data generation unit generates data that correlates the image of the worker's face, the statistical information of each of the more than one work steps of the worker's work, and the dynamic image of each of the more than one work steps. Thus, by identifying the dynamic image that forms the basis of the worker's statistical information, more efficient work analysis can be performed.

[0018] Furthermore, the statistical information generation unit can generate evaluation information representing the evaluation result, which is based on a predetermined benchmark of statistical information about the worker's work generated from the dynamic image. The data generation unit generates data that correlates the image of the worker's face, the generated statistical information about the worker's work, and the evaluation information. Thus, for example, information that should be considered in statistical information analysis, such as work time significantly exceeding or significantly decreasing relative to the target value, can be displayed as an evaluation result, enabling more efficient work analysis.

[0019] Furthermore, the present invention can also capture image processing methods that include at least a portion of the above-described processes, programs for causing a computer to execute these methods, or computer-readable recording media that non-temporarily record such programs. The above-described structures and processes can be combined with each other to constitute the present invention, provided that no technical contradiction arises.

[0020] Invention Effects

[0021] According to the present invention, it is possible to identify individual workers and analyze the content of work based on images acquired by an image sensor. Attached Figure Description

[0022] Figure 1 This is a block diagram illustrating a structural example of an image processing apparatus to which the present invention is applied.

[0023] Figure 2 This is a block diagram illustrating a structural example of a PC (image processing device) according to one embodiment.

[0024] Figure 3 This is a flowchart illustrating an example of the processing flow of a PC involved in one implementation method.

[0025] Figure 4 This is a schematic diagram illustrating a specific example of a display screen involved in one embodiment.

[0026] Figure 5 This is a schematic diagram illustrating a specific example of the display screen involved in Variation Example 1.

[0027] Figure 6 This is a schematic diagram illustrating a specific example of the display screen involved in Variation Example 2.

[0028] Figure 7 This is a schematic diagram illustrating a specific example of the display screen involved in Variation Example 3.

[0029] Figure 8 This is a schematic diagram illustrating a specific example of the display screen involved in Variation Example 4.

[0030] Figure 9 This is a schematic diagram illustrating a specific example of the display screen involved in Variation Example 5.

[0031] Figure 10 This is a schematic diagram illustrating a specific example of the display screen involved in Variation Example 6.

[0032] Figure 11 This is a schematic diagram illustrating a specific example of the display screen involved in Variation Example 7.

[0033] Figure 12 This is a schematic diagram illustrating a specific example of the display screen involved in Variation Example 8. Detailed Implementation

[0034] <Application Example>

[0035] An application example of the present invention will be explained. When analyzing the work content of each worker based on images obtained from image sensors deployed in factory production lines, it is necessary to identify the individual worker. Methods for identifying workers include measuring the intensity of radio waves from the terminal carried by the worker and the access point of a wireless LAN, and measuring the intensity of the measurement based on reading IC tags installed on the worker. However, there are concerns that the accuracy of the measurement decreases due to the influence of various devices on the wireless LAN's radio waves, and that a large number of IC tags need to be prepared for installation on the workers.

[0036] Figure 1 This is a block diagram illustrating a structural example of the image processing apparatus 100 to which the present invention is applied. The image processing apparatus 100 includes an image acquisition unit 101, a face image generation unit 102, a statistical information generation unit 103, a data generation unit 104, and an output unit 105. The image acquisition unit 101 acquires images captured by an image sensor such as an omnidirectional camera disposed on a unit line. The face image generation unit 102 generates images of the face portions of each worker (face images) based on the captured images. In addition, the statistical information generation unit 103 generates statistical information about the work of each worker based on the captured images. Furthermore, the data generation unit 104 generates data that establishes a correlation between the face portion images of the workers generated by the face image generation unit 102 and the statistical information about the work of the workers generated by the statistical information generation unit 103. The output unit 105 outputs the data generated by the data generation unit 104.

[0037] According to the image processing apparatus 100 of the present invention, the operator is identified with high precision based on the image captured by the image sensor, and information representing the analysis content of the operation can be provided for each operator.

[0038] <First Implementation Method>

[0039] The first embodiment of the present invention will be described.

[0040] Figure 2 This is a schematic diagram illustrating a general structural example of the image processing system according to the first embodiment. The image processing system of the first embodiment includes an image sensor, a PC 200 (personal computer; image processing device), and a display device. The image sensor and the PC 200 are interconnected via wired or wireless means, and the PC 200 and the display device are interconnected via wired or wireless means.

[0041] In the first embodiment, it is envisioned that an image sensor captures images of unit lines deployed in a factory, logistics warehouse, or similar location. Furthermore, the area of ​​the unit lines within the captured image becomes a detection area for human detection targeting workers. The image sensor captures images of the unit lines and outputs the captured images to a PC 200. The shooting direction of the image sensor is not particularly limited; in the first embodiment, for example, the image sensor is installed on the ceiling above the floor where the unit lines are deployed, in a direction overlooking the unit lines. The PC 200 performs human detection based on the images captured by the image sensor and outputs the detection results to a display device. Examples of display devices include a monitor and an information processing terminal (smartphone, etc.).

[0042] Furthermore, in the first embodiment, the PC200 is a separate device from the image sensor and the display device, but the PC200 can also be integrated with the image sensor or the display device. Moreover, the location where the PC200 is installed is not particularly limited. For example, the PC200 can be installed in the same room as the image sensor. Additionally, the PC200 can also be a computer in the cloud.

[0043] PC200 includes an input unit 210, a control unit 220, a storage unit 230, and an output unit 240. The control unit 220 includes an image generation unit 221, a statistical information generation unit 222, and a data generation unit 223. Furthermore, the input unit 210, image generation unit 221, statistical information generation unit 222, data generation unit 223, and output unit 240 correspond to the image acquisition unit, image generation unit, statistical information generation unit, data generation unit, and output unit of the present invention, respectively.

[0044] The input unit 210 acquires dynamic image data obtained by capturing unit lines through the image sensor and outputs it to the control unit 220. The dynamic image data acquired by the input unit 210 from the image sensor is stored in the storage unit 230 by the control unit 220. Furthermore, the dynamic image data acquired by the input unit 210 is dynamic image data obtained by capturing unit lines during a specified period (which may also be appropriately set to 1 hour, 1 day, etc.).

[0045] The control unit 220 includes a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), etc. The control unit 220 performs control of various parts within the PC 200 and various information processing.

[0046] The image generation unit 221 detects the face of a worker present in the captured image based on the dynamic image data acquired by the input unit 210. The detection of the worker's face in the dynamic image data acquired by the image sensor can be achieved through facial similarity detection based on existing technology, so detailed explanation is omitted here. Furthermore, the image generation unit 221 determines whether the worker whose face was detected is the worker for whom the work content is being analyzed. In this embodiment, information for identifying the worker whose face was detected and the worker for whom analysis is being conducted, as well as the worker's worker ID, are stored in advance in the storage unit 230 based on the detection position in the dynamic image, the similarity of image features, etc. Based on the information stored in the storage unit 230, the image generation unit 221 identifies the worker for whom analysis is being conducted from the detected face and stores the identified worker's worker ID and the detected position information in the dynamic image in the storage unit 230. In addition, if there is no operator whose face portion is identified as the detected face portion, the image generation unit 221 issues a new operator ID as the operator corresponding to the detected face portion, and stores the issued operator ID and the position information in the detected dynamic image in the storage unit 230.

[0047] In addition, the image generation unit 221 also generates an image of the detected face portion. The image of the operator's face portion generated by the image generation unit 221 can be stored in the storage unit 230 or sent to various parts of the PC 200 that processes the image.

[0048] The statistical information generation unit 222 generates statistical information on the detected work content of each operator based on the dynamic image data stored in the storage unit 230, the data acquired by the input unit 210, and the information generated by the image generation unit 221. As an example of statistical information, a timeline diagram representing the work time in each step of the unit line can be provided. Furthermore, the statistical information may also include information related to the target work time, which serves as an indicator of the work time in each step and can be compared with the work time of each operator. Moreover, the calculation of the work time of each operator for each step based on the image captured by the image sensor and the creation of the timeline diagram based on the calculation results can be achieved using existing technology, therefore detailed explanations are omitted here.

[0049] The data generation unit 223 generates data that associates images of the facial features of each worker determined by the image generation unit 221 with statistical information about the work content of each worker generated by the statistical information generation unit 222. In this embodiment, the data generated by the data generation unit 223 is envisioned to be data such as images that can be displayed by a display device connected to the output unit 240, but it may also be data that can be processed by other information processing devices.

[0050] In addition to the statistical information mentioned above, the storage unit 230 also stores the program executed by the control unit 220, various data used in the control unit 220, etc. For example, the storage unit 230 is an auxiliary storage device such as a hard disk drive or a solid-state drive. The output unit 240 outputs the image generated by the data generation unit 223 to the display device. Alternatively, the image generated by the data generation unit 223 can be stored in the storage unit 230 and output from the output unit 240 at any time.

[0051] Figure 3 This is a flowchart illustrating an example of the processing flow of PC200. After acquiring dynamic image data from the image sensor via input unit 210 and storing it in storage unit 230, PC200 executes... Figure 3 The processing flow.

[0052] First, in step S301, the control unit 220 acquires dynamic image data captured by the image sensor stored in the storage unit 230.

[0053] Next, the image generation unit 221 repeatedly executes steps S302 to S305 for each frame of the motion image data obtained from the storage unit 230. In step S302, the image generation unit 221 detects the face portion of the operator based on the frame of the motion image data obtained from the storage unit 230. Next, in step S303, the image generation unit 221 obtains information from the storage unit 230 based on the detected face portion to determine whether it is the operator of the analysis object, and identifies the operator and the operator of the analysis object corresponding to the face portion detected in step S302. If the image generation unit 221 can identify the operator of the analysis object based on the detected face portion, it establishes an association between the operator's operator ID and the operator's position information in the frame and stores it in the storage unit 230. In addition, if the image generation unit 221 cannot identify the operator of the analysis object from the detected face portion, it issues a new operator ID to the operator of the detected face portion, establishes an association between the issued operator ID and the operator's position information in the frame and stores it in the storage unit 230.

[0054] Furthermore, in step S304, the image generation unit 221 determines the operator's work steps based on the information related to the unit line captured by the image sensor and the operator's position information within the frame stored in the storage unit 230, and establishes a correlation between the determined work steps and the aforementioned operator's position information and stores them in the storage unit 230.

[0055] Next, in step S305, the image generation unit 221 generates an image of the worker's face portion detected in step S302 based on the frame images, associates the generated face portion image with the worker ID, and stores it in the storage unit 230. Furthermore, although the image generation unit 221 is designed to acquire the worker's face portion image for each frame acquired by the input unit 210, it can acquire the worker's face portion image at arbitrary time intervals during the repeated processing of steps S302 to S305, or it can acquire the face portion image of each worker only once.

[0056] Next, in step S306, the image generation unit 221 determines whether the processing steps S302 to S305 have been performed on all frames of the motion image data acquired in step S301. If there are frames for which the processing steps S302 to S305 were not performed (S306: No), the image generation unit 221 returns the processing to step S302 and performs the processing steps S302 to S305 on that frame. On the other hand, if the processing steps S302 to S305 have been performed on all frames of the motion image data (S306: Yes), the image generation unit 221 advances the processing to step S307.

[0057] In step S307, the statistics generation unit 222 generates statistical information such as the operation time and average operation time for each operation step within the unit line, based on the data of each operator stored in the storage unit 230 after the processing in steps S302 to S305. Here, the statistics generated by the statistics generation unit 222 are envisioned as a time sequence diagram representing the operation time of each step. Furthermore, the statistics generation unit 222 associates the generated time sequence diagrams for each operator with the operator ID and stores them in the storage unit 230.

[0058] Next, in step S308, the data generation unit 223 obtains the images of each worker's face generated by the image generation unit 221 and the statistical information generated by the statistical information generation unit 222 from the storage unit 230, and generates data that associates the images of each worker's face with the statistical information based on the worker ID.

[0059] Next, in step S309, the output unit 240 outputs the data generated by the data generation unit 223 in step S308 to the display device. Alternatively, in step S309, the output unit 240 may instead output the data generated by the data generation unit 223 and store that data in the storage unit 230.

[0060] Figure 4 It means in Figure 3This is a schematic diagram of an example of a display screen 400 on a display device for data generated by the data generation unit 223 in the processing flow. The display screen 400 displays facial images 401 to 405 of each worker acquired by the image generation unit 221, identification numbers of each worker ("ID1" to "ID5" in the figure), and a timeline diagram showing the work content of each worker generated by the statistical information generation unit 222.

[0061] In display screen 400, the facial images, identification numbers, and timing diagrams of each worker are configured in a way that ensures the user of display screen 400 understands their interrelationships. In this embodiment, in the unit line captured by the image sensor, the worker's work steps are divided into steps 1 to 5, and the statistics generation unit 222 displays timing diagrams 406 to 410 representing the work time of each generated step. Additionally, display screen 400 also displays a flowchart 411 showing a target value created based on the target work time of each step preset by the statistics generation unit 222.

[0062] Therefore, according to this embodiment, based on the dynamic image data of the image sensor of the shooting unit line, data is generated that establishes a correlation between the facial images of each worker performing work on the unit line and statistical information related to the work. This avoids the costs of installing tags on workers and eliminates concerns about reduced accuracy in worker designation when using communication systems such as wireless LANs to identify workers. It enables high-precision designation of workers and the establishment of a correlation between workers and statistical information related to the work content.

[0063] <Other>

[0064] The above embodiments are merely illustrative examples illustrating the structure of the present invention. The present invention is not limited to the specific embodiments described above, and various modifications can be made within the scope of its technical concept. For example, the structures and processes of the above embodiments can be combined with each other. Modifications to the above embodiments will be described below. Furthermore, in the following description, structures identical to those in the above embodiments will be marked with the same symbols, and detailed descriptions will be omitted.

[0065] <Variation Example 1>

[0066] In Modification 1, during the PC200-based processing described above, the operator corresponding to the detected face image should be recognized as the same operator already stored in the storage unit 230. However, since operator identification is based on image processing, if the identification process fails, there is a possibility that the detected face portion will be associated with a new operator ID. In this case, the above processing results in the statistical information for the same operator being divided into multiple statistical information sets.

[0067] exist Figure 5In the example shown, an image of the face of worker ID "ID2" was mistakenly identified as a new worker and assigned another worker ID "ID3". The worker IDs "ID2" and "ID3", which are different from each other and are classified as a single worker, are combined to generate a timeline diagram associated with each worker ID. As shown in the example of display screen 500, in a display screen 500 that associates the same worker with multiple timeline diagrams, the user operates an input device (not shown) connected to the display device to integrate the facial image and the timeline diagram group, thereby improving the efficiency of statistical information analysis.

[0068] As a specific example, the user operates the cursor 501 displayed on the display screen 500 to select two different facial images 402a and 402b representing the same worker. Then, the user operates the input device, and with the two facial images 402a and 402b selected, a menu 502 is displayed to combine the two facial images 402a and 402b. Furthermore, when the user operates the input device to select menu 502, the user performs the combination of the two facial images 402a and 402b selected via PC 200 and the timing diagram associated with these facial images. Thus, the two sets of facial images 402a and 402b selected on the display screen 500 and the timing diagrams 407 and 408 are combined (merged) into a single set of facial images and a new timing diagram.

[0069] Furthermore, the facial images displayed on the integrated display screen 500 can be selected from either facial images 402a or 402b via user input or processing by the PC 200, or other facial images representing the same operator stored in the storage unit 230 can be used. Additionally, a new sequence diagram can be generated by the statistical information generation unit 222 based on the average of each step calculated from the original sequence diagrams 407 and 408. Alternatively, instead of user operation, two facial images 402a and 402b can be selected, and the identification numbers ("ID2" and "ID3" in the diagram) associated with the facial images 402a and 402b, and the sequence diagrams 407 and 408 can be selected. Furthermore, three or more sets of facial images can be integrated with the flowchart.

[0070] <Variation Example 2>

[0071] In Modification 2, the above-described processing based on PC200 generates a display for the display device. Figure 6The data shown on display screen 600. As shown in the example of display screen 600, in this variation, a menu 601 that automatically executes multiple sets of facial images and flowcharts as in variation 1 is displayed on display screen 600 via PC 200. In variation 2, the background of associating multiple statistical information with the same operator is the same as in variation 1. The differences from variation 1 will be explained below.

[0072] When the display screen 600 is on the display device, and the user operates the input device by pressing the execute button 602 of the menu 601 via the cursor 501, similarly to Modified Example 1, the PC 200 performs the processing of integrating multiple sets of facial images 402a, 402b and timing diagrams 407, 408. The PC 200 can determine the group of integrated facial images 402a, 402b and flowcharts 407, 408 based on the similarity of the facial images.

[0073] <Variation Example 3>

[0074] In Modification 3, the above-described processing by PC200 generates a display that is shown on the display device. Figure 7 The data displayed on screen 700. As shown in the example of screen 700, in this modified example, a portion of the worker's face is missing from the face image 702. Therefore, the user can operate the input device to select a more appropriate face image as face image 702, making it easier to determine the statistical information of each worker.

[0075] As a specific example, the user operates the cursor 501 displayed on the display screen 700 and selects a changed face image 702. Then, the user operates the input device, and with face image 702 selected, a menu 701 is displayed to select face image 702 again. Furthermore, when the user operates the input device to select menu 701, other face images of the operator corresponding to face image 702, generated by the image generation unit 221 and stored in the storage unit 230, are displayed on the selection screen 703. If the user selects an appropriate face image from the multiple face images displayed on the selection screen 703 and presses the selection button 704, the user changes face image 702 to the selected face image. The changed face image is stored in the storage unit 230 in association with the operator ID and statistical information for display of face images in the next and subsequent times.

[0076] <Variation Example 4>

[0077] In variation 4, the above-described processing based on PC200 generates a display that is shown on the display device. Figure 8The data displayed on display screen 800. As shown in the example of display screen 800, in this modified example, when the user selects any face image (face image 405 in the example of the figure) by operating the input device on display screen 400 in modified example 1, more detailed information on the work statistics of the worker corresponding to the selected face image 405 is displayed on display screen 800. As an example, display screen 800 displays a timing diagram showing the work time of each cycle in the unit line (timing diagrams corresponding to C1 to C4 in the figure), a timing diagram showing the average work time, a timing diagram showing the target work time, etc. In addition, the details of the statistical information displayed on display screen 800 are not limited to the information shown in the figure, and can be set appropriately. Thus, the user can switch between viewing the summary information of the work statistics related to multiple workers (display screen 400) and the detailed information of the work statistics related to each worker (display screen 800), and perform statistical information analysis more effectively.

[0078] <Variation Example 5>

[0079] In variation 5, the above-described processing based on PC200 generates a display that is shown on the display device. Figure 9 The data shown in display screen 900. As shown in the example of display screen 900, in this variant, the user can view and listen to dynamic images of each step of the operator.

[0080] As a specific example, when a user operates the cursor 501 displayed on the display screen 900 and selects the step for which they want to view an audiovisual image in the operator's sequence diagram (in the example shown, step 5 of the operator with face image 402 is selected), the user displays an audiovisual menu 901 for the audiovisual image. Then, when the user operates the input device to select the audiovisual menu 901, the selected step's audiovisual image screen 902 is displayed, and the audiovisual image is reproduced. Thus, the user can obtain information related to the job content that is difficult to obtain using only statistical information, allowing for more precise analysis. Furthermore, the audiovisual image displayed on the audiovisual image screen 902 can be generated by the control unit 220 based on the audiovisual image data stored in the storage unit 230, according to each step of the operator, and stored in the storage unit 230. Moreover, while the above example envisions displaying an audiovisual image for each step, it is also possible to display an audiovisual image for a series of steps, including multiple steps or all steps.

[0081] <Variation Example 6>

[0082] In Modification 6, the above-described processing based on PC200 generates a display that is shown on the display device. Figure 10The data shown in display screen 1000. As shown in the example of display screen 1000, in this variant, the user can simultaneously view and listen to dynamic images of multiple operators (steps).

[0083] As a specific example, the user operates the cursor 501 displayed on the display screen 1000, and in the worker's timing diagram, turns on checkboxes 1001-1005 displayed next to the worker's face images 401-405, selecting the worker for whom they want to view / listen to a moving image (in the example shown, the worker with face image 402 and the worker with face image 404 are selected). Then, when the user operates the input device and selects the moving image viewing / listening menu 1006, the moving image screens of the workers whose checkboxes 1001-1005 are turned on are displayed respectively (in the figure, moving image screen 1012 corresponding to worker 402 and moving image screen 1014 corresponding to worker 404), and the moving image is reproduced. Thus, by comparing the actions of so-called novice workers with those of experienced workers during their work, it is expected that information that is difficult to obtain from statistical information can be obtained.

[0084] <Variation Example 7>

[0085] In variation 7, the above-described processing based on PC200 generates a display that is shown on the display device. Figure 11 The data displayed on screen 1100 is shown. As shown in the example of screen 1100, in this modified example, a notification bar 1101 is displayed on screen 1100. This notification bar 1101 notifies the user that the difference between the operator's statistics and the target value's work time is above a threshold. For example, as shown in the figure, the notification bar 1101 stores the notification date and time in the "Date and Time" column and details of the notification content in the "Notification Content" column. In the example shown in the figure, the notification content is "ID5 Cycle 3 Step 5 Gap Large (+6 minutes)," indicating that the operator with ID5 (the operator of face image 405) has a work time in step 5 that is 6 minutes longer than the target value's work time in the third cycle. Furthermore, the information displayed by the notification bar 1101 can be generated by the statistics generation unit 222. For example, the statistics generation unit 222 compares the statistics of each operator's work with the target value's work time, and generates information displayed on the notification bar 1101 when the difference is above a threshold. Therefore, users can efficiently perform more precise statistical analysis while considering the notification information. Furthermore, the type of information displayed in the notification bar 1101 is not limited to the example of a graph; various types of information can be used as notification targets. Additionally, the criteria for deciding whether to display information in the notification bar 1101 can be appropriately determined based on the content of the displayed information.

[0086] <Variation Example 8>

[0087] In variation 8, the above-described processing based on PC200 generates a display that is shown on the display device. Figure 12 The data shown in display screen 1200. Display screen 1200 shows the average work time of operators over different periods (today, recent days, last week, this week, etc. However, it is not limited to these; it could also be the current month, the previous month, etc.). Therefore, by changing the period to compare the same work of operators, it is easy to grasp changes in work efficiency. Furthermore, as shown in the example of display screen 1200, in this modified example, in the time series graph of the operator's work statistics, the results of comparison with the target value are displayed as follows: steps where the work time is more than a predetermined threshold below the target value are shown as good (icon 1202 (Good) in the figure), and steps where the work time is more than a predetermined threshold above the target value are shown as bad (icon 1201 (No Good) in the figure). Additionally, the evaluation information representing the evaluation results of the statistical information can be information on the results of various evaluations, and the prescribed benchmarks used in the evaluation can be appropriately determined based on the evaluation. Furthermore, in display screen 1200, if the user operates the input device and selects a step that is displayed as good or bad (in the example shown, selecting the step displayed as NG icon 1201), then the motion image screen 1203, which displays the motion image frame by frame for the selected step, is regenerated and displayed. Additionally, the motion image displayed in motion image screen 1203 can be generated by the control unit 220 based on the motion image data stored in the storage unit 230, according to each step taken by the operator, and stored in the storage unit 230.

[0088] The embodiments and variations described above can be appropriately combined for implementation. Furthermore, in variation 3, the worker's facial image is changed based on user input, but it can also be configured to automatically change the image based on the facial image stored in the storage unit 230 by the PC 200, for example, by using an image with a larger facial portion, an image where the facial orientation is inferred to be facing forward, or an image with higher reliability for the facial portion. Alternatively, the storage unit 230 can be configured to pre-store the worker's facial image (e.g., a facial image used for an employee ID), employee ID, and name as worker information, and then change the worker's facial image displayed on the display screen described in the embodiments and variations to the worker information stored in the storage unit 230. In this case, the data generation unit 223 uses the worker's facial image generated by the image generation unit 221 and the worker's facial image stored in the storage unit 230 to perform a similarity matching based on the features of each image. Then, based on the matching results, the data generation unit 223 replaces the face image displayed on the display screen with a face image of high similarity from the face images of the workers stored in the storage unit 230, and establishes a correlation between the replaced face image and the time sequence diagram of the worker to generate data for the display screen. Even with these processes, users can more efficiently identify each worker to analyze statistical information.

[0089] In addition, in Variation 4, the display screen 800 shows details of statistical information about the work related to each operator. However, in addition to this, or instead, it may also display details of the work time for each step, information indicating the evaluation results based on each step, and information on the results of the analysis. The type of evaluation or analysis to be used can also be appropriately determined.

[0090] <Postscript 1>

[0091] An image processing apparatus, comprising:

[0092] The image acquisition unit (210) acquires a dynamic image of the shooting operation area through an image sensor;

[0093] The image generation unit (221) generates an image of the face portion of a worker performing work in the work area from the dynamic image;

[0094] The statistical information generation unit (222) generates statistical information about the operator's work based on the dynamic image;

[0095] The data generation unit (223) generates data that associates the generated image of the worker's face with statistical information about the worker's work; and

[0096] The output unit (240) outputs the generated data.

[0097] <Appendix 2>

[0098] An image processing method, characterized by having:

[0099] The step of obtaining a dynamic image of the shooting operation area by means of an image sensor (S301);

[0100] Step (S305) to generate an image of the face portion of a worker performing work in the work area from the dynamic image;

[0101] The step of generating statistical information about the operator's work based on the dynamic image (S307);

[0102] The step of generating data that associates the generated image of the worker's face with statistical information about the worker's work (S308); and

[0103] Step S309: Output the generated data.

[0104] Explanation of reference numerals in the attached figures

[0105] 100: Image processing device; 101: Image acquisition unit; 102: Detection unit; 103: Output unit

[0106] 200: PC (Picture Processing Device)

[0107] 210: Input unit; 230: Storage unit; 240: Output unit

[0108] 220: Control Department

[0109] 221: Image Generation Unit

[0110] 222: Statistical Information Generation Department

[0111] 223: Data Generation Department.

Claims

1. An image processing apparatus, comprising: The image acquisition unit acquires dynamic images of the shooting area through an image sensor; The image generation unit generates an image of the face portion of a worker performing work in the work area from the acquired dynamic image; The statistical information generation unit generates statistical information about the operator's work based on the obtained dynamic image, for each of the more than one work steps contained in the work area. The data generation unit generates data that associates the generated image of the worker's face with the generated statistical information of the worker's work. as well as The output unit outputs the generated data. The input unit accepts input from the user. Based on the acquired dynamic image, the data generation unit generates a dynamic image associated with the statistical information of the operator's work, according to each of the more than one work steps. The data generation unit generates data that correlates the image of the worker's face, statistical information of each step of the worker's work (one or more steps), and dynamic images of each step of the worker's work. Based on the generated data, the output unit outputs a display screen to the display device containing an image of the operator's face and statistical information for each of the more than one work steps. In response to the selection of the display portion corresponding to the specified work step in the statistical information of each of the more than one work steps displayed on the display screen via the input unit, the output unit outputs the dynamic image of the selected work step to the display device based on the mutually associated data, and reproduces and displays it on the display device.

2. The image processing apparatus according to claim 1, characterized in that, The data generation unit generates data that associates images of the facial portions of multiple workers generated by the image generation unit with statistical information of each of the more than one work steps of the multiple workers' jobs generated by the statistical information generation unit. The data generation unit generates new data by merging the statistical information of at least two images from the images of the faces of the multiple workers and the more than one work step associated with the at least two images, based on the data output by the output unit.

3. The image processing apparatus according to claim 2, characterized in that, The at least two images are selected based on user input.

4. The image processing apparatus according to any one of claims 1 to 3, characterized in that, The data generation unit generates new data based on the data output by the output unit, by changing the image of the worker's face to another image of the worker's face generated by the image generation unit.

5. The image processing apparatus according to any one of claims 1 to 3, characterized in that, The data generation unit generates data that associates an image of the operator's face, selected based on a criterion including at least one of the operator's facial features, size, orientation, and facial similarity, with statistical information of each step of the generated operator's job, which is one or more job steps.

6. The image processing apparatus according to any one of claims 1 to 3, characterized in that, The data generation unit generates data that associates the image of the face of a selected worker with statistical information of each step of the worker's job, based on the matching of the pre-acquired images of the worker's face with the generated image of the worker's face.

7. The image processing apparatus according to any one of claims 1 to 3, characterized in that, The statistical information generation unit generates summary and detailed statistical information of the operator's work based on the acquired dynamic images. The data generation unit generates data that correlates the image of the worker's face, the summary information of the generated statistics of the worker's work, and the detailed information of the generated statistics of the worker's work.

8. The image processing apparatus according to any one of claims 1 to 3, characterized in that, The data generation unit generates a dynamic image based on the acquired dynamic image, which is associated with the statistical information of the operator's work. The data generation unit generates data that establishes a correlation between the image of the worker's face, the generated statistical information of the worker's work, and the generated dynamic image.

9. The image processing apparatus according to any one of claims 1 to 3, characterized in that, The statistical information generation unit generates evaluation information representing the evaluation results, which are based on a defined benchmark of statistical information for each of the more than one work steps of the operator's work generated from the acquired dynamic image. The data generation unit generates data that associates an image of the operator's face, statistical information of each step of the operator's work (one or more steps), and evaluation information.

10. An image processing method, characterized in that, have: The steps of obtaining dynamic images of the shooting area using an image sensor; The step of generating an image of the face portion of a worker performing work in the work area from the acquired dynamic image; Based on the obtained dynamic image, the step of generating statistical information about the operator's work for each of the more than one work steps contained in the work area; The step of generating data that associates the generated image of the worker's face with statistical information about the worker's work. The step of outputting the generated data; The step of accepting input from the user; Based on the obtained dynamic image, the step of generating a dynamic image associated with statistical information of the operator's work for each of the more than one work steps; The step of generating data that correlates an image of the operator's face, statistical information of each step of the operator's work (one or more steps), and dynamic images of each step of the operator's work (one or more steps); The step of outputting a display screen containing an image of the operator's face and statistical information of each of the more than one work steps to a display device based on the generated data; as well as In response to the selection of the display portion corresponding to the specified work step from the statistical information of each of the more than one work steps displayed on the display screen, the dynamic image of the selected work step is output to the display device according to the mutually associated data, and is reproduced and displayed on the display device.

11. A computer-readable storage medium storing a program for causing a computer to perform the steps of the image processing method of claim 10.

Images