Information processing device, program, information processing method, and information processing system

Abstract

To provide an information processing device, a program, an information processing method, and an information processing system that can grasp the behavior of a target person regardless of whether the target person is inside or outside the imaging range of an imaging unit.SOLUTION: An information processing device includes a processing unit that is configured to: acquire a plurality of images captured by an imaging unit at a plurality of different time points; determine whether a target person is imaged in each of the acquired plurality of images; derives a position of the target person corresponding to each image in which the target person is determined to be imaged; and upon determining that the target person is not imaged in an image captured during a certain period, of the plurality of images; estimate a position where the target person was present during the period based on first information pertaining to a position of the target person imaged in a first image captured immediately before the period, second information pertaining to a position of the target person imaged in a second image captured immediately after the period, and third information pertaining to length of the period.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a program, an information processing method, and an information processing system.

Background Art

[0002] Conventionally, a technique for determining the position of a target person in a work site from a captured image of the work site captured by an imaging unit such as a camera has been known (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, there has been a problem that the above technique does not function for a target person outside the imaging range of the imaging unit that images the work site.

[0005] An object of this invention is to enable grasping of the behavior of a target person regardless of whether the target person is within or outside the imaging range of the imaging unit.

Means for Solving the Problems

[0006] To solve the above problems, an information processing apparatus according to the present invention acquires a plurality of images captured by an imaging unit at a plurality of different time points, determines whether or not a target person is shown in each of the acquired plurality of images, derives the position of the target person corresponding to each image in which the target person is determined to be shown, If it is determined that the subject is not visible in any of the multiple images taken during a certain period, the location of the subject during that period is estimated based on the first information relating to the subject's position in the first image taken immediately before that period, the second information relating to the subject's position in the second image taken immediately after that period, and the third information relating to the length of that period. It is equipped with a processing unit.

[0007] To solve the above problems, the information processing apparatus according to the present invention is: Multiple images captured by the imaging unit at different points in time are acquired. Determine whether or not the subject is pictured in each of the acquired multiple images. The position of the subject corresponding to each image in which the subject is determined to be present is derived. If it is determined that the subject is not visible in any of the multiple images taken during a certain period, then it is determined whether the subject was in a predetermined location during that period based on first information relating to the subject's position in the first image taken immediately before the period, second information relating to the subject's position in the second image taken immediately after the period, and third information relating to the length of the period. It is equipped with a processing unit.

[0008] To solve the above problems, the program according to the present invention On the computer, A process for acquiring multiple images captured by the imaging unit at multiple different points in time. A process to determine whether or not the subject is pictured in each of the acquired multiple images. A process for deriving the position of the subject corresponding to each image in which the subject is determined to be present. If it is determined that the subject is not captured in any of the multiple images taken during a certain period, the process of estimating the location of the subject during that period is performed based on first information relating to the position of the subject captured in the first image taken immediately before the period, second information relating to the position of the subject captured in the second image taken immediately after the period, and third information relating to the length of the period. Make it run.

[0009] To solve the above problems, the program according to the present invention On the computer, A process for acquiring multiple images captured by the imaging unit at multiple different points in time. A process to determine whether or not the subject is pictured in each of the acquired multiple images. A process for deriving the position of the subject corresponding to each image in which the subject is determined to be present. If it is determined that the subject is not visible in any of the multiple images taken during a certain period, a process is performed to determine whether the subject was in a predetermined location during that period, based on first information relating to the position of the subject in the first image taken immediately before the period, second information relating to the position of the subject in the second image taken immediately after the period, and third information relating to the length of the period. Make it run.

[0010] To solve the above problems, the information processing method according to the present invention is A method of information processing performed by a computer, Multiple images captured by the imaging unit at different points in time are acquired. Determine whether or not the subject is pictured in each of the acquired multiple images. The position of the subject corresponding to each image in which the subject is determined to be present is derived. If it is determined that the subject is not captured in any of the multiple images taken during a certain period, the location of the subject during that period is estimated based on first information relating to the subject's position in the first image taken immediately before the period, second information relating to the subject's position in the second image taken immediately after the period, and third information relating to the length of the period.

[0011] To solve the above problems, the information processing method according to the present invention is A method of information processing performed by a computer, Multiple images captured by the imaging unit at different points in time are acquired. Determine whether or not the subject is pictured in each of the acquired multiple images. The position of the subject corresponding to each image in which the subject is determined to be present is derived. If it is determined that the subject is not captured in any of the multiple images taken during a certain period, then it is determined whether the subject was in a predetermined location during that period based on first information relating to the subject's position in the first image taken immediately before the period, second information relating to the subject's position in the second image taken immediately after the period, and third information relating to the length of the period.

[0012] To solve the above problems, the information processing system according to the present invention is: Multiple images captured by the imaging unit at different points in time are acquired. Determine whether or not the subject is pictured in each of the acquired multiple images. The position of the subject corresponding to each image in which the subject is determined to be present is derived. If it is determined that the subject is not visible in any of the multiple images taken during a certain period, the location of the subject during that period is estimated based on the first information relating to the subject's position in the first image taken immediately before that period, the second information relating to the subject's position in the second image taken immediately after that period, and the third information relating to the length of that period. It includes a processing unit.

[0013] To solve the above problems, the information processing system according to the present invention acquires a plurality of images captured by an imaging unit at a plurality of different time points, determines whether a target person is shown in each of the acquired plurality of images, derives the position of the target person corresponding to each image in which the target person is determined to be shown, Among the plurality of images, if it is determined that the target person is not shown in the images captured during a certain period, based on the first information related to the position of the target person shown in the first image captured immediately before the period, the second information related to the position of the target person shown in the second image captured immediately after the period, and the third information related to the length of the period, it is determined whether the target person was at a predetermined location during the period. It includes a processing unit.

Advantages of the Invention

[0014] According to the present invention, the behavior of the target person can be grasped regardless of whether the target person is within the imaging range of the imaging unit or outside the imaging range.

Brief Description of the Drawings

[0015] [Figure 1] It is a block diagram showing the functional configuration of the behavior analysis system. [Figure 2] It is a diagram showing the floor of a factory. [Figure 3] It is a diagram showing an example of the content of a work area master. [Figure 4] It is a diagram showing an example of the content of an operator master. [Figure 5] It is a diagram showing an example of the content of a work classification master. [Figure 6] It is a diagram showing the identification mark worn by an operator. [Figure 7] It is a diagram showing an example of variations of the identification mark. [Figure 8]This diagram illustrates the method for calculating the worker's position using the coordinate calculation unit. [Figure 9] This diagram illustrates a method for determining the appropriate work based on the results of skeletal detection. [Figure 10] This figure shows an example of a case where the worker is not detected. [Figure 11] This is a flowchart showing the control procedure for behavioral analysis processing. [Figure 12] This is a flowchart showing the control procedure for the work determination process. [Figure 13] This figure shows an example of an behavioral data table. [Figure 14] This figure shows an example of a summary table. [Figure 15] This figure shows an example of an individual summary table. [Figure 16] This is an example of a heatmap. [Figure 17] This figure shows an example of a graph of time spent and distance traveled. [Figure 18] This figure shows an example of a work ratio table. [Figure 19] This diagram shows an example of a representative accommodation location table. [Figure 20] This figure shows an example of a time chart. [Figure 21] This is an example of a graph showing the number of deliveries and the time taken for each delivery. [Figure 22] This figure shows an example of an analytical pie chart. [Modes for carrying out the invention]

[0016] Hereinafter, embodiments of the present invention will be described based on the drawings.

[0017] <Overview of the work analysis system> Figure 1 is a block diagram showing the functional configuration of the behavioral analysis system 1. The behavior analysis system 1 (information processing system) comprises a behavior analysis device 10 (information processing device) and a camera 20 (imaging unit, imaging device). The behavior analysis device 10 is wirelessly or wiredly connected to the camera 20, enabling the transmission and reception of data such as control signals and image data between the behavior analysis device 10 and the camera 20. The behavior analysis system 1 may include two or more cameras 20. In this embodiment, the camera 20 is installed inside the factory and captures video of workers 40 (subjects) working on the production line in the factory.

[0018] Figure 2 shows the factory floor 100. Floor 100 comprises a workroom 2 where a production line is set up, a warehouse 5 where materials used in the manufacture of products are stored, and a passageway 6 connecting the workroom 2 and the warehouse 5. In the following, positions within Floor 100 are represented using the XYZ Cartesian coordinate system shown in Figure 2. The X and Y axes are parallel to the sides of the rectangle formed by the outline of the workroom 2. The Z axis is parallel to the vertical direction.

[0019] The interior of workroom 2 is divided into process area 3 and sub-area 4. Specifically, sub-area 4 is a rectangular area occupying the +X end of workroom 2, and process area 3 is a rectangular area adjacent to sub-area 4 on the -X side. Process area 3 is where workers 40 perform the work of manufacturing products, and workbenches 50 for manufacturing work are installed there. Sub-area 4 is provided with a temporary storage area for materials and equipment used in the manufacturing of products. Passageway 6 extends from the +Y end of sub-area 4 in the +Y direction, then bends in the -X direction and extends to warehouse 5. Warehouse 5 is a rectangular area located separately on the +Y side of process area 3.

[0020] Two cameras, 20a and 20b, are installed in workroom 2. Additionally, one camera, 20c, is installed in warehouse 5. Cameras 20a to 20c are synchronized with each other; that is, cameras 20a to 20c are adjusted so that the same image capture time is recorded for images captured at the same time. Cameras 20a to 20c each capture a trapezoidal imaging range Ra to Rc that radiates from the direction of the camera lens. In Figure 2, the imaging ranges Ra to Rc are represented by dashed lines. The imaging ranges Ra and Rb of cameras 20a and 20b cover most of workroom 2, but there are parts of workroom 2 that are not included in the imaging ranges of either camera 20a or 20b. Furthermore, the imaging range Rc of camera 20c covers almost the entire area of ​​warehouse 5. Since no camera 20 is installed in passageway 6, workers 40 passing through passageway 6 are not captured by any of the cameras 20.

[0021] On floor 100, multiple workers 40 perform their respective tasks. Each worker 40 is assigned specific tasks. For example, one worker 40 performs tasks related to product manufacturing at a workbench 50 in process area 3. Another worker 40 travels back and forth between sub-area 4 and warehouse 5, transporting materials necessary for product manufacturing from warehouse 5 to sub-area 4. Yet another worker operates equipment (not shown) located throughout floor 100.

[0022] Workers 40, who handle tasks such as transporting materials and operating equipment, play a crucial role in maintaining and stabilizing the production line. However, due to the nature of their work, which involves moving around within floor 100, they are not expected to work within standard timeframes like workers 40 who directly perform manufacturing tasks in process area 3. This makes it extremely difficult to determine whether their work is being carried out efficiently. As a result, various problems can arise, sometimes leading to inefficient work.

[0023] In the behavioral analysis system 1 according to this embodiment, the system analyzes the work of worker 40 from images captured by camera 20, while also estimating the location and work of worker 40 outside the imaging range of camera 20. This makes it possible to understand and analyze the behavior of worker 40 regardless of whether they are within or outside the imaging range of camera 20. Furthermore, by collecting and analyzing behavioral data in real time using camera 20 and behavioral analysis device 10, it is possible to perform continuous or immediate behavioral analysis without relying on manual intervention.

[0024] <Configuration of the work analysis system> The configuration of the behavioral analysis device 10 and camera 20 will be described below with reference to Figure 1. The behavioral analysis device 10 includes a processing unit 11 (one or more processing units, a computer, a control unit), a storage unit 12, a display unit 13 (output unit), a communication unit 14, and the like. The various parts of the behavioral analysis device 10 are connected via a bus 15.

[0025] The processing unit 11 includes a CPU (Central Processing Unit) and RAM (Random Access Memory), etc. The processing unit 11 reads and executes the program 121 stored in the memory unit 12 and controls the operation of the behavior analysis device 10 by performing various calculations. The behavior analysis device 10 may have multiple processing units (for example, multiple CPUs), and multiple processing units may share the execution of multiple processes that would otherwise be performed by one processing unit 11 in this embodiment. In this case, multiple processing units may be involved in common processes, or multiple processing units may independently execute different processes in parallel. The processing unit 11 may further include a GPU (Graphics Processing Unit) and a timer, etc.

[0026] The processing unit 11 functions as a mark detection unit 111, a skeleton detection unit 112, an action determination unit 113, a time calculation unit 114, a coordinate calculation unit 115, an undetected time calculation unit 116, and an undetected action estimation unit 117, by executing various processes according to the program 121. The functions of each of these units will be described later.

[0027] The storage unit 12 is a non-temporary recording medium readable by the processing unit 11, which functions as a computer, and stores the program 121 and various data. The storage unit 12 includes non-volatile memory such as an HDD (Hard Disk Drive) or SSD (Solid State Drive). The program 121 is stored in the storage unit 12 in the form of program code that can be read by the computer.

[0028] The memory unit 12 stores a mark detection model 122, a skeleton detection model 123, a task determination model 124, and an action estimation model 125. Details of these machine learning models will be described later.

[0029] Furthermore, the storage unit 12 is provided with an action data area 126. The action data area 126 records data related to the judgment and estimation results of the worker's 40 work and actions, as well as various analysis data generated based on said judgment and estimation results. The action data area 126 also includes databases such as a work area master 1261, a worker master 1262, and a work classification master 1263, which are used for judging, estimating, and analyzing the worker's 40 actions.

[0030] Figure 3 shows an example of the contents of the work area master 1261. Floor 100 is divided into multiple locations (areas) according to their intended use. The work area master 1261 registers the scope and name of each location. The "Location ID" in the work area master 1261 is a unique code associated with each location. The "start coordinates" and "end coordinates" represent the XYZ coordinates of the diagonal of a rectangular area parallel to the XY plane representing each location. Note that each location does not necessarily have to be a rectangle; for example, it may be convex or concave. "Workplace name" refers to the name of each location.

[0031] Figure 4 shows an example of the contents of the worker master 1262. The worker master 1262 contains the assigned work location for each worker performing tasks on floor 100. The "Mark ID" in the worker master 1262 is a unique code associated with each worker 40. As will be described later, each worker 40 wears a different identification mark 60 (see Figures 6 and 7), and the "Mark ID" is identified from the pattern of the identification mark 60. The "Main Work Location ID" is the code for the work location that each worker 40 is assigned (instructed) to be their primary work location. The "Sub-Work Location ID" is the code for the work location (sub-work location) to which each worker 40 is assigned (instructed) to perform incidental tasks. Note that each worker may be assigned two or more sub-work locations. In this case, a column representing the sub-work locations is added to the worker master 1262. The codes for "Main Work Location ID" and "Sub Work Location ID" correspond to the "Location ID" in the work area master 1261.

[0032] Figure 5 shows an example of the contents of the work classification master 1263. The work classification master 1263 contains classifications and registrations of various tasks performed by multiple workers 40 on floor 100. The "Task ID" in the Task Classification Master 1263 is a unique code associated with each task. "Work Classification" refers to the name of each classified task. Examples include "Picking," which involves retrieving materials in warehouse 5, etc.; "Material Organizing," which involves organizing materials in warehouse 5, sub-area 4, etc.; "Outbound Processing," which involves issuing materials out of warehouse 5, etc.; and "Consultation / Communication," which indicates that work has been interrupted due to consultations or communications from managers or other workers. "Category" represents the category to which each task belongs. In this embodiment, "Category" is either "Net Task," which indicates a primary task, or "Ancillary Task," which indicates an incidental task. Here, "Picking" is classified as "Net Task," while "Material Arrangement," "Outbound Processing," and "Consultation / Communication" are classified as "Ancillary Tasks."

[0033] Returning to Figure 1, the storage unit 12 is provided with an image data area 127. The image data area 127 records image data related to the video captured by the camera 20 and transmitted from the camera 20. The image data related to the video includes image data of multiple captured images 30 (frame images) that constitute the video.

[0034] The display unit 13 is equipped with a display device such as a liquid crystal display, and displays various information on the display device according to the display control signal from the processing unit 11.

[0035] The communication unit 14 is composed of a network card or communication module, and transmits and receives data with the camera 20 and external devices (not shown) according to a predetermined communication standard.

[0036] Camera 20 captures video at a predetermined frame rate (e.g., 30fps) and transmits the image data to the behavioral analysis device 10. Camera 20 may capture color video or monochrome video. Furthermore, the imaging resolution of camera 20 is not particularly limited as long as it is within a range that allows for the proper detection of the analysis points 41, which will be described later.

[0037] <Operation of the work analysis system> (Overview of operation) Next, we will explain how the behavioral analysis system 1 works. The following is an overview of the operation of the behavior analysis device 10 of the behavior analysis system 1 regarding the analysis of the worker's behavior 40. The behavioral analysis device 10 first acquires image data of each captured image 30 (frame image) of the video captured by the camera 20. Then, it detects the worker 40 in the acquired image data of the captured images 30 and identifies the worker 40. Hereafter, "image data of the captured images 30" will also be simply referred to as "captured images 30". Next, the XYZ coordinates of worker 40 on the floor 100 are calculated from the coordinates of worker 40 in the captured image 30 (xY coordinates in Figure 8). Based on the calculated position of worker 40, the task being performed by worker 40 is determined. The task determination may be performed in more detail based on the skeletal data of worker 40, as will be described later. If the detected worker 40 was not detected in the captured images 30 from a certain period prior to the detection period (hereinafter referred to as the "undetected period"), the location where the worker 40 was during the undetected period is estimated based on the captured images 30 taken immediately before and after the undetected period. Furthermore, based on the estimated location, the work that the worker 40 was performing during the undetected period is estimated. Furthermore, the calculation results of the worker's position 40, the location estimation results, and the determination and estimation results of the work that worker 40 was performing are recorded and stored as action data in the action data area 126, in association with the time when the captured image 30 was taken. When work data is accumulated in the behavior data area 126, various analyses related to the worker's actions 40 are performed based on the behavior data, and information related to the analysis results is displayed on the display unit 13. The details of these operations are described below.

[0038] (Detection and identification of 40 workers) First, we will explain the operation related to the detection and identification of worker 40. The detection and identification of worker 40 is performed by detecting the identification marks 60a and 60b (identification markers) worn by worker 40 in the captured image 30 of worker 40 using the mark detection model 122.

[0039] Figure 6 shows the identification marks 60a and 60b worn by worker 40. Identification marks 60a and 60b are strip-shaped members with a predetermined pattern drawn on their surface. Identification mark 60a is a headband-like strip-shaped member that is wrapped around the head of worker 40 or the hat worn by worker 40. Identification mark 60b is an armband that is wrapped around the upper arm of worker 40. Worker 40 wears at least one of identification marks 60a or 60b while working. Hereinafter, "identification mark 60" will be used to refer to either identification mark 60a or 60b. The size, width, and pattern of identification mark 60 are determined so that it can be captured by camera 20 regardless of the position, orientation, and posture of worker 40 as they move around the work area. When identification mark 60a is attached to a hat, in addition to the strip-shaped identification mark 60a that goes around the side of the hat, an identification mark with the same pattern may also be attached to the top of the hat.

[0040] Figure 7 shows examples of variations of the identification mark 60. As shown in Figure 7, in this embodiment, multiple types of identification marks 60 with different patterns are provided. The multiple patterns of the identification marks 60 in this embodiment include multiple circles, and at least some of the circle colors, circle sizes, and circle background colors differ from one another. Each worker 40 wears an identification mark 60 with a different pattern. Each of the multiple types of identification marks 60 is associated with a unique mark ID ("A", "B", "C", ...). Note that the patterns of the identification marks 60 are not limited to those shown in Figure 7, and can be any patterns that are distinguishable from one another.

[0041] The mark detection unit 111 of the processing unit 11 detects the identification marks 60 that are captured in the acquired image 30. More specifically, the mark detection unit 111 detects the identification marks 60 using the mark detection model 122. The mark detection model 122 is a machine learning model that, upon receiving image data of the captured image 30 from the mark detection unit 111, recognizes the pattern of the identification marks 60 contained in the captured image 30 and outputs the position of the identification marks 60 in the captured image 30 (x and y coordinates in the captured image 30) and the mark ID of the identification marks 60.

[0042] Alternatively, instead of using the identification mark 60, worker 40 may be identified based on physical characteristics such as their face and skeletal structure. Furthermore, if worker 40 works in a nearly fixed position, the worker 40 can be identified even without wearing the identification mark 60, so the identification mark 60 may be omitted.

[0043] Furthermore, the time calculation unit 114 of the processing unit 11 calculates the time the captured image 30 was captured when the mark detection unit 111 detected the worker 40. If the captured image 30 is being acquired from a video in real time, the time calculation unit 114 obtains the time from the timer of the processing unit 11. If the captured image 30 is from a video that was captured in the past and stored in the storage unit 12, the time calculation unit 114 calculates the elapsed time from the start time of the capture.

[0044] (Calculation of worker 40's position) Next, we will explain the operation of calculating (deriving) the position of worker 40. The position of worker 40 is calculated by the coordinate calculation unit 115 of the processing unit 11. Figure 8 is a diagram illustrating the method for calculating the position of the worker 40 by the coordinate calculation unit 115. The upper part of Figure 8 shows a captured image 30 (frame image) of a video captured by camera 20. The position of pixels within the captured image 30 is represented by xy coordinates, with the x and y axes shown in Figure 8 as the coordinate axes. The lower part of Figure 8 corresponds to a view of the floor 100 (in this case, workroom 2) from the +Z direction at the time of capturing the image 30 in the upper part of Figure 8. The image 30 in the upper part of Figure 8 shows the trapezoidal imaging area Ra in the lower part.

[0045] The coordinate calculation unit 115 first identifies the position coordinates p(x,y) of the worker 40 in the captured image 30. Next, the coordinate calculation unit 115 converts the position coordinates p(x,y) of the worker 40 in the captured image 30 into the XYZ coordinate system of the floor 100 (homography transformation) and calculates the position coordinates P(X,Y,X) of the worker 40 shown in the lower part of Figure 8. The Z coordinate may represent the floor number of the floor 100 (1st floor, 2nd floor, ...).

[0046] (Judgment of the work performed by worker 40) Next, we will explain the operation of determining the work of the worker 40 captured in the image 30. The work determination is performed by the work determination unit 113 of the processing unit 11. The work determination unit 113 determines, for example, that the work performed by worker 40 is the work corresponding to the work location (a work predetermined to be performed by worker 40 in that work location) based on the position coordinates P of worker 40 calculated by the coordinate calculation unit 115 and the work location to which said position coordinates P belong. For example, if worker 40 is in warehouse 5, it may be determined that worker 40 is performing the "picking" work of materials. The work determination unit 113 may also determine whether worker 40 is holding materials, etc., based on the captured image 30, and reflect the result of this determination in the work determination. For example, if worker 40 is in warehouse 5, it may be determined that worker 40 is performing the "picking" work if worker 40 is holding materials, and that worker 40 is performing the "organizing materials" work if worker 40 is not holding materials.

[0047] Furthermore, the work determination unit 113 may determine the work based on the movement of the worker's skeleton 40 detected from the captured image 30. The determination method in this case will be described in detail below.

[0048] Figure 9 illustrates a method for determining the appropriate work based on the results of skeletal detection. The skeleton of worker 40 is detected by the skeleton detection unit 112 of the processing unit 11. The skeleton detection unit 112 detects the skeleton of worker 40 by inputting the portion of the image data of the captured image 30 corresponding to the detected worker 40 into the skeleton detection model 123. The skeleton detection model 123 is a machine learning model that, upon receiving the portion of the image data of the captured image 30 corresponding to worker 40 from the skeleton detection unit 112, detects multiple analysis points 41 corresponding to multiple parts of worker 40 and outputs the position (coordinates) of the analysis points 41. From these analysis points 41, an axis 42 connecting the analysis points 41 may be identified, and the information of this axis 42 may be further taken into account in the work determination described later. If multiple workers 40 are detected in the captured image 30, the skeleton detection unit 112 detects the skeleton for each worker 40.

[0049] Figure 9 shows examples of analysis points 41 and axis lines 42. In this embodiment, analysis points 41 are detected for each of 18 body parts (multiple body parts) of the worker 40. The 18 body parts are, for example, both eyes, both ears, nose, throat, both shoulders, both elbows, both hands, the base of both legs, both knees, and both ankles. However, the location and number of analysis points 41 are not limited to these. Of the above 18 body parts, the detection of analysis points 41 is omitted for body parts that are not captured in the image 30.

[0050] In this specification, detecting multiple analysis points 41 of the worker 40 is referred to as "skeleton detection," and the data containing the coordinates of the detected analysis points 41 is referred to as "skeleton data d" (skeleton data d1 to d7 in Figure 9). Skeleton data d may also include information on the position, length, and orientation of the axis 42.

[0051] The work determination unit 113 makes a work determination based on the skeletal data d. The work determination unit 113 inputs the skeletal data d for each captured image 30 in the captured image group 30G, which consists of the most recent predetermined number of captured images 30, into the work determination model 124. In this embodiment, the predetermined number of frames is "6". The work determination unit 113 obtains skeletal data d1 to d6 from the captured images 30 of frames 1 to 6 that make up the captured image group 30G, for example, as shown in the upper part of Figure 9, and combines them to generate combined skeletal data D. The work determination unit 113 inputs this combined skeletal data D into the work determination model 124. The "predetermined number of frames" may be 5 frames or less, or 7 frames or more.

[0052] The work determination model 124 is a machine learning model that, by inputting a plurality of analysis points 41 extracted from each image 30 in a given image group 30G, outputs a determination result of the work that the worker 40 was performing at the time of acquisition of any of the image 30 included in the image group. Therefore, when the skeletal data combination D (information of the plurality of analysis points 41 detected from the image group 30G) is input to the work determination model 124 from the work determination unit 113, the work determination model 124 determines, based on the time-series changes of the plurality of analysis points 41 in the image group 30G, which of the plurality of tasks included in the process for manufacturing a product was performed by the worker 40 at the time of acquisition of any of the image 30 included in the image group 30G. In this embodiment, the work determination model 124 determines, among the plurality of image 30 included in the image group 30G, which was performed by the worker 40 at the time of acquisition of the last acquired (most recent) image 30. The work determination model 124 may also determine what work the worker 40 was performing at the time of capturing any of the captured images 30 included in the group of captured images 30G, other than the last captured image 30 (for example, an intermediate captured image 30 or the first captured image 30).

[0053] The core judgment logic of the work judgment model 124 uses LSTM (Long Short-Term Memory). LSTM is one of the algorithms of deep neural networks and has a structure that allows the use of short-term memory within the network for a long period of time. Therefore, by using LSTM, it is possible to judge work by taking into account past time-series data. The LSTM in this embodiment is machine-trained to output the judgment result of the worker 40's work (for example, work ID) using skeleton data combination D, which consists of skeleton data d for 6 frames, as training data.

[0054] Furthermore, if a skeleton is not detected in some of the six captured images 30, the work determination model 124 can determine the work from the skeleton data d of the remaining frames, even if the skeleton data d for that captured image 30 is treated as undetected data (blank data, dummy data) that does not contain information related to the skeleton. In the example shown in the upper part of Figure 9, skeleton data d3 and d5 are undetected data, but the work can be determined from the remaining skeleton data d1, d2, d4, and d6.

[0055] As shown in the upper part of Figure 9, based on the image group 30G consisting of captured images 30 from frames 1 to 6, the work of worker 40 at the time of capturing the 6th frame 30 is determined (this is the Nth determination). Then, in the next frame, as shown in the lower part of Figure 9, based on the image group 30G consisting of captured images 30 from frames 2 to 7, the work of worker 40 at the time of capturing the 7th frame 30 is determined (this is the N+1 determination). That is, the oldest captured image 30 (frame 1) from the image group 30G used for determining the Nth determination is excluded, and the latest captured image 30 (frame 7) is added to the image group 30G. Then, skeleton data d2 to d7 are obtained from the captured images 30 from frames 2 to 7, respectively, and skeleton data combined D is generated and input to the work determination model 124. Thus, the work determination unit 113 and the work determination model 124 determine each work based on the combined skeleton data D obtained by shifting the constituent skeleton data d by a predetermined number of frames (one frame at a time in this embodiment). In other words, the work determination unit 113 and the work determination model 124 determine each work based on the group of captured images 30G obtained by shifting the constituent captured images 30 by a predetermined number of frames. By performing work determination for each frame, robustness of the determination can be expected.

[0056] Furthermore, during its training, the task determination model 124 is machine-learned using multiple combined skeleton data D, which are obtained by shifting the constituent skeleton data d by a predetermined number of frames (one frame at a time in this embodiment), as training data, similar to the task determination process. In other words, the task determination model 124 is machine-learned using combined skeleton data D generated from multiple image image groups 30G, which are obtained by shifting the constituent captured images 30 by a predetermined number of frames, as training data. In this way, the training data group itself is given time-series characteristics between the data, and by training this data group, the classification accuracy with respect to time-series changes is improved. In other words, by using an LSTM that makes decisions based on time-series changes, and by giving the LSTM training data time-series features, the classification accuracy is improved by addressing two sets of time-series changes.

[0057] Furthermore, the "specified number of frames" mentioned above is not limited to one frame, but may be two or more frames. Alternatively, instead of LSTM, other decision logics that can perform decisions (estimations) that take time-series data into account may be used, such as RNN (Regressive Neural Network) or Transformer.

[0058] (Estimated behavior of 40 undetected workers) Next, we will describe the operation for estimating the actions of worker 40 who was undetected during the undetected period and was subsequently detected in the captured image 30 during that period. The estimation of the actions of undetected worker 40 is performed by the undetected time calculation unit 116 and the undetected action estimation unit 117 of the processing unit 11. The action estimation may include determining whether or not the undetected worker 40 was in a predetermined location during the undetected period. The action estimation may also include estimating the location where the undetected worker 40 was during the undetected period. Furthermore, the action estimation may include estimating the work that worker 40 was performing during the undetected period. Here, the "location" to be determined or estimated may be, for example, any of the work locations registered in the work area master 1261. The estimation of the location where worker 40 was during the undetected period may include estimating the coordinates where worker 40 was during the undetected period.

[0059] Figure 10 shows an example where worker 40 is not detected. Figure 10 illustrates the case where worker 40 moves from position P1 in sub-area 4, through positions P2, P3, and P4 in passage 6, to position P5 in warehouse 5. In the following, the times when worker 40 was at positions P1 to P5 will be denoted as times T1 to T5, respectively. Time T1 is assumed to be just before worker 40 enters passage 6. Time T2 is assumed to be just after worker 40 enters passage 6. Time T4 is assumed to be just before worker 40 enters warehouse 5. Time T5 is assumed to be just after worker 40 enters warehouse 5.

[0060] As shown in Figure 10, since position P1 within sub-area 4 is included in the imaging range Rb of camera 20b, worker 40 at position P1 at time T1 is imaged by camera 20b. Hereafter, the image 30 captured by camera 20b at time T1 will be referred to as the first image 301. In Figure 10, for convenience, the first image 301 is represented by the imaging range Rb of camera 20b. Also, since position P5 within warehouse 5 is included in the imaging range Rc of camera 20c, worker 40 at position P5 at time T5 is imaged by camera 20c. Hereafter, the image 30 captured by camera 20c at time T5 will be referred to as the second image 302. In Figure 10, for convenience, the second image 302 is represented by the imaging range Rc of camera 20c. On the other hand, since positions P2 to P4 within the passage 6 are not within the imaging range of any of the cameras 20, the worker 40 is not imaged by any of the cameras 20 during the time T2 to T4 when passing through the passage 6.

[0061] In this example, it is assumed that a worker 40 at position P5 is detected from the second image 302 captured by camera 20c at time T5. This worker 40 is not detected in any of the images 30 captured by camera 20 during a certain period prior to this (the period from time T2 to T4 when the worker was moving along the passage 6). Therefore, the undetected time calculation unit 116 first calculates the undetected period during which the worker 40 was not detected. In the example shown in Figure 10, the worker 40 is captured in the first image 301 captured by camera 20b at time T1 and in the second image 302 captured by camera 20c at time T5, and is not captured by any of the cameras 20 during the period from time T2 to T4. Therefore, the undetected time calculation unit 116 determines that the undetected period is from time T2 to time T4. The undetected time calculation unit 116 also calculates the length ΔT of the undetected period (ΔT is the difference between time T4 and time T2). As described above, since cameras 20a to 20c are synchronized with each other, the undetected period can be identified and the length of the undetected period can be calculated based on the image acquisition time of each image by each camera. The first captured image 301 corresponds to the first image taken immediately before the undetected period (a certain period), and the second captured image 302 corresponds to the second image taken immediately after the undetected period (a certain period).

[0062] Next, the undetected activity estimation unit 117 estimates the location of worker 40 during the undetected period based on the position P1 (first information) of worker 40 at time T1 as captured in the first captured image 301, the position P5 (second information) of worker 40 at time T5 as captured in the second captured image 302, and the length ΔT (third information) of the undetected period. In the example shown in Figure 10, the area between position P1 at time T1 and position P5 at time T5, which is not captured by either camera 20, is the passageway 6. Therefore, the undetected activity estimation unit 117 estimates that worker 40 was moving along passageway 6 during the undetected period. Alternatively, the position of worker 40 at each time during the undetected period may be estimated from the length of passageway 6 and the length of the undetected period. Furthermore, if there are multiple paths of different lengths between position P1 and position P5, the movement path taken by worker 40 may be estimated from among the multiple paths based on the length of the undetected period. In this case, the longer the period of undetected activity, the longer the distance traveled through which the organism was estimated to have traveled.

[0063] Furthermore, the undetected behavior estimation unit 117 may estimate the location where worker 40 was during the undetected period by inputting the information of worker 40's position P1 at time T1, worker 40's position P5 at time T5, and the length of the undetected period ΔT into the behavior estimation model 125. The behavior estimation model 125 is a machine learning model that has been trained to output an estimation result of the location where worker 40 was during a certain undetected period by inputting information relating to the position of a certain worker 40 immediately before a certain undetected period, information relating to the position of the worker 40 immediately after the undetected period, and information relating to the length of the undetected period. Therefore, in the example shown in Figure 10, when the information of worker 40's position P1 at time T1, worker 40's position P5 at time T5, and the length of the undetected period ΔT is input into the behavior estimation model 125, the behavior estimation model 125 outputs "Passage 6" as the estimation result of the location where worker 40 was during the undetected period, based on the input information.

[0064] Furthermore, the undetected activity estimation unit 117 estimates the work that worker 40 was performing during the undetected period based on worker 40's position P1 at time T1, worker 40's position P5 at time T5, and the length ΔT of the undetected period. For example, the undetected activity estimation unit 117 may estimate the work that worker 40 was performing as work that is pre-associated with the location where worker 40 was during the undetected period. In the example shown in Figure 10, worker 40 moves between sub-area 4 and warehouse 5 in order to transport materials stored in warehouse 5 to sub-area 4. For this reason, when worker 40 moves from sub-area 4 to warehouse 5 via passage 6, passage 6 is associated with the work of "movement". Also, when worker 40 moves from warehouse 5 to sub-area 4 via passage 6, passage 6 is associated with the work of "transporting" materials. Therefore, if worker 40, who was at position P1 at time T1, moves to position T5 at time T5, the undetected action estimation unit 117 estimates that the work worker 40 was performing was "movement". Alternatively, the undetected activity estimation unit 117 may probabilistically estimate (classify) the work that worker 40 was performing during the undetected period, based on worker 40's position P1 at time T1 and worker 40's position P5 at time T5. Specifically, the undetected activity estimation unit 117 may calculate the probability that worker 40 was performing each of several possible tasks during the undetected period, and estimate the task with the highest probability as the task that worker 40 was performing during the undetected period. Parameters used to calculate the probability may include the contents of the worker master 1262, the attributes of worker 40, and / or tasks associated with passages or rooms between position P1 and position P5.

[0065] Furthermore, the undetected activity estimation unit 117 may estimate the work performed by worker 40 during the undetected period by inputting the information of worker 40's position P1 at time T1, worker 40's position P5 at time T5, and the length ΔT of the undetected period into the activity estimation model 125. In this case, the activity estimation model 125 only needs to be machine-trained to output an estimation result of the work performed by worker 40 during the undetected period by inputting information relating to worker 40's position immediately before a certain undetected period, information relating to worker 40's position immediately after the undetected period, and information relating to the length of the undetected period. For example, if there are multiple rooms for performing different tasks and / or multiple paths of different lengths between the position immediately before and immediately after the undetected period, the activity estimation model 125 can estimate which room was performing which task or which path was moving through during the undetected period based on the input information. In the example shown in Figure 10, when the position P1 of worker 40 at time T1, the position P5 of worker 40 at time T5, and the length ΔT of the undetected period are input to the behavior estimation model 125, the behavior estimation model 125 outputs "movement" as the estimated result of the work that worker 40 was performing during the undetected period, based on the input information. Furthermore, the accuracy of work estimation can be improved by further inputting worker 40's behavioral data (e.g., position coordinates at each time, and / or the work being performed at each time) during a predetermined period leading up to the period of non-detection (the predetermined period immediately preceding the non-detection period), and / or worker 40's behavioral data during a predetermined period after detection (the predetermined period immediately following the non-detection period) into the behavioral estimation model 125. For example, if a portion of the warehouse 5 cannot be imaged by the camera, and worker 40 prepared a cart for transporting parts as an action immediately preceding the non-detection period, and worker 40 headed towards the process area 3 with the cart as an action immediately following the non-detection period, then it can be accurately estimated that the action (work) outside the camera's field of view during the non-detection period was part picking. In addition, by using such input information, the behavioral estimation model 125 can also estimate the order of part picking.

[0066] In the above, the undetected behavior estimation unit 117 estimated the location where worker 40 was during the undetected period. However, instead, it may be determined whether or not worker 40 was in a predetermined location (in this case, the passage 6) based on worker 40's position P1 at time T1, worker 40's position P5 at time T5, and the length ΔT of the undetected period.

[0067] Furthermore, while the above example illustrates the undetected period when worker 40 enters the passageway 6, the worker's actions (location and work) can be estimated using the above method during any undetected period resulting from worker 40 being outside the imaging range of any of the cameras 20. For example, the actions of worker 40 outside the imaging ranges Ra and Rb of cameras 20a and 20b in the workroom 2 shown in Figure 2 can be estimated using the same method as described above.

[0068] (Behavioral analysis processing) Next, we will explain the behavioral analysis process that the processing unit 11 performs in order to realize the actions related to the estimation of the worker 40 described above. Figure 11 is a flowchart showing the control procedure for the behavioral analysis process. The behavioral analysis process begins when a user managing the behavioral analysis system 1 (hereinafter referred to as the "management user") issues an instruction to start the behavioral analysis, and when the camera 20 begins capturing video.

[0069] When the behavioral analysis process is started, the processing unit 11 acquires image data of the latest captured image 30 (frame image) that constitutes the video captured by the camera 20 (step S101).

[0070] The mark detection unit 111 of the processing unit 11 inputs the acquired image 30 to the mark detection model 122 and detects the identification marks 60 contained in the image 30 (step S102).

[0071] The processing unit 11 determines in step S102 whether or not one or more identification marks 60 have been detected (step S103). If it is determined that one or more identification marks 60 have been detected ("YES" in step S103), the time calculation unit 114 of the processing unit 11 calculates (or obtains) the time at which the acquired image 30 was captured (step S104).

[0072] The coordinate calculation unit 115 of the processing unit 11 calculates the position of worker 40 on the floor 100 (step S105). Specifically, the coordinate calculation unit 115 converts the x and y coordinates of worker 40's position in the captured image 30 into the XYZ coordinates of worker 40's position on the floor 100. After that, the undetected action estimation unit 117 of the processing unit 11 performs work determination processing (step S106).

[0073] Figure 12 is a flowchart showing the control procedure for the work determination process. When the work determination process is called, the processing unit 11 extracts an image of a specified range (the portion of the image including the worker 40) corresponding to the identification mark 60 detected in step S102 from the captured image 30 (step S201). The size of the specified range is predetermined so as to include the worker 40 corresponding to the identification mark 60. The processing unit 11 also identifies the detected worker 40 based on the mark ID of the detected identification mark 60.

[0074] The skeleton detection unit 112 of the processing unit 11 inputs the image of the worker 40 extracted in step S201 into the skeleton detection model 123, detects the skeleton of the worker 40, and generates skeleton data d (step S202).

[0075] The processing unit 11 determines in step S202 whether or not a skeleton has been detected (step S203). If it is determined that a skeleton has been detected ("YES" in step S203), the processing unit 11 deletes the oldest skeleton data d from the skeleton data merge D at that point and merges the skeleton data d of the latest frame generated in step S106 into the skeleton data merge D (step S204). Furthermore, if the number of times step S204 is executed after the start of the behavioral analysis process is less than the predetermined number of frames (6 frames in this embodiment), the number of skeletal data d included in the combined skeletal data D after combining the skeletal data d in step S204 will be less than 6 frames. In this case, undetected data may be used for the missing frames, or the work determination process may be terminated without proceeding to step S206 (without determining whether the work is being performed).

[0076] On the other hand, if it is determined in step S203 that no skeleton has been detected ("NO" in step S203), the processing unit 11 deletes the oldest skeleton data d from the skeleton data merge D at that point and merges the undetected data that does not contain data related to skeletons into the skeleton data merge D (step S205).

[0077] When step S204 or step S205 is completed, the work determination unit 113 of the processing unit 11 inputs the skeletal data combination D into the work determination model 124 to determine the work being performed by the worker 40 in the captured image 30 acquired in step S101 (step S206). The work determination unit 113 of the processing unit 11 also records the determination result in the action data area 126, corresponding to the imaging time calculated in step S104 (step S207). When step S207 is completed, the processing unit 11 terminates the work determination process and returns the process to the action analysis process.

[0078] In Figure 12, a flow for analyzing the work of worker 40 using skeletal data d is shown as an example, but the flow for work determination processing is not limited to this. For example, it could be a flow for identifying a work that is pre-associated with the position of worker 40 calculated in step S105.

[0079] Returning to Figure 11, once the work determination process (step S106) is completed, the processing unit 11 determines whether the worker 40, whose identification mark 60 was detected in step S102, was also detected in the previously acquired image 30 (step S107). If it is determined that the worker 40 was not detected in the previously acquired image 30 ("NO" in step S107), the processing unit 11 determines that the worker 40 was not detected during a certain period of not being detected, and executes the following steps S108 to S110.

[0080] The undetected time calculation unit 116 of the processing unit 11 calculates the length of the undetected period (ΔT in the example above) (step S108). Here, the undetected time calculation unit 116 identifies the last captured image 30 (first captured image 301) in which the worker 40 was last seen before the undetected period, and identifies the capture time (T2) of the captured image 30 taken after that captured image 30 as the start time of the undetected period. The undetected time calculation unit 116 also identifies the capture time (T4) of the captured image 30 taken immediately before the captured image 30 (second captured image 302) acquired in step S101 as the end time of the undetected period. Then, the undetected time calculation unit 116 calculates the length of the undetected period (ΔT) from the difference between the end time (T4) and the start time (T2).

[0081] The undetected activity estimation unit 117 of the processing unit 11 estimates the location where worker 40 was during the undetected period (step S109) using the method described above, based on worker 40's position immediately before the undetected period (P1), position immediately after the undetected period (position calculated in step S105) (P5), and the length of the undetected period (ΔT) calculated in step S108.

[0082] Furthermore, the undetected activity estimation unit 117 of the processing unit 11 estimates the work that worker 40 was performing during the undetected period using the method described above, based on the worker's position immediately before the undetected period (P1), position immediately after the undetected period (P5), and the length of the undetected period (ΔT) (step S110).

[0083] If step S110 is completed, or if it is determined in step S107 that worker 40 was also detected in the previously acquired image 30 ("YES" in step S107), the processing unit 11 records the results of the analysis of worker 40's actions in the action data area 126 (step S111). Specifically, the processing unit 11 records the calculation result of worker 40's position in step S105 and the determination result of worker 40's work in step S106 in the action data area 126, corresponding to the imaging time calculated in step S104. If the branch in step S107 is "NO", the estimated results of worker 40's position and work during the undetected period are recorded in the action data area 126 along with the start and end times of the undetected period.

[0084] The processing unit 11 determines whether or not the video contains image data for the next captured image 30 (frame image) (step S112). If it determines that there is image data for the next image ("YES" in step S112), it returns to step S101. If it determines that there is no image data for the next image ("NO" in step S112), the processing unit 11 terminates the behavioral analysis process.

[0085] Steps S101 to S112 are repeatedly executed at intervals corresponding to the frame rate of imaging by the camera 20. For example, if the frame rate is 10fps, 10 frames of images 30 are captured per second, so steps S101 to S112 are repeatedly executed every 1 / 10th of a second. Furthermore, if multiple identification marks 60 are detected in step S103, the processing unit 11 executes the processes in subsequent steps S105 to S111 for each identification mark 60 (i.e., for each corresponding worker 40).

[0086] (Analysis of the behavior of 40 workers) Next, we will explain an example of the results of the analysis of the behavior of 40 workers. Based on the behavioral data accumulated through the above-described behavioral analysis process, analysis data related to the worker's (40) behavior can be generated. The analysis data is stored in the behavioral data area (126). For example, the processing unit (11) generates a behavioral data table (1264) based on the behavioral data accumulated through the above-described behavioral analysis process.

[0087] Figure 13 shows an example of behavioral data table 1264. The behavior data table 1264 extracts the actions of each worker 40 every 10 seconds. Therefore, one row of data in the behavior data table 1264 is generated for each worker 40 at a frequency of once every 10 seconds. In the behavior data table 1264 in Figure 13, data related to the actions of worker 40 corresponding to mark ID "A" (hereinafter referred to as "worker 40A") and worker 40 corresponding to mark ID "B" (hereinafter referred to as "worker 40B"). Worker 40A mainly travels back and forth between sub-area 4 and warehouse 5, transporting materials necessary for product manufacturing from among the materials stored in warehouse 5 to sub-area 4. Worker 40B mainly performs work related to product manufacturing at workbench 50 in process area 3.

[0088] The "Time" column in the behavior data table 1264 represents the time the data row was created. The "Mark ID" is the Mark ID of the identification mark 60 detected by the Mark Detection Unit 111. From the Mark ID, the worker 40 corresponding to that data row can be identified. The "coordinates" refer to the position of worker 40 at the "time" of that data row, and are either position coordinates calculated by the coordinate calculation unit 115 or position coordinates estimated by the undetected action estimation unit 117. "Work Classification" is the result of work determination by the work determination unit 113, or the result of work estimation by the undetected behavior estimation unit 117. Here, the work name is recorded, but the work ID from the work classification master 1263 in Figure 5 may be recorded instead of the work name. The "Estimated Status" indicates whether the data row contains estimated data related to the undetected period. Specifically, the "Estimated Status" is set to "ON" if the "Coordinates" and "Work Classification" in the data row were estimated by the undetected behavior estimation unit 117, and to "OFF" if they were calculated and determined from the workers 40 captured in the image 30. Note that data rows where the "Estimated Status" is "ON" may be calculated in advance and stored in the behavior data table 1264, or they may be processed and calculated each time the data is needed (when visualizing) by the undetected time calculation unit 116 and the undetected behavior estimation unit 117. Furthermore, while it was stated that each data row in the behavior data table 1264 is generated every 10 seconds, this is not limited to that. For example, one data row may be generated when at least one of the "task classification" and "estimated status" changes.

[0089] Furthermore, the processing unit 11 may generate various aggregation tables based on the behavioral data table 1264, depending on the purpose of analysis by the management user. Figure 14 shows an example of summary table 1265. Summary table 1265 is created by combining multiple data rows from the activity data table 1264 that share the same "Mark ID" and "Work Classification" and have consecutive time periods into a single data row, while also adding data items for "Work Location," "Time Spent," and "Distance Traveled." Therefore, one data row in summary table 1265 corresponds to one task performed by a particular worker 40.

[0090] The "Work Location" column in summary table 1265 identifies the work location corresponding to the "Coordinates" based on the "Start Coordinates" and "End Coordinates" of each work location in the work area master 1261 shown in Figure 3. Although the work location name is listed here, the location ID shown in Figure 3 may be recorded instead of the work location name. "Duration" represents the length of time that worker 40 stayed at the "work location" corresponding to that data row and performed the work of the "work classification". "Duration" is calculated as "n × 10" when the data row in the summary table 1265 is a combination of n data rows from the behavior data table 1264. "Distance traveled" represents the distance traveled by worker 40 within the period of that data row. "Distance traveled" is calculated, for example, based on the change in worker 40's position frame by frame (or every predetermined number of frames).

[0091] Furthermore, the processing unit 11 may further process the summary table 1265 to generate an individual summary table 1266. Figure 15 shows an example of an individual summary table 1266. Individual summary table 1266 is obtained by extracting the data row for worker 40 (in this case, worker 40A) corresponding to a certain mark ID (in this case, mark ID "A") from summary table 1265 in Figure 14. From individual summary table 1266, it is possible to understand the changes in worker 40A's actions.

[0092] In the example shown in Figure 15, worker 40A is determined to have performed "material sorting" work in sub-area 4 until "09:07:10" (9:07:10), then is estimated to have moved from sub-area 4 to warehouse 5 via passage 6 until "09:07:30", and is determined to have performed material picking work in warehouse 5 until "09:08:00". Subsequently, is estimated to have moved from warehouse 5 to sub-area 4 via passage 6 until "09:08:20", and is determined to have performed material sorting work in sub-area 4 until "09:08:40". In other words, from the contents of the individual summary table 1266 shown in Figure 15, it can be determined that worker 40A made one round trip between sub-area 4 and warehouse 5 via passage 6 and transported materials from warehouse 5 to sub-area 4 once. By identifying the number of times a combination of tasks, such as "moving," "picking," and "transporting," was repeated among the series of tasks shown in the individual summary table 1266 in Figure 15, the number of times worker 40A transported materials can be determined. Furthermore, by adding the "stay time" for the series of tasks, the time required for transportation can be determined.

[0093] The following describes examples (1) to (8) of behavioral analysis that can be performed based on the behavioral data table 1264, the aggregation table 1265, and the individual aggregation table 1266, etc.

[0094] (1) Display the location of worker 40 on map 70. The processing unit 11 may display a map 70 on the display unit 13 that shows the positions of the workers 40 calculated (derived) based on a plurality of captured images 30. For example, as shown in Figure 2, the processing unit 11 may plot the positions of each worker 40 in real time on a map 70 that provides an overhead view of the floor 100. Alternatively, each worker 40 may be represented on the map 70 by a sign such as a graphic or symbol.

[0095] Furthermore, the processing unit 11 may display on the map 70 the estimated location (or location, if estimated) of the worker 40 during the undetected period, which is estimated based on the multiple captured images 30. For example, the display unit 13 may be made capable of displaying a map 70 at any point in the past, and the estimated location (or position) of the worker 40 may be plotted on the map 70.

[0096] (2) Display heatmap 71 The processing unit 11 may display a heat map 71 on the display unit 13 that represents the time spent at each location of the worker 40, based on the changes in the worker's position based on multiple captured images 30, and the location or position estimation results.

[0097] Figure 16 shows an example of a heatmap 71. The vertical axis of heatmap 71 represents time periods in units of one hour, and the horizontal axis represents the X-coordinate in workspace 2. In heatmap 71, locations with longer dwell times per hour are colored darker.

[0098] (3) Display Graph 72 of Stay Time and Distance Traveled The processing unit 11 may display a stay time / travel distance graph 72 on the display unit 13 based on the changes in the worker's position 40 based on multiple captured images 30, and the location or position estimation results. The stay time / travel distance graph 72 includes information relating to the worker's stay time at a predetermined location for each time period, and information relating to the worker's travel distance (momentum) for each time period.

[0099] Figure 17 shows an example of a 72 graph of time spent and distance traveled. The dwell time / travel distance graph 72 includes a bar graph showing the total time the worker 40 spent at each location for each time period (in units of one hour), and a line graph showing the travel distance at each location for each time period. In the example shown in Figure 17, it can be seen that the worker 40 spent most of the time in process area 3, but the dwell time and travel distance in aisle 6 were longer during the 13:00 (~14:00) time period.

[0100] (4) Display Table 73 of the percentage of time spent The processing unit 11 may display a work ratio table 73 on the display unit 13, which includes information on the proportion of work performed by the worker 40 during each time period, based on the work determination results and estimation results.

[0101] Figure 18 shows an example of a work ratio table 73. The work ratio table 73 shows, in bar graph form, the execution time of net work and ancillary work for each worker 40A to 40C, on a daily and hourly basis. Whether each task belongs to net work or ancillary work is classified based on the work classification master 1263 in Figure 5.

[0102] (5) Display Table 74 of Representative Accommodation Locations The processing unit 11 may display a representative stay location table 74 on the display unit 13, which includes information about the place where the worker 40 stayed the longest during each time period, based on the changes in the worker's position based on multiple captured images 30, and the location or position estimation results.

[0103] Figure 19 shows an example of the representative accommodation location table 74. The representative accommodation table 74 shows, for each worker 40A to 40C, the accommodation location where they stayed the longest during each time period, both daily and hourly. Each cell in the table may be colored with a color corresponding to the location.

[0104] (6) Time Chart 75 The processing unit 11 may display a time chart 75 on the display unit 13, which includes information related to the history of the work performed by the worker 40, based on the work determination result and estimation result.

[0105] Figure 20 shows an example of a time chart 75. Time Chart 75 is a bar graph that shows what work worker 40A performed and when. Here, "movement" (moving from sub-area 4 to warehouse 5), "picking" (taking materials from warehouse 5), "transporting" (carrying materials from warehouse 5 to sub-area 4), and "organizing materials" (organizing materials in warehouse 5) are color-coded and arranged in a bar pattern.

[0106] (7) Graph of the number of transports and transport time 76 The processing unit 11 may display a transport count / transport time graph 76 on the display unit 13, which includes information on the number of times and transport time of transporting materials between two points, based on the changes in the worker's position 40 based on multiple captured images 30 and the location or position estimation results. Alternatively, the display unit 13 may display information on only one of the number of times and transport time of transporting materials.

[0107] Figure 21 shows an example of the transport frequency / transport time graph 76. The transport count / transport time graph 76 includes a bar graph showing the number of times the worker 40 transported materials from warehouse 5 to sub-area 4 for each time period of one hour, and a line graph showing the total time required for transporting materials. The number of material transport counts can be determined, for example, from the number of times the combination of "moving," "picking," and "transporting" operations was repeated in the individual summary table 1266 of Figure 15, as described above. The transport time is obtained by adding the "stay time" for the "moving," "picking," and "transporting" operations. However, the method of calculating the transport time is not limited to this; the stay time related to the "moving" and "transporting" operations may be used as the transport time, or the stay time related to the "material sorting" operation may be used as the transport time, or the stay time related to the "material sorting" operation may be added to the transport time.

[0108] (8) Analysis pie chart 77 The processing unit 11 may display an analysis pie chart 77 on the display unit 13, which includes information related to the work time of the work performed by the worker 40, based on the work determination result, estimation result, and time information.

[0109] Figure 22 shows an example of an analysis pie chart 77. The analysis pie chart 77 shows the work time of a certain worker 40 during a certain period, classified into "net work time," "net work time delay," and "incidental work, etc." "Net work time" represents the work time of tasks belonging to net work that falls within the standard time set as the standard work time for net work. "Net work time delay" is the work time of tasks belonging to net work that are performed in excess of the standard time. Work efficiency may be calculated based on work time and standard time. "Incidental work, etc." is the total time of the breakdown of "material sorting," "outbound processing," "thinking time," and "consultation / communication," and in the analysis pie chart 77 of Figure 22, the work time for each of these breakdowns is also displayed. Note that the breakdown of "incidental work, etc." may be displayed in a separate auxiliary pie chart. By displaying such an analysis pie chart 77, it is possible to visually grasp work losses and waste. Furthermore, the information obtained from the analysis pie chart 77 can be used to calculate the optimal number of personnel, the return on investment when introducing IT equipment and other facilities, and to calculate KPIs (Key Performance Indicators).

[0110] (9) Video display at the time the anomaly occurred The processing unit 11 may detect an anomaly in the work performed by the worker 40 based on the changes in the worker's position based on multiple captured images 30, and the location or position estimation results, and may display the video of the worker 40 captured by the camera 20 at the time the anomaly occurred on the display unit 13. For example, in the stay time / travel distance graph 72 shown in Figure 17, if the trend of the place of stay and the amount of movement during the 13:00 time period differ significantly from other time periods, it may be determined that an anomaly has occurred, and the video of the 13:00 time period may be displayed on the display unit 13. Note that the target of anomaly detection is not limited to the stay time / travel distance graph 72, but a video may be similarly displayed when an anomaly is detected in the map 70, heat map 71, work ratio table 73, representative place of stay table 74, time chart 75, and transport count / transport time graph 76, etc.

[0111] <Effects> As described above, the behavior analysis device 10 as an information processing device according to the above embodiment includes a processing unit 11, which acquires a plurality of captured images 30 taken by the camera 20 at multiple different time points, determines whether or not a worker 40 is captured in each of the acquired plurality of captured images 30, derives the position of worker 40 corresponding to each captured image 30 in which it is determined that a worker 40 is captured, and if it is determined that a worker 40 is not captured in any of the plurality of captured images 30 taken during a certain undetected period, it estimates the location where worker 40 was during the undetected period based on the position P1 (first information) of worker 40 captured in the first captured image 301 (first image) taken immediately before the undetected period, the position P5 (second information) of worker 40 captured in the second captured image 302 (second image) taken immediately after the undetected period, and the length ΔT (third information) of the undetected period. This allows for the estimation of the location of worker 40 during undetected periods outside the camera's imaging range, even if the camera's imaging range does not cover the entire range of worker 40's movements. Therefore, worker 40's movements can be tracked regardless of whether they are within or outside the camera's imaging range. Furthermore, since data related to worker 40's movements can be automatically accumulated, various analyses related to worker 40's movements can be performed with high accuracy and immediately by aggregating and processing this data. This reduces the effort required for behavioral analysis. For example, it becomes possible to immediately detect daily fluctuations in behavior on a production line and take appropriate countermeasures in a timely manner.

[0112] Furthermore, the behavioral analysis device 10 as an information processing device according to the above embodiment includes a processing unit 11, which acquires a plurality of captured images 30 taken by the camera 20 at multiple different time points, determines whether or not a worker 40 is captured in each of the acquired plurality of captured images 30, derives the position of worker 40 corresponding to each captured image 30 in which it is determined that a worker 40 is captured, and if it is determined that a worker 40 is not captured in any of the plurality of captured images 30 taken during a certain undetected period, it determines whether a worker 40 was in a predetermined location during the undetected period based on the position P1 (first information) of worker 40 captured in the first captured image 301 (first image) taken immediately before the undetected period, the position P5 (second information) of worker 40 captured in the second captured image 302 (second image) taken immediately after the undetected period, and the length ΔT (third information) of worker 40. This allows for the determination of whether or not worker 40 was in a predetermined location during the undetected period outside the imaging range, even if the imaging range of camera 20 does not cover the entire range of worker 40's movements. It also allows for the determination of whether or not worker 40 was performing the corresponding task at that location during the undetected period.

[0113] Furthermore, the processing unit 11 estimates the location of worker 40 during the undetected period by inputting position P1, position P2, and the length ΔT of the undetected period into the behavior estimation model 125. The behavior estimation model 125 is machine-trained to output an estimated result of the location of worker 40 during the undetected period by inputting information relating to the position of worker 40 immediately before a certain undetected period, information relating to the position of worker 40 immediately after the undetected period, and information relating to the length of the undetected period. As a result, the location of worker 40 during the undetected period can be estimated with a simple process of inputting position P1, position P2, and the length ΔT of the undetected period into the behavior estimation model 125.

[0114] Furthermore, the processing unit 11 estimates the work that worker 40 was performing at the time the image 30 was captured, based on each captured image 30 in which worker 40 is determined to be present. Based on the above-mentioned positions P1 and P2, and the length ΔT of the undetected period, it estimates the work that worker 40 was performing during the undetected period. This makes it possible to estimate the work that worker 40 was performing during the undetected period when they were outside the camera 20's imaging range, even if the camera 20's imaging range does not cover the entire range of worker 40's movements. Therefore, it is possible to analyze worker 40's actions in more detail, regardless of whether they are within or outside the camera 20's imaging range.

[0115] Furthermore, the processing unit 11 inputs the above-mentioned positions P1 and P2, and the length ΔT of the undetected period, into the behavior estimation model 125 to estimate the work that worker 40 was performing during the undetected period. The behavior estimation model 125 is machine-trained to output an estimated result of the work that worker 40 was performing during the undetected period by inputting information relating to the position of a worker 40 immediately before a certain undetected period, information relating to the position of the worker 40 immediately after the undetected period, and information relating to the length of the undetected period. As a result, the work that worker 40 was performing during the undetected period can be estimated with a simple process of inputting positions P1 and P2, and the length ΔT of the undetected period into the behavior estimation model 125.

[0116] Furthermore, the processing unit 11 further estimates the movement path of the worker 40 during the undetected period based on the above-mentioned positions P1 and P2, and the length ΔT of the undetected period. This makes it possible to understand the worker 40's actions during the undetected period in more detail.

[0117] Furthermore, the processing unit 11 detects identification marks 60 worn by multiple workers 40 in each of the multiple captured images 30, and based on the detection results of the identification marks 60, it determines whether or not a worker 40 is captured in each of the multiple captured images 30, and derives the position of the captured worker 40 for each of the multiple workers 40. This makes it possible to identify each of the multiple different workers 40 and perform behavioral analysis on each worker 40.

[0118] Furthermore, the processing unit 11 displays on the display unit 13 a map 70 showing the positions of the workers 40 derived from the multiple captured images 30, as well as the estimated locations of the workers 40. This allows the position of each worker 40 on the floor 100 to be confirmed on the map 70, regardless of whether the worker 40 is within or outside the imaging range of the camera 20.

[0119] Furthermore, the processing unit 11 displays a heat map 71 on the display unit 13 that represents the time spent at each location of the worker 40, based on the changes in the worker's position and the location estimation results from the multiple captured images 30. This makes it possible to visually and intuitively grasp the time spent at each location of the worker 40, regardless of whether the worker 40 is within or outside the imaging range of the camera 20.

[0120] Furthermore, the processing unit 11 displays a stay time / travel distance graph 72 on the display unit 13, which includes information on the stay time of the worker 40 at predetermined locations for each time period, based on the changes in the worker's position and the location estimation results from multiple captured images 30. This makes it possible to visually and intuitively grasp the stay time of the worker 40 at each location for each time period, regardless of whether the worker 40 is within or outside the imaging range of the camera 20.

[0121] Furthermore, the processing unit 11 displays a stay time / travel distance graph 72 on the display unit 13, which includes information on the travel distance (momentum) of the worker 40 for each time period, based on the changes in the worker 40's position and the location estimation results based on multiple captured images 30. This makes it possible to visually and intuitively grasp the travel distance of the worker 40 at each location for each time period, regardless of whether the worker 40 is within or outside the imaging range of the camera 20.

[0122] Furthermore, the processing unit 11 displays a work ratio table 73 on the display unit 13, which includes information on the proportion of work performed by worker 40 during each time period, based on the estimated work results. This allows for a visual and intuitive understanding of what proportion of work worker 40 was performing, regardless of whether worker 40 was within or outside the imaging range of the camera 20.

[0123] Furthermore, the processing unit 11 displays a representative stay location table 74 on the display unit 13, which includes information about the place where the worker 40 stayed the longest during each time period, based on the changes in the worker 40's position and the location estimation results from the multiple captured images 30. This makes it possible to visually and intuitively grasp the worker 40's representative stay locations, regardless of whether the worker 40 is within or outside the imaging range of the camera 20.

[0124] Furthermore, the processing unit 11 displays a time chart 75 on the display unit 13, which includes information about the history of the work performed by the worker 40, based on the estimated work results. This allows the worker 40's work history to be grasped visually and intuitively, regardless of whether the worker 40 is within or outside the imaging range of the camera 20.

[0125] Furthermore, the processing unit 11 displays a transport count / transport time graph 76 on the display unit 13, which includes information on at least one of the number of times and the transport time of the worker 40 transporting materials between two points, based on the changes in the worker 40's position and the location estimation results based on the multiple captured images 30. This allows the user to visually and intuitively grasp the number of times the worker 40 transported materials and the time required for transport, regardless of whether the worker 40 is within or outside the imaging range of the camera 20.

[0126] Furthermore, the processing unit 11 displays an analysis pie chart 77 on the display unit 13, which includes information related to the work time of the work performed by worker 40, based on the estimated work results. This allows for a visual and intuitive understanding of the work time of worker 40, regardless of whether worker 40 is within or outside the imaging range of the camera 20.

[0127] Furthermore, the processing unit 11 acquires multiple captured images 30 taken by the camera 20 as a video, detects abnormalities in the work performed by the worker 40 based on the changes in the worker's position and the location estimation results based on the multiple captured images 30, and displays the video of the worker 40 taken by the camera 20 at the time the abnormality occurred on the display unit 13. This allows the details of the abnormality to be visually confirmed in the video. In addition, the cause of the abnormality can be easily identified from the situation shown in the video.

[0128] Furthermore, the program 121 according to this embodiment causes the processing unit 11, acting as a computer, to execute the following processes: acquiring multiple captured images 30 taken by the camera 20 at multiple different points in time; determining whether or not a worker 40 is visible in each of the acquired multiple captured images 30; deriving the position of the worker 40 corresponding to each captured image 30 in which it is determined that the worker 40 is visible; and, if it is determined that the worker 40 is not visible in any of the multiple captured images 30 taken during a certain undetected period, estimating the location where the worker 40 was during the undetected period based on the position P1 (first information) of the worker 40 in the first captured image 301 (first image) taken immediately before the undetected period, the position P5 (second information) of the worker 40 in the second captured image 302 (second image) taken immediately after the undetected period, and the length ΔT (third information) of the undetected period. This allows us to estimate the location of worker 40 during undetected periods outside the camera's imaging range, even if the camera's imaging range does not cover the entire range of worker 40's movements. Therefore, it is possible to understand worker 40's movements regardless of whether they are within or outside the camera's imaging range. Furthermore, various analyses related to worker 40's movements can be performed with high accuracy and immediately. This reduces the effort required for behavioral analysis.

[0129] Furthermore, the program 121 according to this embodiment causes the processing unit 11, acting as a computer, to execute the following processes: acquiring multiple captured images 30 taken by the camera 20 at multiple different time points in time; determining whether or not a worker 40 is visible in each of the acquired multiple captured images 30; deriving the position of the worker 40 corresponding to each captured image 30 in which it is determined that the worker 40 is visible; and, if it is determined that the worker 40 is not visible in any of the multiple captured images 30 taken during a certain undetected period, determining whether the worker 40 was in a predetermined location during the undetected period based on the position P1 (first information) of the worker 40 in the first captured image 301 (first image) taken immediately before the undetected period, the position P5 (second information) of the worker 40 in the second captured image 302 (second image) taken immediately after the undetected period, and the length ΔT (third information) of the undetected period. This allows for the determination of whether or not worker 40 was in a predetermined location during the undetected period outside the imaging range, even if the imaging range of camera 20 does not cover the entire range of worker 40's movements. It also allows for the determination of whether or not worker 40 was performing the corresponding task at that location during the undetected period.

[0130] Furthermore, the behavioral analysis method as an information processing method according to this embodiment acquires multiple captured images 30 taken by the camera 20 at multiple different time points, determines whether or not a worker 40 is captured in each of the acquired multiple captured images 30, derives the position of worker 40 corresponding to each captured image 30 in which it is determined that worker 40 is captured, and if it is determined that worker 40 is not captured in any of the multiple captured images 30 taken during a certain undetected period, the location where worker 40 was during the undetected period is estimated based on the position P1 (first information) of worker 40 captured in the first captured image 301 (first image) taken immediately before the undetected period, the position P5 (second information) of worker 40 captured in the second captured image 302 (second image) taken immediately after the undetected period, and the length ΔT (third information) of the undetected period. This allows us to estimate the location of worker 40 during undetected periods outside the camera's imaging range, even if the camera's imaging range does not cover the entire range of worker 40's movements. Therefore, it is possible to understand worker 40's movements regardless of whether they are within or outside the camera's imaging range. Furthermore, various analyses related to worker 40's movements can be performed with high accuracy and immediately. This reduces the effort required for behavioral analysis.

[0131] Furthermore, the behavioral analysis method as an information processing method according to this embodiment acquires multiple captured images 30 taken by the camera 20 at multiple different time points, determines whether or not a worker 40 is captured in each of the acquired multiple captured images 30, derives the position of worker 40 corresponding to each captured image 30 in which it is determined that worker 40 is captured, and if it is determined that worker 40 is not captured in any of the multiple captured images 30 taken during a certain undetected period, it determines whether worker 40 was in a predetermined location during the undetected period based on the position P1 (first information) of worker 40 captured in the first captured image 301 (first image) taken immediately before the undetected period, the position P5 (second information) of worker 40 captured in the second captured image 302 (second image) taken immediately after the undetected period, and the length ΔT (third information) of the undetected period. This allows for the determination of whether or not worker 40 was in a predetermined location during the undetected period outside the imaging range, even if the imaging range of camera 20 does not cover the entire range of worker 40's movements. It also allows for the determination of whether or not worker 40 was performing the corresponding task at that location during the undetected period.

[0132] Furthermore, the behavior analysis system 1 as an information processing system according to this embodiment includes a processing unit 11, which acquires multiple captured images 30 taken by the camera 20 at multiple different time points, determines whether or not a worker 40 is captured in each of the acquired multiple captured images 30, derives the position of worker 40 corresponding to each captured image 30 in which it is determined that a worker 40 is captured, and if it is determined that a worker 40 is not captured in any of the multiple captured images 30 taken during a certain undetected period, it estimates the location where worker 40 was during the undetected period based on the position P1 (first information) of worker 40 captured in the first captured image 301 (first image) taken immediately before the undetected period, the position P5 (second information) of worker 40 captured in the second captured image 302 (second image) taken immediately after the undetected period, and the length ΔT (third information) of the undetected period. This allows us to estimate the location of worker 40 during undetected periods outside the camera's imaging range, even if the camera's imaging range does not cover the entire range of worker 40's movements. Therefore, it is possible to understand worker 40's movements regardless of whether they are within or outside the camera's imaging range. Furthermore, various analyses related to worker 40's movements can be performed with high accuracy and immediately. This reduces the effort required for behavioral analysis.

[0133] Furthermore, the behavior analysis system 1 as an information processing system according to this embodiment includes a processing unit 11, which acquires multiple captured images 30 taken by the camera 20 at multiple different time points, determines whether or not a worker 40 is captured in each of the acquired multiple captured images 30, derives the position of the worker 40 corresponding to each captured image 30 in which it is determined that the worker 40 is captured, and if it is determined that the worker 40 is not captured in any of the multiple captured images 30 taken during a certain undetected period, it determines whether the worker 40 was in a predetermined location during the undetected period based on the position P1 (first information) of the worker 40 captured in the first captured image 301 (first image) taken immediately before the undetected period, the position P5 (second information) of the worker 40 captured in the second captured image 302 (second image) taken immediately after the undetected period, and the length ΔT (third information) of the undetected period. This allows for the determination of whether or not worker 40 was in a predetermined location during the undetected period outside the imaging range, even if the imaging range of camera 20 does not cover the entire range of worker 40's movements. It also allows for the determination of whether or not worker 40 was performing the corresponding task at that location during the undetected period.

[0134] <Other> The above-described embodiments are merely examples of the information processing apparatus, program, information processing method, and information processing system according to the present invention, and are not limited thereto. For example, at least a portion of the processing performed by the processing unit 11 of the behavioral analysis device 10 in the above embodiment may be shared and performed by one or more devices other than the behavioral analysis device 10 in the behavioral analysis system 1. In this case, one or more devices may include a camera 20. For example, an edge AI camera (endpoint AI camera) may be used as the camera 20, and a processing unit provided in the camera 20 may function as at least a part of the mark detection unit 111, skeleton detection unit 112, work determination unit 113, time calculation unit 114, coordinate calculation unit 115, undetected time calculation unit 116, and undetected action estimation unit 117 of the processing unit 11 in the above embodiment. In other words, the processing unit provided in the camera 20 may perform at least a part of the detection of identification marks 60 (identification of worker 40), detection of the skeleton of worker 40, determination of the work of worker 40, calculation of time, calculation of the position coordinates of worker 40, identification of undetected periods, and estimation of actions during undetected periods. In this case, at least a part of the mark detection model 122, skeleton detection model 123, work determination model 124, and action estimation model 125 in the above embodiment may be stored in a storage unit provided in the camera 20. Alternatively, one of the multiple cameras 20 may be designated as an edge AI camera, and the video captured by each camera 20 may be transmitted to the aforementioned edge AI camera. The processing unit provided in the edge AI camera may then perform at least some of the following: detection of identification marks 60 (identification of worker 40), detection of the worker 40's skeleton, determination of the worker 40's work, calculation of time, calculation of the worker 40's position coordinates, identification of undetected periods, and estimation of actions during undetected periods. Furthermore, multiple edge AI cameras connected to each other and capable of sending and receiving data may be used as multiple cameras 20. In this case, based on the video captured by each edge AI camera, a processing unit provided in each edge AI camera may perform at least some of the following: detection of identification marks 60 (identification of worker 40), detection of the worker 40's skeleton, determination of the worker 40's work, calculation of time, calculation of the worker 40's position coordinates, identification of undetected periods, and estimation of actions during undetected periods. The results of these processes may be transmitted to any of the other edge AI cameras upon request from that edge AI camera. In other words, multiple edge AI cameras may share information regarding the worker 40's actions during undetected periods in real time. In addition, an edge AI camera (or behavior analysis device 10) that receives the results of these processes may integrate the behavior data for all workers 40. If a processing unit in the camera 20 performs a portion of the processing previously performed by the processing unit 11 of the behavior analysis device 10, then the processing unit 11 of the behavior analysis device 10 and the processing unit of the camera 20 correspond to "one or more processing units." Furthermore, if the processing unit in the camera 20 performs all of the processing previously performed by the processing unit 11 of the behavior analysis device 10, then the processing unit of the camera 20 corresponds to "one or more processing units," and the camera 20 corresponds to an "information processing device."

[0135] Furthermore, the behavioral analysis device 10 (information processing device) may also be equipped with a camera 20 (imaging unit).

[0136] Furthermore, although the above embodiment was described using an example of analyzing image data of a video captured by camera 20 in real time, the method is not limited to this, and analysis may also be performed using image data of past videos recorded in the image data area 127. In this case, each video needs to be synchronized. That is, the cameras 20 that captured each video need to be synchronized with each other so that the time of each video is synchronized.

[0137] Alternatively, the program, machine learning model, and data stored in the memory unit 12 of the behavioral analysis device 10 may be stored on a server or database on the network, and the necessary data may be obtained from the server or database via the communication unit 14.

[0138] Furthermore, in the above embodiment, the subject whose actions are to be analyzed is exemplified as a worker 40 performing work on the floor 100, but the invention is not limited to this, and any subject performing actions within a range of action that partially overlaps with the imaging range of the camera 20 can be used as the subject of analysis.

[0139] Furthermore, the "first information" may be any information relating to the position of worker 40 as seen in the first image taken immediately before the undetected period, and does not necessarily have to be the position coordinates themselves. Similarly, the "second information" may be any information relating to the position of worker 40 as seen in the second image taken immediately after the undetected period, and does not necessarily have to be the position coordinates themselves. For example, the first and second information may be information representing the area in which worker 40 is located among multiple areas.

[0140] Furthermore, the "third piece of information" may be any information relating to the length of the undetected period, and does not necessarily have to be the length of the undetected period itself. For example, the third piece of information may be information indicating which of the multiple length categories the length of the undetected period belongs to.

[0141] Furthermore, the method of analyzing behavior is not limited to the method using the map 70, heat map 71, stay time / travel distance graph 72, work ratio table 73, representative stay location table 74, time chart 75, transport count / transport time graph 76, and analysis pie chart 77 as exemplified in the above embodiment.

[0142] Furthermore, in the above embodiment, the results of the behavioral analysis (for example, the map 70 in Figure 2, the heatmap 71 in Figure 16, the stay time / travel distance graph 72 in Figure 17, the work ratio table 73 in Figure 18, the representative stay location table 74 in Figure 19, the time chart 75 in Figure 20, the transport count / transport time graph 76 in Figure 21, and the analysis pie chart 77 in Figure 22) were displayed on the display unit 13. However, instead, the results of the behavioral analysis may be output to any output unit. For example, the results of the behavioral analysis may be printed on paper using a printing device as the output unit.

[0143] Furthermore, while the above description discloses examples in which the HDD and SSD of the storage unit 12 are used as the computer-readable medium for the program according to the present invention, the invention is not limited to these examples. Other computer-readable mediums that can be used include information recording media such as flash memory and CD-ROM. In addition, a carrier wave can also be used as a medium for providing the data of the program according to the present invention via a communication line.

[0144] Furthermore, it goes without saying that the detailed configuration and operation of each component of the behavioral analysis system 1 in the above embodiment can be appropriately modified without departing from the spirit of the present invention.

[0145] Although embodiments of the present invention have been described, the scope of the present invention is not limited to the embodiments described above, but includes the scope of the invention as described in the claims and its equivalents. [Explanation of symbols]

[0146] 1. Behavioral Analysis System (Information Processing System) 2. Workroom 3 Process Area 4 Sub-areas 5 Warehouse 6 aisles 10. Behavioral analysis device (information processing device) 11 Processing Section 111 Mark detection unit 112 Skeleton detection unit 113 Work Judgment Department 114 Time calculation section 115 Coordinate Calculation Unit 116 Undetected Time Calculation Unit 117 Undetected Behavior Estimation Unit 12 Storage section 121 Programs 122 Mark Detection Models 123 Skeleton detection model 124 Work Decision Model 125 Behavior Estimation Models (Machine Learning Models) 126 Behavioral Data Area 1261 Work Area Master 1262 Worker Master 1263 Work Classification Master 1264 Behavioral Data Table 1265 Summary Table 1266 Individual Summary Tables 127 Image data area 13 Display Unit (Output Unit) 14 Communications Department 15 bus 20, 20a~20c Camera (imaging unit) 30 Acquired Images (Images) 30G image acquisition data (image group) 40, 40A~40C Workers (Target Persons) 41 analysis points 42 axis 50 workbenches 60, 60a, 60b Identification marks (identification signs) 60b Identification Mark 71 Heatmap 72. Graph showing time spent in a location and distance traveled. 73 Work Ratio Table 74 Representative Accommodation Table 75 Time Chart 76. Graph showing the number of deliveries and delivery time. 77 Analysis Pie Chart 100 floors D Skeletal Data Joining P1 position (first information) P2~P4 position P5 position (secondary information) Ra~Rc imaging range T1~T5 time ΔT: Length of undetected period (Third piece of information) d, d1~d7 Skeletal data

Claims

1. Multiple images captured by the imaging unit at different points in time are acquired. Determine whether or not the subject is pictured in each of the acquired multiple images. The position of the subject corresponding to each image in which the subject is determined to be present is derived. If it is determined that the subject is not visible in any of the multiple images taken during a certain period, the location where the subject was during that period is estimated based on first information relating to the subject's position in the first image taken immediately before the period, second information relating to the subject's position in the second image taken immediately after the period, and third information relating to the length of the period. An information processing device equipped with a processing unit.

2. Multiple images captured by the imaging unit at different points in time are acquired. Determine whether or not the subject is pictured in each of the acquired multiple images. The position of the subject corresponding to each image in which the subject is determined to be present is derived. If it is determined that the subject is not captured in any of the multiple images taken during a certain period, then it is determined whether the subject was in a predetermined location during that period based on first information relating to the position of the subject in the first image taken immediately before the period, second information relating to the position of the subject in the second image taken immediately after the period, and third information relating to the length of the period. An information processing device equipped with a processing unit.

3. The aforementioned processing unit, By inputting the first information, the second information, and the third information into a machine learning model, the location where the subject was during the period is estimated. The aforementioned machine learning model is trained to output an estimated location of a certain subject during a certain period by taking as input information relating to the subject's location immediately before a certain period, information relating to the subject's location immediately after the certain period, and information relating to the length of the certain period. The information processing apparatus according to claim 1.

4. The aforementioned processing unit, Based on each image in which the subject is determined to be present, the task the subject was performing at the time the image was taken is estimated. Based on the first information, the second information, and the third information, the work performed by the subject during the period is estimated. The information processing apparatus according to claim 1.

5. The aforementioned processing unit, By inputting the first information, the second information, and the third information into a machine learning model, the tasks performed by the subject during the period are estimated. The aforementioned machine learning model is trained to output an estimated result of the work that a certain subject was performing during a certain period, by taking as input information relating to the location of a certain subject immediately before a certain period, information relating to the location of a certain subject immediately after the certain period, and information relating to the length of the certain period. The information processing apparatus according to claim 4.

6. The processing unit further estimates the movement path of the subject during the period based on the first information, the second information, and the third information. The information processing apparatus according to claim 1.

7. The aforementioned processing unit, In each of the aforementioned multiple images, the identification tags worn by the multiple subjects are detected. Based on the detection results of the identification mark, a determination is made for each of the multiple images as to whether or not the subject is depicted, and the position of the subject if depicted is derived. The information processing apparatus according to claim 1.

8. The processing unit causes the output unit to output the location of the subject derived from the plurality of images, and a map showing the estimated location of the subject. The information processing apparatus according to claim 1.

9. The processing unit causes the output unit to output a heatmap representing the time spent at each location of the subject, based on the changes in the subject's position based on the plurality of images and the estimation results of the location. The information processing apparatus according to claim 1.

10. The processing unit causes the output unit to output information relating to the time spent by the subject at a predetermined location for each time period, based on the changes in the subject's position based on the plurality of images and the estimation results of the location. The information processing apparatus according to claim 1.

11. The processing unit causes the output unit to output information relating to the subject's activity level for each time period, based on the changes in the subject's position based on the plurality of images and the estimation results of the location. The information processing apparatus according to claim 1.

12. The processing unit, based on the estimation results of the work, causes the output unit to output information relating to the proportion of work performed by the subject during each time period. The information processing apparatus according to claim 4 or 5.

13. The processing unit causes the output unit to output information relating to the location where the subject stayed the longest during each time period, based on the changes in the subject's position based on the multiple images and the location estimation results. The information processing apparatus according to claim 1.

14. The processing unit causes the output unit to output information relating to the history of the work performed by the subject, based on the estimation result of the work. The information processing apparatus according to claim 4 or 5.

15. The processing unit causes the output unit to output information relating to at least one of the number of times and the duration of transport of the material between two points for the subject, based on the changes in the subject's position based on the plurality of images and the location estimation results. The information processing apparatus according to claim 1.

16. The processing unit, based on the estimation result of the work, causes the output unit to output information relating to the work time of the work performed by the subject. The information processing apparatus according to claim 4 or 5.

17. The aforementioned processing unit, Multiple images captured by the aforementioned imaging unit are acquired as a video, Based on the changes in the subject's position based on the multiple images and the estimation results of the location, abnormalities in the work performed by the subject are detected. Of the video of the subject captured by the imaging unit, the video at the time the abnormality occurred is output to the output unit. The information processing apparatus according to claim 1.

18. On the computer, A process for acquiring multiple images captured by the imaging unit at multiple different points in time. A process to determine whether or not the subject is pictured in each of the acquired multiple images. A process for deriving the position of the subject corresponding to each image in which the subject is determined to be present. If it is determined that the subject is not visible in any of the multiple images taken during a certain period, the process involves estimating the location of the subject during that period based on first information relating to the subject's position in the first image taken immediately before the period, second information relating to the subject's position in the second image taken immediately after the period, and third information relating to the length of the period. A program that executes the command.

19. On the computer, A process for acquiring multiple images captured by the imaging unit at multiple different points in time. A process to determine whether or not the subject is pictured in each of the acquired multiple images. A process for deriving the position of the subject corresponding to each image in which the subject is determined to be present. If it is determined that the subject is not visible in any of the multiple images taken during a certain period, a process is performed to determine whether the subject was in a predetermined location during that period, based on first information relating to the position of the subject in the first image taken immediately before the period, second information relating to the position of the subject in the second image taken immediately after the period, and third information relating to the length of the period. A program that executes the command.

20. A method of information processing performed by a computer, Multiple images captured by the imaging unit at different points in time are acquired. Determine whether or not the subject is pictured in each of the acquired multiple images. The position of the subject corresponding to each image in which the subject is determined to be present is derived. If it is determined that the subject is not visible in any of the multiple images taken during a certain period, the location where the subject was during that period is estimated based on first information relating to the subject's position in the first image taken immediately before the period, second information relating to the subject's position in the second image taken immediately after the period, and third information relating to the length of the period. Information processing methods.

21. A method of information processing performed by a computer, Multiple images captured by the imaging unit at different points in time are acquired. Determine whether or not the subject is pictured in each of the acquired multiple images. The position of the subject corresponding to each image in which the subject is determined to be present is derived. If it is determined that the subject is not captured in any of the multiple images taken during a certain period, then it is determined whether the subject was in a predetermined location during that period based on first information relating to the position of the subject in the first image taken immediately before the period, second information relating to the position of the subject in the second image taken immediately after the period, and third information relating to the length of the period. Information processing method.

22. Multiple images captured by the imaging unit at different points in time are acquired. Determine whether or not the subject is pictured in each of the acquired multiple images. The position of the subject corresponding to each image in which the subject is determined to be present is derived. If it is determined that the subject is not visible in any of the multiple images taken during a certain period, the location where the subject was during that period is estimated based on first information relating to the subject's position in the first image taken immediately before the period, second information relating to the subject's position in the second image taken immediately after the period, and third information relating to the length of the period. An information processing system equipped with a processing unit.

23. Multiple images captured by the imaging unit at different points in time are acquired. Determine whether or not the subject is pictured in each of the acquired multiple images. The position of the subject corresponding to each image in which the subject is determined to be present is derived. If it is determined that the subject is not captured in any of the multiple images taken during a certain period, then it is determined whether the subject was in a predetermined location during that period based on first information relating to the position of the subject in the first image taken immediately before the period, second information relating to the position of the subject in the second image taken immediately after the period, and third information relating to the length of the period. An information processing system equipped with a processing unit.

Images