Behavior estimation system

Abstract

To provide a behavior estimation system that can improve the accuracy of estimating work and operations of a movable body at a work site.SOLUTION: Provided is a behavior estimation system 1 estimating the behavior of a movable body on the basis of data related to the movable body, the behavior estimation system comprising: a camera 6 that acquires video data obtained by photographing the movable body from a bird's eye view; operation detection devices 7, 8, 9 that are attached to the movable body and detect operation data of the movable body; and an information processing device 10 that analyzes the video data and the operation data. The information processing device 10 has: a first behavior estimation unit 11 that estimates the behavior of the movable body on the basis of the video data; second behavior estimation units 11, 12, 13, 14 that estimate the behavior of the movable body on the basis of the operation data of the movable body detected by the operation detection devices 7, 8, 9; and a behavior recognition unit 16 that integrates results of the estimation made by the first behavior estimation unit 11 and the second behavior estimation units 11, 12, 13, 14 to estimate the behavior of the movable body.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a system for estimating the actions, such as work or movements, being performed by a moving object contained in video data. [Background technology]

[0002] Patent Document 1 discloses an action recognition device designed to automatically measure the time required for a worker's standard work at a workplace. The device in Patent Document 1 breaks down a worker's standard work into component actions based on video footage of the workplace. For example, in the case of a standard work of putting a product on a shelf, the component actions in Patent Document 1 are broken down into steps such as temporarily placing a box containing the target product, finding and retrieving the target product, and putting the target product on the shelf. The device in Patent Document 1 recognizes component actions by detecting change points, i.e., spatiotemporal feature points, in consecutive image frames that make up the video. Specifically, the device recognizes the worker's component actions by detecting the magnitude of spatiotemporal change in multiple blocks obtained by dividing the image frames. The device is then configured to calculate the overall work time for the standard work by measuring the time required for each component action. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-175268 Summary of the Invention [Problem to be solved by the invention]

[0004] According to Patent Document 1, it is possible to automatically measure the time required for a standard task in response to changes in the worker's posture at the workplace. However, the device in Patent Document 1 measures the time required for a standard task using a camera installed at the workplace that captures a bird's-eye view of the worker. In such a video capturing the entire workplace, there is a possibility that the worker may move and overlap with peripheral equipment or other workers, or that the worker may turn their back to the camera. In such cases, it may be impossible to accurately estimate data related to the work the worker is performing or the worker's movements.

[0005] This invention has been made in light of the above technical problems, and aims to provide a behavior estimation system that can improve the accuracy of estimating the work and movements of moving objects at work sites and the like. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the present invention provides a behavior estimation system that estimates the behavior of a moving body based on data related to the moving body, comprising: a camera that acquires video data taken from a bird's-eye view of the moving body; a motion detection device that is attached to the moving body and detects motion data of the moving body; and an information processing device that analyzes the video data and the motion data, wherein the information processing device has a first behavior estimation unit that estimates the behavior of the moving body based on the video data; a second behavior estimation unit that estimates the behavior of the moving body based on the motion data of the moving body detected by the motion detection device; and a behavior recognition unit that integrates the results estimated by the first behavior estimation unit and the second behavior estimation unit and estimates the behavior of the moving body. [Effects of the Invention]

[0007] The behavior estimation system of the present invention includes a camera that captures video data of a moving object captured from a bird's-eye view, as well as a motion detection device attached to the moving object to detect motion data of the moving object. The behavior estimation system is configured to estimate the behavior of the moving object by integrating the behavior of the moving object estimated from the video data with the behavior of the moving object estimated based on the motion data of the moving object detected by the motion detection device. That is, the behavior estimation system estimates the behavior of the moving object based on multiple information sources, the camera and the motion detection device, and therefore can estimate the behavior of the moving object with higher accuracy than when estimating the behavior of the moving object using only a camera capturing a bird's-eye view of the moving object. Furthermore, because the behavior is estimated based on motion data directly detected by the motion detection device attached to the moving object, the behavior of the moving object can be estimated more reliably. Therefore, it is possible to accurately identify areas for improvement in the workplace, such as so-called overburden, waste, and unevenness, which can be utilized to improve work efficiency at the workplace. [Brief explanation of the drawings]

[0008] [Figure 1] 1A and 1B are diagrams for explaining the overall configuration of a behavior estimation system according to an embodiment of the present invention, in which FIG. 1A is an overall view showing the state in which data on a worker working at a work site is being acquired, and FIG. 1B is a front view showing a glove-type device, which is an example of a behavior detection device worn by a worker. [Figure 2] 1 is a block diagram illustrating a functional configuration of a behavior estimation system according to an embodiment of the present invention. [Figure 3] 4 is a flowchart illustrating an example of control executed by the behavior estimation system according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Next, the present invention will be described based on the embodiments shown in the drawings. Note that the embodiments described below are merely examples of specific embodiments of the present invention, and are not intended to limit the present invention.

[0010] In a behavior estimation system 1 according to an embodiment of the present invention, a worker (mobile object) 3 is filmed working or moving at a work site 2 where line work or the like is performed, and the filmed video data is analyzed to analyze the work and behavior of the worker 3. Furthermore, the behavior estimation system 1 according to an embodiment of the present invention analyzes the behavior of the worker 3 by directly detecting the movement of the worker 3. Figure 1(a) schematically shows an example of a work site 2 that is a target of an embodiment of the present invention.

[0011] At a work site 2 shown in FIG. 1(a), a worker 3 performs a predetermined task, such as assembling parts, on a work object (workpiece) 5 that has been transported by a transport facility 4. Although not shown in the figure, the system is configured so that as the work object 5 is transported, the position where the worker 3 works also moves in the same direction as the transport direction of the work object 5. As shown in FIG. 1, the behavior estimation system 1 mainly comprises a fixed camera 6, a wearable camera 7, an inertial measurement device 8, a glove-type device 9, and an information processing device 10.

[0012] As shown in FIG. 1(a), the fixed camera 6 is an imaging device configured to capture an image of a predetermined area in the work site 2 from above. The fixed camera 6 is fixed to, for example, the ceiling or a wall of the work site 2, and is positioned so as to capture an image of the entire work site 2 including the worker 3. The fixed camera 6 may be configured similarly to a conventionally known camera, such as an RGB camera, and is configured to be able to output captured image data and video (image) data. Third-person image data of the worker 3 captured by the fixed camera 6 from a bird's-eye view is output to the information processing device 10. The fixed camera 6 corresponds to the camera in the embodiment of the present invention.

[0013] The wearable camera (head-mounted camera) 7 is an eye-level camera attached to the worker 3, configured to capture images in the direction the worker 3 is facing. As shown in FIG. 1(a), the wearable camera 7 is attached to the worker 3 so as to capture images in the direction the worker 3 is looking. For example, as shown in FIG. 1, the camera may be attached to the ear of the worker 3 with the image capture direction facing forward of the worker 3, or to the temple of the eyeglasses or between the worker 3's eyebrows. Note that the camera may be configured in the same manner as a conventionally known camera, such as an RGB camera, similar to the fixed camera 6. First-person video data from the viewpoint of the worker 3 captured by the wearable camera 7 can be output to the information processing device 10 in the same manner as third-person video data.

[0014] As shown in FIG. 1(a), the inertial measurement device 8 is attached to the body of the worker 3 and detects the movements of the worker 3. The inertial measurement device 8 is, for example, an inertial motion capture (motion sensor) that measures the angular velocity of the worker 3 around three axes using an acceleration sensor and a gyro sensor. The inertial measurement device 8 is attached to the worker's work clothes, etc., and detects the movements of the worker 3 using sensors attached to the shoulders, waist, and leg joints. The inertial measurement device 8 can measure the linear movement, rotation direction, rotation amount, etc. of the worker 3 as physical quantities. The inertial measurement device 8 may also be a nine-axis sensor equipped with a three-axis geomagnetic sensor. In this case, the direction and orientation of the worker 3 can be further measured.

[0015] As shown in FIG. 1( b), the glove-type device 9 is worn on the hand of the worker 3 and detects the hand movements of the worker 3. The glove-type device 9 is a so-called finger motion capture device that detects the hand movements of the worker 3 using sensors built into the fingertips, finger joints, wrist, and other parts. For example, the glove-type device 9 has pressure sensors attached to the fingertips of the thumb, index finger, and middle finger, and an inertial measurement unit (IMU) as described above attached to the back of the hand. Alternatively, multiple pressure sensors are attached from the fingertips to the entire palm. This makes it possible to detect changes in pressure and movements occurring in the palm and fingertips of the worker 3. The glove-type device 9 may also be equipped with a geomagnetic sensor similar to the inertial measurement device 8, a microphone that collects ambient sounds, and the like. The wearable camera 7, the inertial measurement device 8, and the glove-type device 9 correspond to the motion detection device in this embodiment of the present invention. In other words, the wearable camera 7, the inertial measurement device 8, and the glove-type device 9 acquire motion data of the worker 3.

[0016] The information processing device 10 mainly comprises a processor, a communication unit, a storage unit, etc. The information processing device 10 is configured to perform calculations according to a predetermined program using data acquired from an external source and pre-stored data, and to output the results of the calculations as control command signals. For example, the information processing device 10 executes functions that meet a predetermined purpose by having the processor load a program stored on a recording medium into a working area of ​​the storage unit and execute the program, and by performing various controls through the execution of the program.

[0017] The processor is, for example, a CPU or a DSP. This processor is configured to control the information processing device 10 and perform various information processing operations. The main memory unit includes, for example, a RAM and a ROM. As described above, the main memory unit has a work area for the processor to execute programs. The auxiliary memory unit includes, for example, an EPROM or a hard disk drive. This auxiliary memory unit may also include a portable recording medium, i.e., a removable medium. The auxiliary memory unit also freely stores various programs, various data, and various tables in the recording medium by reading or writing them. The auxiliary memory unit may also store an operating system. The communication unit is a wireless communication circuit connected to an external communication device via wireless communication so as to be able to communicate data. This wireless communication circuit performs communication using cellular communication (mobile communication) such as 5G or 4G (LTE).

[0018] Next, a functional configuration of an information processing device 10 according to an embodiment of the present invention will be described. The behavior estimation system 1 configured as described above is configured to acquire data detected regarding the movements and work of the worker 3 and analyze the data to estimate or recognize the behavior of the worker 3. As such a configuration, the information processing device 10 includes, as shown in FIG. 2 , a video data analysis unit 11, an inertial data analysis unit 12, a glove data analysis unit 13, a work procedure manual acquisition unit 14, a data integration unit 15, a behavior recognition unit 16, and an output unit 17.

[0019] Video data analysis unit 11 analyzes image data or video data captured by fixed camera 6 and wearable camera 7. Video data analysis unit 11 acquires first-person video data captured from the viewpoint of worker 3 by wearable camera 7 and third-person video data captured from a bird's-eye view of worker 3 by fixed camera 6. Then, it analyzes each acquired video data. Note that this video data may be configured so that arbitrary video data is input by the user, or may be configured so that it is automatically captured from each camera. Note that video data analysis unit 11 corresponds to the first behavior estimation unit in the embodiment of the present invention.

[0020] The video data analysis unit 11 analyzes the movements of the worker 3 contained in each acquired video data based on learning data, such as the movements of the worker 3 learned in advance by machine learning and feature quantities of objects and devices located at the work site 2. That is, the video data analysis unit 11 detects the worker 3 from the image frames constituting the video data using a conventionally known deep learning method such as YOLO (YOLOv8) or SSD. The detected work or movements of the worker 3 are then analyzed based on a pre-stored learning model. In particular, with first-person video data, it is possible to detect the direction the worker 3 is looking, the speed at which the scenery is changing, or the shaking of the video data. This makes it possible to estimate whether the worker 3 is walking or looking around.

[0021] The inertial data analysis unit 12 analyzes data (motion data) related to the motion of the worker 3 detected by the inertial measurement device 8. That is, the inertial data analysis unit 12 converts the orientation, speed, acceleration, and angular velocity of each part of the worker 3 measured by the inertial measurement device 8 into the motion of the bones and joints of the worker 3, and analyzes the motion and work of the worker 3. In this case, the motion and work of the worker 3 are analyzed based on a learning model that has been machine-learned based on the work content, walking motion, etc. of the worker 3 that have been stored in advance.

[0022] For example, when worker 3 bends down to enter the interior of a vehicle body being assembled, worker 3 bends his knees, lowers his waist, and bends his spine. Physical quantities such as the magnitude, amount of change, and direction of acceleration and angular velocity that occur in each part of the body of worker 3 due to these movements are detected as data. Based on the data obtained in this way and a learning model stored in advance, inertial data analysis unit 12 can estimate that worker 3 is entering the interior of the vehicle body.

[0023] The glove data analysis unit 13 analyzes data related to the hand movements of the worker 3 detected by the glove-type device 9. The glove data analysis unit 13 analyzes physical quantities input to each sensor of the glove-type device 9 when the worker 3 bends his / her fingers, turns his / her palm, grasps an object, etc. In doing so, the glove data analysis unit 13 analyzes the work content of the worker 3 and changes in parameters that occur in each sensor when grasping an object, based on a learning model that has been machine-learned in advance.

[0024] For example, when a worker 3 grips a tool to tighten a fastening member at a predetermined location, the worker 3 performs actions such as gripping the tool tightly and applying force in the direction of tightening the fastening member. These actions detect physical quantities such as the pressure received by the glove-type device 9 of the worker 3, the direction and angle of wrist movement, and the speed and acceleration of hand movement as data. The glove data analysis unit 13 can estimate that the worker 3 is tightening the fastening member at a predetermined location with the tool based on the data thus obtained and a pre-stored learning model. The video data analysis unit 11, the inertial data analysis unit 12, and the glove data analysis unit 13 correspond to a second behavior estimation unit in this embodiment of the present invention.

[0025] The work procedure manual acquisition unit 14 acquires data on a work procedure manual that describes detailed procedures and work content for work at the work site 2. The work procedure manual describes, for example, taking out wrenches and bolts from designated locations, attaching door trim, or temporarily tightening nuts. The work procedure manual also includes information on the positions and locations where these tasks are to be performed. The work procedure manual acquisition unit 14 acquires the work procedure manual so that the correct order and actions of the work performed by the worker 3 can be referenced and compared.

[0026] The data integration unit 15 acquires and integrates the data analyzed by each of the analysis units 11, 12, and 13 as described above. The data integration unit 15 integrates the analysis results of each analysis unit based on a preset algorithm. That is, the estimation results of the worker's behavior from each analysis unit 11, 12, and 13 are input to the data integration unit 15 as the data analysis results of each analysis unit 11, 12, and 13. For example, each analysis unit 11, 12, and 13 estimates the most likely behavior, the next most likely behavior, etc., from among multiple worker behaviors. The data integration unit integrates the data taking into account the worker's behavior estimated to be most likely by each analysis unit 11, 12, and 13. At this time, the data integration unit may be configured so that the degree to which the estimation results are reflected varies depending on each analysis unit 11, 12, and 13.

[0027] The behavior recognition unit 16 estimates or recognizes the behavior of the worker 3 based on the result of integrating the analysis results of the respective analysis units 11, 12, and 13 by the data integration unit 15. The behavior recognition unit 16 estimates that the behavior determined to be most likely as a result of integrating the analysis results of the respective analysis units 11, 12, and 13 by the data integration unit 15 is the behavior of the worker. In other words, for example, it is estimated or recognized that the worker 3 is walking or fastening a fastening member.

[0028] The output unit 17 displays the results estimated by the behavior recognition unit 16. The output unit 17 is, for example, a PC monitor or a display of a mobile terminal. The output unit 17 displays, for example, third-person video data, and also displays the analyzed worker 3 in the third-person video data so that the worker 3 can be identified, and outputs the behavior of the analyzed worker 3 by displaying the behavior of the worker 3 using symbols, text, or the like. That is, the output unit 17 displays the analyzed worker 3 and its behavior in association with each other.

[0029] Next, an example of control executed by the behavior estimation system 1 configured as described above will be described with reference to Fig. 3. The flowchart shown in Fig. 3 illustrates the processing executed when analyzing the behavior of the worker 3 based on multiple pieces of data.

[0030] In step S1, data is acquired. In step S1, data related to the behavior of worker 3 at work site 2 is acquired by fixed camera 6, wearable camera 7, inertial measurement device 8, and glove-type device 9. For example, third-person video data and first-person video data are acquired by fixed camera 6 and wearable camera 7, respectively. Furthermore, data representing the actual movements of worker 3 as physical quantities is acquired by inertial measurement device 8 and glove-type device 9. In this way, data for estimating or recognizing the behavior of worker 3 is acquired in step S1.

[0031] After acquiring the data for estimating the behavior of worker 3, the process proceeds to step S2. In step S2, the various acquired data are analyzed. In step S2, the first-person video data, third-person video data, and physical quantities of the movements of worker 3 are analyzed for each of devices 6, 7, 8, and 9 to estimate the behavior. That is, the results of analyzing the third-person video data acquired from fixed camera 6, the first-person video data acquired from wearable camera 7, the physical quantities acquired from inertial measurement device 8, and the physical quantities acquired from glove-type device 9 are each obtained.

[0032] After the results of analyzing each type of data are obtained in this manner, the process proceeds to step S3. In step S3, the respective analysis results are integrated. In step S2, the results of estimating the worker's behavior are obtained by each of the analysis units 11, 12, and 13. In step S3, these results are integrated by processing them based on a preset algorithm.

[0033] After the analysis results of the various data are integrated, the process proceeds to step S4. In step S4, the work of the worker 3 is estimated or recognized based on the results integrated in step S3. That is, in step S4, the action determined to be the most likely in step S3 is recognized or estimated as the worker's action. That is, in step S4, the most likely movement or work is estimated as the action of the worker 3 based on the results of integrating the data.

[0034] After the behavior of the worker 3 has been estimated or recognized, the process proceeds to step S5. In step S5, the obtained analysis or estimation result of the behavior of the worker 3 is output. That is, in step S5, the analysis result of the behavior of the worker 3 is displayed on the output unit 17 such as a display. At that time, for example, the work content, time, and accuracy of the worker 3 are also displayed so as to correspond to the analyzed worker 3 shown in the third-person video data.

[0035] As described above, in the behavior estimation system 1 according to the embodiment of the present invention, in addition to the fixed camera 6 capturing bird's-eye images of the worker 3, the behavior of the worker 3 is analyzed using a wearable camera 7 attached to the face or shoulder of the worker 3, an inertial measurement device 8 attached to the worker's clothing, such as work clothes, and a glove-type device 9 worn by the worker 3. The behavior of the worker 3 is recognized by integrating the data analyzed by the devices 6, 7, 8, and 9. In this manner, the behavior estimation system 1 is configured to estimate the behavior of the worker 3 using multiple devices. This allows for more accurate estimation of the behavior of the worker 3, as well as more accurate estimation of specific work content, such as fastening work. This allows for accurate identification of areas for improvement at the workplace 2, such as overburdening, waste, and unevenness, which can be utilized to improve work efficiency at the workplace 2. [Explanation of symbols]

[0036] 1. Behavior estimation system 2. Work site 3. Workers 6 Fixed camera (camera) 7. Wearable camera (motion detection device) 8. Inertial measurement devices (motion detection devices) 9. Glove-type device (motion detection device) 10. Information processing equipment 11 Video data analysis unit (first behavior estimation unit) 12 Inertial data analysis unit (second behavior estimation unit) 13 Globe Data Analysis Unit (Second Behavior Estimation Unit) 14. Work Procedure Manual Acquisition Department 15 Data Integration Department 16 Behavior Recognition Department 17 Output section

Claims

[Claim 1] A behavior estimation system that estimates behavior of a moving object based on data related to the moving object, a camera for acquiring video data obtained by photographing the moving object from above; a motion detection device attached to the moving body and detecting motion data of the moving body; an information processing device that analyzes the video data and the motion data, The information processing device includes: a first behavior estimation unit that estimates a behavior of the moving object based on the video data; a second behavior estimation unit that estimates a behavior of the moving object based on the behavior data of the moving object detected by the behavior detection device; a behavior recognition unit that integrates the results of estimation by the first behavior estimation unit and the second behavior estimation unit and estimates the behavior of the moving object. A behavior estimation system characterized by:

Images