Action recognition device and action recognition method

Abstract

To provide an action recognition device that enables more appropriate work management in production sites. [Solution] The behavior recognition device according to the present invention is a behavior recognition device that recognizes the behavior of a worker in a workspace for producing an object, and comprises: an authentication unit that authenticates the worker in the workspace based on skeletal information of the worker stored in advance; and a voice recognition unit that authenticates the worker's voice, recognizes the start and end of work in a predetermined process related to the object to be produced based on the worker's voice, and obtains the start time and end time of the work.

Description

Technical Field

[0001] The present invention relates to an action recognition device and an action recognition method.

Background Art

[0002] In recent years, technologies for authenticating users of devices have become widespread. Patent Document 1 describes technologies for using physical characteristics such as skeletons and muscles and dynamic characteristics such as walking styles for user authentication, and technologies for a device that logs out when there is no use by the user for a predetermined time.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technology disclosed in Patent Document 1, by using object detection technology and skeleton authentication technology to identify the actions of workers, it is possible to obtain the start time and end time of a predetermined operation in a mass production process. However, this is limited to the case of a mass production process where the actions of workers in a predetermined process are generally uniform. Therefore, even in the same process, in the case of one-by-one production where the movements of workers are diverse, it is desired to have a technology that can obtain actual values regarding working hours and can manage the production site so that only specific workers can perform operations from the perspective of ensuring safety.

[0005] An object of the present invention is to provide an action recognition device that can more appropriately perform work management in a production site.

Means for Solving the Problems

[0006] The behavior recognition device according to the present invention is a behavior recognition device that recognizes the behavior of a worker in a workspace for producing an object, and comprises: an authentication unit that authenticates the worker in the workspace based on skeletal information of the worker stored in advance; and a voice recognition unit that performs voice authentication of the worker, recognizes the start and end of work in a predetermined process related to the object to be produced based on the worker's voice, and obtains the start time and end time of the work. [Effects of the Invention]

[0007] The present invention provides an action recognition device that enables more appropriate work management in production sites. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows a schematic diagram of the workspace related to the behavior recognition device according to one embodiment of the present invention. [Figure 2] This figure shows a worker wearing a wearable device related to an action recognition device according to one embodiment of the present invention. [Figure 3] This is a block diagram showing the functional configuration of an action recognition device according to one embodiment of the present invention. [Figure 4] This figure shows an example of skeletal information in an action recognition device according to one embodiment of the present invention. [Figure 5] This figure shows the hardware configuration of an action recognition device according to one embodiment of the present invention. [Figure 6] This is a flowchart illustrating an action recognition method related to one embodiment of the present invention. [Modes for carrying out the invention]

[0009] The embodiments will be described below with reference to the drawings. Note that the drawings are simplified, and the technical scope of the embodiments should not be narrowly interpreted based on their depiction. Furthermore, the same elements are denoted by the same reference numerals, and redundant explanations are omitted. Also, in the following embodiments, when referring to the number of elements (including quantity, numerical value, amount, range, etc.), unless specifically stated or clearly limited to a particular number in principle, the number is not limited to that particular number and may be greater than or less than that number.

[0010] Furthermore, in the following embodiments, the components are not necessarily essential unless otherwise explicitly stated or considered to be clearly essential in principle. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc., of the components, etc., it shall include those that substantially approximate or resemble their shape, etc., unless otherwise explicitly stated or considered to be clearly not essential in principle. The same applies to the numbers, etc. (including the number of items, numerical values, quantities, ranges, etc.).

[0011] <The process leading to the development of the behavior recognition device 10 according to this embodiment> By using authentication technologies such as object detection and skeletal recognition, it is possible to recognize the actions of workers in a workplace such as a factory where products are manufactured, and to obtain the start and end times of predetermined tasks in predetermined processes.

[0012] In the case of one-of-a-kind production, where the start and end actions of a task vary each time, even within the same process, it is difficult to pinpoint and obtain the start and end times of a specific task performed by that worker.

[0013] Furthermore, in the target workspace, unspecified individuals can enter and exit, and may come into contact with parts and products related to the production process, making it difficult to guarantee safety during work. Therefore, it is desirable that production management equipment be equipped with authentication technology to ensure that only specific workers are allowed to perform tasks in the workspace, thereby guaranteeing safety during work.

[0014] From the above, in the production management scenario, there are issues such as being able to obtain actual values regarding working hours even when performing one - product - one - style production, and ensuring work safety. In order to balance these issues and achieve work leveling and optimization, a technology that automatically and accurately obtains the actual values of work with respect to the planned working hours without causing production hassle is desired.

[0015] Therefore, the action recognition device 10 according to this embodiment has been found. Hereinafter, the action recognition device 10 according to an embodiment of the present invention will be described.

[0016] [Embodiment] FIG. 1 is a diagram showing an overview of the work space 1 related to the action recognition device 10 according to an embodiment of the present invention. In the work space 1, it is assumed that the vehicle 2 is produced in the production area 3 which is a predetermined area. Further, during a predetermined process related to the vehicle 2, it is assumed that the cart 4 is towed by the towing vehicle 5 and parked at the storage area 6. In addition, an imaging device 7 is provided in the work space 1.

[0017] The work space 1 is, for example, a factory or the like where a production line of production objects such as the vehicle 2 is installed. The vehicle 2 is an example of a production object. The production object is not limited to the vehicle 2 and may be various products produced in the work space 1. Further, the cart 4 and the towing vehicle 5 are examples of transport vehicles.

[0018] The cart 4 is used, for example, to transport parts and the like necessary for the production of the vehicle 2. The transport vehicle may include a plurality of carts 4 and may not include the towing vehicle 5. Also, the cart 4 may be parked at the storage area 6 by the operator P driving the towing vehicle 5, or may be parked by the operator P carrying it without using the towing vehicle 5. The operator P who performs work in the work space 1 wears the wearable device 20.

[0019] In addition, the vehicle 2 is equipped with an RFID (Radio Frequency Identification) tag T1, and the cart 4 is equipped with an RFID tag T2. Note that the RFID tag T2 may be mounted on the cart 4 and the towing vehicle 5, or if there are multiple carts 4, it may be mounted on each cart 4. Also, an RFID reader R1 is embedded in the production area 3, and an RFID reader R2 is embedded in the storage yard 6.

[0020] The RFID tags T1 and T2 are an example of a tag device. The tag device is not limited to the RFID tags T1 and T2, and may be, for example, an NFC tag or the like. The RFID tags T1 and T2 are devices that store data to be read by other devices via wireless communication. The RFID tags T1 and T2 include an IC chip containing a memory circuit, a logic circuit, etc., and an antenna element connected to the IC chip.

[0021] The RFID readers R1 and R2 are an example of a reading device. The reading device is not limited to the RFID readers R1 and R2, and may be, for example, an NFC reader or the like. The RFID reader R1 and the RFID reader R2 each include an antenna element and perform short-range wireless communication with the RFID tag T1 and the RFID tag T2, respectively.

[0022] When the vehicle 2 stops in the production area 3, the RFID reader R1 detects the RFID tag T1 mounted on the vehicle 2 and reads the information stored in the RFID tag T1. Then, the RFID reader R1 transmits a signal indicating that the vehicle 2 has been detected to the behavior recognition device 10.

[0023] When the cart 4 stops in the storage yard 6, the RFID reader R2 detects the RFID tag T2 mounted on the cart 4 and reads the information stored in the RFID tag T2. Then, the RFID reader R2 transmits a signal indicating that the cart 4 has been detected to the behavior recognition device 10.

[0024] The imaging device 7 captures images of the worker P's movements, the vehicle 2, the trolley 4, etc., in the workspace 1, and outputs the captured images to the action recognition device 10. If there are multiple workers P in the workspace 1, the movements may become complex, such as the workers P overlapping with the imaging device 7, so multiple imaging devices 7 may be provided. The imaging device 7 is equipped with image detection elements such as a CCD (Charge Coupled device) and a CMOS (Complementary Metal Oxide Semiconductor).

[0025] Figure 2 shows a worker P wearing a wearable device 20 related to an action recognition device 10 according to one embodiment of the present invention. The wearable device 20 is a device formed in the shape of eyeglasses and is called smart glasses or AR glasses. The wearable device 20 comprises temples 21, which are straight frames that rest on the ears, and rims 22, which are frames that surround the eyes and are shaped to rest on the nose.

[0026] The display unit 23 displays an image within the field of view of the worker P wearing the wearable device 20. The display unit 23 is, for example, an organic EL display or liquid crystal display having a display area 24 located inside the rim 22, and displays an image on part or all of this display area 24.

[0027] The miniature camera 25 is a camera fixed to the display unit 23 and captures images of the area around the display unit 23. The miniature camera 25 is fixed facing forward, for example, near the front end of the temple 21, and captures an area similar to the field of view of the worker P. The action recognition device 10 extracts the vehicle 2 and the trolley 4 that appear in the images captured by the miniature camera 25.

[0028] The microphone 26 may be mounted at any suitable position on the wearable device 20. Preferably, the microphone 26 is configured to focus on the voice produced by the worker P.

[0029] The speaker 27 may be mounted at any suitable position on the wearable device 20. The speaker 27 may be a conventional audio speaker, a bone conduction speaker, or a transducer. If the speaker 27 is a bone conduction speaker, it can convert the output signal into vibrations that can be transmitted to the user's bone structure.

[0030] The device controller 28 controls the operation of the wearable device 20 in response to instructions from the behavior recognition device 10. The device controller 28 is, for example, a computer having a processor and memory. The device controller 28 transmits images acquired by the miniature camera 25 to the behavior recognition device 10 as needed.

[0031] The configuration of the wearable device 20 is not limited to the above, and the shape of the wearable device 20 is not limited to the shape shown. The wearable device 20 may include, for example, a pupil position sensor (not shown).

[0032] Figure 3 is a block diagram showing the functional configuration of an action recognition device 10 according to one embodiment of the present invention. The action recognition device 10 comprises a communication unit 11, a detection unit 12, an image processing unit 13, an authentication unit 14, a voice recognition unit 15, and the detection unit 12. The action recognition device 10 is a device that recognizes the actions of worker P in the workspace 1.

[0033] The communication unit 11 includes a communication interface with the imaging device 7, RFID readers R1 and R2, and wearable device 20 in the workspace 1. This communication interface supports, for example, mobile communication standards such as 4G (4th Generation) or 5G (5th Generation), wired LAN (Local Area Network) standards, or wireless LAN standards, but is not limited to these and may support any communication standard.

[0034] The detection unit 12 detects that the vehicle 2, which is the object to be manufactured, is located in the production area 3, which is a predetermined area within the work space 1. The detection unit 12 detects the vehicle 2 based on the signal from the RFID reader R1, which is a reader that detects the RFID tag T1, which is a tag device mounted on the vehicle 2.

[0035] Furthermore, the detection unit 12 detects that the trolley 4 or towing vehicle 5 is present in the storage area 6. The detection unit 12 detects the trolley 4 or towing vehicle 5 based on the signal from the RFID reader R2, which detects the RFID tag T2 mounted on the trolley 4 or towing vehicle 5. In addition, the detection unit 12 detects the vehicle 2, trolley 4, or towing vehicle 5 using a wearable device 20 worn by the worker P.

[0036] The image processing unit 13 processes images of worker P acquired from the imaging device 7, which captures the movements of worker P in the workspace 1. More specifically, the image processing unit 13 extracts images of worker P acquired from the imaging device 7 and generates skeletal information of worker P. The generated skeletal information of worker P is used for skeletal authentication in the authentication unit 14.

[0037] The authentication unit 14 authenticates worker P in the workspace 1 based on the skeletal information of worker P that has been stored in advance. More specifically, the authentication unit 14 performs skeletal authentication by comparing the skeletal information of worker P generated by the image processing unit 13 with the skeletal information of worker P that has been stored in advance. If the skeletal information of worker P matches the skeletal information of worker P that has been stored in advance, the authentication unit 14 authenticates worker P as an authorized worker. The authentication unit 14 may also use a specific unit 122 to calculate the degree of agreement between the skeletal information of worker P and the skeletal information of worker P that has been stored in advance, and perform authentication based on a predetermined degree of agreement.

[0038] The skeletal information of worker P, which has been stored in advance, is stored by a storage means (not shown). The storage means may consist of the ROM (Read Only Memory) or RAM (Random Access Memory) of the action recognition device 10, or it may be a storage means of another device.

[0039] Figure 4 shows an example of skeletal information in an action recognition device 10 according to one embodiment of the present invention. The image processing unit 13 acquires an image including the worker P captured by the imaging device 7 and obtains skeletal information of the worker P by analyzing it using three-dimensional skeletal estimation technology or the like. The image processing unit 13 may generate skeletal information including bones or joints for each part of the worker P's face or body.

[0040] The authentication unit 14 determines a reference coordinate axis such that the center of the waist of worker P, identified from the worker P's skeletal information, becomes the origin. It may then authenticate whether or not worker P is authorized to perform the work by comparing this with the skeletal information of worker P that has been stored in advance.

[0041] In Figure 3, the voice recognition unit 15 performs voice authentication of worker P and recognizes the start and end of work in a predetermined process related to the vehicle 2, which is the object of production, based on the voice of worker P. More specifically, the voice recognition unit 15 performs voice recognition indicating "work start" and "work end" based on the voice acquired from the microphone 26 of the wearable device 20 worn by worker P. The voice recognition unit 15 can acquire the work start time and work end time by worker P.

[0042] Furthermore, the voice recognition unit 15 performs voice authentication of the worker P after the detection unit 12 detects the vehicle 2, which is the object to be produced. Therefore, it is possible to suppress erroneous processing, such as obtaining the start time of work by recognizing, for example, a voice statement such as "start of work" before the start of a predetermined task.

[0043] The notification unit 16 causes the wearable device 20 worn by worker P to notify worker P whether or not the work is permitted. More specifically, if the authentication unit 14 authenticates that worker P is authorized to perform the work, the notification unit 16 causes the speaker 27 of the wearable device 20 to notify worker P that the work is permitted. If worker P is not authorized to perform the work, the notification unit 16 causes the speaker 27 of the wearable device 20 to notify worker P that the work is not permitted.

[0044] Figure 5 shows the hardware configuration of an action recognition device 10 according to one embodiment of the present invention. As shown in the figure, the action recognition device 10 includes a CPU 101, ROM 102, RAM 103, HDD (Hard Disk Drive) 104, HDD (Hard Disk Drive) controller 105, display 106, external device connection I / F (Interface) 107, network I / F 108, bus line 109, keyboard 110, pointing device 111, optical drive 113, and media I / F 115.

[0045] Of these components, the CPU 101 controls the overall operation of the action recognition device 10. The ROM 102 stores programs used to drive the CPU 101, such as the IPL (Initial Program Loader). The RAM 103 is used as the work area for the CPU 101. The HDD 104 stores various data, such as programs.

[0046] The HDD controller 105 controls the reading or writing of various data to the HDD 104 according to the control of the CPU 101. The display 106 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 107 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory or printers.

[0047] The network interface 108 is an interface for data communication using a network. The bus line 109 is an address bus, data bus, etc., for electrically connecting various components such as the CPU 101.

[0048] Furthermore, the keyboard 110 is a type of input means equipped with multiple keys used for inputting characters, numbers, or various instructions. The pointing device 111 is a type of input means used for selecting and executing various instructions, selecting processing targets, moving the cursor, and so on.

[0049] The optical drive 113 controls the reading or writing of various types of data to the DVD-RW 112, which is an example of a removable recording medium. Note that the optical drive 113 is not limited to DVD-RW; it may also be DVD-R or other types. The media interface 115 controls the reading or writing (storage) of data to the recording medium 114, such as flash memory.

[0050] Figure 6 is a diagram showing the flow of the behavior recognition method performed by the behavior recognition device 10 according to the first embodiment of the present invention.

[0051] When vehicle 2 stops in production area 3, if RFID reader R1 detects the RFID tag T1 mounted on vehicle 2 (YES in S101), it executes the process in step S102. If RFID reader R1 does not detect the RFID tag T1 mounted on vehicle 2 (NO in S101), it waits until it detects the RFID tag T1 mounted on vehicle 2.

[0052] When the trolley 4 stops at the storage area 6, if the RFID reader R2 detects the RFID tag T2 mounted on the trolley 4 (YES in S102), it executes the process in step S103. If the RFID reader R2 does not detect the RFID tag T2 mounted on the trolley 4 (NO in S102), it waits until it detects the RFID tag T2 mounted on the trolley 4.

[0053] If the authentication unit 14 performs skeletal authentication based on the skeletal information of worker P in the workspace 1 (YES in S103), it executes the process in step S104. If the authentication unit 14 cannot perform skeletal authentication because worker P is not present in the workspace 1 (NO in S103), it waits until it can detect worker P in the workspace 1.

[0054] If the authentication unit 14 authenticates that worker P is authorized to perform the work in a predetermined process (YES in S104), the notification unit 16 causes the speaker 27 of the wearable device 20 to notify worker P that the work is permitted (S105). If the notification unit 16 finds that worker P is not authorized to perform the work (NO in S104), the notification unit 16 causes the speaker 27 of the wearable device 20 to notify worker P that the work is not permitted (S106).

[0055] If the miniature camera 25 of the wearable device 20 worn by worker P photographs the vehicle 2, which is the object of production, and the detection unit 12 detects the vehicle 2 (YES in S107), the process in step S108 is executed. If the detection unit 12 does not detect the vehicle 2 (NO in S107), the system waits until the vehicle 2 is detected.

[0056] If the voice recognition unit 15 recognizes a voice message indicating "start work" based on the voice acquired from the worker P's wearable device 20 (YES in S108), it acquires the work start time set by worker P (S109) and terminates the process.

[0057] If the speech recognition unit 15 does not recognize the voice indicating "start of work" (NO in S108) but recognizes the voice indicating "end of work" (YES in S110), it obtains the end of work time (S111) and terminates the process. If the speech recognition unit 15 does not recognize the voice indicating "end of work" (NO in S110), it returns to the process in step S108.

[0058] These steps carry out a behavior recognition method according to one aspect of the present invention. However, the behavior recognition method according to one aspect of the present invention may include other steps as appropriate, depending on the measurement conditions, measurement space, etc.

[0059] <Effects of the behavior recognition device 10 according to this embodiment> In this embodiment, the behavior recognition device 10 uses a voice recognition unit 15 to recognize the start and end of work in a predetermined process related to the product being manufactured based on the voice of the worker P. Therefore, even in the case of one-of-a-kind production where the worker P's movements vary, such as the start and end actions of the work being done differently each time, the device can identify and acquire the start and end times of the predetermined work performed by the worker P.

[0060] Furthermore, in this embodiment, the behavior recognition device 10 is authenticated by the authentication unit 14 through skeletal authentication to determine that worker P is authorized to perform a predetermined task. When the authentication unit 14 authenticates that worker P is authorized to perform the task, the notification unit 16 notifies the wearable device 20 that worker P is authorized to perform the task. As a result, it is possible to prevent persons other than the experienced worker P from coming into contact with the production object, thereby ensuring the quality of the production object and the safety of the work.

[0061] Based on the above, the behavior recognition device 10 according to this embodiment can acquire actual values ​​regarding work time even when producing one item at a time in production management situations, and can also ensure work safety. Furthermore, in order to level out and optimize work, it can automatically acquire actual work values ​​in relation to the planned work time, without adding to the production burden.

[0062] Therefore, according to the behavior recognition device 10 of this embodiment, work management can be performed more appropriately in the work space 1, which is the production site.

[0063] Although the present invention has been described above in accordance with the embodiments described above, the present invention is not limited to the configuration of the embodiments described above, and of course includes various modifications, alterations, and combinations that can be made by a person skilled in the art within the scope of the claims of the present patent application. [Explanation of symbols]

[0064] 1. Workspace 2 vehicles 3 Production Area 4 carts 5. Towing Vehicles 6 Storage area 7. Imaging device 10 Behavior recognition device 11 Communications Department 12 Detection unit 13 Image Processing Unit 14. Authentication Department 15. Voice Recognition Unit 16 Hochi Department 20 Wearable Devices P worker R1, R2 RFID Reader T1, T2 RFID tags

Claims

1. An action recognition device that recognizes the actions of workers in a workspace where an object to be manufactured is produced, An authentication unit that authenticates the worker in the workspace based on the worker's skeletal information stored in advance, A voice recognition unit performs voice authentication of the worker, recognizes the start and end of work in a predetermined process related to the production object based on the worker's voice, and obtains the start and end times of the work. An action recognition device equipped with the following features.

2. An image processing unit generates skeletal information of the worker based on an image of the worker obtained from an imaging device that captures the worker's movements in the aforementioned workspace. A notification unit that notifies the worker whether or not to perform the said work via a wearable device worn by the worker, Furthermore, The authentication unit performs skeletal authentication by comparing the skeletal information of the worker generated by the image processing unit with the skeletal information of the worker stored in advance. The notification unit, when the authentication unit authenticates that the worker is authorized to perform the work, causes the wearable device to notify the worker that the work is permitted. The behavior recognition device according to claim 1.

3. The system further includes a detection unit that detects whether the object to be produced is present in a predetermined area within the workspace. The voice recognition unit performs voice authentication of the worker after the detection unit has detected the object to be produced. The behavior recognition device according to claim 1.

4. The detection unit detects the production target based on the signal from a reader that detects a tag device mounted on the production target, and also detects the production target using a wearable device worn by the worker. The behavior recognition device according to claim 3.

5. An action recognition method performed by an action recognition device that recognizes the actions of workers in a workspace where an object to be produced is manufactured, A step of authenticating the worker in the workspace based on the worker's skeletal information stored in advance, The steps include: performing voice authentication of the worker, recognizing the start and end of work in a predetermined process related to the production object based on the worker's voice, and obtaining the start and end times of the work; A method of recognizing behavior, including the following.

Images