Behavior recognition device and behavior recognition method

Abstract

The present invention provides a behavior recognition device capable of more appropriately performing work management on a production site. This behavior recognition device is a behavior recognition device for recognizing the behavior of a worker in a work space in which a production object is produced, and is provided with: an authentication unit for authenticating the worker in the work space on the basis of pre-stored skeleton information of the worker; and a voice recognition unit that performs voice authentication of the worker, recognizes the start and end of a work in a predetermined process related to the production object on the basis of the voice of the worker, and acquires the start time and end time of the work.

Description

Technical Field

[0001] This invention relates to a behavior recognition device and a behavior recognition method. Background Technology

[0002] In recent years, technologies for authenticating users of devices have become widespread. Patent Document 1 describes a technology that uses physical characteristics such as bones and muscles or dynamic characteristics such as walking style for user authentication, as well as a technology related to a device that cancels the device if the user does not use it within a specified time.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2017-058808 Summary of the Invention

[0004] In the technology disclosed in Patent Document 1, the start and end times of specified tasks in mass production processes are obtained by using object detection technology and skeletal recognition technology to determine the actions of operators. However, this is limited to situations where the actions of operators in specified processes are largely the same. Therefore, a technology is desired that can obtain actual values ​​related to task time even in the same process, especially in the case of single-piece, small-batch production where there is a variety of operator actions, and manage the production site from the perspective of ensuring safety, so that only specific operators can perform the tasks.

[0005] The purpose of this invention is to provide a behavior recognition device that enables more appropriate operation management on the production site.

[0006] The behavior recognition device involved in this invention is a behavior recognition device for recognizing the behavior of workers in the work space of a production object. It includes: an authentication unit that authenticates the workers in the work space based on pre-stored skeletal information of the workers; and a voice recognition unit that performs voice authentication of the workers, and based on the workers' voice, identifies the start and end of a specified process related to the production object, and obtains the start time and end time of the work.

[0007] Invention Effects

[0008] This invention provides a behavior recognition device that enables more appropriate operation management on the production site. Attached Figure Description

[0009] Figure 1 This is a schematic diagram showing the working space associated with a behavior recognition device according to an embodiment of the present invention.

[0010] Figure 2This diagram illustrates the state of an operator wearing a wearable device associated with a behavior recognition device according to an embodiment of the present invention.

[0011] Figure 3 This is a block diagram illustrating the functional configuration of a behavior recognition device according to an embodiment of the present invention.

[0012] Figure 4 This is a diagram illustrating an example of skeletal information in a behavior recognition device according to an embodiment of the present invention.

[0013] Figure 5 This is a diagram illustrating the hardware configuration of a behavior recognition device according to an embodiment of the present invention.

[0014] Figure 6 This is a flowchart illustrating a behavior recognition method according to an embodiment of the present invention. Detailed Implementation

[0015] Hereinafter, embodiments will be described with reference to the accompanying drawings. However, since the drawings are simplified, the technical scope of the embodiments should not be narrowly interpreted based on the description in the drawings. Furthermore, the same symbols are used to denote the same elements, and repeated descriptions are omitted. Also, in the following embodiments, when referring to the quantity of elements (including number, value, quantity, range, etc.), the quantity is not limited to a specific number, except in particularly explicit cases or cases where the quantity is clearly limited in principle; it may be more than a specific number or less.

[0016] Furthermore, in the following embodiments, the constituent elements are not necessarily essential, except in cases where they are particularly clear or where they are obviously considered necessary in principle. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc., of constituent elements, etc., it includes those that are substantially similar or analogous to their shape, etc., except in cases where they are particularly clear or where they are obviously not considered necessary in principle. This also applies to the quantities (including number, value, quantity, range, etc.) mentioned above.

[0017] <Discussion process regarding the behavior recognition device 10 involved in the proposed implementation>

[0018] It can use authentication technologies such as object detection and skeleton authentication to identify the behavior of workers in the workspace of factories and other production areas, and obtain the start and end times of specified operations in specified processes.

[0019] Even in the same process, in the case of small-batch production where the start or end actions of the work are different each time and the actions of the operators are varied, it is difficult to determine and obtain the start and end times of the specified work performed by the operator.

[0020] Furthermore, since unspecified individuals can enter and exit the workspace, it is difficult to ensure the safety of operations involving parts related to production processes and contact with production objects. Therefore, in order to ensure the safety of operations within the workspace, production management-related equipment preferably possesses certification technology that allows only specific operators to perform the work.

[0021] As can be seen from the above, in production management scenarios, there are challenges such as obtaining actual values ​​related to operation time and ensuring operation safety, even in single-piece and small-batch production. To address these challenges and achieve standardization and optimization of operations, a technology is desired that can accurately and automatically obtain actual operation values ​​relative to planned operation times without increasing production workload.

[0022] Therefore, the behavior recognition device 10 according to this embodiment was discovered. Hereinafter, a behavior recognition device 10 according to an embodiment of the present invention will be described.

[0023] [Implementation Method]

[0024] Figure 1 This is a schematic diagram showing a workspace 1 related to the behavior recognition device 10 according to an embodiment of the present invention. In the workspace 1, a vehicle 2 is produced in a production area 3, which is a designated area. Furthermore, in a designated process related to the vehicle 2, a trolley 4 is towed by a traction vehicle 5 and parked in a parking area 6. A camera device 7 is also provided in the workspace 1.

[0025] Workspace 1 is, for example, a factory or similar facility with a production line containing production objects such as vehicles 2. Vehicle 2 is one example of a production object. However, the production object is not limited to vehicle 2; it can also be any product produced in workspace 1. Furthermore, trolley 4 and traction vehicle 5 are examples of transport vehicles.

[0026] The trolley 4 is used, for example, to transport components required for the production of vehicle 2. The transport vehicle may include multiple trolleys 4, or it may not include a tractor vehicle 5. Furthermore, the trolley 4 can be parked in the parking area 6 by an operator P driving the tractor vehicle 5, or it can be parked by an operator P moving it without using the tractor vehicle 5. The operator P, who works in the workspace 1, wears wearable equipment 20.

[0027] Furthermore, vehicle 2 is equipped with a Radio Frequency Identification (RFID) tag T1, and trolley 4 is equipped with an RFID tag T2. Additionally, the RFID tag T2 can be mounted on both trolley 4 and the towing vehicle 5, or on each of the multiple trolleys 4. Furthermore, an RFID reader R1 is embedded in the production area 3, and an RFID reader R2 is embedded in the parking area 6.

[0028] RFID tags T1 and T2 are examples of tag devices. Tag devices are not limited to RFID tags T1 and T2; for example, they could also be NFC tags. RFID tags T1 and T2 are devices for storing data that can be read by other devices via wireless communication. RFID tags T1 and T2 include an IC chip including memory circuits and logic circuits, and an antenna element connected to the IC chip.

[0029] RFID readers R1 and R2 are examples of reading devices. However, reading devices are not limited to RFID readers R1 and R2; for example, they could also be NFC readers.

[0030] RFID readers R1 and R2 are equipped with antenna elements and communicate wirelessly with RFID tags T1 and T2 at short range, respectively.

[0031] When vehicle 2 is parked in production area 3, RFID reader R1 detects the RFID tag T1 mounted on vehicle 2 and reads the information stored in the RFID tag T1. Furthermore, RFID reader R1 sends a signal indicating that vehicle 2 has been detected to the behavior recognition device 10.

[0032] When the trolley 4 stops in the parking area 6, the RFID reader R2 detects the RFID tag T2 mounted on the trolley 4 and reads the information stored in the RFID tag T2. Furthermore, the RFID reader R2 sends a signal indicating that the trolley 4 has been detected to the behavior recognition device 10.

[0033] The camera device 7 captures the movements of the worker P, vehicle 2, and trolley 4 in the workspace 1, and outputs the captured images to the behavior recognition device 10. When multiple workers P are present in the workspace 1, their movements become complex due to overlap with each other relative to the camera device 7; therefore, multiple camera devices 7 can be installed. The camera device 7 incorporates image detection elements such as a charge-coupled device (CCD) and a complementary metal-oxide-semiconductor (CMOS).

[0034] Figure 2 This diagram illustrates a worker P wearing a wearable device 20 associated with a behavior recognition device 10 according to an embodiment of the present invention. The wearable device 20 is a device formed in the shape of eyeglasses and is referred to as smart glasses or AR glasses. The wearable device 20 includes a straight frame, i.e., temples 21, for hanging on the ears, and a frame, i.e., a frame 22, that surrounds the eyes and is formed in the shape of hanging on the nose.

[0035] Display 23 displays an image in the field of vision of the worker P wearing the wearable device 20. Display 23 is, for example, an organic EL display or liquid crystal display having a display area 24 disposed inside the frame 22, in which an image is reflected in part or all of the display area 24.

[0036] The small camera 25 is a camera fixed relative to the display 23, and it is used to capture images around the display 23. For example, the small camera 25 is fixed forward near the front end of the temple 21, capturing an area similar to the field of vision of the operator P. The behavior recognition device 10 extracts images from the images captured by the small camera 25 and projects them onto the vehicle 2 and the trolley 4.

[0037] The microphone 26 can be installed in any suitable location within the wearable device 20. The microphone 26 is preferably configured to focus on the voice produced by the operator P.

[0038] The speaker 27 can be installed in any suitable location within the wearable device 20. The speaker 27 can be a conventional audio speaker, a bone conduction speaker, or a transducer. In the case where the speaker 27 is a bone conduction speaker, the speaker 27 is able to convert the output signal into vibrations that can be transmitted to the user's bone structure.

[0039] The device controller 28 controls the operation of the wearable device 20 according to instructions from the behavior recognition device 10. The device controller 28 is, for example, a computer with a processor and memory. The device controller 28 sends images acquired by the small camera 25 to the behavior recognition device 10 at any time.

[0040] Furthermore, the configuration of the wearable device 20 is not limited to the above, and the shape of the wearable device 20 is not limited to the shape shown in the figure. For example, the wearable device 20 may include a pupil position sensor (not shown).

[0041] Figure 3 This is a block diagram illustrating the functional configuration of a behavior recognition device 10 according to an embodiment of the present invention. The behavior recognition device 10 includes a communication unit 11, a detection unit 12, an image processing unit 13, an authentication unit 14, a voice recognition unit 15, and a detection unit 16. The behavior recognition device 10 is a device for recognizing the behavior of a worker P in a workspace 1.

[0042] The communication unit 11 includes communication interfaces with the camera device 7, RFID readers R1 and R2, and wearable device 20 in the workspace 1. These communication interfaces may correspond to mobile communication standards such as 4G or 5G, wired local area network (LAN) standards, or wireless LAN standards, but are not limited to these and may correspond to any communication standard.

[0043] The inspection unit 12 detects that the vehicle 2, which is the object of production, exists in a designated area, namely the production area 3, within the work space 1. The inspection unit 12 detects the vehicle 2 based on the signal from the RFID reader R1, which is a reading device for the tag device, namely the RFID tag T1, mounted on the vehicle 2.

[0044] Furthermore, the detection unit 12 detects the presence of the trolley 4 or the towing vehicle 5 in the parking area 6. The detection unit 12 detects the trolley 4 or the towing vehicle 5 based on the signal from the RFID reader R2, which detects the RFID tag T2 mounted on the trolley 4 or the towing vehicle 5. Moreover, the detection unit 12 detects the vehicle 2, the trolley 4, or the towing vehicle 5 using a wearable device 20 worn by the operator P.

[0045] The image processing unit 13 processes the image of worker P acquired by the camera device 7, which captures the movements of worker P in the workspace 1. More specifically, the image processing unit 13 extracts the image of worker P acquired by the camera device 7 and generates skeletal information of worker P. The generated skeletal information of worker P is used for skeletal authentication in the authentication unit 14.

[0046] The authentication unit 14 authenticates worker P in the workspace 1 based on pre-stored skeletal information of worker P. More specifically, the authentication unit 14 performs skeletal authentication by comparing the skeletal information of worker P generated by the image processing unit 13 with the pre-stored skeletal information of worker P. If the skeletal information of worker P matches the pre-stored skeletal information of worker P, the authentication unit 14 authenticates worker P as an authorized worker to perform the work. The authentication unit 14 may use the determination unit 122 to calculate the degree of consistency between the skeletal information of worker P and the pre-stored skeletal information of worker P, and perform authentication based on the prescribed degree of consistency.

[0047] Additionally, the pre-stored skeletal information of the worker P is stored by a storage component not shown. The storage component may be a read-only memory (ROM) or random access memory (RAM) of the behavior recognition device 10, or it may be a storage component of other devices.

[0048] Figure 4 This diagram illustrates an example of skeletal information in a behavior recognition device 10 according to an embodiment of the present invention. The image processing unit 13 acquires an image of the worker P captured by the camera device 7, and obtains the skeletal information of the worker P through analysis using three-dimensional skeletal estimation techniques, etc. The image processing unit 13 can generate skeletal information, including bones or joints, for the worker P's face or various parts of his body.

[0049] The authentication unit 14 determines a reference coordinate axis, for example, based on the skeletal information of worker P, so that the center of worker P's waist, as determined by the skeletal information, becomes the origin. Furthermore, it can authenticate whether worker P is an authorized worker by comparing it with pre-stored skeletal information of worker P.

[0050] exist Figure 3 In this process, the voice recognition unit 15 performs voice authentication on the operator P, and identifies the start and end of the work in the prescribed process related to the vehicle 2, which is the production target, based on the voice of the operator P. More specifically, the voice recognition unit 15 performs voice recognition of the intention to "start work" and the intention to "end work" based on the voice acquired from the microphone 26 of the wearable device 20 worn by the operator P. The voice recognition unit 15 is able to obtain the start time and end time of the work performed by the operator P.

[0051] Furthermore, the voice recognition unit 15 performs voice authentication of the operator P after the detection unit 12 detects the vehicle 2, which is the production target. Therefore, for example, it is possible to suppress errors such as obtaining the start time of the work by using voice recognition to specify the intention of "starting work" before the start of the work.

[0052] The notification unit 16 informs the worker P via the wearable device 20 whether or not the worker is permitted to perform the work. More specifically, if the authentication unit 14 has authenticated worker P as an authorized worker, the notification unit 16 uses the speaker 27 of the wearable device 20 to inform worker P that the work is permitted. Furthermore, if worker P is not an authorized worker, the notification unit 16 uses the speaker 27 of the wearable device 20 to inform worker P that the work is not permitted.

[0053] Figure 5 This diagram illustrates the hardware configuration of a behavior recognition device 10 according to an embodiment of the present invention. As shown, the behavior recognition device 10 includes a CPU 101, a ROM 102, a RAM 103, a Hard Disk Drive (HDD) 104, a Hard Disk Drive (HDD) controller 105, a display 106, an External Device Interface (I / F) 107, a Network I / F 108, a bus 109, a keyboard 110, a pointing device 111, an optical drive 113, and a Media I / F 115.

[0054] Of these, CPU 101 controls the overall operation of behavior recognition device 10. ROM 102 stores programs used in the CPU 101 driver, such as the Initial Program Loader (IPL). RAM 103 serves as the working area for CPU 101. HDD 104 stores various data, including programs.

[0055] HDD controller 105 controls the reading and writing of various data on HDD 104 according to the control of CPU 101. Display 106 displays various information such as cursor, menu, window, characters, or images. External device connection I / F 107 is an interface for connecting various external devices. These external devices may include, for example, Universal Serial Bus (USB) memory or printers.

[0056] Network I / F 108 is an interface used for data communication via a network. Bus 109 is an address bus or data bus used for electrically connecting various components such as CPU 101.

[0057] Furthermore, the keyboard 110 is an input method equipped with multiple keys for inputting characters, numbers, or various instructions. The pointing device 111 is an input method for selecting and executing various instructions, selecting processing objects, moving the cursor, etc.

[0058] Optical drive 113 controls the reading or writing of various data on DVD-RW 112, which is an example of a removable recording medium. Furthermore, optical drive 113 is not limited to DVD-RW, but can also be DVD-R, etc. Media I / F 115 controls the reading or writing (storage) of data on recording medium 114 such as flash memory.

[0059] Figure 6 This is a flowchart illustrating the behavior recognition method performed by the behavior recognition device 10 according to the first embodiment of the present invention.

[0060] When RFID reader R1 detects RFID tag T1 mounted on vehicle 2 while vehicle 2 is parked in production area 3 (S101 "Yes"), it executes step S102. If RFID tag T1 mounted on vehicle 2 is not detected (S101 "No"), RFID reader R1 waits until RFID tag T1 mounted on vehicle 2 is detected.

[0061] When RFID reader R2 detects RFID tag T2 mounted on trolley 4 while trolley 4 is parked in parking area 6 (S102 "Yes"), it executes step S103. If RFID tag T2 mounted on trolley 4 is not detected (S102 "No"), RFID reader R2 waits until RFID tag T2 mounted on trolley 4 is detected.

[0062] If skeletal authentication is performed based on the skeletal information of worker P within workspace 1 (S103 "Yes"), then authentication unit 14 executes step S104. If skeletal authentication cannot be performed because worker P is not present in workspace 1 (S103 "No"), authentication unit 14 waits until worker P within workspace 1 can be detected.

[0063] If the certification department 14 verifies that the operator P is a pre-approved operator to perform the work in the specified process (S104 "Yes"), the notification department 16 uses the speaker 27 of the wearable device 20 to inform the operator P that the work is permitted (S105). Furthermore, if the operator P is not an approved operator to perform the work (S104 "No"), the notification department 16 uses the speaker 27 of the wearable device 20 to inform the operator P that the work is not permitted (S106).

[0064] If the small camera 25 of the wearable device 20 worn by the operator P takes a picture of the vehicle 2, which is the production target, and the detection unit 12 detects the vehicle 2 (S107 "Yes"), the processing in step S108 is executed. If the detection unit 12 does not detect the vehicle 2 (S107 "No"), it waits until the vehicle 2 is detected.

[0065] If the voice indicating "start work" is received from the wearable device 20 of the operator P ("Yes" in S108), the voice recognition unit 15 receives the start time of the work performed by the operator P (S109) and ends the processing.

[0066] If the voice indicating "start work" is not recognized ("No" in S108) but the voice indicating "end work" is recognized ("Yes" in S110), the voice recognition unit 15 obtains the work end time (S111) and ends the process. If the voice indicating "end work" is not recognized ("No" in S110), the voice recognition unit 15 returns to the process in step S108.

[0067] These steps implement the behavior recognition method according to one aspect of the present invention. However, the behavior recognition method according to one aspect of the present invention may appropriately include other steps depending on the measurement conditions or measurement space, etc.

[0068] <Effects of the behavior recognition device 10 according to this embodiment>

[0069] In the behavior recognition device 10 of this embodiment, the voice recognition unit 15 identifies the start and end of a specified operation in a production process related to the production object based on the voice of the operator P. Therefore, even in the case of small-batch production where the start or end actions of the operation are different each time in the same process, it is possible to determine and obtain the start and end times of the specified operation performed by the operator P.

[0070] Furthermore, in the behavior recognition device 10 of this embodiment, the authentication unit 14 authenticates the worker P as an authorized operator to perform the prescribed work through skeletal recognition. Moreover, when the authentication unit 14 authenticates worker P as an authorized operator, the notification unit 16 instructs the wearable device 20 to inform worker P of the permission to perform the work. Therefore, it is possible to prevent personnel other than the experienced worker P from contacting the production object in advance, thereby ensuring the quality of the production object and the safety of the operation.

[0071] As can be seen from the above, the behavior recognition device 10 according to this embodiment can, in the context of production management, simultaneously acquire actual values ​​related to work time and ensure work safety, even in the case of single-piece, small-batch production. Furthermore, in order to achieve work standardization and optimization, it can accurately and automatically acquire actual work values ​​relative to the planned work time without increasing the workload of production.

[0072] Therefore, the behavior recognition device 10 according to this embodiment can more appropriately manage operations in the workspace 1, which is a production site.

[0073] The present invention has been described above according to the above embodiments, but the present invention is not limited to the above embodiments. Of course, it includes various modifications, alterations and combinations that can be made by those skilled in the art within the scope of the claims of this application.

[0074] Symbol Explanation

[0075] 1-Workspace, 2-Vehicle, 3-Production area, 4-Trolley, 5-Towing vehicle, 6-Parking area, 7-Camera device, 10-Behavior recognition device, 11-Communication department, 12-Detection department, 13-Image processing department, 14-Authentication department, 15-Voice recognition department, 16-Notification department, 20-Wearable device, P-Operator, R1, R2-RFID reader, T1, T2-RFID tag.

Claims

1. A behavior recognition device, which is used to recognize the behavior of workers in the workspace of a production object, characterized in that, have: The authentication department, based on pre-stored skeletal information of the workers, authenticates the workers within the workspace; and The voice recognition unit performs voice authentication of the operator, and based on the operator's voice, identifies the start and end of the operation in the specified process related to the production object, and obtains the start time and end time of the operation.

2. The behavior recognition device according to claim 1, characterized in that, It also has: An image processing unit generates skeletal information of a worker based on an image of the worker acquired from a camera device that captures the worker's movements within the workspace; and The notification department uses a wearable device worn by the worker to inform them whether or not permission is granted to the worker to perform the task. The authentication unit performs skeletal authentication by comparing the skeletal information of the worker generated by the image processing unit with pre-stored skeletal information of the worker. If the certification department verifies that the operator is authorized to perform the operation, the notification department instructs the wearable device to inform the operator of the intention to allow the operation to be performed.

3. The behavior recognition device according to claim 1, characterized in that, It also has: The inspection department detects whether the production object exists within a designated area of ​​the workspace. After the detection unit detects the production object, the voice recognition unit performs voice authentication of the operator.

4. The behavior recognition device according to claim 3, characterized in that, The detection unit detects the production object based on signals from a reading device that detects a tag mounted on the production object, and also detects the production object through a wearable device worn by the operator.

5. A behavior recognition method, which is a behavior recognition method executed by a behavior recognition device, said behavior recognition device being used to recognize the behavior of workers in the workspace of a production object, the behavior recognition method being characterized by comprising the following steps: Based on the pre-stored skeletal information of the workers, the workers within the work space are authenticated; and The operator's voice is authenticated, and based on the operator's voice, the start and end of the operation in the specified process related to the production object are identified, and the start and end times of the operation are obtained.

Images