Work determination device
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- NATIONAL INSTITUTE OF ADVANCED INDUSTRIAL SCIENCE & TECHNOLOGY
- Filing Date
- 2023-02-13
- Publication Date
- 2026-07-31
AI Technical Summary
【0008】 本発明によれば、人が作業を行ったか否かを判定することができる。
Smart Images
Figure 0007898112000001 
Figure 0007898112000002 
Figure 0007898112000003
Abstract
Description
Technical Field
[0001] The present disclosure relates to a work determination device.
Background Art
[0002] The vehicle-mounted device for a transport vehicle disclosed in Patent Document 1 includes a rotation angle detection unit and a control device. The rotation angle detection unit detects the rotation angle of the head of a person boarding the transport vehicle. The control device evaluates the person from the rotation angle of the person's head.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] When evaluating a person from the rotation angle of the head, it may not be possible to appropriately evaluate the person's behavior. For example, a person performs various tasks, but for tasks where the rotation angle of the head does not change, it may not be possible to determine whether the person has performed the task.
Means for Solving the Problems
[0005] The work determination device for solving the above problems includes an imaging device that images a person and a control device. The control device acquires the position information of the person, acquires image data from the imaging device, determines from the position information whether the person is located in a rule area where the work to be performed by the person is set as a rule, and when the person is located in the rule area, determines from the image data whether the person has performed the work set in the rule area.
[0006] The control device can determine from location information whether a person is located within a designated area. If a person is located within a designated area, the control device determines whether the person performed the task set for that area. This determination is made using image data acquired from an imaging device that captures images of people. By using the image data, it is possible to determine whether a person performed the task.
[0007] Regarding the above-mentioned work determination device, the control device may determine whether or not the person performed the work based on the position of the person's joints in the image data. With respect to the above-mentioned work determination device, the control device may determine that the person has performed the work set in the ruled area if the person performs the work set in the ruled area before the elapsed time since the person entered the ruled area reaches a predetermined time. [Effects of the Invention]
[0008] According to the present invention, it is possible to determine whether or not a person has performed the work. [Brief explanation of the drawing]
[0009] [Figure 1] This is a schematic diagram showing the area where people perform their work. [Figure 2] This is a schematic diagram of the vehicle configuration used in the region shown in Figure 1. [Figure 3] Figure 2 is a flowchart showing the work decision control performed by the control device. [Figure 4] This figure shows an example of image data acquired from the imaging device shown in Figure 2. [Figure 5] This figure shows the area containing people extracted from the image data in Figure 4. [Figure 6] This figure shows an example of the content displayed on the display unit for the implementation and non-implementation indications shown in Figure 3. [Modes for carrying out the invention]
[0010] An embodiment of the work determination device will be described. As shown in Figure 1, the work determination system 10 comprises at least one vehicle 20 and a work determination device 60. The vehicle 20 is operated by a person P1 riding in the vehicle 20. The vehicle 20 is, for example, a forklift.
[0011] <Vehicle> Vehicle 20 will operate in a predetermined area A11. Area A11 may be, for example, a factory, logistics warehouse, port, commercial facility, or public facility.
[0012] Area A11 comprises at least one rule area A1. Rule area A1 is an area where the tasks that person P1 should perform are defined as rules. For example, in rule area A1, pointing and calling is defined as the task that person P1 should perform. Pointing and calling is a safety action in which person P1 points and confirms in a specific direction. Rule area A1 in which pointing and calling is defined as the task that person P1 should perform is set, for example, at an intersection or at a building entrance. An intersection includes one where aisles formed by storage shelves in a logistics warehouse intersect.
[0013] As shown in Figure 2, the vehicle 20 includes a vehicle control device 21. The vehicle control device 21 includes a processor 22 and a storage unit 23. The processor 22 is, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a DSP (Digital Signal Processor). The storage unit 23 includes RAM (Random Access Memory) and ROM (Read Only Memory). The storage unit 23 stores program code or instructions configured to cause the processor 22 to execute processing. The storage unit 23, i.e., the computer-readable medium, includes any available medium that can be accessed by a general-purpose or dedicated computer. The vehicle control device 21 may be composed of hardware circuits such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The processing circuit, the vehicle control device 21, may include one or more processors that operate according to a computer program, one or more hardware circuits such as an ASIC or FPGA, or a combination thereof.
[0014] The vehicle 20 is equipped with a drive motor 24. The drive motor 24 is a motor that propels the vehicle 20. The vehicle 20 moves when driven by the drive motor 24. The vehicle 20 is equipped with a motor driver 25. The motor driver 25 controls the travel motor 24. For example, the motor driver 25 controls the travel motor 24 in response to commands from the vehicle control device 21. This adjusts the speed of the vehicle 20.
[0015] The vehicle 20 is equipped with an auxiliary storage device 32. The auxiliary storage device 32 is a non-volatile storage device that can rewrite data. The auxiliary storage device 32 is, for example, a hard disk drive or a solid-state drive.
[0016] The auxiliary storage device 32 stores the map data MD1. The map data MD1 is data representing the structure of the region A11 in the coordinates of the map coordinate system. The map coordinate system may be a two-dimensional coordinate system or a three-dimensional coordinate system. The map coordinate system is a coordinate system with an arbitrary point in the region A11 used by the vehicle 20 as the origin.
[0017] The vehicle 20 includes a communication device 51. The communication device 51 is a network device equipped with a communication control unit, ports, etc., and can transmit and receive information through a communication network. The vehicle control device 21 estimates the self-position of the vehicle 20. The self-position is the coordinates indicating a point of the vehicle 20 in the map coordinate system. Although a point of the vehicle 20 is arbitrary, for example, the central position of the vehicle 20 in the horizontal direction can be cited.
[0018] The vehicle control device 21 estimates the self-position based on the image data acquired from the imaging device p31 described later. The vehicle control device 2i estimates the self-position by matching the feature points obtained from the image data with the map data MD1.
[0019] The vehicle control device 21 transmits the position information indicating the estimated self-position, the image data, and the ID code of the vehicle 20 from the communication device 51 at a predetermined interval. Since the position of the person P1 boarding the vehicle 20 can be regarded as the same as that of the vehicle 20, the position information indicates the position of the person P1 boarding the vehicle 20 in the region A11. The ID code of the vehicle 20 is individual identification information set for each vehicle 20.
[0020] <Work determination device> The work determination device 60 is a device that determines whether the person P1 has performed work according to the rules set in the regular region A1.
[0021] The work determination device 60 includes an imaging device 31. The imaging device 31 is, for example, a digital camera. The imaging device 31 includes an image sensor. The image sensor is, for example, a CCD image sensor (Charge Coupled Device image sensor) or a CMOS image sensor (Complementary Metal Oxide Semiconductor image sensor). The imaging device 31 performs imaging at a predetermined frame rate to generate image data.
[0022] The imaging device 31 is positioned to capture images of a person P1 riding in the vehicle 20 and the surrounding environment of the vehicle 20. In this embodiment, the imaging device 31 is installed on the vehicle 20. The imaging device 31 may be installed, for example, on the head guard. The imaging device 31 may be installed at a fixed position in area A11 and used to constantly monitor a specific location.
[0023] The work determination device 60 includes a control device 61. The control device 61 has a hardware configuration similar to, for example, the vehicle control device 21. The control device 61 includes a processor 62 and a storage unit 63.
[0024] The work determination device 60 includes a communication device 64. The communication device 64 is similar to, for example, the communication device 51 provided in the vehicle 20. The communication device 64 can acquire information transmitted from the communication device 51. The communication device 64 can transmit information to the communication device 51.
[0025] The work determination device 60 includes a display unit 65. The display unit 65 is, for example, a liquid crystal display or an organic electroluminescent display. The display unit 65 is visible to the user of the work determination device 60.
[0026] The work determination device 60 is equipped with an auxiliary storage device 66. The auxiliary storage device 66 stores semantic map data MD2. Semantic map data MD2 is data that assigns meaning to predetermined locations within the region A11 represented by the map coordinate system. Semantic map data MD2 represents the region A11 in which the vehicle 20 is operated. Semantic map data MD2 is data that sets rules for the work that person P1 should perform in the rule region A1, which is part of region A11. In other words, semantic map data MD2 is data that associates the work that person P1 should perform with the coordinates in the map coordinate system. For example, pointing and calling is set as the work that person P1 should perform at the coordinates corresponding to the rule region A1 in the map coordinate system. Semantic map data MD2 is data that reflects the rules set in the rule region A1 into map data MD1.
[0027] <Work Judgment Control> The work determination control performed by the control device 61 will now be described. The work determination control is performed repeatedly at a predetermined control cycle. The work determination control is performed individually for each ID code. That is, if there are multiple vehicles 20 in area A11, the work determination control is performed for each person P1 riding in a vehicle 20.
[0028] As shown in Figure 3, in step S1, the control device 61 acquires location information from the vehicle 20 via the communication device 64. Next, in step S2, the control device 61 acquires image data from the vehicle 20 via the communication device 64. This image data is the image data at the time when the vehicle's own position, indicated by the position information in step S1, is estimated.
[0029] Next, in step S3, the control device 61 sets a work label. The work label is a label that represents the work performed by person P1 who is in the vehicle 20. The control device 61 determines the work that person P1 is performing from the image data. Then, the control device 61 sets a work label corresponding to the work that person P1 is performing. The method for determining the work may be, for example, a method that uses the joint positions of person P1, or a method that uses machine learning. As an example, a method that uses the joint positions of person P1 to determine the work will be described.
[0030] As shown in Figure 4, the control device 61 extracts the range IM2 in which person P1 is visible from the image data IM1. The extraction of the range IM2 in which person P1 is visible may be performed, for example, by extracting feature quantities from the image data IM1. In this case, the control device 61 extracts the range IM2 in which person P1 is visible that is similar to the feature quantities of person P1. The control device 61 may also extract a predetermined range IM2 in the image data IM1 as the range IM2 in which person P1 is visible. For example, the range IM2 in the image data IM1 in which the driver's seat is visible may be extracted as a predetermined range IM2.
[0031] As shown in Figure 5, the control device 61 obtains the position of the joints J1 of person P1 in the image data IM1. The control device 61 performs skeletal estimation for the range IM2. Through skeletal estimation, a model can be obtained in which the joints J1 of person P1, such as the neck, shoulders, elbows, wrists, waist, hips, knees, and ankles, are connected by linear links. The control device 61 obtains the position of the joints J1 of person P1 in the image data IM1 from the model. The position of the joints J1 of person P1 in the image data IM1 is indicated by the X-axis coordinate representing the position of the horizontal pixel and the Y-axis coordinate representing the position of the vertical pixel.
[0032] The control device 61 determines the task being performed by person P1 from the position of the joint J1 of person P1 in the image data IM1. In the case of pointing and calling, the wrist joint J1 is positioned at a predetermined height by pointing and confirming. A range of heights in which the wrist joint J1 is positioned during pointing and calling may be set in advance, and if the wrist joint J1 is positioned within this range, it may be determined that pointing and calling is being performed. For example, the range can be set in the Y-axis coordinates of the image data. Then, if the wrist joint J1 is positioned within this range, the control device 61 may determine that pointing and calling is being performed.
[0033] The work performed by person P1 may include assuming a reverse posture. If the vehicle 20 is a forklift, the posture of person P1 when reversing may be specified in the operation. For example, when traveling between different factories or on a designated passage where cargo handling operations are not normally performed (i.e., a passage where goods are not loaded or unloaded), the operation may specify that the person should reverse while gripping the pillar and looking behind. The control device 61 may determine that the person is assuming a reverse posture if, in a route traveling between different factories or on a designated passage where cargo handling operations are not normally performed, the wrist joint J1 is in a position corresponding to the pillar. The work performed by person P1 may also include checking left and right.
[0034] When the control device 61 identifies the task performed by person P1, it sets a task label corresponding to that task. As shown in Figure 3, in step S4, the control device 61 determines whether or not person P1 is located in the rule area A1. The control device 61 determines that person P1 is located in the rule area A1 if its own position, represented by the position information, is included in the rule area A1. If the determination result in step S4 is affirmative, the control device 61 performs the process in step S5. If the determination result in step S4 is negative, the control device 61 terminates the work determination control.
[0035] In step S5, the control device 61 determines whether or not person P1 performed the task set as a rule in rule area A1. As an example, we will explain the case in which the control device determines whether or not pointing and calling was performed in rule area A1.
[0036] If the control device 61 determines in step S4 that person P1 is located in rule area A1, it recognizes the task set in rule area A1 from the semantic map data MD2. The control device 61 recognizes that pointing and calling is set as the task that person P1 should perform in rule area A1. The control device 61 determines whether the task set in rule area A1 where person P1 is located matches the task represented by the task label. If the task label represents pointing and calling, the control device 61 determines that the task set in rule area A1 has been performed. If person P1 performs the task set in rule area A1 before the elapsed time since person P1 entered rule area A1 reaches a predetermined time, the control device 61 determines that person P1 has performed the task. In this embodiment, if the task represented by the task label becomes pointing and calling before the elapsed time since person P1 entered rule area A1 reaches a predetermined time, the control device 61 affirms the determination result in step S5. If the work indicated by the work label does not become a pointing and calling operation before the predetermined time elapsed since person P1 entered the rule area A1, the control device 61 makes the determination result of step S5 negative. The predetermined time can be set arbitrarily. The predetermined time is the time required for person P1 to perform the work set in the rule area A1 when person P1 enters the rule area A1. The predetermined time is, for example, 3 seconds. The predetermined time may be set to a different time depending on the work set in the rule area A1. Since the work label is set according to the image data IM1, the control device 61 determines from the image data IM1 whether or not person P1 has performed the work set in the rule area A1. If the determination result of step S5 is positive, the control device 61 performs the process of step S6. If the determination result of step S5 is negative, the control device 61 performs the process of step S7.
[0037] In step S6, the control device 61 displays an execution indicator. The execution indicator is a display that notifies the user of the work determination device 60 that person P1 is performing the work set in the rule area A1. For example, as shown in Figure 6, the display unit 65 may display a map representing the rule area A1 and display the trajectory T1 of person P1 on the map. Then, an execution mark 71 may be superimposed on the trajectory T1. In the example shown in Figure 6, the rule area A1 is displayed, but the display unit 65 may display the entire area A11. If there are multiple vehicles 20 in area A11, the display unit 65 may display information corresponding to the person P1 riding in each of the multiple vehicles 20.
[0038] In step S7, the control device 61 displays an "unexecuted" indication. The "unexecuted" indication notifies the user of the work determination device 60 that person P1 has not yet performed the work set in the rule area A1. For example, as shown in Figure 6, a mark 72 indicating "unexecuted" may be superimposed on the trajectory T1. The mark 72 indicating "unexecuted" can be any mark, as long as it is different from the mark 71 indicating "executed".
[0039] [Operation of this embodiment] The control device 61 acquires location information. From the acquired location information, the control device 61 can determine whether or not person P1 is located in the ruled area A1. If person P1 is located in the ruled area A1, the control device 61 can recognize the task set in the ruled area A1 from the semantic map data MD2. The control device 61 determines whether or not person P1 has performed the task set in the ruled area A1. This determination is made using image data IM1 acquired from the imaging device 31 that images person P1. By using image data IM1, it is easier to capture changes in person P1. For example, if one tries to detect the rotation angle of a part of person P1 using a rotation angle detection unit, the rotation angle can only be detected for the part where the rotation angle detection unit is provided. In contrast, the imaging device 31 can capture changes in the part of person P1 that is included in the imaging range.
[0040] [Effects of this embodiment] (1) The control device 61 can determine whether or not person P1 has performed the work by using the image data IM1.
[0041] (2) The control device 61 determines whether or not person P1 has performed the task set in the rule area A1 based on the position of person P1's joints J1 in the image data IM1. Tasks set in the rule area A1 include tasks in which person P1 performs specific actions such as pointing and calling out. When a specific action is performed, the position of person P1's joints J1 changes. Therefore, it is possible to determine whether or not person P1 has performed the task set in the rule area A1 based on the position of joints J1.
[0042] (3) The control device 61 determines that person P1 has performed the task set in the rule area A1 if person P1 performs the task before the elapsed time since person P1 entered the rule area A1 reaches a predetermined time. It may take some time for person P1 to perform the task after entering the rule area A1. Setting a predetermined time prevents the determination of whether or not person P1 has performed the task from becoming excessively strict.
[0043] (4) The control device 61 displays on the display unit 65 whether or not person P1 has performed the task set in the rule area A1. This allows the user of the task determination device 60 to evaluate whether or not person P1 has performed the task set in the rule area A1.
[0044] [Example of changes] The embodiment can be implemented with the following modifications. The embodiment and the following modifications can be combined with each other to the extent that they do not contradict each other technically.
[0045] ○The control device 61 may issue a warning to person P1 if person P1 fails to perform the task set in rule area A1. For example, the control device 61 may transmit a command from the communication device 64 to activate a buzzer or to display a warning on a display unit visible to person P1. The buzzer may be located on the vehicle 20 or in area A11. The display unit visible to person P1 may be located on the vehicle 20 or in area A11. When the control device 61 transmits a command, the display unit 65 may or may not display anything.
[0046] ○The rule area A1 may include an area where checking left and right is set as a task that person P1 should perform. In this case, the rule area A1 is set, for example, at an intersection. Checking left and right is the task of person P1 visually checking left and right.
[0047] ○The rule area A1 may include an area where a pause is set as a task that person P1 should perform. ○The vehicle control device 21 may estimate its own position by comparing the measurement results of the distance measuring sensor with the map data MD1. The distance measuring sensor is, for example, a laser rangefinder, a millimeter-wave radar, or a stereo camera. Estimation of the own position can be achieved by a well-known method of comparing the shape of the surrounding environment estimated from the distance measuring sensor with the shape of the map data MD1.
[0048] ○The vehicle 20 may be equipped with a work determination device. In this case, the work determination control may be performed by the vehicle control device 21. The auxiliary storage device 32 stores semantic map data MD2 in place of or in addition to map data MD1. In this case, the vehicle control device 21 is the control device.
[0049] ○The control device 61 may recognize the task that person P1 should perform from the markers provided in area A11. In this case, the markers are positioned so that they are captured by the imaging device 31 when the vehicle 20 is located in the ruled area A1. The markers are associated with the task that person P1 should perform. The control device 61 recognizes the task that person P1 should perform from the image data IM1. The control device 61 then determines whether the task recognized from the markers matches the task represented by the task label. Since the markers are positioned so that they are captured by the imaging device 31 when the vehicle 20 is located in the ruled area A1, the markers themselves are position information. The markers are, for example, AR (Augmented Reality) markers.
[0050] ○The work determination device 60 may determine whether or not a person P1 who is not riding in the vehicle 20 has performed the work set in the rule area A1. In this case, for example, an imaging device 31 is provided on the helmet worn by person P1 so that person P1 can be imaged by the imaging device 31. Then, a communication device 51 is provided to transmit the image data IM1 obtained by imaging with the imaging device 31 to the work determination device 60. The communication device 51 may be unitized with the imaging device 31. The communication device 51 may be provided separately from the imaging device 31.
[0051] ○The vehicle 20 may be equipped with separate imaging devices for capturing images of person P1 and imaging devices for estimating its own position. ○The work performed by person P1, as determined in step S3, may include performing a forward-moving posture. If vehicle 20 is a forklift, the posture of person P1 when moving forward may be specified in the operation. [Explanation of Symbols]
[0052] A1...Regular area, IM1...Image data, J1...Joint, P1...Person, 31...Imaging device, 60...Work determination device, 61...Control device.
Claims
1. An imaging device for imaging a person riding in a vehicle, Control device and The system includes a memory device that stores semantic map data, which represents a ruled area, which is part of the area in which the vehicle is operated, using coordinates in a map coordinate system, and which sets the tasks that the person should perform in the ruled area as rules. The vehicle estimates its own position in the map coordinate system, The control device is The vehicle acquires the self-position it estimated as the location information of the person, Image data is acquired from the aforementioned imaging device, The location information is used to determine whether the person is located in the specified area. A work determination device that, when the person is located in the ruled area, determines from the image data whether the person has performed the work set in the ruled area.
2. The work determination device according to claim 1, wherein the control device determines whether or not the person performed the work based on the position of the person's joints in the image data.
3. The work determination device according to claim 1 or 2, wherein the control device determines that the person has performed the work set in the ruled area if the time elapsed since the person entered the ruled area reaches a predetermined time.