Determination device

Abstract

To obtain a determination device which can improve accuracy of determination of a direction of the face of a driver wearing a helmet.SOLUTION: A determination device comprises: an acquisition unit which acquires a pickup image in which a helmet attached to the head of a driver of a vehicle is picked up; a determination unit which determines a direction of the face of the driver on the basis of the image of the helmet in the pickup image; and an output processing unit which outputs a determination result of the determination unit.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a determination device. [Background technology]

[0002] Conventionally, images of forklift drivers have been analyzed to detect dangerous driving by the drivers, etc. For example, based on the direction of the driver's face in the image, it is analyzed whether the forklift driver backed up without checking behind the driver. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-176510 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, forklift drivers usually wear helmets, and if the brim of the helmet hides the driver's face, the direction of the driver's face may not be determined. Patent Document 1 discloses a technology for recognizing the driver's line of sight using an eye camera attached in front of the forklift driver, but does not mention what to do if the face is hidden by the helmet.

[0005] The present invention has been made in consideration of the above, and one of its objects is to provide a determination device that can improve the accuracy of determining the direction of the face of a driver wearing a helmet. [Means for solving the problem]

[0006] The determination device of the present invention includes an acquisition unit that acquires an image of a helmet worn on the head of a vehicle driver, a determination unit that determines the direction of the driver's face based on the image of the helmet in the captured image, and an output processing unit that outputs the determination result of the determination unit. Effect of the Invention

[0007] According to the determination device of the present invention, it is possible to improve the accuracy of determining the direction of the face of a driver wearing a helmet. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a system configuration diagram showing an example of a determination system according to an embodiment. [Diagram 2] FIG. 2 is a side view illustrating an example of a forklift according to an embodiment. [Diagram 3] FIG. 3 is a block diagram showing an example of a hardware configuration of the imaging system according to the embodiment. [Figure 4] FIG. 4 is a block diagram illustrating an example of a hardware configuration of the determination device according to the embodiment. [Diagram 5] FIG. 5 is a block diagram illustrating an example of a functional configuration of the determination device according to the embodiment. [Figure 6] FIG. 6 is a diagram showing an example of a captured image according to the embodiment. [Figure 7] FIG. 7 is a diagram showing an example of a captured image according to the embodiment. [Figure 8] FIG. 8 is a diagram showing an example of a captured image according to the embodiment. [Figure 9] FIG. 9 is a diagram showing an example of a captured image according to the embodiment. [Figure 10] FIG. 10 is a diagram for explaining a direction range according to the embodiment. [Figure 11] FIG. 11 is a diagram for explaining the second determination process of the determination device according to the embodiment. [Figure 12]FIG. 12 is a diagram showing an example of a determination result screen displayed on the display device according to the embodiment. [Figure 13] FIG. 13 is a flowchart illustrating an example of a process executed by the determination device according to the embodiment. [Figure 14] FIG. 14 is a diagram for explaining a determination process of a determination device according to a modified example of the embodiment. [Figure 15] FIG. 15 is a diagram for explaining a determination process of a determination device according to a modified example of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment will be described with reference to the drawings. In this specification, components according to the embodiment and descriptions of the components may be described in a plurality of ways. The components and their descriptions are merely examples and are not limited by the expressions in this specification. The components may be identified by names different from those in this specification. The components may also be described by expressions different from those in this specification.

[0010] <Judgment System 1> Fig. 1 is a system configuration diagram showing an example of a determination system 1 according to this embodiment. The determination system 1 shown in Fig. 1 determines the direction of the face of a driver of a forklift 2. The determination system 1 includes an imaging system 10 and a determination device 20.

[0011] The imaging system 10 is mounted on the forklift 2. The determination device 20 is installed in a management facility or the like. The imaging system 10 and the determination device 20 are connected to each other so that they can communicate with each other via a network NW (wide area network) such as the Internet. The determination device 20 may be mounted on the forklift 2, and the determination device 20 and the imaging system 10 may be connected to each other so that they can communicate with each other via a communication cable, for example.

[0012] FIG. 2 is a side view showing an example of a forklift 2 according to an embodiment. As shown in FIG. 2, the forklift 2 has a vehicle body 2a, wheels 2b, etc. The vehicle body 2a is provided with a driver's seat 2c and a handlebar 2d. A driver 3 of the forklift 2 sits in the driver's seat 2c and drives the forklift 2. When driving the forklift 2, the driver 3 wears a helmet 4 on his / her head 3a. The helmet 4 has a body 4a that covers an upper part of the head 3a of the driver 3, and a brim 4b. The brim 4b protrudes forward in the front-rear direction of the helmet 4 from a front part of the body 4a in the front-rear direction of the helmet 4. The forklift 2 is an example of a vehicle. Note that the vehicle is not limited to a forklift.

[0013] The forklift 2 is also provided with two imaging devices 104, 105. The two imaging devices 104, 105 are fixed to the vehicle body 2a. The imaging device 104 is disposed diagonally upward in the front-rear direction of the forklift 2 with respect to the driver's seat 2c, and can image the driver 3 seated in the driver's seat 2c and the helmet 4 worn by the driver 3. The imaging device 105 is disposed diagonally upward in the front-rear direction of the forklift 2 with respect to the driver's seat 2c, and can image the front in the front-rear direction of the forklift 2. Here, the arrow D1 in FIG. 2 indicates the front in the front-rear direction of the forklift 2, and the arrow D2 in FIG. 2 indicates the rear in the front-rear direction of the forklift 2. In the following, unless otherwise specified, the front-rear direction (front, rear), left-right direction (left, right), and up-down direction (up, down) are the front-rear direction, left-right direction, and up-down direction of the forklift 2. The left-right direction is also called the lateral direction. The imaging device 104 is configured as an area camera (area sensor) capable of outputting a color two-dimensional image (captured image) by capturing an image of a subject. The imaging devices 104 and 105 are capable of capturing at least one of a still image and a moving image, for example. In this embodiment, the imaging devices 104 and 105 capture moving images. The two imaging devices 104 and 105 are included in the imaging system 10.

[0014] The images captured by the two imaging devices 104, 105 are transmitted to the determination device 20. The determination device 20 (FIG. 1) determines the orientation of the face 3b of the driver 3 based on at least the captured image obtained by imaging by the imaging device 104.

[0015] <Imaging system 10> Next, the imaging system 10 will be described in detail.

[0016] Fig. 3 is a block diagram showing an example of a hardware configuration of an imaging system 10 according to an embodiment. As shown in Fig. 3, the imaging system 10 includes a control device 101, a communication device 102, a storage device 103, and imaging devices 104 and 105. The imaging system 10 functions as, for example, a drive recorder.

[0017] The control device 101 includes a CPU 111, a ROM 112, a RAM 113, etc. The CPU 111 executes a program stored in the ROM 112, for example, to perform various controls of the imaging system 10 and various calculations.

[0018] The communication device 102 is an element that realizes a communication function for communicating with an external device. In the imaging system 10 of this embodiment, the communication device 102 establishes communication with the determination device 20.

[0019] The storage device 103 is configured with a HDD (Hard Disk Drive) etc. The storage device 103 stores various types of information and data.

[0020] <Judgment device 20> Next, the determination device 20 will be described.

[0021] 4 is a block diagram showing an example of a hardware configuration of the determination device 20 according to this embodiment. As shown in FIG. 4, the determination device 20 includes a control device 201, a communication device 202, a storage device 203, an operation device 204, and a display device 205.

[0022] The control device 201 includes a CPU 211, a ROM 212, a RAM 213, etc. The CPU 211 executes a program stored in the ROM 212, for example, to perform various controls of the determination device 20 and various calculations.

[0023] The communication device 202 is an element that realizes a communication function for communicating with an external device. In the determination device 20 of this embodiment, the communication device 202 establishes communication with the imaging system 10.

[0024] The storage device 203 is configured with a HDD (Hard Disk Drive) or the like. The storage device 203 stores various information and data. For example, the storage device 203 stores an imaging information storage unit 220. The imaging information storage unit 220 stores imaging information. The imaging information includes a captured image captured by the imaging device 104 and related information related to the captured image. The related information includes, for example, a determination result by the determination device 20.

[0025] The operation device 204 is composed of, for example, a keyboard, a mouse, a touch panel switch formed on the display screen of the display device 205, and the like.

[0026] The display device 205 is configured, for example, by a liquid crystal display.

[0027] Fig. 5 is a block diagram showing an example of a functional configuration of the control device 201 of the determination device 20 according to this embodiment. As shown in Fig. 5, the control device 201 has, as functional units, an acquisition unit 221, a determination unit 222, and an output processing unit 224. Details of these functional units will be described later.

[0028] The program executed by the determination device 20 of this embodiment is provided in a state that it is pre-installed in the ROM 212 or the like.

[0029] The program executed by the determination device 20 of this embodiment may be configured to be provided by being recorded in an installable or executable format on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), or a USB memory.

[0030] Furthermore, the program executed by the determination device 20 of this embodiment may be configured to be stored on a computer, such as the determination device 20, connected to a network such as the Internet and provided by being downloaded via the network. Also, the program executed by the determination device 20 of this embodiment may be configured to be provided or distributed via a network such as the Internet.

[0031] The program executed by the judgment device 20 of this embodiment has a modular structure including the above-mentioned units (acquisition unit 221, judgment unit 222, output processing unit 224), and in terms of actual hardware, the CPU 211 reads out the program from the ROM 212 and executes it, thereby loading the above-mentioned units onto a main storage device such as RAM 213, and the acquisition unit 221, judgment unit 222, and output processing unit 224 are generated on the main storage device such as RAM 213.

[0032] The functions of the acquisition unit 221, the determination unit 222, and the output processing unit 224 may be realized by hardware.

[0033] The acquisition unit 221 acquires captured images from the imaging devices 104 and 105. The captured images captured by the imaging device 104 include, for example, a captured image of the helmet 4 worn on the head 3a of the driver 3 of the forklift 2. The captured images captured by the imaging device 105 include a captured image of the surroundings of the forklift 2 in front of the forklift 2.

[0034] 6 to 9 show examples of captured images G obtained by imaging by the imaging device 104. Each captured image G includes a driver image 3G which is an image of the driver 3, and a helmet image 4G which is an image of the helmet 4. The driver image 3G includes a head image 3aG which is an image of the head 3a, and a face image 3bG which is an image of the face 3b. The helmet image 4G includes a main body image 4aG which is an image of the main body 4a, and a brim image 4bG which is an image of the brim 4b.

[0035] The captured image G1 in Fig. 6 is an image in which the facial image 3bG faces forward. The captured image in Fig. 7 is an image in which the facial image 3bG faces to the right (sideways). The captured image G3 in Fig. 8 is an image in which the facial image 3bG faces backward. The captured image G4 in Fig. 9 is an image in which the facial image 3bG faces downward.

[0036] The determination unit 222 determines, based on the captured image G, whether or not the driver 3 checked backward when the forklift 2 moved backward. Specifically, the determination unit 222 determines, based on the captured image G, whether or not the face 3b of the driver 3 turned backward when the forklift 2 moved backward.

[0037] The determination unit 222 determines whether the forklift 2 is moving backwards, for example, based on the captured image G obtained by the imaging device 104 or the captured image obtained by the imaging device 105. At this time, it can determine whether the forklift 2 is moving backwards, for example, based on the difference in the background in the captured image G. In other words, it is possible to know whether the forklift 2 is moving backwards by a known method such as optical flow.

[0038] Furthermore, when determining whether or not the driver 3 has checked backward when the forklift 2 is moving backward, the determination unit 222 determines the orientation of the face 3b of the driver 3 based on the captured image G. For example, as a first determination process, the determination unit 222 determines the orientation of the face 3b of the driver 3 based on the helmet image 4G in the captured image G. Furthermore, as a second determination process, the determination unit 222 determines the orientation of the face 3b of the driver 3 based on the helmet image 4G in the captured image G and feature points of the driver image 3G in the captured image G.

[0039] Fig. 10 is a diagram for explaining the direction range according to the embodiment. As shown in Fig. 10, the direction of the face 3b is, for example, classified into a plurality of (eight, for example) ranges H (hereinafter also referred to as direction ranges H) in a direction perpendicular to the up-down direction of the forklift 2. The eight direction ranges H are partitioned at equal intervals around an axis along the up-down direction of the forklift 2. The eight direction ranges H are a front range H1 including the front and having the front as the center of the range, a rear range H2 including the rear and having the rear as the center of the range, a left range H3 including the left and having the left as the center of the range, a right range H4 including the right and having the right as the center of the range, a left front range H5 including the left front and having the left front as the center of the range, a right front range H6 including the right front and having the right front as the center of the range, a left rear range H7 including the left rear and having the left rear as the center of the range, and a right rear range H8 including the right rear and having the right rear as the center of the range.

[0040] For example, a trained model is used in the first determination process. The trained model has been trained to output the orientation of the helmet 4 when a captured image G including a helmet image 4G is input. The orientation of the helmet 4 is output in, for example, one of eight direction ranges H. The input data during learning of the trained model is the captured image G including the helmet image 4G. At this time, the input data may be the entire captured image G, or may be only the region R of the captured image G that surrounds the helmet image 4G. Furthermore, the teacher data during learning is the orientation (direction range) of the helmet 4. The first determination process is not limited to the above. For example, the first determination process may perform image analysis on the captured image G to extract the helmet image 4G and determine the orientation of the helmet image 4G. Furthermore, the first determination process may determine the orientation of the helmet image 4G by matching the captured image G with a reference captured image.

[0041] The determination unit 222 determines the orientation of the face 3b by assuming that the orientation of the face 3b is the same as the orientation of the helmet image 4G.

[0042] The determination unit 222 determines that the driver 3 has turned backward to check the rear when the orientation of the helmet image 4G, i.e., the orientation of the face 3b, is in the rear range H2, the left rear range H7, or the right rear range H8. The determination unit 222 determines that the driver 3 has not turned backward to check the rear when the orientation of the helmet image 4G, i.e., the orientation of the face 3b, is in the forward range H1, the left front range H5, or the right front range H6.

[0043] The judgment unit 222 performs a second judgment process when it is not possible to accurately estimate (judge) whether the driver 3 is turning around by only the first judgment process, such as when the reliability score output from the model when estimating (judgement) the orientation of the helmet image 4G, i.e., the orientation of the face 3b, is low, or when the orientation of the helmet image 4G, i.e., the orientation of the face 3b, is in the left range H3 or the right range H4.

[0044] In the second determination process, feature point information, which is information on feature points of the helmet image 4G in the captured image G and the driver image 3G in the captured image G, is used. The feature point is, for example, a skeleton point J. The skeleton point J is a feature point that indicates the position of each part of a person, and includes, for example, the upper and lower ends of the face, the center of the ears, and joints (shoulders, waist, groin, elbows, wrists, etc.). The setting of the skeleton points can be realized, for example, by existing technology (OpenPose, etc.). Hereinafter, the left shoulder skeleton point J is referred to as the left shoulder skeleton point J1, the right shoulder skeleton point J is referred to as the right shoulder skeleton point J2, the left ear skeleton point is referred to as the left ear skeleton point J3, and the right ear skeleton point is referred to as the right ear skeleton point J4.

[0045] FIG. 11 is a diagram for explaining the second determination process of the determination device according to the embodiment. FIG. 11 shows a case where the driver 3 turns back to the right rear. As shown in FIG. 11, in the second determination process, the determination unit 222 calculates an angle α (an acute angle) between a line L1 connecting the left shoulder skeleton point J1 and the right shoulder skeleton point J2 and a line M1 along the left-right direction of the forklift 2. When the angle α is equal to or greater than a threshold, the determination unit 222 determines that the driver 3 turns back and the face 3b of the driver 3 faces backward, and the driver 3 has checked the rear. This is because when the driver 3 turns back, one shoulder on the side where the driver turned rises and the other shoulder relatively drops. When the angle α is less than the threshold, the determination unit 222 determines that the driver 3 does not turn back and the face 3b of the driver 3 does not face backward, and the driver 3 has not checked the rear. The threshold is, for example, 1 degree, but is not limited thereto.

[0046] The output processing unit 223 outputs the determination result of the determination unit 222. For example, the output processing unit 223 causes the display device 205 to display a determination result screen on which the determination result is displayed.

[0047] FIG. 12 is a diagram showing an example of a judgment result screen 301 displayed on the display device 205 according to the embodiment. The judgment result screen 301 displays a list of information 302 for each driving to be judged. Each information 302 includes a captured image display field 302a and a judgment result display field 302b. The judgment result display field 302b displays a judgment result for the corresponding driving. The captured image display field 302a displays a frame Gf as a still image when the face 3b of the driver 3 turns toward the rear side of the forklift 2 at the maximum when the forklift 2 moves backward in the captured image G (video). The frame Gf is a thumbnail image. That is, the judgment result screen 301 displays a list of thumbnail images of the captured image G. The frame Gf is extracted from the captured image G by the judgment unit 222 in the judgment process, for example. The frame Gf may be extracted by the output processing unit 223. The captured image G (video) in the captured image display field 302a can be played back by operating the operation device 204.

[0048] Next, the process executed by the determination device 20 will be described with reference to Fig. 13. Fig. 13 is a flowchart showing an example of the determination process executed by the determination device 20 according to the embodiment.

[0049] As shown in FIG. 13, the acquisition unit 221 acquires (receives) a captured image G from the imaging system 10 (S11).

[0050] Next, the determination unit 222 determines the orientation of the face 3b of the driver 3 based on the helmet image 4G in the acquired captured image G (S12). That is, the determination unit 222 performs a first determination process.

[0051] The determination unit 222 determines whether the face 3b of the driver 3 is facing sideways (S13). The face 3b of the driver 3 is facing sideways when the direction of the face of the driver 3 is in the left range H3 or the right range H4.

[0052] If the orientation of the face 3b is not landscape (S13: No), the determination unit 222 proceeds to S15, and if the orientation of the face 3b is landscape (S13: Yes), the determination unit 222 proceeds to S14.

[0053] In S14, the determination unit 222 assigns the feature point information of the driver 3 (for example, information on the skeleton point J) to the determination condition and determines the orientation of the face 3b of the driver 3 (S14). That is, in S14, the determination unit 222 performs a second determination process. After the process of S14, the determination unit 222 proceeds to S15.

[0054] In S15, the determination unit 222 performs a rearward check determination. The rearward check determination is a determination as to whether the driver 3 has checked behind while the forklift 2 is moving backward, based on the result of the previous determination of the direction of the face 3b.

[0055] Next, the output processing unit 223 outputs the determination result of the determination unit 222 in S15, i.e., whether or not the driver 3 checked behind while reversing the forklift 2 (S16). Specifically, the output processing unit 223 causes the display device 205 to display a determination result screen 301.

[0056] <Summary> As described above, the determination device 20 of this embodiment includes the acquisition unit 221, the determination unit 222, and the output processing unit 223. The acquisition unit 221 acquires a captured image G in which the helmet 4 worn on the head 3a of the driver 3 of the forklift 2 (vehicle) is captured. The determination unit 222 determines the orientation of the face 3b of the driver 3 based on the helmet image 4G in the captured image G. The output processing unit 223 outputs the determination result of the determination unit 222.

[0057] According to this configuration, the determination unit 222 determines the orientation of the face 3b of the driver 3 based on the helmet image 4G in the captured image G, so that it is possible to improve the accuracy of determining the orientation of the face 3b of the driver 3 wearing the helmet 4. For example, even if the face 3b is hidden by the helmet 4 and cannot be imaged, it is possible to improve the accuracy of determining the orientation of the face 3b of the driver 3 wearing the helmet 4. Furthermore, according to this configuration, only one imaging device 104 is required to be used for determining the orientation of the face 3b of the driver 3, so that it is easy to simplify the configuration of the determination system 1 and to reduce the cost of the determination system 1.

[0058] The captured image G includes a helmet image 4G and a driver image 3G. The determination unit 222 determines the orientation of the face 3b of the driver 3 based on the helmet image 4G in the captured image G and the feature point (skeleton point J of the shoulder) of the driver image 3G in the captured image G.

[0059] According to this configuration, the judgment unit 222 judges the orientation of the occupant's face 3b based on the helmet image 4G in the captured image G and the feature point (shoulder skeleton point J) of the driver image 3G in the captured image G, thereby further improving the accuracy of judging the orientation of the face 3b of the driver wearing the helmet 4.

[0060] The captured image G is a moving image. The output processing unit 223 causes the display device 205 to display, as a still image, a frame Gf at the time when the face 3b of the driver 3 turns toward the rear side of the forklift 2 to the maximum extent in the captured image G (moving image) while the forklift 2 is moving backward.

[0061] According to this configuration, the frame Gf when the driver's face 3b is turned to the rear side of the forklift 2 at its maximum is displayed as a still image on the display device 205, so that the user of the determination device 20 can easily recognize from the image the direction of the driver's 3 face 3b when the driver's 3 face 3b is turned to the maximum.

[0062] Next, a modified example will be described.

[0063] Fig. 14 is a diagram for explaining the determination process of the determination device 20 according to the modified example of the embodiment. Fig. 15 is a diagram for explaining the determination process of the determination device 20 according to the modified example of the embodiment.

[0064] In this modification, the determination device 20 uses ear skeleton points J (left ear skeleton point J3, right ear skeleton point J4) as feature points in the second determination process.

[0065] As shown in FIG. 14, when the skeleton point J of the ear is located on the opposite side of the brim image 4bG with respect to the center line L2 of the main body image 4aG of the helmet 4, the judgment unit 222 judges that the driver 3 does not turn around to look backward, the face 3b of the driver 3 does not face backward, and the driver 3 is not checking behind.

[0066] When the ear skeleton point J is located on the brim image 4bG side of the center line L2 of the main body image 4aG of the helmet 4, the judgment unit 222 judges that the driver 3 has turned backward, the face 3b of the driver 3 is facing backward, and the driver 3 has checked behind.

[0067] According to this configuration, the judgment unit 222 judges the orientation of the occupant's face 3b based on the helmet image 4G in the captured image G and the feature point (ear skeleton point J) of the driver image 3G in the captured image G, thereby further improving the accuracy of judging the orientation of the face 3b of the driver wearing the helmet 4.

[0068] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0069] 2...Forklift (vehicle) 3. Driver 3G…Driver image (driver image) 3a…Head 3b…face 4. Helmet 4G…Helmet image (helmet image) 20…Judgment device 221…Acquisition Department 222...Judgment section 223...Output processing unit G…Image capture Gf…Frame J... Skeleton points (feature points)

Claims

1. an acquisition unit that acquires an image of a helmet worn on the head of a driver of a vehicle; A determination unit that determines a direction of the driver's face based on an image of the helmet in the captured image; an output processing unit that outputs a determination result of the determination unit; A determination device comprising:

2. The captured image includes an image of the helmet and an image of the driver, The determination unit determines the orientation of the face based on feature points of an image of the helmet in the captured image and an image of the driver in the captured image. The determination device according to claim 1 .

3. The captured image is a video, the output processing unit causes a display device to display, as a still image, a frame in the video in which the driver's face is turned toward the rear side of the vehicle to the maximum extent while the vehicle is moving backward. The determination device according to claim 1 .

Images