Safe driving evaluation program, method, and device

Abstract

The present invention involves: receiving a video including a plurality of frames and obtained by capturing images of an area that includes a driver driving a vehicle and the surroundings of the vehicle; determining the state of the vehicle for each frame on the basis of the differences between the images of an area around the vehicle in the video frames; identifying a target section in the video for evaluating the driver's movement pattern, on the basis of a state pattern indicated by the state determined for each of the consecutive frames; identifying the movement pattern on the basis of the image of the area of the driver in each frame of the identified target section; evaluating the safety of the driving of the vehicle by the driver on the basis of whether or not the identified movement pattern matches the state pattern; and associating and outputting the evaluation result with the video.

Description

Safe driving evaluation program, method, and device

[0001] The disclosed technology relates to a safe driving evaluation program, a safe driving evaluation method, and a safe driving evaluation device.

[0002] Conventionally, evaluations of whether a vehicle, such as a forklift, is being driven safely have been conducted. For example, a vehicle rear image display device has been proposed that determines whether the vehicle is backing up based on detection information from a portable device that is not incorporated into the vehicle. The device displays an image of the rear of the vehicle to the driver when the vehicle is backing up. The device acquires an image of the rear of the vehicle captured by an onboard camera and acquires the vehicle speed by integrating the vehicle's acceleration over time using an acceleration sensor after the vehicle starts moving. The device also sets the vehicle's acceleration direction when the vehicle speed exceeds a predetermined speed as the vehicle's forward direction. If the detected vehicle acceleration direction is opposite to the set forward direction, the device determines that the vehicle is backing up and displays an image of the rear of the vehicle. If the detected vehicle acceleration direction is the same as the set forward direction, the device determines that the vehicle is moving forward and displays guidance information related to the vehicle's movement.

[0003] Also, for example, a driver warning generation device has been proposed that detects the risk of the vehicle starting in the direction opposite to the driver's intended direction and generates a warning. This device detects the position of the vehicle's transmission gear, determines whether the driver is looking toward the front or rear of the vehicle, and detects the risk of the vehicle starting in the direction opposite to the driver's intended direction from the gear position and the driver's looking direction. This device also generates a timing signal to detect the presence or absence of danger, and generates a warning when a danger is detected.

[0004] Furthermore, an information providing device has been proposed that provides a driver with information about what is ahead of the vehicle even when the driver is checking for safety behind the vehicle. When this device detects that the driver of the vehicle is checking for safety behind the vehicle, the display and speaker provide the driver with information about what is ahead of the vehicle.

[0005] JP 2015-069276 A JP 2007-76585 A JP 2010-97270 A

[0006] For example, one method for evaluating whether a driver of a vehicle such as a forklift that handles cargo in a logistics warehouse is driving safely is to check video images from a drive recorder attached to the vehicle. However, manually checking from the video images whether the driver is reliably performing the necessary check operations and maintaining a safe posture in order to drive safely requires a huge amount of man-hours.

[0007] In one aspect, the disclosed technology aims to effectively evaluate safe driving from video images captured of the driving state of a vehicle.

[0008] In one aspect, the disclosed technology accepts a video including multiple frames capturing a driver driving a vehicle and an area surrounding the vehicle. The disclosed technology determines the state of the vehicle for each frame based on image differences in the area surrounding the vehicle between frames of the video. The disclosed technology also identifies a target section in the video for evaluating the driver's behavior pattern based on a status pattern indicated by the status determined for each consecutive frame. The disclosed technology also identifies the behavior pattern based on image differences in the area of ​​the driver between frames in the identified target section. The disclosed technology then evaluates the safety of the driver's driving of the vehicle based on whether the identified behavior pattern corresponds to the status pattern, and outputs the evaluation result in association with the video.

[0009] One aspect is that it has the effect of enabling safe driving to be effectively evaluated from video images of the vehicle's driving state.

[0010] 1 is a block diagram showing a schematic configuration of a safe driving evaluation system; FIG. 2 is a functional block diagram of a safe driving evaluation device; FIG. 3 is a diagram for explaining a threshold for distinguishing between traveling and stopping; FIG. 4 is a diagram for explaining distinguishing between traveling and stopping; FIG. 5 is a diagram for explaining distinguishing between traveling and stopping; FIG. 6 is a diagram for explaining distinguishing between turning; FIG. 7 is a diagram for explaining distinguishing between forward or reverse; FIG. 8 is a diagram for explaining a discrimination range for distinguishing between traveling, reverse, or turning; FIG. 9 is a diagram for explaining distinguishing between traveling, reverse, or turning; FIG. 10 is a diagram for explaining distinguishing between traveling, reverse, or turning; FIG. 11 is a diagram for explaining distinguishing between traveling, reverse, turning, or stopping; FIG. 12 is a diagram for explaining a state pattern for turning; FIG. 13 is a diagram for explaining a state pattern when getting on and off; FIG. 14 is a diagram showing an example of a position that a driver should hold in order to safely get on or off the forklift; FIG. 15 is a diagram showing an example of an evaluation result screen; FIG. 16 is a diagram showing an example of a chapter file; FIG. 17 is a diagram showing an example of a log file; FIG. 18 is a block diagram showing a schematic configuration of a computer that functions as a safe driving evaluation device; FIG. 19 is a flowchart showing an example of a safe driving evaluation process; FIG. 19 is a flowchart showing an example of a specification process;

[0011] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings. In this embodiment, a case where a safe driving evaluation of a forklift is performed will be described as an example. Note that a forklift is an example of a vehicle of the disclosed technology.

[0012] As shown in FIG. 1 , a safe driving evaluation system 100 according to this embodiment includes a safe driving evaluation device 10 and a plurality of user terminals 70 .

[0013] The user terminal 70 is realized by an information processing device such as a personal computer, a tablet terminal, or a smartphone. The user terminal 70 acquires video images captured by a drive recorder attached to the forklift. For example, the drive recorder includes a camera that captures video images of the forklift body, the driver, and an area around the forklift to enable recognition of the forklift's operating status, and a recorder that stores the captured video images. This camera may be, for example, a 360-degree wide-angle camera attached to the top of the forklift's head cover or the tip of the mast, capturing video images of the forklift body from above. The user terminal 70 reads the video images stored in the recorder and uploads them to the safe driving evaluation device 10 via a network. This inputs the video images to the safe driving evaluation device 10. The video images are captured at a predetermined frame rate (e.g., 10 fps), and one video image file contains multiple frames.

[0014] As shown in FIG. 2 , the safe driving evaluation device 10 functionally includes a reception unit 12 , a state determination unit 14 , a section identification unit 16 , a movement identification unit 18 , and an evaluation unit 20 .

[0015] The reception unit 12 receives a moving image input from the user terminal 70 to the safe driving evaluation device 10 and performs preprocessing on the moving image. For example, the reception unit 12 resizes each frame of the moving image to a predetermined size. For example, when deep learning is applied in subsequent processing, the resized size may be set to a size suitable for deep learning processing (e.g., 256 pixels × 256 pixels).

[0016] The receiving unit 12 also normalizes the brightness of all frames included in the moving image, for example, by adaptive histogram equalization processing, etc. This is to prevent erroneous determinations in subsequent functional units due to drastic changes in image brightness, such as when the shooting location changes from indoors to outdoors or from outdoors to indoors.

[0017] The receiving unit 12 also detects frames in which blurring of focus occurs in the moving image. For example, the receiving unit 12 performs edge detection by applying a Laplacian filter to each frame, and if the variance of pixel values ​​of the entire frame calculated from the detection result is smaller than a predetermined threshold (for example, 1,000), the receiving unit 12 determines that the frame is blurred.

[0018] The state determination unit 14 determines the state of the forklift for each frame based on the image difference of the area around the forklift between frames of the video. Specifically, the state determination unit 14 first calculates the optical flow from each frame included in the video, and determines whether the state of the forklift is "moving" or "stopped" for each frame based on the comparison between the magnitude of the optical flow and a first threshold. A large optical flow value indicates a large variation in the image between frames, and therefore it is highly likely that the forklift is moving. On the other hand, a small optical flow value indicates a small variation in the image between frames, and therefore it is highly likely that the forklift is stopped.

[0019] In this embodiment, the first threshold is a larger threshold TH1 between a Ratio threshold set for each video and an absolute threshold common to all videos. For example, the state determination unit 14 may set the Ratio threshold by multiplying a predetermined value by a ratio Ratio calculated by dividing the range between the maximum and minimum median values ​​of the optical flow calculated for each frame across the entire video by the range calculated from the reference video. The reference video may be a video with standard brightness. The value by which the Ratio is multiplied may be, for example, 0.15. The absolute threshold may be, for example, 0.02. Therefore, as shown in FIG. 3 , the state determination unit 14 determines the state of a frame in which the median value of the optical flow is equal to or greater than the threshold TH1, which is the larger of the Ratio threshold and the absolute threshold, as "running," and the state of a frame in which the median value is less than the threshold TH1 as "stopped."

[0020] More specifically, the state determination unit 14 divides each frame into four regions as shown in FIG. 4 . Then, for each region, the state determination unit 14 calculates a vector indicating an optical flow representing the inter-frame fluctuation of the positions of corresponding pixels, and calculates the median of the magnitude of the vector within the region. Note that instead of the median, other statistical values, such as an average value, may be calculated. The reason for dividing the region is to make the state determination less susceptible to the influence of differences in the surrounding scenery reflected in the moving image. Note that the division of the frame is not limited to four divisions as shown in FIG. 4 , and more divisions may be used. Furthermore, the division is not limited to a grid pattern, and the frame may be divided in predetermined angular increments in the circumferential direction from the center of the frame. The state determination unit 14 then calculates a value X by multiplying the median of the optical flow of each region by a moving average for a predetermined number of frames (e.g., 10 frames).

[0021] If the value X calculated for each region is equal to or greater than a threshold value TH1, which is the larger of the ratio threshold value and the absolute threshold value, the state determination unit 14 assigns a flag of "1" to that region, indicating that the region is provisionally determined to be "running." If the value X is less than the threshold value TH1, the state determination unit 14 assigns a flag of "0" to that region, indicating that the region is provisionally determined to be "stopped." For example, if the flag of three-quarters or more of the region in each frame is 1, the state determination unit 14 sets the flag of the entire frame to 1, and otherwise sets the flag to 0.

[0022] Furthermore, the state determination unit 14 assigns a flag of "0" to a frame in which the reception unit 12 detects the occurrence of blurring, regardless of the magnitude of the value X calculated by the moving average of the median values ​​of the optical flows. This is to prevent the state of a frame in which blurring has occurred from being erroneously determined to be "driving."

[0023] As shown in Fig. 5, the state determination unit 14 assigns the determination result "running" to a predetermined number of frames (e.g., three frames) before and after the frame to which the flag "1" is assigned. The state determination unit 14 also assigns the determination result "stopped" to the remaining frames. This prevents excessive switching of the determination result.

[0024] Furthermore, for frames with a determination result of "traveling," the state determination unit 14 further determines whether the forklift is moving forward, backward, or turning based on the distribution of optical flow angles within the frame.

[0025] For example, if the frame is divided into left and right halves as shown in Fig. 6, the optical flow will be in opposite directions in the left and right halves if the forklift is turning, whereas as shown in Fig. 7, the optical flow will be in the same direction in the left and right halves if the forklift is moving forward or backward.

[0026] Therefore, the state determination unit 14 sets a pair of determination ranges at symmetrical positions within the frame so as to include at least the area around the forklift. For example, determination ranges such as those shown in FIG. 8 may be set. In the example of FIG. 8, the determination range shown by the solid line divides the entire frame into two halves, left and right, and is suitable for capturing large movements throughout the frame. The determination range shown by the dotted line divides the driver's side into two halves, left and right, and is less susceptible to the influence of movement of the backrest. The determination range shown by the dashed line is two small areas set on the left and right of the driver's side and is suitable for capturing changes mainly in the floor surface. The determination range shown by the dashed line is two areas set near the support pillar and is suitable for capturing changes mainly in the background. The determination range shown by the two-dot chain line is two small areas set on the left and right of the backrest, and is suitable for capturing changes in the floor surface even if a large object moves on the driver's side.

[0027] When multiple sets of discrimination ranges are set as shown in Fig. 8, the state discrimination unit 14 integrates the discrimination results for each set to discriminate between forward movement, reverse movement, and turning. Specifically, the state discrimination unit 14 calculates the median value of the optical flow angle from each of the left and right discrimination ranges set in the frame. Then, the state discrimination unit 14 assigns a provisional discrimination result indicating "forward movement," "reverse movement," or "turning" to each discrimination range depending on which of the combinations shown in Fig. 6 and Fig. 7 corresponds to the combination of the median values ​​of the left and right optical flow angles.

[0028] The state determination unit 14 integrates the provisional determination results for the multiple determination ranges in each frame to determine the determination result for that frame. For example, the state determination unit 14 may determine the determination result for that frame by a majority vote of the provisional determination results for the multiple determination ranges. Alternatively, as shown in FIG. 9 , the state determination unit 14 may set a frame where the provisional determination results for the multiple determination ranges are all identical as a switching point, and determine the determination results for each frame up to the next switching point as the identical provisional determination result. Alternatively, as shown in FIG. 10 , the state determination unit 14 may determine the determination result for the frame following the frame with the determination result "stop" and for the frame at the start of the video as "none," which does not correspond to forward, reverse, or turning. In this case, the determination results for each frame up to the switching point where the provisional determination results for the multiple determination ranges are all identical may be determined as "none." Note that FIGS. 9 and 10 illustrate a case where three determination ranges are set for each frame.

[0029] Furthermore, the state determination unit 14 may divide the video into predetermined time intervals, take a majority vote among the determination results of each frame within the predetermined time interval, and correct the determination results of each frame within the predetermined time interval. For example, the predetermined time interval may be 1 second, and the frame rate may be 10 fps. In this case, as shown in FIG. 11 , the state determination unit 14 determines the most common determination result ("forward" in the example of FIG. 11 ) among the determination results "forward," "reverse," "turn," and "stop" for the 10 frames as the determination result for those 10 frames. Note that, as shown in FIG. 12 , if the same number of determination results occur in the 10 frames, the determination result that appears earlier in the chronological order may be used. Alternatively, the determination result that appears later in the chronological order may be used. Which determination result to use may be determined experimentally.

[0030] In addition, for each frame, the state determination unit 14 determines that if the distance between the forklift and the driver is equal to or less than a predetermined threshold, it is a riding state in which the driver is riding on the forklift, and if the distance exceeds the threshold, it is a dismounting state in which the driver is dismounting from the forklift.

[0031] Specifically, the state determination unit 14 recognizes a predetermined part of the body, such as the driver's head, for each frame through image processing, and calculates the distance D between the recognized predetermined part and a predetermined point on the forklift. Because the video is captured by a camera fixed to the forklift, the predetermined point on the forklift may be designated in advance as a predetermined position, such as the center of the image. The state determination unit 14 assigns a determination result of "boarding" to frames in which the distance D is equal to or less than a threshold value TH2, and assigns a determination result of "dismounting" to frames in which the distance D exceeds the threshold value TH2. The threshold value TH2 may be set in advance to a value that can distinguish between boarding and dismounting.

[0032] The section specifying unit 16 specifies a target section in the video for evaluating the driver's movement pattern based on the state pattern indicated by the state determined for each of the consecutive frames by the state determining unit 14.

[0033] Specifically, when a state pattern occurs in which the forklift stops after traveling and then reverses or turns, the section identification unit 16 identifies the section that is a predetermined time before the point at which the forklift starts reverses or turns as the target section.

[0034] More specifically, the section identification unit 16 scans the discrimination results from the first frame of the video and searches for a state pattern in which the discrimination results are "driving (forward, backward, or turning)" → "stop" → "backward," or "driving" → "stop" → "turning" → "backward," as shown in FIG. 13 . Hereinafter, these state patterns will be referred to as "direction change state patterns." When a direction change state pattern is found, the section identification unit 16 identifies, as the target section, a section from a predetermined frame (e.g., two frames) before the frame in which the state changes from "stop" to "backward," or from "stop" to "turning," to a predetermined time (e.g., three seconds) before. The predetermined frame and the predetermined time may be set to appropriate values ​​in advance depending on the time when a safety confirmation action is performed while driving corresponding to the corresponding state pattern.

[0035] Furthermore, when a state pattern indicating a change from a riding state to an alighting state or a change from alighting state to a riding state occurs, the section identification unit 16 identifies the time point at which the riding state changes to the alighting state or the time point at which the alighting state changes to a riding state. Then, the section identification unit 16 identifies a section of a predetermined time before and after the identified time point as a target section.

[0036] More specifically, the section identification unit 16 scans the discrimination results from the first frame of the video and searches for a state pattern in which the discrimination result is "get on" → "get off" or "get off" → "get on," as shown in FIG. 14 . Hereinafter, these state patterns will be referred to as "state patterns during getting on and off." When a state pattern during getting on and off is found, the section identification unit 16 identifies, as the target section, a section from a predetermined time before (e.g., 2 seconds before) to a predetermined time after (e.g., 2 seconds after) the frame in which "get on" and "get off" switch. The predetermined time may be set in advance to an appropriate value depending on the time when the getting on or getting off action is performed.

[0037] The movement identification unit 18 identifies a movement pattern of at least one of the driver's movement and line of sight based on the image of the driver's area in each frame of the target section identified by the section identification unit 16.

[0038] Specifically, the movement identification unit 18 acquires skeletal information of the driver in each frame within the target section using a machine learning model such as a pre-trained deep neural network. The movement identification unit 18 then identifies the positions of the shoulders, elbows, wrists, etc., and identifies the direction the driver's arms are pointing as a movement pattern based on the angle between a line connecting both shoulders and a line connecting the elbows and wrists. The movement identification unit 18 may also identify the positions of the nose, shoulders, etc., and identify the driver's line of sight as a movement pattern based on the distance between the nose and left shoulder, the distance between the nose and right shoulder, and the distance between the left shoulder and right shoulder.

[0039] The motion identification unit 18 also identifies the distance between predetermined support points on the forklift and the driver's hands and feet as a motion pattern. The predetermined support points are, for example, points that the driver should support in order to safely get on and off the forklift, as shown in Fig. 15. In the example of Fig. 15, three points are shown as support points: the assist grip, the seat back, and the boarding / exiting step (points indicated by block arrows in Fig. 15).

[0040] The evaluation unit 20 evaluates the safety of the driver's operation of the forklift based on whether the operation pattern identified by the operation identification unit 18 is an operation pattern corresponding to the state pattern, and outputs the evaluation result in association with the video image.

[0041] Specifically, when a target section is identified based on a state pattern of a turn, the evaluation unit 20 determines whether the driver's movement pattern identified in the target section is a movement pattern in which the driver's line of sight points toward the driver's back, or whether the driver's arm points toward the driver's back.

[0042] More specifically, the evaluation unit 20 determines whether or not a change in the pointing direction of the driver's arm within the target section corresponds to a movement of pointing behind the driver, based on the movement pattern of the direction of the driver's arm identified by the movement identification unit 18. Alternatively, the evaluation unit 20 determines whether or not a change in the gaze direction within the target section corresponds to a movement of directing the gaze behind the driver, based on the movement pattern of the driver's gaze direction identified by the movement identification unit 18.

[0043] If the identified movement pattern is one of the movement patterns, the evaluation unit 20 evaluates that the driver has performed a safety check when changing direction, and if the identified movement pattern does not correspond to any of the movement patterns, the evaluation unit 20 evaluates that the driver has not performed a safety check when changing direction.

[0044] Specifically, when a target section is identified based on the state pattern during boarding and disembarking, the evaluation unit 20 determines whether the driver's movement pattern in the target section includes a predetermined safe boarding and disembarking posture that includes supporting one or more support points of the vehicle. More specifically, the evaluation unit 20 determines whether the safe boarding and disembarking posture that simultaneously supports the support points within the target section is adopted based on the movement pattern of the distance between the predetermined support points of the forklift and the driver's hands and feet. The evaluation unit 20 determines whether the support points are being supported based on whether the distance between the support point and the hand or foot corresponding to that support point is equal to or less than a threshold value TH3.

[0045] 15, for example, a safe boarding / exiting posture is assumed to be one in which the left hand holds the assist grip, the right hand holds the back of the seat, and the foot rests on the boarding / exiting step. In this case, the evaluation unit 20 may determine that the boarding / exiting posture is safe when, for example, the distance between the left hand and the assist grip, the distance between the right hand and the back of the seat, and the distance between the right foot or the left foot and the boarding / exiting step are all equal to or less than the threshold value TH3. The threshold value TH3 may be set in advance to a value that can determine that the hand or foot is supporting the support point.

[0046] If a safe boarding / alighting posture is included in the target section, the evaluation unit 20 evaluates that the boarding / alighting operation is being performed safely, and if a safe boarding / alighting posture is not included, the evaluation unit 20 evaluates that the boarding / alighting operation is not being performed safely.

[0047] Furthermore, the evaluation unit 20 adds, as an evaluation result, information indicating whether a movement pattern corresponding to the state pattern has been identified in the target section to a frame of the video corresponding to the target section. Specifically, the evaluation unit 20 adds information indicating that the target section is a target section for turning direction to a target section identified based on a state pattern during turning direction. Furthermore, if the evaluation unit 20 evaluates that a safety check during turning direction has not been performed for a target section for turning direction, it associates an evaluation result such as "poor safety check during turning direction" with the target section. Furthermore, the evaluation unit 20 adds information indicating that the target section is a target section for boarding and disembarking to a target section identified based on a state pattern during boarding and disembarking direction. Furthermore, if the evaluation unit 20 evaluates that a safe boarding and disembarking operation has not been performed for a target section for boarding and disembarking direction, it associates an evaluation result such as "poor safe posture when boarding and disembarking direction" with the target section.

[0048] The moving image with the evaluation result added is transmitted to the user terminal 70 and displayed as an evaluation result screen on a display device provided in the user terminal 70. Note that each frame of the moving image is also given the result of the state determined by the state determination unit 14.

[0049] FIG. 16 shows an example of the evaluation result screen 30. In the example of FIG. 16, the evaluation result screen 30 includes a video playback area 32, status icons 34, a direction change icon 36, and a boarding / disembarking icon 38. Videos are played in the video playback area 32. The status icons 34 identifiably display the corresponding status according to the status determination result assigned to each frame. In the example of FIG. 16, the status icons 34 include icons indicating forward, turning, and reverse, and the icon according to the status determination result assigned to the frame currently displayed in the video playback area 32 is highlighted. Note that in the example of FIG. 16, if the status determination result is "stopped," none of the status icons 34 are highlighted.

[0050] The direction change icon 36 is highlighted when the frame currently displayed in the video playback area 32 is included in the target section for direction changes. Furthermore, the direction change icon 36 is displayed in a display mode such as blinking when an evaluation result such as "poor safety confirmation when changing direction" is associated with the target section. Similarly, the boarding / disembarking icon 38 is highlighted when the frame currently displayed in the video playback area 32 is included in the target section for boarding / disembarking. Furthermore, the boarding / disembarking icon 38 is displayed in a display mode such as blinking when an evaluation result such as "poor safe posture when boarding / disembarking" is associated with the target section.

[0051] The evaluation unit 20 may also create a chapter file in which time information of the video corresponding to the target section is associated with information indicating whether a motion pattern corresponding to the state pattern has been identified in the target section, and output the chapter file together with the video to which the evaluation result is assigned. FIG. 17 shows an example of a chapter file. In the example of FIG. 17 , the chapter file includes data associated with start time information in the video for the target section evaluated as unsafe driving and for frames determined by the reception unit 12 to be blurry. Note that by selecting data in the chapter file, the video may be played in the video playback area 32 on the evaluation result screen 30 from the frame indicated by the start time information of the selected data.

[0052] The evaluation unit 20 may also create and output a log file containing, as evaluation results, the number of target sections identified in the video, the number of times a movement pattern corresponding to a state pattern was identified or not identified in the target section, etc. An example of the log file is shown in Fig. 18. In the example of Fig. 18, in addition to the above-mentioned number of times, etc., information such as an evaluation ID which is identification information of the evaluation result, the total video time of the video, and the time and number of times the state determination result was "forward," "backward," "turning," and "stopped" is also included.

[0053] The safe driving evaluation device 10 may be realized by, for example, a computer 40 shown in Fig. 19. The computer 40 includes a CPU (Central Processing Unit) 41, a GPU (Graphics Processing Unit) 42, a memory 43 as a temporary storage area, and a non-volatile storage device 44. The computer 40 also includes an input / output device 45 such as an input device and a display device, and an R / W (Read / Write) device 46 that controls reading and writing of data from and to a storage medium 49. The computer 40 also includes a communication I / F (Interface) 47 that is connected to a network such as the Internet. The CPU 41, GPU 42, memory 43, storage device 44, input / output device 45, R / W device 46, and communication I / F 47 are connected to one another via a bus 48.

[0054] The storage device 44 is, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage device 44 serving as a storage medium stores a safe driving evaluation program 50 for causing the computer 40 to function as the safe driving evaluation device 10. The safe driving evaluation program 50 includes an acceptance process control command 52, a state determination process control command 54, a section identification process control command 56, an action identification process control command 58, and an evaluation process control command 60.

[0055] The CPU 41 reads the safe driving evaluation program 50 from the storage device 44, loads it into the memory 43, and sequentially executes the control instructions contained in the safe driving evaluation program 50. The CPU 41 operates as the reception unit 12 shown in FIG. 2 by executing the reception process control instruction 52. The CPU 41 also operates as the state determination unit 14 shown in FIG. 2 by executing the state discrimination process control instruction 54. The CPU 41 also operates as the section determination unit 16 shown in FIG. 2 by executing the section identification process control instruction 56. The CPU 41 also operates as the movement determination unit 18 shown in FIG. 2 by executing the action identification process control instruction 58. The CPU 41 also operates as the evaluation unit 20 shown in FIG. 2 by executing the evaluation process control instruction 60. As a result, the computer 40 that executes the safe driving evaluation program 50 functions as the safe driving evaluation device 10. The CPU 41 that executes the program is hardware. A portion of the program may be executed by the GPU 42.

[0056] The functions realized by the safe driving evaluation program 50 may be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.

[0057] Next, the operation of the safe driving evaluation system 100 according to this embodiment will be described.

[0058] A user terminal 70 acquires moving images from a drive recorder attached to a forklift and inputs the moving images to the safe driving evaluation device 10. When the moving images are input to the safe driving evaluation device 10, the safe driving evaluation device 10 executes a safe driving evaluation process shown in Fig. 20. The safe driving evaluation process is an example of a safe driving evaluation method of the disclosed technology.

[0059] In step S10, the receiving unit 12 receives a moving image input from the user terminal 70 to the safe driving evaluation device 10, and performs preprocessing such as resizing, flattening, and blur determination on the moving image.

[0060] Next, in step S12, the state determination unit 14 calculates the optical flow from each frame included in the moving image, and determines for each frame whether the state of the forklift is "running" or "stopped" based on a comparison between the magnitude of the optical flow and a threshold value TH1.

[0061] Next, in step S14, the state determination unit 14 determines whether the state of the forklift is "forward," "backward," or "turning" for the frame with the determination result "traveling" based on the distribution of the optical flow angles within the frame. Next, in step S16, the state determination unit 14 determines whether the forklift is "getting on" or "getting off" for each frame based on the distance between the forklift and the driver.

[0062] Next, in step S20, a specification process is executed. The specification process will now be described in detail with reference to FIG.

[0063] In step S22, the section identification unit 16 scans the discrimination results from the first frame of the moving image and searches for a state pattern of a change of direction in which the discrimination result is "driving (forward, backward, or turning)" → "stop" → "backward," or "driving" → "stop" → "turning" → "backward." If a state pattern of a change of direction is found, the process proceeds to step S24; if not, the process proceeds to step S34.

[0064] In step S24, the section identification unit 16 identifies the section from a predetermined frame before the frame where the state switches from "stop" to "reverse" or from "stop" to "turn" to a predetermined time before as the target section.

[0065] Next, in step S26, the movement identification unit 18 identifies at least one of the movement and line of sight patterns of the driver based on the image of the driver's area in each frame of the target section identified in step S24.

[0066] Next, in step S28, the evaluation unit 20 determines whether the movement pattern identified in step S26 is a movement pattern in which the driver's line of sight points toward the driver's rear, or a movement pattern in which the driver's arm points toward the driver's rear. If it is either of these movement patterns, the process proceeds to step S30, and if it is neither of these movement patterns, the process proceeds to step S32.

[0067] In step S30, the evaluation unit 20 evaluates that the driver has performed a safety check when turning. On the other hand, in step S32, the evaluation unit 20 evaluates that the driver has not performed a safety check when turning.

[0068] Next, in step S34, the section identification unit 16 scans the discrimination results from the first frame of the video and searches for a state pattern at the time of getting on and off in which the discrimination result is "getting on" → "getting off" or "getting off" → "getting on." If a state pattern at the time of getting on and off is found, the process proceeds to step S36; if not found, the identification process ends and the process returns to the safe driving evaluation process ( FIG. 20 ).

[0069] In step S36, the section specification unit 16 specifies, as the target section, a section from a predetermined time before the frame in which "boarding" and "disembarking" are switched to a predetermined time after the frame.

[0070] Next, in step S38, the movement identification unit 18 identifies the distance between a predetermined support point of the forklift and the hands and feet of the driver in the target section identified in step S36 as a movement pattern.

[0071] Next, in step S40, the evaluation unit 20 determines whether the motion pattern identified in step S38 includes a predetermined safe getting-in / out posture that includes support at one or more support points of the vehicle. If the safe posture is included, the process proceeds to step S42, and if not, the process proceeds to step S44.

[0072] In step S42, the evaluation unit 20 evaluates that the driver has boarded and disembarked safely. On the other hand, in step S44, the evaluation unit 20 evaluates that the driver has not boarded or disembarked safely. Then, the identification process ends and the process returns to the safe driving evaluation process (FIG. 20).

[0073] Next, in step S50, the evaluation unit 20 generates and outputs data of an evaluation result confirmation screen in which the evaluation results are associated with the moving images, a chapter file, and a log file, and the safe driving evaluation process ends.

[0074] As described above, in the safe driving evaluation system according to this embodiment, the safe driving evaluation device receives a video including multiple frames capturing an area including a driver operating a forklift and the area surrounding the forklift. The safe driving evaluation device determines the state of the forklift for each frame based on image differences in the area surrounding the forklift between frames of the video. The safe driving evaluation device identifies a target section from the video for evaluating the driver's behavior pattern based on a status pattern indicated by the status determined for each consecutive frame. The safe driving evaluation device also identifies a behavior pattern based on images of the driver's area in each frame of the identified target section. The safe driving evaluation device then evaluates the safety of the driver's vehicle driving based on whether the identified behavior pattern corresponds to the status pattern, and outputs the evaluation result in association with the video. This allows for effective evaluation of safe driving from video capturing the vehicle's driving status.

[0075] In the above embodiment, an example of a situation where a safety check is required is described as when changing direction to reverse, but this is not limiting. For example, the present invention may also be applicable to a case where a forklift driver checks for safety ahead. Possible situations for checking for safety ahead include checking for safety before moving forward, checking for safety on both sides at a T-junction in a work site, and checking whether a loaded load is stable before moving forward. In this case, too, the target section is identified based on a state pattern of moving forward or turning after stopping, a state pattern indicating the loading of a load, etc. Then, the direction in which the driver's arm is pointing or the driver's line of sight in the target section is identified as a movement pattern, and the driver's movement is evaluated in the same manner as in the above embodiment.

[0076] In the above embodiment, the forklift status was described as being determined to be forward, backward, turning, and stopped. However, other factors, such as the load status of the forks, may also be determined. In the above embodiment, the evaluation was described as being based on safety confirmation during direction changes and safe posture during boarding and disembarking. However, this is not limited to these. In addition, the presence or absence of dangerous driving, such as fork operation while traveling with or without a load on the forks (hereinafter referred to as "multitasking"), or sharp turns while loaded or unloaded, may also be evaluated. Figures 16 to 18 described in the above embodiment also include evaluation results related to multitasking. For example, the icons in the lower left corner of the evaluation result screen in Figure 16 are an icon representing the load status and an icon representing multitasking.

[0077] Furthermore, in the above embodiment, a safety coefficient may be calculated and added to the log file created by the evaluation unit, which is an index indicating the degree of safe driving for each item in the video. For example, with regard to safety confirmation during turns, the safety coefficient may be calculated as the number of times that inadequate safety confirmation during turns was confirmed relative to the number of times that a target section was identified based on the state pattern during turns. For example, with regard to safe posture during boarding and disembarking, the safety coefficient may be calculated as the number of times that inadequate safe posture during boarding and disembarking was confirmed relative to the number of times that a target section was identified based on the state pattern during boarding and disembarking. With regard to maneuvering while driving in a loaded or unloaded state, the safety coefficient may be calculated as the time during which the maneuver was performed relative to the driving time in a loaded or unloaded state. With regard to sharp turns, the safety coefficient may be calculated as the number of sharp turns made in a loaded or unloaded state relative to the number of turns made in a loaded or unloaded state. Note that the safety coefficient for maneuvering while driving may be calculated using the number of times instead of the time, and for sharp turns, the safety coefficient may be calculated using the time instead of the number of times.

[0078] Furthermore, in the above embodiment, an example was described in which the safe driving of a forklift was evaluated, but the disclosed technology can be applied to any vehicle that requires safety confirmation and a safe posture when driving.

[0079] In the above embodiment, the safe driving evaluation program is stored (installed) in advance in a storage device, but is not limited to this. The program according to the disclosed technology may be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.

[0080] REFERENCE SIGNS LIST 10 Safe driving evaluation device 12 Reception unit 14 State determination unit 16 Section identification unit 18 Action identification unit 20 Evaluation unit 30 Evaluation result screen 32 Video image playback area 34 State icon 36 Turning icon 38 Getting in and out icon 40 Computer 41 CPU 42 GPU 43 Memory 44 Storage device 45 Input / output device 46 R / W device 47 Communication I / F 48 Bus 49 Storage medium 50 Safe driving evaluation program 52 Reception process control command 54 State determination process control command 56 Section identification process control command 58 Action identification process control command 60 Evaluation process control command 70 User terminal 100 Safe driving evaluation system

Claims

1. receiving a video including a plurality of frames capturing an area including a driver driving a vehicle and the surroundings of the vehicle; determining a state of the vehicle for each frame based on an image difference of an area around the vehicle between frames of the moving image; identifying a target section in the video for evaluating the driver's movement pattern based on a state pattern indicated by the state determined for each of the consecutive frames; Identifying the movement pattern based on an image of the driver's area in each frame of the identified target section; evaluating the safety of the driver's driving of the vehicle based on whether the identified operation pattern is an operation pattern corresponding to the state pattern, and outputting the evaluation result in association with the moving image. A safe driving evaluation program for causing a computer to execute processing including the above.

2. In the case of a state pattern in which the vehicle stops after traveling and then reverses or turns, a section that is a predetermined time before the time when the vehicle starts to reverse or turn is identified as the target section; When the driver's movement pattern in the target section is a movement pattern in which the driver's line of sight points toward the driver's rear, or a movement pattern in which the driver's arms point toward the driver's rear, it is evaluated that the driver has performed a safety check when turning, and when the movement pattern does not correspond to either of these movement patterns, it is evaluated that the driver has not performed a safety check when turning. The safe driving evaluation program according to claim 1 .

3. When the distance between the vehicle and the driver is equal to or less than a predetermined threshold, the vehicle is determined to be in an onboard state in which the driver is on board the vehicle, and when the distance between the vehicle and the driver exceeds the threshold, the vehicle is determined to be in an offboard state in which the driver is off the vehicle; In the case of a state pattern indicating a change from the riding state to the dismounting state, or a change from the dismounting state to the riding state, a section of a predetermined time before and after the time point at which the riding state changes to the dismounting state, or the time point at which the dismounting state changes to the riding state, is identified as the target section; When the driver's movement pattern in the target section includes a predetermined safe getting-in / out posture including support at one or more points of the vehicle, it is evaluated that the driver is performing a safe getting-in / out movement, and when the driver's movement pattern does not include the safe getting-in / out posture, it is evaluated that the driver is not performing a safe getting-in / out movement.

3. The safe driving evaluation program according to claim 1 or 2.

4. A safe driving evaluation program as described in claim 1 or claim 2, which determines whether the state of the vehicle in each frame is moving forward, backward, or turning based on a comparison of the optical flow calculated from each of a set of ranges set at left-right symmetrical positions within the frame so as to include at least the area surrounding the vehicle.

5. A safe driving evaluation program as described in claim 4, wherein when multiple sets of the ranges are set, the discrimination results for each set are integrated to determine whether the state of the vehicle in each frame is moving forward, backward, or turning.

6. 5. The safe driving evaluation program according to claim 4, wherein the state pattern that is most frequently identified among the state pattern discrimination results for each of a predetermined number of consecutive frames is set as the discrimination result for the predetermined number of frames.

7. A safe driving evaluation program as described in claim 1 or claim 2, wherein, as the evaluation result, information indicating whether or not a behavior pattern corresponding to the state pattern has been identified in the target section is added to the frame of the video corresponding to the target section.

8. A safe driving evaluation program as described in claim 1 or claim 2, which outputs, as the evaluation result, a chapter file in which information indicating whether an operation pattern corresponding to the status pattern has been identified in the target section is associated with time information of the video corresponding to the target section.

9. A safe driving evaluation program as described in claim 1 or claim 2, which outputs a log file as the evaluation result, including the number of target sections identified in the video and the number of times an operation pattern corresponding to the status pattern was identified or not identified in the target section.

10. A safe driving evaluation program as described in claim 1 or claim 2, which performs edge detection on each frame of a moving image, and if the variance value calculated from the detection result is smaller than a predetermined threshold, determines that blurring has occurred in the frame, and outputs the determination result together with the evaluation result.

11. receiving a video including a plurality of frames capturing an area including a driver driving a vehicle and the surroundings of the vehicle; determining a state of the vehicle for each frame based on an image difference of an area around the vehicle between frames of the moving image; identifying a target section in the video for evaluating the driver's movement pattern based on a state pattern indicated by the state determined for each of the consecutive frames; Identifying the movement pattern based on an image of the driver's area in each frame of the identified target section; evaluating the safety of the driver's driving of the vehicle based on whether the identified operation pattern is an operation pattern corresponding to the state pattern, and outputting the evaluation result in association with the moving image. A safe driving evaluation method in which a computer executes processes including:

12. a receiving unit that receives a moving image including a plurality of frames, the moving image including a driver driving a vehicle and an area around the vehicle; a state determination unit that determines a state of the vehicle for each frame based on an image difference of an area around the vehicle between frames of the moving image; a section specifying unit that specifies a target section in the video for evaluating the driver's movement pattern based on a state pattern indicated by the state determined for each of the consecutive frames; a movement identification unit that identifies the movement pattern based on an image of the driver's area in each frame of the identified target section; an evaluation unit that evaluates the safety of the driver's driving of the vehicle based on whether the identified operation pattern is an operation pattern corresponding to the state pattern, and outputs the evaluation result in association with the moving image; A safe driving evaluation device including: