Information processing program, device, and method
An information processing device analyzes forklift images to identify dangerous driving conditions and adjusts playback speed, addressing the inefficiency of manual inspections and improving safety evaluations.
Patent Information
- Application Number
- JP2021076785
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-04-28
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-04-28
AI Technical Summary
Manually inspecting images from drive recorders attached to forklifts for safe operation requires significant man-hours, making it inefficient.
An information processing device analyzes moving images from forklifts to determine driving states and operations of movable parts, identifying dangerous driving conditions and adjusting playback speed based on these determinations.
Efficiently identifies dangerous driving conditions in forklift operations and reduces evaluation time by highlighting dangerous sections and adjusting playback speed, enhancing safety evaluation efficiency.
Smart Images

Figure 0007577603000001 
Figure 0007577603000002 
Figure 0007577603000003
Abstract
Description
[Technical field]
[0001] The disclosed technology relates to an information processing program, an information processing device, and an information processing method. [Background technology]
[0002] Regarding detection of dangerous driving of a moving object, for example, a system has been proposed that observes the behavior of the moving object and determines that the moving object is possibly being driven dangerously when the observation result satisfies a specific condition. In this system, the specific condition is, for example, an abnormal approach of a moving object behind the moving object, a sudden acceleration or deceleration of the moving object behind, or repeated sudden acceleration and deceleration.
[0003] Also, a playback device has been proposed that allows a user to view video that has been processed based on the acquired information. In this playback device, a playback window is displayed on the display screen of a display device of a personal computer, and video recorded by a drive recorder is displayed in the video display area of the playback window. Also, this playback device records information that changes with the driving operation of a vehicle, and displays a faster speed for video parts in which the subject's movement does not change than for video parts in which the subject's movement changes. Also, this playback device displays information about the vehicle's situation at the time the video data was recorded.
[0004] Technologies for detecting dangerous driving of cargo handling vehicles such as forklifts have also been proposed. For example, an image processing device has been proposed that provides a driver with information useful for operating a forklift. This image processing device detects objects from an image captured by a camera unit that captures the front of the forklift body, acquires information on the moving speed of the body, and acquires status information on the attitude and height direction position of the forks supported on the front side of the body so that they can be raised and lowered. This image processing device also detects the presence or absence of cargo on the forks or the size of the cargo, and determines the working status of the forklift according to the fork status information and the result of the cargo detection. Then, this image processing device generates a display image according to the determined working status, and displays it on a display unit attached to a position visible from the driver's seat of the forklift.
[0005] Also, for example, a forklift driving support device has been proposed that is mounted on a reach forklift and has a camera that captures at least a part of the surroundings of the reach forklift. This device generates a maximum turning circle on a predicted driving trajectory according to the steering angle, and displays the maximum turning circle superimposed on the image captured by the camera, thereby supporting the determination of the turning start position. [Prior art documents] [Patent documents]
[0006] [Patent Document 1] JP 2020-102092 A [Patent Document 2] JP 2019-32739 A [Patent Document 3] JP 2020-142903 A [Patent Document 4] JP 2020-83572 A Summary of the Invention [Problem to be solved by the invention]
[0007] For example, one method for evaluating whether a moving object such as a forklift that handles cargo in a logistics warehouse is operating safely is to check the video footage from a drive recorder attached to the moving object. However, manually checking the video footage requires a huge amount of man-hours.
[0008] In one aspect, the disclosed technology aims to determine dangerous driving from video images captured of the driving state of a moving object. [Means for solving the problem]
[0009] In one aspect, the disclosed technology determines the running state of a mobile body including a movable part that moves relative to a mobile body body from a video image capturing the driving state of the mobile body, and the operation state of the movable part. Then, the disclosed technology specifies whether or not the mobile body is being driven dangerously in the video image based on the determination results of the running state and the operation state. Effect of the Invention
[0010] One aspect of the present invention is that it has an effect of being able to judge dangerous driving from video images captured of the driving state of a moving object. [Brief description of the drawings]
[0011] [Figure 1] FIG. 2 is a functional block diagram of an information processing device. [Diagram 2] FIG. 2 is a diagram showing an example of one frame of a moving image after preprocessing. [Diagram 3] FIG. 4 is a diagram for explaining determination of a running state. [Figure 4] FIG. 13 is a diagram for explaining the direction of optical flow during turning. [Diagram 5] 1A and 1B are diagrams for explaining the directions of forward and backward optical flows; [Figure 6] FIG. 13 is a diagram for explaining conditions for the direction of optical flow during turning. [Figure 7] 11A and 11B are diagrams for explaining detection of a backrest and specification of a claw movable area. [Figure 8] 13 is a diagram for explaining an angle of an optical flow corresponding to a movable direction of a claw. FIG. [Figure 9] FIG. 11 is a schematic diagram showing an example in which a tape of a specific color is attached to a backrest. [Figure 10] FIG. 13 is a diagram for explaining extraction of a region of a specific color. [Figure 11] FIG. 13 is a diagram for explaining how to specify a person's position. [Figure 12] FIG. 11 is a diagram for explaining the distance between a person's position and an average position. [Figure 13] FIG. 4 is a diagram illustrating an example of a table showing the relationship between a determination result and an operating state. [Figure 14] FIG. 13 is a diagram showing an example of a display of a processed moving image. [Figure 15] FIG. 1 is a block diagram showing a schematic configuration of a computer that functions as an information processing device. [Figure 16] 10 is a flowchart illustrating an example of an information processing routine. [Figure 17] 10 is a flowchart showing an example of a traveling determination process. [Figure 18] 13 is a flowchart illustrating an example of an operation determination process. [Figure 19] 13 is a flowchart showing an example of a boarding / alighting determination process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings. In this embodiment, an example of a case where a forklift driving evaluation is performed will be described. Note that a forklift is an example of a moving body of the disclosed technology, and the forks of the forklift are an example of a moving part that moves relative to the moving body of the disclosed technology.
[0013] As shown in FIG. 1, a moving image is input to the information processing device 10 according to the present embodiment, and the information processing device 10 processes the moving image for driving evaluation and outputs it as a processed moving image. The moving image input to the information processing device 10 is a moving image output from a drive recorder attached to a forklift. For example, the drive recorder includes a camera that captures an area including the forklift body, the movable area of the claws, the occupant, etc., to obtain a moving image that allows the driving state of the forklift to be recognized, and a recorder that stores the captured moving image. For example, this camera may be a 360-degree wide-angle camera that is attached to the top of the head cover or the tip of the mast of the forklift and captures the forklift body side from the top. The moving image stored in the recorder is read out by a user terminal such as a personal computer, and is input to the information processing device 10 by uploading it from the user terminal to the information processing device 10 via a network. The moving image is captured at a predetermined frame rate (for example, 10 fps), and one file of the moving image includes multiple frames.
[0014] 1, the information processing device 10 functionally includes a preprocessing unit 11, a driving determination unit 12, an operation determination unit 13, a boarding / alighting determination unit 14, an identification unit 15, a calculation unit 16, and a generation unit 17. The driving determination unit 12, the operation determination unit 13, and the boarding / alighting determination unit 14 are examples of the determination units of the disclosed technology.
[0015] The pre-processing unit 11 performs pre-processing on the moving image input to the information processing device 10. For example, the pre-processing unit 11 resizes each frame of the moving image to a predetermined size. The resized size may be a size suitable for deep learning processing (e.g., 256 pixels x 256 pixels) when deep learning is applied in the subsequent processing.
[0016] In addition, the pre-processing unit 11 normalizes the luminance of all frames included in the moving image by, for example, adaptive histogram equalization processing, etc. This is to suppress erroneous determinations in each functional unit at the later stage due to abrupt changes in image brightness, such as when the shooting location moves from indoors to outdoors or from outdoors to indoors.
[0017] In addition, the pre-processing unit 11 detects frames in which out-of-focus blur occurs in a moving image. For example, the pre-processing unit 11 applies a Laplacian filter to each frame, and if the variance of pixel values of the entire frame after application of the filter is less than 1,000, the pre-processing unit 11 detects the frame as one in which blur occurs.
[0018] An example of one frame of the preprocessed video is shown in Fig. 2. The preprocessing unit 11 passes the preprocessed video to the driving determination unit 12, the operation determination unit 13, and the getting on / off determination unit 14, respectively.
[0019] The traveling determination unit 12 determines the traveling state of the forklift from the moving image received from the pre-processing unit 11. Specifically, the traveling determination unit 12 calculates an optical flow from each frame included in the moving image, and determines for each frame whether the forklift is traveling or stopped based on a comparison between the magnitude of the optical flow and a first threshold value.
[0020] More specifically, the traveling determination unit 12 calculates a vector indicating an optical flow that represents the image variation between frames by associating corresponding pixels between frames, and calculates the median value of the magnitude of the vector within the frame. Note that other statistical values such as an average value may be calculated instead of the median value. Also, a moving average for a predetermined number of frames may be applied as the statistical value. When the optical flow value is large, it indicates that the image variation between frames is large, and therefore it is highly likely that the forklift is traveling. On the other hand, when the optical flow value is small, it indicates that the image variation between frames is small, and therefore it is highly likely that the forklift is stopped.
[0021] Therefore, as shown in FIG. 3, when the median value X of the optical flow calculated from the frame to be processed is equal to or greater than the threshold value TH1, the running determination unit 12 assigns a flag "1" to the frame indicating a provisional determination result of "running". For example, the threshold value TH1 may be a value that can distinguish the magnitude of the optical flow when the vehicle is running from when the vehicle is stopped. The running determination unit 12 may set the threshold value TH1 to a value obtained by multiplying a ratio Ratio, which is a ratio obtained by dividing the width between the maximum and minimum values of the median value of the optical flow calculated for each frame in the entire video by the width calculated from the reference video, by a predetermined value. As the reference video, a video with standard brightness may be predetermined. Furthermore, when the median value X of the optical flow calculated from the frame to be processed is less than the threshold value TH1, the running determination unit 12 assigns a flag "0" to the frame indicating a provisional determination result of "stopping". Furthermore, the moving determination unit 12 assigns a flag of "0" to a frame in which blurring has been detected by the pre-processing unit 11, regardless of the magnitude of the median value X of the optical flow. This is to prevent a frame in which blurring has occurred from being erroneously determined to be "moving."
[0022] 3, the moving determination unit 12 assigns metadata indicating the determination result "moving" to a predetermined number of frames (e.g., three frames) before and after the frame to which the flag "1" is assigned. In addition, the moving determination unit 12 assigns metadata indicating the determination result "stopped" to the remaining frames. This prevents excessive switching of the determination result.
[0023] In addition, for frames with a judgment result of "moving", the driving judgment unit 12 further judges whether the forklift is moving forward or backward (hereinafter referred to as "forward / reverse") or turning based on the distribution of optical flow angles within the frame.
[0024] For example, if the frame is divided into left and right halves as shown in Fig. 4, the optical flow will be in opposite directions in the left and right regions if the forklift is turning. On the other hand, as shown in Fig. 5, if the forklift is moving forward or backward, the optical flow will be in the same direction in the left and right regions.
[0025] Therefore, the traveling determination unit 12 calculates the median value of the optical flow angles from each of the left and right regions of the frame, and determines whether or not the combination of the optical flow angles in the left and right regions satisfies a predetermined condition. For example, the predetermined condition may be, when the angle ranges are set as shown in Fig. 6, that one of the angles of the left and right regions is included in range 1 and the other angle is included in range 3, or that one angle is included in range 2 and the other angle is included in range 4.
[0026] When the combination of the optical flow angles calculated from the frame to be processed satisfies the condition, the traveling determination unit 12 assigns a flag "1" to the frame indicating a provisional determination result of "turning". On the other hand, when the condition is not satisfied, the traveling determination unit 12 assigns a flag "1" to the frame indicating a provisional determination result of "forward / reverse". In addition, similar to the case of FIG. 3, the traveling determination unit 12 changes the metadata indicating the determination result "running" assigned to a predetermined number of frames (e.g., 5 frames) before and after the frame to which the flag "1" is assigned, to metadata indicating the determination result "turning". In addition, the traveling determination unit 12 changes the metadata indicating the determination result "running" assigned to the remaining frames to metadata indicating the determination result "forward / reverse".
[0027] Furthermore, the running determination unit 12 determines whether the frame with the determination result "stopped" is an appropriate stop or a temporary stop based on the duration of the stopped state. For example, the running determination unit 12 determines whether the frames with the determination result "stopped" are continuous for a number of frames equal to or greater than a second threshold TH2 (e.g., 20 frames). As the threshold TH2, the duration of the stopped state required as an appropriate stop (e.g., 2 seconds) may be previously determined. If they are continuous, the running determination unit 12 changes the metadata of the determination result "stopped" given to each frame included in the continuous section to metadata indicating the determination result "appropriately stopped" indicating an appropriate stop. Furthermore, if the number of consecutive frames with the determination result "stopped" does not reach a predetermined time, the running determination unit 12 changes the metadata of the determination result "stopped" given to each frame to metadata indicating the determination result "temporarily stopped".
[0028] The operation determination unit 13 determines the operation state of the nail from the moving image passed from the preprocessing unit 11. Specifically, the operation determination unit 13 detects an area corresponding to the nail from each frame of the moving image, and determines whether or not the nail is being operated for each frame based on a comparison between the difference between the optical flow in the detected area and the optical flow in the entire frame and a third threshold value.
[0029] More specifically, the operation determination unit 13 detects an area indicating a backrest from each frame using a model trained in advance using a large number of backrest images by a deep learning object detection method such as YOLO (You Only Look Once). The backrest is a load-receiving frame for preventing loads placed on the claw from falling to the rear of the mast, and moves in response to the vertical movement of the claw. Therefore, the movement of the claw is detected by detecting the movement of the backrest.
[0030] For example, the operation determination unit 13 specifies a claw movable area 22 indicating the movable range of the claw as shown in the lower diagram of FIG. 7 based on the area 21 indicating the detected backrest as shown in the upper diagram of FIG. 7 and the mounting position of the camera of the drive recorder. The operation determination unit 13 calculates the optical flow in each frame, and extracts an optical flow within the claw movable area 22 and at an angle corresponding to the movable direction of the claw from the calculated optical flow. In the case of the example of FIG. 7, for example, the angle of the range shown by the shading in FIG. 8 may be predefined as the angle corresponding to the movable direction of the claw. If the difference between the median value X1 of the magnitude of the extracted optical flow and the median value X2 of the magnitude of the optical flow of the entire frame is equal to or greater than the threshold value TH3, the operation determination unit 13 assigns a flag "1" indicating a provisional determination result of "operation in progress" to the frame. On the other hand, if the difference between X1 and X2 is less than the threshold value TH3, the operation determination unit 13 assigns a flag "0" indicating a provisional determination result of "not operating" to the frame. The threshold value TH3 may be determined in advance as a value that allows a distinction to be made between the magnitude of the optical flow when the nail is being manipulated and when it is not being manipulated, for example.
[0031] 3, the operation determination unit 13 assigns metadata indicating the determination result "operation in progress" to a predetermined number of frames (e.g., three frames) before and after the frame to which the flag "1" is assigned. Also, the operation determination unit 13 assigns metadata indicating the determination result "not operating" to the remaining frames.
[0032] Furthermore, the operation determination unit 13 may detect an area of a predetermined specific color within the claw movable area 22 from each frame of the moving image, and determine whether or not the claw is being operated for each frame based on a comparison between the amount of variation between frames of the detected area and a fourth threshold value. The specific color area is, for example, an area showing a marker 24 such as a tape of a specific color attached to a predetermined position of the backrest 23, as shown in FIG. 9. In addition, it is not limited to the case where the marker 24 is attached, but may be an area of the specific color applied to at least a part of the backrest itself. Note that it is preferable to adopt a color that is easily recognized on an image, such as a fluorescent color, as the specific color.
[0033] Specifically, the operation determination unit 13 extracts pixels showing a specific color based on the HSV value of each pixel in the claw movable area 22 for each frame, and generates a binary image in which pixels in the claw movable area 22 from which the specific color can be extracted are expressed in white and pixels from which the specific color cannot be extracted are expressed in black, as shown in FIG. 10. The operation determination unit 13 calculates the change amount C of the area showing the specific color by taking the background difference of the binary image between frames. If the calculated change amount C is equal to or greater than a threshold value TH4, the operation determination unit 13 assigns a flag "1" indicating a provisional determination result of "operation in progress" to the frame. On the other hand, if the change amount C is less than the threshold value TH4, the operation determination unit 13 assigns a flag "0" indicating a provisional determination result of "not operating" to the frame. As the threshold value TH4, for example, a value that can distinguish the magnitude of the change amount C of the area showing the specific color between when the vehicle is moving and when the vehicle is stopped may be previously determined. Then, similar to the above, the operation determination unit 13 assigns metadata indicating the determination result "operation in progress" or "not operating" to each frame based on the flag.
[0034] The operation determination unit 13 may perform a combination of the above-mentioned optical flow-based determination and specific color-based determination. For example, the operation determination unit 13 may assign a flag "1" when the difference between the median values X1 and X2 of the optical flow magnitudes is equal to or greater than a threshold value TH3 and the change amount C of the area showing the specific color is equal to or greater than a threshold value TH4.
[0035] The boarding / disembarking determination unit 14 determines whether the occupant is getting on or off the forklift from the video image passed from the pre-processing unit 11. Specifically, the boarding / disembarking determination unit 14 detects an area corresponding to a person from each frame of the video image. Then, the boarding / disembarking determination unit 14 determines whether the occupant is getting on or off the forklift for each frame based on a comparison between the difference between the position of the area detected in each frame and the average position of the area detected in all frames and a fifth threshold value.
[0036] More specifically, the boarding / disembarking determination unit 14 detects an area showing a person from each frame using a model trained in advance using a large number of person images by an object detection method using deep learning such as YOLO. If two or more areas showing a person are detected from one frame, the area output from the model that has a higher degree of certainty indicating the likelihood that the detected area is a person may be adopted. As shown in FIG. 11, the boarding / disembarking determination unit 14 specifies the coordinates in the frame of the center point of the bounding box 25 detected as an area showing a person for each frame as the position P1 of the person. In addition, the boarding / disembarking determination unit 14 specifies the average of the positions P1 specified from each frame in the entire video as the average position P2 of the person. P2 is a reference point indicating where the area showing the occupant is located on average on the frame.
[0037] As shown in FIG. 12, when the distance D between the person's position P1 and the average position P2 in a frame 27 is equal to or greater than a threshold value TH5, the boarding / alighting determination unit 14 assigns a flag "1" to the frame, indicating a provisional determination result of "getting off". The boarding / alighting determination unit 14 also assigns a flag "1" to a frame in which an area indicating a person is not detected. On the other hand, when the distance D is less than the threshold value TH5, the boarding / alighting determination unit 14 assigns a flag "0" to the frame, indicating a provisional determination result of "getting on". As the threshold value TH5, for example, a value (e.g., 70 pixels) that can distinguish the difference in the positional relationship between getting on and getting off may be previously determined based on the frame size, the angle of view, the area of the driver's seat in the frame, and the like.
[0038] The boarding / alighting determination unit 14 assigns metadata indicating the determination result "getting off" to each frame included in a section where frames with the flag "1" continue for a predetermined time (e.g., 1 second) or more (e.g., 10 frames). The boarding / alighting determination unit 14 also assigns metadata indicating the determination result "getting on" to the other frames.
[0039] The identification unit 15 identifies a section in the video where dangerous driving of the forklift is taking place based on the determination results of the traveling state and the operation state of the claw. Here, a "section" is a division of the video that corresponds to the time axis. A section in the video is identified, for example, by time information or a frame number associated with a frame corresponding to the start of the section, and time information or a frame number associated with a frame corresponding to the end of the section. In this embodiment, dangerous driving is defined as "operation while driving" in which the claw is operated while traveling, and insufficient stopping. Specifically, the identification unit 15 identifies a section in the video where the determination results of the frame are "turning" or "forward / reverse" and "operating" as a section where dangerous driving is taking place. The identification unit 15 also identifies a section in the video where the determination result of the frame is "temporarily stopped" as a section where dangerous driving is taking place.
[0040] For example, the identification unit 15 holds a table showing the relationship between the determination result and the driving state as shown in Fig. 13. In the example of Fig. 13, when the determination result is "turning" and "operating", the driving state is "operation while turning" among "operations while doing other things". Also, when the determination result is "forward / reverse" and "operating", the driving state is "operation while driving" among "operations while doing other things". In Fig. 13, the driving states "operation while turning", "operation while driving", and "insufficient stopping" shown in thick frame are dangerous driving. The "playback speed" included in the table of Fig. 13 will be described later.
[0041] The identification unit 15 refers to this table, identifies the driving state corresponding to the judgment result assigned to each frame, and assigns metadata indicating the type of the identified driving state to each frame. The identification unit 15 then assigns metadata indicating dangerous driving to frames identified as having a driving state of "operating while turning," "operating while driving," or "insufficient stopping." A section in which consecutive frames to which metadata indicating dangerous driving are assigned is a section in which dangerous driving is occurring.
[0042] The calculation unit 16 calculates the ratio of a section where dangerous driving is performed to a section where it is determined that the driver is riding on the forklift in the video as a safety factor. Specifically, the calculation unit 16 calculates the safety factor as a value obtained by dividing the number of frames where the driving state is determined to be either "operating while turning," "operating while traveling," or "insufficient stopping" by the number of frames where the determination result is "riding."
[0043] The generating unit 17 processes the video so that the playback speed of the video is changed based on the result of identifying whether or not the section is dangerously driven. Specifically, the generating unit 17 performs speed-doubling processing on the video so that the playback speed of the section where dangerous driving is not performed is faster than the playback speed of the section where dangerous driving is performed. As the speed-doubling processing, the generating unit 17 may thin out the number of frames in the section where the playback speed is to be increased, or may add control information to the video so that the playback speed is set to a specified speed during playback. In addition, the generating unit 17 may determine a "playback speed" according to the driving state in advance, for example, as shown in the table in FIG. 13, and determine the playback speed of each section of the video by referring to this "playback speed". In the example of FIG. 13, the playback speed is set to 1x speed, i.e., normal speed, 6x speed when riding without dangerous driving, and 10x speed when getting off. By setting the speed in this way, when evaluating the video, the scenes of dangerous driving are played back normally, and other scenes are played back in fast-forward, which can significantly reduce the man-hours required for evaluation.
[0044] Furthermore, the generation unit 17 processes the moving image so that the respective determination results of the driving determination unit 12, the operation determination unit 13, and the getting on / off determination unit 14, the identification result of the identification unit 15, and the safety coefficient calculated by the calculation unit 16 are displayed in association with the moving image. For example, the generation unit 17 adds information to the moving image as shown in Fig. 2 so that information based on the metadata added to each frame is displayed in each frame as shown in Fig. 14.
[0045] In the example of FIG. 14, the processed video (hereinafter, referred to as the "processed video") 28 includes a dangerous driving state display 29 that displays the type of driving state for a section determined to be dangerous driving. The processed video 28 also includes a judgment result display 30 that indicates the judgment result by each judgment unit. In the example of FIG. 14, the judgment result display 30 is a display in which a lamp is associated with a character string indicating the judgment result, and a display in which a lamp corresponding to the relevant judgment result is lit. The processed video 28 also includes an area display 31 that indicates a person detected by the getting on / off judgment unit 14, a safety factor display 32 that indicates the safety factor calculated by the calculation unit 16, and a double speed display 33 that indicates the playback speed determined according to the driving state.
[0046] As described above, the generating unit 17 processes the video by speeding it up and adding information to it, generates a processed video 28, and transmits a file of the processed video 28 to the user terminal. The transmission to the user terminal may be performed by downloading the processed video 28 stored in a predetermined storage area of the information processing device 10 to the user terminal that has made a download request for the processed video 28.
[0047] The information processing device 10 can be realized by, for example, a computer 40 shown in Fig. 15. The computer 40 includes a CPU (Central Processing Unit) 41, a memory 42 as a temporary storage area, and a non-volatile storage unit 43. The computer 40 also includes an input / output device 44 such as an input unit and a display unit, and an R / W (Read / Write) unit 45 that controls reading and writing of data from and to a storage medium 49. The computer 40 also includes a communication I / F (Interface) 46 that is connected to a network such as the Internet. The CPU 41, the memory 42, the storage unit 43, the input / output device 44, the R / W unit 45, and the communication I / F 46 are connected to one another via a bus 47.
[0048] The storage unit 43 can be realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage unit 43 as a storage medium stores an information processing program 50 for causing the computer 40 to function as the information processing device 10. The information processing program 50 has a preprocessing process 51, a driving determination process 52, an operation determination process 53, a boarding / disembarking determination process 54, an identification process 55, a calculation process 56, and a generation process 57.
[0049] The CPU 41 reads out the information processing program 50 from the storage unit 43, expands it in the memory 42, and sequentially executes the processes of the information processing program 50. The CPU 41 executes the pre-processing process 51 to operate as the pre-processing unit 11 shown in FIG. 1. The CPU 41 executes the driving determination process 52 to operate as the driving determination unit 12 shown in FIG. 1. The CPU 41 executes the operation determination process 53 to operate as the operation determination unit 13 shown in FIG. 1. The CPU 41 executes the boarding / disembarking determination process 54 to operate as the boarding / disembarking determination unit 14 shown in FIG. 1. The CPU 41 executes the identification process 55 to operate as the identification unit 15 shown in FIG. 1. The CPU 41 executes the calculation process 56 to operate as the calculation unit 16 shown in FIG. The CPU 41 executes the generation process 57 to operate as the generation unit 17 shown in FIG. 1. As a result, the computer 40 that executes the information processing program 50 functions as the information processing device 10. The CPU 41 that executes the program is hardware.
[0050] The functions realized by the information processing program 50 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit) or the like.
[0051] Next, the operation of the information processing apparatus 10 according to the present embodiment will be described. When a moving image obtained by a drive recorder attached to a forklift is input to the information processing apparatus 10, the information processing routine shown in FIG. 16 is executed in the information processing apparatus 10. Note that the information processing routine is an example of an information processing method of the disclosed technology.
[0052] In step S10, the preprocessing unit 11 acquires the moving image input to the information processing apparatus 10, and performs preprocessing such as resizing, luminance normalization, and detection of frames with blurring on the acquired moving image.
[0053] Next, in step S20, a running determination process is executed. Here, the running determination process will be described with reference to FIG. 17.
[0054] In step S21, the running determination unit 12 calculates an optical flow from each frame included in the preprocessed moving image, and calculates a median value X in the frame of the vector indicating the optical flow.
[0055] Next, in step S22, the running determination unit 12 calculates a ratio Ratio obtained by dividing the width between the maximum value and the minimum value of the median values of the optical flow calculated for each frame in the entire moving image by the width calculated from the reference moving image. The running determination unit 12 sets a value obtained by multiplying Ratio by a predetermined value (for example, 0.15) as a threshold value TH1.
[0056] Next, in step S23, the running determination unit 12 determines whether the median value X of the optical flow is greater than or equal to the threshold value TH1 for each frame. For frames where X ≥ TH1, the process proceeds to step S24, and the running determination unit 12 assigns a flag "1" to that frame. On the other hand, for frames where X < TH1, the process proceeds to step S25, and the running determination unit 12 assigns a flag "0" to that frame. Also, the running determination unit 12 assigns a flag "0" to frames in which blurring has been detected in the preprocessing unit 11.
[0057] Next, in step S26, the moving determination unit 12 assigns metadata indicating the determination result "moving" to a predetermined number of frames (e.g., three frames) before and after the frame to which the flag "1" has been assigned. The moving determination unit 12 also assigns metadata indicating the determination result "stopped" to the remaining frames.
[0058] Next, in step S27, the moving determination unit 12 determines whether or not the distribution of the optical flow angles in the frame for the frame with the determination result "moving" satisfies a predetermined condition. For a frame that satisfies the predetermined condition, the process proceeds to step S28, where the moving determination unit 12 assigns a flag of "1" to that frame. On the other hand, for a frame that does not satisfy the predetermined condition, the process proceeds to step S29, where the moving determination unit 12 assigns a flag of "0" to that frame.
[0059] Next, in step S30, the driving determination unit 12 changes the metadata indicating the determination result "driving" that is assigned to a predetermined number of frames (e.g., 5 frames) before and after the frame to which the flag "1" is assigned, to metadata indicating the determination result "turning." In addition, the driving determination unit 12 changes the metadata indicating the determination result "driving" that is assigned to the remaining frames to metadata indicating the determination result "forward / reverse."
[0060] Next, in step S31, the traveling determination unit 12 determines whether the stopped state is TH2 or more by determining whether the number of frames (e.g., 20 frames) with the determination result "stopped" is consecutive, which is equal to or greater than a threshold value TH2 (e.g., 2 seconds). If the stopped state is TH2 or more, the process proceeds to step S32, where the traveling determination unit 12 changes the metadata of the determination result "stopped" that is assigned to each frame included in the section where the frames with the determination result "stopped" are consecutive, to metadata indicating the determination result "properly stopped". On the other hand, if the stopped state is less than TH2, the process proceeds to step S33, where the metadata of the determination result "stopped" that is assigned to each frame is changed to metadata indicating the determination result "temporarily stopped". Then, the traveling determination process ends, and the process returns to the information processing routine (FIG. 16).
[0061] Next, in step S40, an operation determination process is executed. Here, with reference to FIG. 18, the operation determination process will be described.
[0062] In step S41, the operation determination unit 13 detects a region indicating the backrest from each frame of the moving image transferred from the preprocessing unit 11. Then, the operation determination unit 13 specifies a claw movable region based on the detected region indicating the backrest and the attachment position of the camera of the drive recorder.
[0063] Next, in step S42, the operation determination unit 13 calculates the optical flow within each frame, and extracts, from the calculated optical flow, the optical flow within the claw movable region and at an angle corresponding to the movable direction of the claw. Then, the operation determination unit 13 calculates the median value X1 of the magnitudes of the extracted optical flow and the median value X2 of the magnitudes of the optical flow of the entire frame.
[0064] Next, in step S43, the operation determination unit 13 extracts, from the claw movable region, a region indicating a specific color for each frame, and calculates the change amount C of the region indicating the specific color between frames.
[0065] Next, in step S44, the operation determination unit 13 determines, for each frame, whether the difference between the median values X1 and X2 of the magnitudes of the optical flow is greater than or equal to the threshold value TH3 and the change amount C of the region indicating the specific color is greater than or equal to the threshold value TH4. For frames where |X1 - X2| ≥ TH3 and C ≥ TH4, the process proceeds to step S45, and the operation determination unit 13 assigns a flag "1" to that frame. On the other hand, for frames where |X1 - X2| < TH3 or C < TH4, the process proceeds to step S46, and the operation determination unit 13 assigns a flag "0" to that frame. Note that in this step, it may be set such that a positive determination is made when |X1 - X2| ≥ TH3 or C ≥ TH4.
[0066] Next, in step S47, the operation determination unit 13 assigns metadata indicating the determination result "operating" to a predetermined number of frames (for example, 3 frames) before and after the frame to which the flag "1" is assigned. Further, the operation determination unit 13 assigns metadata indicating the determination result "not operating" to the remaining frames. Then, the operation determination process ends and returns to the information processing routine (FIG. 16).
[0067] Next, in step S50, the boarding / alighting determination process is executed. Here, with reference to FIG. 19, the boarding / alighting determination process will be described.
[0068] In step S51, the boarding / alighting determination unit 14 detects a region indicating a person from each frame of the moving image, and specifies the coordinates within the frame of the center point of the bounding box 25 detected as the region indicating the person as the position P1 of the person. Next, in step S52, the boarding / alighting determination unit 14 specifies the average of the position P1 of the person in the entire moving image specified from each frame as the average position P2 of the person.
[0069] Next, in step S53, the boarding / alighting determination unit 14 determines for each frame whether the distance D between the position P1 of the person and the average position P2 is equal to or greater than the threshold value TH5. For frames where D ≧ TH5, the process proceeds to step S54, and the boarding / alighting determination unit 14 assigns the flag "1" to that frame. On the other hand, for frames where D < TH5, the process proceeds to step S55, and the boarding / alighting determination unit 14 assigns the flag "0" to that frame.
[0070] Next, in step S56, the boarding / alighting determination unit 14 assigns metadata indicating the determination result "alighting" to each frame included in an interval where the frames with the flag "1" are consecutive for a predetermined time (for example, 1 second) or more (for example, 10 frames). Further, the boarding / alighting determination unit 14 assigns metadata indicating the determination result "boarding" to the other frames. Then, the boarding / alighting determination process ends and returns to the information processing routine (FIG. 16).
[0071] Next, in step S60, the identification unit 15 identifies the type of driving state based on the determination result of each frame, and assigns metadata indicating the type of driving state to each frame. The identification unit 15 assigns metadata indicating dangerous driving to frames whose driving state type is "operating while turning," "operating while driving," or "insufficient stopping," thereby identifying sections in the video where dangerous driving is occurring.
[0072] Next, in step S70, the calculation unit 16 calculates a safety coefficient by dividing the number of frames in which the driving state type was identified as either "operating while turning," "operating while driving," or "insufficient stopping" by the number of frames in which the judgment result was "while riding."
[0073] Next, in step S80, the generation unit 17 doubles the speed of the video so that the playback speed of the section where dangerous driving is not occurring is faster than the playback speed of the section where dangerous driving is occurring. The generation unit 17 also processes the video so that information such as the judgment results in each of the above steps S20 to S50, the identification result in the above step S60, and the safety factor calculated in the above step S70 are displayed in association with the video. In this way, the generation unit 17 generates a processed video and transmits a file of the processed video to the user terminal. Then, the information processing routine ends.
[0074] In the above information processing routine, the processes of steps S20, S40, and S50 may be executed in a different order, or may be executed in parallel.
[0075] As described above, the information processing device according to the present embodiment determines the traveling state of the forklift and the operation state of the claw from a video captured of the driving state of the forklift. The information processing device also identifies sections in the video where dangerous driving is occurring based on the determination results of the traveling state and the operation state. The information processing device then changes the playback speed of the video based on the identification result of whether or not the section is dangerous driving. In this way, for the driving state of the forklift where not only the vehicle body is driven but also the claw is operated, sections where dangerous driving is occurring can be identified only from the video, and the playback speed can be changed depending on whether or not the section is dangerous driving. This makes it possible to support efficient driving evaluation.
[0076] In addition, the boarding and alighting status is further judged, a safety factor is calculated from the proportion of dangerous driving during boarding and alighting, and the driving status, operating status, boarding and alighting status, whether or not the driving is dangerous, the safety factor, etc. are displayed in a moving image, thereby supporting more efficient driving evaluation.
[0077] In the above embodiment, the determination of whether the vehicle is in an appropriate stop or a temporary stop is made according to the duration of the stopped state, and the case of a temporary stop is specified as dangerous driving, but the present invention is not limited to this. A stop that does not meet the appropriate stop time may be further classified in stages. For example, if a stop of 2 seconds or more is appropriate, the stopped state may be classified into a temporary stop of 1 second or more but less than 2 seconds, and an instantaneous stop of less than 1 second.
[0078] In addition, in the above embodiment, an example was described in which the driving condition of a forklift was evaluated, but the disclosed technology can be applied to any driving condition of a mobile body that includes a movable part that moves relative to the mobile body main body.
[0079] In the above embodiment, the information processing program is stored (installed) in the storage unit in advance, but the present invention is not limited to this. The program according to the disclosed technology can also be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.
[0080] The following supplementary notes are further disclosed regarding the above embodiment.
[0081] (Appendix 1) determining a running state of a moving object including a movable part that is movable relative to a main body of the moving object and an operation state of the movable part from a moving image captured of the driving state of the moving object; determining whether or not the moving object is being driven dangerously in the moving image based on the determination results of the traveling state and the operation state; An information processing program for causing a computer to execute processing including the above.
[0082] (Appendix 2) 2. The information processing program according to claim 1, further comprising changing a playback speed of the video based on whether or not the section is a section where dangerous driving is occurring.
[0083] (Appendix 3) The information processing program described in Appendix 2, wherein the process of changing the playback speed of the video includes making the playback speed of the section where the dangerous driving is not occurring faster than the playback speed of the section where the dangerous driving is occurring.
[0084] (Appendix 4) An information processing program as described in any one of Supplementary Note 1 to Supplementary Note 3, which identifies a section of the video in which the driving state is determined to be that the moving body is moving and the operation state is determined to be that the movable part is being operated as a section in which the dangerous driving is occurring.
[0085] (Appendix 5) The information processing program according to any one of Supplementary Note 1 to Supplementary Note 4, wherein the process of determining the driving state includes determining whether the moving object is driving or stopped for each frame based on a comparison between the magnitude of the optical flow calculated from each frame included in the moving image and a first threshold value.
[0086] (Appendix 6) An information processing program as described in Appendix 5, which multiplies the first threshold by a ratio of the optical flow statistical value calculated from each frame of a target video to the optical flow statistical value calculated from each frame of a reference video.
[0087] (Appendix 7) 7. The information processing program according to claim 5 or 6, further determining, for a frame determined to be moving, whether the moving body is moving forward, backward, or turning based on the distribution of optical flow angles within the frame.
[0088] (Appendix 8) An information processing program as described in any one of Supplementary Note 1 to Supplementary Note 7, which, when it is determined that the moving body is stopped as the driving state, identifies a section of the video image in which the stopped state continues for a duration of a second threshold value or less as a section in which the dangerous driving is occurring.
[0089] (Appendix 9) The information processing program according to any one of Supplementary Note 1 to Supplementary Note 8, wherein the process of determining the operation state includes detecting a first region indicating the movable part from each frame of the video, comparing a difference between an optical flow in the detected first region and an optical flow in the entire frame with a third threshold value, and detecting a second region indicating a specific color of the movable part or a marker attached to the movable part from each frame of the video, and determining for each frame whether the movable part is being operated or stopped based on at least one of the following: detecting a first region indicating the movable part from each frame of the video, and comparing an amount of variation in the second region between frames with a fourth threshold value.
[0090] (Appendix 10) An information processing program as described in any one of Appendix 1 to Appendix 9, for causing the computer to execute a process further including displaying the judgment results of the driving state and the operation state, and the identification result of the section where the dangerous driving is occurring, in correspondence with the moving image.
[0091] (Appendix 11) Further, a state of a passenger getting on and off the vehicle is determined; calculating a ratio of a section in which the dangerous driving is occurring to a section in which the occupant is determined to be riding on the moving object in the video image as a safety factor; The safety factor is further displayed in association with the moving image. 11. The information processing program according to claim 10.
[0092] (Appendix 12) The information processing program described in Appendix 11, wherein the process of determining the boarding / disembarking state includes detecting a third area indicating the occupant from each frame of the video, and determining for each frame whether the occupant is boarding or disembarking from the moving body based on a comparison between the difference between the position of the third area in each frame and the average position of the third area in the entire video and a fifth threshold.
[0093] (Appendix 13) The information processing program described in Appendix 12, wherein the process of changing the playback speed of the video includes making the playback speed of the section where the occupant is riding on the moving body faster than the playback speed of the section where the occupant is dismounting from the moving body.
[0094] (Appendix 14) a determination unit that determines a running state of a moving body and an operation state of a movable part that is movable relative to a main body of the moving body from a moving image captured of the driving state of the moving body; an identification unit that identifies whether or not dangerous driving of the moving object is being performed in the moving image based on a result of the determination of the traveling state and the operation state; An information processing device comprising:
[0095] (Appendix 15) 15. The information processing device according to claim 14, further comprising a generating unit that changes a playback speed of the video based on the section in which the dangerous driving is occurring.
[0096] (Appendix 16) The information processing device according to claim 15, wherein the generation unit makes a playback speed of the section where the dangerous driving is not occurring faster than a playback speed of the section where the dangerous driving is occurring.
[0097] (Appendix 17) The information processing device according to claim 15 or 16, wherein the identification unit identifies a section of the video in which the driving state is determined to be that the moving body is driving and the operation state is determined to be that the movable part is being operated, as a section in which the dangerous driving is occurring.
[0098] (Appendix 18) The information processing device according to any one of Supplementary Note 15 to Supplementary Note 17, wherein the process of determining the running state by the determination unit includes determining for each frame whether the moving object is running or stopped based on a comparison between the magnitude of the optical flow calculated from each frame included in the moving image and a first threshold value.
[0099] (Appendix 19) determining a running state of a moving object including a movable part that is movable relative to a main body of the moving object and an operation state of the movable part from a moving image captured of the driving state of the moving object; determining whether or not the moving object is being driven dangerously in the moving image based on the determination results of the traveling state and the operation state; An information processing method in which a computer executes a process including the steps of:
[0100] (Appendix 20) determining a running state of a moving object including a movable part that is movable relative to a main body of the moving object and an operation state of the movable part from a moving image captured of the driving state of the moving object; determining whether or not the moving object is being driven dangerously in the moving image based on the determination results of the traveling state and the operation state; A storage medium storing an information processing program for causing a computer to execute a process including the above. [Explanation of symbols]
[0101] 10. Information processing device 11 Pretreatment section 12 Driving judgment unit 13 Operation judgment section 14 Boarding and alighting determination section 15 Specific section 16 Calculation section 17 Generation part 21 Area showing backrest 22 Claw movable area 23 Backrest 24 Marker 25 Bounding Box 27 Frame 28 Edited Videos 29 Dangerous driving condition indication 30 Judgment result display 31 Area display showing people 32 Safety factor display 33x speed display 40 Computer 41 CPU 42 Memory 43 Storage section 49 Storage medium 50 Information Processing Program
Claims
1. determining a running state of a moving object including a movable part that is movable relative to a main body of the moving object and an operation state of the movable part from a moving image captured of the driving state of the moving object; An information processing program for causing a computer to execute a process including identifying whether or not dangerous driving of the moving object is being performed in the moving image based on a determination result of the traveling state and the operation state, The process of determining the running state includes determining whether the moving object is running or stopped for each frame based on a comparison between a magnitude of an optical flow calculated from each frame included in the moving image and a first threshold value, and further determining, for a frame determined to be running, whether the moving object is moving forward, backward, or turning based on a distribution of angles of the optical flow within the frame. Information processing program.
2. An information processing program as described in claim 1, comprising changing the playback speed of the video based on whether or not a section of the video is a section where the dangerous driving is taking place.
3. The information processing program according to claim 2 , wherein the process of changing the playback speed of the video includes making the playback speed of the section in which the dangerous driving is not occurring faster than the playback speed of the section in which the dangerous driving is occurring.
4. An information processing program as claimed in any one of claims 1 to 3, which identifies a section of the video image in which the driving state is determined to be that the moving body is moving and the operation state is determined to be that the movable part is being operated as a section in which the dangerous driving is taking place.
5. The information processing program according to claim 1 , further comprising multiplying the first threshold by a ratio of an optical flow statistical value calculated from each frame of a target video to an optical flow statistical value calculated from each frame of a reference video.
6. An information processing program as described in any one of claims 1 to 5, wherein when the driving state is determined to be that the moving body is stopped, a section of the video image in which the duration of the stopped state is less than a second threshold value is identified as a section in which the dangerous driving is occurring.
7. The process of determining the operation state includes detecting a first area indicating the movable part from each frame of the video, comparing the difference between the optical flow in the detected first area and the optical flow in the entire frame with a third threshold value, and detecting a second area indicating a specific color of the movable part or a marker attached to the movable part from each frame of the video, and determining for each frame whether the movable part is being operated or stopped based on at least one of the following: detecting a first area indicating the movable part from each frame of the video, and comparing the amount of variation in the second area between frames with a fourth threshold value.
8. An information processing program as described in any one of claims 1 to 7, for causing the computer to execute a process further including displaying the results of the determination of the driving state and the operation state, and the results of identifying the section where the dangerous driving is occurring, in correspondence with the moving image.
9. Further, a state of a passenger getting on and off the vehicle is determined; calculating a ratio of a section in which the dangerous driving is occurring to a section in which the occupant is determined to be riding on the moving object in the video image as a safety factor; The safety factor is further displayed in association with the moving image. The information processing program according to claim 8.
10. The information processing program described in claim 9, wherein the process of determining the boarding / disembarking state includes detecting a third area indicating the occupant from each frame of the video, and determining for each frame whether the occupant is boarding or disembarking from the moving body based on a comparison between the difference between the position of the third area in each frame and the average position of the third area in the entire video and a fifth threshold value.
11. The information processing program according to claim 10 , wherein the process of changing the playback speed of the moving image includes making the playback speed of the section in which the occupant is riding on the moving body faster than the playback speed of the section in which the occupant is dismounting from the moving body.
12. a determination unit that determines a running state of a moving body and an operation state of a movable part that is movable relative to a main body of the moving body from a moving image captured of the driving state of the moving body; and a determination unit that determines whether or not dangerous driving of the moving object is being performed in the moving image based on a determination result of the traveling state and the operation state, the determination unit determines whether the moving object is moving or stopped for each frame based on a comparison between a magnitude of an optical flow calculated from each frame included in the moving image and a first threshold, and for a frame determined to be moving, determines whether the moving object is moving forward, backward, or turning based on a distribution of angles of the optical flow within the frame; Information processing device.
13. determining a running state of a moving object including a movable part that is movable relative to a main body of the moving object and an operation state of the movable part from a moving image captured of the driving state of the moving object; An information processing method in which a computer executes a process including identifying whether or not dangerous driving of the moving object is being performed in the moving image based on a determination result of the traveling state and the operation state, The process of determining the running state includes determining whether the moving object is running or stopped for each frame based on a comparison between a magnitude of an optical flow calculated from each frame included in the moving image and a first threshold value, and further determining, for a frame determined to be running, whether the moving object is moving forward, backward, or turning based on a distribution of angles of the optical flow within the frame. Information processing methods.
Citation Information
Patent Citations
System for warning dangers during operation of e.g. fork-lift truck, has portable radio frequency identification sending- and receiving- device releasing warning signal when warning device is provided in danger area of mobile machine
DE102009030953A1
Driving force control device for industrial vehicle
JP2001116136A
Vehicle dispatching system and method
JP2009015684A
Background difference processing device, background difference processing method and computer program
JP2012104053A
Video monitoring system, video processing server, and method for browsing and delivering monitored video
JP2014192776A
