Playback device and playback method, as well as their programs, recording device and control method for the recording device, etc.

Abstract

The present invention provides a playback device that allows users to view video processed based on acquired information. [Solution] A playback device for playing back recorded video data processes the recorded video data based on acquired information and controls the display of the video based on the processed video data. A playback window 130 is displayed on the display screen of the playback device, and the video recorded by the drive recorder is displayed in the video display area 141 of the playback window 130. The frame 143 of the user's field of view is displayed, and the video is processed and displayed so that the area outside the field of view 142 is blurred.

Description

Technical Field

[0001] This invention relates to, for example, a playback device, a playback method, its program, a recording device, a control method for the recording device, and the like.

Background Art

[0002] There is known an in-vehicle video recording device that captures video of the vehicle surroundings using a camera installed in a vehicle and records the surrounding video, vehicle speed, etc. when an impact is applied to the vehicle due to an event (incident) such as sudden braking, sudden steering, or a collision. There is also known a playback device that plays back the video captured by such a drive recorder. (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the invention described in Patent Document 1, it was only possible to play back the captured video such as the video before and after the occurrence of an event.

[0005] This invention aims, for example, to increase the likelihood of displaying images that are more suitable for the user than before. The purpose of this invention is not limited to this, and the applicant intends to obtain rights through divisional applications, amendments, etc., for configurations that aim to obtain effects from the parts of the configuration disclosed in this specification and drawings, etc. For example, there are problems disclosed in this specification that can be reinterpreted as "the problem is that..." where "can be done". Each problem is described independently, and the applicant intends to obtain rights to the configurations that solve these problems individually through divisional applications, amendments, etc. Even if a problem is implicitly understood from the description in the specification, the applicant intends to include a part of the configuration described in this specification in the claims through amendment or divisional application. [Means for solving the problem]

[0006] (1) A playback device for playing back recorded video data, which may process the recorded video data based on acquired information and perform control to display a video based on the processed video data.

[0007] In this way, users can view not the recorded video itself, but a video processed based on the acquired information. By processing the video in a way that is suitable for the user viewing it, it is possible to obtain a video that is suitable for the user. Rather than simply processing the recorded video, the recorded video data is processed based on the acquired information, which increases the likelihood of displaying a video that is more suitable for the user viewing it. For example, in a playback device that plays back video data recorded in a vehicle, it is preferable to process the recorded video data based on information acquired in the vehicle and to control the device to display a video based on the processed video data. In particular, it is preferable to process the recorded video data based on information acquired in the vehicle at the same time as the video data recorded in the vehicle and to control the device to display a video based on the processed video data. The same time period may be within a predetermined time range, which may be a few seconds, for example, but it is especially preferable for the same time period to be simultaneous. The information to be acquired should preferably be information that changes in accordance with the driver's actions, and especially information related to the driver's own behavior. Information related to the driver's own state should be information that shows a relationship between the driver's own state and the video recorded by the vehicle. It would be good to do so. The relationship can be an objective relationship, but it is especially good to make it a subjective relationship of the driver themselves. A subjective relationship would be one in which the driver thinks things looked like, and for example, it would be good to make it the information necessary to make the driver infer "how things looked." However, it is best to acquire information other than the car's speed.

[0008] The recorded video data may be, for example, video data recorded by a recording device, or video data recorded on a memory card such as an SD (Secure Digital) card, microSD card, or XQD® card. The acquired information may be, for example, information input by the user viewing the video. By having the user input information and the playback device acquire that information, the likelihood of displaying video suitable for the user viewing the video increases, so that processing appropriate for the user viewing the video is performed. The acquired information may also be predetermined information. Information may be acquired by reading such predetermined information. The processing may be, for example, simply a change in the video data before and after processing.

[0009] (2) The processing may include a processing that displays the user's field of view in the video.

[0010] This makes it easier to understand the user's field of view in the video. Knowing the user's field of view makes it easier to distinguish between parts of the image that the user cannot see and parts that they can see. The user's field of view may be fixed, determined according to the user's age, or measured and determined for each user. In particular, the processing should be a process that displays the user's field of view in the video based on the acquired information. In particular, it is good to perform a processing process that displays the user's field of view in the video based on the range based on the information for adjusting the field of view input from the field of view input means and the acquired information. Displaying the field of view in the video means displaying it in a manner that makes it possible to distinguish between what is inside and outside the field of view. For example, the field of view may be displayed by enclosing it in a frame, or the area outside the field of view may be filled in or blurred. The video may be, for example, video recorded by a recording device using a wide-angle camera, or video recorded by a recording device using a 360-degree camera.

[0011] (3) If there is an object that requires attention in the video outside the user's field of view, it is advisable to notify the user accordingly.

[0012] This makes it easier for the user to understand that there are objects that require attention in areas they cannot see. Whether or not there are objects that require attention outside the user's field of view can be determined, for example, by detecting objects in each frame of the recorded video using image recognition, and determining that an object exists if the range of the detected object in that frame is within the range outside the field of view, and does not exist if it is not. In particular, the object detection process should preferably omit at least a part of the object search by image recognition within the field of view in the frame's video, and preferably not be done at all. It is preferable to perform the object search by image recognition only in the range outside the field of view in the frame's video. For example, if the video is acquired by a recording device, the user will know that there are objects that require attention in areas they cannot see, increasing the likelihood that the user will be able to pay attention to areas that are not visible when driving by moving their head. The objects that require attention may be, for example, pedestrians, bicycles, cars, motorcycles, etc. To inform the user of this, it is preferable to display the location in the video.

[0013] (4) The processing method displays the user's gaze during the recording of the video data in the video. It is preferable to use this processing method.

[0014] This makes it easier to understand where the user was looking when recording video data. In particular, the processing should be a process that displays the user's gaze on the video at the time of recording based on the acquired information. In particular, the user's gaze should be displayed on the video based on the user's gaze input from an input means for inputting the user's gaze at the time of recording video data and the previously acquired information. For example, the user's eye movements should be recorded when recording video data, and the user's commands should be displayed on the video based on that record. The gaze is, for example, the direction the user is looking, and may also be called their line of sight. Displaying the gaze on the video means, for example, displaying the gaze sequentially as points on the video, or displaying the trajectory of the gaze on the video. The user's gaze can be detected, for example, by photographing the user with a shooting means when recording video data and detecting the gaze from the captured image obtained by that shooting means, or by having the user wear a gaze detection device such as glasses for gaze detection and detecting the gaze using the gaze detection device such as glasses.

[0015] (5) The processing described above may be a processing that moves the field of view in the direction the user is facing when recording the video data and displays it in the video.

[0016] This method makes it clear which direction the user is facing when recording video data, and makes it easier to understand the field of view in that direction. It also makes it easier to understand which part of the video was visible when recording video data. In particular, it is preferable to acquire information indicating the direction detected by a detection means that detects the direction the user is facing, and then move the field of view based on that acquired information to display it on the video. The field of view can be moved by processing the frames so that the range inside and outside the field of view can be distinguished at regular intervals. For example, the field of view can be moved relatively, or it can be moved relatively by moving the video.

[0017] (6) For example, the processing may be a processing that displays in the video the area that the user has not seen for a certain period of time when recording the video data.

[0018] This makes it easier to identify areas that the user has not looked at for a certain period of time. In particular, it is preferable to obtain information from a detection means that detects areas that the user has not looked at for a certain period of time, and then process the image to display the areas that the user has not looked at for a certain period of time based on the obtained information. It is preferable to detect the user's gaze and then detect the areas that the user has not looked at for a certain period of time based on the detected user gaze information and the information for a certain period of time. The user's gaze can be detected, for example, by photographing the user with a shooting means when recording video data and detecting the gaze from the captured image obtained by that shooting means, or by having the user wear a gaze detection device such as glasses for gaze detection and detecting the gaze using the gaze detection device. The period of time can be, for example, the time while the video data is recorded by a recording device, the time while the vehicle is stopped at an intersection, while turning right or left at an intersection, or while changing lanes. Displaying the areas that have not been looked at for a certain period of time for a certain period of time means, for example, by filling in or blurring the areas that have not been looked at for a certain period of time.

[0019] (7) For example, the processing may include a process that displays on the video whether the user's line of sight at the time of recording the video data matches the direction the user should be looking.

[0020] This makes it easier to determine whether the user is looking in the direction they should be looking or not. In particular, information is obtained from an input device that provides information indicating whether the user's gaze during video data recording matches the direction the user should be looking, and based on the obtained information, it is determined whether the user's gaze during video data recording matches the direction the user should be looking. It is preferable to perform a processing step that displays on the video whether or not this is the case. Alternatively, it is preferable to perform a processing step that obtains information input from an input means that inputs information indicating the user's gaze at the time of recording the video data and information input from an input means that inputs information that the user should look at, and based on the obtained information, displays on the video whether or not the user's gaze at the time of recording the video data matches the direction the user should look at. Alternatively, it is preferable to perform a processing step that obtains information from a detection means that detects whether or not the user's gaze at the time of recording the video data matches the direction the user should look at, and based on the obtained information, displays on the video whether or not the user's gaze at the time of recording the video data matches the direction the user should look at. For example, when video data is being recorded by a recording device, the direction the user should look at is predetermined according to the surrounding circumstances. It is also preferable to calculate the degree of agreement between the user's gaze and the direction the user should look at.

[0021] (8) For example, when the video data is recorded by a recording device, the direction the user should look in, in determining whether the user's line of sight and the direction the user should look in coincide at the time the video data is recorded, should vary depending on the operation of the car while the video data is being recorded.

[0022] By doing so, it becomes easy to understand that the direction in which the user looks varies according to the driving of the vehicle or the operation of driving the vehicle by the user at the time of recording video data. Since the direction in which the user looks varies according to the driving of the vehicle or the operation of driving the vehicle by the user at the time of recording video data, it becomes easy to understand whether or not the user is looking in the direction that should be looked at according to the driving of the vehicle, or whether or not the user is looking in the direction that should be looked at according to the operation of driving the vehicle. In particular, it is preferable to determine the direction in which the user should look based on information from input means for inputting information about the driving of the vehicle or the operation of driving the vehicle by the user at the time of recording video data. The direction to be looked at, for example, when the operation of driving the vehicle is going straight, should be a direction of looking substantially forward, and when changing lanes, should be a direction of looking at the rearview mirror, the door mirror of the driving lane on the side where the lane is changed, the side window on the side where the lane is changed, and the like. Also, as an operation of driving the vehicle, when the steering wheel is operated relatively largely after operating the wiper for a right turn or a left turn, the direction should be to look diagonally forward and diagonally backward on the side where the steering wheel is operated, and when the steering wheel is operated relatively slightly after operating the wiper for a lane change, the direction should be to look at the rearview mirror, the door mirror of the driving lane on the side where the lane is changed, the side window on the side where the lane is changed, and the like.

[0023] (9) For example, the processing may be a processing for obtaining information indicating whether or not the order in which the user looked at the time of recording video data matches the order that should be looked at in advance, and performing the processing to be displayed on the video based on the obtained information.

[0024] By doing so, it becomes easy to see whether the order in which the user viewed matches the order that should be viewed in advance. In particular, based on the information from the detection means that detects whether the order in which the user viewed matches the order that should be viewed in advance at the time of recording the video data, information indicating whether the order in which the user viewed matches the order that should be viewed in advance may be displayed on the video. Regarding the order in which the user viewed at the time of recording the video data, for example, it can be found by analyzing the video data obtained by shooting the user who was driving a vehicle and detecting the user's line of sight in time series. Regarding the order that should be viewed in advance, it can be found based on the information input from the input means that inputs the information indicating that order. For example, when playing back the recorded video data, an instructor or the like may input from the input means the order in which the video represented by the played-back video data should be viewed. The order that should be viewed in advance may vary depending on the situation at the time of recording, for example. For example, if the video data is of the time when a vehicle starts, an example of the order that should be viewed is to look ahead, look at the rearview mirror or behind, look at the right door mirror or fender mirror, and look at the right window. The degree of coincidence between the order in which the user viewed and the order that should be viewed in advance may be calculated.

[0025] (10) For example, the processing may be processing for displaying on the video the location that the user should view.

[0026] By doing so, it becomes easy to see from the video the location that the user should view. In particular, it is advisable to obtain the information input from the input means that inputs the information indicating the location that the user should view, and perform processing for displaying on the video the location that the user should view based on the obtained information. For example, if the video data is a video seen from a driver during driving, it becomes easy to see the locations that the driver should view, and it is often useful for driving guidance. The processing may be, for example, when the location to view the rearview mirror is concerned, marking the video portion of that rearview mirror, or when the location to view the door mirror is concerned, marking the video portion of that door mirror.

[0027] (11) For example, if the user is not looking at a part of the video data that they should be looking at, the image of that scene may be displayed.

[0028] This approach makes it easier to identify situations where the user is not looking at what they should be looking at. In particular, when recording video data, it is advisable to display the screen of a scene where the user is not looking at what they should be looking at, based on information entered through an input device that provides information indicating the user is not looking at what they should be looking at. Whether or not the user is looking at what they should be looking at can be determined using gaze information obtained from a detection device that detects the user's gaze. For example, if the video data represents footage of the user driving, it becomes easier to identify situations where the user is not looking at what they should be looking at, which is likely to help improve driving skills. Also, if the video data represents footage of the user driving, evaluation can be performed based on images of scenes where the user is not looking at what they should be looking at.

[0029] (12) For example, the display speed of the portion of the video where there is no change in the movement of the subject may be set to be faster than the display speed of the portion of the video where there is a change in the movement of the subject.

[0030] This increases the likelihood of quickly viewing the entire video. In particular, to determine whether a video section shows no change in the subject's movement or does, a detection method can be used to detect whether a section shows no change in the subject's movement or does. This detection method should detect the difference between images between frames, and if the difference is less than a certain level, it should be considered that there is no change in the subject's movement, and if the difference is greater than a certain level, it should be considered that there is a change in the subject's movement. For example, if the video section showing no change in the subject's movement is not considered a relatively important part, and the video section showing change in the subject's movement is considered a relatively important part, then it becomes more likely that viewers will be able to view the less important parts in less time and view the relatively important parts in more detail. The video section showing no change in the subject's movement includes not only cases where there is no change in the subject's movement at all, but also cases where there is little change in the subject's movement, or where the change in the subject's movement is below a certain level. The video section showing change in the subject's movement includes not only cases where there is even a slight movement of the subject, but also cases where the change in the subject's movement is above a certain level.

[0031] (13) For example, video data is recorded by a recording device, and the user's evaluation of the car's driving is performed on the portion of the video in which there is a change in the movement of the subject.

[0032] This makes it easier for users to evaluate the car's driving in video segments where the subject's movement changes. In particular, to determine whether a video segment shows no change in the subject's movement or does, a detection means can be used to detect whether the video segment shows no change in the subject's movement or does. That detection means is The system detects the difference between images between frames, assuming that if the difference is less than a certain level, there is no change in the subject's movement, and if the difference is greater than a certain level, there is a change in the subject's movement. In particular, the evaluation of driving should be performed by analyzing the video, and an evaluation method should be used that gives a high rating to driving that closely follows predetermined traffic rules, and a low rating to driving that closely disregards predetermined traffic rules. For example, in driving evaluation, locations where the subject's movement changes, such as when turning right, left, or changing lanes, can be considered more important than locations where the vehicle is stopped at a traffic light and there is no change in the subject's movement. Driving evaluation can be performed in such situations.

[0033] (14) For example, the video data is recorded by a recording device, and the video is displayed based on the portion of the video that includes the customer.

[0034] This makes it easier to view footage that includes passengers. To determine if a portion of the footage includes passengers, the video data can be analyzed, and if a person other than the driver is detected inside the vehicle, that portion of the footage should be considered to include passengers. For example, if recording devices are installed in commercial vehicles such as buses and taxis, and video data is recorded by such devices, it becomes possible to display footage of those commercial vehicles that includes passengers.

[0035] (15) For example, based on keywords that describe the situation in the video portion, it is possible to find the video portion related to those keywords from within the video.

[0036] This method makes it easier to find video segments related to the keyword within the video. In particular, it is good to detect the keyword entered from an input means that inputs a keyword describing the situation of the video segment, and then use an analysis means to analyze the video and find the video segment related to the detected keyword. The situation of the video segment can be an abstract situation such as bright or dark, or a situation that includes a specific subject such as a bullet train or a red car.

[0037] (16) For example, the video data is recorded by a recording device, and based on keywords that describe the condition of the car at the time the video data was recorded, it is possible to find the video portion related to those keywords from the video.

[0038] This approach increases the likelihood of finding video segments related to keywords describing the car's condition at the time of recording. The system detects keywords entered into an input method for recording video data, and then uses an analysis method to analyze the video and find the segments related to the detected keywords. Keywords describing the car's condition could include, for example, "the car accelerated," "the car reached a certain speed," or "the turn signal was activated."

[0039] (17) For example, the video data may be recorded by a recording device and a warning may be issued if the user is not looking in the direction they should be looking while driving the car.

[0040] This approach makes it easier to warn the user if they are not looking in the correct direction. In particular, it is advisable to warn based on information entered by an input device that provides information indicating that the user was not looking in the correct direction. Such an input device should display the user's gaze, detected by a detection device that detects the user's gaze while driving, on the played-back video, and input information indicating that the displayed gaze was determined not to be looking in a predetermined direction. Warnings could include, for example, adding a warning mark to the displayed video, coloring part or all of the displayed video with a specific color, or outputting an alarm. That's good.

[0041] (18) For example, the video data is recorded by a recording device, and the processing is a processing that distinguishes between the range visible to the user and the range not visible to the user while the user is driving the car, and displays the video accordingly.

[0042] This makes it easier to distinguish between areas the user can see and areas they cannot see. In particular, it is preferable to perform a processing operation that distinguishes between areas the user can see and areas they cannot see based on information input from an input means that provides information indicating the areas the user can see, and then displays the image accordingly. For example, areas that the user cannot see can be blurred, filled with a semi-transparent color, or completely filled in.

[0043] (19) For example, the processing may be performed on the recorded video data in such a way that the display range of the video is changed according to the orientation of the device that displays the video based on the processed video data.

[0044] This makes it easier to view the image according to the orientation of the display device. It is preferable to change the display range of the image based on information from a detection means that detects the orientation of the display device based on the processed image data. For example, it is preferable to record the entire 360-degree range during recording and then display the image corresponding to the orientation of the display device.

[0045] (20) For example, the video data is recorded by a recording device, and the processing is performed on the recorded video data in such a way that the display range of the video is changed with the direction of travel of the car as the reference direction, according to the orientation of the device that displays the video based on the processed video data.

[0046] In this way, the display range of the image, with the direction of the car's movement as the reference direction, is changed, increasing the likelihood that the image can be viewed according to the orientation of the display device. It is preferable to change the display range of the image based on information from a detection means that detects the orientation of the display device based on the processed image data. This allows viewing not only of the image in the direction of the car's movement while it is in motion, but also of images in directions other than the direction of movement.

[0047] (21) For example, information regarding the condition of the car at the time the video data was recorded may be displayed.

[0048] This makes it easier to understand the car's status. The information regarding the car's status may include, for example, the car's speed, acceleration, and position. In particular, it is preferable to display the information on the display screen of the display device by a display control means based on the information input from an input means that inputs information regarding the car's status when recording video data. The information regarding the car's status may be displayed overlaid on the video, or it may be displayed in a separate area from the area where the video is displayed. For example, the information regarding the car's status may be displayed in the area above, below, to the left, or above the video.

[0049] (22) For example, video data is recorded by a recording device, and an image of a map showing the range of movement of the car at the time the video data was recorded is displayed, and at a specified location on the displayed map image, a video based on the video data recorded by the recording device is displayed.

[0050] In particular, the vehicle's movement range during video data recording is detected by a detection means, image data representing a map image of the detected movement range is acquired by an acquisition means, the map image represented by the acquired image data is displayed on the display screen of the display device by a display control means, a location on the displayed map image is specified by a specification means, video data recorded at the specified location is found in the video data by an image analysis means, and the found video is displayed on the display screen of the display device by the display control means. In this way, it becomes easier to view the video at the specified location from the map image. Even if the time of the video you want to see is unknown, if the location is known, you can view the video at that location.

[0051] (23) For example, it is preferable to display at least one of the following: an image of the displayed map based on video data recorded by the user's recording device at a specified location within the displayed map image, and an image of the displayed map based on video data recorded by a third party's recording device at a specified location within the displayed map image.

[0052] In this way, it becomes more likely that users can view not only video data recorded by their own recording device, but also video data recorded by a third party's recording device. In particular, a command is sent from the transmitting means to a server or other location where video data recorded by a third party's recording device is stored at a specified location on the displayed map image. The receiving means receives the video data sent from the server or other location in response to the command, and the display control means displays the video based on the video data recorded by the third party's recording device on the display screen of the display device. Both the video based on the video data recorded by the user's recording device and the video based on the video data recorded by the third party's recording device may be displayed simultaneously, or they may be displayed in a switchable manner. The video based on the video data recorded by the third party's recording device may be from the same season as the video data recorded by the user's recording device, or from a different season. Furthermore, video footage based on video data recorded by a third-party recording device may depict the same weather conditions as when the video data was recorded by the user's recording device, or it may depict different weather conditions.

[0053] (24) For example, the name of the recorded video data, including the location where it was filmed, may be displayed as the name that identifies the recorded video data.

[0054] In this way, a name including the shooting location can be used as a name to identify the video data. This makes it easier to find video data at a specific location. In particular, it is preferable to detect the shooting location from data indicating the recording location of the recorded video data using a detection means, change the name including the detected shooting location to a name that identifies the recorded video data using a renaming means, and then display the changed name on the display screen of the display device using a display control means as the name that identifies the recorded video data. In particular, it is preferable to change the file name that stores the recorded data to a name that includes the shooting location of the recorded video data using a renaming means. For example, the file name of the video file that stores the video data may include the shooting location. The name of the shooting location itself may also be used as the name that identifies the video data.

[0055] (25) For example, if multiple video data are recorded, a name that identifies the video data of the same shooting location may be displayed together.

[0056] This method makes it easier to determine whether the video data is from the same shooting location. In particular, it is advisable to control the display device using a display control means to group together the names used to identify video data from the same shooting location and display them consecutively on the display screen. Locations with many shaded areas can be considered everyday shooting locations such as commuter or school routes, while locations with few identical shooting locations can be considered non-everyday shooting locations such as travel destinations. This increases the likelihood of easily selecting video data from everyday shooting locations or non-everyday shooting locations.

[0057] (26) For example, it is a good idea to prioritize displaying names that appear less frequently.

[0058] This method allows for the display of less frequently occurring names as a priority, making it easier to select the corresponding video data. In particular, it is desirable to control the display device using a display control means to prioritize the display of less frequently occurring names on the display screen. A name can be judged as less frequently occurring if its frequency of occurrence is below a predetermined threshold, for example. Prioritizing display means, for example, if multiple identifying names are displayed in a list format in a vertical direction, they should be displayed at the top, and if they are displayed sequentially, they should be displayed in an earlier order.

[0059] (27) For example, it is a good idea to display an image of a map of the shooting location.

[0060] In this way, in addition to the video based on the video data, there is a higher probability that a map of the filming location can also be viewed. In particular, it is preferable to use a search means to find an image of the map of the filming location based on the data indicating the location of the filming location, and then use a display control means to display the found map image on the display screen of the display device.

[0061] (28) For example, the video data is recorded by a recording device, and it is preferable to display a map image that shows the range of movement of the car at the time the video data was recorded, and to display information regarding the operation of the car at the time the video data was recorded on the map image.

[0062] This makes it easier to understand what kind of car operations were performed at the location shown in the map image. By looking at the information about car operations displayed on the map image, it becomes possible to understand what kind of car operations are likely to be required when driving a car in the location represented by the map image. It is preferable to use a search means to find the map image representing the range of travel of the car from the data indicating the range of travel, a display control means to control the display device to display the found map image on the display screen, input information about car operations from the input means, and the display control means to control the display device to display the input information on the map image. Car operations can be, for example, using the turn signal, making a sudden turn, or making a sudden brake.

[0063] (29) For example, the video data is recorded by a recording device, and the video is displayed based on the video data after the processing, as well as information based on a comparison between the driving of a third party and the driving of the user at the location where the video data was recorded.

[0064] This makes it easier to compare the driving of a third party with the driving of the user. This comparison increases the likelihood that the user can objectively evaluate their own driving skills compared to those of a third party. In particular, it is preferable to input data indicating the location where the video data was recorded from an input means, acquire information indicating the driving of a third party at the recording location indicated by the input data using an acquisition means, use the acquired information about the third party's driving to perform a comparison between the user's driving and the third party's driving using a comparison means, and control the display device using a display control means to display information based on the comparison on the display screen. The information based on the comparison between the third party's driving and the user's driving may include, for example, the ranking of the third party's driving and the ranking of the user's driving, or the average evaluation of the third party's driving and the evaluation value of the user's driving.

[0065] (30) For example, the video data is recorded by the user's recording device, and it is preferable to display the video based on the processed video data and the video based on the video data recorded in the recording device in accordance with the operation of a third party at the location where the video data is recorded.

[0066] This makes it easier to compare video footage of the user's driving with video footage of a third party's driving. By watching a third party's driving, the user is more likely to understand the strengths and weaknesses of their own driving. In particular, it is preferable to input data indicating the location where the video data is recorded from an input means, acquire video data recorded in accordance with the third party's driving at the recording location indicated by the input data using an acquisition means, and control the display device using a display control means to display the video based on the acquired video data recorded in accordance with the third party's driving, as well as the video obtained in accordance with the user's driving, on the display screen. The location where the video data is recorded is, for example, the location where video data obtained from the user driving a car equipped with a recording device is recorded.

[0067] (31) For example, information obtained by at least one of vehicle-to-infrastructure communication and vehicle-to-vehicle communication is recorded in association with video data, and control is performed to display the information obtained by at least one of vehicle-to-infrastructure communication and vehicle-to-vehicle communication in relation to the video based on the processed video data.

[0068] This makes the information obtained through at least one of the vehicle-to-infrastructure communication and vehicle-to-vehicle communication easier to understand. In particular, it is preferable to read the information obtained through at least one of the vehicle-to-infrastructure communication and vehicle-to-vehicle communication using a reading means, and to control the display device using a display control means to display the read information on a display screen. The information obtained through at least one of the vehicle-to-infrastructure communication and vehicle-to-vehicle communication may be, for example, information obtained based on ITS (Intelligent Transport Systems). The information obtained through vehicle-to-infrastructure communication may be, for example, information transmitted from roadside devices installed on the road, and the information obtained through vehicle-to-vehicle communication may be information communicated between vehicles. Displaying in relation to the video means that it is sufficient if it is related to the video, and it may be displayed on the video, displayed in a separate area from the video, or displayed when the video is being displayed.

[0069] (32) For example, the processing process may be a processing process that corrects to the memory color.

[0070] This method makes it easier to obtain images based on memory colors. In particular, it is good to use this as a correction method to adjust the image to match memory processing. For example, it increases the likelihood of obtaining an image that is relatively close to what the user remembers. Memory colors can be defined as, for example, the colors that the user remembers as an image, or the colors that they remember for a specific object. Correction to memory colors can be done, for example, by changing the brightness, hue, saturation, etc., through user operation while viewing the image.

[0071] (33) For example, the processing may be a processing that blurs at least a part of the image based on the recorded video data.

[0072] This increases the likelihood of seeing an image that is at least partially blurred. In particular, it is good to process the image using a blurring method that blurs the image. You can blur a specified area of ​​the image, or you can blur the area outside the specified area. You can also make it possible to switch between blurring the specified area and blurring the area outside the specified area.

[0073] (34) For example, when outputting audio for the video based on the processed video data, the communication robot may be further provided, which faces the display screen that displays the video, and when outputting audio to a viewer of the video based on the processed video data, which faces the viewer.

[0074] In this way, there is a higher probability of obtaining a communication robot that, when outputting audio about a video, faces the display screen showing the video, and when outputting audio to the viewer of the video, faces the viewer. In particular, the communication robot should have an audio output means, a head rotation means that can move its neck between the display screen and the viewer, a determination means that analyzes the video displayed on the display screen and determines whether to output audio about the video displayed on the display screen or to the viewer, and a head rotation control means that controls the head rotation means so that when outputting audio about a video, it faces the display screen showing the video, and when outputting audio to the viewer of the video, it faces the viewer.

[0075] (35) For example, control may be performed to change the manner in which the video based on the processed video data is displayed based on information about roads.

[0076] This makes it easier to change the display mode of the video based on information about roads. In particular, it is good to analyze the video represented by the video data, detect information about roads contained in the video, and change the display mode of the video based on the processed video data based on the detected information about roads. The information about roads may be, for example, the width of the road or road signs. For example, the display mode may be changed by changing the brightness, hue, saturation, etc. of the video based on the information about roads, or by displaying videos with different camera angles.

[0077] (36) For example, it may be a recording device that records the video data to be played back in the playback device.

[0078] This increases the likelihood of recording video data to be played back on the playback device. For example, it could be a drive recorder that records video data to be played back on the playback device. The recording device may be installed in four-wheeled vehicles such as automobiles, motorcycles, bicycles, etc.

[0079] (37) For example, the recording device may be controlled to record the video data in high quality based on specified conditions.

[0080] This method makes it easier to record video data in high quality based on specified conditions. In particular, it is preferable to specify the conditions using a specifying means, input the specified conditions using an input means, and control the recording device using a recording control means to record video data in high quality based on the input conditions. The specified conditions may be, for example, location and time. High quality may be achieved, for example, by increasing the frame rate above a certain threshold or by setting the resolution higher than a certain resolution.

[0081] (38) For example, at least one of the playback device and the recording device may be equipped with a specifying means for specifying conditions, and control may be performed to record the video data in high quality based on the conditions specified by the specifying means.

[0082] In this way, the likelihood of recording high-quality video data based on the conditions specified by the designated means increases. In particular, by specifying the conditions by the designated means and using the input means... It is preferable to input specified conditions and control the recording device with a recording control means to record video data in high quality based on the input conditions. When the playback device is equipped with a specifying means, for example, it is preferable to record data representing the conditions specified by the specifying means on a recording medium for recording video data, load such a recording medium into the recording device, and read the data representing the specified conditions from the recording medium, thereby recording video data in high quality in the recording device based on the specified conditions.

[0083] (39) For example, the condition may be either that the speed of the vehicle on which the recording device is installed exceeds a certain level, or that the vehicle on which the recording device is installed has driven to a designated location.

[0084] This makes it easier to record high-resolution video data when the car's speed exceeds a certain level or when it is driving through a designated location. In particular, it is preferable that the car's speedometer is photographed using a camera installed in the recording device, the captured video is analyzed by an analysis means, and high-resolution recording is performed when a determination means determines that the car's speed has exceeded a certain level. Alternatively, the car's position may be detected by a position detection means, and high-resolution recording may be performed when a determination means determines that the car is driving through a designated location.

[0085] (40) For example, when recording the video data with a recording device, it is preferable to record information obtained by at least one of vehicle-to-infrastructure communication and vehicle-to-vehicle communication in association with the video data.

[0086] This increases the likelihood of recording information obtained through at least one of vehicle-to-infrastructure communication and vehicle-to-vehicle communication in association with video data. In particular, it is preferable to receive the information transmitted through at least one of vehicle-to-infrastructure communication and vehicle-to-vehicle communication using a receiving means, and to record the received information in association with video data obtained by filming. Specifically, it is preferable to record the information transmitted through at least one of vehicle-to-infrastructure communication and vehicle-to-vehicle communication in the header of the video file storing the video data obtained by filming.

[0087] (41) For example, control may be performed to transmit data representing a first calculated value uniquely determined from the video data recorded by the recording device to the blockchain system.

[0088] This increases the likelihood that data representing a first calculated value uniquely determined from video data can be transmitted to the blockchain system. The data representing the first calculated value can be recorded in the blockchain. In particular, it is preferable to control the transmission means to calculate a first calculated value uniquely determined from video data recorded by a recording device and transmit data representing the calculated first calculated value to the blockchain system. The calculation means for calculating the first calculated value may be provided in the recording device or in the playback device, and the transmission means may also be provided in the recording device or in the playback device. Since the blockchain is relatively unlikely to be tampered with, the fact that the data representing the first calculated value is recorded in the blockchain indicates that there is a high probability that the data representing the first calculated value has not been tampered with, and it becomes more likely that the video data has not been tampered with by using the data representing the first calculated value and the video data.

[0089] (42) For example, the video data recorded by the recording device is MPEG-based video data, and the data representing a first calculated value uniquely determined for each unit, with the immediate vicinity of the P-frame used as the unit for dividing the video data, is recorded in the block chain.

[0090] This approach increases the likelihood of recording data representing a first calculated value for each unit, separated just before the P-frame, in the blockchain. Since a P-frame cannot be reproduced as a single image without an I-frame, calculating a first calculated value for each unit separated just before the P-frame would likely make the calculation of that first calculated value more difficult. In particular, it is preferable to control the transmission means to use the point just before the P-frame as the unit for separating the video data, calculate a first calculated value that is uniquely determined for each unit, and transmit data representing the calculated first calculated value to the blockchain system.

[0091] (43) For example, control may be implemented to record data representing a second calculated value uniquely determined from event data in the blockchain.

[0092] This increases the likelihood that data representing a second calculated value uniquely determined by event data can be recorded in the blockchain. This makes it difficult to tamper with the second calculated value. In particular, it is preferable to control the transmission means to calculate a second calculated value uniquely determined from event data using a calculation means and to transmit data representing the calculated second calculated value to the blockchain system. By comparing the second calculated value with the event data, it is possible to determine whether the event data has been tampered with, and since it is difficult to tamper with the second calculated value, it also becomes difficult to tamper with the event data.

[0093] (44) For example, it is preferable to record data indicating whether the video data recorded in the recording device is from manual operation or automatic operation, in association with the video data.

[0094] This approach increases the likelihood of determining whether the video data was recorded during manual or autonomous driving. In particular, it is advisable to input identification data from an input means to indicate whether the video data was recorded during manual or autonomous driving, and to record the input identification data in association with the video data. Sometimes, it is advisable to record the identification data in the header area of ​​the video file in which the video data is recorded. Manual driving can be defined as, for example, when a person operates the steering wheel of a car, while autonomous driving is when the steering wheel is operated by machine control without human intervention.

[0095] (45) For example, the image quality of the video data recorded by the recording device may be changed depending on whether the vehicle is being driven manually or automatically.

[0096] This increases the likelihood of obtaining different image quality data for manual driving and autonomous driving. In particular, it is preferable to input identification data indicating whether the system is in manual or autonomous driving mode from an input means, and then change the image quality of the video data recorded when the identification data indicates manual driving versus when it indicates autonomous driving using a modification means. The image quality may be increased during manual driving and decreased during autonomous driving, or vice versa.

[0097] (46) For example, a communication robot could be operated during autonomous driving.

[0098] This approach increases the likelihood of operating the communication robot during autonomous driving. During autonomous driving, for example, the user does not need to drive, so they can communicate with the communication robot without interfering with driving. In particular, it is preferable for the input means of the recording device to input identification data indicating that autonomous driving is in progress, and for the transmission means of the recording device to send an operation command to the communication robot in response to such identification data being input to the input means.

[0099] (47) For example, it is possible to display video based on recorded video data and control the display mode of the video in accordance with the determination that there is a high probability that an event will occur.

[0100] This approach makes it easier to change the display mode of the video in response to a determination that there is a high probability of an event occurring. In particular, it is preferable to install a display device on a recording device installed in a vehicle, display video on the display screen of the display device, analyze the video using an analysis means while the vehicle is in motion, and, in response to a determination means determining that there is a high probability of an event occurring in front of the vehicle as a result of the analysis, control the display device using a display control means to change the display mode of the video on the display screen. For example, by recognizing that the display mode of the video has changed, it becomes clear that there is a high probability of an event occurring, and preparations can be made for that event. For example, if the video data is recorded by a drive recorder installed in a moving car, a change in the video mode indicates that, for example, there is a possibility of some kind of event occurring in a few seconds, such as a sudden pedestrian or a car suddenly appearing.

[0101] (48) For example, it is advisable to record data representing the communication status of radio waves received by a communication device.

[0102] This method makes it easier to record data representing the communication status of communication radio waves. In particular, it is good to receive the communication radio waves using a communication device that supports reception of communication radio waves and record data representing the communication status of the communication radio waves. Specifically, the data representing the communication status should indicate whether communication is in progress or whether communication has been interrupted. For example, the communication radio waves should be radio waves based on LTE (Long Term Evolution), Wi-Fi, or Bluetooth (registered trademark). When displaying video, the display mode may be changed according to the communication status.

[0103] (49) For example, the recording mode of the video data may be changed based on the communication radio waves received by the communication device.

[0104] This method makes it easier to change the video data recording mode based on communication radio waves. In particular, it is good to receive communication radio waves using a communication device that supports the reception of communication radio waves, and then change the video data recording mode based on the received communication radio waves. The recording mode can be anything as long as the recording method changes. For example, you could set the recording mode to high quality when receiving communication radio waves and to low quality when not receiving communication radio waves. The reverse is also possible.

[0105] (50) For example, video data representing the image of the driver who has been called in and the video data obtained by the driver's driving are recorded in association.

[0106] This increases the likelihood of recording video data representing the driver during roll call in association with video data obtained from the driver's driving. In particular, it is preferable to input video data representing the driver during roll call using an input means, and then record the input video data in association with video data obtained from the driver's driving, which is captured by a camera installed in the recording device. It is especially preferable to store these video data in the same folder. For example, it becomes possible to display video obtained from the driver's driving in association with the video of the driver during roll call. For example, it becomes possible to display video of the driver during roll call and video of the driver's driving simultaneously.

[0107] (51) For example, control may be performed to record in association the video data obtained by a recording device installed on a truck driven by the driver and the video data obtained by a recording device installed on a forklift operated by the driver.

[0108] This makes it easier to record and associate video data of truck driving with video data of forklift driving. In particular, it is preferable to transmit video data obtained from a camera installed on a recording device attached to the truck, which captures the truck driver's driving, to a server, and video data obtained from a camera installed on a recording device attached to the forklift, which captures the forklift driver's driving, to the server, and record these video data on the server in association with each other. In particular, it is preferable to create a video folder for each driver on the server and store the video data of the driver when driving the truck and the video data when driving the forklift in the same video file. Alternatively, when driving the truck, the video data of the truck driving may be recorded on a recording medium loaded in the recording device attached to the truck, and when driving the forklift, the recording medium may be removed from the recording device attached to the truck and loaded into the recording device attached to the forklift, and the video data of the forklift driving may be stored in the same video file as the video data of the truck driving. For example, it becomes possible to compare and view video of a driver driving a truck with video of the same driver driving a forklift.

[0109] (52) For example, the system should be controlled to display information that drivers should pay attention to, overlaid on the actual scenery visible from a moving car.

[0110] This approach increases the likelihood of displaying information that drivers should pay attention to while driving, overlaid on the real-world scenery. This allows drivers to see the real-world scenery while simultaneously understanding the information they should be aware of while driving. In particular, it would be beneficial to display information that drivers should pay attention to while viewing the real-world scenery using a Mixed Reality (MR) device. Specifically, for example, the user could wear MR glasses, viewing the real-world scenery through the glasses, and simultaneously displaying information that drivers should pay attention to in a semi-transparent form overlaid on the real-world scenery on the surface of the glasses.

[0111] (53) For example, it is advisable to implement control that notifies the user if the vehicle has traveled to the same location as the vehicle has traveled to previously.

[0112] This makes it easier to determine whether a location has been traveled before. Knowing that a location has been traveled before allows for new considerations, such as whether to follow the same route or a different route. In particular, the recording medium loaded into the recording device contains data indicating previous travel locations. The previously traveled location represented by the recorded data is compared with the current location, and if they match, the user is notified that they are traveling in the same location as before through an audio output from the audio output device or a display on the display screen of the display device. The GPS (global positioning system) receiver installed in the recording device is ideally used to detect the current location.

[0113] (54) For example, it is preferable to take a picture of the car's speedometer and perform control to record the video data based on the captured speedometer.

[0114] This increases the likelihood of recording video data based on the captured speedometer reading. In particular, it is desirable to install a camera in the vehicle at a position that allows it to capture the speedometer, photograph the vehicle's speedometer with the camera, analyze the resulting video using an analysis device, and start recording video data when the speedometer indicates a speed that triggers video data recording. For example, it is desirable to start recording video data when the vehicle's speedometer exceeds 80 km / h.

[0115] (55) For example, a recording device attached to an automated guided vehicle may be used to record the video data based on the detection of approaching objects around the automated guided vehicle.

[0116] This increases the likelihood of recording video data based on the detection of approaching objects around the automated guided vehicle (AGV). In particular, it is advisable to use a camera installed in the recording device to photograph the area around the AGV, and when the detection means detects the approach of an object around the AGV, record the video data obtained from the camera's image. For example, this would allow us to see what happens when the AGV comes into contact with an object.

[0117] (56) For example, a recording device attached to a forklift may be used to record the video data based on at least one of the operation of the forklift and the fact that a driver is on the forklift.

[0118] This makes it easier to record video data based on at least one of the forklift's movement and the driver getting on the forklift. In particular, it is preferable to start recording video data when the detection means detects at least one of the forklift's movement and the driver getting on the forklift. Specifically, it is preferable to use a camera installed in the recording device to photograph the inside of the forklift, and by analyzing the video data obtained from the photography, to detect whether the forklift has moved or whether the driver has gotten on the forklift, and to start recording video data in response to that detection. Forklift movement may be, for example, the forks moving. For example, the recording of video data may be stopped when the driver gets off the forklift.

[0119] (57) Event recording trigger information may be recorded in the header of the file that stores the video data.

[0120] This increases the likelihood of recording event trigger information in the file header. For example, it increases the possibility of using that trigger information to perform various actions.

[0121] (58) For example, a playback method for playing back recorded video data, characterized in that the recorded video data is processed based on acquired information, and control is performed to display a video based on the processed video data.

[0122] In this way, users are more likely to see processed video based on acquired information, rather than the recorded video itself. By processing the video in a way that is suitable for the user viewing it, a video more appropriate for the user can be obtained. Rather than processing the recorded video, the recorded video data is processed based on acquired information, so there is a higher possibility of displaying a video that is more suitable for the user viewing it.

[0123] (59) In the above playback method, it is preferable to control the recording device to record the video data to be played back.

[0124] This increases the likelihood that the video data to be played back can be recorded on the recording device.

[0125] (60) For example, a computer-readable program for playing back recorded video data, which causes the computer to process the recorded video data based on acquired information and to perform control to display a video based on the processed video data. It would be best to have a program that can read data.

[0126] In this way, users can view not the recorded video itself, but a video processed based on the acquired information. By applying processing tailored to the user viewing the video, a user-friendly video can be obtained. Since the recorded video data is processed based on the acquired information, rather than the recorded video itself, there is a higher possibility of displaying a video that is more suitable for the user viewing the video.

[0127] The inventions described in (1) to (60) above can be combined in any way. For example, one may combine all or part of the configuration of the invention described in (1) with at least part of the configuration of at least one of the inventions described in (2) and onward. In particular, it is preferable to combine the invention described in (1) with at least part of the configuration of at least one of the inventions described in (2) and onward. Alternatively, one may extract any configuration from the inventions described in (1) to (60) and combine the extracted configurations. The applicant of this application intends to obtain rights to inventions that include these configurations. [Effects of the Invention]

[0128] In this way, users can view not the recorded video itself, but a video processed based on the acquired information. By applying processing tailored to the user viewing the video, a user-friendly video can be obtained. Since the recorded video data is processed based on the acquired information, rather than the recorded video itself, there is a higher possibility of displaying a video that is more suitable for the user viewing the video. The effects of the present invention are not limited thereto, and the effects produced by the components of the structure disclosed in this specification and the drawings are also disclosed. The present invention intends to obtain rights to the components that produce such effects through divisional applications, amendments, etc. For example, the phrases "can do..." in this specification are descriptions that specify the effects produced, and there are components that produce effects even without such descriptions. Furthermore, there are effects that can be grasped by the component even without such descriptions. [Brief explanation of the drawing]

[0129] [Figure 1] This is a side view of a dashcam. [Figure 2] This is a block diagram showing the electrical configuration of a drive recorder. [Figure 3] This shows the file structure of the video file. [Figure 4] This is a flowchart showing the record processing procedure. [Figure 5]This flowchart shows the procedure for generating header data. [Figure 6] This is a flowchart showing the video data recording process. [Figure 7] This flowchart shows the data output processing procedure. [Figure 8] This flowchart shows the data output processing procedure. [Figure 9] This is an example of the image displayed on the screen. [Figure 10] This is a block diagram showing the electrical configuration of a personal computer. [Figure 11] This is a flowchart showing the playback process. [Figure 12] This is a flowchart showing the playback process. [Figure 13] This is a flowchart showing the playback process. [Figure 14] This is an example of a playback window. [Figure 15] This is an example of a playback window. [Figure 16] This is an example of a playback window. [Figure 17] This is an example of a playback window. [Figure 18] This is an example of a playback window. [Figure 19] This is an example of a playback window. [Figure 20] This is an example of a playback window. [Figure 21] This is an example of a playback window. [Figure 22] This is an example of a playback window. [Figure 23] This is an example of a playback window. [Figure 24] This is an example of a playlist display area. [Figure 25] This is an example of a playlist display area. [Figure 26] This shows I-frames and P-frames. [Figure 27] This is a flowchart showing the record processing procedure. [Figure 28] This is a flowchart showing the processing procedure for a dashcam installed in a truck. [Figure 29] This flowchart shows the processing procedure for a drive recorder attached to a forklift. [Figure 30] This is a flowchart showing the record processing procedure. [Figure 31] This is a flowchart showing the processing procedure for an automated guided vehicle (AGV). [Figure 32] This is a flowchart showing the processing procedure for operational evaluation. [Figure 33] This is a flowchart showing the playback process. [Figure 34] This is a flowchart showing the playback process. [Figure 35] This is a flowchart showing the playback process. [Figure 36] This is an example of a map image. [Figure 37] This is a flowchart showing the playback process. [Figure 38] This is an example of a playback window. [Figure 39] This is an example of a playback window. [Figure 40] This is an example of a personal computer and a communication robot. [Figure 41] This is a block diagram showing the electrical configuration of a communication robot. [Figure 42] This is a flowchart showing the processing steps of a communication robot. [Figure 43] This shows a communication robot placed inside a vehicle. [Modes for carrying out the invention]

[0130] (1) Recording process Figures 1 through 9 illustrate the recording process using a drive recorder.

[0131] Figure 1 shows an embodiment of the present invention and is a side view of drive recorder 1 (for example, an example of a recording device).

[0132] The drive recorder 1 comprises a mounting section 2 and a camera section 4.

[0133] Mounting section 2 is fixed to the windshield (not shown) of the vehicle to which the drive recorder 1 is installed, and allows for the insertion and removal of the memory card. A display screen 3 for displaying the recorded video is formed on a part of the outer surface of mounting section 2. Furthermore, mounting section 2 is equipped with a DC (direct current) jack (not shown) for connection to the vehicle via a converter unit (not shown). In addition, mounting section 2 is equipped with predetermined buttons (not shown).

[0134] The camera unit 4 incorporates a front camera (not shown in Figure 1) that primarily captures the area in front (the direction of travel of the vehicle to which the drive recorder 1 is attached) and a rear camera (not shown in Figure 1) that primarily captures the area behind the vehicle. The camera unit 4 is equipped with a front lens 5 that focuses an image of the subject on the image plane of the front camera, and a rear lens 6 that focuses an image of the subject on the image plane of the rear camera. The front camera can capture the area in front of the vehicle, to the left and right in front of the vehicle, and up and down in front of the vehicle, while the rear camera can capture the area behind the vehicle, to the left and right in rear of the vehicle, and up and down in rear of the vehicle. Therefore, the front camera and rear camera together can capture an omnidirectional view centered on the drive recorder 1.

[0135] The camera unit 4 can move forward and backward within a predetermined angle θ relative to the mounting unit 2, enabling shooting at a relatively suitable angle regardless of the angle of the windshield.

[0136] Figure 2 is a block diagram showing the electrical configuration of drive recorder 1.

[0137] The overall operation of drive recorder 1 is controlled by controller 10.

[0138] The drive recorder 1 is equipped with a memory card reader / writer 95. When a memory card 25 is inserted into the drive recorder 1, the memory card reader / writer 95 reads the data recorded on the memory card 25 and writes data to the memory card 25. The memory card 25 contains an operation program, which will be described later. The controller 10 reads this operation program and installs it into the drive recorder 1. The controller 10 reads the operation program installed into the drive recorder 1 and controls each part of the drive recorder 1 according to the operation program. If the operation program of the drive recorder 1 is stored on a recording medium other than the memory card 25, it may be read from such a recording medium. In this case, an interface such as a driver corresponding to the recording medium will be provided in the drive recorder. Alternatively, it may be received by a communication circuit 17 and installed into the drive recorder 1.

[0139] As described above, the drive recorder 1 is equipped with a front camera 11 and a rear camera 12. The video data captured by the front camera 11 and the rear camera 12 are input to the controller 10. The drive recorder 1 is also equipped with a display device having a display screen 3, and the display control device 13 controls the display on the display device 14. Furthermore, the drive recorder 1 is equipped with a clock 15, a vehicle-to-infrastructure / vehicle-to-infrastructure information receiving circuit 16, a communication circuit 17, and an acceleration sensor 18. The time data representing the time output from the clock 15 is output to the controller 10. The vehicle-to-infrastructure / vehicle-to-infrastructure information receiving circuit 16 receives, for example, data representing the surrounding conditions transmitted from roadside units placed on roads, etc., based on an advanced road information system (an example of vehicle-to-infrastructure information), and data representing the vehicle itself transmitted from the vehicle (an example of vehicle-to-vehicle information), and outputs the received data to the controller 10. The vehicle-to-infrastructure / vehicle-to-infrastructure information receiving circuit 16 uses, for example, a communication circuit based on DSRC (Dedicated Short Range Communications). The communication circuit 17 communicates with, for example, a communication robot if one is placed inside the vehicle. The output data from the communication circuit 17 is input to the controller 10. The acceleration sensor 18 detects the acceleration of the vehicle to which the drive recorder 1 is installed and outputs the detected acceleration data to the controller 10.

[0140] Furthermore, the drive recorder 1 is equipped with input devices 20 such as buttons, memory 21 such as RAM (Random Access Memory), and a GPS (Global Positioning System) receiver 22. Output signals from the input devices 20, output data from the memory 21, and output data from the GPS receiver 22 are also input to the controller 10.

[0141] Figure 3 shows the file structure of the video files that store the video data output from the front camera 11 and the video data output from the rear camera 11.

[0142] The video file includes a header recording area 31 and a video recording area 32. The header recording area 31 is an area for recording header data, and the video recording area 32 is an area for recording video data.

[0143] Figure 4 is a flowchart showing the recording process of drive recorder 1.

[0144] The recording process shown in Figure 4 is repeated for each frame (or any other frame) captured until the recording process is complete.

[0145] When the engine of a vehicle to which the drive recorder 1 is installed is started, the vehicle's battery supplies power to the drive recorder. Then, the recording process shown in Figure 4 begins. Also, a video file is generated on the memory card 25. When the recording process begins, the video data output from the front camera 11 and the rear camera 12 is input to the display control device 13 via the controller 10. The captured video is displayed on the display screen 3 of the display device 14 (see Figure 9). The video displayed on the display screen 3 is either the video obtained by the front camera 11 or the video obtained by the rear camera 12, according to the program recorded on the memory card 25, but it is also possible to display both the video obtained by the front camera 11 and the video obtained by the rear camera 12. Alternatively, the default display may be the video obtained by the front camera 11 or the video obtained by the rear camera 12, and a toggle button may be provided on the input device 20 to switch between the two depending on the operation of the toggle button.

[0146] First, header data is generated and temporarily recorded in memory 21 (step 41), followed by the temporary recording of video data in memory 21 (step 42). Next, output processing of various data is performed (step 43), and the data temporarily recorded in memory 21 is written to memory card 25 (step 44).

[0147] In Figure 4, after the generation of header data and temporary recording of the generated header data to memory 21 (step 41), temporary recording of video data to memory 21 is performed (step 42), followed by output processing of various data, etc. (step 43). However, the order of these processes does not matter. In addition, the header data and video data are temporarily recorded to memory 21, the header data and video data recorded in memory 21 are read, and the read header data and video data are written to the memory card 25. However, if possible, the data may be written directly to the memory card 25 without being temporarily stored in memory 21. Furthermore, two or more of the processes of header data generation and recording (step 41), video data recording (step 42), and data output processing (step 43) may be performed in parallel.

[0148] When the engine of the vehicle to which drive recorder 1 is installed is stopped, and the power supply to drive recorder 1 is cut off, the recording process shown in Figure 4 is terminated.

[0149] Figure 5 is a flowchart showing the header data generation process (the process described in step 41 of Figure 4), illustrating the process of the controller 10.

[0150] If the vehicle to which drive recorder 1 is installed is capable of autonomous driving, data indicating that autonomous driving is in progress (driving that is not based on the vehicle's driver's input) is transmitted from the vehicle to drive recorder 1. For example, by connecting drive recorder 1 to the vehicle's in-vehicle network and detecting the data output to the in-vehicle network indicating whether autonomous driving or manual driving (driving that is based on the vehicle's driver's input), the controller 10 of drive recorder 1 recognizes whether the vehicle is in autonomous or manual driving. Furthermore, if the vehicle and drive recorder 1 are capable of wireless communication and the vehicle transmits data indicating whether autonomous or manual driving is in progress, that data is transmitted to the drive recorder. The controller 10 of the drive recorder 1 may recognize whether the vehicle is in automatic or manual driving mode by receiving the signal in the communication circuit 17 of the drive recorder 1.

[0151] The controller 10 of the drive recorder 1 determines whether the vehicle is in autonomous driving mode or manual driving mode (step 51). If the vehicle is in autonomous driving mode, data indicating that the data recorded on the memory card 25 is video data captured during autonomous driving is temporarily recorded in the memory 21 as header data (step 52). Similarly, if the vehicle is in manual driving mode, data indicating that the data recorded on the memory card 25 is video data captured during manual driving is temporarily recorded in the memory 21 as header data (step 53). For example, data indicating whether video data captured within a certain time period was taken during autonomous driving or manual driving is temporarily recorded on the memory card 25. Of course, it is also possible to create an autonomous driving video folder to store video data taken during autonomous driving and a manual driving video folder to store video data taken during manual driving, and store video data taken during autonomous driving in the autonomous driving video folder and video data taken during manual driving in the manual driving video folder. The storage of video data in a time-based or autonomous driving video folder, or in a manual driving video folder, establishes an association between the video data from autonomous driving or manual driving and the video data itself.

[0152] Next, the controller 10 checks whether the vehicle-to-infrastructure / vehicle-to-infrastructure information receiving circuit 16 has received vehicle-to-infrastructure communication information or vehicle-to-vehicle communication information (step 54). If vehicle-to-infrastructure communication information or vehicle-to-vehicle communication information is received (YES in step 54), data representing the received communication information is temporarily recorded in memory 21 along with data representing the time the communication information was received (step 55). Since the time the communication information was received is known, it is also possible to determine what kind of video was being filmed at that time, and when playing back the filmed video, the vehicle-to-infrastructure communication information or vehicle-to-vehicle communication information can be output simultaneously. In this way, for example, vehicle-to-infrastructure communication information or field-of-view vehicle-to-vehicle communication information can be associated with video data by time. If neither vehicle-to-infrastructure communication information nor vehicle-to-vehicle communication information is received (NO in step 54), the process in step 55 is skipped, and the received information is not temporarily recorded in memory 21.

[0153] Furthermore, the system checks whether the communication circuit 17 has received a communication signal (step 56). If the communication circuit 17 receives a communication signal (YES in step 56), data representing the communication status of that signal is temporarily recorded in the memory 21 (step 57). For example, if the communication circuit 17 receives radar waves emitted from speed enforcement devices installed on the road, such as speed cameras, or radio signals used by emergency vehicles, data indicating the content of those signals is temporarily recorded in the memory 21. If the communication circuit 17 does not receive a communication signal (NO in step 56), the process in step 57 is skipped, and data representing the communication status of the communication signal is not recorded in the memory 21.

[0154] Furthermore, the driver's gaze is detected by the controller 10 from the video data obtained by the rear camera 12 (step 58). The gaze data representing the detected gaze is also temporarily written to memory 21 along with data representing the time (step 59). This allows us to know when and in which direction the driver was looking. The eyes are detected from the video, and the gaze is determined by the positional relationship between the pupil, iris, and cornea (the so-called black part of the eye) and the conjunctiva (the so-called white part of the eye). Similarly, the video data obtained by the rear camera 12 is analyzed to detect the direction the driver of the vehicle is facing, and the data representing the detected direction is also temporarily written to memory 21 along with data representing the time of shooting.

[0155] Figure 6 is a flowchart showing the video data recording process (the process described in step 42 of Figure 4).

[0156] During the video data recording process, the controller 10 confirms whether the vehicle is in autonomous or manual operation mode (step 61). If the vehicle is in autonomous operation mode, relatively low-resolution video data is recorded in memory 21 (step 62). For example, the controller 10 controls the front camera 11 and rear camera 12 so that the number of pixels per frame represented by the video data output from the front camera 11 and rear camera 12 is reduced to less than a first threshold (resolution reduction), or so that the number of frames read per unit time is reduced to less than a first number of frames (frame rate reduction). This results in relatively low-resolution video data. If the vehicle is in manual operation mode, relatively medium-resolution video data is recorded in memory 21 (step 63). For example, the front camera 11 and rear camera 12 are controlled by the controller 10 so that pixel decimation is performed so that the number of pixels per frame represented by the video data output from the front camera 11 and rear camera 12 is greater than a first threshold, or so that the number of frames read per unit time is greater than a first number of frames. This results in relatively medium-quality video data. In this way, the quality of the video data can be adjusted by adjusting the resolution and frame rate. During autonomous driving, relatively low-quality video data is recorded in memory 21, and during manual driving, relatively medium-quality video data is recorded in memory 21. However, it may also be the case that relatively medium-quality video data is recorded in memory 21 during autonomous driving, and relatively low-quality video data is recorded in memory 21 during manual driving. It is sufficient that the quality of the video data recorded during autonomous driving is different from the quality of the video data recorded during manual driving.

[0157] Next, the controller 10 checks whether the specified conditions have been met (step 64). For example, the vehicle speed may exceed a certain level, the vehicle may be driving in a specified location, or, if the input device 20 of the drive recorder 1 is equipped with a high-definition setting switch (an example of a setting means), the operation of that high-definition setting switch is an example of a condition being met. Whether the vehicle speed has exceeded a certain level can be detected by analyzing the video obtained by the rear camera 12, if the drive recorder 1 is mounted on the vehicle so that the rear camera 12 can capture the vehicle's speedometer, or it can be detected from the distance traveled calculated based on the GPS receiver 22 and the time taken to travel that distance, calculated based on the clock 15. Whether the vehicle is driving in a specified location can be determined based on the output data from the GPS receiver 22.

[0158] If the specified conditions are met (YES in step 64), relatively high-resolution video data is temporarily recorded in memory 21 (step 65). For example, the front camera 11 and rear camera 12 are controlled by the controller 10 to stop pixel decimation of the video data output from the front camera 11 and rear camera 12, or to increase the number of frames read per unit time compared to the number of frames at medium resolution. This results in relatively high-resolution video data. If the specified conditions are not met (NO in step 64), the process in step 65 is skipped, and the process of recording high-resolution video data in memory 21 is not performed.

[0159] Furthermore, it is checked whether a command to change the recording mode has been issued to the controller 10 (step 66). If a command to change the recording mode has been issued (YES in step 66), the recording mode is changed (step 67); if no command to change the recording mode has been issued (NO in step 66), the recording mode is not changed. For example, as described above, if the communication circuit 17 receives radar waves emitted from speed enforcement devices installed on the road, such as speed cameras, or communication radio waves such as those used by emergency vehicles, it may be determined that a command to change the recording mode has been issued. As a result of the command to change the recording mode, the recording mode can be changed to, for example, to record high-resolution video data in the memory 21.

[0160] In the video data recording process shown in Figure 6, after confirming whether the vehicle is in autonomous or manual driving mode (step 61), a confirmation of whether the specified conditions have been met (step 64) is performed. However, the confirmation of whether the specified conditions have been met (step 64) may be performed first, followed by the confirmation of whether the vehicle is in autonomous or manual driving mode (step 61), or both of these confirmations (steps 61 and 64) may be performed in parallel.

[0161] Figure 7 is a flowchart showing the data output processing procedure (processing procedure for step 43 in Figure 4).

[0162] In the data output processing procedure, the controller 10 also checks whether the vehicle is in automatic or manual driving mode (step 71). If the vehicle is in automatic driving mode, a start command is sent from the communication circuit 17 to the communication robot (not shown) located inside the vehicle (step 72). If the vehicle is in manual driving mode, no start command is sent from the communication circuit 17 to the communication robot located inside the vehicle. If the vehicle is in manual driving mode, a stop command may be sent to the communication robot. If the vehicle is in automatic driving mode, the driver is not manually operating the vehicle, so the operation of the communication robot does not interfere with driving. On the other hand, the driver can communicate with the communication robot, allowing them to make effective use of their time. If the vehicle is in manual driving mode, the communication robot located inside the vehicle does not operate, so any interference such as the driver being distracted by the communication robot talking to them can be prevented. In addition, if a stop command is sent to the communication robot during manual driving mode, the communication robot can be stopped if it is currently operating.

[0163] Next, the controller 10 checks whether there is a high probability of an event occurring (step 73). For example, if the road narrows or there is a lot of pedestrian traffic while driving, it is considered that there is a high probability of an event occurring, such as the driver suddenly applying the brakes. If there is a high probability of an event occurring (YES in step 73), the display control device 13 is controlled by the controller 10 to change the display mode of the video displayed on the display screen 3 of the drive recorder 1 (step 74). For example, the display mode of the video can be changed by displaying the words "Caution!" on the display screen, or by superimposing a semi-transparent red or yellow color on the entire video displayed on the display screen 3 or on a part of the video, such as the area around the video. If there is no high probability of an event occurring (NO in step 73), the display mode of the video displayed on the display screen 3 of the drive recorder 1 cannot be changed.

[0164] Furthermore, if the vehicle is in motion (or even stopped), the controller 10 also checks whether the current location is the same as a previously visited location (step 75). The memory card 25 installed in the drive recorder 1 has GPS data indicating previously visited locations recorded in the video file header recording area. The controller 10 compares the GPS data recorded in the header recording area with the GPS data received by the GPS receiver 22 installed in the drive recorder 1 attached to the vehicle while it is in motion, thereby confirming whether the current location is the same as a previously visited location. If the vehicle is driving in the same location as before (YES in step 75), the drive recorder 1 notifies the driver that it is the same location (step 76). If the vehicle is not driving in the same location as before (NO in step 75), the drive recorder 1 does not notify the driver that it is the same location.

[0165] Figure 9 shows an example of the display screen 3 of drive recorder 1.

[0166] The display screen 3 of the drive recorder 1 shows the footage captured by the front camera 11. An image is displayed. At the bottom of display screen 3, the text string 7 "You are driving in the same area as you have driven in before" is displayed. By seeing this text string 7, the driver can see that they are driving in the same area as they have driven in before. The driver can then tell whether they are driving in an area that they have the impression of having driven in due to déjà vu, or an area they have actually driven in before. Whether it is the same area as a previously driven location does not have to be an exact match, but it is sufficient if it falls within a certain range.

[0167] Although the string 7 indicates that the location is the same as a previously driven location, the display of a predetermined mark or a change in the display mode of the display screen 3 may be used instead of displaying string 7. The driver will know that they are driving in the same location as a previously driven location by seeing such a mark or by confirming a change in the display mode. In addition, if the drive recorder 1 is equipped with a speaker, the driver may be notified that they are driving in the same location as a previously driven location by outputting voice or sound from the speaker instead of displaying string 7, etc. In any case, it is sufficient for the driver to know that they are driving in the same location as a previously driven location.

[0168] Returning to Figure 8, once one frame of video data is temporarily recorded in memory 21, the controller 10 calculates the hash value of the recorded video data (an example of a first calculated value uniquely determined from the video data) (step 77). The data representing the calculated hash value is transmitted to the blockchain system via the communication circuit 17 (step 78). The blockchain system records the data representing the calculated hash value in the blockchain (a block of transactions). Since the blockchain is difficult to tamper with, it becomes possible to verify whether the video data has been tampered with using the hash value. The memory card 25 also records the data representing the calculated hash value. To verify whether the video data has been tampered with, the data representing the hash value transmitted to the blockchain system is compared with the data representing the hash value recorded on the memory card 25. If the hash values ​​match, it is determined that the video data has not been tampered with; if the hash values ​​do not match, it is determined that the video data has been tampered with. It goes without saying that the video data recorded on the memory card 25 and the data representing its hash value are recorded in association with each other on the memory card 25. For example, the data representing the hash value is recorded in the video file header recording area 31, along with data indicating which time period of video data the hash value was calculated from.

[0169] In this way, when the header data and video data are temporarily recorded in memory 21, the recorded header data and video data are read from memory 21, the header data is written to the header recording area of ​​the video file, and the video data is written to the video recording area of ​​the video file.

[0170] (2) Playback process Figures 10 to 23 illustrate the playback process using playback viewer software.

[0171] Figure 10 is a block diagram showing the electrical configuration of a tablet personal computer (an example of a playback device).

[0172] The overall operation of the tablet-type personal computer (hereinafter referred to as "personal computer") 80 is controlled by the CPU (Central Processing Unit) 80.

[0173] The personal computer 80 is equipped with a display device 82. This display device 82 is The display control device 81 is controlled by the CPU 90. The personal computer 80 is also equipped with an accelerometer 93, and the output signal from the accelerometer 93 is input to the CPU 90. Furthermore, the personal computer 80 includes a memory card reader / writer 95 that reads data recorded on an SSD (solid state drive) 94 and a memory card 25, which are accessed by the CPU 90, and writes data to the memory card 25.

[0174] Furthermore, the personal computer 80 includes input devices 96 such as a keyboard and mouse, memory 97 such as RAM, and communication circuits 98 for connecting to a network such as the Internet.

[0175] The operating program, described later, is received via the internet by the communication circuit 98 of the personal computer 80 and installed on the personal computer 80. The CPU 90 reads and executes the installed program, thereby controlling each component. The operating program may also be stored on a recording medium such as a memory card 25, and the operating program may be read from such a recording medium and installed on the personal computer 80.

[0176] Figures 11 to 13 are flowcharts showing the playback process in the personal computer 80. Figures 14 to 24 are examples of playback windows displayed on the display screen of the display device 92 of the personal computer 80.

[0177] The memory card 25 that was installed in drive recorder 1 is removed from drive recorder 1 and installed in personal computer 80. When the operating program installed on personal computer 80 is started by the user, display device 92 The playback window 130 shown in Figure 14 is displayed on the screen.

[0178] Figure 14 shows an example of the playback window 130.

[0179] The playback window 130 has a menu bar 131 at its top, and below this menu bar 131 is a video display area 141. Furthermore, below the video display area 141 are a map image display area 151, a vehicle information display area 161, and a playlist display area 171.

[0180] Menu bar 131 displays various items, and by selecting each item, a pull-down menu appears, allowing for detailed commands. Menu bar 131 displays the strings "File," "View," "Playback," "Tools," and "Information," as well as folder icons 132, camera icons 133, printer icons 134, reel icons 135, FD (flexible disk) icons 136, log icons 137, and gear icons 138. By pulling down the strings "File," "View," "Playback," "Tools," and "Information," a pull-down menu corresponding to the respective item appears, allowing for detailed commands. Clicking the folder icon 132 selects a folder, clicking the camera icon 133 converts the image to a still image, clicking the printer icon 134 prints the image, clicking the reel icon 135 converts the image to a video, clicking the FD (flexible disk) icon 136 backs up the data, clicking the log icon 137 converts the image to log data, and clicking the gear icon 138 changes the recording settings. The video display area 141 is the area that displays the video recorded by the drive recorder 1, etc. The map image display area 151 is the area near the shooting location of the video displayed in the video display area 141. This is the area for displaying the map image. The vehicle information display area 161 is the area for displaying information such as the vehicle's speed at the time of video recording. The vehicle information display area 161 includes a vehicle status display area 162 that displays the status of the vehicle to which the drive recorder 1 that recorded the video displayed in the video display area 141 is installed, operation buttons 163 that give operation commands such as playing and stopping the video, a time display area 166 that displays the recording time of the video displayed in the video display area 141, and an acceleration graph display area 168 that displays the vehicle's acceleration in a graph. The playlist display area 171 is the area for displaying a list of filenames of video files recorded on the memory card 25.

[0181] When the folder icon 132 is clicked and a folder containing video files is selected, the filenames of the video files stored in that folder are displayed in the playlist display area 171 as clickable files, as shown in Figure 14. The recording start time of the video is used as the filename, but it goes without saying that other times can also be used as the filename.

[0182] When any of the filenames displayed in the playlist display area 171 is clicked using the mouse, which is included in the input device 96, the corresponding video file is read from the memory card 25. Playback of the video data stored in the read video file begins (Figure 11, step 111). Various header data corresponding to the recording time of the video data are recorded in the header recording area of ​​the video file, and the recorded header data is also read when the video at the recording time is displayed, and various superimposing processes are performed on the video according to the content of the read header data.

[0183] When playback of video data (the video data to be played may be footage or recordings taken when the user viewing the video was driving, or footage or recordings taken when a user other than the user viewing the video was driving) begins, the video data is displayed as a video in the video display area 141, as shown in Figure 15. At the same time, data representing GPS information recorded in the video file containing the video data is transmitted via the internet to a website that transmits data representing a map image. Then, from that website, image data representing a map of the vicinity of the location where the video displayed in the video display area 141 was recorded is transmitted to the personal computer 80. When the image data representing the map is received by the personal computer 80, the map image is displayed in the map image display area 151. The map image also displays an arrow icon 152 that represents the location displayed in the video display area 141. Furthermore, the vehicle status display area 162 of the vehicle information display area 161 displays the vehicle's speed, longitudinal acceleration (X), lateral acceleration (Y), vertical acceleration (Z), latitude (N), and longitude (E) at the time the video displayed in the video display area 141 was being recorded. The time display area 166 displays the recording time of the video displayed in the video display area 141, and the acceleration graph display area 168 displays an acceleration graph.

[0184] Since the video data is recorded in all directions, it is possible to display not only images of subjects visible from the direction of travel of the vehicle, but also images of subjects behind (opposite direction to the direction of travel), to the left of the vehicle (left relative to the direction of travel), or to the right of the vehicle (right relative to the direction of travel). Figure 16 shows a playback window 130 with images of subjects behind in the video display area 141, Figure 17 shows a playback window 130 with images of subjects to the right in the video display area 141, and Figure 18 shows a playback window 130 with images of subjects to the left in the video display area 141. For example, if you drag to the right using the mouse in the state shown in Figure 15, the image will gradually switch from images of subjects in the direction of travel to images of subjects to the right, and as shown in Figure 17, the image of subjects to the right will be displayed in the video display area 141. If you drag further to the right using the mouse in the state shown in Figure 17, the image of subjects to the right will be displayed. The image gradually switches from the subject's image to the image of the subject behind, as shown in Figure 16, which is displayed in the image display area 141. If you drag further to the right using the mouse in the state shown in Figure 16, the image gradually switches from the image of the subject behind to the image of the subject to the left, as shown in Figure 18, which is displayed in the image display area 141. If you drag further to the right using the mouse in the state shown in Figure 18, the image gradually switches from the image of the subject to the left to the image of the subject in the direction of travel, as shown in Figure 15, which is displayed in the image display area 141. In this way, images of subjects in all directions can be displayed in the image display area 141. In the above embodiment, the mouse is dragged to the right, but if you drag to the left, similar to the above, the image gradually switches from the image of the subject in the direction of travel as shown in Figure 15 to the image of the subject to the left as shown in Figure 18, then gradually switches to the image of the subject behind as shown in Figure 16, then gradually switches to the image of the subject to the right as shown in Figure 17, and then gradually switches back to the image of the subject in the direction of travel as shown in Figure 15.

[0185] In this embodiment, when an image is displayed in the image display area 141, the field of view is superimposed on the image (Figure 11, step 111).

[0186] Figure 19 shows an example of a playback window 130 that displays the user's field of view.

[0187] In the video display area 141, a frame 143 indicating the field of view is superimposed by the CPU 90 onto the video represented by the video data (this is an example of video processing). The video portion outside the field of view area 142 is blurred by the CPU 90 (hatching is shown to indicate that it is blurred). The field of view is predetermined for each user's age, and the data representing the field of view is pre-recorded in the SSD 94. By specifying the user's age, the data representing the field of view corresponding to the specified age is read from the SSD 94, and the frame 143 indicating the field of view and the area outside the field of view area 142 are blurred as shown in Figure 19 (this is an example of control to process the video data based on the acquired information and display the video based on the processed video data). Of course, the field of view of each user may be detected in advance, and data representing the user's unique field of view may be pre-recorded in the SSD 94. The frame 143 indicating the field of view for each user will be superimposed on the video. Because users know their field of view, they can recognize which areas are visible and which are not.

[0188] Furthermore, the CPU 90 also determines whether there are objects requiring attention (for example, infants) in the area 142 outside the field of view (Figure 11, step 113). If there are objects requiring attention in the area 142 outside the field of view (YES in Figure 11, step 113), a warning (an example of notification to the user) is given to the user (step 114). For example, the area where the object requiring attention is displayed may be surrounded by a frame or a mark may be displayed. Whether there are objects requiring attention in the area 142 outside the field of view can be determined by pre-defining the objects requiring attention, having the CPU 90 perform video analysis, and detecting whether the CPU 90 detects that the image of the object is displayed in the area 142 outside the field of view. The user will know that there are objects requiring attention outside the field of view, and will understand that turning their head as needed to effectively widen their field of view while driving will help prevent accidents. If there are no objects requiring attention in the area 142 outside the field of view (NO in Figure 11, step 113), the process in step 114 is skipped, and no warning is given. The system only needs to notify the user if there is an object requiring attention in the area 142 outside the field of view; therefore, it does not necessarily need to be displayed in the area 142 outside the field of view. For example, the user could be notified by sound output.

[0189] Furthermore, the CPU 90 superimposes a mark 144 indicating the user's gaze onto the video display area 141 (this is an example of a processing step to display the user's gaze on the video) (Figure 11, step 115). Since the data indicating the user's gaze is recorded in the header recording area 31 of the video file in association with the video recording time, by reading the data indicating the user's gaze from the header recording area 31, a mark representing the user's gaze can be superimposed on the video displayed in the video display area 141 at the recording time of the displayed video. Because the user's gaze is superimposed on the video, it is possible to confirm whether the user is looking at subjects that they should be looking at (for example, traffic lights, oncoming cars, pedestrians, etc.).

[0190] In the example shown in Figure 19, the field of view frame 143 and the mark 144 indicating the user's line of sight are displayed regardless of the user's line of sight. That is, the field of view frame 143 indicates the area that would be the field of view when the user views the image displayed in the image display area 141. Therefore, regardless of the user's line of sight while driving, the field of view frame 143 is displayed in the approximate center of the image display area 141. For example, even if the user's line of sight is not in the center of the image but at the edge of the image displayed in the image display area 141, the field of view frame 143 will still be displayed in the approximate center of the image display area 141. Alternatively, the field of view frame 143 may be displayed in the image display area 141 centered on the user's actual line of sight. This would allow the user's actual field of view while driving to be determined. In this case as well, blurring of the area 142 outside the field of view may be performed. The area within the field of view frame 143 is an example of the area that the user can see while driving the vehicle, and the area outside the field of view 142 is an example of the area that the user cannot see while driving the vehicle. This is an example of how these areas can be distinguished and displayed.

[0191] If the direction the user is facing changes during the recording time of the video displayed in the video display area 142 (YES in step 116 of Figure 11), the field of view frame 143 is moved in the direction of the change and superimposed on the video (step 117 of Figure 11). Whether the direction the user is facing has changed can be determined based on the data indicating the direction the user is facing, which is recorded in the header recording area of ​​the video file. If the direction the user is facing does not change (NO in step 116 of Figure 11), the field of view frame 143 is not moved.

[0192] Figure 20 shows an example of how the field of view frame 143 is moved.

[0193] Assume that when recording the video displayed in the video display area 141, the user's facing direction changes from approximately directly in front of the direction of travel to the lower right. In this case, the user's position is moved from the position of the field of view frame 143 to the position of the field of view frame 143A. Also, the user's position is moved from the position of the line of sight mark 144 to the position of the line of sight mark 144A. The user can recognize that the field of view has changed because the direction they are facing has changed. When the direction the user is facing changes, the relative positions of the video displayed in the video display area 141 and the field of view frame 143 (143A) also change. For this reason, it may be possible to ensure that the relative positions of the video and the field of view frame 143 do not change (for example, so that the field of view frame 143 is displayed approximately in the center of the video). For example, similar to how the video displayed in the video display area 141 changes when dragged with the mouse, the video in the direction the user is facing has changed may be displayed.

[0194] Next, it is determined whether the user's line of sight and the direction the user should be looking coincide (Figure 12, step 118). If they do not coincide (NO in Figure 12, step 118), a mark indicating the area to be looked at is displayed on the video display area 141 (Figure 12, step 119). If they coincide (YES in Figure 12, step 118), the mark indicating the area to be looked at is not displayed on the video display area 141. However, if they coincide, the user may be notified of this coincidence (displayed on the video display area 141, sound output, etc.). Needless to say, the direction to be looked at will differ depending on the driving during recording (right turn, left turn, lane direction, etc.) or the user's driving operations (brake operation, turn signal lever operation, etc.).

[0195] Figure 21 shows an example of a display showing markers indicating areas to look at.

[0196] If the user was supposed to look at the lower right corner of the image displayed in the image display area 141, but did not (for example, was looking straight ahead), then mark 144 will be displayed in the image display area 141 indicating that the lower right corner was the area the user should have looked at (this is an example of processing that displays the area the user should have looked at in the image). The CPU 90 determines where the area to look is by analyzing the displayed image, or by reading event data indicating events that occurred when the image was recorded (such as operation of the turn signal lever or sudden braking) if such event data is recorded in the header recording area of ​​the image file. For example, if the user changed lanes from the left lane to the right lane by operating the turn signal lever, but did not look at the right side mirror, mark 145 will be displayed as the area that the user should have looked at. In addition, if the user is supposed to check a predetermined location, such as when turning right or left, the system may display a location to look at if the user has not looked at that location.

[0197] Furthermore, if the user is not looking at the part they should be looking at, the video display in the video display area 141 may be temporarily paused, and an image (still image) of that scene may be displayed in the video display area 141. In a video, the scene that the user is not looking at ends in an instant, whereas an image of the paused state is displayed in the video display area 141, which can better draw the user's attention.

[0198] Furthermore, for parts of the video displayed in the video display area 141 where there is little to no change in the movement of the subject (YES in step 120 of Figure 12), the CPU 90 makes the display speed relatively faster (step 121 of Figure 12). For parts of the video where there is significant change in the movement of the subject (NO in step 120 of Figure 12), the display speed is not made relatively faster. Whether there is little to no change in the movement of the subject can be determined by comparing the difference between video frames. Parts of the video where there is little to no change in the movement of the subject are not considered to be important scenes such as pedestrians suddenly appearing or oncoming vehicles veering into the road, so there is no problem in increasing the display speed. Also, parts of the video where the vehicle's position is not changing are considered to be less important because the vehicle is stationary and accidents are less likely to occur. For this reason, the display speed may also be increased for parts of the video where the vehicle's position is not changing.

[0199] Furthermore, the CPU 90 also determines whether the orientation of the display device 92 has been changed (Figure 12, step 122). If the orientation has been changed (YES in Figure 12, step 122), the display range of the image displayed in the image display area 141 is changed according to that orientation (Figure 12, step 123). If the orientation has not been changed (NO in Figure 12, step 122), the display range of the image is not changed according to the changed orientation. Whether the orientation of the display device 92 has been changed, and in which direction it has been changed, can be detected by the acceleration sensor 93. Suppose the display screen of the display device 92, which was facing a certain direction, is displaying an image of the direction of travel of the vehicle (see Figure 17). If the display device 92, which was facing a certain direction, is turned to the right, the image of the right direction (see Figure 17) will be displayed in the image display area 141, similar to when the image displayed in the image display area 141 is dragged to the right. Similarly, if the display device 92, which was facing a certain direction, is turned to the left, the image to the left (see Figure 18) will be displayed in the image display area 141, just as if the image displayed in the image display area 141 were dragged to the left. Also, if the display device 92, which was facing a certain direction, is turned to the rear, the image to the rear (see Figure 18) will be displayed in the image display area 141. Changing the display range of the image with the direction of travel of the vehicle as the reference direction in this way is one example of image data processing. This enables a pseudo-VR (virtual reality) display.

[0200] Furthermore, information regarding the vehicle's status is displayed at the time of recording of the video shown in the video display area 141. The CPU 90 determines whether information regarding the vehicle's status is recorded in the video file's header recording area 31 (Figure 12, step 124). If information regarding the vehicle's status is recorded (YES in Figure 12, step 124), that information is displayed in the video display area 141. If information regarding the vehicle's status is not recorded (NO in Figure 12, step 124), that information is not displayed in the video display area 141.

[0201] Figure 22 shows an example of information regarding the vehicle's status being displayed in the video display area 141.

[0202] A speech bubble 146 in the first display mode is displayed in the video display area 141. This speech bubble 146 displays information about the vehicle's status at the time the video displayed in the video display area 141 was recorded. In the example shown in Figure 22, the text "Brakes applied!!" is displayed in the speech bubble 146, indicating that the vehicle's brakes (emergency braking) were applied at the time the video displayed in the video display area 141 was recorded. Data representing information about the vehicle's status is also associated with time data and recorded in the video file's header recording area 31, and is read from that header recording area 31. However, for example, the CPU 90 may infer the situation that occurred to the vehicle from a sudden change in acceleration and display it in the speech bubble 146 as information about the vehicle's status. For example, if the vehicle's longitudinal acceleration suddenly changes in the negative direction at a certain recording time, it can be assumed that emergency braking was applied, so the same speech bubble 146 as shown in Figure 22 may be displayed in the video when such acceleration occurs.

[0203] Furthermore, if the user specifies any location in the map image displayed in the map image display area 151 using the mouse (YES in step 126 of Figure 13), data representing the specified location is sent to the server, and data indicating the latitude and longitude of that location is sent from the server to the personal computer 80. When the personal computer 80 receives the data indicating the latitude and longitude, it finds the scene of the video displayed in the video display area 141 that was filmed at the location closest to that latitude and longitude, and the video from that scene is displayed in the video display area 141 (step 127 of Figure 13). If no location in the map image is specified (NO in step 126 of Figure 13), the process in step 127 is skipped, and the video is not displayed by specifying a location in the map image. Furthermore, while the video of a specified location in the map image is displayed in the video display area 141, the video displayed in the video display area 141 may be either video recorded by a drive recorder installed in the vehicle the user is riding in, or video recorded by a drive recorder installed in a third party's vehicle, or both. In this case, the server stores data representing video recorded by a drive recorder installed in a third party's vehicle, and the video data recording the location specified by the user is read from the server and transmitted to the personal computer 80. If the server stores a large amount of data representing video recorded by a drive recorder installed in a third party's vehicle, it is preferable to have the user select which video to display on the personal computer 80 from among these videos, and then display the selected video on the display screen of the display device 92.

[0204] Furthermore, the CPU 90 determines whether vehicle-to-infrastructure communication or vehicle-to-vehicle communication information is recorded in the video file header recording area 31 at the time the video displayed in the video display area 141 is recorded (Figure 13, step 128). If vehicle-to-infrastructure communication or vehicle-to-vehicle communication information is recorded (YES in Figure 13, step 128), that vehicle-to-infrastructure communication or vehicle-to-vehicle communication information is displayed in the video display area 141. If vehicle-to-infrastructure communication and vehicle-to-vehicle communication information are not recorded (NO in Figure 13, step 128), that vehicle-to-infrastructure communication or vehicle-to-vehicle communication information is not displayed in the video display area 141.

[0205] Figure 23 shows an example of information from vehicle-to-infrastructure communication (the same applies to vehicle-to-vehicle communication) being displayed in the video display area 141.

[0206] In the video display area 141, a second display mode, which is different from the first display mode of the aforementioned (see Figure 22) of the speech bubble 146, is displayed. This speech bubble 147 displays vehicle-to-infrastructure communication information at the time the video displayed in the video display area 141 was recorded. In the example shown in Figure 23, the text "Vehicle on the left ahead" is displayed in the speech bubble 147, indicating that a vehicle was present to the left in front of the vehicle at the time the video displayed in the video display area 141 was recorded. Data representing vehicle-to-infrastructure communication and vehicle-to-vehicle communication information is also recorded in the video file's header recording area 31, associated with the time data, and read from that header recording area 31. Users who obtain vehicle-to-infrastructure communication and vehicle-to-vehicle communication information can then check what vehicle operations were performed based on that information at the time the video was recorded (for example, whether the vehicle slowed down).

[0207] In the playback processing procedure shown in Figure 11, the following processes are performed in order: checking whether there is an object of attention outside the field of view (Figure 11 step 113), checking whether the direction the user is facing has changed (Figure 11 step 113), checking whether the user's line of sight and the direction the user should be looking coincide (Figure 12 step 118), checking whether there has been any change in the movement of the subject (Figure 12 step 120), checking whether the orientation of the display device 92 has been changed (Figure 12 step 122), checking whether there is information regarding the status of the vehicle (Figure 12 step 124), checking whether a position in the map image has been specified (Figure 13 step 126), and checking whether information for vehicle-to-vehicle communication or vehicle-to-infrastructure communication has been obtained (Figure 13 step 128). However, the order of these processes is arbitrary, and each process may be performed in parallel.

[0208] (3) Variant Figure 24 shows a portion of the playlist display area 171 included in the playback window 130.

[0209] In the embodiment described above, the file name of the video file displayed in the playlist display area 171 uses the date and time the video data was recorded. However, in this embodiment, instead of the date and time the video data was recorded, the location where the video data was recorded (an example of a filming location) is used as the file name of the video file.

[0210] The location where the video data is recorded is used as the filename for the video file as follows. For example, when a video file is generated in drive recorder 1, the latitude and longitude of the location where it is generated are read by GPS receiver 22, and this latitude and longitude data is transmitted from communication circuit 17 to the server via the internet. The server finds the location of these latitude and longitude values, receives data representing the found location via the internet, and uses the location obtained from the received data as the filename for the video file. Alternatively, the latitude and longitude data recorded in the header recording area 31 of the video file corresponding to the recorded date and time may be transmitted to the server via the internet during playback, the data representing the found location is received by the server via the internet, and the location obtained from the received data is used as the filename for the video file. In Figure 24, the filenames are displayed in the order they were recorded. Since the location at the time of recording is included in the filename, it becomes immediately clear where the video was filmed.

[0211] Figure 25 also shows a portion of the playlist display area 171 included in the playback window 130.

[0212] In Figure 25, the video file containing the location of the recording, which appears infrequently, is shown. The names are displayed in the playlist display area 171 so that they appear higher (a possible example of prioritizing display). The filenames of video files containing frequently occurring recording locations are displayed lower (at the bottom). Frequently occurring recording locations are thought to be mostly from daily life recordings, such as during commutes, while infrequently occurring recording locations are thought to be mostly from non-daily life recordings, such as during travel. To make it easier to play video files recorded in daily life, the filenames of video files containing infrequently occurring recording locations are displayed higher.

[0213] Furthermore, videos from the same location may be displayed together in the playlist display area 171. This makes it easier to distinguish between video files from the same location and video files from different locations. Additionally, the video file name only needs to include the location at the time of recording, and may also include the recording date and time in addition to the location. Furthermore, a map image of the location corresponding to the video file name displayed in the playlist display area 171 may be displayed, for example, in the video display area 141. Displaying a map makes it easier to understand which location the video file is from, rather than just displaying the name. For example, the video file name including the location may be sent from the communication circuit 98 to a map server on the Internet (not shown), the map data sent from the map server is received by the personal computer 80, and the map image is displayed in the video display area 141. By selecting the map image displayed in the video display area 141, playback of the corresponding video file may be started.

[0214] Figure 26 shows an example of a frame structure based on MPEG (Moving Picture Experts Group).

[0215] In Figure 26, elements labeled "I" represent I-frames, and elements labeled "P" represent P-frames. One I-frame is followed by three P-frames, and these one I-frame and three P-frames constitute a GOP (Group of Picture).

[0216] As shown in Figure 8, step 77, when calculating the hash value, the unit used to divide the video data is the frame immediately before the P-frame, not the frame immediately before the I-frame, and the controller 10 calculates a hash value for each of these units. For example, the controller 10 calculates a hash value for the P-frame, the I-frame, and the three P-frames following the I-frame as a single unit. The hash value calculated in this way is then transmitted to the blockchain system.

[0217] Alternatively, instead of calculating a hash value from video data, when various events (for example, sudden braking or skidding) occur in the drive recorder, the controller 10 may calculate a hash value (an example of a second calculated value uniquely determined from event data) from the data representing those events, and transmit the calculated hash value to the blockchain system.

[0218] Figure 27 is a flowchart showing an example of the recording process procedure for drive recorder 1.

[0219] In this embodiment, the driver's (for example, professional driver) behavior during roll call is filmed and recorded. The recorded video data is stored in a video file and sent to a server for each driver, where it is stored. Drive recorder 1 accesses the server and reads the video file containing the driver's roll call video data (step 181). The controller 10 of drive recorder 1 records the roll call video file onto the memory card 25 installed in drive recorder 1 (step 182).

[0220] Next, the driver's name and date are stored in the video file that contains the video data of the driving. The video file is generated by the controller 10 with the specified filename (step 183). The driver's name may be pre-stored in the memory 21 of the drive recorder 1, or it may be entered from the input device 20. When the engine of the vehicle to which the drive recorder 1 is installed is started, recording of video data during driving begins, and until recording ends (YES in step 185), the video data continues to be stored in the video file as described above.

[0221] In this way, a video file containing video data from the roll call and a video file containing video data from the driving are recorded on the memory card 25. When such a memory card 25 is inserted into the personal computer 80, the video from the roll call and the video from the driving can be displayed in the video display area 141 of the playback window 130. The video file containing the video data from the roll call and the video file containing the video data from the driving are associated by the driver's name (this is an example of association).

[0222] In this embodiment, the drive recorder 1 stores the video file from the roll call and the video file from the driving session on the memory card 25. However, the video file from the roll call and the video file from the driving session may be stored on the memory card 25 during playback. For example, the memory card 25 containing the video file from the driving session is inserted into the personal computer 80, and the file names of the video files are displayed in the playlist display area 172. When any video file name is selected from the video file names displayed in the playlist display area 172, data representing the driver's name, recorded in the header recording area 31 of the video file corresponding to the selected video file name, is sent to the server recording the roll call video file, and the roll call video file is sent from that server to the personal computer 80. When the roll call video file is received by the personal computer 80, the video from the roll call, represented by the driver's roll call video file, and the video from the driving session, represented by the driver's driving video file, are displayed in the video display area 141. Preferably, the video footage from the roll call and the video footage from the driving scene are displayed simultaneously and side-by-side in the video display area 141, but this is not necessarily required.

[0223] Figures 28 and 29 are flowcharts illustrating an example of the recording process for drive recorder 1. Figure 28 shows an example of the recording process when drive recorder 1 is installed on a truck, and Figure 29 is a flowchart illustrating the recording process for drive recorder 1 installed on a forklift when the driver of the truck moves the cargo loaded on the truck using a forklift.

[0224] Referring to Figure 28, while the truck is in operation, video data is recorded by the drive recorder 1 installed in the truck onto the memory card 25 installed in the drive recorder 1 (step 191). When recording of video data to the memory card 25 is complete, data representing the truck driver's name and data representing the date and time are stored in the header recording area of ​​the video file, and the video file with video data recorded in the video recording area of ​​the video file is sent from the drive recorder 1 to the server (step 192). Data representing the driver's name is pre-stored in the memory 21 of the drive recorder 1. Furthermore, the drive recorder sends an association command with the video file from the forklift to the server (step 193). For example, a command is sent from the drive recorder attached to the truck to the server to associate a video file that was transmitted from the same location as the last location (where the truck was parked) represented by the video data stored in the video file transmitted from the truck's drive recorder 1, and that contains video data recorded by the drive recorder attached to the forklift within a few minutes of the truck's parking time.

[0225] Referring to Figure 29, when the drive recorder 1 attached to the forklift detects that a driver has approached the forklift, recording begins by the front camera 11 and rear camera 12 included in the drive recorder 1 (step 201).

[0226] The video footage captured by the front camera 11 or the rear camera 12 is analyzed, and if it is detected that the driver has boarded the forklift and the forklift has started to operate, or if the driver has boarded the forklift (YES in step 202), recording of the video data to the memory card 25 installed in the drive recorder 1 attached to the forklift begins (step 203). Once recording is complete, the video file is sent to the server (step 205).

[0227] On the server, if there are two video files where the recording end time and recording start time are approximately the same, and the location at the end of recording and the location at the start of recording are approximately the same, then, as described above, these video files will be treated as, for example, video files of the same driver switching from a truck to a forklift. By sending these video files representing a series of videos to the personal computer 80, the video of the truck driving and the video of the forklift driving can be displayed in the video display area 141 of the playback window 130.

[0228] In the above embodiment, the driver operates a truck and then a forklift, but the reverse is also possible: the driver operates a forklift and then a truck, and separate video files are obtained for each operation. Alternatively, a series of video files may be obtained showing the driver operating a forklift to load cargo onto a truck, then driving the truck to the destination, and finally unloading the cargo from the truck at the destination using a forklift.

[0229] Figure 30 is a flowchart showing an example of the recording process procedure for a drive recorder 1 installed in a vehicle.

[0230] The drive recorder 1 is mounted on the upper inside of the vehicle's windshield in a position that allows it to capture the vehicle's speedometer. When the vehicle's engine starts, recording begins with the front camera 11 and the rear camera 12, and the rear camera 12 captures the vehicle's speedometer (step 211). The video of the vehicle's speedometer is analyzed by the controller 10 to determine whether the vehicle's speed has reached, for example, 30 km / h or more (this is an example of a speed above a certain level, but other speeds are also acceptable) (step 212). If the speed is 30 km / h or more (YES in step 212), recording of video data to the memory card 25 begins (step 213). If the vehicle's speed falls below 30 km / h (YES in step 214), recording of video data to the memory card 25 stops (step 215).

[0231] Thus, when the vehicle speed exceeds 30 km / h, the captured video data is recorded to the memory card 25, and when the speed falls below 30 km / h, the recording of the captured video data to the memory card 25 is stopped. When the vehicle is being driven at a certain speed or higher, the video data obtained through recording is recorded to the memory card 25. In this embodiment, when the vehicle speed exceeds 30 km / h, the captured video data is recorded to the memory card 25, and when the speed falls below 30 km / h, the recording of the captured video data to the memory card 25 is stopped. However, it is also possible to record the captured video data to the memory card 25 when the vehicle speed exceeds 30 km / h, but not stop when the speed falls below 30 km / h, or to record the captured video data to the memory card 25 regardless of the vehicle speed, and stop when the speed falls below 30 km / h.

[0232] Figure 31 is a flowchart showing an example of the recording process procedure for a drive recorder installed in an automated guided vehicle (AGV).

[0233] The aforementioned drive recorder 1 is mounted on the automated guided vehicle (AGV). The AGV is equipped with a laser sensor that detects the approach of surrounding objects to the AGV.

[0234] When the automated guided vehicle's (AGV) engine is started (the AGV's power is turned on), power is supplied to the drive recorder 1. Then, the front camera 11 and rear camera 12 installed in the drive recorder 1 begin recording (step 221). When the laser sensor detects the approach of an object (an example of a surrounding object, which could also be another AGV) to the AGV (YES in step 222), recording of video data to the memory card 25 installed in the drive recorder 1 begins (step 223). If the laser sensor does not detect the approach of a surrounding object to the AGV (NO in step), recording of video data to the memory card 25 does not begin. Even if a surrounding object accidentally comes into contact with the AGV, the incident can be recorded. Thus, the vehicles include not only those with a driver, but also those without a driver.

[0235] Figure 32 is a flowchart showing an example of the processing procedure for operational evaluation.

[0236] A user drives a vehicle equipped with drive recorder 1, removes the memory card 25 containing the recorded video data from drive recorder 1, and inserts it into personal computer 80. The playback window 130 shown in Figure 14 is displayed on the display screen of display device 92, and the video data recorded on memory card 25 is played back (step 231). If there is movement in the subject of the video being played back (YES in step 232), the video is analyzed, and a driving evaluation is performed based on that video (step 233). For example, if the brakes are applied suddenly when there is movement in the subject, or if other vehicles such as pedestrians, bicycles, motorcycles, or cars get too close, the driving evaluation will be low. Whether the brakes were applied suddenly can be determined from the acceleration data, and whether pedestrians or other vehicles are too close can be determined from the relative size of the pedestrians or other vehicles in the video. If there is no movement in the subject of the video (NO in step 232), no driving evaluation is performed for that part of the video. Steps 232 and 233 are repeated until the video ends. In video footage where the subject is stationary, the vehicle is considered to be stationary, making the driving evaluation largely meaningless. Since driving evaluation is not performed in such locations, it is less likely that unnecessary evaluations will be made. Driving evaluation is performed only when the subject is moving, and not when the subject is stationary. Alternatively, driving evaluation could be performed only when the vehicle is moving, and not when the vehicle is stationary.

[0237] Figure 33 is a flowchart showing an example of the playback processing procedure for video data recorded by drive recorder 1.

[0238] In this embodiment, the drive recorder 1 is installed in a vehicle that transports passengers, such as a bus, taxi, hired car, or train, but it does not necessarily have to be installed in such a vehicle.

[0239] The memory card 25, which was installed in the drive recorder 1, is removed from the drive recorder 1 and installed in the personal computer 80. The playback window 130 shown in Figure 14 is displayed on the display screen 14 of the personal computer 80, and the video data recorded on the memory card 25 is played back (step 241).

[0240] The CPU 90 analyzes the video and determines whether or not there are passengers in the video (step 242). For example, if there are people sitting in the seats of the vehicle except for the driver, it can be determined that there are passengers. If there are passengers in the video (YES in step 242), the video is displayed in the video display area 141 (step 243). If there are no passengers in the video (NO in step 242), the video display in the video display area 141 is stopped. Steps 242 and 243 are repeated until the video ends (NO in step 244). In the embodiment described above, the video display in the video display area 141 is stopped when there are no passengers in the video, but the video may be displayed, but the display speed may be increased, or the video may be skipped to the part of the video that contains passengers.

[0241] Figure 34 is a flowchart showing an example of a playback procedure for recorded video data. While it is preferable that the data was recorded by drive recorder 1, it may also be recorded by other recording devices. This playback process involves finding the desired scene within the video.

[0242] The memory card 25 containing the video data is inserted into the personal computer 80, and the playback window 130 is displayed on the display screen of the display device 92. When the user clicks the word "Tools" in the menu bar 131 of the playback window 130, a pull-down menu including the "Search" item appears. When the "Search" item is selected from the pull-down menu, a pop-up window for entering keywords appears on the playback window 130. The user enters keywords describing the situation of the video portion they want to find into the pop-up window using the input device 96 (step 251). The CPU 90 analyzes the video data and finds video footage related to the entered keywords (step 258). The video footage of the found scene is displayed in the video display area 141 (step 254). The keywords describing the situation of the video portion entered by the user do not have to be clear, such as "video containing a Shinkansen," but can also be vague, such as "bright video." The video portion corresponding to the vague keyword is found. For example, if the keyword is "bright video," the video portion with relatively high brightness is found. Alternatively, keywords describing the vehicle's condition at the time of video data recording can be entered to find the corresponding video footage. For example, keywords such as the vehicle's condition when the drive recorder recording the video data was installed, speeding, sudden acceleration, or turn signal activation can be used to find the corresponding video portion. Since the data describing the vehicle's condition is recorded along with the recording time in the header recording area 31 of the video file where the video data is stored, this data can be used to find the corresponding video portion. Furthermore, keywords that search for video footage after a dynamic movement, such as finding the video footage after a red car overtakes a blue car, can be entered to find the corresponding video portion.

[0243] Figure 35 shows a flowchart illustrating an example of the playback processing procedure for video data recorded by Tribe Recorder 1.

[0244] The memory card 25 that was installed in drive recorder 1 is removed from drive recorder 1 and installed in personal computer 80. The playback window 130 shown in Figure 14 is displayed on the display screen 92 of personal computer 80.

[0245] Data indicating the latitude and longitude of the vehicle's movement position, recorded in the header recording area 31 of the video file stored on the memory card 25, is read, and the vehicle's movement range is detected by the CPU 90 of the personal computer 80 (step 261). A request command for map image data including the detected movement range is sent from the personal computer 80 to the server, and the server sends the map image data back to the personal computer 80. Data is transmitted. When the map image data is received by the personal computer 80, the map image represented by the received image data is displayed in the video display area 141 (step 262). Furthermore, data representing the vehicle operation information at the time of recording the video data is read from the video file header recording area 31, and the vehicle operation status performed at each location is displayed on the map image displayed in the video display area 141 (step 264).

[0246] Figure 35 is an example of a map image displayed in the video display area 141.

[0247] In the map image, the vehicle's movement is indicated by a dashed line 148. This dashed line 148, representing the movement, is obtained from data indicating the vehicle's location, which is recorded in the header recording area 31. Furthermore, within the movement indicated by the dashed line, the vehicle's operation status is displayed by a callout 146. By viewing the map image, the user can relatively easily grasp the vehicle's movement range and also relatively easily understand what kind of vehicle operations were performed at what locations during that movement.

[0248] Figure 37 is a flowchart showing an example of the playback processing procedure for video data recorded by the drive recorder 1. In this embodiment, video obtained from the user's driving and video obtained from a third party's driving are displayed in the video display area 141.

[0249] The memory card 25 installed in the vehicle being driven by the user is removed, and the removed memory card 25 is installed in the personal computer 80. The video data obtained from the user's driving of the vehicle is read from the memory card 25 and played back (step 271). In addition, data indicating the recording location (vehicle movement range) of the video data obtained from the user's driving (data indicating latitude and longitude) is read from the video file header recording area 31 and sent to a server where many video files obtained from the driving of a third party are stored. The server finds a video file that stores video data obtained from driving in the same location as the user's driving and sends it to the personal computer 80. It is sufficient to find a third party's video file that closely matches the user's direction and range of movement, even if it does not perfectly match. The found third party's video file is received by the personal computer 80 (step 273), and the video obtained from the user's driving and the video obtained from the third party's driving are displayed in the video display area 141 (step 274).

[0250] Figure 38 shows an example of the playback window 130.

[0251] The video display area 141 is divided into two video display areas 141A and 141B, left and right (or up and down). The left video display area 141A displays video obtained by the user's driving (an example of video based on processed video data), and the right video display area 141B displays video obtained by a third party driving, represented by a video file found on the server. In the video display area 141A, a speech bubble 146A is displayed indicating the user's vehicle operation information at the time the video displayed in the video display area 141A was recorded. It goes without saying that the operation information displayed in the speech bubble 146A is obtained from acceleration data, etc., recorded in the header recording area 31 of the video file where the video data is stored. Similarly, a speech bubble 146B indicating the third party's vehicle operation information at the time the video displayed in the video display area 141B was recorded is also displayed in the video display area 141B. It is preferable that the CPU 90 performs playback control so that the display of the video is synchronized so that the position indicated by the video displayed in video display area 141A and the position indicated by the video displayed in video display area 141B are approximately the same.

[0252] The video of the user's driving displayed in the video display area 141A and the video displayed in the video display area 141B By comparing your driving with that of a third party, you can improve your driving skills by comparing your own driving with that of others. In particular, the 146A and 146B vents allow you to see the similarities and differences in your driving, so you can also see whether your driving is consistent with or different from that of others.

[0253] Furthermore, the driving of the user may be compared with that of a third party, and information representing an evaluation of the user's driving (an example of information based on the comparison between the driving of the third party and the driving of the user) may be displayed in the video display area 141. For example, when the evaluation is that the driving skill is worse than that of the third party, caution information such as "Let's improve the driving skill by referring to the driving of the third party." may be displayed, and when an evaluation that the driving skill is higher than that of the third party is obtained, information representing a high evaluation such as "The driving technique seems to be high." may be displayed.

[0254] FIG. 39 is an example of the playback window 130.

[0255] At the time of recording the video displayed in the video display area 141, alphabetic characters indicating the locations that the driver of the vehicle should look at and the order thereof are superimposed on the video. The example shown in FIG. 39 is, for example, a video at the time of lane change, and it is preferable to view the locations indicated by the alphabet in alphabetical order. At the time of recording the video data, the line of sight of the driver is detected as described above, and it is detected by the CPU 90 whether the line of sight of the driver is moving in alphabetical order through the locations indicated by the alphabet. When the line of sight of the driver is moving in alphabetical order through the locations indicated by the alphabet, for example, a character string such as "This is an ideal driving." is displayed in the video display area 141. The user can confirm that the driving skill is not bad. When the line of sight of the driver is not moving through the locations indicated by the alphabet, a character string such as "There is an unconfirmed location." is displayed in the video display area 141 to inform the user that there is an unconfirmed location. Also, when the line of sight of the driver is not moving in alphabetical order, a character string such as "The confirmation order is not ideal." is displayed in the video display area 141 to inform the user that the confirmation order is not good. The user can know whether the movement of the line of sight during driving is appropriate.

[0256] Furthermore, in the example shown in FIG. 19, although a processing operation of blurring the area 142 outside the visual field range was performed, it is also possible to detect the range not viewed by the user within a certain period of time and blur the non-viewed range. For example, it is also possible to detect the range not viewed by the user from the start to the end of a right or left turn and blur the non-viewed range. Needless to say, the line of sight of the user who is the driver is detected at the time of recording video data, and the non-viewed range of the user is detected using the data of the line of sight. For example, when the non-viewed range is detected, as shown in FIG. 19, the non-viewed range is blurred or semi-transparented in the same manner as the area 142 outside the visual field range.

[0257] Furthermore, when playing back the video data recorded by the drive recorder 1, the CPU 90 may perform correction (an example of a processing operation) so as to be the memory color of the driver. In order to correct to the memory color, the contrast, tone, and saturation of the obtained video are adjusted, and it may be determined in advance how much each is adjusted, or several preset adjustment values may be determined and selected at the time of playback.

[0258] Furthermore, a communication robot may be used together with the personal computer 80 during video playback.

[0259] FIG. 40 is an example of a communication robot 280 placed near the personal computer 80.

[0260] The communication robot 280 includes a body 281 and a head 282. An eye camera is provided inside the eye 283 of the head 282, and a speaker is provided in the mouth 284. The head 282 can rotate clockwise and counterclockwise relative to the body 281. The communication robot 280 is positioned to the right of the personal computer 80. The user sits in front of the personal computer 80. In this way, the positions of the personal computer 80, the communication robot 280, and the user are predetermined. The communication robot 280 is controlled to face the display device 92 when displaying the playback window 130 on the display screen of the personal computer 80's display device 92 and outputting audio comments about the video displayed in the video display area 141, and to face the user when outputting audio comments to the user viewing the video.

[0261] Figure 41 is a block diagram showing the electrical configuration of the communication robot 280.

[0262] The communication robot 280 includes a CPU 291 that controls the overall operation, a memory 292 that stores data, a wireless communication circuit 294 that communicates with the personal computer 80, the drive recorder 1, etc., and a wireless LAN (local area network) board 293 connected to the wireless communication circuit 294. The communication robot 280 also includes a head motor 296 that rotates the head 282, an eye camera 298, a speaker 300, and a microphone 301. The head motor 296 is controlled by a motor control board 295, and video data captured by the eye camera 298 is taken into the communication robot 280 by a sensor board 297. Furthermore, audio output from the speaker 300 and audio input from the microphone 301 are controlled by an audio input / output board 299.

[0263] Figure 42 is a flowchart showing the processing procedure of the communication robot 280. It goes without saying that the operation program of the communication robot 280 is stored in memory 292.

[0264] The display screen of the display device 92 of the personal computer 80 displays the playback window 130 described above, and the video recorded by the drive recorder 1 is displayed in the video display area of ​​the playback window 130. The personal computer 80 and the communication robot 280 communicate, and acceleration data is transmitted from the personal computer 80 to the communication robot 280 (other data, such as data recorded in the header recording area 31 of the video file and event data, may also be transmitted from the personal computer 80 to the communication robot 280. It is sufficient that data is transmitted from the personal computer 80 to the communication robot 280 that allows the communication robot 280 to advise the user, etc., after viewing the video).

[0265] Based on the acceleration data transmitted to the communication robot 280, if the CPU 291 of the communication robot 280 determines that there has been a sudden change in acceleration in the longitudinal direction of the vehicle to which the drive recorder 1 recording the displayed video is attached (step 311), it is assumed that the vehicle has applied the brakes suddenly. Then, as shown in Figure 40, the communication robot 280, which was facing the user, has its head 282 rotated 90 degrees clockwise (step 312) to face the display device 92 of the personal computer 80. The display screen of the display device 92 shows the video of the scene in which the brakes were applied suddenly, and when the communication robot 280 faces the display device 92, a voice message indicating that it is a dangerous driving situation is output from the speaker 300 (step 313). For example, the voice message "This kind of driving is dangerous." is output. The voice data is stored in the memory 292.

[0266] Subsequently, the head 282 of the communication robot 280 is rotated 90 degrees counterclockwise (step 314), and the communication robot 280 faces the user. When the communication robot 280 faces the user, a voice message is output from the speaker 300 warning the user not to engage in dangerous driving like the scene shown on the display screen of the display device 92 (step 315). For example, the message "Let's drive safely" is output. Because the warning comes from the communication robot 280 and not from a human instructor, the user is more likely to heed the warning.

[0267] Figure 43 shows the communication robot 280 described above placed on the dashboard 311 inside the vehicle.

[0268] As described above, the drive recorder 1 is mounted on the top of the windshield 310. The drive recorder 1 and the communication robot 280 communicate, and as described above, when the drive recorder 1 recognizes that the vehicle is in autonomous driving mode, it sends an operation start command to the communication robot 280. The communication robot 280 then operates and begins to talk to the vehicle's driver or other relevant person.

[0269] Furthermore, when playing back video data recorded by the drive recorder 1, the CPU 90 may analyze the video data for road information (for example, information such as the road narrowing or being a dead end) on which the vehicle to which the drive recorder 1 is installed is traveling, and change the display of the video shown in the video display area 141 according to the analysis results. For example, a string indicating the detected road information may be superimposed on the video, or some of the hue, brightness, and saturation of the whole or a part of the video may be adjusted. As the video displayed in the video display area 141 changes, it is possible to know that there has been a change in the road conditions.

[0270] Furthermore, event recording trigger information may be recorded in the video file header recording area 31. For example, information about operating the turn signal, sudden braking, or wipers may be recorded as event recording trigger information. Additionally, while the drive recorder 1 is recording, the user may sit in the passenger seat and wear a glasses-type display device on their head. The display screen of this display device will show driving caution information, and the view in front of the vehicle while driving (an example of the actual scenery visible from a moving car) will be visible through the display screen. The driving caution information will be displayed superimposed on the view in front of the vehicle. Needless to say, the road to be driven on is predetermined, and caution information for that road will be displayed on the display device. Alternatively, a third party other than the user (for example, a driving instructor) may ride in the vehicle and select the caution information to be displayed on the display device, and the selected caution information will be displayed on the user's display device.

[0271] In the above embodiment, the drive recorder 1 was capable of recording in all directions, including the front, sides, and rear of the vehicle, but it does not necessarily have to be capable of recording in all directions.

[0272] Furthermore, in addition to normal continuous recording and event recording, the drive recorder 1 may also perform other types of recording. For example, for event recording, the acceleration sensor and trigger switch will record before and after the event. This will be recorded separately from the continuous recording as an event record. The 30 seconds before and after the acceleration sensor reacts (1 minute) may also be recorded. For example, if the first event occurs, and then the second event occurs, for example, if the car hits the guardrail the first time, and then shakes and hits the guardrail a second time, the recording may be made. If the recording times overlap, they may be treated as a single event. If a slot becomes available, you can record it as a separate event, or if they don't overlap, you can process them as two separate events.

[0273] Furthermore, the aforementioned drive recorder 1 may be used on a factory line to ensure smooth operation, by building the line and using cameras to monitor and adjust the line. When assembling with machine tools, cameras may be temporarily installed at points where problems are likely to occur to capture images of the assembly process. When setting up a new line in a factory, cameras may be temporarily installed for about three months to verify any malfunctions. Records may be taken before and after an alert is issued by a machine tool in the factory, or images may be taken before and after triggers such as green, yellow, or red lights on the machine tool. The video data obtained from such cameras may be displayed in the playback window 130 described above. Only events may be captured, or continuous capture may be performed.

[0274] Even with 360-degree drive recorders (drive recorders that record in all directions) and wide-angle camera drive recorders, the range of human vision is not the same as the camera's imaging range, and the visible range varies depending on a person's age and individual differences. During senior driver training and license renewal, a narrow field of view is dangerous and often leads to hitting bumps, so it would be good to record driving footage using a drive recorder and show instances of hitting bumps during the training. By displaying the visible range of the driver's seat, the training can be conducted without criticizing the driver, but rather by saying, "If you can't see, please look properly," or "Turn your head so you can see." For example, it would be good to display a frame so that the visible range is clear during playback.

[0275] When recording driving conditions (scenery, landscape) with drive recorder 1, the driver's eye movements can be recorded simultaneously, and if the field of view is shifted, a vector or circle indicating where the driver is looking within the field of view can be displayed. Recording where the driver's gaze was directed while driving and displaying it in the playback window 130 should be adjusted so that the displayed range and frame are shifted in the direction the driver is looking. If drive recorder 1 can capture a full 360-degree (omnidirectional) image, the entire area in front of, to the side of, and behind the vehicle is recorded, so the direction the driver is facing can be determined by image recognition. Accordingly, the displayed frame should be moved. In addition to the present, the playback window 130 should display not only the present but also the extent to which the driver was checking things in the present, past, and future, such as looking straight ahead now but not in the past. For example, the areas that were not being looked at could be filled in. Areas that were not being looked at at all can be identified. The extent to which the head was turned, the gaze, and the extent to which the driver was looking can be determined. Drive recorder 1 can be connected to a gaze detection sensor, and the gaze captured by the sensor can be displayed. It can identify areas the user is not looking at. The imaging time changes depending on the driving situation, such as normal driving, high-speed driving, and driving at intersections. For example, the time spent driving in a driving school, or the time before and after stopping at an intersection. The viewing range is predetermined at each point in the driving school, and the system judges how well the user's actions match that range. It would be good to have a standard, such as a minimum percentage of viewing required to pass. Even in normal driving, it would be good to judge how much the user looks when stopped at an intersection, or how much they look at different speeds, such as between 40 km / h and 80 km / h, or above 80 km / h. At high speeds, you don't look to the sides, but you do look in the rearview mirror, while in city driving, you must also pay attention to the sides. When driving slowly at 20 km / h or less, you must look 360 degrees, checking mirrors and blind spots while driving. The viewing range changes based on the turn signal, brakes, and steering angle. If you are turning to the right, you must look to the right relatively accurately. By using the playback window 130, it is possible to see, for example, whether the user checked before activating the turn signal or before turning the steering wheel. Furthermore, during elderly driver training, it would be beneficial to have elderly individuals actually drive within the driving school grounds, attach a dashcam, and then later show the footage in a playback window 130, demonstrating how they were driving.A simulator would suffice, but elderly drivers won't be convinced unless they actually drive, so showing them real-life footage helps them understand. This has two benefits: it helps the driver understand and makes it easier for the instructor to understand. It would also be good to provide information for the instructors. Consider the areas that haven't been viewed as information for the instructors. Alternatively, you could check if the driver is looking at the instruments (speedometer, etc.) by having them pre-determine points of focus such as the mirrors. You could also record the driving behavior with a drive recorder 1 while the driver is heavily intoxicated and wearing glasses, and show them the footage.

[0276] Since buses have over a dozen mirrors, it would be good to evaluate whether the driver is looking at all of them, in addition to the tachograph, by also evaluating their visual perception. Displaying the driver's line of sight (gaze) would allow for an assessment of what percentage of their driving time they are looking at a particular spot. It would also be possible to evaluate where the driver is looking and where they are not, before and after using the turn signal. Buses and other commercial vehicles always have routines, including manuals for checking at bus stops and before departure, and forklifts also require checking in all directions (front, back, left, and right), so it would be possible to evaluate whether the driver is operating according to the normal routines for buses, trains, trucks, and forklifts. Passenger movements could also be shown.

[0277] By recording the viewing state and eye movement history while the video is playing, and analyzing the moments before and after triggers such as activating a turn signal, it's possible to determine if the driver is properly checking blind spots. In driving, it's necessary to look at the right place at the right time, which differs from normal eye-tracking analysis. In the video, the car is moving, and the points of focus are predetermined before triggers, turn signals, and turning the steering wheel. The conditions change over time. Sometimes the timing of the look is delayed. It's important that the analysis is appropriate in correspondence with time. There are places that should be looked at. The system shows what should be looked at and provides a guiding example to show whether the user was looking at the bottom. If approaching an intersection, look here; if turning, look here. The system considers the type of trigger, what to use as a trigger, speed, turn signal, location (intersection), red light, road sign, checking left and right at a crosswalk, looking behind when backing up (not just looking ahead), checking mirrors, only looking at the backup monitor, looking left and right. It's strange to check these things, such as whether the driver is looking ahead while backing up, or driving while looking behind. Eye gaze analysis can determine the driver's orientation from the video footage. It may also detect whether the driver's eyes are open and display this information during playback. Scenes of poor driving could be shown as thumbnails, similar to film reels. For example, since there are specific actions that the driver should be performing, such as not checking mirrors when changing lanes, or not using the turn signal or accelerator, the system could extract and display thumbnails of the parts where these actions are not being performed correctly.

[0278] When scoring driving, it may be excluded from scoring when the vehicle is stationary. Scoring is based on whether it moves smoothly, stops, makes a sharp turn, or brakes suddenly. When the building is moving in the video, the previous vehicle is not moving, and at which location and for how long it was moving, it is considered as the evaluation target. When the vehicle is stationary, it is not considered as the evaluation target. Exclude the stationary part for evaluation. You may skip the part where the vehicle is stationary, fast forward to the next part, and play the video. You may also advance the video when the signal changes from red to blue. When the scorer watches the video, the time can be shortened. Display not only the outer video but also the inner video when the customer gets on, only when the customer gets on. If you play it only when there is a change in the customer, it will become a special feature when the customer gets on. This is because for a taxi, it is sufficient to see the customer's face.

[0279] Also, when searching, if it is surely bright, find a bright video. If there was a Shinkansen, find the Shinkansen video and pick up the Shinkansen video. In this way, it is advisable to pick up from the general things those that meet the conditions from the video. When there is a one-week video and you are told something ambiguous, you will start to search for it. When you want to show it in a presentation and you know it was around there, you can search for it. You may also search for a situation where the light is like this in the parking lot of a supermarket. It is advisable to find it using as keys specific items, such a situation, this time, the recorded state of the car, the speed it reached, sudden acceleration, or the use of the blinker. After a red car overtakes a blue car, Search for the video after a dynamic movement, and it is advisable to find the desired scene from after this continuous scene in the video.

[0280] You may show the actual image at the top and the recognized image at the bottom, discriminate the vehicle type by the image, or show it by the contour. From such ambiguity of the scene, you may present candidates for the desired video scene and let it be selected. For each image, tag it like what is here, and note how long it was there. It may be set that if it has stopped for more than 5 minutes, or you may attach a voice recognition function to the software that plays the video.

[0281] During playback, it would be good to display where the driver was looking. If it is determined that the driver is looking away, a mark can be placed in the playback window 130, or the image can be turned red the moment the driver looks away. This is because the 360-degree camera captures the image and shows where the driver is looking. If the driver is looking away, an alarm may sound. If the driver closes their eyes, an alarm may sound. The playback can be skipped to that section, or only that section can be played. If the camera is not a wearable camera like a 360-degree camera, the direction the user is looking can be determined and that direction can be displayed. A wearable camera is also acceptable. Instead of a wearable camera, a wearable sensor can be used, such as one that is placed on the head or attached to the user using something like a headband. There should be a sensor that can determine which direction the driver is facing. An accelerometer (camera) can be attached to the ear, or a sensor (camera) can be attached to a woman's hair clip. A sensor (camera) can be clipped to the brim of a hat. You could combine those sensors with a 360-degree image to show which direction you're facing. You can analyze it using a 360-degree image. You could also indicate that there's a frame area showing the field of view on the opposite side, or on the back side. You could also indicate that the frame is displayed, but it's not currently visible. You could make the outer frame green, or make the visible area appear clearly, with everything else appearing gray. You could make the clear area expand as you check. You can see at a glance the area you're not looking at. You could clear the area when there's no head movement. You could display the visible and invisible areas only before and after turning the steering wheel, or before and after using the turn signal. You could display the angle the camera is pointing on the map below. You could display both the angle of your line of sight and the angle of the car. You could display the angle of the video itself. You could show how much is visible based on the direction of the car and the direction of your line of sight. You could display the current field of view, or the field of view of the area being displayed.

[0282] A 360-degree camera drive recorder may be equipped with VR (virtual reality) display capabilities. Since the 360-degree camera captures the entire 360 ​​degrees, it can produce images similar to VR on a flat surface. In this case, the reference direction can be the front camera side, the direction of shooting, or north. It is necessary to determine how much of the VR field of view should be mapped to the direction with north as the reference direction. In the case of drive recorder 1, the forward direction is forward. It may also be a playback device that shows the direction in which the main unit is moved. This range is the range that the user can see, but it can be shown that a person 10 years younger can see more. A comparison function may also be included. It can be seen that what you can see now will be reduced to this much in 10 years or 30 years. A function may be provided that allows instructors to switch the angle range for each age. It may also be possible to switch by age. VR display may be based on the direction the driver can see. Playback software for the video recorded by drive recorder 1 that can be displayed on a VR device may be provided. With VR devices, you can display acceleration data as an overlay or as an OSD (On-Screen Display). You can even display acceleration data in 3D. While the image itself cannot be 3D, it can be displayed semi-transparently in front of the visible surface within the sphere. You can also place panels in areas where they are not needed. You can tell just by moving your head. Where your gaze is directed should be in front of you; overlay-like elements should also be in front. The intensity can even change depending on how long you are looking at it. Alternatively, the image could be made denser or sparser depending on how often the viewer is looking. When the user turns sideways, they could see the side view; when they turn upwards, they could see the image above. The direction the user is facing can be detected using an accelerometer, geomagnetic sensor, or gyroscope. The direction of forward is determined by the fixed camera position. The center of the sphere can be considered the front. The camera is always facing forward. The image data can then be fed into the framework.

[0283] From the perspective of an administrator, there's no point in watching footage when the vehicle is stopped at a traffic light or otherwise stationary, so it would be good to have a function that automatically recognizes and skips such footage in the playback software (program).

[0284] On a map, the vehicle's movement path could be displayed, and a function could be added to jump to a specified point on that path. The time of day corresponding to a point on the map would be indicated, and the time would be displayed when the cursor is moved over, allowing the user to find the image from that time. Street View could also be displayed. A function could be added to switch between Street View images and dashcam footage. Street View includes both past and present images, so it would be possible to see what the scenery looked like a year ago. Footage from three years ago of a route driven could be viewed. Footage before and after a building was constructed could also be viewed. It would be good to display this in a separate window. It would also be good to be able to switch to older maps. If the date of recording is included, it would be possible to display footage of a desired season, such as autumn or spring. If the user drives the same route for commuting, summer and winter footage could be displayed. The current footage of the route being driven could also be displayed. On rainy days, it would be possible to show what the scenery looked like.

[0285] In the playlist display area 172 of the playback window 130, information indicating the location of the video can be displayed. For example, information indicating that the video traveled from Okazaki City to Toyota City can be displayed. This can be displayed as text or as a zoomed-out map. This makes it much easier to understand and use. If municipal and area data is available, it becomes possible to display "from XX City to YY City". By using the Location API (Application Programming Interface) and sending latitude and longitude to a search server, information such as "the location is XX City" will be returned, and this information can be output. The destination of the video file can be automatically assigned. When videos are lined up, if you commute by car every day and don't care about the commute videos, and only want to see travel videos, categorizing the videos from here to here as "commuter" makes selection easier. For example, in the case of XX City, it is good to categorize videos where the starting and ending points are the same and bring the rarer ones to the top. This is because if you have only been to a place once or so, it is likely to be a travel video.

[0286] The video file header recording area 31 stores GPS data periodically, such as per frame, every second, or every 0.5 seconds. To name the video file after a specific place name, such as "Higashi-Okazaki," the coordinates can be sent to the location search server's API, and the information returned from the server can be used to create the video file name. GPS data can be sent to a server that stores map images, and the map can be retrieved and displayed from that server. If the personal computer 80 has map data and place name information, there is no need to send it to an external server. Event information can also be displayed. Event data can be displayed in the playback window 130, showing information such as "When you traveled this road, events occurred at this scene, at this scene." Where sudden braking occurred, etc., can also be shown on the map. For a commute route, events that occurred and changed at each point can be summarized and displayed. The route can be displayed, and information such as "When you traveled this road, these events occurred" can be overlaid on the map. For example, if you tap on a road on a map, such as National Route 248, events that occurred while traveling on National Route 248 will be displayed on the map for a certain period, for example, five years, and at that location. This shows areas that are particularly dangerous. All logs are displayed, and clicking on them brings up logs, heart rate data, and they can be categorized, such as by cycling. Areas where minor incidents occurred, where the accelerator was pressed hard, or specific locations prone to accidents can be identified, compiled, and displayed. Commuting video files can be categorized and displayed, and clicking on them will map them all onto a map.

[0287] Alternatively, some kind of driving evaluation could be performed at regular intervals, for example, from this intersection to that intersection, and this evaluation could be automatically saved to the server. When playback is started, other people's evaluation data will also be accumulated for each section, so it would be good if the evaluation of one's own driving—whether it is reckless or safe—could be displayed in the playback window 130 while playback is in progress, allowing the user to compare their driving to that of others. Below the video, a checkmark or a crying face could appear depending on the evaluation, or a score could be displayed, or the average score of other users could be shown.

[0288] Furthermore, the video footage could be saved to the server at regular intervals, allowing for comparison of the same location. For example, a comparison could be made between specific points along a road, such as National Route 1. Driving scores are assigned based on acceleration, speed, sensors, gyroscope, and braking technique, but the question remains whether that score is truly acceptable or unacceptable for that particular road. On winding roads, the steering wheel is naturally turned multiple times, so a rating of excessive weaving might not be appropriate, as this is due to the curves of the road. Instead of absolute evaluation, relative evaluations could be used, or footage of other drivers could be displayed. The video could show the top-ranked driver, indicate that the user is ranked 30th, and present examples of excellent driving. While there's the question of whether it's acceptable to display the top-ranked video, a permission function could be provided. A prompt could ask, "Is it okay to provide this?", "Do you want to make it viewable by others?", or include an upload button. The camera (drive recorder 1) itself could have communication capabilities to upload or transmit score data. You can confirm when installing the playback software on Personal Computer 80, or you can do it all at once during installation, or you can let users choose to upload each video file individually, or you can have an permission button appear the moment they try to upload. You can confirm the video files before uploading, but you can upload the scores without confirming them. You can also have a rule that you will not link users to information about where they were. You can delete date data when uploading. The day of the week and time are acceptable, but it is better to delete any personally identifiable information. If only the location and time are included, and the car model and personal name are not included, it is acceptable as long as it is not possible to tell whose data it is.

[0289] It would be good to be able to specify the resolution for each location: roads with no obstacles, such as farm roads, are less likely to have accidents, so they will be filmed (recorded) but at a lower resolution; and roads in urban areas, where there is a risk of sudden appearances, will be filmed (recorded) at a higher resolution. It would be good to be able to specify that the range from here to here should be recorded at a low resolution, and this area at a high resolution. This will allow for longer recording times. The location is specified using the playback window 130. The user can specify in advance the range that is acceptable to record. It would be good to circle this range on the map and set the playback window 130 to film (record) at a low resolution when entering this range. Insert the memory card 25 with the recording settings into the drive recorder 1. Then it will operate accordingly. If the drive recorder 1 has an LCD display, the settings can also be made on the drive recorder 1. You can also adjust the resolution here when parking surveillance. You can also change the resolution while driving, because there is a place where absolutely nothing will happen. You can just film (record) in detail using event recording. If all the data is stored on the cloud (server) side, it's possible to know whether an event will occur or not in a given location, allowing for resolution adjustment on the cloud side. Low-probability areas are recorded at low resolution (frame rate), while high-probability areas are recorded at high resolution. Since this is an event switching mechanism, the events occur... Sometimes it's good to use high resolution. You can record in high resolution at your destination and in low resolution during the journey there. You don't need to record when driving on the highway. You can also record only when an event occurs (event recording). For example, if you're going to Kyoto, you can record in high resolution while you're in Kyoto city and stop recording once you leave Kyoto. You can also record at one frame per second or a low frame rate. You can pre-set the locations where you want to upload videos to video sharing sites. For example, if you go to a race track, you can record the track in high quality and not record the journey there. You can specify areas to record at a high frame rate (high quality) instead of areas to record at a low frame rate (low quality). When parked with parking surveillance enabled, you can record at a low frame rate.

[0290] For example, if you decide to film in Kyoto in high resolution, it's a good idea to record the journey there and back as well. If you don't record, the dashcam is useless, and even if you try to set it after arriving in Kyoto, you can't because you're driving. For example, you could record in high resolution only near Kiyomizu-dera Temple, but you might forget to set it, so this prevents that from happening. Also, it's a good idea to record in high resolution while driving on mountain passes. You can record on highways in high resolution and on ordinary roads in low resolution. You could record on the Metropolitan Expressway in high resolution and not record on regular highways, or you could set it to record when the speed exceeds 100 km / h, or when acceleration occurs in corners. Area settings can be changed by providing a slide switch on the dashcam 1 to switch between high resolution and low resolution.

[0291] A 760MHz ITS service system, providing advanced road information, has been operating as an experimental service since around 2016, and it would be beneficial if this information could be recorded together with the information from Drive Recorder 1. This information includes, for example, information about other vehicles in the vicinity of the vehicle (position, direction of movement, acceleration, etc.), traffic lights (how many meters ahead, time until they turn red, etc.), road signs (how many meters ahead, speed limit, etc.), pedestrians (position, direction of movement, etc.), emergency vehicle approach notifications, sudden braking warnings, traffic congestion warnings, malfunctioning signal vehicle warnings, sign notifications, motorcycle approach notifications, signal timing notifications, construction information notifications, and weather warnings. ITS has both vehicle-to-infrastructure communication and vehicle-to-vehicle communication. Vehicle-to-infrastructure communication is communication between the road and vehicles, and vehicle-to-vehicle communication is communication between vehicles. In vehicle-to-infrastructure communication, information such as vehicle number information and information obtained from sensors of surrounding pedestrians is transmitted. In vehicle-to-vehicle communication, information such as the direction the vehicle is going and the current accelerator status is transmitted. For example, if vehicle-to-infrastructure data is received and saved along with the video, then, for instance, if an LED traffic light is flashing, and the frame rate of the camera is synchronized with the frame at which the flashing has stopped, the light of the traffic light may not appear in the video at all, and in some situations, it may remain dark indefinitely. Even if the light of the traffic light is not visible, the ITS information can record the state of the traffic light as data. If the ITS information indicates that the signal is green, then the information that the signal is green can be displayed at the edge of the playback window 130. Since information is beginning to be provided using a dedicated frequency, it can be received by a receiver, recorded together with the video from the drive recorder, and displayed in the playback window 130.

[0292] It would be good to record ITS service information with drive recorder 1 and display that information. It would be good if drive recorder 1 could record information such as a car coming from the side, a bicycle coming, crossing on a red light, crossing on a yellow light, not braking before the yellow light, etc., onto memory card 25. It would be good if drive recorder 1 could receive and record information about other vehicles in the vicinity of your vehicle (position, direction of movement, acceleration), how many meters ahead there is a traffic light, how many meters ahead it will turn red, the type of road sign, and also detect and transmit pedestrians. It would be good if drive recorder 1 could record information such as when an emergency vehicle is approaching, and when the car in front brakes suddenly, information is sent between vehicles via a bucket brigade, so it would be good if drive recorder 1 could record such information onto memory card 25. Also, it would be good if it could record information such as how many vehicles ahead braked suddenly, as this can be known in advance. In the case of road-vehicle-to-infrastructure traffic, this information is not transmitted. Information is transmitted from an antenna near a specific point. When a vehicle enters an area of ​​about 5 meters from that antenna, the drive recorder 1 automatically receives the information. Between vehicles, vehicles that can receive information from the road can also transmit their own information. This includes information such as current speed in kilometers per hour, turning left based on the car navigation system, and whether the brakes were applied. When receiving information between vehicles and the road, the drive recorder receives it. The drive recorder also receives information between vehicles, and can receive information transmitted by neighboring cars because they use the same frequency. It is best for the drive recorder to only receive and not transmit. The ITS device is a transceiver, and the drive recorder 1 records the communication between the car equipped with the drive recorder and other cars. It simply intercepts the information. It would also be possible to install software in the drive recorder that allows it to transmit information to other cars after receiving it.

[0293] This system would be useful in situations where there is a building at an intersection and a car approaches from a blind spot, allowing one car to communicate with another to prevent accidents by recording how the other car behaves, whether it's braking or not. It could also be used with cameras at the intersection to alert drivers of pedestrians. Since the system is designed to encourage caution when turning, it would be helpful for administrators to monitor if they can see when a hidden vehicle is present and the brakes are applied. It would also be beneficial if it recorded instances where a car failed to brake despite a pedestrian being detected, even though the car's display showed the signal was red and wouldn't turn green for another 10 seconds, resulting in poor fuel efficiency.

[0294] In the playback window 130, information about the car to which the drive recorder 1 is installed is displayed at the bottom. For example, speed and turn signal operation status. In addition to such information, it would be good to display the information obtained from the ITS system in a separate location.

[0295] For example, since we know that there is a car a certain number of meters ahead to the right, we could use an overlay to show the time remaining until the traffic light appears the moment the traffic light is visible, if there is a traffic light a certain number of meters ahead where the car is located. This could also be displayed as text. It could be displayed as text or an icon. The callout is just one example of an icon.

[0296] Since dashcam footage is used as objective evidence of the accident, it is desirable that it cannot be tampered with. It is also desirable that any tampering be detectable. Furthermore, the footage recorded by dashcam 1 is not a recording of the scenery as remembered by the driver, but rather objective footage recorded in accordance with the use of the cameras included in dashcam 1 (front camera 11, rear camera 12) at that location. This is because the installation location of dashcam 1 is different from the driver's line of sight, and the recording system of dashcam 1 is different from the human visual system. The differences between a dashcam's recording system and the human eye's imaging system include, of course, differences in field of view, sensitivity, head movement, eye movement, and optical system, as well as differences in the imaging sensor and the structure of the eye. However, the most important difference lies in the cognitive structure of the brain. Therefore, it would be beneficial to obtain the image remembered by the driver and equip the dashcam 1 with a function that can produce an image as close to that as possible. Based on the differences between the objective image recorded by the dashcam 1 and the image remembered by the driver, the driver can be educated or any distortions in the driver's perception can be corrected.

[0297] It would be good to be able to process the colors and other aspects of the video recorded by Drive Recorder 1 so that it matches what the driver expected. It would be good to process the video to reflect the objective perception and display an image that closely resembles the impression the driver felt.

[0298] To detect tampering, the hash value of the data recorded when recording video from drive recorder 1 is calculated, and the hash value is stored in relation to the video, as well as in a blockchain on a P2P (peer-to-peer) network connected to the network. Previously, one entity stored all the video data on a server, but with a distributed data database, everyone connected to the internet can share that data, and each server will have the same data. This involves attaching blocks, which are thrown onto the internet, to the existing blockchain, creating a chain that continues to grow. The data in that chain will be the same data recorded in the database on all servers. The key feature is that since everyone has the same data, tampering can be detected unless the majority of people tamper with it.

[0299] The way the data is created involves calculating the hash value of the previous block and always including it in the next block when linking blocks together. It's impossible to tamper with a single block; to do so, all subsequent blocks would have to be tampered with, making tampering extremely difficult. The computational load required to calculate the value of a blockchain is set so that it takes about 10 minutes for Bitcoin, and recalculating all the blocks would be impossible time-wise. The entire flow of a contract can be recorded on the blockchain, or a bundle of contracts such as import / export procedures, including the people who transport goods within the destination country, the people who make payments, the people who exchange currency, and how orders are handled, can all be recorded on the chain. For example, information such as "it passed through here on this date" and "it was loaded onto the ship" can all be recorded on the chain so that it can be verified later. It can also be used in areas where interests conflict and where a verification mechanism is needed. Furthermore, in a contract where an advertiser buys advertising space and, for example, agrees to run several 15-second commercials between 7 and 8 pm during prime time over a month, it's impossible to verify whether all the ads were actually run. Similarly, with transit advertising, it's impossible to verify where all the ads were displayed. With online advertising, it's impossible to verify whether the ad actually received that many impressions or was displayed on the target user's browser. However, if an audit is needed between the advertiser and the person actually running the ad, blockchain technology can be used. For example, if an ad is run on the blockchain, it can be linked together in a chain, indicating that the ad was run 1000 times and that it has now been run 800 times.

[0300] To prevent tampering, video data should be recorded on a blockchain. For example, in the case of video based on Motion JPEG (Joint Photographic Experts Group), where data is stored in each frame, the hash value of that frame can be calculated and transmitted as a block, allowing the blockchain system to link it into a chain, thus preventing tampering. Alternatively, the hash value of the entire video file for a certain range, such as one minute, could be calculated.

[0301] In MPEG-based video data, splitting the data immediately before an I-frame makes it easy to tamper with, so it's better to split it immediately before a P-frame.

[0302] In a public blockchain, like Bitcoin, anyone can participate and view all the data in the blocks flowing through it. In contrast, a consortium-type, private blockchain is closed off, for example, within a banking association, and can only be used among the banks that are members of that association. For example, you can use a public blockchain system. Not just hash values, but also location information, such as the current location of a drive recorder, can be sent to the blockchain system. This sends information about who is where on what date. For example, if information that drive recorder 901-1 is located in Okazaki City, Aichi Prefecture is sent to the blockchain system, the server managing all the blockchains will broadcast this information via P2P. Video data, etc., can also be transmitted to the blockchain system. If you send it to [a specific platform / website], you can sell it to any user who wants it.

[0303] Similarly, advertisements can also be delivered. For example, you could send an advertisement for a store to users in a specific location, such as Okazaki City, Aichi Prefecture, and then pinpoint the advertisement to only those who are just before the store. Maintaining a blockchain incurs costs, but a small amount is deducted as a fee when a transaction occurs and distributed to the user managing the server and the user who first created that block. The information about the currency paid can also be recorded on the blockchain. For example, if someone asks you to take a picture for 100 yen, the 100 yen is deducted from the account of the person who made the request, about 80 yen goes to the user of the drive recorder, and 20 yen is distributed to the person maintaining the server.

[0304] It would be beneficial to be able to zoom in and out of the video using a remote control or similar device while the video is playing. Furthermore, it would be good to be able to move a frame using a cursor, for example, and store the position of that moved frame in relation to the video's time. Ideally, when the frame is moved during playback, its position should be recorded, and later, based on that recorded frame position information, it would be possible to create a video stream by extracting only that portion of the video.

[0305] It would be beneficial if the field of view could be manually adjusted rather than automatically detected. For example, if the frame is moved using a joystick, mouse, or keyboard, the system could remember the moved position and display that frame during playback. This would allow others to recognize the range of vision a person had, as it varies depending on factors such as concentration levels while driving, age, and eye conditions. For instance, a person could claim that they saw this range or that they were looking at this range. It would also be beneficial to allow for subjective adjustments to brightness and color, as well as adjustments to remembered colors. In particular, it would be good to display both objective and subjective footage, allowing for a comparison between the footage captured by the camera and the footage the user believes they actually saw. Displaying two different videos of the same scene would allow a person to explain in court that while the camera actually recorded it this way, they saw it this way. In terms of criminal liability, two important factors are the objective occurrence of the accident and its consequences, and intent. It is crucial to prove whether the person hit the vehicle intentionally or negligently. Objective images cannot be manipulated, but subjective images can be presented in a way that differs from what one actually perceives, or both can be presented together.

[0306] Furthermore, the hash values ​​of the vehicle's location information and event information may be calculated and transmitted to a blockchain system. This makes it difficult to tamper with event information such as sudden deceleration or acceleration.

[0307] Furthermore, in order to adjust the brightness, color, and memory color to the subjectively perceived values, a brightness adjustment slider may be provided on the drive recorder 1, and the brightness, color, and memory color may be adjusted by operating the slider. Alternatively, brightness adjustment buttons, color adjustment buttons, and memory color adjustment buttons may be provided on the playback window 130, and the brightness may be adjusted by operating these buttons.

[0308] Furthermore, in the case of autonomous driving of vehicles such as automobiles, it is preferable to record data in the video file header recording area 31, as described above, that identifies whether the drive recorder 1 is recording during manual driving where a driver is operating the vehicle, or during autonomous driving where the vehicle is operating automatically. In this case, it is preferable to also record the level of autonomous driving in the video file header recording area 31. When identification data indicating whether autonomous driving or manual driving is in progress, and data indicating the level of autonomous driving if autonomous driving is in progress, are transmitted within the vehicle's CAN (Controller Area Network), the drive recorder 1 receives the data communicated within the CAN (for example, the drive recorder (By connecting the driver 1 to the CAN bus or receiving data communicated within the CAN bus via the communication circuit 17), the drive recorder 1 can record identification data indicating whether the vehicle is in autonomous or manual driving mode, and if it is in autonomous driving mode, data indicating the level of autonomous driving, in the header recording area 31 of the video file. When playing back the video data, the playback window 130 can display whether the video displayed in the video display area 141 is from autonomous driving mode or manual driving mode, and if it is from autonomous driving mode, the level of autonomous driving can also be displayed.

[0309] The drive recorder 1 may be configured to record video data with different image quality settings depending on whether the vehicle is driving autonomously or manually. For example, it may record at a relatively high resolution when driving autonomously and at a relatively low resolution when driving manually, or vice versa. Furthermore, the image quality may be adjusted according to the level of autonomous driving. The higher the level of autonomous driving, the higher the image quality may be recorded, or conversely, the higher the level of autonomous driving, the lower the image quality may be recorded.

[0310] Furthermore, if the drive recorder includes multiple cameras (front camera 11, rear camera 12), the camera recording may be switched depending on whether the vehicle is in autonomous or manual driving mode. For example, video data obtained only from the front camera 11 may be recorded during autonomous driving, and video data obtained only from the rear camera 12 may be recorded during manual driving, or vice versa. In addition, during playback, only the autonomous driving footage may be played back, and the manual driving footage may be skipped, or vice versa. Furthermore, video data recorded at one level of autonomous driving may be played back, and video data recorded at other levels may be skipped.

[0311] Furthermore, when playing back video data, the communication robot may provide guidance on the driving techniques shown in the played-back video. While direct explanations from an instructor to a user can sometimes anger the user, a communication robot saying "This was dangerous" reduces the psychological burden on both the instructor and the user. The communication robot is positioned near the instructor and user who are viewing the display device 92 of the personal computer 80 playing the video. For example, it is preferable to always place the communication robot to the right of the display device 92 of the personal computer 80, and to pre-determine the seating positions of the instructor and user. The video data playback software (program) may also allow settings such as whether the communication robot is placed to the right or left of the display device 92 of the personal computer 80. Of course, the communication robot may also be equipped with a camera that captures a 360-degree view of its surroundings, recognizing the positions of the display device 92, instructor, and user. The robot can recognize individuals in uniform as instructors and other individuals as users.

[0312] Furthermore, the system may detect the width of the road included in the video footage obtained by the drive recorder 1 or the video data obtained by the drive recorder 1's recording, and modify the video processing based on that road width. For example, in places where the road is narrow, the system may display video in the left and right directions, or change the type of video during playback according to the road conditions (narrow, gravel road, one-way street, etc.). Also, at intersections, if the drive recorder 1 is capable of displaying video during recording, it may display video of the road in front, and similarly display video of the road in front during playback. Additionally, for example, the system may display video of the right or left side of the road on the display screen 3 of the drive recorder 1, so that the road on the right or left side is visible.

[0313] Furthermore, when recording video data by drive recorder 1, drive recorder The system can analyze video footage from a short distance ahead of the vehicle to which the camera is installed, perform image analysis to determine if a dangerous situation is occurring, and switch the video feed to a camera that better shows the danger a few seconds before the danger occurs. Alternatively, the display mode can be switched to show the current video feed, for example, if there is a danger in 2 seconds. For instance, if a car is crossing in front of the vehicle in 2 seconds, the system might switch from the sphere mode (showing the situation in front, behind, and to the sides of the vehicle) to a mode that only shows the front. This allows the dangerous situation to be seen in 2 seconds. The system can then switch back to the original sphere mode. In this way, the display mode can be automatically changed when something unusual is likely to occur. Furthermore, the system could automatically analyze all video files in advance, detect any anomalies, and mark the video to be played. Whether something is dangerous can be determined by whether there is a large change in the video or a large change in acceleration. The display can be changed without the user having to operate a switch button. For example, it would be preferable to switch a few seconds before a dangerous event is likely to occur, for example, 5 seconds beforehand.

[0314] Furthermore, the drive recorder 1 may receive communication data from the engine starter and car security system, and use the received data as a trigger for event recording. The drive recorder 1 may be equipped with communication circuits based on LTE (Long Term Evolution), Wi-Fi, and Bluetooth®, record data representing the status of each communication, and perform processing to change the state of the displayed video according to the respective communication status. The display change processing may be a change in the display format of the video displayed on the display screen 3 of the drive recorder 1, or a change in the display format of the video in the playback window 130 during playback. For example, when a command signal to turn the car security system on or off is received, the system may automatically switch to video recording of the left and right sides of the vehicle, around the doors. Also, when a Wi-Fi signal is received, it is assumed that the vehicle has entered the home area, so the video may be stopped. When there are multiple means of communication, it is advisable to decide which video to display according to each means of communication. The display format may be switched according to the means of communication or the content of the communication. If Bluetooth® is used to unlock car doors or start the engine, the system may switch from displaying the surrounding area video to displaying the interior video as well, depending on the reception of the Bluetooth signal. It may also capture the interior of the vehicle as well as the surrounding area. This allows the system to capture the interior state at the moment the engine starts. When using the remote starter, the system may switch back to the original state after about 30 seconds. When unlocking the doors, the system may maintain recording both the surrounding area and the interior of the vehicle until the doors are locked, and then switch back to recording only the surrounding area or only the front of the vehicle.

[0315] Furthermore, in the operations of buses, taxis, and logistics companies, roll calls are always conducted. For example, alcohol checks are mandatory to check if drivers are in good physical condition. The video of these roll calls is recorded. It may be possible to view the roll call video and the drive recorder video together in the playback window 130. For example, by specifying the roll call video file, it would be possible to play the video recorded by the drive recorder 1 of the driver who was rolled over. For example, if drivers are rolled over in order and then board the vehicles in order, the roll call video and the drive recorder 1 video can be associated based on their temporal relationship.

[0316] If a drive recorder 1 is installed on both a forklift and a vehicle such as a truck, it would be beneficial to transmit the video recorded by either drive recorder 1 to a server so that the video of the vehicles (truck and forklift) driven by the same driver can be viewed as a continuous sequence of events throughout the day. For example, if a driver drives a truck to a certain location, then drives a forklift at that location, and then gets into a vehicle to go to a delivery destination, it would be beneficial to be able to view this entire sequence of video. By combining this with roll call video, it would be possible to see the driver's entire daily routine.

[0317] Furthermore, a composite reality system can be used for driving instruction. For example, the user drives once, and the video is recorded on a drive recorder. Then, in the playback window 130, the video is played back and the driving skills are automatically evaluated. During this automatic evaluation, information on points to pay attention to is added to the video data. Subsequently, the user wears MR glasses and drives or rides in the passenger seat over the same location, and the information is displayed on the MR glasses. The user can see the information on points to pay attention to superimposed on the actual scenery. For example, information such as where to look for traffic lights and where to look for stop lines is displayed on the lenses (glass) of the MR glasses, and the user can confirm this information while looking at the actual scenery.

[0318] Additionally, the driving history data can be recorded in the memory 21 of drive recorder 1, and when approaching a previously visited location, it can display when, where, and what route was taken. The driving history data can also be used to display or announce whether the location has been visited before.

[0319] Furthermore, the rear camera 12 of the drive recorder 1 may capture and recognize the vehicle's meter, which can then be used as a trigger to start recording video data. Alternatively, the vehicle's interior light may be captured, and if it is detected that the interior light is on, event recording may be started and data indicating that the interior light is on may be sent to a server. From that server, for example, data indicating that the interior light is on may be sent to the vehicle's user. The user will then know that the interior light is on.

[0320] One option is to equip automated guided vehicles (AGVs) in factories with drive recorders (cameras). AGVs are equipped with LIDAR (Light Detection and Ranging) sensors, which can use lasers to scan at high speed and detect surrounding objects. When an approaching object is detected, an event recording can be triggered (recording for a set period, for example, several tens of seconds to several minutes, depending on the trigger).

[0321] Image recognition can be used to trigger event recording the moment the forklift's forks move. Alternatively, the front camera 11 and rear camera 12 of the drive recorder 1 can be used to record, and when the forklift moves, this can be used as a trigger. Furthermore, event recording can be started when a person wearing a helmet gets on the forklift, or conversely, when a person without a helmet gets on. In this case, recording can start when a person gets on and stop when they get off. The start and end of recording playback can also be determined by when a person gets on or off. A characteristic of forklifts is that while they are owned by a company, they are driven not only by the company's employees but also by transportation companies. Because the driver changes, recording can be started when such a person gets on, or when a person approaches, or until 3 minutes after they get off, then stop. Forklift accidents are a problem, but this could help determine who caused the damage.

[0322] Furthermore, when a photograph is taken, Exif information is attached, and text data is embedded within the JPEG data. The exposure and shutter speed used at the time of shooting are embedded. This information may be displayed in the playback window 130 using playback software. Still image files may be generated from video footage, and the Exif information may be recorded in the header recording area of ​​the still image file. For example, it is preferable to record the information displayed in the playback window 130 as Exif information in the header recording area. For example, a series of acceleration changes in the video may be historically recorded in the header recording area of ​​the still image file. GPS information and speed, the maximum acceleration when moving within a certain range, the speed of the car, whether the brakes were applied, whether the turn signals were used, etc., may be recorded as Exif information in the header recording area. The entire range of speed may be recorded, or all values ​​for the types of events that occurred within plus or minus 3 seconds before and after a certain event occurred, if the drive recorder has multiple cameras. In some cases, information such as which camera's footage was recorded, how many cameras were recording, and where the camera was mounted on the vehicle may be recorded as Exif information in the header recording area. Furthermore, if a drive recorder is installed in an autonomous vehicle in a factory, the trigger cause, whether the indicator light error was red or yellow, the status of the forklift, the amount of rotation of the forklift's turn, the status of the gyroscope, battery, etc., may be recorded as Exif information in the header recording area. Information obtained from AWS (Automatic Guided Vehicle), CWS (Collision Warning System), stoppers, vehicle-to-vehicle communication, and vehicle-to-infrastructure communication may also be recorded as Exif information in the header recording area. Information such as whether there are people or not, whether there is a driver or not, whether there is a person in the passenger seat or not, whether there is a person sitting in the back seat or not, whether it is a commercial vehicle or a different vehicle, a taxi, a bus, and whether there are people inside the bus or not may also be used as triggers, and this information may also be recorded as Exif information in the header recording area. Furthermore, the information recorded as Exif data may be sent to a blockchain system and recorded in the blockchain as a transaction.

[0323] Furthermore, points may be awarded to the user in accordance with the use of playback software (program) that displays the playback window 130. For example, when connected to the internet, the imported data may be uploaded and points added when the import button is pressed. If it is a communication-type drive recorder 1, once permission is granted, video data can be transmitted automatically.

[0324] The scope of the present invention is not limited to the configurations explicitly described in the specification, but also includes combinations of various aspects of the present invention disclosed herein. While the configurations for which patent protection is sought are specified in the appended claims, the present invention intends to include configurations disclosed herein that are not currently specified in the claims in the future.

[0325] The present invention is not limited to the configuration described in the embodiments above. The components of each embodiment and modification described above may be arbitrarily selected and combined. Furthermore, any component of each embodiment and modification may be arbitrarily combined with any component described in the means for solving the invention or any component that embodies any component described in the means for solving the invention. We intend to obtain rights for these as well in amendments or divisional applications of this application.

[0326] The scope of the present invention is not limited to the configurations explicitly described in the specification, but also includes combinations of various aspects of the present invention disclosed herein. While the configurations for which patent protection is sought are specified in the appended claims, the present invention intends to include configurations disclosed herein that are not currently specified in the claims in the future. The present invention is not limited to the configuration described in the embodiments above. The components of each embodiment and modification described above may be arbitrarily selected and combined. Furthermore, any component of each embodiment and modification may be arbitrarily combined with any component described in the means for solving the invention or any component that embodies any component described in the means for solving the invention. We intend to obtain rights for these as well in amendments or divisional applications of this application. Furthermore, the applicant intends to acquire rights to the overall design or a partial design by filing an application for amendment to the design application. The drawing depicts the entire device with solid lines, but it is a drawing that includes not only the overall design but also the partial design claimed for a part of the device. For example, it is a drawing that includes a partial design for a part of the device, as well as a part of the device that is unrelated to a component. The part of the device may be a component of the device, or a part of that component. The drawing includes the overall design as well as the solid lines. I intend to obtain a patent for a partial design in which any part of the design is represented by a dashed line. [Explanation of symbols]

[0327] 1: Drive recorder, 2: Mounting section, 3: Display screen, 4: Camera section, 5: Front lens, 6: Rear lens, 7: Text input field, 10: Controller, 11: Front camera, 12: Rear camera, 13: Display control device, 14: Display device, 15: Clock, 16: Infrastructure / vehicle-to-infrastructure information receiving circuit, 17: Communication circuit, 18: Accelerometer, 20: Input device, 21: Memory, 22: GPS receiver, 25: Memory card, 31: Header recording area, 32: Video recording area 80: Computer, 81: Display control device, 82: Display device, 90: CPU, 92: Display device, 93: Accelerometer, 95: Memory card reader / writer, 96: Input device, 97: Memory, 98: Communication circuit, 130: Playback window, 131: Menu bar, 132: Folder icon, 133: Camera icon, 134: Printer icon, 135: Reel icon, 136: FD icon, 137: Log icon, 138: Gear icon 141: Video display area, 141A: Video display area, 141B: Video display area, 142: Area outside the field of view, 143: Frame, 143A: Frame, 144: Mark, 144A: Mark, 145: Mark, 146: Speech bubble, 146A: Speech bubble, 146B: Speech bubble, 147: Speech bubble, 148: Dashed line, 151: Map image display area, 152: Icon, 161: Vehicle information etc. display area, 162: Vehicle status display area, 163: Operation button, 166: Time display area, 168: Acceleration graph display area, 17 1: Playlist display area, 172: Playlist display area, 280: Communication robot, 281: Torso, 282: Head, 283: Eyes, 284: Mouth, 291: CPU, 292: Memory, 293: Communication LAN board, 294: Wireless communication circuit, 295: Motor control board, 296: Head motor, 297: Sensor board, 298: Eye camera, 299: Audio input / output board, 300: Speaker, 301: Microphone, 310: Windshield, 311: Dashboard

Claims

1. A playback device for playing back recorded video data, During playback of the aforementioned video data, the frame is moved in response to input from the input device. The position of the moved frame is stored in relation to the time of the video data, The system controls the display of the stored position of the frame during playback. A regeneration device characterized by the following features.

2. The control includes a control that generates a video stream in which only that portion is extracted, based on the stored information about the position of the frame. The regeneration apparatus according to feature 1.

3. A program for a computer to implement the functions of the playback device described in Claim 1 or 2.

Images