System, program, trained model, training model generation method and generation device, etc.

Abstract

To provide a technique to detect scene of a specific image which can be captured in a vehicle.SOLUTION: Moving image data obtained by imaging a subject by a drive recorder or the like is inputted to a learned learning model (step 93) to obtain a feature image from the learning model (step 93). When the coincidence between a desired image and the feature image is equal to or larger than a threshold (YES at step 95 and step 96), event recording is performed (step 98).SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present invention relates to, for example, a system, a program, a trained model, a method and device for generating a trained model, etc. [Background technology]

[0002] In conventional technology, the occurrence of a specific event in a shooting device such as a drive recorder triggers the creation of event recording information. In such a device, it has been considered to efficiently collect shooting data obtained by shooting under abnormal conditions (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-016227 A Summary of the Invention [Problem to be solved by the invention]

[0004] It would be desirable to have a technique for efficiently collecting images of specific scenes, such as scenes desired by a user, in addition to image data obtained by photographing in abnormal conditions.

[0005] In view of the above-mentioned problems, one object of the present invention is to provide a technique for detecting a specific image scene that can be captured by a vehicle.

[0006] The object of the present invention is not limited to this, and the applicant intends to obtain rights for configurations that aim to obtain effects from parts of the configurations disclosed in this specification and drawings, etc., by filing a divisional application, amendment, etc. For example, this specification discloses problems in which the parts described as "can" and "is possible" in this specification are read as "is the problem." The problems are described as being independent, and the applicant intends to obtain rights for the configurations for solving each problem separately by filing a divisional application, amendment, etc. Even if the problems are implicitly understood from the description in the specification, the applicant intends to include part of the configurations described in this specification in the scope of the patent claim by amendment or divisional application. In addition, the applicant has disclosed a configuration that solves a problem that combines these independent problems, and intends to obtain rights for it. [Means for solving the problem]

[0007] (1) The system of the present invention has a function of detecting an image representing a specific image scene that can be captured by a vehicle from images captured by the vehicle at each of a number of points in time, based on data defining the scene.

[0008] In this manner, a technique can be provided for detecting specific image scenes that can be captured by a vehicle.

[0009] (2) The vehicle has a function of taking images using a camera installed in the vehicle and a function of recording images taken by the camera at each of multiple points in time on a recording medium, and the detection function may detect, from among the images recorded on the recording medium at each point in time, an image that represents a scene that has a degree of similarity with the scene of the specific image that is equal to or greater than a threshold value.

[0010] In this way, it is possible to detect, from images captured by a camera installed in a vehicle, an image showing a scene whose degree of match with a scene in a specific image is equal to or exceeds a threshold value.

[0011] (3) The above data may be images taken at a single point in time.

[0012] If a specific image is from a single point in time, the time to detect the scene can be reduced.

[0013] (4) The above data may be images taken at two or more points in time.

[0014] The accuracy of the scene detection can be improved if the specific images are images from two or more points in time. The images from two or more points in time are preferably images that are continuous in time, such as a video (images in which the movement of the main subject is continuous within the images).

[0015] (5) The above data may be generated by deep learning by inputting an image identical to the specific image or multiple images similar to the specific image.

[0016] In this way, deep learning techniques can be used to detect specific image scenes that can be captured by the vehicle.

[0017] (6) The above data is a first learned model obtained by machine learning using the above specific image as training data, and the detection function may detect an image representing the above scene by inputting images taken at each of the above points in time into the learned model.

[0018] In this way, a particular frame can be detected relatively quickly.

[0019] (7) The specific scenes may include, for example, at least one of a scene that may lead to an accident or disaster, and a scene that has actually led to an accident or disaster.

[0020] In this way, scenes that may lead to an accident or disaster, or scenes that are close to scenes that have led to an accident or disaster, can be detected from the video.

[0021] (8) The image display device may further include a warning control means for controlling a warning device to issue a warning in response to detection of an image representing the scene.

[0022] For example, the above system can be installed in a vehicle and used to record video while shooting. When a scene similar to a specific image is found in the video, the driver of the vehicle can be alerted that the situation is similar to the scene in the specific image.

[0023] (9) The warning control means may control the warning device so as to change the contents of the warning according to the type or degree of danger of the scene.

[0024] In this way, the driver of the vehicle can easily grasp the degree of caution required.

[0025] (10) It is preferable that the image processing apparatus further comprises a first recording control means for controlling a recording device so as to record an image of the detected scene on a recording medium.

[0026] In this way, an image of a frame of a particular scene can be recorded, making it easier to refer to the image of that frame later, for example.

[0027] (11) The detecting function may detect an image representing the scene based on the data and a physical quantity indicating the traveling state of the vehicle.

[0028] In this way, the desired scene can be detected with higher accuracy by taking into account the vehicle's driving conditions.

[0029] (12) The recording medium may further include a playback control means for controlling a playback device to play back the images taken by the vehicle at each of the above-mentioned time points recorded on the recording medium, and an alarm control means for controlling an alarm device to alarm an image representing the detected scene in association with the image played back in the playback device under the control of the playback control means.

[0030] In this way, it becomes easier to detect a scene similar to a specific image when playing back an image recorded on a recording medium.

[0031] (13) The recording medium may further include a generating means for storing images taken by the vehicle at each of the times from when a recording start command is given to when a recording stop command is given, the time period being considered as one period, and a second recording control means for controlling the recording device to record the image generated by the generating means in the one period.

[0032] In this way, the missing portion of the video can be restored.

[0033] (14) The generating means may generate an image corresponding to the missing portion by inputting at least one of a second image before or after a first portion corresponding to the missing portion in images taken at each of multiple time points in another vehicle, and inputting at least one image before or after the missing data into a trained second learning model that has been subjected to machine learning with the first portion as output.

[0034] In this way, a video of the missing portion can be generated relatively accurately.

[0035] (15) A program for implementing the functions of the above system on a computer may be provided.

[0036] In this way, by installing the program in the device, it is possible to detect specific image scenes that can be captured by the vehicle in the device.

[0037] (16) The trained model of the present invention uses specific images that can be captured by a vehicle as training data, takes images captured by the vehicle at each of multiple time points as input, and detects images that represent the scene of the specific images from the input images.

[0038] In this way, it is possible to provide a trained model for detecting specific image scenes that can be captured by a vehicle.

[0039] (17) A method for generating a learning model according to the present invention uses specific images that can be captured by a vehicle as training data, generates a learning model in which the input is an image captured by the vehicle at each of a number of time points, and the output is an image representing the scene of the specific image from the input images.

[0040] In this way, it is possible to generate a learning model that can detect a relatively large number of images of a specific scene and that can detect images of a specific scene with high accuracy.

[0041] (18) The specific images may be images captured by a plurality of image capturing devices.

[0042] In this way, it is possible to generate a learning model that can detect a relatively large number of images of a specific scene and that can detect images of a specific scene with high accuracy.

[0043] (19) The above-mentioned photographing device may be a drive recorder installed in a vehicle, and the above-mentioned specific image may represent a scene when a recording command is given to the drive recorder, or a scene when an impact is applied to the vehicle in which the drive recorder is installed.

[0044] In this way, it is possible to generate a learning model that detects images of the scene when a recording command is given to the drive recorder or when an impact is applied to the vehicle in which the drive recorder is installed.

[0045] (20) It is preferable to detect an image showing a scene in which an object is moving at a speed or acceleration greater than a certain value.

[0046] In this way, it is possible to generate a learning model that detects images containing objects moving at a speed or acceleration above a certain level.

[0047] (21) A learning model for detecting an image representing the scene may be generated based on the data and a physical quantity indicating the traveling state of the vehicle.

[0048] In this way, by taking into account the vehicle's driving conditions, it is possible to generate a learning model that can detect a desired scene with greater accuracy.

[0049] (22) A first image, which is an image of a predetermined period among images taken at each of a plurality of time points, and a second image, which is at least one image before or after the first image, are used as training data, and the first image is used as input and the second image is used as output for learning. A third image, which is taken at each of a plurality of time points including a missing portion during the predetermined period, is used as input, and a fourth image, which is an image of the missing portion, is estimated from the input third image.

[0050] In this way, a learning model for generating images of missing parts can be generated.

[0051] (23) The trained model generation device according to the present invention generates a trained model using any one of the trained model generation methods (17) to (22).

[0052] In this way, a learning model can be generated that can provide a trained model for detecting specific image scenes that can be captured by a vehicle.

[0053] The inventions shown in (1) to (22) above can be combined in any way. For example, at least a part of the configuration of at least one of the inventions from (2) to (23) may be added to all or a part of the configuration of the invention shown in (1). In particular, it is preferable to add at least a part of the configuration of at least one of the inventions from (2) to (23) to the invention shown in (1). In addition, any configuration may be extracted from the inventions shown in (1) to (23) and the extracted configurations may be combined.

[0054] The applicant of this application intends to obtain rights to the invention including these configurations. Furthermore, even if there is a statement such as "in the case of" or "when", this is not intended to describe the configuration as being limited to that case or time. These are merely examples of better configurations, and the applicant intends to obtain rights to configurations other than these cases or times. Furthermore, the descriptions in an order are not limited to this order. Configurations in which some parts have been deleted or the order changed are also disclosed, and the applicant intends to obtain rights to these configurations. Effect of the Invention

[0055] According to the present invention, a technique for detecting a specific image scene that can be captured by a vehicle can be provided.

[0056] The effects of the invention of this application are not limited to this, and effects achieved from the configuration disclosed in the specification and drawings are also disclosed, and the applicant intends to obtain rights to the configuration achieving said effects through divisional applications, amendments, etc. For example, in this specification, statements such as "can" and "is possible" are statements that clearly indicate the effects achieved, and there are also parts that show effects even without statements such as "can" and "is possible." Furthermore, there are effects that can be understood from the configuration even without such statements. [Brief description of the drawings]

[0057] [Figure 1] This is an example of a drive recorder. [Diagram 2] This shows a drive recorder being installed inside a car. [Diagram 3] FIG. 2 is a block diagram showing the electrical configuration of the drive recorder. [Figure 4] FIG. 2 is a block diagram showing an electrical configuration of the server. [Diagram 5] This is an example of a learning model. [Figure 6] 1 is an example of a target image. [Figure 7]1 is an example of a flowchart for generating a learning model. [Figure 8] 1 is an example of a learning model for generating multiple target images. [Figure 9] (A) to (C) are examples of target images. [Figure 10] FIG. 1 is a block diagram showing the electrical configuration of a generative adversarial network that generates a target image. [Figure 11] This is an example of a learning model. [Figure 12] 13 is a flowchart showing a processing procedure for recording an event. [Figure 13] This is an example of a learning model. [Figure 14] 13 is a flowchart showing a procedure of a degree of coincidence calculation process. [Figure 15] 1 is an example of a frame. [Figure 16] 1 is an example of a frame constituting a moving image. [Figure 17] This is an example of a learning model. [Figure 18] An example of a personal computer. [Figure 19] 2 is an example of a recording format. [Figure 20] 1 is an example of a playback window. [Figure 21] 13 is a flowchart showing a playback process procedure. [Figure 22] 1 is an example of a playback window. [Diagram 23] 1 is an example of a playback window. [Figure 24] (A) and (B) show the relationship between the frame number and the address position information. [Diagram 25] 13 is a flowchart showing a repair process procedure. [Figure 26] 2 is an example of a recording format. [Figure 27] This is an example of a learning model. [Figure 28] This shows how a learning model is trained. [Figure 29]13 is a flowchart showing a learning model generation process procedure. [Diagram 30] 13 is a flowchart showing a procedure for generating a missing portion of a moving image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0058] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the embodiment shown below is one embodiment of the present invention, and the contents of the present invention should not be interpreted as being limited based on the following description.

[0059] [How the idea for the present invention came about] In current drive recorders, event recording is automatically (passively) performed according to the detection results of sensors such as acceleration sensors, but the only way for a user to actively record an event is by using a switch, and if the sensors do not react and the user is in a situation where he or she cannot press the switch (for example, when the driver is not in the car while driving or parking), video recording cannot be performed. Therefore, the inventor has come up with a solution in which, using methods such as deep learning, accidents and dangerous situations in which the sensors do not react are learned from video data, and when the same or similar situations occur, event recording is automatically performed (first and second embodiments, etc.). In this way, this solution complements event recording with machine learning, and learns videos of situations in which the sensors do not react. In addition, the inventor has come up with the idea that, instead of recording events by shooting such situations as accidents, it is possible to learn scenery while driving and points that the user wants to take as teacher images in advance, and automatically determine similar situations and record them with the drive recorder. The inventors thought that if the drive recorder could be used as a recording device other than for recording images of accidents, etc., the uses of the drive recorder would increase (third embodiment, etc.).

[0060] [Outline of each embodiment] Each embodiment described below relates to a technology for detecting an image representing a specific image scene from images taken by a vehicle at each of a plurality of time points based on data defining the scene of the specific image that can be captured by the vehicle. In each embodiment, the specific image is an image defining a desired scene (also called a desired image). "Detecting an image representing the scene" includes generating a trigger for executing a predetermined process. The predetermined process is event recording in the first and second embodiments, and includes displaying information about the image representing the scene, and when the displayed information is selected, displaying an event-recorded video including the image representing the scene based on video data representing the corresponding video, fast-forwarding to a point where the image of the scene is displayed according to this selection during video playback, and jumping (cueing) to the point during image playback. Not limited to these, "detecting an image representing the scene" may include a process of associating identification information indicating that the image has been detected with the image. In each embodiment, a method for finding a frame representing a scene having a degree of match with a desired image scene equal to or greater than a threshold value among a plurality of frames constituting a video represented by video data will be described. A video is composed of images (still images in this case, also referred to as frames in the embodiment) at each of a plurality of time points arranged on a time axis. When a video is played based on video data, the images are played sequentially along the time axis. A specific image may be an image at one time point (i.e., a still image), or may be an image at two or more time points, or may be an image at two or more time points taken consecutively in time to represent frames that constitute a video. A desired image may be any image, such as an image of an accident such as a car accident (not limited to a car accident), a scene that may lead to a disaster such as a natural disaster (not limited to a natural disaster), a scene that led to an accident or disaster, or an image of a desired landscape. In addition, it may be an image of a scene where a recording command is given to a drive recorder by an operation button or the like as described later, or an image of a scene when an impact is applied to a vehicle equipped with a drive recorder.

[0061] [First Example] 1 to 11 show a first embodiment. In the first embodiment, a target image is generated using deep learning using a plurality of images that are the same as a desired image or an image that is similar to the desired image, or a plurality of images that are the same as the desired image and an image that is similar to the desired image, and the generated target image is used as the desired image. Generation of a trained model is also described.

[0062] FIG. 1 is a perspective view of a drive recorder 1 (one example of a system) seen obliquely from the rear. In the following description, the drive recorder 1 is a device that is installed in a vehicle after the vehicle is installed in the present embodiment. The vehicle is, for example, a private automobile or a commercial automobile (passenger car). However, the vehicle may also be a special automobile such as a bus, truck, or forklift, or a public transportation vehicle such as a train, monorail, or linear motor car.

[0063] An SD card slot 10 is formed on one side of the housing of the drive recorder 1. A display 11 is formed over almost the entire back surface of the housing of the drive recorder 1, and a plurality of operation buttons 12 are provided on both the left and right sides of the display 11. A joint rail 13 is provided on the top surface of the drive recorder 1. Although not shown in FIG. 1, a camera lens is provided on the front surface of the drive recorder. A DC jack for the drive recorder 1 is formed on a side not shown in FIG. 1, and a speaker and an HD (high definition) output terminal are formed on the bottom surface of the drive recorder 1.

[0064] The camera including the lens captures, for example, an image in front of the vehicle. Preferably, the camera may also capture images of the interior and exterior of the vehicle (the side of the vehicle, the rear, etc.). The DC jack is a jack for connecting to a DC power source via a power cable. The SD card insertion slot 10 is an insertion slot for inserting an SD card [an example of a memory card, which may be other memory cards such as an xD picture card, or may be a memory other than a memory card such as a USB (Universal Serial Bus) memory]. The speaker outputs sound and audio. The HD output terminal is a terminal for connecting to other information devices via a cable. The joint rail 13 is for attaching a joint for mounting the drive recorder 1 on the vehicle. The display 11 displays various images such as images captured by the camera of the drive recorder 1. The operation button 12 is for inputting various commands to the drive recorder 1 by the user's operation.

[0065] FIG. 2 shows a view of the vehicle in which the drive recorder 1 is mounted, viewed from inside the vehicle cabin toward the front.

[0066] A rearview mirror 4 is provided near the center of the upper part of the vehicle's windshield 3. A drive recorder 1 is fixed to the vehicle by a joint at a position adjacent to the rearview mirror 4 on the passenger side (on the left side as viewed from the front in FIG. 2).

[0067] A DC jack of the drive recorder 1 is connected to a cigarette lighter socket 5 via a power cable 6. When the accessory power of the vehicle is turned on, power is supplied from the cigarette lighter socket 5 to the drive recorder 1. Note that the drive recorder 1 is not limited to the form shown in Figs. 1 and 2, and may be, for example, one equipped with a celestial sphere camera or a semi-celestial sphere camera.

[0068] FIG. 3 is a block diagram showing the electrical configuration of the drive recorder 1. As shown in FIG.

[0069] The drive recorder 1 includes a controller 20. The controller 20 includes a CPU (Central Processing Unit) 20a that controls the overall operation of the drive recorder 1, a ROM (Read Only Memory) 20b that stores the operating program of the drive recorder 1, a RAM (Random Access Memory) 20c that temporarily stores data, and a timer 20d. The controller 20 has functions such as a GPS information processing program, an image processing program, and a communication processing program.

[0070] The drive recorder 1 also includes a GPS (Global Positioning System) receiver 16, a camera 17, an SD card reader / writer 14, and an acceleration sensor 18. The GPS receiver 16 detects the position of the drive recorder 1, and thus the position of the vehicle in which the drive recorder 1 is mounted can be known. The camera 17 captures images of the front of the vehicle in which the drive recorder 1 is mounted, as described above. The SD card reader / writer 14 reads data recorded on the inserted SD card 23 by inserting the SD card 23 into the SD card insertion slot 10, as described above, and writes the data to the SD card 23. The acceleration sensor 18 detects the up-down, left-right, and front-back accelerations given to the drive recorder 1, and thus the up-down, left-right, and front-back accelerations of the vehicle in which the drive recorder 1 is mounted can be known.

[0071] The GPS data output from the GPS receiver 16 indicating the position of the drive recorder 1, the video data indicating the video captured by the camera 17, and the signal indicating the acceleration detected by the acceleration sensor 18 are each input to the controller 20. The controller 20 executes a GPS information processing program to record the GPS data output from the GPS receiver 16 on an SD card 23 inserted in the SD card slot 10. The controller 20 also executes a video processing program to record the video data obtained from the camera 17 on the SD card 23 in association with the time.

[0072] Furthermore, the drive recorder 1 includes a speaker 15 for outputting the above-mentioned audio and the like, a display 11 for displaying images and the like, an operation button 12, a communication circuit 19, and an LTE module 21.

[0073] When audio data, etc. is output from the controller 20 to the speaker 15, audio, etc. is output from the speaker 15, and when video data, etc. is output from the controller 20 to the display 11, video, etc. is displayed on the display 11. Various commands from the operation buttons 12 are input to the controller 20.

[0074] The communication circuit 19 functions as a communication means for wirelessly communicating with an external device, such as a server, a personal computer, a smartphone, or a tablet terminal. The controller 20 has a function of transmitting data such as video to an external device via the communication circuit 19 by executing a communication processing program. The communication circuit 19 may be one that complies with a short-range wireless communication standard such as the WiFi standard or Bluetooth (registered trademark). The short-range wireless communication standard can be applied to communication between a work vehicle (such as a forklift) operating within a premises and an external device. The mobile communication system standard can be applied to communication between a vehicle moving within a wider area and an external device.

[0075] The LTE (Long Term Evolution) module 21 is a communication circuit that complies with the mobile phone communication standard. A communication circuit that complies with the standards of other mobile communication systems such as 4G may be used instead of the LTE module 21. The LTE module 21 can transmit video data captured by the camera 17 to an external server.

[0076] The operating program of the drive recorder 1 may be pre-recorded on the SD card 23, or may be received by the LTE module 21 via the Internet or the like and recorded on the SD card 23, or the SD card 23 may be inserted into a personal computer or the like and connected to the Internet via the personal computer so that the operating program can be downloaded from the personal computer and recorded on the SD card 23.

[0077] FIG. 4 is a block diagram showing the electrical configuration of the server 30 that communicates with the drive recorder 1.

[0078] The server 30 includes a control device 31 that controls the entire operation.

[0079] A communication device for communicating with the drive recorder 1 and other client computers, a memory 33 for storing data, and a hard disk drive 34 are connected to the control device 31 of the server 30. The hard disk drive 34 accesses a hard disk 35 for storing video data and the like, and reads the video data recorded on the hard disk 35 and records the video data and the like on the hard disk 35.

[0080] FIG. 5 is an example of a learning model 40.

[0081] 5 is a learning model 40 that uses deep learning to input multiple images and generate a target image. The learning model 40 includes an input layer 41, an intermediate layer (hidden layer) 42, and an output layer 43.

[0082] Each of the pixels P1 to PN that make up an arbitrary image (including not only the desired image and images similar to the desired image, but also other images) is input from the input layer 41 of the learning model 40, and learning is performed by deep learning through the intermediate layer 42 and the output layer 43 for a huge number of arbitrary images. After that, when an arbitrary image that contains many desired images and images similar to the desired image is input from the input layer 41 of the learning model 40, a neuron 43a that responds to the desired image and images similar to the desired image can be found. If an unknown image is input from the input layer 41 of the learning model 40 and the neuron 43a responds, it can be determined that the input unknown image is the desired image or an image similar to the desired image. In this way, the learning model 40 can recognize images.

[0083] Furthermore, by tracing back from the reacting neuron 43a to the input layer 41 side, it is possible to generate a target image that is most likely to be the desired image. The target image generated in this way becomes the desired image generated by deep learning (unsupervised learning).

[0084] For example, by inputting images that may lead to an accident, images that may lead to a disaster, images that have led to accidents, images that have led to disasters, other accidents, and a huge number of images unrelated to disasters into learning model 40 and learning through deep learning, it becomes possible to determine a neuron (the first neuron) that reacts when an image that may lead to an accident is input to learning model 40, a neuron (the second neuron) that reacts when an image that may lead to a disaster is input to the learning model, a neuron (the third neuron) that reacts when an image that led to an accident is input to the learning model, and a neuron (the fourth neuron) that reacts when an image that led to a disaster is input to the learning model.

[0085] By tracing back from the first reacting neuron to the input layer 41 side, a target image resembling an image that may lead to an accident can be generated, by tracing back from the second reacting neuron to the input layer 421, a target image resembling an image that may lead to a disaster can be generated, by tracing back from the third reacting neuron to the input layer 41 side, a target image resembling an image that has led to an accident can be generated, and by tracing back from the fourth reacting neuron to the input layer 41 side, a target image resembling an image that has led to a disaster can be generated. The generated target images can be used as the desired image.

[0086] FIG. 6 shows an example of a target image 50 generated as described above, which is an example of an image that may lead to an accident.

[0087] The target image 50 is an image generated based on deep learning and is different from an actually captured image. However, it is possible that the target image 50 may be almost identical to the actually captured image by chance.

[0088] As an example of this embodiment, frames of scenes that match the scene of the target image 50 at a threshold value or higher are found from among the frames that make up the moving image.

[0089] FIG. 7 is a flowchart showing the procedure of the target image generation process.

[0090] The target image generation processing procedure shown in FIG. 7 may be performed at any timing as long as a large number of images are obtained. In addition, it may be performed in the drive recorder 1 if image data representing a large number of images is recorded in the SD card 23 or the database 22, or it may be performed in the server 30 if image data representing a large number of images is recorded in the hard disk 35 of the server 30. When images are collected from drive recorders (multiple drive recorders) installed in multiple vehicles, it is preferable that the target image generation processing procedure is performed in the server 30. Furthermore, if the Internet is accessible, the target image generation processing may be performed in either the drive recorder 1 or the server 30 by reading a large number of images from the Internet, even if a large number of images are not recorded in the SD card 23, the database 22, or the hard disk 35. Furthermore, the target image generation processing may be performed in a client computer (personal computer).

[0091] A large number of arbitrary images are input to the learning model 40 (step 61), and deep learning is performed in the learning model 40 (step 62). A target image is obtained as described above (step 63).

[0092] It goes without saying that the program of the learning model 40 is installed in the controller 20 of the drive recorder 1 or the control device 31 of the server 30.

[0093] FIG. 8 illustrates a method for generating multiple target images.

[0094] The first learning model 40A, the second learning model 40B and the third learning model 40C each include an input layer, an intermediate layer and an output layer, similar to the learning model shown in FIG.

[0095] The image input to the first learning model 40A is the first image, the image input to the second learning model 40B is the second image, and the image input to the third learning model 40C is the third image. As described with reference to FIG. 5, when a huge number of images are input to the first learning model 40A and the model is trained by deep learning, a neuron that responds to a specific image (e.g., an image that is likely to lead to an accident) is found, and an image that represents the characteristics of the specific image is obtained by tracing the neuron to the input layer. For example, a first target image that represents the characteristics that are likely to lead to an accident is obtained. Similarly, when a huge number of images are input to the second learning model 40B and the model is trained by deep learning, a neuron that responds to a specific image (e.g., an image that is likely to lead to a disaster) is found, and an image that represents the characteristics of the specific image is obtained by tracing the neuron to the input layer. For example, a second target image that represents the characteristics that are likely to lead to a disaster is obtained. When a huge number of images are input to the third learning model 40C and it is trained by deep learning, neurons that respond to a specific image (for example, the image that led to the accident) are found, and by tracing the neurons to the input layer, an image that represents the characteristics of the specific image is obtained. For example, a third target image that represents the characteristics of the image that led to the accident is obtained. In this way, multiple types of target images are obtained.

[0096] In FIG. 8, multiple types of target images are obtained using multiple learning models 40A-40C, but even in a single learning model 40 as shown in FIG. 5, it is possible to find the first target image, the second target image, the third target image, etc. by tracing the input layer 41 and searching for neurons that respond to the first target image, neurons that respond to the second target image, neurons that respond to the third target image, etc.

[0097] In the example shown in Fig. 8, it is preferable that the first image, the second image, and the third image are images taken consecutively in time series. As a result, the first target image, the second target image, and the third target image obtained also represent images that are consecutive in time series. For example, if the first target image, the second target image, and the third target image represent the characteristics of images that are likely to lead to an accident, when frames close to the first target image, the second target image, and the third target image appear in sequence in the video, it can be determined that there is a higher possibility of an accident in such a situation, and the accuracy is higher than when determining that there is a higher possibility of an accident based on only one target image.

[0098] FIGS. 9(A), (B) and (C) are examples of the first target image, the second target image and the third target image generated as described above.

[0099] It can be seen that the child depicted as the subject approaches the car and the possibility of an accident increases as the image progresses from the first target image 51 to the second target image 52 to the third target image 53. If the first target image 51, the second target image 52 and the third target image 53 are set as desired images and a scene similar to the first target image 51, the second target image 52 and the third target image 53 is found in the video, it can be determined that there is a high probability that an accident will occur in that scene.

[0100] In the above embodiment, three target images, the first target image 51, the second target image 52 and the third target image 53, are set as the desired images, but two or four or more target images may be generated and set as the desired images.

[0101] FIG. 10 shows another method for generating a target image, and is a block diagram showing the electrical configuration of a generative adversarial network.

[0102] Feature data representing the features of a desired image is input to a generator 71. In the generator 71, pseudo image data representing a pseudo image of the desired image is generated from the input feature data. The generated pseudo image data is input from the generator 71 to a classifier 72. Image data representing the desired image is also input to the classifier 72. The classifier 72 classifies whether the desired image represented by the image data representing the desired image and the pseudo image represented by the pseudo image data are the same or similar images. Data representing the classification result is input to both the generator 71 and the weighting modifier 73. The generator 71 is adjusted based on the data representing the classification result, and the weighting of the features of the classifier 72 is changed by the weighting modifier 73.

[0103] By repeating the generation of pseudo image data in the generator 71, the classification in the classifier 72, the adjustment based on the classification result in the generator 71, and the change of the weight of the classifier 72 in the weight changer 73, the classifier 72 determines that the pseudo image represented by the pseudo image data generated by the generator 71 is identical to or similar to the desired image. The pseudo image determined in this way is set as the target image. Multiple images of the same type may be set as desired images, and a pseudo image obtained by repeating the above-mentioned process may be set as the target image. The target image obtained in this way becomes the desired image to be found from the frames constituting the video.

[0104] In the above-mentioned embodiment, a target image is generated using an arbitrary image and the generated target image is used as the desired image, or a pseudo image is generated using a desired image and the generated pseudo image is used as the desired image, but when an image that is likely to lead to an accident or disaster, an image that led to an accident or disaster, or an image of a scene that the user wants to find is already known, the target image, pseudo image, etc. may be used as the desired image without generating the target image, pseudo image, etc. using deep learning, etc. For example, when there is a video of an event record (for example, a video recorded when an impact is applied to a vehicle when a drive recorder mounted on a vehicle records a video), one or more frames constituting the video of the event record may be used as the desired image.

[0105] Figure 11 shows a method for training a learning model to generate a trained model.

[0106] 13 shows another example of generating a target image, and an example of a learning model.

[0107] The learning model 80 includes an input layer 81 , an intermediate layer (hidden layer) 82 , and an output layer 83 .

[0108] A desired image is used as training data, and data representing pixels P1 to PN constituting the desired image is input to input layer 81 in the order in which the image is composed. The features, weighting, and the like of input layer 81, intermediate layer 82, and output layer 83 are adjusted as necessary so that data identical to pixels P1 to PN input to input layer 81 via intermediate layer 82 is output from output layer 83 in the same order as the order in which the desired image is composed. In other words, machine learning is performed so that the output of learning model 80 is approximately the same as the input, and the features, weighting, and the like of learning model 80 are adjusted as necessary.

[0109] It is also possible to reduce the number of data representing pixels input to the input layer 81, or reduce the number of data representing pixels output from the output layer 83. The features, weights, and the like of the learning model 80 may be adjusted as necessary so that data representing pixels at the same position in the image is approximately the same between the input and the output.

[0110] When data representing any image (e.g., each frame constituting a video) is input to a learning model that has been trained in this way, the closer the image matches the desired image, the closer the data for each pixel output from output layer 83 will be to the data for pixels P1 to PN that constitute the desired image.

[0111] According to the first embodiment, a trained model can be obtained that can generate a desired image and can be used to detect a frame of a scene that is similar to the scene of the desired image from among multiple frames that make up a video.

[0112] [Second Example] 12 to 16 show the second embodiment. In the second embodiment, a frame having a high degree of matching with a desired image is found from among a plurality of frames constituting video data by using a trained learning model 80 generated with reference to FIG. 11. In the second embodiment, video data captured by a shooting device is recorded, the recorded video data is read, and a frame having a high degree of matching with a desired image is found from among a plurality of frames constituting the read video data. In the following, the acceleration detected by the acceleration sensor 18 is used to detect the desired image, and a process of determining whether or not there is an object moving at a certain speed or acceleration or higher in the video is used in combination, but at least it is sufficient that the desired image is detected by applying an image captured by the drive recorder 1 to the trained model generated by the method described in the first embodiment.

[0113] FIG. 12 is a flowchart showing the procedure of the event recording process performed by the drive recorder 1 shown in FIGS. 1 and 2 during continuous recording.

[0114] The program for carrying out the processing procedure shown in FIG. 12 may be installed in the controller 20 of the drive recorder, or may be stored in an SD card (an example of a recording medium storing a program) 23, or may be transmitted via a network, received by the communication circuit 19, and installed in the drive recorder 1.

[0115] When the drive recorder 1 is mounted on a vehicle, when the engine of the vehicle is started, the camera 17 of the drive recorder 1 starts shooting, and the SD card reader / writer 14 starts recording (continuous recording) the video data obtained by shooting to the SD card 23. Continuous recording starts when the engine of the vehicle is started, and ends when the engine of the vehicle is turned off. Even when the engine of the vehicle is not running, continuous recording starts when a recording start command is given to the controller 20 from the operation button 12, and ends when a recording end command (an example of a recording stop command) is given to the controller 20 from the operation button 12. The continuous recording from start to end is one video of the continuous recording. In this embodiment, the video data obtained by shooting with the camera 17 is also given to the RAM 20c and recorded therein. The video data recorded in the RAM 20c is recorded periodically for a certain period of time, and is repeatedly overwritten after the certain period has elapsed.

[0116] In this embodiment, when an event is detected by the acceleration sensor 18 or when a recording command is given by the operation button 12 (YES in step 91), the event is recorded in the event recording area of ​​the SD card 23 (step 98).

[0117] If no event is detected by the acceleration sensor 18 or a recording command (NO in step 91), the controller 20 (an example of a first recording control means) reads video data recorded in the RAM 20c (an example of a recording medium), and determines whether or not there is an object moving at a speed or acceleration equal to or greater than a certain level in the video represented by the read video data (step 92). This object may or may not be determined in advance. It may be the relative speed or acceleration between the vehicle in which the drive recorder 1 is mounted and the object, or it may be the absolute speed or acceleration of the object.

[0118] If the video represented by the read video data contains an object moving at a speed or acceleration equal to or greater than a certain level (YES in step 92), the video data read from RAM 20c is input frame by frame to the trained learning model 80 (step 93). A learning process is performed in the learning model 80, and image data representing feature images is obtained frame by frame from the output layer 83 of the learning model 80 (step 94). Although the learning model 80 is described as hardware, in reality it is implemented by software in the controller 20.

[0119] The degree of match between the obtained feature image and the desired image is calculated by the controller 20 (step 95).

[0120] FIG. 13 shows how video data is input frame by frame to a trained learning model 80.

[0121] Pixels P11 to P1N constituting one frame are input to an input layer 81 according to the arrangement of one frame, and data of pixels P21 to P2N is output via an intermediate layer 82 and an output layer 83 as image data representing a feature image.

[0122] The data of these pixels P21 to P2N that represent the characteristic image are compared with the data of each pixel that constitutes the desired image, and the controller 20 performs a process of calculating the degree of coincidence between the characteristic image and the desired image.

[0123] FIG. 14 is a flowchart showing the procedure of the matching degree calculation process.

[0124] First, the degree of agreement between the desired image and the feature image is calculated as the first degree of agreement (step 131). Next, the degree of agreement between the vehicle information (an example of driving information) when the desired image is obtained from the drive recorder 1 mounted on the vehicle (an example of the second vehicle) and the frame input to the learning model 80 when the feature image is obtained is obtained from the drive recorder 1 mounted on the vehicle (an example of the first vehicle) is calculated as the second degree of agreement (step 132). The driving information includes physical quantities indicating the state of the vehicle, such as the vehicle speed, the attitude of the vehicle body, and the acceleration of the vehicle. Such physical quantities are physical quantities that change as the vehicle travels. In addition to this, the driving information may include information that changes as the vehicle travels, such as the location (position information) of the vehicle. Furthermore, an overall degree of agreement is calculated from the first degree of agreement and the second degree of agreement (step 133).

[0125] 14 uses vehicle information when the desired image was obtained, so if the desired image is a virtual target image generated using a trained learning model, the first degree of match will be used to determine the overall degree of match without calculating the second degree of match. However, even if the desired image is a virtual target image, if vehicle information about the image used to obtain the target image is known, the second degree of match may be calculated by using the vehicle information to calculate average vehicle information.

[0126] Returning to FIG. 12, if the calculated degree of coincidence (total degree of coincidence) is equal to or greater than the threshold (YES in step 96), a frame that represents a scene having a degree of coincidence with the desired image scene equal to or greater than the threshold is detected among the frames constituting the moving image represented by the moving image data recorded in the RAM 20c, and an event recording command is generated from the controller 20 (an example of a warning control means) and a warning is issued (step 97). The warning may be a voice warning output from the speaker 15 (an example of a warning device) of the drive recorder 1, a character or image warning displayed on the display 11 (an example of a warning device), or a warning by emitting or blinking an LED (light emitting diode). The warning may also be issued using a speaker, a display screen, or the like provided in the vehicle in which the drive recorder 1 is mounted. The warning informs the driver of the vehicle that the driving situation is close to the scene of the desired image (the driving situation is being reached). For example, if the desired image is an image that is likely to lead to an accident, the driver can drive more carefully. Furthermore, when the event recording command is generated, the event is recorded in the event recording area of ​​the SD card 23 (step 98).

[0127] In the above-described embodiment, when the object is moving at a speed or acceleration above a certain level (YES in step 92), the photographed data is input into the trained learning model; however, the photographed data may be input into the trained learning model regardless of whether the object is moving at a speed or acceleration above a certain level, or the photographed data may be input into the trained learning model without determining whether the object is moving at a speed or acceleration above a certain level.

[0128] Fig. 15 is an example of one of a plurality of frames constituting a moving image represented by the moving image data read from RAM 20c. Fig. 16 shows a plurality of frames constituting a moving image represented by the moving image data read from RAM 20c.

[0129] As shown in Fig. 16, a moving image represented by the moving image data is composed of a plurality of frames FR1 to FRe, and it is assumed that the degree of match (overall degree of match) obtained when a frame FRn shown in Fig. 15 and Fig. 16 is input to the learning model 80 is equal to or greater than a threshold value. Then, data representing frames representing a moving image shot for a few seconds to a few tens of seconds (or other time) before the frame FRn and frames representing a moving image shot for a few seconds to a few tens of seconds (or other time) after the frame FRn are recorded in the event recording area of ​​the SD card 23 by the SD card reader / writer 14.

[0130] The warning in the above embodiment may be changed depending on the type or danger of the scene. For example, in a high-risk scene, the warning may be given by both voice and display, or the volume may be increased so that the driver can recognize the high-risk scene. In a low-risk scene, such as a scene similar to a favorite place, the driver may be notified by a relatively quiet voice.

[0131] FIG. 17 shows a modification of the learning model 80 shown in FIG.

[0132] The learning model 105 shown in FIG. 17 includes an input layer 106, an intermediate layer 107, and an output layer 108, with the output layer 108 being a logistic regression layer.

[0133] A plurality of frames constituting a moving image are input to the learning model 105 one by one. For example, data representing pixels P31 to P3N constituting one frame is input to the input layer 106 of the learning model 105 according to the pixel arrangement. In such a case, when a frame has a high degree of match with a desired image, identification data is output from the output layer 83, which is a logistic regression layer. When the identification data is output, it is determined that the degree of match between the desired image and the frame input to the learning model 105 is equal to or greater than a threshold, and an event recording command is generated from the controller 20. The learning model 105 is also a trained learning model generated by inputting image data representing a large number of desired images from the input layer 106 and training the model so that identification data is output from the output layer 108 when image data representing the desired image is input from the input layer 106.

[0134] In addition, when the identification data is output, an event recording command may be generated if the degree of match (the second degree of match described above) between the vehicle information at the time when the frame input to the learning model 105 was obtained and the vehicle information corresponding to the desired image is equal to or greater than a threshold value.

[0135] Event recording can be performed based on the captured video even without an event recording command from the acceleration sensor 18 or an event recording command from the operation button 12. Furthermore, when calculating the overall degree of coincidence using vehicle information, not only the captured video but also the vehicle information is used, so that more appropriate event recording can be performed according to the vehicle situation.

[0136] In the above-described embodiment, when a frame similar to a desired image is detected from a moving image, an event recording is performed and the moving image data is recorded on the SD card 23. However, it is also possible to record identification information for identifying the frame similar to the desired image on the SD card 23 or other recording medium without recording the moving image data.

[0137] In the above-described embodiment, the overall degree of coincidence is used to determine whether or not an event has been recorded, but the second degree of coincidence may not be used and instead the first degree of coincidence may be determined based on whether or not the first degree of coincidence is equal to or greater than a threshold value. However, by combining the physical quantity indicating the vehicle's running state, it is possible to expect the effect of detecting the frame of the desired scene with higher accuracy.

[0138] [Third Example] This embodiment proposes an automatic playback support function in the viewer software for recorded data. In the case of video data in drive recorders for management purposes, the data is often long and generally problem-free. If it is an event, the viewer software will display the occurrence point, but it can be a pain for the administrator to discover and check the attention points that are not events. Therefore, the inventor came up with a solution that makes it easier for the administrator to check by having the system learn the desired video situation using deep learning, etc., and by identifying and detecting as attention points separate from events when there is video that is the same or similar to the desired situation in the recorded data, and displaying it in the viewer software, etc., or by returning to normal speed at the event occurrence point or attention point during fast-forward playback.

[0139] Additionally, this embodiment proposes a function for restoring data such as video recorded on a recording medium. For example, when a recording format to be described later is adopted, the data itself is recorded even if the video is in an abnormal state (even if there is a missing part) because the data is written sequentially to the storage medium instead of the FAT method used in Windows (registered trademark) and the like. However, when a special abnormality occurs, a certain sector of the video data is erased with 0 or other information is written, so that the video cannot be played back with the viewer software. In this case, the contents of the recording medium may be checked sector by sector and restored manually, but the inventor has come up with a solution that not only learns the status of the recording medium using a method such as deep learning and restores the erased data from the data before and after the missing part, but also restores the above recording format to a normal state so that it can be confirmed with the viewer software and the like. The details of this embodiment will be described below.

[0140] 18 to 29 show the third embodiment, which is related to the playback of video recorded by a recording device such as the drive recorder 1. In the second embodiment, when a frame highly matching a desired image is found while video data is being recorded in a recording device such as the drive recorder 1, event recording is performed. In the third embodiment, a frame highly matching a desired image is found while playing back video data recorded continuously on an SD card 23 (or other recording medium), and event recording is performed. In the third embodiment, the continuously recorded video data is played back using a personal computer, but if a recording device such as the drive recorder 1 has a playback function, playback can also be performed in such a recording device, and the following processing can also be performed.

[0141] FIG. 18 is a block diagram showing the electrical configuration of a personal computer 110 that plays back moving images.

[0142] The overall operation of the personal computer 110 is governed by a controller 115 .

[0143] The personal computer 110 is provided with a display device 111, and images and the like are displayed on the display screen of the display device 111 under the control of a display control device 112. A communication device 113 for communicating with the Internet and the like, and a memory 114 are connected to the control device 115. The personal computer 110 is provided with an input device 116 such as a keyboard and a mouse, and commands given from the input device 116 are input to the control device 115.

[0144] In addition, the control device 115 is also connected to a hard disk drive 117 which reads data recorded on a hard disk 118 and records the data on the hard disk 118, and an SD card reader / writer 119 which reads data recorded on an SD card and records the data on an SD card 23.

[0145] FIG. 19 shows the recording format of the SD card 23.

[0146] In the recording area of ​​the SD card 23, a file system area 121, a constant recording area 122, and an event recording area 125 are formed.

[0147] In the file system area 121, dedicated software for reproducing the data recorded in the continuous recording area 122 and the event recording area 125 is recorded.

[0148] A management area 123 and a recording area 124 are formed in the constant recording area 122. Various setting information is recorded in the management area 123 by dedicated software recorded in the file system area 121. Image data representing frames constituting a moving image is recorded in the recording area 124 in the order of frame numbers. A header recording area 131, a frame image data recording area 132 and a footer recording area 133 are formed for each frame in the recording area 124. Additional information such as the frame number, address position and shooting time is recorded in the header, and additional information other than the additional information recorded in the header is recorded in the footer.

[0149] Frames 1 to E recorded in recording area 124 of continuous recording area 122 represent one period of continuous recording of a moving image.

[0150] Like the constant recording area 122, the event recording area 125 is formed with a management area 126 and a recording area 127. Various setting information is also recorded in the management area 126 by dedicated software recorded in the file system area 121. Image data representing frames constituting a moving image is recorded in the recording area 127 in the order of frame numbers. The recording area 127 is also formed with a header recording area 131, a frame image data recording area 132, and a footer recording area 133 for each frame. Additional information such as the frame number, address position, and shooting time is recorded in the header, and additional information other than the additional information recorded in the header is recorded in the footer.

[0151] Frames 1 to E recorded in recording area 127 of event recording area 125 represent one moving image of the event recording.

[0152] FIG. 20 is an example of a playback window 150.

[0153] When an SD card 23 is inserted into the personal computer 110 and the icon of the SD card 23 that appears on the display screen of the display device 111 is clicked, an icon of the dedicated software appears. When the icon of the dedicated software is clicked, the dedicated software is started and a playback window 150 is displayed on the display screen of the display device 111.

[0154] The playback window 150 is formed with a video display area 151 for displaying the selected video, an event record list display area 152, a map image display area 153, a vehicle information etc. display area 160, a continuous recording list display area 161, and an acceleration display area 163.

[0155] The video display area 151 is an area for displaying video (continuously recorded video, event recorded video) recorded by the drive recorder 1 or the like. The event record list display area 152 is an area for displaying an event record list in the form of recording time (or in a form other than recording time). The event record list displayed in the event record list display area 152 displays all event records recorded on the SD card 23. When a continuous recorded video is selected, the event record list display area 152 may display an event record video that was recorded during the shooting of the selected continuous recorded video.

[0156] The map image display area 153 is an area that displays an image of a map in the vicinity of the shooting location of the image displayed in the image display area 151. The map image is displayed in the map image display area 153 by transmitting data representing the position of the vehicle from the personal computer 110 to a map server existing on the Internet, and transmitting data representing a map in the vicinity of the position of the vehicle from the map server to the personal computer 110. The vehicle information display area 160 is an area that displays information such as the speed of the vehicle when the image was recorded. The vehicle information display area 160 includes a vehicle status display area 158 that displays the status of the vehicle to which the drive recorder 1 that recorded the image displayed in the image display area 151 is attached, an operation button 157 that gives operation commands such as playing and stopping the image, an interior highlight button 154 that gives an instruction for processing to brighten the image inside the vehicle, a normal display button 155 that gives an instruction for processing to stop the brightening of the image inside the vehicle and perform normal display, a time display area 156 that displays the recording time of the image displayed in the image display area 151, and an acceleration graph display area 159 that displays the acceleration of the vehicle in a graph.

[0157] The continuous recording list display area 161 is an area that displays a list of continuous recording videos recorded on the SD card 23. The acceleration display area 162 displays an image of a car. In front of the image of the car, the letters "+X" are displayed with an arrow pointing forward, next to the image of the car, the letters "+Y" are displayed with an arrow pointing horizontally, and above the image of the car, the letters "+Z" are displayed with an arrow pointing upward. The acceleration of the car in the forward or backward direction is displayed in front of the image of the car, the acceleration of the car in the lateral direction is displayed horizontally of the image of the car, and the acceleration of the car in the vertical direction is displayed above the image of the car. Since the acceleration of the car is displayed together with the exterior of the car, it is relatively easy to see in which direction the acceleration is acting on the car.

[0158] FIG. 21 is a flow chart showing the playback process procedure, and FIGS. 22 and 23 show an example of the playback window 150. As shown in FIG.

[0159] When a desired event recorded video or continuous recorded video is selected from the event recording list or continuous recording list displayed in the event recording list display area 152 or continuous recording list display area 161 of the playback window 150, video data representing the selected video is read from the continuous recording area 122 of the SD card 23 by the SD card reader / writer 119 (step 141). In this embodiment, it is assumed that a continuous recorded video has been selected.

[0160] When a continuous recording moving image is selected, a representative image of the selected continuous recording moving image is displayed in the video display area 151 of the playback window 150, as shown in Fig. 22 (step 142). When the playback button of the operation buttons 157 is pressed (YES in step 143), the read video data is played back by the control device 115 (a playback device, an example of a playback control means) (step 144). The selected continuous recording moving image is now displayed in the video display area 151.

[0161] When the stop button among the operation buttons 157 is pressed during playback of the continuously recorded moving image (YES in step 145), the control device 115 stops the moving image being played back (step 146).

[0162] When a desired event record moving image is selected from the event record list, the moving image data representing the selected event record moving image is also read from the event recording area 125 of the SD card 23 by the SD card reader / writer 119 (step 141).

[0163] When an event record video is selected, a representative image of the selected event record video is displayed in the video display area 151 of the playback window 150, as shown in Fig. 23 (step 142). The representative image of the event record video may be an image captured at the time when an event recording command was issued, or may be the first image of the event record video.

[0164] When the play button among operation buttons 157 is pressed (YES in step 143), the control device 115 performs a playback process of the read video data (step 144). The selected event record video is displayed in the video display area 151. When the stop button among operation buttons 157 is pressed during playback of the event record video (YES in step 145), the control device 115 stops the video being played (step 146).

[0165] 12 may be performed during the playback process of the continuously recorded video, and when a desired image is detected, the control device 115 may record the event in the event recording area 125. When the event is recorded, the desired image is additionally displayed in the event record list display area 152. Preferably, information indicating that the image is related to the desired image may be recorded in the header of a frame constituting the continuously recorded video in which the desired image is detected during playback, and the control device 115 or the display control device 112 (an example of a notification control means) may control the display device 111 so that, when the video is played, the time of the found event record and the like are displayed (an example of a notification) in the playback window 150 of the display device 111 (an example of a notification device).

[0166] 24(A) and 24(B) are examples of tables showing the relationship between frame numbers of frames constituting one video and address position information.

[0167] As described above, the frame number and address position information are recorded in the header recording area 131 regardless of whether it is a continuous recording video or an event recording video, so there is no need to create the tables shown in Figures 24(A) and 24(B). However, tables such as those shown in Figures 24(A) and 24(B) may be generated and recorded in the management area 123 of the continuous recording area 122 or the management area 126 of the event recording area 125.

[0168] As described above, frame numbers and address position information (information indicating the address of the beginning of a frame) corresponding to frames constituting a continuous recording moving image or an event recording moving image are recorded in the header recording area 131. Moving image data representing a continuous recording moving image or an event recording moving image according to this embodiment is continuously recorded in the order of frame numbers in the recording area 124 of the continuous recording area 122 or the recording area 127 of the event recording area 125. For example, as shown in Fig. 24(A), one moving image composed of frames from frame 1 to frame E is recorded frame by frame in the recording area 124 of the continuous recording area 122 or the recording area 127 of the event recording area 125.

[0169] In such cases, at least one of the frame number and address location information may not be recorded correctly, resulting in an abnormal state. For example, the frame numbers are not consecutive even though they should have been recorded consecutively, the address location information indicates an address location that is significantly larger than the address location information immediately before, or the address location information indicates an address location that is earlier than the address location information immediately before, etc. In such cases, even if the video data itself is normal, that part of the video may not be playable.

[0170] For example, in Figure 24(A), the frame numbers and address position information are all recorded normally, but in Figure 24(B), as shown by the hatching, the frame numbers and address position information for frames 11 to 20 have abnormal values, and the video represented by these frames 11 to 20 often cannot be played back normally.

[0171] In this embodiment, if at least one of the frame number and the address position information is not recorded correctly, they are rewritten to have the correct values. Also, if the video data is also not recorded correctly, for example, if there is a missing piece of video data, the missing video data is generated and written to the frame that was not recorded correctly.

[0172] FIG. 25 is a flowchart showing the procedure of the restoration process, and FIG. 26 shows the recording format of the SD card 23.

[0173] The repair process may be started when the SD card 23 is inserted into the personal computer 110 and the playback software is started, or when a specific video is specified for video playback, the repair process for that video may be started, or a repair mode button or the like may be provided in the playback window 150 and the repair process may be started when the repair mode button is pressed.

[0174] When the restoration process of the desired moving image is started, the control device 115 reads the frame numbers recorded in the header recording area 131 of the frames constituting the moving image (step 171). If the read frame numbers are not consecutive, the control device 115 determines that the non-consecutive frame numbers have not been recorded normally (YES in step 172). As shown in formula 1, the address position information of a frame plus the amount of moving image data of that frame is the address position information of the frame next to that frame, so if this relationship is broken, the control device 115 determines that the address position information has not been recorded normally (YES in step 172).

[0175] Address location information of a frame + video data amount of that frame = address location information of the next frame ··· Equation 1

[0176] If at least one of the frame number and the address position information has not been recorded normally (NO in step 172), the control device 115 repairs the improperly recorded frame number or address position information (or the frame number and the address position information) (step 173). As described above, if the frame number has not been recorded normally, the control device 115 rewrites the frame numbers so that they are consecutive in the order of the frames that make up the moving image. If the address position information has not been recorded normally, the control device 115 rewrites the address position information so that it has a normal value to satisfy the relationship shown in formula 1.

[0177] Next, the control device 115 judges whether the video data recorded in the recording area 124 or 127 is recorded normally (step 174). If the scenes represented by the frames are too different, for example, if a car is shown in one frame, but the next frame is completely black or completely white, making it impossible to recognize the object, the frame is judged to have been recorded normally. However, even if the scene suddenly changes in a frame, if it is judged to be a frame that constitutes a video of the same shooting scene based on shooting position information, etc., it is judged to have been recorded normally. For example, in the case of an accident, the scene suddenly changes, and this is to prevent such a case from being judged to have been recorded normally. A user may visually check and judge a frame that is not recorded normally.

[0178] If the video data is not normally recorded (NO in step 174), the abnormally recorded video data is generated and the missing portion of the video (missing data) is repaired (step 175). A method for repairing the missing portion of the video will be described later. With reference to FIG. 26, if the video data recorded in frames 10 to 20 of the video data recorded in the recording area 124 of the continuous recording area 122 is not normally recorded as shown by hatching, the video data recorded in frames 10 to 20 is repaired. When the missing portion of the video is repaired, it is preferable to record information indicating that the repaired frame is not a frame obtained by shooting but a repaired frame as additional information in the header recording area 131 or the footer recording area 131. This is to make it possible to distinguish between a real scene actually obtained by shooting and a pseudo scene generated later.

[0179] When a missing portion of a moving image is restored, the amount of data of the frame may change, and the address position information of the restored frame is likely to change as well. For this reason, the address position information is corrected according to Equation 1 (step 176).

[0180] In this embodiment, a learning model is used to restore the video data, but it is also possible to generate video data by image interpolation using frames before and after the frame to be restored without using a learning model.

[0181] Fig. 27 and Fig. 28 show a method for generating a learning model. Fig. 27 shows an example of a learning model, and Fig. 28 shows how the learning model is trained. Fig. 29 is a flowchart showing the procedure of a learning model generation process.

[0182] In FIG. 27 and FIG. 28, ten frames from frames 21 to 30 (an example of the second video data, teacher data, and first portion) are generated by machine learning using frames 1 to 20 before frame 21 and frames 31 to 50 after frame 30 (an example of the first video data, teacher data, and second image). Frames 1 to 50 are frames actually captured in another vehicle. For example, frames actually captured in another vehicle are frames captured by a drive recorder mounted on another vehicle that traveled in the same place or another place where the same or similar scene (e.g., scenery) can be captured. The machine learning is a supervised learning in which frames 1 to 20 and frames 31 to 50 are input and frames 21 to 30 are output. Frames other than frames 21 to 30 can be generated in the same manner.

[0183] Learning model 180 includes an input layer 181, an intermediate layer 182, and an output layer 183. Frames 1 to 20, and frames 31 to 50 are input to each neuron in the input layer 181 in the order in which they were shot in the order of frame numbers. As shown in Fig. 28, data for pixel E1, data for pixel E2, and so on are input to input layer 181 of learning model 180 in the order of the pixel arrangement for each pixel in each frame. In this way, video data (frames 1 to 20, frames 31 to 50) excluding a portion of the video data (frames 21 to 30) is input to input layer 181 (step 191).

[0184] Data input from input layer 181 is output from output layer 183 via intermediate layer 182. Learning model 180 is trained so that data corresponding to frames 21 to 29 are output from each neuron in output layer 183 in accordance with the pixel arrangement. As shown in FIG. 28, frames 1 to 20 and frames 31 to 50 are input in the order in which they were shot in frame number order, and when frames 21 to 20 are output from output layer 183, learning of the learning model is completed. In this manner, learning is performed so that the output is a fixed video portion (frames 21 to 30). By similarly performing multiple learning processes for other video data (step 192), it becomes possible to generate a learning model for a video of a defect video portion by using video frames before and after the defect video portion.

[0185] 27 to 29, the input has 40 frames and the output has 10 frames, but the number of input frames and the number of output frames can be changed without being limited to these numbers of frames. In addition, if the number of input frames and the number of output frames are both kept constant and the number of frames of the defect video to be generated is smaller than the number of output frames, frames can be thinned out so that the video data representing the output frames matches the number of frames of the defect video to be generated, and if the number of frames of the defect video to be generated is greater than the number of output frames, the number of frames can be adjusted by interpolation or the like using the video data representing the output frames.

[0186] Preferably, a plurality of learning models 180 suitable for various scenes are generated, and the defect video portion is generated using the learning model corresponding to the scene. For example, learning models 180 corresponding to the scene before or after the defect video portion are prepared, such as a learning model 180 used to generate a defect video when a car ahead has an accident, and a learning model 180 used to generate a defect video when a bird, a bicycle, a pedestrian, etc. suddenly appears ahead, and the defect video portion is generated using the learning model 180 corresponding to the scene.

[0187] Furthermore, when generating a defect video, if the video is from a drive recorder mounted on a vehicle, a learning model 180 may be used that was generated using a video that has a high degree of match with the vehicle information of the vehicle mounted on the drive recorder that captured the defect video.

[0188] FIG. 30 is a flowchart showing the procedure for generating a missing portion of a moving image.

[0189] When video data of a defective portion is found among the video data recorded on the SD card 23, the video data before and after the defective portion are input frame by frame to the trained learning model 180 in the order in which they were shot (step 201).

[0190] The input video data is trained in a trained learning model 180 (an example of a second learning model) (step 202), video data (an example of a third image) including a missing portion is input to the learning model 180 by the control device 115 (an example of a video data generating means), and video data (an example of a fourth image) representing the video data of the missing portion is output from the learning model 180 in frame order (step 203). The video data of the missing portion thus obtained is recorded in the missing portion of the recording area 124 or 127 by the control device 115 (an example of a second recording control means) (step 204).

[0191] Even if there is a recording error such as a frame number recorded on the memory card 23, it can be repaired, and even if there is a missing part in the video, it can be repaired.

[0192] In the above description, in response to "detecting an image representing the above scene," information relating to the image representing the scene is displayed in the event record list display area 152, and when the displayed information is selected, an event recorded video including the image representing the scene is displayed based on video data representing the corresponding video. However, this is not limited to this, and in response to this selection, during playback of a video recorded on the SD card 23, it may be possible to fast forward to the point at which the image of the scene is to be displayed, or to jump (cue) to that point during playback of the image.

[0193] In the above-described embodiments, various processes are performed based on video data, but the processes may be performed based on data other than video data. The data other than video data may be composed of a set of still image data representing still images captured in chronological order.

[0194] In the above-mentioned embodiment, the drive recorder 1 has the acceleration sensor 18 to detect the state of the vehicle (driving state, etc.), but if the acceleration sensor 18 is not used for generating a learning model or detecting a scene, it may not have this sensor. In the second embodiment, the drive recorder 1 may not execute the process of step 91, or may not execute the process of detecting an event by the acceleration sensor 18 among the processes of step 91. In this case, the second degree of agreement is not used, and an image of a specific scene is detected based on whether or not the first degree of agreement is equal to or greater than a threshold value. In such a case, the drive recorder 1 may not have the acceleration sensor 18. If the drive recorder 1 detects the state of the vehicle using a sensor other than the acceleration sensor 18, the drive recorder 1 may be configured not to have the sensor.

[0195] In the above-described embodiment, the functions realized using deep learning technology may be realized using other machine learning or other alternative means, if available. For example, pattern matching technology may be used as a means for detecting images with a high degree of match.

[0196] The scope of the present invention is not limited to the configurations explicitly described in the specification, but includes combinations of various aspects of the present invention disclosed in this specification. The configurations of the present invention that are to be patented are specified in the attached claims, but it is our intention to include configurations disclosed in this specification in the claims in the future, even if they are not currently specified in the claims.

[0197] The present invention is not limited to the configuration described in the above-mentioned embodiment. The components of each of the above-mentioned embodiments and modifications may be arbitrarily selected and combined. Any components of each of the embodiments and modifications may be arbitrarily combined with any components described in the means for solving the invention or any components that embody any components described in the means for solving the invention. We intend to acquire rights to these as well through amendments to this application or divisional applications. Even if there is a description such as "in the case of" or "when", this is not intended to describe a configuration that is limited to that case or time. We also disclose configurations that are not in these cases or times, and we intend to acquire rights. Furthermore, the parts described in order are not limited to this order. We also disclose configurations in which some parts are deleted or the order is changed, and we intend to acquire rights.

[0198] In addition, we intend to obtain rights to the overall design or partial design by converting to a design registration application. The drawings depict the entire device in solid lines, but they include not only the overall design but also partial designs claimed for a part of the device. For example, some parts of the device may be partial designs, and the drawings include some parts of the device as partial designs regardless of the parts. A part of the device may be a part of the device, or a part of that part. We intend to obtain rights to the overall design, as well as partial designs in which any part of the solid line part of the drawing is shown as a broken line part. In addition, the modules, parts, and components inside the device's casing, all of which are shown in the drawings, are subject to trade independently, and we intend to obtain rights to them by converting to a design registration application. [Explanation of symbols]

[0199] 1: Drive recorder, 3: Windshield, 4: Room mirror, 5: Cigarette lighter socket, 6: Power cable, 10: SD card slot, 11: Display, 12: Operation buttons, 13: Joint rail, 14: SD card reader / writer, 15: Speaker, 16: GPS receiver, 17: Camera, 18: Acceleration sensor, 19: Communication circuit, 20: Controller, 20c: RAM, 20d: Timer, 21: LTE module, 22: Database, 23: SD card, 30: Server, 31: Control device, 33: Memory, 34: Hard disk disk drive, 35: hard disk, 40: learning model, 40A: first learning model, 40B: second learning model, 40C: third learning model, 41: input layer, 42: intermediate layer, 43: output layer, 43a: neuron, 50: target image, 51: first target image, 52: second target image, 53: third target image, 71: generator, 72: discriminator, 73: weight changer, 80: learning model, 81: input layer, 82: intermediate layer, 83: output layer, 105: learning model, 106: input layer, 107: intermediate layer, 108: output layer, 110: personal computer data, 111: display device, 112: display control device, 113: communication device, 114: memory, 115: control device, 116: input device, 117: hard disk drive, 118: hard disk, 119: SD card reader / writer, 121: file system area, 122: continuous recording area, 123: management area, 124: recording area, 125: event recording area, 126: management area, 127: recording area, 131: header recording area, 132: frame image data recording area, 133: footer recording area, 150: playback window, 151: video display display area, 152: event record list display area, 153: map image display area, 154: interior highlight button, 155: normal display button, 156: time display area, 157: operation button, 158: vehicle state display area, 159: acceleration graph display area, 160: vehicle information etc. display area, 161: record list display area, 162: acceleration display area, 163: acceleration display area, 180: learning model, 181: input layer, 182: intermediate layer, 183: output layer, 421: input layer, E1: pixel, E2: pixel, FR1: frame, FRn: frame, P1: pixel, P21: pixel

Claims

1. A system having a function of detecting an image representing a specific image scene from images taken by a vehicle at each of a plurality of time points based on data defining the specific image scene that can be taken by a vehicle, the system comprising: The function of taking pictures using cameras installed in the vehicle, A function of recording images taken by the camera at each of a plurality of points in time on a recording medium; having The above detection function is Detecting an image representing a scene having a degree of similarity with a scene of the specific image equal to or greater than a threshold value among the images recorded on the recording medium at each point in time, when an object moving at a speed or acceleration equal to or greater than a certain value is present in the image; system.

2. The data is generated by deep learning by inputting an image identical to the specific image or a plurality of images similar to the specific image. The system of claim 1 .

3. a warning control means for controlling a warning device to issue a warning in response to detection of an image representing the specific image scene; The system of claim 1 or 2, further comprising:

Images