Image recording device

Abstract

To improve a technique for synchronizing video data and control data.SOLUTION: An image recording device 20 includes a control unit 21 that performs control of acquiring video data D1 consisting of one or more images of a vehicle VH captured by an imaging device 30 provided in the vehicle VH, and control data of the vehicle VH including a time stamp acquired from the vehicle VH when each image was captured, and embedding and saving, as metadata, the acquired control in the image data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image recording device. [Background technology]

[0002] Patent document 1 discloses an in-vehicle video recording device that includes a frame memory circuit that stores video data converted from video signals captured by multiple cameras in frame units, a control unit that controls one of the frame memory circuits to output the video data in chronological order, a video data compression circuit that compresses the frame-unit video data output from one of the frame memory circuits, a digital recording unit that records the compressed video data on a memory card, and a recording frame rate setting unit that sets the frame rate output from each frame memory circuit based on a predetermined setting value. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-028635 Summary of the Invention [Problem to be solved by the invention]

[0004] Video data acquired from a vehicle and its control data are analyzed to diagnose vehicle malfunctions. Control data is also called time-series data. When analyzing data, there is a need to check the value of the control data at a certain moment in the video. Control data includes, for example, records and timestamps of "diagnosis communication" performed to diagnose vehicle malfunctions, etc. In the diagnosis communication, data for fault diagnosis is transmitted from the ECU. "ECU" is an abbreviation for electronic control unit. Generally, data is transmitted between ECUs using an in-vehicle network such as CAN. "CAN" is an abbreviation for Controller Area Network.

[0005] Currently, a timestamp is not recorded for each image (frame) that makes up video data. Instead, the shooting time for each image is calculated based on the video's recording start time and its frame rate (FPS). The video data and control data are synchronized based on this calculated shooting time. "FPS" stands for frames per second. This is because existing video data formats (such as the MP4 standard) do not allow for embedding metadata such as timestamps. The official name for "MP4" is MPEG Layer-4 Part 14. "MPEG" stands for Moving Pictures Expert Group. Previously, linking video data and control data (time-series data) required preprocessing, which involved assigning a timestamp to each image in the video data, which was time-consuming. Therefore, there was room for improvement in the technology for synchronizing video data and control data.

[0006] An object of the present disclosure is to improve techniques for synchronizing video data and control data. [Means for solving the problem]

[0007] The image recording device according to the present disclosure includes a control unit that acquires video data consisting of one or more images captured by an imaging device installed in a vehicle and control data for the vehicle including a timestamp acquired from the vehicle when each image was captured, and embeds the acquired control data as metadata in the image data and stores it. [Effects of the Invention]

[0008] The present disclosure provides an improved technique for synchronizing video data and control data, which makes it easier to handle video data and control data. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating a configuration of an image recording apparatus according to an embodiment of the present disclosure. [Figure 2] 4 is a flowchart illustrating an operation of the image recording device according to an embodiment of the present disclosure. [Figure 3] 1A to 1C are diagrams for explaining an image recording method according to an embodiment of the present disclosure. [Figure 4] FIG. 10 is a diagram for explaining playback of data recorded by an image recording device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0011] In each drawing, the same or corresponding parts are denoted by the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

[0012] The configuration of an image recording device 20 according to this embodiment will be described with reference to FIG.

[0013] In this embodiment, the image recording device 20 is mounted on a vehicle VH. The image recording device 20 includes a control unit 21, a storage unit 22, and a communication unit .

[0014] The control unit 21 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for specific processing. "CPU" is an abbreviation for central processing unit. "GPU" is an abbreviation for graphics processing unit. An example of the programmable circuit is an FPGA. "FPGA" is an abbreviation for field-programmable gate array. An example of the dedicated circuit is an ASIC. "ASIC" is an abbreviation for application specific integrated circuit. The control unit 21 controls each part of the image recording device 20 and executes processing related to the operation of the image recording device 20.

[0015] The storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, RAM or ROM. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read only memory. RAM is, for example, SRAM or DRAM. "SRAM" is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. ROM is, for example, EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read only memory. The storage unit 22 functions as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 22 stores data used in the operation of the image recording device 20 and data obtained by the operation of the image recording device 20. The storage unit 22 may store video data D1 acquired from an imaging device 30 provided in the vehicle VH, control data D2 for the vehicle VH, and composite data D1' in which the video data D1 and the control data D2 are integrated. The video data D1, control data D2, composite data D1', and the imaging device 30 will be described later.

[0016] The communication unit 23 includes at least one communication interface. The communication interface is, for example, an interface compatible with a mobile communication standard such as LTE, 4G standard, or 5G standard, an interface compatible with short-range wireless communication such as Bluetooth (registered trademark), or a LAN interface. "LTE" is an abbreviation for Long Term Evolution. "4G" is an abbreviation for 4th generation. "5G" is an abbreviation for 5th generation. The communication unit 23 receives data used in the operation of the image recording device 20 and transmits data obtained by the operation of the image recording device 20. In this embodiment, the communication unit 23 communicates with the imaging device 30.

[0017] The functions of the image recording device 20 are realized by executing a program according to this embodiment on a processor serving as the control unit 21. That is, the functions of the image recording device 20 are realized by software. The program causes a computer to execute the operations of the image recording device 20, thereby causing the computer to function as the image recording device 20. That is, the computer functions as the image recording device 20 by executing the operations of the image recording device 20 in accordance with the program.

[0018] The program can be stored on a non-transitory computer-readable medium. Examples of non-transitory computer-readable media include flash memory, magnetic recording devices, optical disks, magneto-optical recording media, and ROMs. The program can be distributed by selling, transferring, or lending portable media such as SD cards, DVDs, or CD-ROMs that store the program. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for digital versatile disc. "CD-ROM" is an abbreviation for compact disc read only memory. The program can also be distributed by storing it in the storage of a server and transferring it from the server to another computer. The program can also be provided as a program product.

[0019] A computer temporarily stores a program stored on a portable medium or transferred from a server in its main storage device. The computer then reads the program stored in the main storage device using a processor and executes processing in accordance with the read program. The computer may also read the program directly from a portable medium and execute processing in accordance with the program. The computer may also execute processing in accordance with the received program each time a program is transferred from a server to the computer. Processing may also be executed through a so-called ASP-type service that achieves its function simply by issuing an execution command and obtaining the results, without transferring the program from the server to the computer. "ASP" is an abbreviation for application service provider. Programs include information used for processing by a computer that is equivalent to a program. For example, data that does not directly instruct a computer but has properties that define computer processing falls under the category of "equivalent to a program."

[0020] Some or all of the functions of the image recording device 20 may be realized by a programmable circuit or a dedicated circuit as the control unit 21. In other words, some or all of the functions of the image recording device 20 may be realized by hardware.

[0021] An overview of this embodiment will be described with reference to FIG.

[0022] The image recording device 20 acquires video data D1 and control data D2. The video data D1 is composed of one or more images (frames) Fi captured by an imaging device 30 provided in the vehicle VH. The control data D2 includes a timestamp acquired from the vehicle VH when each image Fi was captured. The image recording device 20 embeds the acquired control data D2 as metadata in the data of the images Fi and saves it.

[0023] According to this embodiment, control data D2 is added to video data D1 as metadata. Therefore, it is possible to read data in a state where control data D2 is linked to one or more images Fi constituting the video data D1, without performing preprocessing such as calculating the shooting time for each image Fi in the video data D1 and adding the calculated shooting time to each image Fi. This improves the technology for synchronizing the video data D1 and the control data D2. As a result, it becomes easier to handle the video data D1 and the control data D2.

[0024] In this embodiment, the imaging device 30 provided in the vehicle VH includes a camera that generates an image of a subject within its field of view. The camera may be a monocular camera or a stereo camera. The camera includes an optical system such as a lens and an imaging element such as a CCD image sensor or a CMOS image sensor. "CCD" is an abbreviation for charge-coupled device. "CMOS" is an abbreviation for complementary metal oxide semiconductor. In this embodiment, the imaging device 30 captures images of the periphery of the vehicle VH. The imaging device 30 may also capture images of the interior of the vehicle VH. In this embodiment, the imaging device 30 continuously captures one or more images Fi at a predetermined frame rate. In this embodiment, the imaging device 30 outputs data composed of the one or more captured images Fi as video data D1.

[0025] In this embodiment, the imaging device 30 is connected to the image recording device 20 as an external input device. Any connection method, such as USB, HDMI (registered trademark), or Bluetooth (registered trademark), can be used. "USB" is an abbreviation for Universal Serial Bus. "HDMI (registered trademark)" is an abbreviation for High-Definition Multimedia Interface. The control unit 21 of the image recording device 20 acquires the video data D1 output by the imaging device 30. Specifically, the control unit 21 communicates with the imaging device 30 via the communication unit 23 and receives the video data D1 output from the imaging device 30. The imaging device 30 may be an external sensor. The external sensor may be, for example, an in-vehicle camera such as a front camera or a rear camera, LiDAR, millimeter-wave radar, laser radar, or an ultrasonic sensor, as long as it acquires information about the external world as video. "LiDAR" is an abbreviation for light detection and ranging. The imaging device 30 may be built into the image recording device 20 instead of being connected to the image recording device 20 as an external input device.

[0026] The operation of the image recording device 20 according to this embodiment will be described with reference to Figures 2 to 4. This operation corresponds to the image recording method according to this embodiment. That is, the image recording method according to this embodiment includes steps S1 to S3 shown in Figure 2. Hereinafter, each step in the flowchart will be identified by an S and a number.

[0027] In S1 of FIG. 2, the control unit 21 of the image recording device 20 acquires video data D1. The video data D1 may be acquired by any procedure, but is acquired, for example, by the following procedure. In this embodiment, the imaging device 30 provided in the vehicle VH captures an image of the periphery of the vehicle VH. Specifically, the imaging device 30 captures an image of the periphery of the vehicle VH by continuously capturing one or more images Fi of the periphery of the vehicle VH at a predetermined frame rate. The imaging device 30 may also capture an image of the interior of the vehicle VH. For example, the imaging device 30 may capture an image of a malfunction location in the interior of the vehicle VH. The control unit 21 acquires data consisting of one or more images Fi captured by the imaging device 30 as video data D1. The control unit 21 stores the acquired video data D1 in the storage unit 22. As shown in FIG. 3, one or more images (frames) Fi constituting the video data D1 include images F1 to F3.

[0028] In S2 of FIG. 2, the control unit 21 of the image recording device 20 acquires control data D2. The control data D2 may be acquired by any procedure, but is acquired, for example, by the following procedure. In this embodiment, the control data D2 includes a record of diagnostic communication in the vehicle VH. As an example, assume that when the image F1 is captured by the imaging device 30 in S1, fault diagnosis data is transmitted from the ECU as diagnostic communication in the vehicle VH. The control unit 21 monitors the diagnostic communication in the vehicle VH to acquire a transmission record Ri of the fault diagnosis data from the ECU together with a timestamp as control data D2. The control unit 21 stores the acquired control data D2 in the storage unit 22. As shown in FIG. 3, the control data D2 includes transmission records R1 to R3 as transmission records Ri of the fault diagnosis data from the ECU. Each of the transmission records R1 to R3 is assigned a timestamp.

[0029] In S3 of FIG. 2, the control unit 21 of the image recording device 20 embeds the control data D2 acquired in S2 as metadata in the video data D1 acquired in S1. Specifically, the control unit 21 superimposes the control data D2 on the images Fi constituting the video data D1 and outputs the data in a data format in which the video data D1 and the control data D2 are integrated. Alternatively, the control unit 21 assigns, as metadata, to one or more images Fi constituting the video data D1, a timestamp included in the control data D2 acquired from the vehicle VH when each image Fi was captured. As an example, assume that, in S2, a transmission record R1 of fault diagnosis data transmitted as diagnostic communication from the ECU when the image F1 was captured by the imaging device 30 is acquired as the control data D2. The control unit 21 assigns, as metadata, the timestamp of the transmission record R1 indicated in the control data D2 to the data of the image F1 constituting the video data D1. At this time, the control unit 21 may treat the control data D2 as an attached file of the video data D1 and link the two data using the timestamp of the metadata assigned to the image F1. FIG. 3 shows an example of composite data D1' in which the video data D1 and the control data D2 are integrated. In the example of FIG. 3, the composite data D1' is composed of images F1' to F3' in which the timestamps of the transmission records R1 to R3 indicated in the control data D2 are assigned to the data of the images F1 to F3 that make up the video data D1. The control unit 21 stores the obtained composite data D1' in the storage unit 22.

[0030] As described above, according to this embodiment, the image recording device 20 acquires video data D1 consisting of one or more images Fi captured by the imaging device 30 provided in the vehicle VH, and control data D2 of the vehicle VH including timestamps acquired from the vehicle VH when each image Fi was captured. The image recording device 20 embeds the acquired control data D2 as metadata in the data of the images Fi and saves it as composite data D1'.

[0031] According to this configuration, the control data D2 is added to the video data D1 as metadata, and the resulting data is saved as composite data D1'. Therefore, by playing back the composite data D1', it is possible to read out the data in a state in which the control data D2 is linked to one or more images Fi constituting the video data D1. This improves the technique for synchronizing the video data D1 and the control data D2. For example, as shown in FIG. 4, when the composite data D1' is played back, a value indicated by the control data D2 is displayed on the image represented by the video data D1. As a result, it becomes easier for an analyst U1 attempting to diagnose a problem in the vehicle VH based on the video data D1 and the control data D2 to associate and handle the video data D1 and the control data D2.

[0032] In the above-described embodiment, the image recording device 20 is described as being mounted on the vehicle VH, but this is not limited to this. The image recording device 20 may be provided outside the vehicle VH and connected to the vehicle VH so as to be able to communicate with it via a network such as the Internet.

[0033] The present disclosure is not limited to the above-described embodiments. For example, multiple blocks shown in the block diagrams may be integrated, or a single block may be divided. Instead of executing multiple steps shown in the flowcharts in chronological order as described, steps may be executed in parallel or in a different order depending on the processing capabilities of the device executing each step, or as needed. Other modifications are possible within the scope of the present disclosure. [Explanation of symbols]

[0034] 20 Image recording device 21 Control section 22 Memory section 23 Communications Department 30 Imaging device D1 Video data D2 Control data D1' Composite Data Fi,F1,F2,F3,Fi',F1',F2',F3' Image Ri, R1, R2, R3 transmission record VH vehicle U1 Analyst

Claims

[Claim 1] An image recording device comprising a control unit that acquires video data consisting of one or more images captured by an imaging device installed in a vehicle and control data for the vehicle including a timestamp acquired from the vehicle when each image was captured, and embeds the acquired control data as metadata in the image data and stores it.

Images