Image processing device, image recording device, image playback device, image processing method, image recording method, and image playback method

Abstract

To provide an image processing device capable of reducing the time lag between video data captured by multiple cameras.SOLUTION: The image processing device includes: an acquisition unit that acquires a first image taken by a first imaging unit and a second image taken by a second imaging unit; a location information acquisition unit that acquires location information of a vehicle equipped with the first imaging unit and the second imaging unit based on the distance traveled from a reference point; and a processing unit that adds the acquired location information and a synchronization signal to synchronize the first image and the second image to the visible or invisible areas of the first image and the second image.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an image recording device, an image playback device, an image processing method, an image recording method, and an image playback method for a monitoring system that monitors train tracks. [Background technology]

[0002] For example, as disclosed in Patent Documents 1 and 2, it has been known for some time that in order to ensure the safety of railway and other vehicle operations, cameras are mounted on railway vehicles and images of the tracks and surrounding areas captured by the cameras while the railway vehicles are traveling are monitored. The monitored objects captured by the cameras are varied, including, for example, electric train lines (overhead electric train lines, hereinafter also referred to as "catenary lines") installed along the tracks to supply power to the vehicles. Electric train lines are made up of electric wires such as contact wires and feeder lines, and equipment that supports the electric wires. Examples of such equipment include catenary poles (support poles, utility poles), beams, hangers, suspension wires, anti-vibration pipes, anti-vibration fittings, and the like. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-168268 [Patent Document 2] Japanese Patent Publication No. 2020-017824 Summary of the Invention [Problem to be solved by the invention]

[0004] When monitoring electric railroad tracks, multiple cameras are installed on the roof of an electric vehicle, and the condition of the electric railroad tracks installed along the tracks is captured by a camera installed on the electric vehicle while it is in motion. For example, the condition of overhead poles, beams, hangers, etc. is simultaneously captured by two cameras with different field of view. The cameras used are high-resolution cameras capable of capturing video data with a quality of, for example, 4K or higher, so that loose overhead and suspension wires and missing bolts on overhead poles, beams, hangers, etc. can be confirmed. The video data captured by the high-resolution cameras is output at a frame rate of, for example, 60 fps and transmitted to a recording device or the like installed on the vehicle via a signal converter such as an optical converter, and the video data captured by each camera is recorded.

[0005] However, there have been cases where frame misalignment has occurred between video data of a monitored object captured simultaneously by two cameras with different field of view. This frame misalignment is thought to be caused by, for example, variations in processing delays in the compression process of video data in signal converters and recording devices that make up the transmission path. When frame misalignment has occurred, for example, a supervisor monitoring the railroad tracks has had to manually rewind or fast-forward multiple frames of video data to understand the images of the monitored object captured simultaneously by multiple cameras, which has been a burden.

[0006] The present disclosure has been made in consideration of the above circumstances, and aims to provide a technology for suppressing time lag between video data captured by multiple cameras. [Means for solving the problem]

[0007] In order to solve the above problems, the following measures are adopted. That is, an image processing device according to one aspect of the present disclosure includes: A first image captured by the first imaging unit and a second image captured by the second imaging unit are acquired. The acquisition part that will benefit you, a location information acquisition unit that acquires location information based on a travel distance from a reference point of a vehicle equipped with the first imaging unit and the second imaging unit; a processing unit that adds the acquired position information and a synchronization signal for synchronizing the first image and the second image to a visible area or an invisible area of ​​the first image and the second image; Equipped with.

[0008] This makes it possible to reduce time lag between video data captured by multiple cameras based on the synchronization signals and location information added to the visible or invisible areas of the first and second images. [Effects of the Invention]

[0009] The present disclosure provides a technology for suppressing time lag between video data captured by multiple cameras. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 2 is a diagram illustrating a monitoring camera connected to an image processing device according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating the positions of a plurality of cameras provided on the roof of a vehicle. [Figure 3] FIG. 10 is a diagram showing an example of an image of a monitoring target captured by a camera. [Figure 4] FIG. 10 is a diagram showing another example of an image of a monitored object captured by a camera. [Figure 5] FIG. 10 is a diagram showing another example of an image of a monitored object captured by a camera. [Figure 6] FIG. 1 is a diagram illustrating a monitoring system of a comparative example. [Figure 7] FIG. 10 is a diagram illustrating another monitoring system of a comparative example. [Figure 8] 1 is a diagram illustrating a schematic configuration of a monitoring system according to an embodiment. [Figure 9] FIG. 1 is a diagram illustrating an example of an image processing apparatus according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The configurations of the following embodiments are examples, and the present disclosure is not limited to the configurations of the embodiments. Furthermore, the following embodiments can be combined as much as possible.

[0012] First Embodiment FIG. 1 is a diagram illustrating a monitoring camera connected to an image processing device according to this embodiment. As shown in the dashed rectangular frame A1 in FIG. 1, the monitoring camera is mounted on the roof of an electric vehicle (hereinafter simply referred to as a "vehicle") 50 traveling on the railroad tracks and captures the condition of the electric railroad tracks installed along the tracks. The monitoring camera (hereinafter simply referred to as a "camera") simultaneously captures the condition of monitored objects, such as electric wires such as electric wires and feeder lines, overhead poles (support poles, utility poles) that support the electric wires, beams, hangers, suspension wires, and vibration-control pipes, using multiple cameras with different field of view. For example, as shown by Z1#1 to Z1#13 in FIG. 1, video images capturing the condition of the monitored objects on the electric railroad tracks are captured by multiple cameras installed on the roof of a vehicle 50 traveling on the railroad tracks in the direction of the solid arrow A2.

[0013] FIG. 2 is a diagram illustrating the positions of multiple cameras installed on the roof of a vehicle 50. FIG. 2 illustrates a top view of eight cameras (2a to 2h) installed on the roof top surface 50a of the vehicle 50. Each camera (2a to 2h) is combined with a mount (5a to 5h) for fixing the camera to the roof top surface 50a and an illuminator (4a to 4h) for illuminating the object to be photographed, to form a camera unit. An air conditioner 51 surrounded by an air conditioning work fence 51a is installed on the roof top surface 50a of the vehicle 50. Hereinafter, the cameras (2a to 2h) will be collectively referred to as camera 2.

[0014] For example, camera 2a, together with mount 5a and illuminator 4a, constitutes an overhead line camera unit, and camera 2a and illuminator 4a are fixed to mount 5a. Similarly, camera 2b, together with mount 5b and illuminator 4b, constitutes an overhead line camera unit, and camera 2b and illuminator 4b are fixed to mount 5b. Each overhead line camera unit is provided with a utility pole detection sensor (3a, 3b) that detects an overhead line pole. The utility pole detection sensors (3a, 3b) measure the distance to an overhead line pole illuminated by, for example, a laser, and output a binary detection signal indicating an active status when the measured distance is equal to or less than a certain threshold, and an inactive status otherwise. Hereinafter, the active status will also be referred to as an on state, and the inactive status will also be referred to as an off state.

[0015] Camera 2c, together with mount 5c and illuminator 4c, constitutes a side camera unit, and camera 2c and illuminator 4c are fixed to mount 5c. Cameras 2d, 2e, and 2f, together with mounts 5d, 5e, and 5f and illuminators 4d, 4e, and 4f, constitute a side camera unit, and each camera is fixed to mounts (5d, 5e, 5f) together with its corresponding illuminator (4d, 4e, 4f).

[0016] Furthermore, the camera 2g, together with the mount 5g and the illuminator 4g, constitute a front-rear camera unit, and the camera 2g and the illuminator 4g are fixed to the mount 5g. Similarly, the camera 2h, together with the mount 5h and the illuminator 4h, constitute a front-rear camera unit, and the camera 2h and the illuminator 4h are fixed to the mount 5h.

[0017] If the direction of the front and rear camera units equipped with camera 2g is set to the direction of travel of vehicle 50, camera 2a simultaneously captures the state of the electric rails installed on the left side along the tracks, and camera 2b simultaneously captures the state of the objects on the electric rails installed on the right side along the tracks. Similarly, for example, cameras 2c, 2d, 2e, and 2f constituting the side camera units simultaneously capture the state of the electric rails installed on the left and right sides along the tracks. Furthermore, cameras 2g and 2h installed in the front and rear camera units simultaneously capture the state of, for example, beams installed between overhead line poles on either side of the tracks.

[0018] 3 to 5 are diagrams showing examples of images captured by cameras installed on a vehicle 50. Fig. 3 shows, for example, images Z2 and Z3 captured by cameras 2c and 2d constituting the side camera unit. Fig. 4 shows, for example, images Z4 and Z5 captured by cameras 2a and 2b constituting the overhead line camera unit. Fig. 5 shows, for example, image Z6 captured by camera 2g constituting the front / rear camera unit.

[0019] As shown in the dashed frame 62 of photographed image Z2, the state of a hanger that suspends the contact wire 60 from the suspension wire 61, which is provided on the right side of the traveling direction of the vehicle 50, is photographed by camera 2c. Similarly, as shown in the dashed frame 62 of photographed image Z3, the state of a hanger that suspends the contact wire 60 from the suspension wire 61, which is provided on the left side of the traveling direction of the vehicle 50, is photographed by camera 2d. As photographed images Z2 and Z3 are taken simultaneously by cameras 2c and 2d from a traveling vehicle 50, missing bolts that secure the suspension wire 61 and the contact wire 60 of the hanger 62, etc., can be identified from the images.

[0020] The same applies to photographed images Z4 and Z5 in Fig. 4. As shown in the dashed frame 63 of photographed image Z4, the state of the power feeder branch line 64, which is provided on the right side of the traveling direction of the vehicle 50 and connects the power feeder line 65 and the trolley wire 60, is photographed via camera 2a. Also, as shown in the dashed frame 63 of photographed image Z5, the state of the power feeder branch line 64, which is provided on the left side of the traveling direction of the vehicle 50 and connects the power feeder line 65 and the trolley wire 60, is photographed via camera 2b. Photographed images Z4 and Z5 are photographed at the same time via camera 2a and camera 2b. By taking a photograph, missing metal fittings to which the feeder wire 65, the feeder branch wire 64, and the trolley wire 60 are connected, and tears in the coating of the feeder branch wire 64, etc. can be identified through the image.

[0021] As shown in photographed image Z6 in Fig. 5, objects monitored by cameras (2a to 2h) mounted on the roof of vehicle 50 include beam 71, high-voltage power line 66, anti-vibration pipe 72, and anti-vibration metal fitting 72a, which are installed between overhead line poles 70 installed on the left and right sides of the tracks. From photographed image Z6, it is possible to grasp the fixed state of overhead line pole 70 and beam, distortion of anti-vibration pipe 72, missing anti-vibration metal fitting 72a, and the degree of bending of high-voltage power line 66, feeder line 65, and feeder branch line 64, etc.

[0022] In this way, the cameras (2a to 2h) installed in the vehicle 50 are high-resolution cameras that can capture images of the state of the monitored object with high image quality of, for example, 4K or higher so that the state of the monitored object can be confirmed from the captured video. The video captured by the cameras (2a to 2h) is output at a frame rate of, for example, 60 fps, and transmitted to a recording device or the like installed in the vehicle via a signal converter such as an optical converter, and the video data captured by each camera is recorded.

[0023] Fig. 6 is a diagram illustrating a monitoring system 100 that monitors the state of a train track using video images captured by a monitoring camera, as a comparative example. For the sake of explanation, Fig. 6 illustrates an example of processing of two types of video images captured by a camera 2a and a camera 2b.

[0024] In FIG. 6, video images captured by cameras 2a and 2b are output in an uncompressed state to signal converters (hereinafter simply referred to as "converters") (41a, 41b) such as optical converters. Camera 2a and converter 41a, and camera 2b and converter 41b are connected using, for example, a high-speed transmission cable conforming to the 12G-SDI (Serial Digital Interface) standard, which is capable of transmitting video images of 4k resolution or higher. The video images converted into video by converters 41a and 41b are connected again via optical cables to converters 42a and 42b, which are signal converters such as optical converters, respectively. Converters 42a and 42b are connected via optical cables to image recording device 20. Image recording device 20 is provided, for example, in the leading or trailing car of vehicle 50. In order to accommodate the length of the wiring that passes through multiple vehicles to reach the vehicle 50 in which the image recording device 20 is installed, the video images output from each camera are transmitted via multiple optical cables and converters (41a, 42a, 41b, 42b).

[0025] The image recording device 20 includes a CPU (Central Processing Unit), an MPU (Micro Processor A computer such as a PC that includes a processor such as a Serial Communication Unit (SCM) and a storage medium such as a memory. The memory includes flash memory, RAM (Random Access Memory), ROM (Read Only Memory), etc. The image recording device 20 provides functions that meet a predetermined purpose by controlling peripheral devices through execution of programs stored in the memory, etc.

[0026] As shown in FIG. 6, the image recording device 20 includes image capture boards 21a and 21b and an external storage device 22. The external storage device 22 is, for example, an SSD (Solid State Drive) externally attached to the image recording device 20. In this embodiment, the external storage device 22 is a "storage The converter 42a is connected to the image capture board 21a via an optical cable, and the converter 42b is connected to the image capture board 21b via an optical cable. The image capture board 21a converts the optical signal output from the converter 42a and acquires video data that conforms to the 12G-SDI standard and that was captured by the camera 2a. The image capture board 21a then encodes and compresses the acquired video data, and stores it in the external storage device 22 as a video file 101a captured by the camera 2a. An example of such a compression encoding method is the h.265 format. The same is true for the image capture board 21b. Video data that is compliant with the 12G-SDI standard and that is captured by the camera 2b is encoded and compressed using the h.265 format compression encoding method, and is stored in the external storage device 22 as a video file 101b captured by the camera 2b.

[0027] The image reproducing device 30 includes a CPU (Central Processing Unit), an MPU (Micro Processor A computer such as a PC that includes a processor such as a Serial Communication Unit (SCM) and a storage medium such as a memory. The memory includes a flash memory, a RAM (Random Access Memory), a ROM (Read Only Memory), etc. The image playback device 30 executes programs stored in the memory, etc., to control peripheral devices and provide functions that meet a predetermined purpose.

[0028] The image playback device 30 acquires the video files 101a and 101b stored in the external storage device 22 of the image recording device 20. The image playback device 30 then decodes each acquired video file to play back the video data captured by the cameras 2a and 2b before encoding, and displays it on a display device such as an LCD (Liquid Crystal Display) monitor. The display device of the image playback device 30 displays, for example, the state of the monitored object of the train tracks captured by the cameras 2a and 2b, as shown in Figures 3 to 5.

[0029] However, there have been cases where frame lags have occurred between video data of a monitored object captured simultaneously by cameras 2a and 2b. These frame lags are caused by variations in processing delays in the video data compression process in converters 41a and 41b and recording devices that make up the transmission path.

[0030] For example, let F1 to F5 enclosed by a rectangular frame in Fig. 6 be video frames captured by each camera. An instruction signal for synchronizing the timing of the start of capture is input to cameras 2a and 2b. Camera 2a starts capturing images of the monitored object on the train tracks in accordance with the instruction signal, and outputs video frames to converter 41a at a predetermined frame rate (e.g., 60 fps) in the order of F1, F2, F3, F4, and F5. The same is true for camera 2b. Video frames of the monitored object, which have been captured in accordance with the instruction signal, are output to converter 41b in the order of F1, F2, F3, F4, and F5.

[0031] Converters 41a and 41b convert 12G-SDI video images into optical signals and output them to converters 42a and 42b, respectively. Due to variations in processing performance between the devices that make up converters 41a and 41b, the timing of the conversion into optical signals differs between converters 41a and 41b. For this reason, as shown in FIG. 6, video frame F3 captured by camera 2b is output from converter 41b at the same time as video frame F2 is output from converter 41a. A similar difference in processing timing occurs between converters 42a and 42b, and video frame F4 captured by camera 2b is output from converter 42b at the same time as video frame F3 is output from converter 42a.

[0032] In the image capture boards 21a and 21b of the image recording device 20, differences in the processing performance and timing of the compression encoding process between the devices that make them up will also cause frame lag between the image capture boards 21a and 21b, which depends on the compression encoding, etc. For example, the image capture board 21a will generate a compressed moving image file 101a at the timing of moving image frame F4, and the image capture board 21b will generate a compressed moving image file 101b at the timing of moving image frame F5.

[0033] As shown in the solid line balloon 102, the image playback device 30 reads and plays back the video frames compressed in the video files 101a and 101b, for example, from the beginning. , and displayed on the display screen of an LCD monitor. The video data of the played video files 101a and 101b is played frame by frame, resulting in a frame lag between the played images. As shown in FIG. 6, the video images of the monitored object, which were started to be captured at the same time, are played back from video frame 4 on camera 2a and from video frame 5 on camera 2b, resulting in a frame lag between the played images. If a frame lag occurs between the played images, for example, a monitor monitoring the train tracks must manually rewind or fast-forward the frames of the multiple video data. This creates a burden on the monitor, as they must understand the images of the monitored object captured simultaneously by multiple cameras.

[0034] FIG. 7 is a diagram illustrating another comparative example. FIG. 7 illustrates a processing configuration for eight channels of video images, in which timestamp information triggered by detection signals from utility pole detection sensors (3a, 3b) is added when compressed video data is stored in memory. In the configuration shown in FIG. 7, the transmission paths for eight channels of camera images are divided into two groups (#1, #2), and images for each group are recorded by two image recording devices (20#1, 20#2). The channels of group #1 may be configured, for example, as channels corresponding to each of the cameras (2a, 2b, 2g, 2h), or as channels corresponding to each of the cameras (2a, 2c, 2e, 2h). Group #2 is configured as channels corresponding to the four cameras other than group #1. An instruction signal generated by signal generator 80 for synchronizing the timing of the start of recording is input to the cameras belonging to group #1 via distributor 81a. Furthermore, an instruction signal generated by the signal generator 80 for synchronizing the timing of starting shooting is input to the cameras belonging to group #2 via distributor 81b.

[0035] Uncompressed video images captured by camera 2#1, a group of four cameras constituting group #1, are input to image capture board 21#1 of image recording device 20#1 via converters 41 (41#1) and 42 (42#1) corresponding to each of the four systems. Similarly, uncompressed video images captured by camera 2#2, a group of four cameras constituting group #2, are input to image capture board 21#2 of image recording device 20#2 via converters 41 (41#2) and 42 (42#2) corresponding to each of the four systems. In each image capture board belonging to groups #1 and #2, the optical signal output from converter 42 is converted, and video data conforming to the 12G-SDI standard captured by each camera is acquired (SDI CONV). Then, each image capture board encodes the acquired video data. The video files for each camera generated by each image capture board are stored in the external storage device 22 via, for example, a USB (Universal Serial Bus). In the external storage device 22, the eight video files captured by the cameras (2a to 2h) are stored in the corresponding SSDs.

[0036] In the embodiment shown in FIG. 7, the utility pole detection signal detected by the utility pole detection sensors (3a, 3b) is used as a trigger to transmit the DIO (Data Input / Output) of the management device 90 that manages time stamp information. The management device 90 receives the data via a LAN provided in the vehicle 50. The management device 90 is connected to two image recording devices (20#1, 20#2) through the management device 90. When the utility pole detection signal is in an ON state indicating utility pole detection, the management device 90 outputs time information managed by itself as time stamp information to the two image recording devices (20#1, 20#2). When storing a moving image file after image compression in a ring buffer memory, each image recording device adds the time stamp information output from the management device 90 to the moving image file. Note that time management for each moving image frame in the moving image file is handled by the image recording device 20 that generates the moving image file.

[0037] However, even with the configuration shown in Figure 7, there were cases where discrepancies in the timestamp information between image frames of a monitored object occurred. Specifically, the video files of each camera stored in the external storage device 22 were played back using the image playback device 30, and the timestamp information and the time information for the image frames were analyzed. As a result, it was found that when the timestamp was used as a key common to the eight video files, discrepancies could occur between the management times of the eight image frames in which the monitored object was captured. It was confirmed that such discrepancies in the timestamp information depended at least on variations in the processing delay in the video data compression process.

[0038] FIG. 8 is a diagram illustrating the schematic configuration of a monitoring system 1 according to this embodiment. FIG. 8 illustrates an example of processing two types of video images captured using cameras 2a and 2b, similar to the monitoring system 100 shown in FIG. 6. An instruction signal (camera synchronization signal) for instructing the timing of starting capture is input to each of cameras 2a and 2b. The monitoring system 1 according to this embodiment differs from the monitoring system 100 of the comparative example in that it includes an image processing device 10. The following mainly describes the differences from the monitoring system 100 of the comparative example.

[0039] As shown in FIG. 8, the image processing device 10 according to this embodiment is provided between the cameras (2a, 2b) and the converters (41a, 41b). The image processing device 10 and the cameras (2a, 2b), and the image processing device 10 and the converters (41a, 41b), are connected using high-speed transmission cables conforming to the 12G-SDI standard, which is capable of transmitting video images with 4K or higher image quality. The image processing device 10 receives utility pole detection signals and position information detected by utility pole detection sensors (3a, 3b). The position information is, for example, distance information based on the travel distance of a vehicle 50 equipped with the camera 2 from a reference point. An example of such position information is kilometers measured from the starting point where the vehicle began traveling. The position information is output, for example, from a management device 90 that manages the travel position of the vehicle 50.

[0040] The image processing device 10 of this embodiment adds a synchronization signal and position information to the visible area or invisible area of ​​the video captured by the cameras (2a, 2b) to synchronize the frames of the video captured by the cameras 2a and 2b.

[0041] The image processing device 10 acquires video images of a monitored object on the train tracks, which have been captured by cameras 2a and 2b in response to an instruction signal. As shown in FIG. 8, cameras 2a and 2b output video frames of the monitored object, which have been captured in response to an instruction signal, in the order of F1, F2, F3, F4, and F5 at a frame rate of 60 fps. The image processing device 10 acquires the video frames captured by cameras 2a and 2b in the order in which they are output. In this embodiment, camera 2a corresponds to an example of a "first imaging unit," and the video frames captured by camera 2a correspond to an example of a "first image." Camera 2b corresponds to an example of a "second imaging unit," and the video frames captured by camera 2b correspond to an example of a "second image." The image processing device 10's acquisition of the video frames captured by cameras 2a and 2b corresponds to an example of an "acquisition unit."

[0042] The image processing device 10 also acquires utility pole detection signals and position information detected by the utility pole detection sensors (3a, 3b). In this embodiment, the utility pole detection sensors (3a, 3b) correspond to an example of a "sensor." The image processing device 10 that acquires the utility pole detection signals detected by the utility pole detection sensors (3a, 3b) corresponds to an example of a "processing unit," and the image processing device 10 that acquires the position information corresponds to an example of a "position information acquisition unit."

[0043] The image processing device 10 then uses the utility pole detection signals detected by the utility pole detection sensors (3a, 3b) and the position information as time stamp information to generate a time stamp of the video frames of the cameras (2a, 2b). The image processing device 10 adds binary bit information, such as "1" when the utility pole detection signal is in an on state indicating utility pole detection and "0" when the utility pole detection signal is in an off state, to a predetermined area of ​​the video frame acquired from each camera. Assume that such bit information is added to each video frame from the cameras (2a, 2b). For example, assume that there are N video frames from each of the cameras (2a, 2b). Assume also that video frames 1 to N-1 are each assigned a "0" indicating an off state, and the Nth video frame is assigned a "1" indicating an on state. Even if a misalignment occurs between the cameras 2a and 2b in video frames 1 to N-1 of these video frames, the downstream device (image recording device 20) can recognize frame synchronization based on the bit information of "1" indicating an on state added to the Nth video frame and can restore synchronization from video frame N onwards.

[0044] Such a predetermined area can be predetermined in the captured video as an area that does not affect the monitoring of the monitored object. The same applies to the location information as for the utility pole detection signal. For example, distance information in units of kilometers can be added as binary data to a predetermined area of ​​a video frame. The image processing device 10 may also add the above-mentioned timestamp information to the management area (invisible area) of the video frame captured by each camera. In this embodiment, the utility pole detection signal corresponds to an example of a "synchronization signal," and the image processing device 10, which adds the utility pole detection signal and location information as timestamp information to the visible or invisible area of ​​the video frame, is an example of a "processing unit." Adding the utility pole detection signal and location information as timestamp information to the visible or invisible area of ​​the video frame in the image processing device 10 can be considered an example of an image processing method.

[0045] As shown in the solid-line rectangular arrow box in FIG. 8, video frames (F1 to F5) captured by camera 2a and to which timestamp information Tm has been added are output to converter 41a via a high-speed transmission cable conforming to the 12G-SDI standard. Similarly, video frames (F1 to F5) captured by camera 2b and to which timestamp information Tm has been added are output to converter 41b via the high-speed transmission cable. Note that the information shown in the solid-line rectangular arrow box (T1 to T5) represents, for example, an index number of the video frame based on the timestamp information Tm. Examples of such index numbers include time information managed by the image processing device 10 and a count value starting from the input of the timestamp information Tm. The image processing device 10 includes a counter that increments its count value at regular intervals of unit time steps such as 1 ms or 10 ms. The counter resets the count value, for example, when the timestamp information Tm is input. The image processing device 10 then adds the count value, which has been counted based on the timestamp information Tm, to each video frame. At the very least, it is necessary to know the position (time distance) of a video frame relative to the timestamp information Tm input at a predetermined interval. The count value added to each video frame allows the time distance from the timestamp information Tm of that frame to be added as index information. The image processing device 10 can add binary data indicating an index number to the visible or invisible area of ​​a video frame in the same way as the timestamp information Tm. Assigning index numbers is expected to improve time management for video frames.

[0046] In FIG. 8, it is assumed that timestamp information Tm is added to video frame F5 captured by each of cameras 2a and 2b, for example. Uncompressed video images output from image processing device 10 to converters (41a, 41b) are converted into optical signals and output to converters (42a, 42b). Although the processing timing for converting into optical signals differs between converters 41a and 41b, the converted video images are output to converters 42a and 42b at timings starting with video frame F2. Due to the difference in processing timing, converters (42a, 42b) output video frame F4 captured by camera 2b from converter 42b at the timing of video frame F3 output from converter 42a. The video images captured by each of cameras 2a and 2b are , and the video frames are output to image capture boards 21a and 21b of image recording device 20 with a delay between them. In this embodiment, converters 41a and 42a provided on the transmission path of the video captured by camera 2a correspond to an example of a "first signal converter," and converters 41b and 42b on the transmission path of the video captured by camera 2b correspond to an example of a "second signal converter." Furthermore, image capture board 21a corresponds to an example of a "first encoder," and image capture board 21b corresponds to an example of a "second encoder."

[0047] Frame shifts also occur in the image capture boards 21a and 21b of the image recording device 20 due to differences in processing performance and processing timing, such as compression encoding. However, the image recording device 20 can compress video frames F5 by synchronizing their pre-compression frame positions based on timestamp information Tm added to each video frame F5. The image recording device 20 generates image files 23a and 23b between the image capture boards 21a and 21b, where the video frame positions are synchronized and compressed based on the timestamp information Tm. In the image file 23a, for example, video frames F4 and F5 synchronized by the timestamp information Tm are compressed together with the timestamp information and index number. Similarly, in the image file 23b, video frame F5 synchronized by the timestamp information Tm is compressed together with the timestamp information and index number. The generated image files 23a and 23b are stored in the external storage device 22. In this embodiment, the image file 23a corresponds to an example of a "first image file," and the image file 23b corresponds to an example of a "second image file." Furthermore, image recording device 20, which synchronizes and compresses the positions of each pre-compression video frame based on timestamp information Tm added to video frames F5 from cameras 2a and 2b, is an example of a "recording unit." Furthermore, the process of image recording device 20 synchronizing and compressing the positions of each pre-compression video frame based on timestamp information Tm added to video frames F5 from cameras 2a and 2b can be considered an example of an image recording method.

[0048] The image playback device 30 acquires the image files 23a and 23b stored in the external storage device 22 of the image recording device 20. The image playback device 30 then decodes each acquired image file to play back the uncompressed video data captured by the cameras 2a and 2b, and displays it on a display device such as an LCD monitor. When playing back the video data from the compressed image files 23a and 23b, the image playback device 30 synchronizes the video frame positions based on the timestamp information Tm added to the video frames F5 of the cameras 2a and 2b. As shown in the solid-line speech bubble 31 in FIG. 8, the image playback device 30 can play back the decoded video frames in synchronization with the timestamp information Tm, eliminating any misalignment between the video frames captured by the cameras 2a and 2b. As shown in the dashed rectangular frame, the video frames F5 with the timestamp information Tm can be used to synchronize the video frame positions captured by the cameras 2a and 2b, allowing the playback of video images of a monitored object. In this embodiment, the image playback device 30 that acquires the image files 23a and 23b stored in the external storage device 22 of the image recording device 20 is an example of an "image acquisition unit." The image playback device 30 that decodes each of the image files 23a and 23b is an example of an "image decomposition unit." The image playback device 30 that synchronizes the video frame positions based on the timestamp information Tm added to the video frames F5 of the cameras 2a and 2b when playing back video data is an example of a "playback unit." The image playback device 30 that synchronizes the video frame positions based on the timestamp information Tm added to the video frames F5 of the cameras 2a and 2b when playing back video data can be considered an example of an "image playback method."

[0049] 9 is a diagram for explaining a form in which index numbers are added to video frames as timestamp information in the image processing device 10. In FIG. 9, the processing paths for the eight camera images are divided into two groups (#1, #2), and the processing paths for each group are In this example, two image recording devices (20#1, 20#2) record the images.

[0050] In FIG. 9, the ternary signal generator and two splitters correspond to the signal generator 80 and splitters 81a and 81b shown in FIG. 7, and input instruction signals to synchronize the timing of the start of image capture to the eight camera systems. As described in FIG. 7, camera 2#1 belonging to the system of group #1 may be composed of cameras (2a, 2b, 2g, 2h), for example, or may be composed of cameras (2a, 2c, 2e, 2h). Camera 2#2 belonging to the system of group #2 is composed of a system corresponding to the four cameras other than camera 2#1. Hereinafter, camera 2#1 will be described as being composed of cameras 1 to 4, and camera 2#2 will be described as being composed of cameras 5 to 8. Similarly, images 1 to 8 will be described as moving images captured by cameras 1 to 8, respectively. Furthermore, encoders 1 to 4 of image recording device 20#1 correspond to the four encoder boards that make up image capture board 21#1 of image recording device 20#1 in FIG. 7. Similarly, encoders 5 to 8 of image recording device 20#2 correspond to the four encoder boards that make up image capture board 21#2 of image recording device 20#2 in Fig. 7. Files 1 to 8 correspond to the eight video files that have been encoded and compressed by encoders 1 to 8, respectively.

[0051] As shown in FIG. 9, images 1 to 8, which are eight channels of video captured by cameras 1 to 8, are input to an image processing device 10 in an uncompressed state via, for example, a high-speed transmission cable conforming to the 12G-SDI standard, which is capable of transmitting video of 4k resolution or higher. The image processing device 10 has, for example, a switching device such as a relay for each of the eight channels of input video. The switching device outputs images 1 to 8 connected to an input terminal (IN) to an index addition circuit connected to an output terminal (OUT) according to the active / inactive status of an enable signal. The enable signal is, for example, a utility pole detection signal detected by utility pole detection sensors (3a, 3b). The enable signal causes the video output from the eight channels of cameras 2 to be output simultaneously (at the same timing) to the index addition circuit.

[0052] The image processing device 10 receives, for example, a binary status signal from a management device 90 installed in the vehicle 50, for assigning index numbers to the eight streams of images 1 to 8. This status signal may be, for example, a trigger signal whose status value changes when the kilometer value acquired as mileage information changes, or a trigger signal indicating a change in the minute or other unit of time information managed by the management device 90. As shown in FIG. 9 , the index assignment circuit is reset when the trigger signal changes from inactive to active, and the counter value, internal time information, etc., counted upon the reset are assigned to images 1 to 8. As described above, binary data indicating the counter value, internal time information, etc., are assigned as index numbers to the visible or invisible areas of images 1 to 8. The eight streams of images 1 to 8 to which index numbers have been assigned are output as uncompressed moving images conforming to the 12G-SDI standard to a transmission path to which the image recording device 20 is connected.

[0053] In the image recording device 20, eight systems of images 1 to 8, each assigned an index number, are acquired, and are encoded and compressed by encoders 1 to 8, respectively, and stored in the external storage device 22. In the external storage device 22, the video files (files 1 to 8) captured by the eight systems of cameras 2 are stored, for example, in the corresponding SSDs.

[0054] As described above, the image processing device 10 according to this embodiment can acquire eight channels of video images captured by the camera 2 in an uncompressed state through a high-speed transmission cable conforming to the 12G-SDI standard, which is capable of transmitting video images of 4k resolution or higher. In addition, the image processing device 10 can acquire distance information based on the travel distance of the vehicle 50 on which the camera 2 is installed from a reference point (the travel time of the vehicle The image processing device 10 can acquire, as location information, a distance (kilometers measured in kilometers from the starting point where the video was started) in units of kilometers. The image processing device 10 can then add a synchronization signal for synchronizing the eight video streams and location information to the visible or invisible area of ​​each of the eight video streams. According to this embodiment, it is possible to reduce time lags between video data captured by multiple cameras based on the synchronization signals and location information added to the visible or invisible area of ​​each of the eight video streams.

[0055] The image processing device 10 according to this embodiment acquires utility pole detection signals detected by the utility pole detection sensors (3a, 3b) and can add timestamp information using the acquired detection signals as synchronization signals to each of the eight channels of video. The image processing device 10 can also add, as a synchronization signal, any one of the following information: a count value triggered by the detection signal when counting is started, time information starting from the input of the detection signal, and an index number. The image processing device 10 can output the eight channels of video, in which the position information and synchronization signals are recorded in the visible or invisible region, to the optical signal converters that form the respective transmission paths, thereby enabling transmission to distant locations within the vehicle 50.

[0056] <Other embodiments> The above-described embodiment is merely an example, and the present disclosure may be modified as appropriate within the scope of the description without departing from the gist of the disclosure. The processes and means described in the present disclosure may be freely combined and implemented as long as no technical contradiction occurs.

[0057] Furthermore, a process described as being performed by one device may be shared and executed by multiple devices. Alternatively, a process described as being performed by different devices may be executed by a single device. In a computer system, the hardware configuration (server configuration) by which each function is realized can be flexibly changed. For example, eight channels of video output from the image processing device 10 may be input to the image recording device 20 without going through a converter. By connecting the image processing device 10 and the image recording device 20 with a 12G-SDI cable and capturing the video, it is possible to avoid differences in processing timing caused by variations in processing performance between the converters. Furthermore, each counter shown in FIG. 9 may be a single counter common to the eight channels of video.

[0058] The present disclosure can also be realized by providing a computer program implementing the functions described in the above embodiments to a computer, and having one or more processors in the computer read and execute the program. Such a computer program may be provided to the computer via a non-transitory computer-readable storage medium connectable to the computer's system bus or via a network. Non-transitory computer-readable storage media include any type of disk, such as a magnetic disk (e.g., a floppy disk, a hard disk drive (HDD), etc.) or an optical disk (e.g., a CD-ROM, a DVD disk, a Blu-ray disk, etc.). It also includes any type of medium suitable for storing electronic instructions, such as a read-only memory (ROM), a random-access memory (RAM), an EPROM, an EEPROM, a magnetic card, a flash memory, or an optical card. [Explanation of symbols]

[0059] 1,100 monitoring system 2a, 2b, 2c, 2d, 2e, 2f, 2g, 2h Camera 3a, 3b Utility pole detection sensor (sensor) 10 Image processing device 20 Image recording device 21a, 21b Image capture board 22 External storage device 30 Image reproduction device 50 vehicles

Claims

1. an acquisition unit that acquires a first image captured by the first imaging unit and a second image captured by the second imaging unit; a location information acquisition unit that acquires location information based on a travel distance from a reference point of a vehicle equipped with the first imaging unit and the second imaging unit; a processing unit that adds the acquired position information and a synchronization signal for synchronizing the first image and the second image to a visible area or an invisible area of ​​the first image and the second image; An image processing device comprising:

2. The processing unit acquires a detection signal output by a sensor that detects a support pole of an electric rail provided along a track on which the vehicle runs, and The image processing device according to claim 1 , wherein the acquired detection signal is added as the synchronization signal.

3. The image processing device according to claim 2, wherein the processing unit adds, as the synchronization signal, one of the following information: a count value whose counting was started in response to the acquired detection signal, time information starting from the input of the acquired detection signal, and an index number.

4. 2. The image processing device according to claim 1, wherein the processing unit outputs a first image in which the position information and the synchronization signal are recorded in a visible area or an invisible area to a first signal converter, and outputs a second image in which the position information and the synchronization signal are recorded in a visible area or an invisible area to a second signal converter.

5. a first encoder connected to the first signal converter; a second encoder connected to the second signal converter; a recording unit that encodes the first image acquired by the first encoder and the second image acquired by the second encoder by synchronizing frame positions between the first image and the second image in accordance with position information and a synchronization signal added to the visible or invisible area of ​​each of the first image and the second image, and records a first image file for the encoded first image and a second image file for the encoded second image in a storage unit; An image recording device comprising:

6. an image acquisition unit that acquires the encoded first image file and the encoded second image file; an image decoding unit that decodes a first image before encoding from the first image file and decodes a second image before encoding from the second image file; a playback unit that plays back the first image and the second image by synchronizing frame positions between the first image and the second image in accordance with position information and a synchronization signal added to the visible area or the invisible area of ​​the first image and the second image; An image reproduction device comprising:

7. Obtaining a first image captured by a first imaging unit and a second image captured by a second imaging unit; acquiring position information based on a travel distance from a reference point of a vehicle equipped with the first imaging unit and the second imaging unit; adding the acquired position information and a synchronization signal for synchronizing the first image and the second image to the visible or invisible areas of the first image and the second image; An image processing method comprising:

8. acquiring a first image from a first encoder coupled to a first signal converter; acquiring a second image from a second encoder coupled to a second signal converter; encoding the first image and the second image by synchronizing frame positions between the first image and the second image according to position information and a synchronization signal added to the visible area or the invisible area of ​​each of the first image and the second image, and recording a first image file for the first image after encoding and a second image file for the second image after encoding; Image capture methods, including:

9. obtaining an encoded first image file and a second image file; decoding a first image before encoding from the first image file and a second image before encoding from the second image file; playing the first image and the second image by synchronizing frame positions between the first image and the second image according to position information and a synchronization signal added to the visible or invisible areas of the first image and the second image; An image reproduction method including:

Images