System and program and the like

Abstract

To provide a system and the like that are more excellent than before.SOLUTION: A system 1 comprises: cameras 10-1 to 10-4; and a camera video recording circuit 40 that compress video imaged by the camera 10-1 to 10-4 and stores it in a storage medium 60. A second video signal conversion circuit 30 distributes video signals transmitted between the cameras 10-1 to 10-4 and the camera image recording circuit 40 to an image recognition circuit 50 for implementing an image recognition function, on the basis of the video signals. The camera video recording circuit 40 records the video of the cameras 10-1 to 10-4 in the storage medium 60, when receiving a trigger signal TG depending on a result of recognition by the image recognition function.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to systems, programs, and the like.

Background Art

[0002] Conventionally, a system equipped with a camera and recording the video of the camera has been known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For example, in a system having a function of recording the video of such a camera, there have been various problems. One of the objects of the present invention is to provide a system or the like that is superior to the prior art.

[0005] The object of the invention of the present application is not limited to this, and the applicant also has the intention of obtaining rights by divisional application, amendment, etc. for a configuration that aims to obtain an effect resulting from a part of the configuration disclosed in this specification and drawings. For example, the problems obtained by reading the parts described as "can" in this specification as "is a problem" are disclosed in this specification. The problems are described as independent ones, and the applicant also has the intention of obtaining rights by divisional application, amendment, etc. alone for the configuration for solving each problem. Even if the problems are implicitly grasped from the description of the specification, the applicant has the intention of making a part of the configuration described in this specification the scope of claims by amendment or divisional application. In addition, the applicant also discloses a configuration for solving the problems obtained by combining these independent problems and has the intention of obtaining rights.

Means for Solving the Problems

[0006] The present invention can be understood as an invention in the following aspects, for example. (1) A system comprising a camera and a camera video recording circuit that compresses and stores the video captured by the camera in a storage means, the system being provided with signal distribution means for distributing a signal transmitted between the camera and the camera video recording circuit to an electronic component that realizes a predetermined function based on the signal.

[0007] In this way, the video captured by the camera can be compressed and stored in the storage means, and in the electronic component that realizes a predetermined function, a predetermined function can be realized based on the distributed signal between the camera and the camera video recording circuit.

[0008] The camera can be various types of cameras, but can be a CCD (Charge Coupled Device) camera, and particularly preferably a CMOS (Complementary Metal - Oxide - Semiconductor) camera. The camera can also be a camera for capturing still images, but particularly preferably a camera for capturing moving images. The camera can be a special camera such as an infrared camera, but preferably a camera that captures visible light, and particularly preferably a visible light camera. The camera may be a camera that obtains a monochrome video, but preferably a camera that obtains a color video. The camera is preferably, for example, a camera in which color filters of red (R), green (G), and blue (B) are arranged in a plane. The resolution of the camera is preferably VGA or higher.

[0009] The camera may refer to, for example, a part including an imaging element, or may refer to the imaging element itself.

[0010] In particular, the camera is preferably a camera installed in a vehicle. In particular, the camera is preferably a camera retrofitted after the vehicle manufacturer ships the vehicle.

[0011] The video compression unit provided in the camera video recording circuit can use various algorithms, but it is preferably to use an encoder with a standardized algorithm, and in particular, it is preferably to include at least one of an H.264 encoder or an H.265 encoder. In particular, the encoder of the video provided in the camera video recording circuit is preferably not a software process but a hardware configuration. The encoder is preferably a SoC (System-on-a-Chip) rather than a simple FPGA (Field-Programmable Gate Array). Thus, the camera video recording circuit may be a camera video recording chip. The camera video recording circuit is preferably configured with a hard macro rather than a soft macro.

[0012] It is particularly preferable that the camera video recording circuit is sold as a dedicated chip for camera video recording. The camera video recording circuit is preferably an LSI (Large Scale Integrated circuit) for camera video recording such as an ASSP for camera video recording or a SoC for camera video recording, and in particular, the camera video recording circuit is preferably a SoC for a drive recorder.

[0013] The storage means is preferably a removable storage means, and particularly preferably a card-shaped medium such as an SD (registered trademark) card or a micro SD (registered trademark) card.

[0014] The signal transmitted between the camera and the camera video recording circuit may be a control signal in particular, but it is preferably to include at least one of a video signal or a video synchronization signal. The video synchronization signal is preferably at least one of a pixel clock, a vertical synchronization signal, or a horizontal synchronization signal, and particularly preferably at least two of them. In particular, it is preferably a pixel clock and a vertical synchronization signal.

[0015] The video signal for transmitting the camera image may, for example, serially transmit the images synchronized in video channel units, or may transmit them in parallel. Alternatively, it may be transmitted in a combination of serial and parallel. In particular, it may be superimposed with data such as the time, event information, and sensor information at that time on a per-frame basis of the video, or superimposed on the video. It may be superimposed with data such as the time, event information, and sensor information at that time on a synchronized unit, or superimposed on the video.

[0016] When the video signal is to be distributed, for example, a video signal of a predetermined protocol may be input, and it may be output by distributing it to a plurality of output destinations with the same protocol as the input protocol. For example, in the case of a camera that outputs an MIPI CSI-2 signal as the video signal, the signal distribution means may be an MIPI CSI-2 signal splitter chip.

[0017] When the video signal is to be distributed, for example, a video signal of a predetermined protocol may be input, and it may be output with a protocol different from the input protocol but with the same protocol signal for a plurality of output destinations. For example, in the case of a camera that outputs an FPD-LINK III signal as the video signal, the signal distribution means may be a circuit (for example, a dual-port output deserializing hub such as a 1-port input 2-port output) that inputs the FPD-LINK III signal and outputs it as, for example, a plurality of MIPI CSI-2 signals. In particular, the port that outputs as an MIPI CSI-2 signal may have a function of outputting a replica of a certain port to other ports. In particular, it may be a deserializing hub chip.

[0018] The electronic component for realizing a predetermined function may particularly include at least one of another camera or an image recognition circuit separate from the camera. When the electronic component for realizing a predetermined function includes a camera, the predetermined function may be a function of imaging another area including a range different from that of the camera. When the electronic component for realizing a predetermined function includes an image recognition circuit, the predetermined function may be an image recognition function.

[0019] The signal distribution means may be, for example, simply a wiring splitter. However, it is preferably provided with a function of distributing signals in such a way that synchronization of signals transmitted between a plurality of distributed signal lines can be achieved. In particular, it is preferably a circuit (for example, a chip) provided with a function of distributing signals in such a way that synchronization of signals transmitted between a plurality of distributed signal lines can be achieved.

[0020] The camera and the camera video recording circuit may be provided in the same housing. However, they may be provided in separate housings, and the two housings may be configured to be connected by connection means such as cables and connectors. The camera and the camera video recording circuit may be configured to be connected via a cable of 1 meter or more. In particular, the camera and the camera video recording circuit may be connected via a cable routed inside a vehicle.

[0021] (2) The electronic component has another camera different from the camera, the signal includes a video synchronization signal generated by the synchronization signal generation means, and based on the distributed synchronization signal, the video signals of each of the camera and the other camera are output to the camera video recording circuit.

[0022] In this way, a synchronized video signal can be obtained between the camera and another camera on the camera video recording circuit side. For example, a video synchronized between a camera and another camera can be input to the camera video recording circuit, and the video based on the input synchronized signals of the plurality of videos can be compressed and recorded. For example, when the recorded signals of the plurality of videos are played back simultaneously, a plurality of synchronized videos can be viewed. Note that the compression of the video may be performed separately for each of the plurality of cameras in separate files, but it is preferably performed in one file.

[0023] (3) The camera has the synchronization signal generation means, and the signal distribution means is preferably configured to distribute the synchronization signal to the other camera.

[0024] By doing so, the video signal output from the other camera becomes a video signal based on the synchronization signal generated in the camera, facilitating synchronization between the two cameras. In particular, it is preferable to adopt this configuration and to configure the camera and the other camera to be provided in the same housing. This exhibits particularly excellent effects when the routing distance of the synchronization signal is short.

[0025] As for the camera, it is particularly preferable that the synchronization signal is generated in the imaging device. By doing so, it is not necessary to use an externally synchronizable imaging device that captures images in synchronization with the input synchronization signal, and a low-cost and simple configuration can be achieved.

[0026] (4) It is preferable that the synchronization signal generation means is provided outside the camera and the other camera, and the signal distribution means distributes the synchronization signal to the camera and the other camera.

[0027] By doing so, it is easy to synchronize the video signal output from the other camera with the video signal output from the other camera.

[0028] In particular, it is preferable to adopt this configuration and to configure the camera and the other camera to be provided in different housings. This exhibits particularly excellent effects when the routing distance of the synchronization signal is relatively long.

[0029] (5) It is preferable that the camera video recording circuit has the synchronization signal generation means, and the signal distribution means distributes the synchronization signal to the camera and the other camera.

[0030] By doing so, the possibility of obtaining synchronized video signals from the camera and the other camera at a timing suitable for video recording processing and the like can be increased.

[0031] (6) A first video signal conversion circuit having first video signal conversion means for converting signals output from the camera and the other camera into video signals for transmission via a cable, and a second video signal conversion circuit having second video signal conversion means for converting the video signal transmitted via the cable into a video signal that can be input to the camera video recording circuit. The synchronization signal generation means is provided in the first video signal conversion circuit or the second video signal conversion circuit, and the signal distribution means is provided in the first video signal conversion circuit or the second video signal conversion circuit, and the synchronization signal may be configured to be distributed to the camera and the other camera.

[0032] In this way, it is possible to easily input and record video signals synchronized by both cameras from the camera and the other camera to the camera video recording circuit via a cable.

[0033] The cable may be of a relatively long distance. The relatively long distance may be a distance longer than the length required for wiring within the camera unit including at least one of the camera or the other camera, and particularly preferably 1 meter or more.

[0034] The first video signal conversion means may be, for example, a serializer, and the second video signal conversion means may be, for example, a deserializer. The first video signal conversion circuit may particularly be a serializer chip, and the second video signal conversion circuit may particularly be a deserializer chip. When the camera and the other camera are in-vehicle cameras, a transmission distance between the first video signal conversion circuit and the second video signal conversion circuit may preferably use one capable of transmitting several meters.

[0035] For example, it may also include a first camera unit having the camera and a first first video signal conversion circuit, a second camera unit having the other camera and a second first video signal conversion circuit, and a recording unit having a second video signal conversion circuit and a camera video recording circuit. The first camera unit and the recording unit may be connected by a first cable, and the second camera unit and the recording unit may be connected by a second cable to transmit respective video signals. For example, a first camera unit in which the camera (first camera) attached to the front glass of a vehicle is installed to image the front of the vehicle, and a second camera unit in which the other camera (second camera) attached to the rear glass of the vehicle is installed to image the rear of the vehicle may be provided. A recording unit provided on the dashboard or under the passenger seat or the like may be connected to both camera units by separate cables. Alternatively, the recording unit and the first camera unit may be configured as one unit.

[0036] For example, it may also be configured such that a first camera unit having the camera and a first first video signal conversion circuit and a second camera unit having the other camera and a second first video signal conversion circuit are connected by a first cable, and a recording unit having a second video signal conversion circuit and a camera video recording circuit and the second camera unit are connected by a second cable to transmit respective video signals. For example, it may also be configured such that a single camera unit includes a plurality of cameras including the camera and the other camera and a first video signal conversion circuit. The first video signal conversion circuit converts the video signals of these plurality of cameras to flow through a single cable and transmits them to a recording unit having a second video signal conversion circuit and a camera video recording circuit via the single cable, so that the recording means records the videos of the plurality of cameras. For example, a camera unit attached to the front glass or the like of a vehicle, the camera unit including the camera (first camera) for imaging the front of the vehicle and the other camera (second camera) for imaging the rear of the vehicle, and a recording unit provided on the dashboard or under the passenger seat or the like may be configured to be connected by a cable.

[0037] Note that, for example, the second video signal conversion circuit may convert the video signals of a plurality of cameras that are converted by the first video signal conversion circuit and transmitted via the relatively long-distance one cable into signals for relatively short-distance transmission required for the wiring in the recording unit.

[0038] For example, the second video signal conversion circuit may be configured to output the video signals of a plurality of cameras to flow through one transmission path to the camera video recording circuit, and the camera video recording circuit may be configured to input this. For example, the second video signal conversion circuit may divide the video signals of two cameras into packets or the like of a predetermined data size, respectively, and transmit the packets or the like alternately. The video signals of a larger number of cameras may be interleaved and output to one transmission path.

[0039] Alternatively, for example, the second video signal conversion circuit may be configured to output the video signals of a plurality of cameras to flow through different transmission paths for each camera to the camera video recording circuit, and the camera video recording circuit may be configured to input these. In this way, in different transmission paths for each camera or in the camera video recording circuit connected thereto, processing for each camera unit is easy. For example, it becomes relatively easy to superimpose and record or output information different from the video for each camera frame for each camera.

[0040] One cable may desirably have as few signal lines as possible, and for example, a twisted pair cable or the like may be used.

[0041] The transmission path may desirably have as few signal lines as possible, and for example, several signal lines (for example, printed wiring on a substrate) or the like may be used. Between the plurality of cameras including the camera and the other camera and the first video signal conversion circuit, it may be connected via at least one of printed wiring or a flexible cable. Between the second video signal conversion circuit and the camera video recording circuit, it may be connected via at least one of printed wiring or a flexible cable.

[0042] (7) It is preferable that the lengths of the signal wirings from the signal distribution means to the camera and the other camera are substantially the same.

[0043] In this way, the possibility that the video signal from the camera and the video signal from the other camera become synchronized signals can be increased.

[0044] (8) The electronic component includes an image recognition circuit that performs a predetermined image recognition process and is provided separately from the camera video recording circuit. The signal to be transmitted includes a video signal. The image recognition circuit outputs information regarding the result of the image recognition process performed based on the distributed video signal to the camera video recording circuit. The camera video recording circuit may be configured to compress the video captured by the camera and store it in the storage means based on the information regarding the result.

[0045] In this way, it is possible to surely perform compression of the camera video and storage in the storage means, and it is also possible to perform recording based on the result of the image recognition process of the same camera video.

[0046] The camera video recording circuit is relatively inexpensive, but since it is designed specifically for video compression and storage, there is a problem that it is difficult to add functions such as image recognition (for example, little information disclosure, few remaining resources, etc.). For example, the SoC for a drive recorder is inexpensive but is designed for a drive recorder, so there is a problem that it is difficult to add functions such as image recognition (for example, little information disclosure, few remaining resources, etc.).

[0047] On the one hand, there is a problem that the image recognition circuit used for image recognition lacks highly reliable software for video compression and recording. The image recognition circuit may be, for example, a general-purpose SoC (SoC for image recognition) that can be utilized by compiling OSS (Open Source Software) or the like. In particular, it is preferably an SoC equipped with an accelerator circuit for image recognition, an SoC equipped with an accelerator circuit for deep learning, or the like. Note that, for example, there is also a problem that it is costly to mount a video encoder such as H.264 on a general-purpose SoC or an image recognition circuit that is particularly an FPGA. Further, even if a general-purpose SoC has a video encoder, especially when it is to be adopted as a drive recorder, there is a problem that the software for the drive recorder lacks high reliability, and on the other hand, it is extremely costly to develop it from scratch in-house.

[0048] For example, a MIPI CSI-2 splitter (e.g., a splitting chip) is used to distribute and input the same camera video signal to a camera video recording circuit and an image recognition circuit. The camera video recording circuit and the image recognition circuit are connected by a communication line. The image recognition circuit recognizes the state of the object in the image. When the recognition result indicates a situation where recording is required, a trigger signal is passed to the camera video recording circuit via this communication line. When the camera video recording circuit receives the trigger signal from the image recognition circuit, it may record the video signal it has received in the storage means as an event.

[0049] The communication line may transmit the trigger data as a parallel communication line or a serial communication line. For example, information on the type of the recognized object may be transmitted as this data. Alternatively, it may be a simple communication line between GPIOs (General - purpose input / output). For example, connect the GPIO output port of the SoC for image recognition to the trigger input port of the SoC for the drive recorder. When, as a result of image recognition by the SoC for image recognition, a situation where recording is necessary occurs, the signal of the GPIO output port of the SoC for image recognition is changed. When there is a change in the signal of the trigger input port of the SoC for the drive recorder, the SoC for the drive recorder may record the video for a predetermined number of seconds before and after as a trigger generation by the recording means. Further, signals from sensors connected to the camera video recording circuit may be transmitted to the image recognition circuit. For example, if the camera video recording circuit is the SoC for the drive recorder and the image recognition circuit is the SoC for image recognition, the acceleration sensor, switch, and GPS module are connected to the SoC for the drive recorder, and the information received by the SoC for the drive recorder from the acceleration sensor, switch, and GPS module is output to the SoC for image recognition in real - time. The SoC for image recognition may receive this information from the SoC for the drive recorder in real - time, use it for image recognition, and output the result to the SoC for the drive recorder. Such functions of both SoCs may be realized by executing programs for realizing such functions stored in each SoC respectively.

[0050] For example, the signals of each of the four cameras are serialized by a serializer and input to a deserialization hub. The deserialization hub outputs a serial signal in which the signals from the same four cameras are interleaved to the first port and the second port respectively. For example, the camera video recording circuit may be connected to the first port and the image recognition circuit may be connected to the second port.

[0051] For example, in an image recognition circuit, when it is detected that an intrusion has occurred beyond a prohibited sign even though the prohibited sign has been recognized, an entry prohibition trigger is output to a camera video recording circuit. When the camera video recording circuit receives the entry prohibition trigger from the image recognition circuit, it encodes and records the video for 15 seconds before and after that as an entry prohibition trigger video in a storage means.

[0052] Also, for example, when it is detected that a person has entered the vicinity of a forklift, the image recognition circuit outputs a surrounding person recognition trigger to the camera video recording circuit. For example, when the camera video recording circuit receives the surrounding person recognition trigger from the image recognition circuit, it encodes and records the video for 30 seconds before and after that as a surrounding person recognition trigger video in a storage means.

[0053] For example, signals of four interleaved cameras are separated into videos of the four cameras in each of the image recognition circuit and the camera video recording circuit. The image recognition circuit may perform a process of determining whether to output a trigger from any one of the videos of the four cameras, or may be configured to output a trigger when a state where a trigger is required is recognized in any of the videos. Alternatively, a process of comprehensively determining whether to apply a trigger may be performed. It is preferable to make it possible to set which of these is adopted.

[0054] In particular, it is preferable to provide a synchronization signal to a deserialization hub, a camera video recording circuit, and an image recognition circuit to achieve synchronization. The synchronization signal may be a pixel clock or a vertical synchronization signal. The deserialization hub may synchronize and output a clock to four serializers. Alternatively, the deserialization hub may generate a clock and provide the clock to the camera video recording circuit and the image recognition circuit to achieve synchronization. The image recognition circuit may be an FPGA instead of an SoC, or may be various other computers.

[0055] In the above example, although an example in which the MIPI CSI-2 branching function is present in the deserialization hub is described, the brancher may be an external device.

[0056] The function for setting the camera (and / or the other camera), for example, the part including the camera control part among the functions of the ISP (Image Signal Processor), may be configured to use, for example, the one on the camera video recording circuit side (Pattern 1), the one on the image recognition circuit side (Pattern 2), or a configuration that switches and uses both sides (Pattern 3). Also, it may be either a configuration using an SoC with built-in ISP or a configuration using an external ISP. The installation position of the external ISP may be either on the camera video recording circuit side. However, since the picture creation required for image recognition is different from the picture creation that can be compressed and recorded on the recording means and played back, etc., the problem is how to control the camera, such as the shutter speed, frame rate, exposure time, etc. In this regard, it is particularly advisable to do as in (9).

[0057] (9) The camera video recording circuit may set the camera.

[0058] By doing so, it is possible to prevent conflicts in settings and relatively well achieve both the recording of camera video and image recognition based on the camera video. For example, the inventors have found that a configuration adopting the above-mentioned (Pattern 1) exhibits excellent effects. In many cases, the images used for image recognition have lower requirements for the quality of the images required for recording. In particular, it is advisable to have a function for recognizing images by deep learning. In such a configuration, generally, the requirements for the number of pixels, frame rate, etc. can be low. It is particularly good if it is within the camera video recording circuit as the camera video recording circuit side.

[0059] However, regarding the functions of the ISP related to focus and exposure, it may be better to set the camera according to the requirements necessary for image processing on the image recognition circuit side. Therefore, it is advisable to have a system that sets the content of the camera settings on the camera video recording circuit side in advance, taking into account the settings related to exposure and focus that are necessary on the image recognition circuit side. For example, taking into account the settings related to exposure and focus that are necessary on the image recognition circuit side in advance, and adjusting within a range where the camera video does not become too bad, it is advisable to set the ISP (for example, the ISP in the camera video recording circuit) on the camera video recording circuit side.

[0060] (10) It is advisable to coordinately adjust the settings of the camera between the camera video recording circuit and the image recognition circuit.

[0061] In this way, it becomes easier to achieve both the compression and recording of camera video and the improvement of the accuracy of image recognition processing.

[0062] The image recognition circuit and the camera video recording circuit may communicate with each other and have a function of mutually grasping the state of camera control.

[0063] For example, instead of setting the camera settings in advance, or while setting in advance, the image recognition circuit side generates an adjustment request signal for exposure and focus and outputs it to the camera video recording circuit. When the camera video recording circuit receives this, it is advisable to have a system that performs the setting by the setting function on the camera video recording circuit side. For example, instead of setting the camera settings in advance, or while setting in advance, the image recognition circuit side outputs an adjustment request signal for exposure and focus to the camera video recording circuit, and when the camera video recording circuit receives this, it is advisable to adjust the settings of the ISP on the camera video recording circuit side accordingly. Note that it is particularly advisable to configure the information related to the camera settings set on the camera video recording circuit side to be transmitted to the image recognition circuit and used by the image recognition circuit in the image recognition process.

[0064] For example, when performing label recognition with an image recognition circuit, it is advisable to calculate the area corresponding to the road shoulder in the image as the priority area for exposure calculation. Basically, at this time, the control of the camera is performed from the camera video recording circuit side. However, in advance, the exposure priority of this area should be set to be slightly higher. When the exposure of the label area is insufficient in the image recognition circuit, an exposure correction signal should be output from the image recognition circuit to the camera video recording circuit. When the camera video recording circuit receives this signal, it is advisable to output a signal to increase the exposure of the camera to the camera side, and so on.

[0065] In this way, the control entity of the camera itself should basically be the camera video recording circuit side, and it is advisable to configure it so that the image recognition circuit side does not directly control the camera. Also, an adjustment signal should be output from the image recognition circuit side to the camera video recording circuit side, and the configuration should be such that the image recognition circuit side that receives this adjustment signal indirectly controls the camera. For example, the control entity of the camera itself should basically be the camera video recording circuit, and it is advisable to configure the image recognition circuit not to directly control the camera. Also, it is advisable to configure it for indirect control.

[0066] Regarding the focus setting, it is advisable to make it unnecessary by setting the camera to pan focus.

[0067] As another example, as described above (Pattern 3), for example, the control entity of the camera may be switched in a time-division manner. In particular, the recording frame rate of the video should be set to a frame rate sufficient for video recording and playback (for example, a frame rate at a level that cannot be captured by the human visual system (for example, about 24 to 30 fps)), while the frame rate of the camera should be set to a higher frame rate (for example, 120 fps). When the frames output by the camera are sequentially numbered as "1", "2", "3", "4", "1", "2", "3", "4", ···, it is advisable to make the ones for "1" be from the camera video recording circuit side and the ones for "3" be from the image recognition circuit side. For example, "2" and "4" can be used as the switching times.

[0068] The switching may be performed, for example, by physically switching both the ISP on the camera video recording circuit side and the ISP on the image recognition circuit side. However, ultimately, the ISP should switch the settings of the ISP on the camera video recording circuit side in a time-division manner. All of the multiple cameras should be controlled by the camera video recording circuit side, and the image recognition circuit side should not directly control any of the multiple cameras. It may be indirectly controlled.

[0069] Among the multiple cameras, the cameras indirectly controlled by the image recognition circuit should be some of them. These some cameras should be those with a large divergence between the requirements necessary for recognition and the requirements necessary for image recording / playback (for humans to view the video).

[0070] There may be one camera. For example, a MIPI CSI-2 splitter may be used to split the camera signal into two.

[0071] (11) It is provided with recognition video signal generation means for generating a signal with a reduced amount of information compared to the video signal input to the camera video recording circuit, and the image recognition circuit may be configured to perform the image recognition process based on the video signal with the reduced amount of information.

[0072] In this way, the amount of information processed by the image recognition circuit can be reduced in advance, the recognition process can be performed at a lower cost and at high speed, and the video can be compressed with excellent image quality and recorded on the recording means.

[0073] For example, the signal from the camera is not branched and is input to the camera video recording circuit. The camera video recording circuit may be configured to reduce the amount of information, such as by thinning out the video signal from the camera, and output it from another port such as a USB port to the image recognition circuit. The thinning out of the image may be, for example, a process such as thinning out pixels or thinning out frames. Generally, in image recognition, especially in the case of deep learning, the amount of information of the image necessary for recognition may be much smaller compared to when a human checks the video.

[0074] In this example, the decimation is to be performed by the camera video recording circuit. However, there is a possibility that the load on the camera video recording circuit may increase or that resources may be insufficient. Also, there is a possibility that the camera video recording circuit may not be accessible in the first place. Therefore, it is advisable to branch the video signal from the camera, and while one of the branched video signals is directly input to the video recording circuit, the decimated video signal is input to the image recognition circuit. For example, this function may be configured using an FPGA. Alternatively, instead of an FPGA, a single ISP chip may be used, and the output destination may be switched for each frame using the output port switching function of the ISP chip.

[0075] (12) The image recognition circuit may record information based on the video signal distributed from the signal distribution means in the second recording means.

[0076] In this way, information based on the video signal distributed from the signal distribution means is recorded in the second recording means.

[0077] The second recording means is preferably a recording means different from the first recording means. In particular, it is preferably a removable recording medium different from the first recording means. As the information based on the video signal distributed from the distribution means, it is preferably information obtained by processing the video signal distributed and input to the image recognition circuit. For example, it is advisable to record video information that surrounds the recognized object and superimposes its attributes as character information. Also, together with or instead of such video information, for example, the trigger signal sent to the camera video recording circuit may be recorded together with the time. For example, information regarding the coordinates and type of the recognized object may be recorded in association with the time.

[0078] The inventions described in (1) to (12) above can be arbitrarily combined. For example, at least a part of the configuration of at least one of the inventions from (2) and later may be added to all or a part of the configuration of the invention described in (1). In particular, it is preferable to use an invention in which at least a part of the configuration of at least one of the inventions from (2) and later is added to the invention described in (1). In particular, it is preferable to have a configuration that includes both the configuration of (2) and the configuration of (8). Also, any configuration may be extracted from the inventions described in (1) to (12), and the extracted configurations may be combined. The applicant of this application intends to obtain rights for inventions including these configurations. Also, even if there is a description such as "in the case of ~" or "when ~", it is not described as a configuration limited to that case or that time. These show examples of better configurations, and the applicant also intends to obtain rights for configurations other than these cases and times. Also, the order in the descriptions with an order is not limited to this order. The applicant also discloses configurations in which some parts are deleted or the order is changed, and intends to obtain rights for them.

Advantages of the Invention

[0079] According to the present invention, a system or the like superior to the conventional ones can be provided.

[0080] The effects of the invention of this application are not limited to this, and the effects resulting from the parts of the configuration disclosed in this specification and the drawings and the like are also disclosed, and the applicant also intends to obtain rights for the configurations that exhibit the effects by divisional applications, amendments, etc. For example, the parts described as "can ~" in this specification are descriptions that clearly show the effects, and even if there is no description of "can ~", there are parts that show the effects. Also, even if there is no such description, there are effects grasped by the said configuration.

Brief Description of the Drawings

[0081]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Mode for Carrying Out the Invention

[0082] Hereinafter, embodiments will be described in detail with reference to the drawings. The following embodiments are examples of the embodiments of the present disclosure, and the present disclosure is not limited to these embodiments. In the drawings referred to in the present embodiment, the same parts or parts having the same function are given the same reference numerals or similar reference numerals (reference numerals with only A, B, etc. appended after the numbers), and the repeated description thereof may be omitted. In addition, in each of the drawings referred to in the following description, in order to make each member, each region, etc. recognizable in size, the scale may be different from the actual one.

[0083] FIG. 1 is a block diagram showing the configuration of a system 1 according to an embodiment. The system 1 is preferably an in-vehicle device mounted on a vehicle, and more preferably an in-vehicle device that functions as a drive recorder. The vehicle is preferably a forklift, but may also be a work vehicle other than a forklift or a vehicle other than a worker (for example, a private or public passenger car, a bus or a train).

[0084] Hereinafter, first, the transmission of a video signal in the system 1 and the processing using the video signal will be described, and then the transmission of a synchronization signal and the processing using the synchronization signal will be described.

[0085] System 1 includes cameras 10-1 to 10-4, first video signal conversion circuits 20-1 to 20-4, a second video signal conversion circuit 30, a camera video recording circuit 40, an image recognition circuit 50, a storage medium 60, an information input unit 70, and a display unit 80. Hereinafter, when there is no need to particularly distinguish each of cameras 10-1 to 10-4, they may be collectively referred to as "camera 10", and when there is no need to particularly distinguish each of first video signal conversion circuits 20-1 to 20-4, they may be collectively referred to as "first video signal conversion circuit 20". In this embodiment, there are four cameras 10 and four first video signal conversion circuits 20 each, which handle video signals of four video channels. However, cameras 10 and first video signal conversion circuits 20 may each be three or less or five or more, and may handle video signals of three or less or five or more video channels. Note that each circuit of the above-described system 1 is realized by, for example, physically separated circuits (e.g., chips), but may also be realized by circuits (e.g., the same chip) in which two or more circuits are fabricated on the same substrate.

[0086] Camera 10 is an electronic component that captures an image and generates a video signal indicating the captured video. Camera 10 can be various types of cameras, can be a CCD camera, and particularly preferably a CMOS camera. Camera 10 can also be a camera for capturing still images, but particularly preferably a camera for capturing moving images. Camera 10 can be a special camera such as an infrared camera, but is preferably a camera that captures visible light (e.g., light in a wavelength range of 400 to 700 nm), and particularly preferably a visible light camera. Camera 10 may be a camera that obtains a monochrome video, but is preferably a camera that obtains a color video. Camera 10 is preferably, for example, a camera in which color filters of red (R), green (G), and blue (B) are arranged in a plane. The resolution of the camera is preferably VGA or higher. Camera 10 may refer to, for example, a portion including an image sensor, or may refer to the image sensor itself. Particularly, camera 10 is preferably a camera installed in a vehicle. Particularly, camera 10 is preferably a camera retrofitted after the vehicle manufacturer ships the vehicle.

[0087] In this embodiment, the video signal generated and output by the camera 10 is data in digital format including the gradation values of each pixel of a plurality of pixels. The video signal preferably includes 8-bit (256 gradations) data for each of the color components of red (R), green (G), and blue (B), but the color components and the number of gradations are not limited to this.

[0088] The camera 10-1 is communicably connected to the first video signal conversion circuit 20-1. The camera 10-2 is communicably connected to the first video signal conversion circuit 20-2. The camera 10-3 is communicably connected to the first video signal conversion circuit 20-3. The camera 10-4 is communicably connected to the first video signal conversion circuit 20-4. Each of the cameras 10-1 to 10-4 and each of the first video signal conversion circuits 20-1 to 20-1 may be connected by either a wired or wireless transmission path, but it is preferable to be connected using a wired transmission path. In the example of FIG. 1, the camera 10-1 outputs video signals A1, A2, A3, A4 in chronological order. The camera 10-2 outputs video signals B1, B2, B3 in chronological order. The camera 10-3 outputs video signals C1, C2, C3, C4 in chronological order. The camera 10-4 outputs video signals D1, D2 in chronological order. In the cameras 10-1 to 10-4, the video signals with smaller numbers attached after "A" to "D" are output in order. Also, it is assumed that the video signals with the same number are output from the cameras 10-1 to 10-4 at the same time. For example, the video signals A1, B1, C1, D1 are output at the same time. Each of the video signals A1 to A4, B1 to B3, C1 to C4, D1, D2 is a video signal of a predetermined unit, and for example, it is preferable to be a video signal indicating one frame (also referred to as one frame) of video.

[0089] Cameras 10-1 to 10-4 may each have a different imaging area. The area imaged by camera 10 may include, for example, one or both of the inside and outside of the vehicle. The outside area may be any one of the front (front face), right front diagonal, right side (true side), right rear diagonal, rear (true rear), left rear diagonal, left side (true side), and left front diagonal of the vehicle, but other areas may also be used. Cameras 10-1 to 10-4 may include, for example, at least a camera (first camera) attached to the windshield of the vehicle and a camera (second camera) installed to image the rear of the vehicle.

[0090] The first video signal conversion circuit 20 includes first video signal conversion means for receiving an input of a video signal from camera 10 and converting it into a video signal in serial format. The first video signal conversion circuit 20 may be a serializer, particularly a serializer chip. The video signal in serial format is a signal obtained by converting the video signal input from camera 10 into a serial data stream. The first video signal conversion circuit 20-1 converts the video signal from camera 10-1, the first video signal conversion circuit 20-2 converts the video signal from camera 10-2, the first video signal conversion circuit 20-3 converts the video signal from camera 10-3, and the first video signal conversion circuit 20-4 converts the video signal from camera 10-4. The first video signal conversion circuits 20-1 to 20-4 output the interleaved converted video signals to the second video signal conversion circuit 30. Each of the first video signal conversion circuits 20-1 to 20-4 and the second video signal conversion circuit 30 may be connected either by wire or wirelessly, but it is preferable to be connected using a single wired cable. The first video signal conversion circuits 20-1 to 20-4 convert it into a video signal for transmission via such a cable. This cable may be a relatively long-distance cable. The relatively long distance may be a distance longer than the length required for wiring within the camera unit including at least one of camera 10 or another camera 10, and particularly preferably 1 meter or more.

[0091] When the camera 10 is an in-vehicle camera, it is preferable to use a signal transmission cable with a transmission distance of several meters between the first video signal conversion circuit 20 and the second video signal conversion circuit 30.

[0092] The second video signal conversion circuit 30 includes second video signal conversion means for converting the video signals of the cameras 10-1 to 10-4 transmitted from each of the first video signal conversion circuits 20-1 to 20-4 via cables into video signals that can be input to the camera video recording circuit 40. The second video signal conversion circuit 30 is preferably a deserialiser, particularly a deserialiser chip. Further, the second video signal conversion circuit 30 also functions as signal distribution means for distributing the video signals transmitted between the cameras 10-1 to 10-4 and the camera video recording circuit 40 to the image recognition circuit 50. For example, the second video signal conversion circuit 30 may have an MIPI CSI-2 branching function. In this way, the video captured by the camera 10 can be compressed and stored in the storage medium 60, and the image recognition function can be realised in the image recognition circuit 50 based on the distributed signals between the camera 10 and the camera video recording circuit 40.

[0093] The second video signal conversion circuit 30 has a first port 31 and a second port 32. The camera video recording circuit 40 is communicably connected to the first port 31, and the image recognition circuit 50 is communicably connected to the second port 32. Such a second video signal conversion circuit 30 can also be specified as a deserialiser hub.

[0094] The second video signal conversion circuit 30 may be connected to each of the camera video recording circuit 40 and the image recognition circuit 50 either by wire or wirelessly, but it is preferably connected using a wired cable. The second video signal conversion circuit 30 outputs the video signal S1 to the camera video recording circuit 40 and the video signal S2 to the image recognition circuit 50 based on the interleaved video signals from the cameras 10-1 to 10-4. The second video signal conversion circuit 30 distributes the video signals from the cameras 10-1 to 10-4 by outputting the same image signals S1 and S2 from each of the first port 31 and the second port 32 at the same timing. The second video signal conversion circuit 30 inputs, for example, a video signal of a predetermined protocol and distributes and outputs it to a plurality of output destinations in the same protocol as the input protocol. For example, if the camera 10 outputs a MIPI CSI-2 signal as the video signal, the second video signal conversion circuit 30 is preferably a MIPI CSI-2 signal splitter chip. The second video signal conversion circuit 30 may input, for example, a video signal of a predetermined protocol and output signals of the same protocol for a plurality of output destinations in a protocol different from the input protocol. For example, if the camera 10 outputs an FPD-LINK III signal as the video signal, the second video signal conversion circuit 30 is preferably a chip (for example, a dual-port output deserializing hub with 1-port input and 2-port output) having a port for inputting the FPD-LINK III signal and outputting it as, for example, a plurality of MIPI CSI-2 signals. In particular, the port that outputs as a MIPI CSI-2 signal preferably has a function of outputting a replica of a certain port to other ports. In particular, it is preferably a deserializing hub chip.

[0095] In system 1, one cable preferably has as few signal lines as possible, for example, a twisted pair cable may be used. As the transmission line, it is preferable to use as few signal lines as possible, for example, several signal lines (for example, printed wiring on a substrate) may be used. Between the camera 10 and a plurality of cameras 10 including another camera 10 different from this and the first video signal conversion circuit 20, it is preferable to connect via at least one of printed wiring or a flexible cable. Between the second video signal conversion circuit 30 and the camera video recording circuit 40, it is preferable to connect via at least one of printed wiring or a flexible cable.

[0096] The second video signal conversion circuit 30 selects the video signals of the cameras 10-1, 10-2, 10-3, 10-4 in a predetermined order, and outputs the selected video signals from each of the first port 31 and the second port 32. Here, in the example of FIG. 1, the second video signal conversion circuit 30 outputs the video signals S1 and S2 by switching the video signals in units of one frame in the order of cameras 10-1, 10-2, 10-3, 10-4, 10-1, 10-2, 10-3, 10-4 ···. Specifically, the second video signal conversion circuit 30 outputs the video signals S1 and S2 from each of the first port 31 and the second port 32 in the order of video signals A1, B1, C1, D1, A2, B2, C2, D2, A3, B3, C3, A4, C4. Note that the 30 does not necessarily have a function of distributing video signals (MIPI CSI-2 branching function), and in this case, an external distributor may be used.

[0097] The camera video recording circuit 40 has a function of compressing the video signal S1 from the first port 31 and storing it in the storage medium 60. The video compression unit (not shown) provided in the camera video recording circuit 40 can use various algorithms, but it is preferably to use an encoder with a standardized algorithm, and in particular, it is preferably to include at least one of an H.264 encoder or an H.265 encoder. In particular, the encoder of the video provided in the camera video recording circuit 40 is preferably configured by hardware rather than software processing. The camera video recording circuit 40 is preferably an FPGA, but it is particularly desirable to be an SoC. Also, the camera video recording circuit 40 is preferably configured by a hard macro rather than a soft macro.

[0098] It is particularly good if the camera video recording circuit 40 is sold as a dedicated chip for camera video recording. The camera video recording circuit 40 is preferably an LSI for camera video recording such as an ASSP for camera video recording or an SoC for camera video recording, and in particular, the camera video recording circuit is preferably an SoC for a drive recorder.

[0099] When the trigger signal TG is input from the image recognition circuit 50, the camera video recording circuit 40 performs event recording to store the video signal for a period corresponding to the input in the storage medium 60. This period is preferably, for example, 30 seconds before and after the event occurs, but other periods may also be used.

[0100] The image recognition circuit 50 functions as an electronic component that realizes a predetermined function using the signal transmitted between the camera 10 and the camera video recording circuit 40 distributed by the second video signal conversion circuit 30. The image recognition circuit 50 recognizes the video indicated by the video signal S2 from the second port 32, and determines whether an event that triggers event recording has occurred. When the image recognition circuit 50 determines that the event has occurred, it outputs a trigger signal TG that triggers the start of event recording to the camera video recording circuit 40. For example, when the image recognition circuit 50 recognizes the state of an object in the video and, as a result, determines that event recording is required, it outputs the trigger signal TG to the camera video recording circuit 40. In this way, the image recognition circuit 50 may comprehensively determine whether to generate the trigger signal TG from the videos of the four cameras 10-1 to 10-4.

[0101] In this way, the image recognition circuit 50 outputs information regarding the result of the image recognition process performed based on the video signal S2 to the camera video recording circuit 40. The image recognition circuit 50 may output information indicating the position of the recognized object in the video to the camera video recording circuit 40. Based on the information regarding the result of the image recognition process, the camera video recording circuit 40 compresses the video indicated by the video signal S1 and stores it in the storage medium 60. In this way, it is possible to reliably compress and store the video of the camera 10 in the storage medium 60, and it is also possible to perform recording based on the result of the image recognition process of the video of the same camera 10.

[0102] Here, an example of event recording will be described. For example, when the image recognition circuit 50 recognizes a no-entry sign based on a video but intrusion is detected beyond the sign, the image recognition circuit 50 outputs a trigger signal TG1 indicating a no-entry trigger to the camera video recording circuit 40. When the camera video recording circuit 40 receives the input of the trigger signal TG1, it encodes and records a video signal showing the video for 15 seconds before and after that as a no-entry trigger video on the storage medium 60. For example, when the image recognition circuit 50 recognizes that a person has intruded into the vicinity of a forklift, the image recognition circuit 50 outputs a trigger signal TG2 indicating a surrounding person recognition trigger to the camera video recording circuit 40. When the camera video recording circuit 40 receives the input of the trigger signal TG2, it encodes and records a video signal showing the video for 30 seconds before and after that as a surrounding person recognition trigger video on the storage medium 60. Here, only the video of the video channel that triggered the output of the trigger signal TG1 or TG2 may be recorded, or the videos of all video channels may be recorded. Note that various modifications are conceivable for the event that triggers the output of the trigger signal and the period for which event recording is targeted. For example, event recording may be performed when the information input via the information input unit 70 satisfies a predetermined condition.

[0103] For example, the same camera video signal is distributed and input to the camera video recording circuit 40 and the image recognition circuit 50 using an MIPI CSI-2 splitter (for example, a splitting chip). The camera video recording circuit 40 and the image recognition circuit 50 are connected by a communication line. The image recognition circuit 50 performs image recognition on the state of the object. When the recognition result indicates a situation where recording is necessary, the image recognition circuit 50 passes a trigger signal to the camera video recording circuit via this communication line. When the camera video recording circuit receives the trigger signal from the image recognition circuit, it may event-record the video signal it has received in the storage means.

[0104] The communication line may transmit the trigger signal TG and other data as a parallel communication line or a serial communication line. For example, information on the type of the recognized object may be transmitted as this data. Alternatively, it may be a simple communication line between GPIOs. For example, the GPIO output port of the SoC for image recognition is connected to the trigger input port of the SoC for the drive recorder. When a situation where recording is necessary occurs as a result of image recognition in the SoC for image recognition, the signal of the GPIO output port of the SoC for image recognition is changed. When there is a change in the signal of the trigger input port of the SoC for the drive recorder, the SoC for the drive recorder may record the video for a predetermined number of seconds before and after as a trigger generation to the recording means. Further, signals from sensors connected to the camera video recording circuit 40 may be transmitted to the image recognition circuit 50. For example, if the camera video recording circuit 40 is the SoC for the drive recorder and the image recognition circuit 50 is the SoC for image recognition, the acceleration sensor, switch, and GPS module are connected to the SoC for the drive recorder, and the information received by the SoC for the drive recorder from the acceleration sensor, switch, and GPS module is output to the SoC for image recognition in real time. The SoC for image recognition may receive this information from the SoC for the drive recorder in real time, use it for image recognition, and output the result to the SoC for the drive recorder. Such functions of both SoCs may be realized by executing programs for realizing such functions stored in each SoC respectively.

[0105] The image recognition circuit 50 may record information based on the video signal distributed from the second video signal conversion circuit 30 in the second recording means. In this way, information based on the video signal distributed from the second video signal conversion circuit 30 is recorded in the second recording means. The second recording means may be a recording means different from the first recording means. In particular, it may be a removable recording medium different from the first recording means. If the first recording means is the storage medium 60, the second recording means is a recording means different from the storage medium 60. As the information based on the distributed video signal, it is preferable to use information obtained by processing the video signal distributed and input to the image recognition circuit 50. For example, it is preferable to record video information that surrounds the recognized object and superimposes its attributes as character information. Also, together with such video information, or instead of such video information, for example, the trigger signal TG output to the camera video recording circuit 40 may be recorded together with the time. For example, information regarding the coordinates and type of the recognized object may be recorded in association with the time.

[0106] In addition to the function of performing event recording, the camera video recording circuit 40 may have a function of performing continuous recording. In this case, the camera video recording circuit 40 records video in a predetermined storage area secured in the storage medium 60, and when the storage area is full, it records the video by overwriting in order from the oldest video. These storage areas may be separated or overwriting may be prohibited using an overwrite prohibition flag so that the video related to event recording is not lost due to the continuous recording function.

[0107] The storage medium 60 functions as a storage means for storing the video recorded by the camera video recording circuit 40. The storage medium 60 may be either internal storage means or external storage means of the system 1, but it is particularly preferably a removable external storage means, and particularly preferably a card-shaped medium such as an SD card or a microSD card. The video stored in the storage medium 60 can be displayed on the display unit 80 or displayed using an external terminal. The display unit 80 is, for example, a liquid crystal display or another type of display device.

[0108] The information input unit 70 receives input of information from the outside. The information input unit 70 may be provided with various switches, for example, including a switch for turning on or off the power of the system 1. The information input unit 70 may be provided with a sensor for a drive recorder. The information input unit 70 may include, for example, an acceleration sensor (for example, functioning as a G sensor) for detecting an impact that triggers event recording. Further, the information input unit 70 may be provided with a GPS (Global Positioning System) sensor for measuring the current position. Between the camera video recording circuit 40 and the image recognition circuit 50, various signals corresponding to the information input via the information input unit 70 may be exchanged.

[0109] Subsequently, the transmission of the synchronization signal and the processing using the synchronization signal will be described. The signal transmitted between the camera 10 and the camera video recording circuit 40 includes the video synchronization signal generated by the synchronization signal generation means. The system 1 may be configured to output the video signals of each of the cameras 10-1 to 10-4 to the camera video recording circuit 40 based on the synchronization signal distributed together with the video signal.

[0110] In this way, the camera video recording circuit 40 can obtain video signals synchronized with the cameras 10-1 to 10-4. For example, the camera video recording circuit 40 can input video signals synchronized among the cameras 10-1 to 10-4 and compress and record a video based on the input synchronized video signals of a plurality of cameras. For example, when the recorded video signals of a plurality of cameras are played back simultaneously, a plurality of synchronized videos can be viewed. Note that the compression of the video may be performed in separate files for each of the plurality of cameras, but it is preferable to perform it in one file.

[0111] In system 1, in order to synchronize the operations of the plurality of cameras 10, the first video signal conversion circuits 20-1 to 20-4, the camera video recording circuit 40, and the image recognition circuit 50, the synchronization signal is also distributed to the image recognition circuit 50. Thus, as the signal transmitted between the camera 10 and the camera video recording circuit 40, it may be a control signal in particular, but it is preferable that it includes at least either the video signal or the video synchronization signal. As the synchronization signal, it is preferable to include at least one of the pixel clock, the vertical synchronization signal, and the horizontal synchronization signal, and particularly preferably at least two of them. In particular, it is preferable to use the pixel clock and the vertical synchronization signal. The second video signal conversion circuit 30 may output the clock to the four first video signal conversion circuits 20-1 to 20-4 in synchronization. For the generation entity and distribution method of the synchronization signal, for example, the following (Method 1) or (Method 2) may be adopted.

[0112] (Method 1) Using one camera 10 as the parent to generate a synchronization signal and distributing it to the other cameras 10, the first video signal conversion circuits 20, the second video signal conversion circuit 30, the camera video recording circuit 40, and the image recognition circuit 50.

[0113] (Method 1) Any one of the cameras 10 has a synchronization signal generation means. And the one camera 10 as the signal distribution means distributes this synchronization signal to the other cameras 10, the first video signal conversion circuits 20, the second video signal conversion circuit 30, the camera video recording circuit 40, and the image recognition circuit 50.

[0114] In this way, the video signal output from the other camera 10 becomes a video signal based on the synchronization signal generated in one camera 10, and it becomes easier to synchronize these cameras 10. In particular, it is preferable to adopt this configuration and to configure the one camera 10 and the other camera 10 to be provided in the same housing. It exhibits an excellent effect particularly when the routing distance of the synchronization signal is short. Among the cameras 10, it is particularly preferable that it is generated in the imaging element. In this way, it is not necessary to use an externally synchronizable imaging element that captures images in synchronization with the input synchronization signal, and a low-cost and simple configuration can be achieved.

[0115] (Method 2) Any one of the first video signal conversion circuit 20, the second video signal conversion circuit 30, the camera video recording circuit 40, and the image recognition circuit 50 generates a synchronization signal and distributes it to the other elements.

[0116] (Method 2) The synchronization signal generation means is provided outside the camera 10, and the synchronization signal generated by the synchronization signal generation means is distributed to the cameras 10-1 to 10-4 and the other elements.

[0117] If (Method 2) is adopted, it is easy to synchronize the video signal output from one camera 10 with the video signal output from another camera 10. In particular, when this configuration is adopted and the camera 10 and another camera 10 are provided in different enclosures, it is particularly effective when the routing distance of the synchronization signal is relatively long.

[0118] In (Method 2), the camera video recording circuit 40 may have a synchronization signal generation means. In this case, the camera video recording circuit 40 as the signal distribution means distributes the synchronization signal to the cameras 10-1 to 10-4 and the other elements. In this way, the possibility of obtaining synchronized video signals from the camera and another camera at a timing suitable for the video recording process can be increased.

[0119] In (Method 2), the synchronization signal generation means may be provided in either the first video signal conversion circuit 20 or the second video signal conversion circuit 30. In this case, the first video signal conversion circuit 20 or the second video signal conversion circuit 30 as the signal distribution means distributes the synchronization signal to the cameras 10-1 to 10-4 and the other elements. In this way, it is possible to easily input and record the video signals synchronized by both cameras from the camera and the other camera into the camera video recording circuit via a cable.

[0120] In particular, it is advisable to synchronize by supplying a synchronization signal to the second video signal conversion circuit 30, the camera video recording circuit 40, and the image recognition circuit 50. The synchronization signal may be a pixel clock or a vertical synchronization signal. The second video signal conversion circuit 30 may synchronize and output a clock to the four first video signal conversion circuits 20-1 to 20-4. Alternatively, the second video signal conversion circuit 30 may generate a clock and supply the clock to the camera video recording circuit 40 and the image recognition circuit 50 to achieve synchronization. The image recognition circuit 50 may be an FPGA instead of an SoC, or may be various other types of computers.

[0121] The method of processing based on the synchronization signal is not limited to either (Method 1) or (Method 2). For example, the synchronization signal generation means may be provided separately from the elements of the above-described System 1.

[0122] The above signal distribution means may be, for example, simply a wiring splitter. However, it is preferably provided with a function to distribute in such a way that synchronization of the signals transmitted between the plurality of distributed signal lines can be achieved. In particular, it is preferably a circuit (for example, a chip) provided with a function to distribute in such a way that synchronization of the signals transmitted between the plurality of distributed signal lines can be achieved.

[0123] Also, the camera 10 and the camera video recording circuit 40 may be configured to be provided in the same housing. However, they may be configured to be provided in separate housings, and the two housings may be connected by connection means such as a cable or a connector. The camera 10 and the camera video recording circuit 40 may be configured to be connected via a cable of 1 meter or more. In particular, the camera 10 and the camera video recording circuit 40 are preferably connected via a cable routed inside a vehicle.

[0124] The configuration described with reference to FIG. 1 may be modified as shown in FIG. 2. FIG. 2 shows the parts of the configuration described with reference to FIG. 1 that are different from FIG. 1. In this example, the video signal from the camera 10 is input to the camera video recording circuit 40 without being distributed. The camera video recording circuit 40 supplies the video signals from the cameras 10-1 to 10-4 to the image recognition circuit 50. The camera video recording circuit 40 includes recognition video signal generation means for generating a signal with a reduced amount of information compared to the input video signal, and reduces the amount of information by thinning out the video signal, for example, and outputs it to the image recognition circuit 50 via another port such as a USB (Universal Serial Bus) port. Thinning out the video signal may be performed by processes such as thinning out pixels or frames. Generally, in image recognition, especially in the case of deep learning, the amount of information of the image required for recognition may be much smaller compared to when a human checks the video.

[0125] The configuration described with reference to FIG. 2 may be further modified as shown in FIG. 3. FIG. 3 shows the parts of the configuration described with reference to FIG. 1 that are different from FIG. 1, extracted and shown. In the example of FIG. 2, thinning out was performed by the camera video recording circuit 40, but there is a possibility that the processing load may increase or there may be a shortage of resources. Therefore, as shown in FIG. 3, an image processing unit 90 may be provided between the camera 10 and the camera video recording circuit 40 and the image recognition circuit 50. The image processing unit 90 branches the video signal from the camera, outputs the original video signal to the camera video recording circuit 40, and outputs the thinned-out video signal to the image recognition circuit 50. The image processing unit 90 is, for example, an FPGA, but may be configured using a single ISP chip. The image processing unit 90 may be configured to switch the output destination for each frame using the output port switching function of the ISP chip.

[0126] In the system 1, there may be one camera 10. For example, a splitter exemplified by an MIPI CSI-2 splitter may be used to split the signal of the camera into two. In this case, it is equivalent to a configuration in which any one of the cameras 10-1 to 10-4 described with reference to FIG. 3 is used and the image processing unit 90 is used as the splitter.

[0127] In system 1, the video signal for transmitting the video of camera 10 may, for example, serially transmit video synchronized in video channel units, or may transmit it in parallel. Alternatively, it may be transmitted in a serial and parallel combination. In particular, it is preferable to superimpose data such as the time, event information, and sensor information at that time on a per-frame basis of the video, or superimpose it on the video. It may be superimposed with data such as the time, event information, and sensor information at that time on a synchronized unit, or superimposed on the video.

[0128] Also, it is preferable that the signal wiring lengths from the signal distribution means to camera 10 and another camera 10 are substantially the same. By doing so, the possibility that the video signal from camera 10 and the video signal from another camera 10 become synchronized signals can be increased.

[0129] Next, with reference to FIGS. 4 to 7, a connection example between cameras 10-1 to 10-4 and the processing device 100 will be described. The processing device 100 is a device that performs predetermined processing based on the video signals output by cameras 10-1 to 10-4. The processing device 100 may be, for example, either the camera video recording circuit 40 or the image recognition circuit 50.

[0130] In the example of FIG. 4, cameras 10-1 to 10-4 are provided in one camera unit, and these are connected to a single first video signal conversion circuit 20. The first video signal conversion circuit 20 outputs the video signal Sa1 to the second video signal conversion circuit 30 via one cable. The first video signal conversion circuit 20 outputs the same video signal Sa2 as the video signal Sa1 to the processing device 100 via one cable. The video signal Sa2 is a signal that keeps the video signal Sa1 in serial format in serial format. The processing device 100 executes processing using the video signal Sa2 input via one cable.

[0131] In the example of FIG. 5, a plurality of cameras 10-1 to 10-4 are provided in one camera unit, and these are connected to a single first video signal conversion circuit 20. The first video signal conversion circuit 20 outputs a video signal Sb1 to the second video signal conversion circuit 30 via one cable. The video signal Sb1 is the same signal as the video signal Sa1. The second video signal conversion circuit 30 outputs video signals Sb21, Sb22, Sb23, and Sb24 using different cables for each of the cameras 10-1 to 10-4. The video signals Sb21 to Sb24 are signals obtained by converting the serial-form video signal Sb1 into a parallel form. The processing device 100 executes processing using the video signals Sb21 to Sb24 input via different cables for each camera 10.

[0132] In the example of FIG. 6, a plurality of cameras 10-1 to 10-4 are provided in one camera unit, and these are connected to different first video signal conversion circuits 20-1 to 20-4 for each camera. The first video signal conversion circuits 20-1 to 20-4 each output video signals Sc11, Sc12, Sc13, and Sc14 to the second video signal conversion circuit 30 via one cable. The second video signal conversion circuit 30 outputs a video signal Sc2 to the processing device 100 via one cable. The processing device 100 executes processing using the video signal Sc2 input via the cable. The video signal Sc2 is a signal obtained by converting the parallel-form video signals Sc1 to Sc4 into a serial form. The system 1 in FIG. 1 corresponds to this connection example.

[0133] In the example of FIG. 7, a plurality of cameras 10-1 to 10-4 are provided in one camera unit, and these are connected to different first video signal conversion circuits 20-1 to 20-4 for each camera. The system 1 in FIG. 1 corresponds to this connection example. The first video signal conversion circuits 20-1 to 20-4 each output video signals Sd11, Sd12, Sd13, Sd14 to the second video signal conversion circuit 30 via one cable. The second video signal conversion circuit 30 outputs video signals Sd21, Sd22, Sd23, Sd24 using different cables for each of the cameras 10-1 - 10-4. The video signals Sd21 to Sd24 are signals that keep the parallel video signals Sd11 to Sd14 in parallel form. The processing device 100 executes processing using the video signals Sb21 to Sb24 input via different cables for each camera 10.

[0134] Based on the configurations of FIGS. 1, 6, and 7, a system may be configured that includes a first camera unit having a camera 10 and a first video signal conversion circuit 20, a second camera unit having another camera 10 and a second first video signal conversion circuit 20, and a recording unit having a second video signal conversion circuit 30 and a camera video recording circuit 40. The first camera unit and the recording unit are connected by a first cable, and the second camera unit and the recording unit are connected by a second cable to transmit respective video signals. For example, a first camera unit in which the camera (first camera) attached to the front glass of a vehicle is installed to image the front of the vehicle, and a second camera unit in which another camera (second camera) attached to the rear glass of the vehicle is installed to image the rear of the vehicle may be provided, and a recording unit provided on the dashboard or under the passenger seat or the like may be connected to both camera units by different cables. Alternatively, the recording unit and the first camera unit may be configured as one unit.

[0135] For example, a first camera unit having a camera 10 and a first first video signal conversion circuit 20, and a second camera unit having another camera 10 and a second first video signal conversion circuit 20 may be connected by a first cable, and a recording unit having a second video signal conversion circuit 30 and a camera video recording circuit 40 and the second camera unit may be connected by a second cable so as to transmit respective video signals.

[0136] Based on the configurations of FIGS. 1 and 4 and 5, a system may be configured such that one camera unit includes a camera 10, a plurality of cameras 10 including another camera 10 different from this, and a first video signal conversion circuit 20. The first video signal conversion circuit 20 converts the video signals of these plurality of cameras 10 so as to flow through one cable, and transmits them via the one cable to a recording unit including a second video signal conversion circuit 30 and a camera video recording circuit 40 to record the videos of the plurality of cameras 10 on a storage medium 60. For example, a camera unit attached to the front glass of a vehicle or the like, the camera unit including a camera 10 (first camera) that images the front of the vehicle and another camera 10 (second camera) that images the rear of the vehicle, and a recording unit provided on the dashboard or under the passenger seat or the like may be configured to be connected by a cable.

[0137] Incidentally, for example, the second video signal conversion circuit 30 may convert the video signals of the plurality of cameras 10 that are converted by the first video signal conversion circuit 20 and transmitted via one cable having a relatively long distance into signals for transmission having a relatively short distance required for the wiring in the recording unit.

[0138] Based on the configurations of FIGS. 1 and 4, 5, for example, the second video signal conversion circuit 30 may be configured to output the video signals of the plurality of cameras 10 to flow through one cable to the camera video recording circuit 40 so that these are input to the camera video recording circuit 40. For example, the second video signal conversion circuit 30 may divide the video signals of two cameras into packets or the like of a predetermined data size respectively, and transmit the packets or the like alternately. The video signals of a larger number of cameras may be interleaved and output to one transmission path.

[0139] Alternatively, based on the configurations of FIGS. 1 and 6, 7, for example, the second video signal conversion circuit 30 may be configured to output the video signals of the plurality of cameras 10 to flow through different cables for each camera to the camera video recording circuit 40 so that these are input to the camera video recording circuit 40. In this way, within different transmission paths for each camera 10 or within the camera video recording circuit 40 connected thereto, processing for each camera 10 unit is facilitated. For example, it becomes relatively easy to superimpose and record or output information different from the video in units of camera frames for each camera 10.

[0140] The camera video recording circuit 40 is relatively inexpensive, but since it is designed specifically for video compression and storage, there is a problem that it is difficult to add functions such as image recognition (for example, there is little information disclosure, few remaining resources, etc.). For example, although the SoC for a drive recorder is inexpensive, since it is designed for a drive recorder, there is a problem that it is difficult to add functions such as image recognition (for example, there is little information disclosure, few remaining resources, etc.).

[0141] On the other hand, there is a problem that the image recognition circuit 50 used for image recognition lacks highly reliable software for video compression and recording. The image recognition circuit may be, for example, a general-purpose SoC (SoC for image recognition) that can be utilized by compiling OSS or the like. In particular, it is preferably an SoC equipped with an accelerator circuit for image recognition, an SoC equipped with an accelerator circuit for deep learning, or the like. Note that, for example, there is also a problem that it is costly to mount a video encoder such as H.264 on a general-purpose SoC or an image recognition circuit that is particularly an FPGA. Further, even if a general-purpose SoC is equipped with a video encoder, especially when it is to be adopted as a drive recorder, the software for the drive recorder lacks high reliability, and on the other hand, there is a problem that it is extremely costly to develop it from scratch in-house.

[0142] In contrast, in the system of the present embodiment, the camera video recording circuit 40 and the image recognition circuit 50 are realized by different circuits such as SoCs, and the video signal and the synchronization signal are distributed as described above, so that a system or the like superior to the conventional one can be provided.

[0143] Furthermore, the following configuration may be adopted.

[0144] (Camera control example 1) The function for setting the camera 10 (and / or another camera 10) (for example, a part including the part for controlling the camera among the functions of the ISP) may be, for example, configured to use the one on the side of the camera video recording circuit 40 (pattern 1), the one on the side of the image recognition circuit 50 (pattern 2), or a configuration that switches between and uses both sides (pattern 3). Also, it may be either a configuration using an SoC with built-in ISP or a configuration using an external ISP. As the installation position of the external ISP, it may be on either side of the camera video recording circuit 40. However, since the image drawing required for image recognition is different from the image drawing that can be compressed and recorded on the storage medium 60 and reproduced, etc., the problem is how to control the camera, such as the shutter speed, frame rate, exposure time, etc. Therefore, the camera video recording circuit 40 may have a function for setting the camera 10.

[0145] In this way, it is possible to prevent setting conflicts and relatively well balance the recording of the video of the camera 10 and the image recognition based on the video of the camera 10. For example, the inventors have found that an excellent effect can be achieved with the configuration adopting the above-mentioned (pattern 1). In many cases, the quality requirements of the images used for image recognition are lower than those required for recording. In particular, it is preferable to have a function for recognizing images by deep learning. In such a configuration, generally, the requirements for the number of pixels, frame rate, etc. can be low. It is particularly good if it is within the camera video recording circuit 40 as the camera video recording circuit 40 side.

[0146] However, regarding the functions of the ISP related to focus and exposure, it may be better to set the camera according to the requirements necessary for image processing on the side of the image recognition circuit 50. Therefore, it is advisable to adopt a system in which the content of the camera settings on the side of the camera video recording circuit 40 is set in advance, taking into account the settings related to exposure and focus that are necessary on the side of the image recognition circuit 50. For example, taking into account the settings related to exposure and focus that are necessary on the side of the image recognition circuit 50 in advance, and adjusting within a range where the camera video does not become too bad, for example, set the ISP (for example, the ISP in the camera video recording circuit 40) on the side of the camera video recording circuit 40.

[0147] (Camera Control Example 2) The camera settings are coordinated and adjusted between the camera video recording circuit 40 and the image recognition circuit 50. By doing so, it becomes easier to achieve both compression and recording of camera video and improvement in the accuracy of image recognition processing.

[0148] As shown in FIGS. 8(a) and 8(b), it is advisable to provide a function in which the image recognition circuit 50 and the camera video recording circuit 40 communicate with each other to grasp the state of the control of the camera 10.

[0149] For example, instead of setting the settings of the camera 10 in advance, or setting in advance and generating an adjustment request signal for exposure and focus on the side of the image recognition circuit 50 and outputting it to the camera video recording circuit 40. When the camera video recording circuit 40 receives this, the setting is performed by the setting function on the side of the camera video recording circuit 40. For example, instead of setting the camera settings in advance, or setting in advance and outputting an adjustment request signal for exposure and focus to the camera video recording circuit 40 on the side of the image recognition circuit 50. When the camera video recording circuit 40 receives this, it is advisable to adjust the ISP settings on the side of the camera video recording circuit 40 accordingly. Note that the information related to the camera settings set on the side of the camera video recording circuit 40 is transmitted to the image recognition circuit 50. It is particularly advisable to configure the image recognition circuit 50 to receive this and use it in the image recognition process.

[0150] For example, when the image recognition circuit 50 performs label recognition, it is advisable to calculate the area corresponding to the road shoulder in the image as the priority area for exposure calculation. Basically, at this time, the control of the camera is performed from the camera video recording circuit 40 side. However, in advance, the exposure priority of this area is set to be slightly higher. When the exposure of the label area is insufficient in the image recognition circuit 50, an exposure correction signal is output from the image recognition circuit 50 to the camera video recording circuit 40. When the camera video recording circuit 40 receives this signal, it is advisable to output a signal to increase the exposure of the camera to the camera side, etc.

[0151] In this way, the control entity of the camera 10 itself should basically be on the camera video recording circuit 40 side, and the image recognition circuit 50 side should be configured not to directly control the camera. Also, an adjustment signal is output from the image recognition circuit 50 side to the camera video recording circuit 40 side, and the image recognition circuit 50 side that receives this adjustment signal should be configured to indirectly control the camera. For example, the control entity of the camera 10 itself should basically be the camera video recording circuit 40, and the image recognition circuit 50 should be configured not to directly control the camera. Also, it should be configured to control indirectly.

[0152] Note that the setting regarding focus may be made unnecessary by setting the camera to pan focus.

[0153] As another example, as described above (Pattern 3), for example, the control entity of the camera may be switched in a time-division manner. In particular, the recording frame rate of the video should be set to a frame rate sufficient for video recording and playback (for example, a frame rate at a level that cannot be captured by the human eye (for example, about 24 to 30 fps)), while the frame rate of the camera should be set to a higher frame rate (for example, 120 fps). When numbers such as "1", "2", "3", "4", "1", "2", "3", "4",... are sequentially assigned to the frames output by the camera, the one for "1" should be from the camera video recording circuit 40 side, and the one for "3" should be from the image recognition circuit 50 side. For example, "2" and "4" may be used as switching times.

[0154] The switching may be performed by physically switching both the ISP on the camera video recording circuit 40 side and the ISP on the image recognition circuit 50 side. However, it is preferable that the ISP switches the settings of the ISP on the camera video recording circuit 40 side in a time-division manner. It is preferable that all of the plurality of cameras 1-0 are controlled by the camera video recording circuit 40 side, and the image recognition circuit 50 side does not directly control any of the plurality of cameras. It may be controlled indirectly.

[0155] The cameras indirectly controlled by the image recognition circuit 50 may be some of the plurality of cameras. These some cameras may be some cameras with a large divergence between the requirements necessary for recognition and the requirements necessary for recording / playing back images (for humans to view videos).

[0156] In the configuration described with reference to FIG. 8, the image recognition circuit 50 may be replaced with a processing device that performs processes other than the image recognition process.

[0157] The functions of the system 1 described in the above-described embodiments can be realized by one or more programs and can be realized by the cooperation of one or more hardware resources. When the functions of the system 1 are realized using a program, the program for realizing this function may be provided in a state stored in a computer-readable recording medium such as various magnetic recording media, optical recording media, magneto-optical recording media, semiconductor memories, etc. Further, this program may be distributed via a network. Further, the present invention can also be grasped as a signal processing method.

[0158] The scope of the present invention is not limited to the configurations explicitly described in the specification, but also includes combinations of various aspects of the present invention disclosed in this specification within its scope. Among the present invention, the configuration for which a patent is sought is specified in the appended claims. However, even a configuration that is not currently specified in the claims, the configuration disclosed in this specification has the intention of being the subject of claims in the future.

[0159] The present invention is not limited to the configurations described in the above-described embodiments. The constituent elements of the above-described embodiments and modification examples may be arbitrarily selected and combined. Also, any constituent element of each embodiment or modification example may be arbitrarily combined with any constituent element described in the means for solving the invention or a constituent element embodying any constituent element described in the means for solving the invention. With respect to these, there is an intention to obtain rights in the amendment or divisional application of the present application. Also, even if there is a description such as "in the case of ~" or "when ~", it is not described as a configuration limited to that case or that time. The present invention also discloses configurations that are not these cases or times, and has an intention to obtain rights. Also, the places with descriptions in a specific order are not limited to this order. The present invention also discloses configurations in which some parts are deleted or the order is changed, and has an intention to obtain rights.

[0160] Also, by filing a change application for a design application, there is an intention to obtain rights for the overall design or a partial design. Although the drawings depict the entire apparatus in solid lines, the drawings include not only the overall design but also partial designs claimed for a part of the apparatus. For example, not only can a part of the members of the apparatus be a partial design, but the drawings also include a partial design of a part of the apparatus regardless of the members. As a part of the apparatus, it may be a part of the members of the apparatus or a part of the members. Regarding the overall design, of course, there is an intention to claim the rights for a partial design in which an arbitrary part of the solid line part of the drawing is changed to a broken line part.

Explanation of Reference Numerals

[0161] 1: System 1, 10-1 to 10-4: Cameras, 20, 20-1 to 20-4: Second video signal conversion circuits, 30: Deserializer, 31: First port, 32: Second port, 50: Image recognition circuit, 60: Storage medium, 70: Information input unit, 80: Display unit, 90: Image processing unit, 100: Processing device

Claims

1. A system including a SoC for a drive recorder and a SoC for image recognition, The system is characterized in that the output port of a SoC for image recognition is connected to the input port of the SoC for the drive recorder, and when image recognition by the SoC for image recognition results in a situation requiring recording, the signal at the output port of the SoC for image recognition is changed, and when there is a change in the signal at the input port of the SoC for the drive recorder, the SoC for the drive recorder has a function of recording an image in a recording means for a predetermined number of seconds before and after the change.

2. The camera's video signal transmitted via the cable A distribution means is provided for distributing the SoC for the drive recorder and the SoC for the image recognition. The system of claim 1 .

3. the image recognition SoC has a function of outputting information regarding a result of the image recognition process to the drive recorder SoC; The SoC for the drive recorder has a function of recording an image in the recording means based on information on the result of the image recognition processing output from the SoC for the drive recorder.

3. The system according to claim 1 or 2, characterized in that

4. The image recognition SoC has a function of outputting information indicating the position of a recognized object in the image to the drive recorder SoC.

4. A system according to claim 1, wherein:

5. the image recognition SoC has a function of outputting a trigger signal indicating a no entry trigger to the drive recorder SoC when an intrusion beyond a no entry sign is detected despite the image recognition SoC recognizing the no entry sign based on the image, The SoC for the drive recorder has a function of, when receiving an input of a trigger signal indicating the no-entry trigger from the SoC for image recognition, recording an image for a predetermined time before and after the input as a no-entry trigger image in the recording means.

5. A system according to claim 1, wherein:

6. the image recognition SoC has a function of outputting a trigger signal indicating a surrounding person recognition trigger to the drive recorder SoC when it recognizes that a person has entered the vicinity of the forklift; The SoC for the drive recorder has a function of, when receiving an input of a trigger signal indicating the surrounding person recognition trigger from the SoC for image recognition, recording an image of a predetermined time before and after the input as an entry prohibition trigger image in the recording means.

5. A system according to claim 1, wherein:

7. The image recognition SoC further includes a function of recording information based on an input image in a second recording means which is different from the recording means in which the image recognition SoC records information.

7. A system according to claim 1, wherein:

8. The image recognition SoC stores information obtained by processing an input image as information to be recorded in a second recording means. The system of claim 7 .

9. The image recognition SoC has a function of recording information related to the coordinates and type of a recognized object in association with the time of recording in the second recording means.

9. The system according to claim 7 or 8, characterized in that

Images