Systems and programs

Abstract

To provide a system that is more excellent than before.SOLUTION: A system 1 comprises: cameras 10-1 to 10-4; and a camera image recording circuit 40 which compresses images captured by the cameras 10-1 to 10-4 to store the images in a storage medium 60. A second image signal conversion circuit 30 distributes an image signal transmitted between the cameras 10-1 to 10-4 and the camera image recording circuit 40 to an image recognition circuit 50 that realizes an image recognition function on the basis of the image signal. The camera image recording circuit 40 records the images of the cameras 10-1 to 10-4 in the storage medium 60 when receiving a trigger signal TG according to the result of recognition by the image recognition function.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a system, a program, and the like. [Background technology]

[0002] 2. Description of the Related Art Systems that include a camera and record images captured by the camera have been known. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-196066 A Summary of the Invention [Problem to be solved by the invention]

[0004] For example, there have been various problems in systems having a function of recording images from such cameras. One of the objects of the present invention is to provide a system that is superior to conventional systems.

[0005] The purpose of the present invention is not limited to this, and the applicant intends to obtain rights for configurations that aim to obtain effects from parts of the configurations disclosed in this specification and drawings, etc., by filing a divisional application, amendment, etc. For example, this specification discloses a problem in which the part described as "can be" in this specification is read as "the problem is to be". The problems are described as being independent, and the applicant intends to obtain rights for the configurations for solving each problem separately by filing a divisional application, amendment, etc. Even if the problems are implicitly understood from the description in the specification, the applicant intends to include part of the configurations described in this specification in the scope of the patent claim by amendment or divisional application. In addition, the applicant has disclosed a configuration that solves a problem that combines these independent problems, and intends to obtain rights for it. [Means for solving the problem]

[0006] The present invention can be understood as the following aspects, for example. (1) A system having a camera and a camera image recording circuit that compresses images captured by the camera and stores them in a memory means, and preferably the system also has a signal distribution means that distributes signals transmitted between the camera and the camera image recording circuit to electronic components that realize specified functions based on the signals.

[0007] In this way, the image captured by the camera can be compressed and stored in the memory means, and in the electronic components that realize the specified functions, the specified functions can be realized based on the signals distributed between the camera and the camera image recording circuit.

[0008] The camera may be of various types, but may be a CCD (Charge Coupled Device) camera, and may preferably be a CMOS (Complementary Metal-Oxide-Semiconductor) camera. The camera may be a camera for capturing still images, and may preferably be a camera for capturing moving images. The camera may be a special camera such as an infrared camera, but may preferably be a camera for capturing visible light, and may preferably be a visible light camera. The camera may be a camera for obtaining monochrome images, but may preferably be a camera for obtaining color images. The camera may be, for example, a camera with red (R), green (G), and blue (B) color filters arranged on a plane. The camera resolution may be VGA or higher.

[0009] The camera may refer to, for example, a part equipped with an image sensor, but may also refer to the image sensor itself.

[0010] In particular, the camera may be a camera installed in the vehicle, and in particular, the camera may be a camera that is retrofitted after shipment from the vehicle manufacturer.

[0011] The video compression section in the camera video recording circuit can use various algorithms, but it is preferable to use an encoder with a standardized algorithm, and in particular, it is preferable to use at least one of an H.264 encoder and an H.265 encoder. In particular, it is preferable that the video encoder in the camera video recording circuit is configured with hardware, not with software processing. It is preferable that the encoder is a SoC (System-on-a-Chip) rather than a simple FPGA (Field-Programmable Gate Array). In this way, the camera video recording circuit may be a camera video recording chip. It is preferable that the camera video recording circuit is configured with a hard macro, not a soft macro.

[0012] It is particularly preferable that the camera video recording circuit is a dedicated chip for camera video recording that is sold on the market. The camera video recording circuit may be, for example, an ASSP for camera video recording, an LSI (Large Scale Integrated circuit) for camera video recording, such as an SoC for camera video recording, and in particular, the camera video recording circuit may be an SoC for a drive recorder.

[0013] The storage means is preferably a removable storage means, and more preferably a card-shaped medium such as an SD (registered trademark) card or a micro SD (registered trademark) card.

[0014] The signal transmitted between the camera and the camera video recording circuit may be a control signal, but preferably includes at least one of a video signal and a video synchronization signal. The video synchronization signal may be at least one of a pixel clock, a vertical synchronization signal, and a horizontal synchronization signal, and more preferably at least two of them. In particular, the pixel clock and the vertical synchronization signal are preferably used.

[0015] The video signal transmitting the camera video may be, for example, a serial transmission of video synchronized on a video channel basis, or a parallel transmission. Alternatively, a combination of serial and parallel transmission may be used. In particular, data such as the current time, event information, and sensor information may be superimposed on the video on a frame-by-frame basis. Data such as the current time, event information, and sensor information may be superimposed on the synchronized unit, or may be superimposed on the video.

[0016] In the case of distributing a video signal, for example, a video signal of a specific protocol may be input, and distributed to multiple output destinations using the same protocol as the input protocol. For example, if the camera outputs a MIPI CSI-2 signal as the video signal, the signal distribution means may be a MIPI CSI-2 signal splitter chip.

[0017] In the case of distributing a video signal, it is preferable to input a video signal of a predetermined protocol and output signals of the same protocol to multiple output destinations, which are different from the input protocol. For example, in the case of a camera that outputs an FPD-LINK III signal as a video signal, the signal distribution means is a circuit having a port that inputs an FPD-LINK III signal and outputs it as, for example, multiple MIPI CSI-2 signals (for example, a dual port output deserializer hub with one port input and two ports output). In particular, the port that outputs the MIPI CSI-2 signal should have a function of outputting a replica of a certain port to another port. In particular, it is preferable to use a deserializer hub chip.

[0018] The electronic component that realizes the predetermined function may include at least one of a camera separate from the camera and an image recognition circuit. When the electronic component that realizes the predetermined function includes a camera, the predetermined function may be a function of capturing an image of another area including a range different from that of the camera. When the electronic component that realizes the predetermined function includes an image recognition circuit, the predetermined function may be an image recognition function.

[0019] The signal distribution means may be, for example, a simple wiring branch, but it is particularly preferable that the signal distribution means has a distribution function so as to synchronize signals transmitted between the multiple distributed signal lines, and it is particularly preferable that the signal distribution means is a circuit (e.g., a chip) having a distribution function so as to synchronize signals transmitted between the multiple distributed signal lines.

[0020] The camera and the camera image recording circuit may be provided in the same housing, but it is preferable to provide them in separate housings and connect the two housings with a connecting means such as a cable or connector. The camera and the camera image recording circuit are preferably connected via a cable of at least 1 meter. In particular, it is preferable that the camera and the camera image recording circuit are connected via a cable that is routed inside the vehicle.

[0021] (2) The electronic component may have another camera different from the camera, and the signal may include a video synchronization signal generated by a synchronization signal generating means, and the video signals of the camera and the other camera may be output to the camera video recording circuit based on the distributed synchronization signal.

[0022] In this way, a video signal synchronized between the camera and another camera can be obtained on the camera video recording circuit side. For example, a video signal synchronized between the camera and another camera can be input to the camera video recording circuit, and the video based on the input synchronized video signals can be compressed and recorded. For example, when the recorded video signals are played back simultaneously, the synchronized videos can be viewed. Although the video may be compressed into separate files for each of the multiple cameras, it is preferable to compress the video into a single file.

[0023] (3) It is preferable that the camera has the synchronization signal generating means, and the signal distributing means distributes the synchronization signal to the other camera.

[0024] In this way, the video signal output from the other camera is based on the synchronization signal generated within the other camera, making it easier to synchronize the two cameras. It is particularly advantageous to adopt this configuration and to have the other camera and the other camera housed in the same housing. This is particularly effective when the distance over which the synchronization signal must be routed is short.

[0025] In the camera, it is preferable to generate the signal within the image sensor. This eliminates the need for an image sensor capable of external synchronization that captures images in sync with an input synchronization signal, and allows for a low-cost and simple configuration.

[0026] (4) The synchronization signal generating means may be provided outside the camera and the other camera, and the signal distributing means may distribute the synchronization signal to the camera and the other camera.

[0027] In this way, it is easy to synchronize the video signal output from the other camera with the video signal output from the other camera.

[0028] In particular, it is preferable to adopt this configuration and provide the camera and the other camera in different housings, which is particularly effective when the synchronization signal has to be routed over a relatively long distance.

[0029] (5) It is preferable that the camera image recording circuit has the synchronization signal generating means, and the signal distributing means distributes the synchronization signal to the camera and the other camera.

[0030] This increases the possibility of obtaining synchronized video signals from the camera and another camera, for example, at a timing that matches the video recording process.

[0031] (6) A first video signal conversion circuit having a first video signal conversion means for converting signals output from the camera and the other camera into video signals for transmission via a cable, and a second video signal conversion circuit having a second video signal conversion means for converting the video signals transmitted via the cable into video signals that can be input to the camera video recording circuit, wherein the synchronization signal generation means is provided within the first video signal conversion circuit or the second video signal conversion circuit, and the signal distribution means is provided within the first video signal conversion circuit or the second video signal conversion circuit, and the synchronization signal is distributed to the camera and the other camera.

[0032] In this way, the video signals synchronized between the camera and the other camera can be easily input to a camera video recording circuit via a cable and recorded.

[0033] The cable may be a relatively long cable, which may be a length longer than the length required for wiring within a camera unit including at least one of the camera and the further camera, and may in particular be one meter or more.

[0034] The first video signal conversion means may be, for example, a serializer, and the second video signal conversion means may be, for example, a deserializer. The first video signal conversion circuit may be, in particular, a serializer chip, and the second video signal conversion circuit may be, in particular, a deserializer chip. When the camera and the other camera are in-vehicle cameras, the transmission distance between the first video signal conversion circuit and the second video signal conversion circuit may be, in particular, several meters.

[0035] For example, the vehicle may include a first camera unit having the camera and a first first video signal conversion circuit, a second camera unit having the other camera and a second first video signal conversion circuit, and a recording unit having a second video signal conversion circuit and a camera video recording circuit, and a first cable is connected between the first camera unit and the recording unit, and a second cable is connected between the second camera unit and the recording unit to transmit the respective video signals. For example, the vehicle may include a first camera unit installed so that the camera (first camera) attached to the windshield of the vehicle captures an image of the front of the vehicle, and a second camera unit installed so that the other camera (second camera) captures an image of the rear of the vehicle, and a recording unit installed on the dashboard or under the passenger seat is connected to both camera units by separate cables. Alternatively, the recording unit and the first camera unit may be configured as one unit.

[0036] Also, for example, a first camera unit having the camera and a first first video signal conversion circuit and a second camera unit having the other camera and a second first video signal conversion circuit may be connected by a first cable, and a recording unit having a second video signal conversion circuit and a camera video recording circuit and the second camera unit may be connected by a second cable, and the respective video signals may be transmitted. Also, for example, one camera unit may be provided with a plurality of cameras including the camera and the other camera, and a first video signal conversion circuit, and the first video signal conversion circuit may convert the video signals of the plurality of cameras to flow through a single cable, and transmit the signals to a recording unit provided with a second video signal conversion circuit and a camera video recording circuit via the single cable, so that the videos of the plurality of cameras are recorded in the recording means. For example, a camera unit that is attached to the windshield of a vehicle, and that includes the camera (first camera) that captures an image in front of the vehicle and the other camera (second camera) that captures an image behind the vehicle, may be configured to be connected by a cable to a recording unit provided on the dashboard or under the passenger seat.

[0037] For example, the second video signal conversion circuit may convert the video signals of multiple cameras converted by the first video signal conversion circuit and transmitted over the single cable over a relatively long distance into signals for transmission over a relatively short distance required for wiring within the recording unit.

[0038] For example, the second video signal conversion circuit may be configured to output video signals from multiple cameras to the camera video recording circuit so that they flow through one transmission path, and the camera video recording circuit may input these. For example, the second video signal conversion circuit may divide the video signals from two cameras into packets or the like of a predetermined data size, and transmit the packets or the like alternately. Video signals from a larger number of cameras may be output in an interleaved manner through one transmission path.

[0039] Alternatively, for example, the second video signal conversion circuit may be configured to output video signals from multiple cameras to the camera video recording circuit so that the video signals are sent to different transmission paths for each camera, and the camera video recording circuit may input the video signals. In this way, processing for each camera can be easily performed in the different transmission paths for each camera or in the camera video recording circuit connected to the transmission paths. For example, it becomes relatively easy to record and output information other than the video by superimposing it on each camera frame for each camera.

[0040] It is advisable to use as few signal lines as possible in one cable, for example a twisted pair cable.

[0041] The number of signal lines as small as possible should be used as the transmission path, for example, a few signal lines (e.g., printed wiring on a substrate). The camera, the other camera, and the first video signal conversion circuit should be connected via at least one of printed wiring and a flexible cable. The second video signal conversion circuit and the camera video recording circuit should be connected via at least one of printed wiring and a flexible cable.

[0042] (7) It is preferable that the lengths of signal wiring from the signal distribution means to the camera and the other camera are approximately the same.

[0043] In this way, it is possible to increase the possibility that the video signal from the camera and the video signal from the other camera will be synchronized.

[0044] (8) The electronic components may include an image recognition circuit that performs a predetermined image recognition processing and is provided separately from the camera image recording circuit, and the signal to be transmitted may include a video signal, and the image recognition circuit may output information regarding the results of the image recognition processing performed based on the distributed video signal to the camera image recording circuit, and the camera image recording circuit may compress the image captured by the camera based on the information regarding the results and store it in the memory means.

[0045] In this way, it is possible to reliably compress the camera image and store it in the storage means, and it is also possible to perform recording based on the results of image recognition processing of the same camera image.

[0046] Although camera video recording circuits are relatively inexpensive, they are designed specifically for compressing and storing video, which makes it difficult to add functions such as image recognition (e.g., little information disclosure, few remaining resources, etc.) For example, SoCs for drive recorders are inexpensive, but because they are designed for drive recorders, it makes it difficult to add functions such as image recognition (e.g., little information disclosure, few remaining resources, etc.).

[0047] On the other hand, there is a problem that there is no highly reliable image recognition circuit for image recognition as software for compressing and recording images. For example, it is recommended to use a general-purpose SoC (SoC for image recognition) that can be used by compiling OSS (Open Source Software) and the like. In particular, it is recommended to use an SoC equipped with an accelerator circuit for image recognition or an accelerator circuit for deep learning. There is also the problem that it is costly to install an image encoder such as H.264 on a general SoC, or especially on an image recognition circuit that is an FPGA. In addition, even if a general SoC is equipped with an image encoder, there is a problem that there is a lack of reliable software for drive recorders, especially when trying to use it as a drive recorder, and there is also the issue that it is extremely costly to create it from scratch in-house.

[0048] For example, a MIPI CSI-2 splitter (e.g., a splitter chip) is used to split and input the same camera video signal to the camera video recording circuit and the image recognition circuit. The camera video recording circuit and the image recognition circuit are connected by a communication line, and the image recognition circuit performs image recognition of the state of the target object, and if the result of the recognition indicates that recording is required, a trigger signal is passed to the camera video recording circuit via this communication line. When the camera video recording circuit receives a trigger signal from the image recognition circuit, it is preferable to event-record the video signal it has received in a storage means.

[0049] The communication line may be a parallel communication line or a serial communication line to transmit trigger data. For example, information on the type of recognized object may be transmitted as this data. Alternatively, it may be a communication line between simple GPIOs (General-purpose input / output). For example, the GPIO output port of the SoC for image recognition is connected to the trigger input port of the SoC for drive recorder, and when the result of image recognition by the SoC for image recognition indicates a situation requiring recording, the signal of the GPIO output port of the SoC for image recognition is changed, and when there is a change in the signal of the trigger input port of the SoC for drive recorder, the SoC for drive recorder records the image for a predetermined number of seconds before and after the trigger occurrence in the recording means. Furthermore, signals from sensors connected to the camera image recording circuit may be transmitted to the image recognition circuit. For example, if the camera video recording circuit is an SoC for a drive recorder and the image recognition circuit is an SoC for image recognition, the acceleration sensor, switch, and GPS module are connected to the SoC for the drive recorder, and the SoC for the drive recorder receives information from the acceleration sensor, switch, and GPS module and outputs the information to the SoC for image recognition in real time. The SoC for image recognition may receive this information from the SoC for the drive recorder in real time, use it for image recognition, and output the results to the SoC for the drive recorder. The functions of both SoCs may be realized by executing programs for realizing these functions stored in each SoC.

[0050] For example, each of the signals from four cameras is serialized by a serializer and input to a deserializer hub. The deserializer hub outputs a serial signal that interleaves the signals from the same four cameras to the first and second ports. For example, it is recommended to connect a camera video recording circuit to the first port and an image recognition circuit to the second port.

[0051] For example, when the image recognition circuit recognizes a no entry sign but detects an intrusion beyond the sign, it outputs a no entry trigger to the camera video recording circuit. When the camera video recording circuit receives the no entry trigger from the image recognition circuit, it encodes and records images for 15 seconds before and after the no entry trigger in the storage means as no entry trigger images.

[0052] Also, for example, when the image recognition circuit detects that a person has entered the vicinity of the forklift, it outputs a surrounding person recognition trigger to the camera video recording circuit. For example, when the camera video recording circuit receives a surrounding person recognition trigger from the image recognition circuit, it encodes and records images for 30 seconds before and after the trigger in the storage means as surrounding person recognition trigger images.

[0053] For example, the interleaved signals from four cameras are processed by an image recognition circuit and a camera video recording circuit, which each separate the signals into four camera images. The image recognition circuit may perform processing to determine whether or not to output a trigger from one of the images from the four cameras, or may output a trigger when a state requiring a trigger is recognized in any of the images. Alternatively, the image recognition circuit may perform processing to determine whether or not to activate a trigger overall. It is preferable to be able to set which of these methods is to be adopted.

[0054] In particular, it is a good idea to provide a synchronization signal to the deserializer hub, the camera video recording circuit, and the image recognition circuit to achieve synchronization. The synchronization signal can be a pixel clock or a vertical synchronization signal. The deserializer hub can synchronize and output a clock to the four serializers. Alternatively, the deserializer hub can generate a clock and provide it to the camera video recording circuit and the image recognition circuit to achieve synchronization. The image recognition circuit can be an FPGA instead of an SoC, or it can be any other type of computer.

[0055] In the above example, the MIPI CSI-2 branching function is provided within the deserializer hub, but the branching device may be external.

[0056] The function for setting the camera (and / or the other camera) (e.g., a part of the ISP (Image Signal Processor) function including a part for controlling the camera) may be, for example, a configuration using one on the camera image recording circuit side (Pattern 1), a configuration using one on the image recognition circuit side (Pattern 2), or a configuration using one on both sides by switching (Pattern 3). Also, it may be a configuration using either an SoC with an ISP built in or an external ISP. The external ISP may be installed either on the camera image recording circuit side. However, since the image creation required for image recognition is different from the image creation that is compressed and recorded in a recording means for playback, etc., it becomes an issue as to how the camera should be controlled, such as the shutter speed, frame rate, and exposure time. In this regard, it is particularly preferable to adopt (9).

[0057] (9) It is preferable that the camera image recording circuit sets the camera.

[0058] In this way, it is possible to prevent conflicts in settings, and to relatively effectively balance camera image recording and image recognition based on the camera image. For example, the inventors have found that a configuration that employs the above-mentioned (Pattern 1) produces excellent effects. In many cases, the images used for image recognition are of low quality relative to the image quality required for recording. It is particularly good to have a function for recognizing images by deep learning, and in such a configuration, the requirements for the number of pixels, frame rate, etc. are generally low. As for the camera image recording circuit side, it is particularly good to have it within the camera image recording circuit.

[0059] However, with regard to the focus and exposure functions of the ISP, it may be better to set the camera to match the requirements for image processing on the image recognition circuit side. Therefore, it is a good idea to have a system in which the camera settings on the camera image recording circuit side are set in advance, taking into account the settings related to exposure and focus required on the image recognition circuit side. For example, it is a good idea to set the ISP on the camera image recording circuit side (for example, the ISP in the camera image recording circuit) in advance, taking into account the settings related to exposure and focus required on the image recognition circuit side, and adjusting them to a range where the camera image does not become too bad.

[0060] (10) It is preferable that the camera settings are adjusted in a coordinated manner between the camera image recording circuitry and the image recognition circuitry.

[0061] This makes it easier to both compress and record camera images and improve the accuracy of image recognition processing.

[0062] It is preferable that the image recognition circuit and the camera image recording circuit communicate with each other to have the function of mutually understanding the state of camera control.

[0063] For example, instead of setting the camera settings in advance, or in addition to setting them in advance, the image recognition circuit side may generate an exposure or focus adjustment request signal and output it to the camera video recording circuit, and when the camera video recording circuit receives this, the system may perform the settings using the setting function of the camera video recording circuit side. For example, instead of setting the camera settings in advance, or in addition to setting them in advance, the image recognition circuit side may output an exposure or focus adjustment request signal to the camera video recording circuit, and when the camera video recording circuit receives this, the ISP settings of the camera video recording circuit side may be adjusted accordingly. It is particularly preferable to configure the camera settings information set on the camera video recording circuit side to be transmitted to the image recognition circuit, which receives this and uses it in the image recognition processing.

[0064] For example, when recognizing signs using an image recognition circuit, the area in the image that corresponds to the shoulder of the road can be calculated as the priority area for exposure calculation. In this case, the camera is basically controlled from the camera image recording circuit, but the exposure priority of this area can be set slightly higher in advance, and if the image recognition circuit finds that the sign area is underexposed, the image recognition circuit can output an exposure correction signal to the camera image recording circuit, and when the camera image recording circuit receives this signal, it can output a signal to increase the camera exposure to the camera side.

[0065] In this way, it is preferable to configure the camera itself so that the camera image recording circuit side is the main control entity, and the image recognition circuit side does not directly control the camera. Also, it is preferable to configure the image recognition circuit side to output an adjustment signal to the camera image recording circuit side, and the image recognition circuit side that receives this adjustment signal controls the camera indirectly. For example, it is preferable to configure the camera itself so that the camera image recording circuit side is the main control entity, and the image recognition circuit does not directly control the camera. Also, it is preferable to configure it so that it controls indirectly.

[0066] In addition, it is recommended that you do not set the focus as the camera should be set to pan focus.

[0067] As another example, as the above-mentioned (Pattern 3), the camera control entity may be switched in a time-division manner. In particular, the video recording frame rate may be set to a frame rate sufficient for video recording and playback (for example, a frame rate that does not stutter for human vision (for example, about 24 to 30 fps)), while the camera frame rate may be set to a higher frame rate (for example, 120 fps), and when the frames output by the camera are numbered in order as "1", "2", "3", "4", "1", "2", "3", "4", etc., "1" may be the camera video recording circuit side and "3" the image recognition circuit side. For example, "2" and "4" may be the switching time.

[0068] The switching can be done by physically switching both the ISP on the camera image recording circuit side and the ISP on the image recognition circuit side, but it is better to switch the ISP setting on the camera image recording circuit side in a time-division manner. It is better to have the camera image recording circuit side control all the multiple cameras, and not have the image recognition circuit side directly control any of the multiple cameras. It is better to have them controlled indirectly.

[0069] The cameras that the image recognition circuit indirectly controls should be a subset of multiple cameras, and these subsets should be cameras where the requirements for recognition are significantly different from the requirements for recording and playing back images (for humans to view the images).

[0070] There can be just one camera, or you can split the camera signal into two using, for example, a MIPI CSI-2 splitter.

[0071] (11) It is preferable to provide a recognition video signal generating means for generating a signal having a reduced amount of information compared to the video signal input to the camera video recording circuit, and the image recognition circuit is configured to perform the image recognition processing based on the video signal having the reduced amount of information.

[0072] In this way, the amount of information to be handled by the image recognition circuit can be reduced in advance, and the recognition process can be performed at higher speed and at lower cost, and the image can be compressed with excellent image quality and recorded on the recording means.

[0073] For example, the signal from the camera may not be branched but may be input to a camera video recording circuit, and the camera video recording circuit may reduce the amount of information by thinning out the video signal from the camera, and output the signal to the image recognition circuit from another port such as a USB port. Thinning out the image may be performed by thinning out pixels or frames, for example. In general, in image recognition, and particularly in deep learning, the amount of image information required for recognition can be significantly smaller than that required for a human to check the image.

[0074] In this example, the thinning out is performed by the camera video recording circuit, but there is a possibility that the load on the camera video recording circuit will be high or that resources will be insufficient. In addition, it may be impossible to make changes to the camera video recording circuit in the first place. Therefore, it is a good idea to split the video signal from the camera and input one of the split video signals directly to the video recording circuit, while inputting the thinned video signal to the image recognition circuit. For example, this function can be configured using an FPGA. Alternatively, instead of an FPGA, a standalone ISP chip can be used and the output destination can be switched for each frame using the output port switching function of the ISP chip.

[0075] (12) The image recognition circuit may record information based on the video signal distributed from the signal distribution means in a second recording means.

[0076] In this way, information based on the video signal distributed by the signal distribution means is recorded on the second recording means.

[0077] The second recording means may be a recording means different from the first recording means. In particular, the second recording means may be a removable recording medium different from the second recording means. The information based on the video signal distributed from the distribution means may be information obtained by processing the video signal distributed and input to the image recognition circuit. For example, video information that surrounds the recognized object and overlays its attributes as text information may be recorded. In addition to or instead of such video information, a trigger signal sent to a camera video recording circuit may be recorded together with the time. For example, information regarding the coordinates and type of the recognized object may be recorded in association with the time.

[0078] The inventions shown in (1) to (12) above can be arbitrarily combined. For example, the invention may be configured by adding at least a part of the configuration of at least one of the inventions after (2) to all or a part of the configuration of the invention shown in (1). In particular, the invention shown in (1) may be configured by adding at least a part of the configuration of at least one of the inventions after (2). In particular, the invention may be configured by including both the configuration of (2) and the configuration of (8). In addition, any configuration may be extracted from the inventions shown in (1) to (12) and the extracted configurations may be combined. The applicant of this application intends to obtain rights to the inventions including these configurations. In addition, even if there is a description such as "in the case of" or "when", this is not intended to describe the configuration as being limited to that case or time. These are examples of better configurations, and the applicant intends to obtain rights to configurations other than these cases or times. In addition, the parts described in order are not limited to this order. Configurations in which some parts have been deleted or the order has been changed are also disclosed, and the applicant intends to obtain rights. Effect of the Invention

[0079] According to the present invention, it is possible to provide a system etc. that is superior to conventional systems.

[0080] The effects of the invention of this application are not limited to this, and effects obtained from the configuration disclosed in the specification and drawings, etc. are also disclosed, and the intention is to obtain rights to the configuration that achieves the effects by divisional applications, amendments, etc. For example, in this specification, the phrase "can" is a description that clearly indicates the effect that is achieved, and there are also parts that show the effect even without the phrase "can." Also, there are effects that can be understood from the configuration even without such a description. [Brief description of the drawings]

[0081] [Figure 1] 1 is a block diagram showing a configuration of a system according to an embodiment; [Diagram 2] FIG. 13 is a block diagram showing a modified example of the system configuration. [Diagram 3] FIG. 13 is a block diagram showing a modified example of the system configuration. [Figure 4] FIG. 2 is a diagram illustrating an example of a connection between a camera and a processing device. [Diagram 5] FIG. 2 is a diagram illustrating an example of a connection between a camera and a processing device. [Figure 6] FIG. 2 is a diagram illustrating an example of a connection between a camera and a processing device. [Figure 7] FIG. 2 is a diagram illustrating an example of a connection between a camera and a processing device. [Figure 8] FIG. 2 is a diagram illustrating an example of a connection between a processing device and a recording device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0082] Hereinafter, the embodiments will be described in detail with reference to the drawings. The embodiments shown below are examples of the embodiments of the present disclosure, and the present disclosure is not limited to these embodiments. In the drawings referred to in the present embodiment, the same parts or parts having similar functions are given the same or similar symbols (symbols with A, B, etc. added after the numbers), and repeated explanations may be omitted. In addition, in each of the drawings referred to in the following description, the scale may be different from the actual scale in order to make each member, each region, etc. recognizable.

[0083] 1 is a block diagram showing the configuration of a system 1 according to an embodiment. The system 1 may be an on-board device mounted on a vehicle, and in particular may be an on-board device that functions as a drive recorder. The vehicle may be a forklift, but may also be a work vehicle other than a forklift, or a vehicle other than that of a worker (for example, a private or public passenger car, bus, or train).

[0084] In the following, first, the transmission of a video signal in the system 1 and the processing using the video signal will be described, and then the transmission of a synchronizing signal and the processing using the synchronizing signal will be described.

[0085] The system 1 includes cameras 10-1 to 10-4, first video signal conversion circuits 20-1 to 20-4, second video signal conversion circuit 30, camera video recording circuit 40, image recognition circuit 50, storage medium 60, information input unit 70, and display unit 80. Hereinafter, when it is not necessary to distinguish between the cameras 10-1 to 10-4, they may be collectively referred to as "cameras 10", and when it is not necessary to distinguish between the first video signal conversion circuits 20-1 to 20-4, they may be collectively referred to as "first video signal conversion circuits 20". In this embodiment, there are four cameras 10 and four first video signal conversion circuits 20, and they handle video signals of four video channels. However, there may be three or less cameras 10 and five or more cameras 10 and five or more video channels, respectively, and they may handle video signals of three or less video channels. Each circuit of the above-described system 1 is realized, for example, by a physically separated circuit (e.g., a chip), but two or more circuits may be realized by circuits fabricated on the same substrate (e.g., the same chip).

[0086] The camera 10 is an electronic component that captures an image and generates a video signal that indicates the captured image. The camera 10 can be any type of camera, but can be a CCD camera, and is preferably a CMOS camera in particular. The camera 10 can be a camera for capturing still images, but is preferably a camera for capturing moving images in particular. The camera 10 can be a special camera such as an infrared camera, but is preferably a camera for capturing visible light (for example, light in a wavelength range of 400 to 700 nm), and is preferably a visible light camera in particular. The camera 10 may be a camera that can obtain monochrome images, but is preferably a camera that can obtain color images. The camera 10 may be a camera in which, for example, red (R), green (G), and blue (B) color filters are arranged on a plane. The resolution of the camera is preferably VGA or higher. The camera 10 may refer to a part that includes an image sensor, but may also refer to the image sensor itself. In particular, the camera 10 is preferably a camera installed in a vehicle. In particular, it is preferable that the camera 10 is a camera that is attached to the vehicle after it is shipped by the vehicle manufacturer.

[0087] In this embodiment, the video signal generated and output by the camera 10 is digital data including the gradation values ​​of each of a plurality of pixels. The video signal preferably includes 8-bit (256 gradations) data for each of the red (R), green (G), and blue (B) color components, but the color components and number of gradations are not limited to this.

[0088] The camera 10-1 is communicatively connected to the first video signal conversion circuit 20-1. The camera 10-2 is communicatively connected to the first video signal conversion circuit 20-2. The camera 10-3 is communicatively connected to the first video signal conversion circuit 20-3. The camera 10-4 is communicatively connected to the first video signal conversion circuit 20-4. The cameras 10-1 to 10-4 may be connected to the first video signal conversion circuits 20-1 to 20-1 by either a wired or wireless transmission path, but it is preferable to connect them using a wired transmission path. In the example of FIG. 1, the camera 10-1 outputs video signals A1, A2, A3, and A4 in chronological order. The camera 10-2 outputs video signals B1, B2, and B3 in chronological order. The camera 10-3 outputs video signals C1, C2, C3, and C4 in chronological order. Camera 10-4 outputs video signals D1 and D2 in chronological order. Cameras 10-1 to 10-4 output video signals in ascending order of the numbers suffixed to "A" to "D". Video signals with the same number are output simultaneously by cameras 10-1 to 10-4. For example, video signals A1, B1, C1, and D1 are output simultaneously. Each of video signals A1 to A4, B1 to B3, C1 to C4, D1, and D2 is a video signal of a predetermined unit, and may be a video signal showing an image of one frame, for example.

[0089] The cameras 10-1 to 10-4 may each capture a different area. The area captured by the camera 10 may include, for example, one or both of the inside and outside of the vehicle. The area outside the vehicle may be any of the areas in front (front), right diagonally forward, right (directly to the side), right diagonally rear, rear (directly behind), left diagonally rear, left (directly to the side), and left diagonally forward, but may be another area. The cameras 10-1 to 10-4 may include, for example, at least a camera (first camera) attached to the windshield of the vehicle and a camera (second camera) installed to capture an image of the rear of the vehicle.

[0090] The first video signal conversion circuit 20 includes a first video signal conversion means for receiving a video signal input from the camera 10 and converting it into a video signal in a serial format. The first video signal conversion circuit 20 may be a serializer, particularly a serializer chip. The serial format video signal is a signal obtained by converting the video signal input from the camera 10 into a serial data string. The first video signal conversion circuit 20-1 converts the video signal from the camera 10-1, the first video signal conversion circuit 20-2 converts the video signal from the camera 10-2, the first video signal conversion circuit 20-3 converts the video signal from the camera 10-3, and the first video signal conversion circuit 20-4 converts the video signal from the camera 10-4. The first video signal conversion circuits 20-1 to 20-4 output the interleaved converted video signals to the second video signal conversion circuit 30. Each of the first video signal conversion circuits 20-1 to 20-4 and the second video signal conversion circuit 30 may be connected either by wire or wirelessly, but it is preferable that they are connected using a single wired cable. The first video signal conversion circuits 20-1 to 20-4 convert the video signals into video signals for transmission via such a cable. This cable should be a relatively long-distance cable. The relatively long distance should be a distance longer than the length required for wiring in a camera unit including at least one of the camera 10 and another camera 10, and should particularly be one meter or more.

[0091] When the camera 10 is an in-vehicle camera, it is particularly preferable that the transmission distance between the first video signal conversion circuit 20 and the second video signal conversion circuit 30 be several meters.

[0092] The second video signal conversion circuit 30 includes a second video signal conversion means for converting the video signals of the cameras 10-1 to 10-4 transmitted from the first video signal conversion circuits 20-1 to 20-4 via cables into video signals that can be input to the camera video recording circuit 40. The second video signal conversion circuit 30 may be a deserializer, particularly a deserializer chip. The second video signal conversion circuit 30 also functions as a signal distribution means for distributing the video signals transmitted between the cameras 10-1 to 10-4 and the camera video recording circuit 40 to the image recognition circuit 50. For example, the second video signal conversion circuit 30 may have a MIPI CSI-2 branching function. In this way, the video captured by the camera 10 can be compressed and stored in the storage medium 60, and the image recognition function can be realized in the image recognition circuit 50 based on the distributed signal between the camera 10 and the camera video recording circuit 40.

[0093] The second video signal conversion circuit 30 has a first port 31 and a second port 32. A camera video recording circuit 40 is communicatively connected to the first port 31, and an image recognition circuit 50 is communicatively connected to the second port 32. Such a second video signal conversion circuit 30 can also be specified as a deserializer hub.

[0094] The second video signal conversion circuit 30 may be connected to the camera video recording circuit 40 and the image recognition circuit 50 either by wire or wirelessly, but it is preferable to connect them using a wired cable. The second video signal conversion circuit 30 outputs a video signal S1 to the camera video recording circuit 40 and a video signal S2 to the image recognition circuit 50 based on the interleaved video signals from the cameras 10-1 to 10-4. The second video signal conversion circuit 30 distributes the video signals from the cameras 10-1 to 10-4 by outputting the same image signals S1 and S2 from the first port 31 and the second port 32 at the same timing. The second video signal conversion circuit 30, for example, inputs a video signal of a predetermined protocol, and distributes and outputs the signal to multiple output destinations using the same protocol as the input protocol. For example, if the camera 10 outputs a MIPI CSI-2 signal as a video signal, the second video signal conversion circuit 30 may be a MIPI CSI-2 signal splitter chip. The second video signal conversion circuit 30 may, for example, input a video signal of a predetermined protocol and output signals of the same protocol to multiple output destinations, which are different from the input protocol. For example, in the case of a camera 10 that outputs an FPD-LINK III signal as a video signal, the second video signal conversion circuit 30 may be a chip equipped with a port that inputs an FPD-LINK III signal and outputs it as, for example, multiple MIPI CSI-2 signals (for example, a dual port output deserializer hub with one port input and two ports output). In particular, the port that outputs the MIPI CSI-2 signal may have a function of outputting a replica of a certain port to another port. In particular, it is preferable to use a deserializer hub chip.

[0095] In the system 1, one cable should have as few signal lines as possible, for example, a twisted pair cable. A transmission path should have as few signal lines as possible, for example, several signal lines (for example, printed wiring on a substrate). A connection between the camera 10 and a plurality of cameras 10, including another camera 10, and the first video signal conversion circuit 20 should be made via at least one of a printed wiring or a flexible cable. A connection between the second video signal conversion circuit 30 and the camera video recording circuit 40 should be made via at least one of a printed wiring or a flexible cable.

[0096] The second video signal conversion circuit 30 selects the video signals of the cameras 10-1, 10-2, 10-3, and 10-4 in a predetermined order, and outputs the selected video signals from each of the first port 31 and the second port 32. Here, in the example of FIG. 1, the second video signal conversion circuit 30 outputs the video signals S1 and S2 by switching the video signals in units of one frame in the order of the cameras 10-1, 10-2, 10-3, 10-4, 10-1, 10-2, 10-3, 10-4, etc. Specifically, the second video signal conversion circuit 30 outputs the video signals S1 and S2 from the first port 31 and the second port 32 in the order of the video signals A1, B1, C1, D1, A2, B2, C2, D2, A3, B3, C3, A4, and C4. It should be noted that the 30th does not need to have the function of distributing a video signal (MIPI CSI-2 branching function), in which case an external distributor may be used.

[0097] The camera video recording circuit 40 has a function of compressing the video signal S1 from the first port 31 and storing it in the storage medium 60. The video compression section (not shown) in the camera video recording circuit 40 can use various algorithms, but it is preferable to use an encoder with a standardized algorithm, and it is particularly preferable to have at least one of an H.264 encoder and an H.265 encoder. In particular, the video encoder in the camera video recording circuit 40 is preferably configured by hardware, not by software processing. The camera video recording circuit 40 is preferably an FPGA, and more preferably an SoC. In addition, it is preferable that the camera video recording circuit 40 is configured by a hard macro, not a soft macro.

[0098] It is particularly preferable that the camera video recording circuit 40 is sold as a dedicated chip for camera video recording. The camera video recording circuit 40 may be, for example, an ASSP for camera video recording, an LSI for camera video recording such as an SoC for camera video recording, and in particular, the camera video recording circuit may be an SoC for a drive recorder.

[0099] When the trigger signal TG is input from the image recognition circuit 50, the camera video recording circuit 40 performs event recording by storing the video signal for a period corresponding to the input in the storage medium 60. This period may be, for example, 30 seconds before and after the occurrence of the event, but may be another period.

[0100] The image recognition circuit 50 functions as an electronic component that realizes a predetermined function using a signal distributed by the second video signal conversion circuit 30 and transmitted between the camera 10 and the camera video recording circuit 40. The image recognition circuit 50 recognizes the video represented by the video signal S2 from the second port 32, and judges whether an event that triggers event recording has occurred. If the image recognition circuit 50 judges that the event has occurred, it outputs a trigger signal TG that triggers the start of event recording to the camera video recording circuit 40. For example, the image recognition circuit 50 recognizes the state of an object in the video, and if the result of the recognition indicates that the event recording is necessary, it outputs the trigger signal TG to the camera video recording circuit 40. In this way, the image recognition circuit 50 may comprehensively judge whether to generate the trigger signal TG from the videos of the four cameras 10-1 to 10-4.

[0101] In this way, the image recognition circuit 50 outputs information about the results of the image recognition process performed based on the video signal S2 to the camera video recording circuit 40. The image recognition circuit 50 may output information indicating the position of a recognized object in the video to the camera video recording circuit 40. The camera video recording circuit 40 compresses the video represented by the video signal S1 based on the information about the results of the image recognition process and stores it in the storage medium 60. In this way, it is possible to reliably compress the video of the camera 10 and store it in the storage medium 60, and it is also possible to record based on the results of the image recognition process of the same video of the camera 10.

[0102] Here, an example of event recording will be described. For example, when the image recognition circuit 50 recognizes a no-entry sign based on the image, but detects an intrusion beyond the sign, the image recognition circuit 50 outputs a trigger signal TG1 indicating a no-entry trigger to the camera video recording circuit 40. When the camera video recording circuit 40 receives the input of the trigger signal TG1, it encodes and records a video signal showing an image for 15 seconds before and after the trigger signal TG1 as a no-entry trigger image in the storage medium 60. When the image recognition circuit 50 recognizes, for example, that a person has intruded into the vicinity of the forklift, it outputs a trigger signal TG2 indicating a surrounding person recognition trigger to the camera video recording circuit 40. When the camera video recording circuit 40 receives the input of the trigger signal TG2, it encodes and records a video signal showing an image for 30 seconds before and after the trigger signal TG2 as a surrounding person recognition trigger image in the storage medium 60. Here, only the image of the video channel that triggered the output of the trigger signal TG1 or TG2 may be recorded, or the images of all the video channels may be recorded. Note that various modifications are possible regarding the event that triggers the output of the trigger signal and the period that is the subject of the event recording. For example, event recording may be performed when information input via the information input unit 70 satisfies a predetermined condition.

[0103] For example, a MIPI CSI-2 splitter (e.g., a splitter chip) is used to split and input the same camera video signal to the camera video recording circuit 40 and the image recognition circuit 50. The camera video recording circuit 40 and the image recognition circuit 50 are connected by a communication line, and the image recognition circuit 50 performs image recognition of the state of the target object, and if the result of the recognition indicates that recording is necessary, a trigger signal is passed to the camera video recording circuit via this communication line. When the camera video recording circuit receives a trigger signal from the image recognition circuit, it is preferable to event-record the video signal it has received in a storage means.

[0104] The communication line may be a parallel communication line or a serial communication line to transmit the trigger signal TG and other data. For example, information on the type of recognized object may be transmitted as this data. Alternatively, it may be a simple communication line between GPIOs. For example, the GPIO output port of the SoC for image recognition is connected to the trigger input port of the SoC for drive recorder, and when the result of image recognition by the SoC for image recognition indicates a situation requiring recording, the signal of the GPIO output port of the SoC for image recognition is changed, and when there is a change in the signal of the trigger input port of the SoC for drive recorder, the SoC for drive recorder records the image for a predetermined number of seconds before and after the trigger occurrence in the recording means. Furthermore, signals from sensors connected to the camera image recording circuit 40 may be transmitted to the image recognition circuit 50. For example, if the camera video recording circuit 40 is an SoC for a drive recorder and the image recognition circuit 50 is an SoC for image recognition, the acceleration sensor, switch, and GPS module are connected to the SoC for the drive recorder, and the SoC for the drive recorder receives information from the acceleration sensor, switch, and GPS module and outputs the information to the SoC for image recognition in real time. The SoC for image recognition may receive this information from the SoC for the drive recorder in real time, use it for image recognition, and output the results to the SoC for the drive recorder. The functions of both SoCs may be realized by executing programs for realizing these functions stored in each SoC.

[0105] The image recognition circuit 50 may record information based on the video signal distributed from the second video signal conversion circuit 30 in the second recording means. In this way, information based on the video signal distributed from the second video signal conversion circuit 30 is recorded in the 22nd recording means. The second recording means may be a recording means different from the first recording means. In particular, it may be a removable recording medium different from the first recording means. If the first recording means is a storage medium 60, the second recording means is a recording means different from the storage medium 60. The information based on the distributed video signal may be information obtained by processing the video signal distributed and input to the image recognition circuit 50. For example, video information that surrounds the recognized object and superimposes its attributes as text information may be recorded. In addition to such video information, or instead of such video information, for example, the trigger signal TG output to the camera video recording circuit 40 may be recorded together with the time. For example, information on the coordinates and type of the recognized object may be recorded in association with the time.

[0106] The camera video recording circuit 40 may have a function of continuous recording in addition to the function of event recording. In this case, the camera video recording circuit 40 records video in a predetermined memory area secured in the storage medium 60, and when the memory area becomes full, records video by overwriting the oldest video first. To prevent the loss of video related to the event recording by the continuous recording function, these memory areas may be separated, or overwriting may be prohibited using an overwrite prohibition flag.

[0107] The storage medium 60 functions as a storage means for storing the images recorded by the camera image recording circuit 40. The storage medium 60 may be either an internal storage means of the system 1 or an external storage means, but is preferably a removable external storage means, and is particularly preferably a card-shaped medium such as an SD card or a microSD card. The images stored in the storage medium 60 can be displayed on the display unit 80 or by using an external terminal. The display unit 80 is, for example, a liquid crystal display or another type of display device.

[0108] The information input unit 70 accepts input of information from the outside. The information input unit 70 may include various switches including, for example, a switch for turning on or off the power of the system 1. The information input unit 70 may include a sensor for a drive recorder. The information input unit 70 may include, for example, an acceleration sensor (functioning as, for example, a G sensor) for detecting an impact that triggers event recording. The information input unit 70 may also include a GPS (Global Positioning System) sensor for measuring the current position. Various signals according to the information input via the information input unit 70 may be exchanged between the camera video recording circuit 40 and the image recognition circuit 50.

[0109] Next, the transmission of the synchronization signal and the processing using the synchronization signal will be described. The signal transmitted between the camera 10 and the camera video recording circuit 40 includes a video synchronization signal generated by a synchronization signal generating means. The system 1 is preferably configured to output the video signals of the cameras 10-1 to 10-4 to the camera video recording circuit 40 based on the synchronization signal distributed together with the video signals.

[0110] In this way, the camera video recording circuit 40 can obtain video signals synchronized between the cameras 10-1 to 10-4. For example, the camera video recording circuit 40 can input video signals synchronized between the cameras 10-1 to 10-4, compress and record video based on the input synchronized video signals. For example, when the recorded video signals are played back simultaneously, the synchronized video can be viewed. Note that the video may be compressed into separate files for each of the multiple cameras, but it is preferable to compress the video into a single file.

[0111] In the system 1, in order to synchronize the operations of the multiple cameras 10, the first video signal conversion circuits 20-1 to 20-4, the camera video recording circuit 40, and the image recognition circuit 50, the synchronization signal is also distributed to the image recognition circuit 50. In this way, the signal transmitted between the camera 10 and the camera video recording circuit 40 may be a control signal, but it is preferable to have at least one of a video signal or a video synchronization signal. The synchronization signal may be at least one of a pixel clock, a vertical synchronization signal, and a horizontal synchronization signal, and more preferably at least two of them. In particular, it is preferable to have a pixel clock and a vertical synchronization signal. The second video signal conversion circuit 30 may synchronize and output a clock to the four first video signal conversion circuits 20-1 to 20-4. The generation entity and distribution method of the synchronization signal may be, for example, the following (Method 1) or (Method 2).

[0112] (Method 1) A synchronization signal is generated with one camera 10 as a parent and distributed to another camera 10, the first video signal conversion circuit 20, the second video signal conversion circuit 30, the camera video recording circuit 40, and the image recognition circuit 50.

[0113] In (Method 1), any one of the cameras 10 has a synchronization signal generating means. Then, the one camera 10, which acts as a signal distributing means, distributes this synchronization signal to another camera 10, the first video signal converting circuit 20, the second video signal converting circuit 30, the camera video recording circuit 40, and the image recognition circuit 50.

[0114] In this way, the video signal output from the other camera 10 becomes a video signal based on the synchronization signal generated in the first camera 10, making it easier to synchronize these cameras 10. In particular, it is preferable to adopt this configuration and have the first camera 10 and the other camera 10 housed in the same housing. This is particularly effective when the synchronization signal is routed over a short distance. In the camera 10, it is particularly preferable that the signal is generated in the imaging element. In this way, it is not necessary to use an imaging element capable of external synchronization that captures images in synchronization with the input synchronization signal, and a low-cost and simple configuration can be achieved.

[0115] (Method 2) Any one of the first video signal conversion circuit 20, the second video signal conversion circuit 30, the camera video recording circuit 40, and the image recognition circuit 50 generates a synchronization signal and distributes it to each of the other elements.

[0116] In (Method 2), a synchronization signal generating means is provided outside camera 10, and the synchronization signal generated by this synchronization signal generating means is distributed to cameras 10-1 to 10-4 and other elements.

[0117] Method 2 makes it easy to synchronize the video signal output from one camera 10 with the video signal output from another camera 10. It is particularly effective to adopt this configuration and provide the camera 10 and the other camera 10 in different housings. This is particularly effective when the synchronization signal is routed over a relatively long distance.

[0118] In (Method 2), the camera video recording circuit 40 may have a synchronization signal generating means. In this case, the camera video recording circuit 40 as a signal distribution means distributes the synchronization signal to the cameras 10-1 to 10-4 and other elements. In this way, it is possible to increase the possibility of obtaining synchronized video signals from the camera and another camera at a timing that matches the video recording process.

[0119] In (Method 2), a synchronization signal generating means may be provided in either the first video signal conversion circuit 20 or the second video signal conversion circuit 30. In this case, the first video signal conversion circuit 20 or the second video signal conversion circuit 30 serving as a signal distribution means distributes the synchronization signal to the cameras 10-1 to 10-4 and other elements. In this way, the video signals synchronized between the camera and the other camera can be easily input to a camera video recording circuit via a cable and recorded.

[0120] In particular, a synchronization signal may be provided to the second video signal conversion circuit 30, the camera video recording circuit 40, and the image recognition circuit 50 to achieve synchronization. The synchronization signal may be a pixel clock or a vertical synchronization signal. The second video signal conversion circuit 30 may synchronously output a clock to the four first video signal conversion circuits 20-1 to 20-4. Alternatively, a clock may be generated in the second video signal conversion circuit 30, and the clock may be provided to the camera video recording circuit 40 and the image recognition circuit 50 to achieve synchronization. The image recognition circuit 50 may be an FPGA instead of an SoC, or may be any other type of computer.

[0121] The method of processing based on the synchronization signal is not limited to either (Method 1) or (Method 2), and for example, a synchronization signal generating means may be provided separately from the elements of the system 1 described above.

[0122] The above signal distribution means may be, for example, a simple wiring branch, but it is particularly preferable that the means has a distribution function so as to synchronize signals transmitted between the multiple distributed signal lines, and it is particularly preferable that the means is a circuit (e.g., a chip) having a distribution function so as to synchronize signals transmitted between the multiple distributed signal lines.

[0123] Also, the camera 10 and the camera video recording circuit 40 may be configured to be provided in the same housing, but it is preferable to provide them in separate housings and connect the two housings with a connecting means such as a cable or a connector. It is preferable to connect the camera 10 and the camera video recording circuit 40 via a cable of 1 meter or more. In particular, it is preferable to connect the camera 10 and the camera video recording circuit 40 via a cable that is routed inside the vehicle.

[0124] The configuration described in FIG. 1 may be modified as shown in FIG. 2. In FIG. 2, the configuration described in FIG. 1 is mainly shown with different parts from FIG. 1. In this example, the video signal from the camera 10 is input to the camera video recording circuit 40 without being distributed. The camera video recording circuit 40 supplies the video signals from the cameras 10-1 to 10-4 to the image recognition circuit 50. The camera video recording circuit 40 includes a recognition video signal generating means for generating a signal with a reduced amount of information compared to the input video signal, and reduces the amount of information by thinning the video signal, and outputs the reduced information to the image recognition circuit 50 via another port such as a USB (Universal Serial Bus) port. The video signal may be thinned out by, for example, thinning out pixels or thinning out frames. This is because the amount of information required for image recognition in general image recognition, particularly in deep learning, can be significantly smaller than that required for humans to check the video.

[0125] The configuration described in FIG. 2 may be further modified as shown in FIG. 3. In FIG. 3, the configuration described in FIG. 1 is illustrated by focusing on the parts different from FIG. 1. In the example of FIG. 2, the thinning is performed by the camera video recording circuit 40, but this may increase the processing load or may cause a shortage of resources. Therefore, as shown in FIG. 3, an image processing unit 90 may be provided between the camera 10, the camera video recording circuit 40, and the image recognition circuit 50. The image processing unit 90 branches the video signal from the camera, outputs the video signal as it is to the camera video recording circuit 40, and outputs the thinned video signal to the image recognition circuit 50. The image processing unit 90 is, for example, an FPGA, but may be configured using a single ISP chip. The image processing unit 90 may be configured to switch the output destination for each frame using the output port switching function of the ISP chip.

[0126] In the system 1, the number of cameras 10 may be one. For example, the camera signal may be split into two using a splitter such as a MIPI CSI-2 splitter. In this case, it is equivalent to a configuration in which one of the cameras 10-1 to 10-4 described in FIG. 3 is used and the image processing unit 90 is used as a splitter.

[0127] In system 1, the video signal transmitting the video of camera 10 may be transmitted in serial or parallel, for example, with the video synchronized in video channel units. Alternatively, serial and parallel transmission may be combined. In particular, data such as the current time, event information, and sensor information may be superimposed on the video in units of frames of the video, or may be superimposed on the video. Data such as the current time, event information, and sensor information may be superimposed on the synchronized units, or may be superimposed on the video.

[0128] It is also advisable to make the lengths of the signal wiring from the signal distribution means to the camera 10 and to the other camera 10 approximately the same. This increases the possibility that the video signal from the camera 10 and the video signal from the other camera 10 will be synchronized.

[0129] Next, an example of connection between the cameras 10-1 to 10-4 and the processing device 100 will be described with reference to Fig. 4 to Fig. 7. The processing device 100 is a device that performs predetermined processing based on the video signals output by the cameras 10-1 to 10-4. The processing device 100 may be, for example, either the camera video recording circuit 40 or the image recognition circuit 50.

[0130] In the example of FIG. 4, cameras 10-1 to 10-4 are provided in one camera unit, and these are connected to a single first video signal conversion circuit 20. The first video signal conversion circuit 20 outputs a video signal Sa1 to the second video signal conversion circuit 30 via a single cable. The first video signal conversion circuit 20 outputs a video signal Sa2, which is the same as the video signal Sa1, to the processing device 100 via a single cable. The video signal Sa2 is a signal that retains the serial format of the video signal Sa1. The processing device 100 executes processing using the video signal Sa2 input via the single cable.

[0131] In the example of FIG. 5, a single camera unit is provided with a plurality of cameras 10-1 to 10-4, which are connected to a single first video signal conversion circuit 20. The first video signal conversion circuit 20 outputs a video signal Sb1 to the second video signal conversion circuit 30 via a single cable. The video signal Sb1 is the same as the video signal Sa1. The second video signal conversion circuit 30 outputs video signals Sb21, Sb22, Sb23, and Sb24 using different cables for each of the cameras 10-1 to 10-4. The video signals Sb21 to Sb24 are signals obtained by converting the video signal Sb1, which is in serial format, into parallel format. The processing device 100 executes processing using the video signals Sb21 to Sb24 input via different cables for each camera 10.

[0132] In the example of FIG. 6, a single camera unit is provided with a plurality of cameras 10-1 to 10-4, which are connected to different first video signal conversion circuits 20-1 to 20-4 for each camera. The first video signal conversion circuits 20-1 to 20-4 each output video signals Sc11, Sc12, Sc13, and Sc14 to the second video signal conversion circuit 30 via a single cable. The second video signal conversion circuit 30 outputs a video signal Sc2 to the processing device 100 via a single cable. The processing device 100 executes processing using the video signal Sc2 input via the cable. The video signal Sc2 is a signal obtained by converting the parallel video signals Sc1 to Sc4 into a serial signal. The system 1 of FIG. 1 corresponds to this connection example.

[0133] In the example of FIG. 7, a single camera unit is provided with a plurality of cameras 10-1 to 10-4, which are connected to different first video signal conversion circuits 20-1 to 20-4 for each camera. The system 1 of FIG. 1 corresponds to this connection example. Each of the first video signal conversion circuits 20-1 to 20-4 outputs video signals Sd11, Sd12, Sd13, and Sd14 to the second video signal conversion circuit 30 via a single cable. The second video signal conversion circuit 30 outputs video signals Sd21, Sd22, Sd23, and Sd24 using different cables for each of the cameras 10-1 to 10-4. The video signals Sd21 to Sd24 are signals that are parallel video signals Sd11 to Sd14, but remain in parallel format. The processing device 100 executes processing using the video signals Sb21 to Sb24 inputted via different cables for each camera 10.

[0134] Based on the configurations of Figures 1, 6 and 7, a system may be configured that includes a first camera unit having a camera 10 and a first video signal conversion circuit 20, a second camera unit having another camera 10 and a second first video signal conversion circuit 20, and a recording unit having a second video signal conversion circuit 30 and a camera video recording circuit 40, with a first cable connecting the first camera unit and the recording unit, and a second cable connecting the second camera unit and the recording unit, and transmitting the respective video signals. For example, the system may include a first camera unit installed so that the camera (first camera) attached to the windshield of the vehicle captures an image of the front of the vehicle, and a second camera unit installed so that the other camera (second camera) captures an image of the rear of the vehicle, and a recording unit installed on the dashboard or under the passenger seat is connected to both camera units by separate cables. Alternatively, the recording unit and the first camera unit may be configured as one unit.

[0135] Also, for example, a first camera unit having a camera 10 and a first first video signal conversion circuit 20 and a second camera unit having another camera 10 and a second first video signal conversion circuit 20 may be connected by a first cable, and a recording unit having a second video signal conversion circuit 30 and a camera video recording circuit 40 and the second camera unit may be connected by a second cable, and the respective video signals may be transmitted.

[0136] 1 and 4 and 5, a system may be configured in which one camera unit includes a camera 10 and a plurality of cameras 10 including another camera 10, and a first video signal conversion circuit 20, and the first video signal conversion circuit 20 converts the video signals of the plurality of cameras 10 to flow through one cable, and transmits the signals to a recording unit including a second video signal conversion circuit 30 and a camera video recording circuit 40 via the single cable, and records the videos of the plurality of cameras 10 in a storage medium 60. For example, a camera unit that is attached to the windshield of a vehicle and includes a camera 10 (first camera) that captures an image in front of the vehicle and another camera 10 (second camera) that captures an image behind the vehicle may be connected to a recording unit provided on the dashboard or under the passenger seat, etc., via a cable.

[0137] For example, the second video signal conversion circuit 30 may convert the video signals of multiple cameras 10 converted by the first video signal conversion circuit 20 and transmitted over a single cable over a relatively long distance into signals for transmission over a relatively short distance, which is necessary for wiring within the recording unit.

[0138] 1 and 4 and 5, for example, the second video signal conversion circuit 30 may be configured to output video signals from a plurality of cameras 10 to the camera video recording circuit 40 so as to flow them over one cable, and these may be input to the camera video recording circuit 40. For example, the second video signal conversion circuit 30 may divide the video signals from the two cameras into packets or the like of a predetermined data size, and transmit the packets or the like alternately. Video signals from a greater number of cameras may be output in an interleaved manner over one transmission path.

[0139] 1 and 6 and 7, for example, the second video signal conversion circuit 30 may be configured to output video signals from a plurality of cameras 10 to the camera video recording circuit 40 so that the video signals are sent to different cables for the respective cameras, and these are input to the camera video recording circuit 40. In this way, processing can be easily performed for each camera 10 in the different transmission paths for the respective cameras 10 or in the camera video recording circuit 40 connected to these. For example, it becomes relatively easy to record and output information separate from the video by superimposing it on each camera frame for each camera 10.

[0140] The camera video recording circuit 40 is relatively inexpensive, but is designed specifically for compressing and storing video, so there is a problem that it is difficult to add functions such as image recognition (e.g., little information disclosure, few remaining resources, etc.) For example, SoCs for drive recorders are inexpensive, but are designed for drive recorders, so there is a problem that it is difficult to add functions such as image recognition (e.g., little information disclosure, few remaining resources, etc.).

[0141] On the other hand, there is a problem that the image recognition circuit 50 used for image recognition does not have a highly reliable software for compressing and recording video. The image recognition circuit may be a general-purpose SoC (SoC for image recognition) that can be used by compiling OSS, etc. In particular, it may be an SoC equipped with an accelerator circuit for image recognition or an SoC equipped with an accelerator circuit for deep learning. There is also a problem that it is costly to mount an image encoder such as H.264 on a general SoC, or especially on an image recognition circuit that is an FPGA. Even if a general SoC has an image encoder, there is a problem that there is a lack of reliable software for drive recorders, especially when it is used as a drive recorder, and there is also a problem that it is extremely costly to create one from scratch in-house.

[0142] In contrast, in the system of this embodiment, the camera video recording circuit 40 and the image recognition circuit 50 are realized as different circuits such as SoCs, and the video signal and synchronization signal are distributed as described above, so that a system that is superior to the conventional systems can be provided.

[0143] Furthermore, the following configuration may be adopted.

[0144] (Camera control example 1) The function for setting the camera 10 (and / or another camera 10) (for example, a part of the ISP function including a part for controlling the camera) may be, for example, a configuration using the one on the camera video recording circuit 40 side (Pattern 1), a configuration using the one on the image recognition circuit 50 side (Pattern 2), or a configuration using both sides by switching (Pattern 3). Also, it may be a configuration using an SoC with an ISP built in, or a configuration using an external ISP. The external ISP may be installed on either the camera video recording circuit 40 side. However, since the image creation required for image recognition is different from the image creation that is compressed and recorded in the storage medium 60 for playback, etc., it becomes an issue as to what state the camera should be controlled in, such as the shutter speed, frame rate, and exposure time. Therefore, the camera video recording circuit 40 may have a function for setting the camera 10.

[0145] In this way, it is possible to prevent conflicts in settings, and to relatively effectively balance the recording of the image from the camera 10 and the image recognition based on the image from the camera 10. For example, the inventors have found that a configuration that adopts the above-mentioned (Pattern 1) produces excellent effects. In many cases, the image used for image recognition is of low quality relative to the image quality required for recording. In particular, it is preferable to have a function for recognizing images by deep learning, and in such a configuration, the requirements for the number of pixels, frame rate, etc. are generally low. As for the camera image recording circuit 40 side, it is particularly preferable to have it within the camera image recording circuit 40.

[0146] However, with regard to the focus and exposure functions of the ISP, it may be better to set the camera in accordance with the requirements for image processing on the image recognition circuit 50 side. Therefore, it is preferable to configure a system in which the camera settings on the camera image recording circuit 40 side are set in advance, taking into consideration the settings related to exposure and focus required on the image recognition circuit 50 side. For example, it is preferable to set the ISP on the camera image recording circuit 40 side (for example, the ISP in the camera image recording circuit 40) in advance, taking into consideration the settings related to exposure and focus required on the image recognition circuit 50 side, and adjusting, for example, within a range in which the image of the camera does not become too bad.

[0147] (Camera control example 2) The camera settings are adjusted in a coordinated manner between the camera image recording circuit 40 and the image recognition circuit 50. In this way, it becomes easier to achieve both the compression and recording of the camera image and the improvement of the accuracy of the image recognition processing.

[0148] As shown in FIGS. 8(a) and 8(b), it is preferable that the image recognition circuit 50 and the camera video recording circuit 40 communicate with each other to grasp the state of the control of the camera 10.

[0149] For example, instead of setting the camera 10 in advance, or in addition to setting the camera 10 in advance, the image recognition circuit 50 generates an exposure or focus adjustment request signal and outputs it to the camera video recording circuit 40. When the camera video recording circuit 40 receives this, the image recognition circuit 50 outputs an exposure or focus adjustment request signal to the camera video recording circuit 40. When the camera video recording circuit 40 receives this, the image recognition circuit 50 adjusts the ISP settings of the camera video recording circuit 40 accordingly. Note that information on the camera settings set on the camera video recording circuit 40 side is transmitted to the image recognition circuit 50. It is particularly preferable that the image recognition circuit 50 receives this and uses it in the image recognition process.

[0150] For example, when recognizing signs using the image recognition circuit 50, the area in the image that corresponds to the shoulder of the road can be calculated as the priority area for exposure calculation. In this case, the camera is basically controlled from the camera image recording circuit 40, but the exposure priority of this area can be set slightly higher in advance, and if the image recognition circuit 50 finds that the sign area is underexposed, the image recognition circuit 50 can output an exposure correction signal to the camera image recording circuit 40, and when the camera image recording circuit 40 receives this signal, it can output a signal to increase the camera exposure to the camera side.

[0151] In this way, it is preferable to configure the camera 10 itself so that the camera video recording circuit 40 is the main controller, and the image recognition circuit 50 does not directly control the camera. It is also preferable to configure the image recognition circuit 50 to output an adjustment signal to the camera video recording circuit 40, and the image recognition circuit 50 receiving this adjustment signal indirectly controls the camera. For example, it is preferable to configure the camera 10 itself so that the camera video recording circuit 40 is the main controller, and the image recognition circuit 50 does not directly control the camera. It is also preferable to configure the camera to control indirectly.

[0152] It is advisable to use a pan focus camera so that focus settings are not required.

[0153] As another example, as the above-mentioned (Pattern 3), the camera control entity may be switched in a time-division manner. In particular, the video recording frame rate may be set to a frame rate sufficient for video recording and playback (for example, a frame rate at a level that does not cause stuttering in the human eye (for example, about 24 to 30 fps)), while the camera frame rate may be set to a higher frame rate (for example, 120 fps), and when the frames output by the camera are numbered in order as "1", "2", "3", "4", "1", "2", "3", "4", etc., "1" may be the camera video recording circuit 40 side and "3" the image recognition circuit 50 side. For example, "2" and "4" may be the switching time.

[0154] The switching may be performed by physically switching both the ISP on the camera image recording circuit 40 side and the ISP on the image recognition circuit 50 side, but it is preferable that the ISP is switched by time sharing the setting of the ISP on the camera image recording circuit 40 side. It is preferable that the camera image recording circuit 40 side controls all of the multiple cameras 1-0, and that the image recognition circuit 50 side does not directly control any of the multiple cameras. It is preferable that they are indirectly controlled.

[0155] The cameras that are indirectly controlled by the image recognition circuit 50 may be a part of a plurality of cameras. These cameras may be a part of cameras where the requirements for recognition are largely different from the requirements for recording and playing back images (for humans to view images).

[0156] In the configuration described in FIG. 8, the image recognition circuit 50 may be replaced with a processing device that performs processing other than image recognition processing.

[0157] The functions of the system 1 described in the above-mentioned embodiment may be realized by one or more programs and by the cooperation of one or more hardware resources. When the functions of the system 1 are realized by using a program, the program for realizing the functions may be provided in a state stored in a computer-readable recording medium such as various magnetic recording media, optical recording media, magneto-optical recording media, and semiconductor memories. The program may also be distributed via a network. The present invention may also be understood as a signal processing method.

[0158] The scope of the present invention is not limited to the configurations expressly described in the specification, but includes combinations of various aspects of the present invention disclosed herein. The configurations of the present invention for which a patent is sought are specified in the appended claims, but it is the intention of the present inventors to include in the claims any configurations disclosed in this specification that are not currently specified in the claims.

[0159] The present invention is not limited to the configurations described in the above-mentioned embodiments. The components of each of the above-mentioned embodiments and modifications may be arbitrarily selected and combined. Any components of each of the embodiments and modifications may be arbitrarily combined with any components described in the means for solving the invention or any components that embody any components described in the means for solving the invention. We intend to acquire rights to these as well through amendments to this application or divisional applications, etc. Furthermore, even if there is a description such as "in the case of..." or "when...", this is not intended to describe a configuration that is limited to that case or time. We also disclose configurations that are not in these cases or times, and we intend to acquire rights to them. Furthermore, the parts described in order are not limited to this order. We also disclose configurations in which some parts are deleted or the order is changed, and we intend to acquire rights to them.

[0160] In addition, we intend to obtain rights to the overall design or partial design by filing a conversion application to a design application. The drawings show the entire device in solid lines, but they include not only the overall design but also partial designs claimed for some parts of the device. For example, it is a drawing that includes some parts of the device as partial designs, regardless of the parts, as well as a drawing that includes some parts of the device as partial designs. A part of the device may be a part of the device, or it may be a part of that part. We intend to obtain rights to not only the overall design, but also partial designs in which any part of the solid line parts of the drawings is drawn as a dashed line part. [Explanation of symbols]

[0161] 1: system 1, 10-1 to 10-4: cameras, 20, 20-1 to 20-4: second video signal conversion circuit, 30: deserializer, 31: first port, 32: second port, 50: image recognition circuit, 60: storage medium, 70: information input unit, 80: display unit, 90: image processing unit, 100: processing device

Claims

1. A system including a plurality of cameras, an SoC for image recognition, and an SoC for a drive recorder, the SoC for the drive recorder receives an input of a video signal indicating an image captured by each of the plurality of cameras, converts the video signal, outputs information obtained by converting the video signal to the SoC for image recognition, and outputs predetermined information received from at least one of an acceleration sensor, a switch, and a GPS module in real time to the SoC for image recognition; the SoC for image recognition receives information obtained by converting the video signal from the SoC for the drive recorder, receives the predetermined information in real time from the SoC for the drive recorder, performs a predetermined image recognition process using the information obtained by converting the video signal and the predetermined information, and outputs information regarding a result of the image recognition process to the SoC for the drive recorder; The SoC for the drive recorder records images based on the image signals output from the plurality of cameras based on the information regarding the results. system.

2. The plurality of cameras includes a first camera mounted on a windshield of the vehicle and a second camera mounted on a rear windshield of the vehicle. The system of claim 1 .

Images