Systems and programs, etc.

Abstract

We will provide systems and programs that are superior to those of the past. [Solution] System 1 comprises cameras 10-1 to 10-4 and a camera video recording circuit 40 that compresses the images captured by cameras 10-1 to 10-4 and stores them in a storage medium 60. The second video signal conversion circuit 30 distributes the video signals S1 and S2 transmitted between cameras 10-1 to 10-4 and the camera video recording circuit 40 to an image recognition circuit 50 that implements an image recognition function based on the video signals. When the camera video recording circuit 40 receives a trigger signal TG according to the result of recognition by the image recognition function, it records the images from cameras 10-1 to 10-4 in the storage medium 60.

Description

Technical Field

[0001] The present invention relates to systems, programs, and the like.

Background Art

[0002] Conventionally, a system equipped with a camera and recording the video of the camera has been known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For example, in a system having a function of recording the video of such a camera, there were various problems. One object of the present invention is to provide a system and the like that are superior to the prior art.

[0005] The object of the invention of the present application is not limited to this, and the applicant also has the intention of obtaining rights by divisional application, amendment, etc. for a configuration aimed at obtaining an effect resulting from a part of the configuration disclosed in this specification and drawings. For example, the problem of reading the part described as "can be" in this specification as "is a problem" is disclosed in this specification. The problems are described as independent ones, and the applicant also has the intention of obtaining rights by divisional application, amendment, etc. alone for the configuration for solving each problem. Even if the problem is implicitly grasped from the description of the specification, the applicant has the intention of making a part of the configuration described in this specification the scope of the claims by amendment or divisional application. Also, a configuration for solving a problem combining these independent problems is disclosed, and the applicant has the intention of obtaining rights.

Means for Solving the Problems

[0006] The present invention can be understood, for example, as an invention in the following embodiments. (1) A system comprising a camera and a camera video recording circuit that compresses images captured by the camera and stores them in a storage means, wherein the system comprises a signal distribution means that distributes signals transmitted between the camera and the camera video recording circuit to electronic components that perform predetermined functions based on the signals.

[0007] In this way, the image captured by the camera can be compressed and stored in a storage means, and the electronic component that performs a predetermined function can perform that function based on the signal distributed between the camera and the camera image recording circuit.

[0008] The camera can be of various types, but it can be a CCD (Charge Coupled Device) camera, and a CMOS (Complementary Metal-Oxide-Semiconductor) camera is particularly preferable. The camera can be a camera for capturing still images, but a camera for capturing video is particularly preferable. The camera can be a special camera such as an infrared camera, but a camera that captures visible light is preferable, and a visible light camera is particularly preferable. The camera can be a camera that can produce monochrome images, but a camera that can produce color images is preferable. The camera can be, for example, a camera with red (R), green (G), and blue (B) color filters arranged on a plane. The resolution of the camera should be VGA or higher.

[0009] The term "camera" may refer to the part that includes, for example, an image sensor, or it may refer to the image sensor itself.

[0010] In particular, the cameras should be cameras installed on the vehicle. Specifically, the cameras should be cameras that are added after the vehicle has been shipped by the manufacturer.

[0011] The video compression section of the camera video recording circuit can use various algorithms, but it is preferable to use an encoder with a standardized algorithm, and it is especially preferable to have at least one of either an H.264 encoder or an H.265 encoder. In particular, the video encoder of the camera video recording circuit should be hardware-based rather than software-based. The encoder should preferably be a System-on-a-Chip (SoC) rather than a simple FPGA (Field-Programmable Gate Array). Thus, the camera video recording circuit may be a camera video recording chip. It is preferable that the camera video recording circuit is composed of hard macros rather than soft macros.

[0012] The camera video recording circuit is preferably a chip sold specifically for camera video recording. The camera video recording circuit can be a Large Scale Integrated Circuit (LSI), such as an ASSP or SoC for camera video recording, and is particularly well-suited to being an SoC for dashcams.

[0013] The storage means is preferably a removable storage means, and is particularly preferably a card-shaped medium such as an SD® card or a microSD® card.

[0014] The signals transmitted between the camera and the camera video recording circuit may be control signals, but it is preferable that they include at least one of either a video signal or a video synchronization signal. The video synchronization signal may be at least one of a pixel clock, a vertical synchronization signal, or a horizontal synchronization signal, and it is preferable that it includes at least two of these. In particular, it is preferable that it includes a pixel clock and a vertical synchronization signal.

[0015] The video signal transmitting camera footage may be transmitted serially, synchronized on a per-video-channel basis, or in parallel. Alternatively, a combination of serial and parallel transmission may be used. In particular, it is desirable to superimpose data such as the time, event information, and sensor information onto each video frame, or to superimpose it onto the video itself. Data such as the time, event information, and sensor information may be superimposed onto synchronized units, or to superimpose it onto the video itself.

[0016] If the system is designed to distribute video signals, it is preferable to, for example, input a video signal of a predetermined protocol and distribute it to multiple output destinations using the same protocol as the input signal. For example, if a camera outputs a MIPI CSI-2 signal as a video signal, the signal distribution means should be a MIPI CSI-2 signal splitter chip.

[0017] When distributing video signals, it is preferable to, for example, input a video signal of a predetermined protocol and output a signal of the same protocol to multiple output destinations, even if the protocol is different from the input protocol. For example, if a camera outputs an FPD-LINK III signal as a video signal, the signal distribution means should be a circuit equipped with ports that input an FPD-LINK III signal and output it as, for example, multiple MIPI CSI-2 signals (e.g., a dual-port output deserializer hub with 1 input and 2 outputs). In particular, the ports that output as MIPI CSI-2 signals should have the function of outputting a replica of one port to another port. A deserializer hub chip is particularly suitable for this purpose.

[0018] The electronic component that implements the predetermined function may include at least one of either another camera or an image recognition circuit. When the electronic component that implements the predetermined function includes a camera, the predetermined function may be a function that captures an image of a different area, including a range different from that of the aforementioned camera. When the electronic component that implements the predetermined function includes an image recognition circuit, the predetermined function may be an image recognition function.

[0019] The signal distribution means may, for example, simply branch the wiring, but it is preferable to have a function that distributes the signals so that they are synchronized between the multiple distributed signal lines, and it is especially preferable to have a circuit (for example, a chip) that has a function that distributes the signals so that they are synchronized between the multiple distributed signal lines.

[0020] The camera and camera video recording circuit may be housed in the same enclosure, but it is preferable to house them in separate enclosures and connect them using cables, connectors, or other means of connection. The camera and camera video recording circuit should be connected via a cable of 1 meter or longer. In particular, the camera and camera video recording circuit should be connected via a cable routed within the vehicle.

[0021] (2) The electronic component may have another camera different from the camera, and the signal may include a synchronization signal of the video generated by the synchronization signal generation means, and based on the distributed synchronization signal, the video signals of the camera and the other camera may be output to the camera video recording circuit.

[0022] In this way, the camera video recording circuit can obtain synchronized video signals between one camera and another. For example, the camera video recording circuit can input synchronized video signals between one camera and another, and then compress and record video based on these multiple synchronized video signals. For example, when the recorded video signals are played back simultaneously, multiple synchronized videos can be viewed. Note that video compression can be performed on separate files for each of the multiple cameras, but it is preferable to perform it on a single file.

[0023] (3) The camera may have the synchronization signal generation means, and the signal distribution means may be configured to distribute the synchronization signal to the other camera.

[0024] By doing so, the video signal output from the other camera becomes a video signal based on the synchronization signal generated in the camera, making it easier to synchronize the two cameras. In particular, it is preferable to adopt this configuration and to configure the camera and the other camera to be provided in the same housing. This exhibits particularly excellent effects when the wiring distance of the synchronization signal is short.

[0025] As for the camera, it is particularly preferable that the synchronization signal is generated in the imaging device. By doing so, it is not necessary to use an externally synchronizable imaging device that captures images in synchronization with the input synchronization signal, and a low-cost and simple configuration can be achieved.

[0026] (4) It is preferable that the synchronization signal generation means is provided outside the camera and the other camera, and the signal distribution means distributes the synchronization signal to the camera and the other camera.

[0027] By doing so, it is easy to synchronize the video signal output from the other camera with the video signal output from the other camera.

[0028] In particular, it is preferable to adopt this configuration and to configure the camera and the other camera to be provided in different housings. This exhibits particularly excellent effects when the wiring distance of the synchronization signal is relatively long.

[0029] (5) It is preferable that the camera video recording circuit has the synchronization signal generation means, and the signal distribution means distributes the synchronization signal to the camera and the other camera.

[0030] By doing so, the possibility of obtaining synchronized video signals from the camera and the other camera at a timing suitable for video recording processing and the like can be increased.

[0031] (6) The system comprises a first video signal conversion circuit having a first video signal conversion means for converting signals output from the camera and the other camera into video signals for transmission via a cable, and a second video signal conversion circuit having a second video signal conversion means for converting the video signals transmitted via the cable into video signals that can be input to the camera video recording circuit, wherein the synchronization signal generation means is provided within the first video signal conversion circuit or the second video signal conversion circuit, and the signal distribution means is provided within the first video signal conversion circuit or the second video signal conversion circuit, and the synchronization signal is distributed to the camera and the other camera.

[0032] In this way, synchronized video signals from both the aforementioned camera and the other camera can be easily input to the camera video recording circuit via a cable and recorded.

[0033] The cable should be relatively long. Relatively long means a distance longer than the length required for wiring within the camera unit, which includes at least one of the camera or the other camera, and is particularly good if it is 1 meter or more.

[0034] The first video signal conversion means may be, for example, a serializer, and the second video signal conversion means may be, for example, a deserializer. The first video signal conversion circuit may be a serializer chip, and the second video signal conversion circuit may be a deserializer chip. The transmission distance between the first video signal conversion circuit and the second video signal conversion circuit should be such that a transmission of several meters is possible, especially if the camera and another camera are in-vehicle cameras.

[0035] Alternatively, the system may include a first camera unit having the camera and a first video signal conversion circuit, a second camera unit having the other camera and a second first video signal conversion circuit, and a recording unit having the second video signal conversion circuit and a camera video recording circuit, with the first camera unit and the recording unit connected by a first cable, and the second camera unit and the recording unit connected by a second cable, to transmit the respective video signals. For example, the system may include a first camera unit in which the camera (first camera) mounted on the windshield of the vehicle is positioned to capture images of the area in front of the vehicle, and a second camera unit mounted on the rear window of the vehicle is positioned to capture images of the area behind the vehicle, with a recording unit located on the dashboard or under the passenger seat connected to both camera units by separate cables. Alternatively, the recording unit and the first camera unit may be configured as a single unit.

[0036] Alternatively, for example, a first camera unit having the camera and a first video signal conversion circuit and a second camera unit having the other camera and a second first video signal conversion circuit may be connected by a first cable, and a recording unit having a second video signal conversion circuit and a camera video recording circuit may be connected to the second camera unit by a second cable, thereby transmitting the respective video signals. Alternatively, for example, a single camera unit may be equipped with multiple cameras, including the aforementioned camera and the other camera, and a first video signal conversion circuit. The first video signal conversion circuit converts the video signals from these multiple cameras to flow through a single cable, and transmits this signal via the single cable to a recording unit equipped with a second video signal conversion circuit and a camera video recording circuit, so that the recording means can record the images from the multiple cameras. For example, a camera unit mounted on the windshield of a vehicle may be equipped with the aforementioned camera (first camera) that images the front of the vehicle and the other camera (second camera) that images the rear of the vehicle, and a recording unit provided on the dashboard or under the passenger seat may be connected by a cable.

[0037] For example, the second video signal conversion circuit may convert the video signals from multiple cameras, which have been converted by the first video signal conversion circuit and transmitted over a relatively long distance via the aforementioned single cable, into signals suitable for transmission over a relatively short distance, which are necessary for wiring within the recording unit.

[0038] For example, the second video signal conversion circuit may be configured to output video signals from multiple cameras to the camera video recording circuit so that they flow through a single transmission path, which the camera video recording circuit then inputs. For example, the second video signal conversion circuit may divide the video signals from two cameras into packets of a predetermined data size and transmit the packets alternately. Alternatively, it may interleave the video signals from a larger number of cameras into a single transmission path and output them.

[0039] Alternatively, for example, the second video signal conversion circuit may be configured to output the video signals from multiple cameras to the camera video recording circuit, sending them through different transmission paths for each camera, which the camera video recording circuit then inputs. This makes it easier to process each camera individually within the different transmission paths for each camera or within the camera video recording circuit connected to them. For example, it becomes relatively easy to overlay and record or output information other than the video on a per-camera frame basis for each camera.

[0040] A single cable should ideally have as few signal lines as possible; for example, twisted-pair cables are a good choice.

[0041] The transmission path should ideally consist of as few signal lines as possible, such as a few signal lines (e.g., printed circuit board wiring). The connection between the camera and the other camera, as well as the other cameras, and the first video signal conversion circuit should be made via at least one of printed circuit board wiring or a flexible cable. The connection between the second video signal conversion circuit and the camera video recording circuit should be made via at least one of printed circuit board wiring or a flexible cable.

[0042] (7) The lengths of the signal wiring from the signal distribution means to the camera and the other camera should be approximately the same.

[0043] In this way, the likelihood of the video signal from the aforementioned camera and the video signal from the other camera being synchronized can be increased.

[0044] (8) The electronic component includes an image recognition circuit that performs predetermined image recognition processing and is provided separately from the camera image recording circuit, wherein the transmitted signal includes a video signal, and the image recognition circuit outputs information regarding the result of the image recognition processing performed based on the distributed video signal to the camera image recording circuit, and the camera image recording circuit compresses the video captured by the camera based on the information regarding the result and stores it in the storage means.

[0045] This method ensures reliable compression of camera footage and storage in a memory device, and also allows for recording based on the results of image recognition processing of the same camera footage.

[0046] Camera video recording circuits are relatively inexpensive, but because they are designed specifically for video compression and storage, it is difficult to add functions such as image recognition (for example, limited information disclosure, limited remaining resources, etc.). For example, SoCs for dashcams are inexpensive, but because they are designed for dashcams, it is difficult to add functions such as image recognition (for example, limited information disclosure, limited remaining resources, etc.).

[0047] On the other hand, image recognition circuits used for image recognition have a problem in that there is no reliable software for video compression and recording. It would be good to use a general-purpose SoC (System-on-a-Chip) that can be used by compiling OSS (Open Source Software), for example, for image recognition circuits. It would be particularly good to use an SoC equipped with an image recognition accelerator circuit or an SoC equipped with a deep learning accelerator circuit. However, there is also the problem that it is costly to put a video encoder such as H.264 on a general SoC, or especially an FPGA image recognition circuit. Furthermore, even if a general SoC is equipped with a video encoder, when it comes to adopting it, especially for a dashcam, there is a lack of reliable software for dashcams, and developing it from scratch in-house would be extremely costly.

[0048] For example, the same camera video signal can be distributed and input to both the camera video recording circuit and the image recognition circuit using a MIPI CSI-2 splitter (e.g., a splitter chip). The camera video recording circuit and the image recognition circuit are connected by a communication line. The image recognition circuit recognizes the state of the object, and if the recognition results in a situation requiring recording, it sends a trigger signal to the camera video recording circuit via this communication line. When the camera video recording circuit receives the trigger signal from the image recognition circuit, it should record the video signal it is receiving as an event in a storage means.

[0049] The communication line may be a parallel or serial communication line to transmit trigger data. For example, information about the type of recognized object may be transmitted as this data. Alternatively, it may be a simple GPIO (General-purpose input / output) communication line. For example, the trigger input port of the drive recorder SoC may be connected to the GPIO output port of the image recognition SoC. When the image recognition SoC determines that recording is necessary as a result of image recognition, the signal at the GPIO output port of the image recognition SoC is changed. The drive recorder SoC, upon detecting the change in the signal at its trigger input port, may record video for a predetermined number of seconds before and after the trigger in the recording means. Furthermore, signals from sensors connected to the camera video recording circuit may also be transmitted to the image recognition circuit. For example, if the camera video recording circuit is an SoC for a dashcam and the image recognition circuit is an SoC for image recognition, then the accelerometer, switch, and GPS module are connected to the dashcam SoC, and the information received by the dashcam SoC from the accelerometer, switch, and GPS module is output to the image recognition SoC in real time. The image recognition SoC receives this information from the dashcam SoC in real time, uses it for image recognition, and outputs the results back to the dashcam SoC. The functions of both SoCs can be realized by executing programs stored in each SoC that implement these functions.

[0050] For example, the signals from each of the four cameras are serialized by a serializer and input to a deserializer hub. The deserializer hub outputs interleaved serial signals from the same four cameras to its first and second ports, respectively. For example, a camera video recording circuit could be connected to the first port and an image recognition circuit to the second port.

[0051] For example, in an image recognition circuit, if it detects that a vehicle has entered beyond a "no entry" sign despite the circuit recognizing the sign, it outputs a "no entry" trigger to the camera video recording circuit. When the camera video recording circuit receives the "no entry" trigger from the image recognition circuit, it encodes the video footage from 15 seconds before and after the trigger as "no entry" trigger footage and records it in a storage device.

[0052] Furthermore, for example, if the image recognition circuit detects that a person has entered the vicinity of a forklift, it outputs a surrounding person recognition trigger to the camera video recording circuit. For example, when the camera video recording circuit receives the surrounding person recognition trigger from the image recognition circuit, it encodes the video footage for 30 seconds before and after the trigger as the surrounding person recognition trigger video and records it in a storage means.

[0053] For example, the signals from four interleaved cameras are separated into video feeds from each of the four cameras by both the image recognition circuit and the camera video recording circuit. The image recognition circuit may determine whether to output a trigger from any one of the video feeds from the four cameras, or it may output a trigger if a trigger-required condition is recognized in any of the video feeds. Alternatively, it may determine whether to apply a trigger based on all of the video feeds combined. It would be desirable to allow the system to configure which of these approaches to adopt.

[0054] It is particularly important to synchronize the deserializer hub, camera video recording circuit, and image recognition circuit by providing a synchronization signal. Suitable synchronization signals include pixel clocks and vertical synchronization signals. The deserializer hub should output a synchronized clock to the four serializers. Alternatively, the deserializer hub can generate a clock, which can then be provided to the camera video recording circuit and image recognition circuit for synchronization. The image recognition circuit may be an FPGA instead of an SoC, or it may be any other type of computer.

[0055] The example above describes a system where the deserializer hub has MIPI CSI-2 branching functionality, but the branching device can also be external.

[0056] The function for configuring the aforementioned camera (and / or the aforementioned other camera) (for example, the part of the ISP (Image Signal Processor) function that includes the camera control part) may be configured to use the camera video recording circuit side (Pattern 1), the image recognition circuit side (Pattern 2), or a configuration that switches between using the functions of both sides (Pattern 3). Furthermore, each configuration may use either an SoC with a built-in ISP or an external ISP. The external ISP may be installed on either the camera video recording circuit side or the other side. However, since the image processing required for image recognition and the image processing required to compress and record on the recording means for playback are different, the question arises as to how to control the camera state, such as shutter speed, frame rate, and exposure time. In this regard, (9) is particularly recommended.

[0057] (9) The camera video recording circuit may perform the camera settings.

[0058] This approach prevents configuration conflicts and allows for a relatively smooth balance between recording camera footage and image recognition based on that footage. For example, the inventors have found that adopting the configuration described above (Pattern 1) yields excellent results. The images used for image recognition are often of lower quality than those required for recording. It is particularly beneficial to have a function that recognizes images using deep learning, and in such a configuration, the requirements for pixel count, frame rate, etc., are generally low. On the camera footage recording circuit side, it is especially good to implement this within the camera footage recording circuit itself.

[0059] However, regarding the focus and exposure functions of the ISP, it is sometimes better for the image recognition circuit to configure the camera according to the requirements for image processing. Therefore, it is advisable to have a system where the camera settings on the camera video recording circuit are configured in advance, taking into account the exposure and focus settings required by the image recognition circuit. For example, the ISP settings on the camera video recording circuit (for example, the ISP within the camera video recording circuit) can be configured in advance, taking into account the exposure and focus settings required by the image recognition circuit, and adjusting them within a range where the camera image does not become too poor.

[0060] (10) It is preferable to coordinate the settings of the camera between the camera video recording circuit and the image recognition circuit.

[0061] This approach makes it easier to achieve both compression and recording of camera footage and improved accuracy in image recognition processing.

[0062] The image recognition circuit and the camera video recording circuit should communicate with each other and have a function to mutually understand the control status of the camera.

[0063] For example, instead of pre-setting the camera settings, or while pre-setting them, the image recognition circuit generates exposure and focus adjustment request signals and outputs them to the camera video recording circuit. The camera video recording circuit then receives these signals and performs the settings according to its own setting function. For example, instead of pre-setting the camera settings, or while pre-setting them, the image recognition circuit outputs exposure and focus adjustment request signals to the camera video recording circuit. The camera video recording circuit then adjusts its ISP settings accordingly. It is particularly preferable to transmit information regarding the camera settings to be set by the camera video recording circuit to the image recognition circuit, which then receives and uses this information in its image recognition processing.

[0064] For example, when an image recognition circuit recognizes a sign, it is good practice to calculate the area corresponding to the road shoulder in the image as the priority area for exposure calculation. In this case, the camera is basically controlled from the camera video recording circuit side, but it is good practice to set the exposure priority of this area to be slightly higher beforehand, and if the exposure of the sign area is underexposed by the image recognition circuit, the image recognition circuit should output an exposure compensation signal to the camera video recording circuit, and when the camera video recording circuit receives this signal, it should output a signal to the camera to increase the camera's exposure.

[0065] In this way, the camera's control is primarily handled by the camera video recording circuit, and the image recognition circuit should not directly control the camera. Furthermore, the image recognition circuit should output an adjustment signal to the camera video recording circuit, which then indirectly controls the camera. For example, the camera's control is primarily handled by the camera video recording circuit, and the image recognition circuit should not directly control the camera. It should also be configured to control the camera indirectly.

[0066] Regarding focus settings, it's best to set the camera to pan-focus and disregard any further settings.

[0067] As another example, as mentioned above (Pattern 3), the camera control entity could be switched using time-division multiplexing. In particular, the video recording frame rate should be sufficient for recording and playing back video (for example, a frame rate that does not cause stuttering for human vision (e.g., around 24 to 30 fps)), while the camera frame rate should be higher (e.g., 120 fps). When the frames output by the camera are numbered sequentially as "1", "2", "3", "4", "1", "2", "3", "4"..., "1" should be from the camera video recording circuit side, and "3" should be from the image recognition circuit side. For example, "2" and "4" could be the switching time.

[0068] The switching could be done by physically switching both the ISP on the camera video recording circuit and the ISP on the image recognition circuit, but it is best to have the ISP settings on the camera video recording circuit be switched in a time-division multiplexer manner. All cameras should be controlled by the camera video recording circuit, and the image recognition circuit should not directly control any of the cameras. It is best to control them indirectly.

[0069] The cameras that the image recognition circuit indirectly controls should be a subset of the multiple cameras. These subsets should be those where there is a large discrepancy between the requirements necessary for recognition and the requirements necessary for image recording and playback (for humans to view the images).

[0070] One camera is sufficient. For example, you could use a MIPI CSI-2 splitter to split the camera signal into two.

[0071] (11) The camera video recording circuit is provided with a recognition video signal generation means that generates a signal with reduced information content than the video signal input to the camera video recording circuit, and the image recognition circuit is configured to perform the image recognition processing based on the video signal with reduced information content.

[0072] In this way, the amount of information handled by the image recognition circuit can be reduced in advance, allowing recognition processing to be performed at a lower cost and faster, while also enabling the video to be compressed and recorded on the recording device with excellent image quality.

[0073] For example, the signal from the camera could be routed directly into the camera video recording circuit without branching, and the camera video recording circuit could reduce the amount of information in the video signal from the camera by thinning it out, before outputting it to the image recognition circuit via another port such as a USB port. Image thinning could be done by, for example, thinning out pixels or thinning out frames. Generally, in image recognition, especially in the case of deep learning, the amount of image information required for recognition can be significantly smaller than what a human can see when viewing an image.

[0074] In this example, the decimation is performed in the camera video recording circuit, but this may increase the load on the camera video recording circuit or lead to insufficient resources. Furthermore, it may not be possible to modify the camera video recording circuit at all. Therefore, it is advisable to split the video signal from the camera, feeding one branch directly into the video recording circuit, while inputting the decimated video signal to the image recognition circuit. For example, this function can be implemented using an FPGA. Alternatively, instead of an FPGA, a standalone ISP chip could be used, and the output destination could be switched frame by frame using the ISP chip's output port switching function.

[0075] (12) The image recognition circuit may record information based on the video signal distributed from the signal distribution means in the second recording means.

[0076] In this way, the second recording means records information based on the video signal distributed from the signal distribution means.

[0077] The second recording means may be a recording means separate from the first recording means. In particular, it may be a removable recording medium separate from the second recording means. The information based on the video signal distributed from the distribution means may be information obtained by processing the video signal that has been distributed and input to the image recognition circuit. For example, video information may be recorded that surrounds the recognized object and overlays its attributes as text information. Alternatively, along with or instead of such video information, a trigger signal sent to the camera video recording circuit may be recorded along with its time. For example, information regarding the coordinates and type of the recognized object may be recorded in association with its time.

[0078] The inventions described in (1) to (12) above can be combined in any way. For example, one may combine all or part of the configuration of the invention described in (1) with at least part of the configuration of at least one of the inventions described in (2) and onward. In particular, it is preferable to have an invention that combines the invention described in (1) with at least part of the configuration of at least one of the inventions described in (2) and onward. It is especially preferable to have a configuration that includes both the configuration of (2) and the configuration of (8). Alternatively, one may extract any configuration from the inventions described in (1) to (12) and combine the extracted configurations. The applicant of this application intends to obtain rights to inventions that include these configurations. Furthermore, even if there is a description such as "in the case of..." or "when...", it is not meant to be a configuration that is limited to that case or time. These are merely examples of better configurations, and the applicant intends to obtain rights to configurations that do not fall under these cases or times. Also, even if there is a sequence of descriptions, it is not limited to that order. Configurations with some parts deleted or the order rearranged are also disclosed, and the applicant intends to obtain rights to them as well. [Effects of the Invention]

[0079] According to the present invention, it is possible to provide a system that is superior to conventional systems.

[0080] The effects of the present invention are not limited thereto, and the effects produced by the components of the structure disclosed in this specification and the drawings are also disclosed. The present invention intends to obtain rights to the components that produce such effects through divisional applications, amendments, etc. For example, the phrases "can do..." in this specification are descriptions that specify the effects produced, and there are components that produce effects even without such descriptions. Furthermore, there are effects that can be grasped by the component even without such descriptions. [Brief explanation of the drawing]

[0081] [Figure 1] This block diagram shows the configuration of a system according to one embodiment. [Figure 2] This is a block diagram showing a variation of the system configuration. [Figure 3] This is a block diagram showing a variation of the system configuration. [Figure 4] This diagram shows an example of connecting a camera and a processing unit. [Figure 5] This diagram shows an example of connecting a camera and a processing unit. [Figure 6] This diagram shows an example of connecting a camera and a processing unit. [Figure 7] This diagram shows an example of connecting a camera and a processing unit. [Figure 8] This figure shows an example of a connection between a processing unit and a recording device. [Modes for carrying out the invention]

[0082] The embodiments will be described in detail below with reference to the drawings. The embodiments shown below are examples of embodiments of the present disclosure, and the present disclosure is not limited to these embodiments. In the drawings referenced in these embodiments, the same parts or parts having similar functions are denoted by the same or similar reference numerals (simply a number followed by A, B, etc.), and repeated explanations may be omitted. In addition, in the drawings referenced in the following description, the scale may be different from that of the actual parts in order to make each component, area, etc. recognizable.

[0083] Figure 1 is a block diagram showing the configuration of System 1 according to one embodiment. System 1 may be an in-vehicle device mounted on a vehicle, and in particular may be an in-vehicle device that functions as a drive recorder. The vehicle may be a forklift, but it may also be a work vehicle other than a forklift, or a vehicle other than a worker's vehicle (for example, a private or official passenger car, bus, or train).

[0084] The following sections will first describe the transmission of video signals and processing using video signals in System 1, and then describe the transmission of synchronization signals and processing using synchronization signals.

[0085] System 1 includes cameras 10-1 to 10-4, first video signal conversion circuits 20-1 to 20-4, a second video signal conversion circuit 30, a camera video recording circuit 40, an image recognition circuit 50, a storage medium 60, an information input unit 70, and a display unit 80. Hereinafter, when there is no need to distinguish between cameras 10-1 to 10-4, they will be collectively referred to as "camera 10," and when there is no need to distinguish between first video signal conversion circuits 20-1 to 20-4, they will be collectively referred to as "first video signal conversion circuit 20." In this embodiment, there are four cameras 10 and four first video signal conversion circuits 20, which handle video signals from four video channels. However, there may be three or fewer cameras 10 and five or more first video signal conversion circuits 20, which handle video signals from three or fewer or five or more video channels. Each circuit of System 1 described above is implemented by, for example, physically separated circuits (e.g., chips), but two or more circuits may also be implemented by circuits fabricated on the same substrate (e.g., the same chip).

[0086] Camera 10 is an electronic component that captures images and generates a video signal representing the captured image. Camera 10 can be various types of cameras, but it can be a CCD camera, and is particularly good as a CMOS camera. Camera 10 can be a camera for capturing still images, but is particularly good as a camera for capturing video. Camera 10 can be a special camera such as an infrared camera, but is best suited to capturing visible light (for example, light in the wavelength range of 400 to 700 nm), and is particularly good as a visible light camera. Camera 10 may be a camera that produces monochrome images, but is best suited to producing color images. Camera 10 may be, for example, a camera with red (R), green (G), and blue (B) color filters arranged on a plane. The resolution of the camera should be VGA or higher. Camera 10 may refer to, for example, the part equipped with the image sensor, or it may refer to the image sensor itself. Camera 10 is particularly good as a camera installed in a vehicle. Camera 10, in particular, should be a camera that is added after the vehicle is shipped from the manufacturer.

[0087] In this embodiment, the video signal generated and output by the camera 10 is digital data that includes the grayscale value of each pixel of a plurality of pixels. The video signal may contain 8 bits (256 grayscales) of data for each of the red (R), green (G), and blue (B) color components, but the number of color components and grayscales are not limited to this.

[0088] Camera 10-1 is connected to the first video signal conversion circuit 20-1 in a communicative manner. Camera 10-2 is connected to the first video signal conversion circuit 20-2 in a communicative manner. Camera 10-3 is connected to the first video signal conversion circuit 20-3 in a communicative manner. Camera 10-4 is connected to the first video signal conversion circuit 20-4 in a communicative manner. Each of the cameras 10-1 to 10-4 and each of the first video signal conversion circuits 20-1 to 20-1 may be connected by either a wired or wireless transmission line, but it is preferable to use a wired transmission line. In the example in Figure 1, camera 10-1 outputs video signals A1, A2, A3, and A4 in chronological order. Camera 10-2 outputs video signals B1, B2, and B3 in chronological order. Camera 10-3 outputs video signals C1, C2, C3, and C4 in chronological order. Camera 10-4 outputs video signals D1 and D2 in chronological order. Cameras 10-1 to 10-4 output video signals in ascending order of the number appended to "A" to "D". Video signals with the same number are output simultaneously by cameras 10-1 to 10-4. For example, video signals A1, B1, C1, and D1 are output simultaneously. Each of the video signals A1 to A4, B1 to B3, C1 to C4, D1, and D2 is a video signal of a predetermined unit, but it is preferable that they represent, for example, one frame (also called one picture) of video.

[0089] Cameras 10-1 to 10-4 should each capture different areas. The area captured by camera 10 may include, for example, either the interior and exterior of the vehicle, or both. The exterior area may be the front (front), the front right, the right (side), the rear right, the rear (directly behind), the rear left, the left (side), and the front left, but other areas may also be included. Cameras 10-1 to 10-4 should, for example, include at least a camera mounted on the vehicle's windshield (first camera) and a camera installed to capture the rear of the vehicle (second camera).

[0090] The first video signal conversion circuit 20 includes a first video signal conversion means that receives a video signal input from the camera 10 and converts it into a serial video signal. The first video signal conversion circuit 20 may be a serializer, particularly a serializer chip. A serial video signal is a signal obtained by converting the video signal input from the camera 10 into a serial data sequence. The first video signal conversion circuit 20-1 converts the video signal from camera 10-1, the first video signal conversion circuit 20-2 converts the video signal from camera 10-2, the first video signal conversion circuit 20-3 converts the video signal from camera 10-3, and the first video signal conversion circuit 20-4 converts the video signal from camera 10-4. The first video signal conversion circuits 20-1 to 20-4 output the interleaved converted video signals to the second video signal conversion circuit 30. Each of the first video signal conversion circuits 20-1 to 20-4 and the second video signal conversion circuit 30 may be connected by wire or wireless means, but it is preferable that they be connected using a single wired cable. The first video signal conversion circuits 20-1 to 20-4 convert the signal into a video signal for transmission via such a cable. This cable should be a relatively long cable. A relatively long cable is one that is longer than the length required for wiring within a camera unit that includes at least one of the cameras 10 or another camera 10, and is particularly preferable to be 1 meter or more.

[0091] The transmission distance between the first video signal conversion circuit 20 and the second video signal conversion circuit 30 should be such that, in particular, if the camera 10 is an in-vehicle camera, it is desirable to use a circuit that can transmit signals over a distance of several meters.

[0092] The second video signal conversion circuit 30 includes a second video signal conversion means that converts the video signals from cameras 10-1 to 10-4, transmitted via cables from each of the first video signal conversion circuits 20-1 to 20-4, into video signals that can be input to the camera video recording circuit 40. The second video signal conversion circuit 30 may be a deserializer, particularly a deserializer chip. The second video signal conversion circuit 30 also functions as a signal distribution means that distributes the video signals transmitted between cameras 10-1 to 10-4 and the camera video recording circuit 40 to the image recognition circuit 50. For example, the second video signal conversion circuit 30 may have a MIPI CSI-2 branching function. In this way, the video captured by camera 10 can be compressed and stored in the storage medium 60, and the image recognition circuit 50 can realize an image recognition function based on the distributed signals between camera 10 and camera video recording circuit 40.

[0093] The second video signal conversion circuit 30 has a first port 31 and a second port 32. The camera video recording circuit 40 is communicated to the first port 31, and the image recognition circuit 50 is communicated to the second port 32. Such a second video signal conversion circuit 30 can also be identified as a deserializer hub.

[0094] The second video signal conversion circuit 30 and the camera video recording circuit 40 and image recognition circuit 50 may be connected by wire or wireless, but it is preferable to connect them using wired cables. Based on the interleaved video signals from cameras 10-1 to 10-4, the second video signal conversion circuit 30 outputs video signal S1 to the camera video recording circuit 40 and video signal S2 to the image recognition circuit 50. The second video signal conversion circuit 30 distributes the video signals from cameras 10-1 to 10-4 by outputting the same image signals S1 and S2 from the first port 31 and the second port 32 at the same timing. The second video signal conversion circuit 30 can, for example, input a video signal of a predetermined protocol and distribute and output it to multiple output destinations using the same protocol as the input protocol. For example, if camera 10 outputs a MIPI CSI-2 signal as a video signal, the second video signal conversion circuit 30 can be a MIPI CSI-2 signal splitter chip. The second video signal conversion circuit 30 may, for example, input a video signal of a predetermined protocol and output a signal of the same protocol to multiple output destinations, even if the protocol is different from the input protocol. For example, if the camera 10 outputs an FPD-LINK III signal as a video signal, the second video signal conversion circuit 30 may be a chip equipped with ports that input an FPD-LINK III signal and output it as multiple MIPI CSI-2 signals (for example, a dual-port output deserializer hub with one input and two outputs). In particular, the ports that output as MIPI CSI-2 signals should have the function of outputting a replica of one port to another port. A deserializer hub chip is particularly suitable.

[0095] In System 1, a single cable should preferably have as few signal lines as possible, for example, a twisted-pair cable. The transmission path should preferably have as few signal lines as possible, for example, a few signal lines (for example, printed circuit board wiring) should be used. Camera 10 and multiple cameras 10, including another camera 10, and the first video signal conversion circuit 20 should be connected via at least one of printed circuit board wiring or a flexible cable. The second video signal conversion circuit 30 and the camera video recording circuit 40 should be connected via at least one of printed circuit board wiring or a flexible cable.

[0096] The second video signal conversion circuit 30 selects the video signals from cameras 10-1, 10-2, 10-3, and 10-4 in a predetermined order and outputs the selected video signals from the first port 31 and the second port 32, respectively. In the example shown in Figure 1, the second video signal conversion circuit 30 outputs video signals S1 and S2 by switching the video signals in the order of cameras 10-1, 10-2, 10-3, 10-4, 10-1, 10-2, 10-3, 10-4, etc., on a frame-by-frame basis. Specifically, the second video signal conversion circuit 30 outputs video signals S1 and S2 from the first port 31 and the second port 32, respectively, in the order of video signals A1, B1, C1, D1, A2, B2, C2, D2, A3, B3, C3, A4, C4. Furthermore, the 30th unit does not necessarily have a function for distributing video signals (MIPI CSI-2 splitting function), and in this case, an external splitter may be used.

[0097] The camera video recording circuit 40 has the function of compressing the video signal S1 from the first port 31 and storing it in the storage medium 60. The video compression unit (not shown) provided in the camera video recording circuit 40 can use various algorithms, but it is particularly good to use an encoder with a standardized algorithm, and it is especially good to have at least one of either an H.264 encoder or an H.265 encoder. In particular, it is good that the video encoder provided in the camera video recording circuit 40 is implemented in hardware rather than being processed in software. The camera video recording circuit 40 may be an FPGA, but it is especially desirable to use an SoC. Furthermore, it is desirable that the camera video recording circuit 40 be composed of hard macros rather than soft macros.

[0098] The camera video recording circuit 40 is preferably sold as a dedicated chip for camera video recording. The camera video recording circuit 40 may be an LSI for camera video recording, such as an ASSP or SoC for camera video recording, and the camera video recording circuit is particularly preferably an SoC for a dashcam.

[0099] When the camera video recording circuit 40 receives a trigger signal TG from the image recognition circuit 50, it performs event recording, storing the video signal for a period corresponding to that input in the storage medium 60. This period may be, for example, 30 seconds before and after the event occurs, but it may also be a different period.

[0100] The image recognition circuit 50 functions as an electronic component that realizes a predetermined function using signals transmitted between the camera 10 and the camera video recording circuit 40, which are distributed by the second video signal conversion circuit 30. The image recognition circuit 50 recognizes the image shown by the video signal S2 from the second port 32 and determines whether an event that triggers event recording has occurred. If the image recognition circuit 50 determines that the event has occurred, it outputs a trigger signal TG to the camera video recording circuit 40, which triggers the start of event recording. For example, the image recognition circuit 50 recognizes the state of an object in the image, and if the recognition results in a situation where event recording is necessary, it outputs a trigger signal TG to the camera video recording circuit 40. In this way, the image recognition circuit 50 may determine whether to generate a trigger signal TG by comprehensively analyzing the images from the four cameras 10-1 to 10-4.

[0101] In this manner, the image recognition circuit 50 outputs information regarding the results of the image recognition processing performed based on the video signal S2 to the camera video recording circuit 40. The image recognition circuit 50 may also output information indicating the position of the recognized object in the video to the camera video recording circuit 40. Based on the information regarding the results of the image recognition processing, the camera video recording circuit 40 compresses the video indicated by the video signal S1 and stores it in the storage medium 60. In this way, the compression of the video from camera 10 and its storage in the storage medium 60 can be reliably performed, and recording based on the results of image recognition processing of the same video from camera 10 can also be performed.

[0102] Here, an example of event recording is described. For example, if the image recognition circuit 50 recognizes a no-entry sign based on the video, but detects intrusion beyond the sign, it outputs a trigger signal TG1 indicating a no-entry trigger to the camera video recording circuit 40. When the camera video recording circuit 40 receives the input of trigger signal TG1, it encodes and records the video signal showing the footage for 15 seconds before and after the signal as a no-entry trigger video on the storage medium 60. For example, if the image recognition circuit 50 recognizes that a person has entered the vicinity of a forklift, it outputs a trigger signal TG2 indicating a surrounding person recognition trigger to the camera video recording circuit 40. When the camera video recording circuit 40 receives the input of trigger signal TG2, it encodes and records the video signal showing the footage for 30 seconds before and after the signal as a surrounding person recognition trigger video on the storage medium 60. Here, only the video from the video channel that triggered the output of trigger signal TG1 or TG2 may be recorded, or the video from all video channels may be recorded. Various variations are possible regarding the events that trigger the output of the trigger signals and the period covered by event recording. For example, event recording may be performed when information input via the information input unit 70 meets predetermined conditions.

[0103] For example, the same camera video signal is distributed and input to the camera video recording circuit 40 and the image recognition circuit 50 using a MIPI CSI-2 splitter (e.g., a splitter chip). The camera video recording circuit 40 and the image recognition circuit 50 are connected by a communication line. The image recognition circuit 50 recognizes the state of the object using image recognition, and if the recognition results in a situation requiring recording, it sends a trigger signal to the camera video recording circuit via this communication line. When the camera video recording circuit receives the trigger signal from the image recognition circuit, it should record the video signal it is receiving as an event in a storage means.

[0104] The communication line may be a parallel communication line or a serial communication line to transmit trigger signals TG and other data. For example, information about the type of recognized object may be transmitted as this data. Alternatively, it may be a simple communication line between GPIOs. For example, the trigger input port of the drive recorder SoC may be connected to the GPIO output port of the image recognition SoC. When the image recognition SoC determines that recording is necessary as a result of image recognition, the signal at the GPIO output port of the image recognition SoC is changed. The drive recorder SoC, upon detecting the change in the signal at its trigger input port, may record video for a predetermined number of seconds before and after the change in the recording means. Furthermore, signals from sensors connected to the camera video recording circuit 40 may also be transmitted to the image recognition circuit 50. For example, if the camera video recording circuit 40 is an SoC for a drive recorder and the image recognition circuit 50 is an SoC for image recognition, then the accelerometer, switch, and GPS module are connected to the drive recorder SoC, and the information received by the drive recorder SoC from the accelerometer, switch, and GPS module is output to the image recognition SoC in real time. The image recognition SoC receives this information from the drive recorder SoC in real time, uses it for image recognition, and outputs the results to the drive recorder SoC. The functions of both SoCs can be realized by executing programs stored in each SoC to implement these functions.

[0105] The image recognition circuit 50 may record information based on the video signal distributed from the second video signal conversion circuit 30 in the second recording means. In this way, the 22nd recording means records information based on the video signal distributed from the second video signal conversion circuit 30. The second recording means may be a recording means different from the first recording means. In particular, it may be a removable recording medium different from the first recording means. If the first recording means is a storage medium 60, then the second recording means is a recording means different from the storage medium 60. The information based on the distributed video signal may be information obtained by processing the video signal that has been distributed and input to the image recognition circuit 50. For example, video information may be recorded that surrounds the recognized object and overlays its attributes as text information. Alternatively, along with or instead of such video information, the trigger signal TG output to the camera video recording circuit 40 may be recorded along with its time. For example, information regarding the coordinates and type of the recognized object may be recorded in association with its time.

[0106] The camera video recording circuit 40 may have a function for continuous recording in addition to the function for event recording. In this case, the camera video recording circuit 40 records video to a predetermined storage area reserved on the storage medium 60, and when the storage area becomes full, it records video by overwriting the oldest video first. To prevent video related to event recording from being lost due to the continuous recording function, these storage areas may be separated, or overwriting may be prohibited using an overwrite prohibition flag.

[0107] The storage medium 60 functions as a storage means for storing images recorded by the camera image recording circuit 40. The storage medium 60 may be either an internal or external storage means of the system 1, but it is particularly preferable to have a removable external storage means, and especially preferable to have a card-shaped medium such as an SD card or microSD card. The images stored in the storage medium 60 can be displayed on the display unit 80 or displayed using an external terminal. The display unit 80 is, for example, a liquid crystal display or other type of display device.

[0108] The information input unit 70 accepts information input from an external source. The information input unit 70 may be equipped with various switches, including, for example, a switch for turning the power of system 1 on or off. The information input unit 70 may also be equipped with sensors for the drive recorder. The information input unit 70 may include, for example, an acceleration sensor (for example, functioning as a G-sensor) for detecting an impact that triggers event recording. The information input unit 70 may also be equipped with a GPS (Global Positioning System) sensor for measuring the current location. Various signals corresponding to the information input via the information input unit 70 may be exchanged between the camera video recording circuit 40 and the image recognition circuit 50.

[0109] Next, the transmission of the synchronization signal and the processing using the synchronization signal will be described. The signal transmitted between camera 10 and camera video recording circuit 40 includes the video synchronization signal generated by the synchronization signal generation means. System 1 is configured to output the video signals of cameras 10-1 to 10-4 to the camera video recording circuit 40 based on the synchronization signal distributed along with the video signal.

[0110] In this way, the camera video recording circuit 40 can obtain synchronized video signals with cameras 10-1 to 10-4. For example, the camera video recording circuit 40 can input synchronized video signals from cameras 10-1 to 10-4, compress and record video based on these multiple synchronized video signals. For example, when the recorded video signals are played back simultaneously, multiple synchronized videos can be viewed. Although video compression may be performed on separate files for each of the cameras, it is preferable to perform it on a single file.

[0111] In System 1, in order to synchronize the operation of multiple cameras 10, first video signal conversion circuits 20-1 to 20-4, camera video recording circuit 40, and image recognition circuit 50, the synchronization signal is also distributed to the image recognition circuit 50. Thus, the signal transmitted between the camera 10 and the camera video recording circuit 40 may be a control signal, but it is preferable to include at least one of either a video signal or a video synchronization signal. The synchronization signal may be at least one of the pixel clock, vertical synchronization signal, and horizontal synchronization signal, and it is preferable to include at least two of them. In particular, it is preferable to use the pixel clock and the vertical synchronization signal. The second video signal conversion circuit 30 may output a synchronized clock to the four first video signal conversion circuits 20-1 to 20-4. For example, the following (Method 1) or (Method 2) can be adopted as the source of the synchronization signal and the distribution method.

[0112] (Method 1) One camera 10 acts as the parent to generate a synchronization signal, which is then distributed to another camera 10, a first video signal conversion circuit 20, a second video signal conversion circuit 30, a camera video recording circuit 40, and an image recognition circuit 50.

[0113] (Method 1) involves one of the cameras 10 having a synchronization signal generation means. This camera 10, acting as a signal distribution means, then distributes this synchronization signal to another camera 10, a first video signal conversion circuit 20, a second video signal conversion circuit 30, a camera video recording circuit 40, and an image recognition circuit 50.

[0114] In this configuration, the video signal output from another camera 10 becomes a video signal based on the synchronization signal generated within the first camera 10, making it easier to synchronize these cameras 10. In particular, it is preferable to adopt this configuration and to house both the first camera 10 and the second camera 10 in the same housing. This is especially effective when the distance over which the synchronization signal needs to be routed is short. In particular, it is preferable to generate the synchronization signal within the image sensor of the camera 10. In this way, it is not necessary to use an externally syncable image sensor that captures images in synchronization with the input synchronization signal, resulting in a low-cost and simple configuration.

[0115] (Method 2) One of the first video signal conversion circuit 20, the second video signal conversion circuit 30, the camera video recording circuit 40, and the image recognition circuit 50 generates a synchronization signal and distributes it to the other elements.

[0116] (Method 2) involves a synchronization signal generation means provided outside the camera 10, which distributes the synchronization signal generated by this means to cameras 10-1 to 10-4 and the other elements.

[0117] Method 2 makes it easier to synchronize the video signal output from one camera 10 with the video signal output from another camera 10. In particular, it is preferable to adopt this configuration and to house camera 10 and the other camera 10 in different housings. This is especially effective when the distance over which the synchronization signal needs to be routed is relatively long.

[0118] In (Method 2), the camera video recording circuit 40 may have a synchronization signal generation means. In this case, the camera video recording circuit 40, acting as a signal distribution means, distributes the synchronization signal to cameras 10-1 to 10-4 and other elements. This increases the likelihood of obtaining synchronized video signals from the camera and other cameras at timings that match the video recording process.

[0119] In (Method 2), a synchronization signal generation means may be provided in either the first video signal conversion circuit 20 or the second video signal conversion circuit 30. In this case, the first video signal conversion circuit 20 or the second video signal conversion circuit 30, acting as a signal distribution means, distributes the synchronization signal to the cameras 10-1 to 10-4 and the other elements. In this way, it is easy to input the synchronized video signals from the camera and the other camera to the camera video recording circuit via a cable and record them.

[0120] In particular, it is preferable to synchronize the second video signal conversion circuit 30, the camera video recording circuit 40, and the image recognition circuit 50 by providing a synchronization signal. The synchronization signal can be a pixel clock or a vertical synchronization signal. The second video signal conversion circuit 30 should output a synchronized clock to the four first video signal conversion circuits 20-1 to 20-4. Alternatively, the second video signal conversion circuit 30 can generate a clock and provide that clock to the camera video recording circuit 40 and the image recognition circuit 50 to synchronize them. The image recognition circuit 50 may be an FPGA instead of an SoC, or it may be any other type of computer.

[0121] The method of processing based on the synchronization signal is not limited to either (Method 1) or (Method 2), and for example, the synchronization signal generation means may be provided separately from the elements of System 1 described above.

[0122] The above signal distribution means may, for example, simply branch the wiring, but it is particularly desirable to have a function that distributes the signals so that they are synchronized between the multiple distributed signal lines, and especially a circuit (for example, a chip) that has a function that distributes the signals so that they are synchronized between the multiple distributed signal lines.

[0123] Furthermore, while the camera 10 and the camera video recording circuit 40 may be housed in the same enclosure, it is preferable to house them in separate enclosures and connect them using cables, connectors, or other means of connection. The camera 10 and the camera video recording circuit 40 should be connected via a cable of 1 meter or longer. In particular, the camera 10 and the camera video recording circuit 40 should be connected via a cable routed within the vehicle.

[0124] The configuration described in Figure 1 may be modified as shown in Figure 2. Figure 2 mainly shows the parts of the configuration described in Figure 1 that differ from those in Figure 1. In this example, the video signal from camera 10 is input to camera video recording circuit 40 without being distributed. Camera video recording circuit 40 supplies video signals from cameras 10-1 to 10-4 to image recognition circuit 50. Camera video recording circuit 40 is equipped with a recognition video signal generation means that generates a signal with reduced information content compared to the input video signal, and outputs the reduced information content of the video signal, for example, by decimating the video signal, to the image recognition circuit 50 via another port such as a USB (Universal Serial Bus) port. Decimation of the video signal can be done by processes such as decimating pixels or decimating frames. Generally, in image recognition, especially in the case of deep learning, the amount of image information required for recognition can be significantly smaller than what a human can see when viewing an image.

[0125] The configuration described in Figure 2 may be further modified as shown in Figure 3. Figure 3 shows the parts of the configuration described in Figure 1 that differ from those in Figure 1. In the example in Figure 2, the decimation is performed by the camera video recording circuit 40, but this may increase the processing load or lead to insufficient resources. Therefore, as shown in Figure 3, an image processing unit 90 may be provided between the camera 10 and the camera video recording circuit 40 and the image recognition circuit 50. The image processing unit 90 branches the video signal from the camera and outputs the original video signal to the camera video recording circuit 40, and the decimated video signal to the image recognition circuit 50. The image processing unit 90 is, for example, an FPGA, but may also be configured using a standalone ISP chip. The image processing unit 90 may be configured to switch the output destination for each frame using the output port switching function of the ISP chip.

[0126] In System 1, there may be only one camera 10. For example, the camera signal may be split into two using a splitter, as exemplified by the MIPI CSI-2 splitter. In this case, it is equivalent to the configuration in Figure 3 where one of the cameras 10-1 to 10-4 is used and the image processing unit 90 is used as the splitter.

[0127] In System 1, the video signal transmitting the video from camera 10 may be transmitted serially, synchronized on a per-video-channel basis, or in parallel. Alternatively, it may be transmitted in a combination of serial and parallel. In particular, it is preferable to superimpose data such as the time, event information, and sensor information onto each video frame, or to superimpose it onto the video. Data such as the time, event information, and sensor information may be superimposed onto the synchronized units, or to superimpose it onto the video.

[0128] Furthermore, it is preferable to make the length of the signal wiring from the signal distribution means to camera 10 and the other camera 10 approximately the same. Doing so increases the likelihood that the video signal from camera 10 and the video signal from the other camera 10 will be synchronized signals.

[0129] Next, with reference to Figures 4 to 7, an example of the connection between cameras 10-1 to 10-4 and the processing unit 100 will be described. The processing unit 100 is a device that performs predetermined processing based on the video signals output by cameras 10-1 to 10-4. The processing unit 100 may be, for example, either the camera video recording circuit 40 or the image recognition circuit 50.

[0130] In the example shown in Figure 4, cameras 10-1 to 10-4 are provided in a single camera unit and connected to a single first video signal conversion circuit 20. The first video signal conversion circuit 20 outputs video signal Sa1 to the second video signal conversion circuit 30 via a single cable. The first video signal conversion circuit 20 outputs the same video signal Sa2 as video signal Sa1 to the processing unit 100 via a single cable. Video signal Sa2 is a signal that retains the serial format of video signal Sa1. The processing unit 100 performs processing using the video signal Sa2 input via a single cable.

[0131] In the example shown in Figure 5, multiple cameras 10-1 to 10-4 are provided in one camera unit, and these are connected to a single first video signal conversion circuit 20. The first video signal conversion circuit 20 outputs video signal Sb1 to the second video signal conversion circuit 30 via a single cable. Video signal Sb1 is the same signal as video signal Sa1. The second video signal conversion circuit 30 outputs video signals Sb21, Sb22, Sb23, and Sb24 using different cables for each of the cameras 10-1 to 10-4. Video signals Sb21 to Sb24 are signals obtained by converting the serial video signal Sb1 into a parallel format. The processing unit 100 performs processing using the video signals Sb21 to Sb24 input via different cables for each camera 10.

[0132] In the example shown in Figure 6, a single camera unit is equipped with multiple cameras 10-1 to 10-4, and these are connected to different first video signal conversion circuits 20-1 to 20-4 for each camera. Each of the first video signal conversion circuits 20-1 to 20-4 outputs video signals Sc11, Sc12, Sc13, and Sc14 to the second video signal conversion circuit 30 via a single cable. The second video signal conversion circuit 30 outputs video signal Sc2 to the processing unit 100 via a single cable. The processing unit 100 performs processing using the video signal Sc2 input via the cable. Video signal Sc2 is a signal obtained by converting the parallel video signals Sc1 to Sc4 into a serial format. System 1 in Figure 1 corresponds to this connection example.

[0133] In the example shown in Figure 7, multiple cameras 10-1 to 10-4 are provided in a single camera unit, and these are connected to different first video signal conversion circuits 20-1 to 20-4 for each camera. System 1 in Figure 1 corresponds to this connection example. Each of the first video signal conversion circuits 20-1 to 20-4 outputs video signals Sd11, Sd12, Sd13, and Sd14 to the second video signal conversion circuit 30 via a single cable. The second video signal conversion circuit 30 outputs video signals Sd21, Sd22, Sd23, and Sd24 using different cables for each of the cameras 10-1 to 10-4. Video signals Sd21 to Sd24 are signals that retain the parallel format of the parallel video signals Sd11 to Sd14. The processing unit 100 performs processing using the video signals Sb21 to Sb24 input via different cables for each camera 10.

[0134] Based on the configurations in Figures 1 and 6,7, a system may be configured to transmit video signals from each camera unit, which includes a first camera unit having a camera 10 and a first video signal conversion circuit 20; a second camera unit having another camera 10 and a second first video signal conversion circuit 20; and a recording unit having a second video signal conversion circuit 30 and a camera video recording circuit 40. The first camera unit and the recording unit are connected by a first cable, and the second camera unit and the recording unit are connected by a second cable. For example, a first camera unit may be provided, in which the camera (first camera) mounted on the windshield of the vehicle is positioned to capture images in front of the vehicle, and a second camera unit may be provided, in which the other camera (second camera) mounted on the rear window of the vehicle is positioned to capture images behind the vehicle. A recording unit provided on the dashboard or under the passenger seat is connected to both camera units by separate cables. Alternatively, the recording unit and the first camera unit may be configured as a single unit.

[0135] Alternatively, for example, a first camera unit having a camera 10 and a first video signal conversion circuit 20 may be connected to a second camera unit having another camera 10 and a second first video signal conversion circuit 20 by a first cable, and a recording unit having a second video signal conversion circuit 30 and a camera video recording circuit 40 may be connected to the second camera unit by a second cable, thereby transmitting the respective video signals.

[0136] Based on the configurations in Figures 1 and 4, 5, a system may be configured in which a single camera unit includes multiple cameras 10, including camera 10 and another camera 10, and a first video signal conversion circuit 20, the first video signal conversion circuit 20 converts the video signals from these multiple cameras 10 so that they flow through a single cable, and transmits them via this single cable to a recording unit equipped with a second video signal conversion circuit 30 and a camera video recording circuit 40, so that the video from the multiple cameras 10 is recorded on a storage medium 60. For example, a camera unit mounted on the windshield of a vehicle may be configured to include a camera unit with a camera 10 (first camera) that captures images in front of the vehicle and another camera 10 (second camera) that captures images behind the vehicle, and a recording unit located on the dashboard or under the passenger seat may be connected by a cable.

[0137] For example, the second video signal conversion circuit 30 may convert the video signals from multiple cameras 10, which have been converted by the first video signal conversion circuit 20 and transmitted over a relatively long distance via a single cable, into signals suitable for relatively short-distance transmission required for wiring within the recording unit.

[0138] Based on the configurations in Figures 1 and 4, 5, for example, the second video signal conversion circuit 30 may be configured to output the video signals from multiple cameras 10 to the camera video recording circuit 40 so that they flow through a single cable, which is then input to the camera video recording circuit 40. For example, the second video signal conversion circuit 30 may divide the video signals from two cameras into packets of a predetermined data size and transmit the packets alternately. Alternatively, the video signals from a larger number of cameras may be interleaved into a single transmission path and output.

[0139] Alternatively, based on the configurations in Figures 1 and 6,7, the second video signal conversion circuit 30 may be configured to output the video signals from multiple cameras 10 to the camera video recording circuit 40 so that they flow through different cables for each camera, and these are input to the camera video recording circuit 40. In this way, processing can be easily performed on a per-camera 10 basis within the different transmission paths for each camera 10 or within the camera video recording circuit 40 connected to them. For example, it becomes relatively easy to superimpose and record or output information other than video on a per-camera frame basis for each camera 10.

[0140] The camera video recording circuit 40 is relatively inexpensive, but because it is designed specifically for video compression and storage, it has problems in that it is difficult to add functions such as image recognition (for example, it has limited information disclosure and limited remaining resources). For example, a dashcam SoC is inexpensive, but because it is designed for dashcams, it has problems in that it is difficult to add functions such as image recognition (for example, it has limited information disclosure and limited remaining resources).

[0141] On the other hand, the image recognition circuit 50 used for image recognition has the problem that there is no reliable software for video compression and recording. The image recognition circuit should be a general-purpose SoC (SoC for image recognition) that can be used by compiling open-source software, for example. In particular, it would be good to use an SoC equipped with an image recognition accelerator circuit or an SoC equipped with a deep learning accelerator circuit. However, there is also the problem that it is costly to put a video encoder such as H.264 on a general SoC, or especially on an FPGA image recognition circuit. Furthermore, even if a general SoC is equipped with a video encoder, there is a problem that reliable software for dashcams is scarce, and developing it from scratch in-house would be extremely costly.

[0142] In contrast, the system of this embodiment implements the camera video recording circuit 40 and the image recognition circuit 50 using different SoCs or other circuits, and distributes the video signal and synchronization signal as described above, thereby providing a system that is superior to conventional systems.

[0143] Furthermore, the following configuration should be adopted.

[0144] (Camera control example 1) The function for configuring camera 10 (and / or another camera 10) (for example, the part of the ISP function that controls the camera) may be configured to use the one on the camera video recording circuit 40 side (Pattern 1), the one on the image recognition circuit 50 side (Pattern 2), or a configuration that switches between using the ones on both sides (Pattern 3). Furthermore, each configuration may use either an SoC with a built-in ISP or an external ISP. The external ISP may be installed on either the camera video recording circuit 40 side. However, since the image processing required for image recognition and the image processing required to compress and record on the storage medium 60 for playback are different, the question arises as to how to control the camera's state, such as shutter speed, frame rate, and exposure time. Therefore, the camera video recording circuit 40 may have a function for configuring camera 10.

[0145] This approach prevents configuration conflicts and allows for relatively smooth coexistence between recording video from camera 10 and image recognition based on that video. For example, the inventors have found that adopting the configuration described above (Pattern 1) yields excellent results. The images used for image recognition are often of lower quality than those required for recording. It is particularly beneficial to have a function that recognizes images using deep learning, and in such a configuration, the requirements for pixel count, frame rate, etc., are generally low. On the camera video recording circuit 40 side, it is especially good to have it within the camera video recording circuit 40.

[0146] However, regarding the focus and exposure functions of the ISP, it may be better to configure the camera on the image recognition circuit 50 side according to the requirements necessary for image processing. Therefore, it is advisable to have a system in which the camera settings on the camera video recording circuit 40 side are configured in advance, taking into account the exposure and focus settings required on the image recognition circuit 50 side. For example, it is advisable to configure the ISP on the camera video recording circuit 40 side (for example, the ISP within the camera video recording circuit 40) in advance, taking into account the exposure and focus settings required on the image recognition circuit 50 side, and adjusting them within a range where the camera image does not become too poor.

[0147] (Camera control example 2) The camera settings are coordinated between the camera video recording circuit 40 and the image recognition circuit 50. This makes it easier to achieve both compression and recording of camera images and improved accuracy of image recognition processing.

[0148] As shown in Figures 8(a) and 8(b), it is preferable that the image recognition circuit 50 and the camera video recording circuit 40 communicate with each other and have a function to mutually understand the control status of the camera 10.

[0149] For example, instead of pre-setting the settings of camera 10, or while pre-setting them, the image recognition circuit 50 generates exposure and focus adjustment request signals and outputs them to the camera video recording circuit 40. When the camera video recording circuit 40 receives these signals, it performs the settings according to its own setting function. For example, instead of pre-setting the camera settings, or while pre-setting them, the image recognition circuit 50 outputs exposure and focus adjustment request signals to the camera video recording circuit 40. When the camera video recording circuit 40 receives these signals, it is preferable to adjust the ISP settings on the camera video recording circuit 40 accordingly. The information regarding the camera settings to be set by the camera video recording circuit 40 is transmitted to the image recognition circuit 50. It is particularly preferable for the image recognition circuit 50 to receive this information and use it in its image recognition processing.

[0150] For example, when the image recognition circuit 50 recognizes a sign, it is good practice to calculate the area corresponding to the road shoulder in the image as the priority area for exposure calculation. In this case, the camera is basically controlled from the camera video recording circuit 40, but it is good practice to set the exposure priority of this area to be slightly higher in advance, and if the exposure of the sign area is underexposed by the image recognition circuit 50, the image recognition circuit 50 outputs an exposure compensation signal to the camera video recording circuit 40, and when the camera video recording circuit 40 receives this signal, it outputs a signal to the camera to increase the camera's exposure.

[0151] Thus, the camera 10 itself should be primarily controlled by the camera video recording circuit 40, and the image recognition circuit 50 should not directly control the camera. Furthermore, the image recognition circuit 50 should output an adjustment signal to the camera video recording circuit 40, and the image recognition circuit 50, upon receiving this adjustment signal, should indirectly control the camera. For example, the camera 10 itself should be primarily controlled by the camera video recording circuit 40, and the image recognition circuit 50 should not directly control the camera. Alternatively, it should be configured to control it indirectly.

[0152] Furthermore, it's best to set the camera to pan-focus and therefore eliminate the need for any focus-related settings.

[0153] As another example, as mentioned above (Pattern 3), the camera control entity may be switched using time-division multiplexing. In particular, the video recording frame rate should be sufficient for recording and playing back video (for example, a frame rate that does not cause stuttering to the human eye (e.g., around 24 to 30 fps)), while the camera frame rate should be higher (e.g., 120 fps). When the frames output by the camera are numbered sequentially as "1", "2", "3", "4", "1", "2", "3", "4", ..., "1" should be from the camera video recording circuit 40, and "3" should be from the image recognition circuit 50. For example, "2" and "4" could be the switching time.

[0154] The switching could be done by physically switching both the ISP on the camera video recording circuit 40 and the ISP on the image recognition circuit 50, but it is preferable that the ISP simply switches the settings of the ISP on the camera video recording circuit 40 in a time-division multiplexing manner. All of the cameras 1-0 should be controlled by the camera video recording circuit 40, and the image recognition circuit 50 should not directly control any of the cameras. It is preferable to control them indirectly.

[0155] The cameras that the image recognition circuit 50 indirectly controls should be a subset of the multiple cameras. These subsets should be those cameras where there is a large discrepancy between the requirements necessary for recognition and the requirements necessary for image recording and playback (for humans to view the images).

[0156] In the configuration described in Figure 8, the image recognition circuit 50 may be replaced with a processing unit that performs processing other than image recognition.

[0157] The functions of System 1 described in the above-described embodiments may be implemented by one or more programs and by the coordination of one or more hardware resources. When the functions of System 1 are implemented using a program, the program for implementing this function may be provided stored on a computer-readable recording medium such as various magnetic recording media, optical recording media, magneto-optical recording media, or semiconductor memory. This program may also be distributed via a network. Furthermore, the present invention can also be understood as a signal processing method.

[0158] The scope of the present invention is not limited to the configurations explicitly described in the specification, but also includes combinations of various aspects of the present invention disclosed herein. While the configurations for which patent protection is sought are specified in the appended claims, the present invention intends to include configurations disclosed herein that are not currently specified in the claims in the future.

[0159] The present invention is not limited to the configurations described in the embodiments above. The components of each embodiment and modification described above may be arbitrarily selected and combined. Furthermore, any component of each embodiment and modification may be arbitrarily combined with any component described in the means for solving the invention or any component that embodies any component described in the means for solving the invention. The present application intends to obtain rights to these as well through amendments or divisional applications. In addition, even if there is a description such as "in the case of..." or "when...", it is not meant to be a configuration that is limited to that case or time. Configurations that do not fall under these cases or times are also disclosed, and the present application intends to obtain rights to them. Furthermore, even if there is a sequence of descriptions, it is not limited to that order. Configurations with some parts deleted or the order rearranged are also disclosed, and the present application intends to obtain rights to them.

[0160] Furthermore, the applicant intends to obtain rights to the overall design or a partial design by filing an application for amendment to the design application. The drawing depicts the entire device with solid lines, but it is a drawing that includes not only the overall design but also partial designs claimed for parts of the device. For example, it is a drawing that includes not only a partial design for a part of the device, but also a partial design for a part of the device regardless of whether it is a part or not. A part of the device may be a part of the device's components, or a part of a component. The applicant intends to obtain rights not only to the overall design, but also to a partial design where any part of the solid lines in the drawing is represented by dashed lines. [Explanation of Symbols]

[0161] 1: System 1, 10-1~10-4: Camera, 20, 20-1~20-4: Second video signal conversion circuit, 30: Deserializer, 31: First port, 32: Second port, 50: Image recognition circuit, 60: Storage medium, 70: Information input unit, 80: Display unit, 90: Image processing unit, 100: Processing unit

Claims

1. A system comprising an SoC for a dashcam and an SoC for image recognition, A system characterized in that the output port of an image recognition SoC is connected to the input port of the drive recorder SoC, and when the image recognition SoC determines that recording is necessary, the signal at the output port of the image recognition SoC is changed, and the drive recorder SoC has a function to record images for a predetermined number of seconds before and after the change in the signal at the input port of the drive recorder SoC in a recording means.

2. The video signal from the camera transmitted via the cable The system includes distribution means for distributing data between the aforementioned SoC for the drive recorder and the aforementioned SoC for image recognition. The system according to claim 1, characterized by the following:

3. The aforementioned SoC for image recognition includes a function to output information regarding the results of the image recognition processing to the SoC for the drive recorder. The aforementioned SoC for the drive recorder includes a function to record an image in the recording means based on information regarding the results of image recognition processing output from the SoC for the drive recorder. The system according to claim 1 or 2, characterized by the above.

4. The aforementioned SoC for image recognition is equipped with a function to output information indicating the position of the recognized object in the image to the SoC for the drive recorder. A system according to any one of claims 1 to 3, characterized by the above.

5. The aforementioned SoC for image recognition has a function to output a trigger signal indicating a no-entry trigger to the SoC for the drive recorder if, despite recognizing a no-entry sign based on the image, intrusion beyond the sign is detected. The aforementioned SoC for the drive recorder has a function that, upon receiving a trigger signal indicating the no-entry trigger from the image recognition SoC, records images for a predetermined time before and after the trigger as no-entry trigger images in the recording means. A system according to any one of claims 1 to 4, characterized by the above.

6. The aforementioned SoC for image recognition has a function to output a trigger signal indicating surrounding person recognition trigger to the SoC for the drive recorder when it recognizes that a person has entered the vicinity of the forklift. The aforementioned SoC for the drive recorder has a function that, upon receiving an input of a trigger signal indicating the surrounding person recognition trigger from the aforementioned SoC for image recognition, records images for a predetermined time before and after the trigger as entry-prohibition trigger images in the recording means. A system according to any one of claims 1 to 4, characterized by the above.

7. The aforementioned SoC for image recognition is further equipped with a function to record information based on an input image in a second recording means, which is a different recording means from the recording means used by the SoC for image recognition. A system according to any one of claims 1 to 6, characterized by the above.

8. The aforementioned SoC for image recognition includes information obtained by processing the input image as information to be recorded in the second recording means. The system according to claim 7, characterized by the following:

9. The aforementioned SoC for image recognition is equipped with a function to record information related to the coordinates and type of the recognized object, associated with the time, as information to be recorded in the second recording means. The system according to claim 7 or 8, characterized by the above.

Images