Video transmission system, vehicle, and video transmission method

Abstract

This invention relates to an image transmission system, a vehicle, and an image transmission method. It enables efficient transmission of multiple images. The image transmission system comprises: a first transmitter that uses a first image transmission standard to superimpose two or more first image signals to generate a second image signal; a second transmitter that uses a second image transmission standard different from the first image transmission standard to superimpose a third image signal and the second image signal to generate a fourth image signal; a first receiver that receives the fourth image signal transmitted via a single cable and separates the second image signal from the third image signal; and a second receiver that separates the two or more first image signals from the second image signal.

Description

Technical Field

[0001] This invention relates to the transmission of images. Background Technology

[0002] There are techniques for making image transmission more efficient. In this regard, for example, Patent Document 1 discloses a system in which image signals are multiplexed to achieve efficient image transmission in a system connecting multiple cameras.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2005-328479 Summary of the Invention

[0006] The problem the invention aims to solve

[0007] The purpose of this disclosure is to efficiently transmit multiple images.

[0008] Solution for solving the problem

[0009] One embodiment of this disclosure is an image transmission system comprising: a first transmitter that uses a first image transmission standard to superimpose two or more first image signals to generate a second image signal; a second transmitter that uses a second image transmission standard different from the first image transmission standard to superimpose a third image signal with the second image signal to generate a fourth image signal; a first receiver that receives the fourth image signal transmitted via a single cable and separates the second image signal from the third image signal; and a second receiver that separates the two or more first image signals from the second image signal.

[0010] One embodiment of this disclosure is a vehicle, wherein the vehicle has: a plurality of cameras for capturing images of the exterior of the vehicle; a first device for superimposing two or more first image signals using a first image transmission standard to generate a second image signal, and superimposing a third image signal with the second image signal using a second image transmission standard different from the first image transmission standard to generate a fourth image signal; and a second device for receiving the fourth image signal transmitted via a single cable, separating the second image signal from the third image signal, and separating the two or more first image signals from the second image signal.

[0011] Alternatively, the above-mentioned system, the method executed by the vehicle, the program for causing a computer to execute the method, or the computer-readable storage medium that non-volatilely stores the program can be listed as other options.

[0012] Invention Effects

[0013] According to this disclosure, multiple images can be transmitted efficiently. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the vehicle system according to the implementation method.

[0015] Figure 2 This is a schematic diagram of a system that overlays image signals using GVIF.

[0016] Figure 3 This is a schematic diagram of a system that overlays image signals using MIPI.

[0017] Figure 4 This is an example of an image displayed by the vehicle terminal 20 when the vehicle is reversing.

[0018] Figure 5 This is a schematic diagram of the processing performed by the image transmission system in the first embodiment.

[0019] Figure 6 This is a schematic diagram of the image signals transmitted between ECU10 and vehicle terminal 20.

[0020] Figure 7 This is a block diagram that roughly represents an example of the structure of vehicle 100.

[0021] Figure 8 This diagram illustrates the functional modules of the control unit 11 and the flow of data between these modules.

[0022] Figure 9 This is an example of selection data sent by the vehicle terminal 20.

[0023] Figure 10 This diagram illustrates the functional modules of the control unit 21 and the flow of data between these modules.

[0024] Figure 11 This is an example of the data referenced by the vehicle terminal 20 when selecting a camera.

[0025] Figure 12 This is a flowchart of the processes performed by ECU10 and vehicle terminal 20.

[0026] Figure 13 This is a module structure diagram of the control unit 11 in the second embodiment.

[0027] Figure 14 This is a diagram illustrating the location of a camera installed on a vehicle.

[0028] Explanation of reference numerals in the attached figures

[0029] 100…Vehicle; 10…ECU; 20…Vehicle terminal; 11, 21…Control unit; 12, 22…Storage unit; 13, 23…Communication unit; 14…Image interface; 24…Input / output unit; 30A~30E…Camera. Detailed Implementation

[0030] In recent years, the use of in-vehicle cameras has been increasing. In addition to dashcams, other types of in-vehicle cameras include rear cameras, side cameras, stereo cameras for autonomous driving, and cameras for driver monitoring. Figure 14 This diagram illustrates a camera mounted externally on a vehicle. As shown, the vehicle is equipped with multiple cameras to monitor multiple directions. These cameras are connected to an electronic control unit (ECU) that manages the images. Furthermore, this ECU is connected to in-vehicle devices (navigation systems), etc.

[0031] In image transmission, predetermined image transmission standards are utilized. Examples of image transmission standards include GVIF (Gigabit Video Interface) (registered trademark) and MIPI (Mobile Industry Processor Interface) (registered trademark).

[0032] Image transmission can be achieved via image transmission and reception modules and cables connecting the modules to each other. However, if the number of onboard cameras increases, there is a problem of an increase in the number of image transmission and reception modules and cables.

[0033] One approach to solving this problem is to overlay transmitted images. By overlaying transmitted image signals, the number of modules and cables used in image transmission and reception can be reduced.

[0034] On the other hand, increasing the number of superimposed images requires more hardware, potentially leading to increased costs. While standards exist that can superimpose multiple images using a single hardware device, these require uniform image resolution and frame rate, making them unsuitable for vehicles equipped with various cameras. Furthermore, image superimposition can result in a decrease in bit rate, leading to a reduction in image quality, such as image quality. In particular, safety-related images (e.g., images from rear cameras) must meet specified quality requirements, sometimes prohibiting any reduction in image quality (such as image quality degradation).

[0035] The image transmission system of the present invention can transmit multiple images in superposition while ensuring quality requirements.

[0036] One of the solutions disclosed herein is an image transmission system.

[0037] Specifically, the device comprises: a first transmitter that uses a first image transmission standard to superimpose two or more first image signals to generate a second image signal; a second transmitter that uses a second image transmission standard different from the first image transmission standard to superimpose a third image signal with the second image signal to generate a fourth image signal; a first receiver that receives the fourth image signal transmitted through a single cable and separates the second image signal from the third image signal; and a second receiver that separates the two or more first image signals from the second image signal.

[0038] Two or more first image signals and third image signals are image signals transmitted between different devices via a single cable. When the image transmission system is applied to a vehicle, the two or more first image signals and third image signals can also be image signals generated by an onboard camera.

[0039] The first transmitter superimposes two or more first image signals to generate a second image signal. Then, the second transmitter further superimposes a third image signal onto the second image signal to generate a fourth image signal.

[0040] The first transmitter uses a different image transmission standard than the second transmitter.

[0041] For example, a first transmitter uses a first image transmission standard capable of superimposing multiple images to superimpose two or more first image signals. The resulting second image signal is transmitted to a second transmitter, and the second and third image signals are superimposed. The second transmitter uses a second image transmission standard to perform this superposition, generating a fourth image signal.

[0042] The first and second receivers separate the superimposed image signals in reverse order. That is, from the fourth image signal, the third image signal is separated from the second image signal using the second image transmission standard. Then, using the first image transmission standard, the second image signal is separated into two or more first image signals.

[0043] Therefore, it is possible to combine multiple different image transmission standards to overlay and separate images.

[0044] If images are transmitted overlaid using a single image transmission standard, the following problems will occur.

[0045] When the number of transmitted image signals changes, the frequency band allocated to each image signal changes, thus changing the image quality.

[0046] It is difficult to mix images with different resolutions and frame rates.

[0047] On the other hand, by combining multiple different image transmission standards, it is possible to ensure the quality requirements of the image while simultaneously transmitting multiple image signals with different resolutions and frame rates.

[0048] The following describes specific embodiments of this disclosure based on the accompanying drawings. Unless otherwise specified, the hardware structures, module structures, and functional structures described in each embodiment are not intended to limit the scope of protection to these embodiments.

[0049] (First Implementation)

[0050] Reference Figure 1 An overview of the vehicle system according to the first embodiment will be described. The vehicle system of this embodiment is configured to include a vehicle 100.

[0051] Vehicle 100 is a car equipped with multiple cameras. Examples of these multiple cameras include rear cameras, side cameras, dashcam cameras, autonomous driving cameras, and driver monitoring cameras.

[0052] Vehicle 100 can also be a vehicle capable of monitoring the exterior of the vehicle from the driver's seat. Additionally, vehicle 100 can be a vehicle capable of autonomous or semi-autonomous driving. Multiple cameras in vehicle 100 are connected to ECU 10.

[0053] ECU10 is an electronic control unit that processes image signals acquired by multiple cameras. ECU10 is also known as a camera ECU.

[0054] ECU 10 is configured to acquire image signals from multiple cameras and select and provide image signals based on requests from other electronic control units and the vehicle terminal 20. It should be noted that... Figure 1 The example shown is a single ECU, but the vehicle 100 can also be configured to include multiple other ECUs. Examples of other ECUs include, for example, an engine ECU, a body ECU, an autonomous driving ECU, a power control ECU, etc. ECU 10 can either manage only camera images or perform other processing based on the camera images (e.g., autonomous driving, driver assistance, etc.).

[0055] The in-vehicle terminal 20 is an information terminal installed in the vehicle 100. Also known as an infotainment terminal, the in-vehicle terminal 20 provides information (such as traffic information and route guidance) and entertainment (such as music and animation) to the vehicle's occupants. The in-vehicle terminal 20 can function independently, like a car navigation system, or it can collaborate with devices such as smartphones.

[0056] In addition, the vehicle terminal 20 can also be configured to communicate with the network via a communication module.

[0057] The vehicle-mounted terminal 20 is configured to output camera images based on the vehicle's status. For example, when the gear is in reverse, the vehicle-mounted terminal 20 can also receive images corresponding to the rear camera from the ECU 10 and output them via a display. Additionally, when the vehicle 100's speed is below a certain speed and the vehicle 100 is located at an intersection with poor visibility, the vehicle-mounted terminal 20 can also receive images corresponding to the left and right front cameras from the ECU 10 and output them via a display.

[0058] Alternatively, the vehicle terminal 20 can also be configured to output camera images based on the driver's operation. For example, when the function of monitoring the omnidirectional movement of the vehicle is activated, the vehicle terminal 20 can also receive images corresponding to multiple cameras from the ECU 10, combine them to generate an image from a viewpoint overlooking the vehicle 100 (hereinafter referred to as an omnidirectional image), and output it via a display.

[0059] Here, we will explain the problems in image transmission.

[0060] The transmission of images within the vehicle (e.g., transmission between ECU10 and vehicle terminal 20) can be performed using a predetermined image transmission standard. For example, if GVIF (Gigabit Video Interface) (registered trademark), one of the image transmission standards, is used, image signals can be transmitted serially at high speed via a single cable.

[0061] Figure 2 (A) is a schematic diagram of a system for transmitting images via GVIF. As shown, transmission is achieved by serializing the image signal using a transmitter. The serialized signal is decoded (deserialized) at the receiving end (receiver), thereby extracting the image signal.

[0062] On the other hand, when multiple cameras generate image signals, they need to be aggregated and transmitted. For example, this could be a situation where, to generate a 360-degree view, images from multiple cameras capturing the exterior of a vehicle are transmitted simultaneously. One method to achieve this is... Figure 2 As shown in (B), there is a method that includes multiple units (transmitters and receivers) for transmitting and receiving video signals, as well as transmission cables. However, this method requires hardware corresponding to the number of cameras, incurring costs.

[0063] As another method, such as Figure 2As shown in (C), there is a method for superimposing multiple image signals. The transmitter (serializer) used in the GVIF standard has the function of superimposing image signals onto each other, allowing simultaneous transmission of image signals from multiple cameras. The transmitter sequentially transmits multiple image signals through time division. The superimposed images can be separated by the receiver (deserializer).

[0064] It should be noted that the overlay in this disclosure refers to a process for sequentially transmitting multiple image signals in a transmission path. Examples of such processes include frame interleaving, which switches the source of the image signal every frame, and line interleaving, which switches the source of the image signal every line.

[0065] However, this method also has its problems. One issue is that the transmitter used by GVIF can only add one image signal to the existing image signal. That is, if you want to transmit image signals generated by three cameras, you need three transmitters and three receivers. This creates a cost problem.

[0066] On the other hand, by utilizing MIPI (Mobile Industry Processor Interface), one of the image transmission standards (registered trademark), it is also possible to overlay multiple image signals. Figure 3 This is a schematic diagram of a system that overlays image signals via MIPI. For example, there are known transmitters that convert multiple (maximum 4 channels) MIPI standard image signals into serial signals.

[0067] However, when using this method, if the number of image signals transmitted simultaneously increases, the bandwidth of each signal is compressed, resulting in a decrease in quality.

[0068] Figure 4 This is an example of an image displayed by the on-board terminal 20 when the vehicle is reversing. When the vehicle is reversing, for example... Figure 4 As shown in (A), sometimes the output corresponds to the image from the rear camera. In this case, the transmitted image is of one type. On the other hand, as... Figure 4 As shown in (B), sometimes, in addition to the image corresponding to the rear camera, images from multiple cameras used to generate a 360-degree view are also transmitted simultaneously. In this case, four types of images are transmitted. That is, the frequency band allocated to a camera changes depending on the method of image display.

[0069] To ensure security, it is preferable to transmit the image corresponding to the rear camera with high priority. However, in the aforementioned method, when displaying a 360-degree view, the bandwidth of the camera image corresponding to the rear camera is compressed. If the bandwidth is compressed, adverse effects such as reduced frame rate, reduced image quality, and image freezing may occur, potentially posing security risks.

[0070] Furthermore, the MIPI standard presents a problem of unstable transmission when images of different resolutions and frame rates are superimposed. In other words, it is difficult to use only a high-resolution camera for the rear camera.

[0071] To address this issue, the image transmission system in this embodiment uses a MIPI standard transmitter to overlay multiple camera images, and then uses a GVIF standard transmitter to further overlay the overlaid image signals. Figure 5 This is a schematic diagram of the processing performed by the image transmission system in this embodiment. Additionally, Figure 6 It is a diagram illustrating the transmission frequency band of the image.

[0072] Here, image signal 1 represents the image signal from the aforementioned rear camera (i.e., the high-priority image signal). Image signals 2 to 4 represent images from other cameras (e.g., front camera, side camera, etc.).

[0073] In the GVIF standard, a maximum of two sets of image signals can be superimposed on a single transmitter, therefore the transmission bandwidth is allocated as shown in reference numerals 601 and 602. That is, for image signal 1 (i.e., the image signal from the rear camera), the bandwidth shown in reference numeral 601 can be guaranteed to a minimum.

[0074] Furthermore, since the image signal obtained by superimposing according to a predetermined standard (MIPI) is further superimposed according to a different standard (GVIF), it is possible to mix images with different resolutions and frame rates.

[0075] Next, the various components of vehicle 100 will be described in detail.

[0076] Figure 7 It is a general representation Figure 1 The diagram shows an example of the structure of a vehicle 100. The vehicle 100 is configured to have an ECU 10 and an on-board terminal 20. These components are interconnected via cables for transmitting image signals and cables for transmitting control signals. It should be noted that in this example, the ECU 10 and the on-board terminal 20 are shown as components of the vehicle 100, but the vehicle 100 may also house multiple electronic control units responsible for vehicle control, such as an engine ECU and a body ECU.

[0077] First, let's explain ECU10.

[0078] ECU 10 is an electronic control unit that processes image signals acquired by multiple cameras in the vehicle. ECU 10 is also referred to as a camera ECU. ECU 10 selects and provides image signals based on requests from other electronic control units and the vehicle terminal 20. It should be noted that in this embodiment, ECU 10 only manages camera images, but it can also perform other processing based on camera images (e.g., autonomous driving, driver assistance, etc.). In this case, ECU 10 can also be referred to as an ADAS-ECU, etc.

[0079] The ECU10 is configured to have a control unit 11, a storage unit 12, a communication unit 13, and an image interface 14.

[0080] The control unit 11 is a computing unit that executes predetermined programs to realize various functions of the ECU 10. The control unit 11 can be implemented, for example, by a CPU.

[0081] Based on instructions received from an external source (typically the vehicle terminal 20), the control unit 11 performs processing to select and send image signals generated by multiple cameras. For example, if the vehicle terminal 20 requests to display a panoramic image of the vehicle 100, the control unit 11 selects the camera used to generate the panoramic image and sends the corresponding image signal to the vehicle terminal 20.

[0082] Details regarding the processing performed by the control unit 11 will be described later.

[0083] Storage unit 12 is a storage device that includes a main storage device and an auxiliary storage device. The auxiliary storage device stores the operating system (OS), various programs, various tables, etc. By loading the programs stored therein into the main storage device and executing them, the various functions that meet the predetermined purpose, as described below, can be achieved.

[0084] The communication unit 13 is an interface unit for connecting the ECU 10 and the vehicle terminal 20. In this embodiment, the ECU 10 and the vehicle terminal 20 are connected via two paths: a path for transmitting images and a path for transmitting control signals. Examples of paths for transmitting images include twisted-pair cables and coaxial cables. In this embodiment, a single cable is used to transmit multiple images simultaneously. Additionally, examples of paths for transmitting control signals include vehicle networks. For example, control signals can be transmitted via CAN (Controller Area Network) based networks or Ethernet (registered trademark). It should be noted that the path for transmitting control signals can also be shared with other ECUs, etc.

[0085] The communication unit 13 is the interface connecting them. The communication unit 13 may also include a hardware interface for transmitting images and a hardware interface for transmitting control signals (e.g., a CAN controller).

[0086] The image interface 14 is an interface unit for connecting multiple cameras mounted on the vehicle. In this embodiment, the image interface 14 receives image signals conforming to the MIPI standard (e.g., MIPI CSI-2) from the cameras.

[0087] Cameras 30A to 30E are multiple cameras mounted on vehicle 100. Examples of cameras mounted on vehicle 100 include cameras for recording (driving recorder), cameras for autonomous driving, cameras for driver safety confirmation, and cameras for driver monitoring. Furthermore, examples of camera placement include, for example, front, front side, side, and rear. Figure 7 In this example, vehicle 100 is configured with camera 30A as a rear camera, camera 30B as a front camera, and cameras C to E as side cameras. The rear camera 30A is used for safety confirmation when the vehicle is reversing, etc. The front camera 30B is used for driver assistance and for recording with a dashcam. The side cameras 30C to E are used for confirming blind spots. Furthermore, a 360-degree view can be generated based on the images acquired by these cameras.

[0088] These cameras are connected to the ECU10 via MIPI standards (e.g., MIPI CSI-2).

[0089] Next, the functions of the control unit 11 will be explained.

[0090] Figure 8 This diagram illustrates the functional modules of the control unit 11 and the flow of data between these modules. The functional modules shown can be implemented by executing programs stored in storage components such as ROM using a CPU or similar device.

[0091] First, multiple image signals acquired via the image interface 14 are input to the control unit 11. In this example, the image signal from camera 30A, which is a rear camera, is referred to as the first camera signal. The image signals from cameras 30B to 30E are referred to as the second camera signals. In this embodiment, the first camera signal is the image signal with the highest processing priority. This is because the first camera signal contains images capturing the area behind the vehicle and therefore should be transmitted in real time (for example, if frames are dropped while the vehicle is reversing, pedestrians crossing behind the vehicle may not be captured). In contrast, the second camera signal is used to generate auxiliary images (omnidirectional images), and therefore has a relatively lower priority compared to the first camera signal.

[0092] Unlike the first camera signal, which must be sent to the vehicle terminal 20, the second camera signal is appropriately selected based on instructions from the vehicle terminal 20. For example, if the vehicle terminal 20 requests an image from a side camera, only the image signal from the side camera is selected.

[0093] The image selection unit 111 selects the signal to be transmitted from a plurality of second camera signals. In this embodiment, the vehicle terminal 20 sends data of a designated camera ("selection data" in the figure) to the image selection unit 111, and the image selection unit 111 responds to this and selects the second camera signal. Figure 9 (A) is an example of selecting data.

[0094] For example, when the vehicle terminal 20 sends selection data for an image to generate an omnidirectional image, the image selection unit 111 selects a second camera signal for generating the omnidirectional image.

[0095] In this embodiment, the "first camera signal" and the "second camera signal selected by the image selection unit 111" are superimposed.

[0096] The image processing unit 112 adjusts the multiple second camera signals that are being superimposed. For example, the image processing unit 112 performs processing on each of the multiple second camera signals to make their resolution and frame rate consistent. For example, when superimposing image signals that conform to the MIPI standard, it is preferable to make the resolution and frame rate of all image signals consistent in order to smooth the communication speed. The resolution can be adjusted by stretching or shrinking the image, or by adding color or cropping.

[0097] It should be noted that the processing performed by the image processing unit 112 is not limited to processing for superimposing image signals. For example, the processing of adjusting the frame rate of the image (e.g., converting a 30fps image signal to 29fps) may also be considered based on the light emission cycle of the LED signal (e.g., 60Hz).

[0098] The first overlay unit 113 performs the process of overlaying multiple second camera signals selected by the image selection unit 111. As a result, the multiple image signals are converted into a single image signal containing multiple channels. It should be noted that the selection of the second camera signals and the overlay of the signals can be freely implemented within the standard range of the image signals (e.g., within the maximum bit rate range). The converted signal can be, for example, set to CSI-2.

[0099] The second overlay unit 114 performs the process of overlaying the image signal superimposed by the first overlay unit 113 with the first camera signal. Therefore, as Figure 6As shown, an image signal with a nested structure can be obtained. Image signals 1 to 4 follow the MIPI standard, and the image signal generated by the second overlay 114 (reference numeral 603) follows the GVIF standard.

[0100] The image signal generated by the second overlay unit 114 is sent to the vehicle terminal 20.

[0101] Next, for Figure 7 The vehicle-mounted terminal 20 shown will be described.

[0102] The in-vehicle terminal 20 is a device that provides information to the occupants of the vehicle 100, and is also known as a car navigation system, infotainment system, or car audio head unit. The in-vehicle terminal 20 can provide navigation and entertainment to the occupants of the vehicle. The in-vehicle terminal 20 may also have the function of downloading traffic information, road map data, music, and dynamic images through communication with the external network of the vehicle 100.

[0103] The vehicle-mounted terminal 20 can be constructed from a general-purpose computer. That is, the vehicle-mounted terminal 20 can be configured as a computer with processors such as CPU and GPU, main storage devices such as RAM and ROM, and auxiliary storage devices such as EPROM, hard disk drives, and removable media. The auxiliary storage devices store the operating system (OS), various programs, various tables, etc., and by executing the programs stored therein, various functions that meet the predetermined purpose can be achieved, as described later. However, some or all of the functions can also be implemented using hardware circuits such as ASICs or FPGAs.

[0104] The vehicle terminal 20 is configured to have a control unit 21, a storage unit 22, a communication unit 23, and an input / output unit 24.

[0105] The control unit 21 is the component responsible for controlling the vehicle terminal 20. The control unit 21 is composed of information processing units such as CPU (Central Processing Unit) and GPU (Graphics Processing Unit).

[0106] The control unit 21 provides information to the occupants of the vehicle. This information may include, for example, traffic information, navigation information, music, videos, radio broadcasts, and digital television broadcasts. The control unit 21 outputs this information via the input / output unit 24.

[0107] Furthermore, the control unit 21 provides driver assistance functions utilizing an in-vehicle camera. Examples of such driver assistance functions include monitoring the area behind the vehicle when reversing, and monitoring left and right traffic conditions at intersections with poor visibility. When providing driver assistance functions utilizing an in-vehicle camera, the control unit 21 requests the transmission of images from the ECU 10. The control unit 21 can also perform processing to generate an output image based on the image signal transmitted from the ECU 10, and processing to generate a user interface screen.

[0108] Details regarding the processing performed by Control Unit 21 will be described later.

[0109] The storage unit 22 is a component for storing information and is composed of storage media such as RAM, disk, and flash memory. The storage unit 22 stores various programs executed by the control unit 21, as well as the data used by those programs.

[0110] Communication unit 23 is an interface unit used to connect vehicle terminal 20 and ECU 10. Like communication unit 13, communication unit 23 has two interfaces: one corresponding to the path for transmitting images and one corresponding to the path for transmitting control signals. The interface corresponding to the path for transmitting images is, for example, a twisted-pair cable or a coaxial cable. The interface corresponding to the path for transmitting control signals is, for example, a CAN (Controller Area Network).

[0111] The input / output unit 24 is a component that accepts user input operations and provides prompts to the user. Specifically, it consists of a touch panel and its control components, and a liquid crystal display and its control components. In this embodiment, the touch panel and the liquid crystal display are combined into a single touch panel display. Additionally, the input / output unit 24 may also include a speaker or similar device for outputting sound.

[0112] Next, the functions of the control unit 21 will be explained.

[0113] Figure 10 This diagram illustrates the functional modules of the control unit 21 and the flow of data between these modules. The functional modules shown can be implemented by executing programs stored in storage components such as ROM by a CPU or similar device.

[0114] The selection unit 211 sends data (selection data) to the ECU 10 for specifying the camera to acquire images.

[0115] Based on the driver's operation, the selection unit 211 determines which camera's image to request from the ECU 10. For example, when the vehicle is reversing 100 degrees, if... Figure 4As illustrated in (A), sometimes the image from the rear camera is output. In this case, the rear camera is selected. Additionally, when the vehicle is reversing 100 degrees, as... Figure 4 As illustrated in (B), sometimes the output includes images from the rear camera and 360-degree views. In this case, select all cameras used with the external monitor, including the rear camera.

[0116] The camera used can be determined based on either the driver's operation or the vehicle's state. For example, when the gear is in "reverse," the camera used for reversing can be selected. Therefore, the vehicle terminal 20 can also store data that associates the vehicle's state (mode) with the camera used, and generate selection data based on this data. The selection unit 211 can, for example, be based on... Figure 11 The example data is used to generate selection data. Such data can also be stored in storage unit 22.

[0117] The first separation unit 212 receives the image signal sent from the ECU 10 via the communication unit 23 and separates the image signal. For example... Figure 6 As illustrated, the image signals transmitted and received between ECU10 and vehicle terminal 20 conform to the GVIF standard. By separating them, two image signals conforming to the MIPI standard can be obtained. Figure 6 (Ref. reference numerals 601 and 602 in the accompanying drawings). The first camera signal (ref. reference numeral 601) (i.e., the image signal generated by the rear camera) in the separated image signal is sent to the information providing unit 214.

[0118] Image signals other than the first camera signal (reference numeral 602) are sent to the second separation unit 213.

[0119] The second separation unit 213 separates the image signal sent from the first separation unit 212 and acquires multiple second camera signals. These multiple second camera signals are then sent to the information providing unit 214 in the same manner as the first camera signals.

[0120] The information providing unit 214 generates a user interface screen for the vehicle occupants (driver) based on the signals from the first and second cameras. For example, the information providing unit 214 embeds the images captured by the cameras into a predetermined graphic and outputs it. At this time, guide lines indicating the vehicle's position, expected trajectory, etc., can also be superimposed. Alternatively, processing can be performed to generate a single image (such as a 360-degree view) based on images captured by multiple cameras.

[0121] (Processing flow)

[0122] Figure 12This is a flowchart of the processes performed by ECU10 and vehicle terminal 20.

[0123] First, in step S11, the selection unit 211 determines the camera to acquire the image. This step can be performed either when the driver performs a predetermined operation or when the vehicle is in a predetermined state. The selection unit 211 determines the camera to use based on the driver's operation or the vehicle's state and sends the data used to specify that camera (selection data) to the ECU 10.

[0124] In step S12, ECU10 (image selection unit 111) determines one or more cameras for acquiring image signals based on selection data, and selects the image signal from that camera. The selected image signal (second camera signal) is then handed over to image processing unit 112.

[0125] Next, in step S13, the image processing unit 112 performs preprocessing for superimposing the image signals. Specifically, this involves frame rate conversion, resolution conversion, etc., on the second camera signal. It should be noted that any preprocessing used for superimposing multiple image signals can be performed in this step, except for the illustrated processing. It should also be noted that this step can be omitted if unnecessary. The second camera signal processed by the image processing unit 112 is then passed to the first superimposing unit 113.

[0126] Next, in step S14, the first superposition unit 113 superimposes the input signals from multiple second cameras to generate a single image signal.

[0127] Next, in step S15, the second overlay unit 114 overlays the second camera signal superimposed by the first overlay unit 113 with the first camera signal. The superimposed image signal is then sent to the vehicle terminal 20 via an interface for transmitting images.

[0128] In step S16, the first separation unit 212 separates the received image signal to obtain a first camera signal. The separated image signal is then sent to the second separation unit 213. In step S17, the second separation unit 213 separates the received image signal to obtain a second camera signal. The first camera signal and the second camera signal are then sent to the information providing unit 214.

[0129] The information providing unit 214 provides information to the occupants of the vehicle based on the received signals from the first camera and the second camera.

[0130] As explained above, the ECU10 of this embodiment superimposes image signals generated by multiple cameras in multiple stages according to two different image transmission standards. This allows for the superimposed transmission of multiple image signals while ensuring the desired quality requirements are met. Furthermore, for image signals requiring higher quality, the resolution and frame rate can be set higher than those of other image signals.

[0131] (Second Implementation)

[0132] In the first embodiment, the image signal generated by the rear camera is transmitted with high priority. Alternatively, it can be specified from outside the ECU10 which image signal to prioritize (i.e., which image signal to process as the first camera signal).

[0133] The second embodiment is an embodiment in which the vehicle terminal 20 instructs the ECU 10 on the priority of the image signal.

[0134] Figure 13 This is a block diagram of the ECU10 (control unit 11) in the second embodiment. The difference between this embodiment and the first embodiment is that the image selection unit 111 selects both the first camera signal and the second camera signal, and the selection data sent from the vehicle terminal 20 includes a priority specification.

[0135] Figure 9 (B) is an example of selection data in the second embodiment. As shown in the figure, in this embodiment, the selection data is data that establishes an association between the utilized cameras according to each priority. In this example, it is shown that the camera with the highest priority is the rear camera. That is, the first camera signal becomes the image signal generated by the rear camera. The second camera signal becomes the image signal generated by the other cameras.

[0136] The image selection unit 111 selects image signals based on the selection data and determines their output destination according to each priority. Specifically, the image signal corresponding to the camera with the highest priority is sent to the second overlay unit 114 as the first camera signal, and the other image signals are sent to the first overlay unit 113 as the second camera signals.

[0137] The subsequent processing is the same as in the first implementation method.

[0138] As explained above, according to the second embodiment, the priority of the image signals can be dynamically specified. Thus, for example, it is possible to prioritize the image from the rear camera in a scenario where the vehicle is reversing, and to prioritize the image from the side camera in a scenario where vehicles are merging.

[0139] (Variation example)

[0140] The above-described implementation is merely an example, and the present invention can be implemented with appropriate modifications without departing from its spirit.

[0141] For example, the processes and components described in this disclosure can be freely combined and implemented as long as they do not create technical contradictions.

[0142] In addition, in the description of the implementation method, it is assumed that multiple cameras output image signals that conform to the MIPI standard, but the image signals output by the cameras may also conform to other standards.

[0143] Furthermore, in the description of the implementation method, it is assumed that the transmission between ECU10 and vehicle terminal 20 is performed using image signals conforming to the GVIF standard, but image transmission between devices can also utilize other standards. However, the standard that can be used preferably satisfies at least one of the following conditions.

[0144] (1) is a standard that can guarantee the predetermined quality requirements for at least one image signal to be superimposed and transmitted.

[0145] The predetermined quality requirements include, for example, real-time requirements. This enables the real-time transmission of the first camera signal.

[0146] (2) is a standard that can always allocate a frequency band of more than a predetermined value for at least one image signal to be superimposed and transmitted.

[0147] Thus, for example, it is possible to allocate a frequency band above a predetermined value to the first camera signal and allocate the remaining frequency band to the second camera signal.

[0148] (3) is a standard that allows multiple resolutions and frame rates to coexist.

[0149] Furthermore, the processes described as being performed by one device can also be executed by multiple devices. Alternatively, the processes described as being performed by different devices can also be executed by one device. In a computer system, it is possible to flexibly change the hardware architecture (server architecture) used to implement each function.

[0150] This disclosure can also be implemented by supplying a computer program with the functions described in the above embodiments to a computer, which has one or more processors that read and execute the program. Such a computer program can be provided to the computer either by a non-volatile computer-readable storage medium connectable to the computer's system bus or via a network. Non-volatile computer-readable storage media include, for example, any type of disk such as a hard disk (floppy disk, hard disk drive (HDD), etc.), an optical disk (CD-ROM, DVD, Blu-ray disc, etc.), a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card, a flash memory, an optical card, or any type of medium suitable for storing electronic instructions.

Claims

1. An image transmission system, wherein, This image transmission system is applied to vehicles and has the following features: The first transmitter uses a first image transmission standard to superimpose two or more first image signals to generate a second image signal; The second transmitter uses a second image transmission standard different from the first image transmission standard to superimpose the third image signal with the second image signal to generate a fourth image signal; A first receiver receives the fourth image signal transmitted via a single cable and separates the second image signal from the third image signal. as well as The second receiver separates the two or more first image signals from the second image signal. The third image signal is an image signal that requires real-time conditions and is generated by the rear camera of the vehicle. The first image signal is an image signal generated by a camera located outside the rear. A frequency band exceeding a predetermined value is allocated to the third image signal, and the remaining frequency band is allocated to the second image signal that includes two or more of the first image signals. The second image transmission standard is a standard that can guarantee the real-time requirement for the third image signal and a standard that can allocate a frequency band above the predetermined value to the third image signal.

2. The image transmission system according to claim 1, wherein, The third image signal is a single image signal, and the number of the first image signals included in the second image signal is determined based on external specifications.

3. The image transmission system according to claim 1, wherein, The predetermined electronic control unit mounted on the vehicle has the first transmitter and the second transmitter.

4. The image transmission system according to claim 3, wherein, An onboard terminal equipped with the first receiver and the second receiver in the vehicle sends first data to the electronic control unit, specifying two or more cameras that generated the first image signal. The first transmitter in the electronic control unit superimposes the first image signals generated by two or more cameras specified by the first data.

5. A vehicle, wherein, The vehicle has: Multiple cameras capture images of the outside of the vehicle; The first device uses a first image transmission standard to superimpose two or more first image signals to generate a second image signal, and uses a second image transmission standard different from the first image transmission standard to superimpose a third image signal with the second image signal to generate a fourth image signal. as well as The second device receives the fourth image signal transmitted via a single cable, separates the second image signal from the third image signal, and separates the two or more first image signals from the second image signal. The third image signal is an image signal that requires real-time conditions and is generated by the rear camera of the vehicle. The first image signal is an image signal generated by a camera located outside the rear. A frequency band exceeding a predetermined value is allocated to the third image signal, and the remaining frequency band is allocated to the second image signal that includes two or more of the first image signals. The second image transmission standard is a standard that can guarantee the real-time requirement for the third image signal and a standard that can allocate a frequency band above the predetermined value to the third image signal.

6. The vehicle according to claim 5, wherein, The first device is an electronic control unit that manages the plurality of cameras. The second device is a vehicle-mounted terminal.

7. An image transmission method, wherein, This image transmission method is applied to vehicles, and the image transmission method includes: In the first step, a first image transmission standard is used to superimpose two or more first image signals to generate a second image signal. In the second step, a second image transmission standard, different from the first image transmission standard, is used to superimpose the third image signal with the second image signal to generate a fourth image signal. The third step involves transmitting the fourth image signal via a single cable. The fourth step involves receiving the fourth image signal and separating the second image signal from the third image signal; and The fifth step involves separating the two or more first image signals from the second image signal. The third image signal is an image signal that requires real-time conditions and is generated by the rear camera of the vehicle. The first image signal is an image signal generated by a camera located outside the rear. A frequency band exceeding a predetermined value is allocated to the third image signal, and the remaining frequency band is allocated to the second image signal that includes two or more of the first image signals. The second image transmission standard is a standard that can guarantee the real-time requirement for the third image signal and a standard that can allocate a frequency band above the predetermined value to the third image signal.

8. The image transmission method according to claim 7, wherein, It also includes the step of simultaneously outputting the separated third image signal and the separated two or more first image signals.

Images