Binocular camera access method and system

Abstract

The invention relates to a binocular camera access system which comprises a first optical sensor, a second optical sensor, a deserializer, a first processor and an optional second processor. In the system, a first optical sensor and a second optical sensor are respectively connected to two data stream input ports of a deserializer, and two data stream output ports of the deserializer are respectively connected to a first processor and a second processor. And outputting the two paths of input data streams through one or two data stream output ports of the deserializer. A binocular camera access method is applied to the system, and the method further comprises an initialization step, an optical sensor MIPI fault processing step and an optical sensor hot plug fault processing step. According to the invention, the binocular camera can flexibly access a single processor or double processors, can be compatible with selection of different computing power processor chips and different application scenes, and can effectively reduce the overall hardware cost.

Description

Technical Field

[0001] The present invention relates to a binocular camera access method and system, belonging to the technical field of assisted driving. Background Art

[0002] At present, most assisted driving systems use forward-looking binocular perception solutions. There are three existing solutions for the access of forward-looking binocular cameras:

[0003] like Figure 1 Solution 1 shown: In this solution, the binocular camera sends two images to a high-computing power processor chip at the same time through a deserializer chip. The high-computing power processor chip has a high cost.

[0004] like Figure 2 Solution 2 shown: In this solution, the binocular camera sends two images to two processor chips respectively through two deserializer chips. Although this solution can select two lower-cost processor chips to reduce the overall cost of the processor chip, it also increases the cost of a deserializer chip.

[0005] like Figure 3 Solution 3 shown: In this solution, the binocular camera sends two images simultaneously to a high-computing ADAS (Advanced Driver Assistance Systems) processor chip SOC1 through a deserializer chip for perception algorithm processing, and sends the two images to another processor chip SOC2 located in the cockpit through SOC1, and SOC2 then transmits the images to the cockpit vehicle display screen for display. In this solution, the image enters SOC1 and then is transmitted to SOC2 for display, which will greatly increase the vehicle display delay. In addition, it also occupies additional computing power of SOC1, increasing costs. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a binocular camera access method and system, which enable the binocular camera to be flexibly connected to a single processor or a dual processor, is compatible with different application scenarios, and can effectively reduce the overall hardware cost.

[0007] According to a first aspect of the present invention, a binocular camera access method is provided, which is applied to a binocular camera, wherein the binocular camera includes a first optical sensor and a second optical sensor, and the method includes:

[0008] Setting a data stream working mode of the deserializer, wherein the data stream working mode is determined according to a processor configuration mode;

[0009] The deserializer receives the first optical sensor and the second optical sensor data streams;

[0010] The deserializer processes the data stream according to the data stream working mode and outputs the data stream to the processor through one or two ports.

[0011] According to one embodiment, the binocular camera access method further includes the following initialization steps:

[0012] Initializing and configuring the deserializer, the first optical sensor, and the second optical sensor;

[0013] Sending an exposure synchronization signal to the deserializer to synchronize input data streams of the first optical sensor and the second optical sensor;

[0014] According to the processor configuration mode, the first optical sensor data stream is enabled, the second optical sensor data stream is enabled, and the deserializer data stream is enabled.

[0015] According to one embodiment, the binocular camera access method further includes the following first optical sensor MIPI fault processing step:

[0016] receiving a notification signal indicating that the first optical sensor fails;

[0017] According to the processor configuration mode, the MIPI link of the relevant processor input port is reset, the first optical sensor data stream is reset, and the deserializer data stream is reset.

[0018] According to one embodiment, the binocular camera access method further includes the following second optical sensor MIPI fault processing step:

[0019] receiving a notification signal indicating that the second optical sensor fails;

[0020] According to the processor configuration mode, the MIPI link of the relevant processor input port is reset, the second optical sensor data stream is reset, and the deserializer data stream is reset.

[0021] According to one embodiment, the binocular camera access method further includes the following first optical sensor hot plug failure processing step:

[0022] monitoring the connection state of the first optical sensor until a change in state from disconnected to connected is detected, and then reinitializing the first optical sensor;

[0023] According to the processor configuration mode, the MIPI link of the relevant processor input port is reset, the first optical sensor data stream is reset, and the deserializer data stream is reset.

[0024] According to one embodiment, the binocular camera access method further includes the following second optical sensor hot plug failure processing step:

[0025] monitoring the connection state of the second optical sensor until a change in state from disconnected to connected is detected, and then reinitializing the second optical sensor;

[0026] According to the processor configuration mode, the MIPI link of the relevant processor input port is reset, the second optical sensor data stream is reset, and the deserializer data stream is reset.

[0027] According to a second aspect of the present invention, a binocular camera access system is provided, applying the binocular camera access method described in the first aspect of the present invention, the system comprising:

[0028] a first optical sensor, a second optical sensor, a deserializer, a first processor, and an optional second processor;

[0029] The first optical sensor and the second optical sensor are respectively connected to the two data stream input ports of the deserializer, and the two data stream output ports of the deserializer are respectively connected to the first processor and the second processor, and by setting the data stream working mode of the deserializer, the two input data streams of the input port are processed and output through one or two data stream output ports of the deserializer;

[0030] The first processor is also used to set the deserializer and control initialization and fault handling of the system.

[0031] According to one embodiment, the data stream working mode of the deserializer is set to process the input data streams of the first optical sensor and the second optical sensor and output them to the first processor and the second processor respectively.

[0032] According to one embodiment, the data stream working mode of the deserializer is set to process the input data streams of the first optical sensor and the second optical sensor and output them to the first processor and / or the second processor.

[0033] According to one embodiment, the first optical sensor and the second optical sensor both include a sensor, a serializer and a memory; the deserializer also includes a main control channel and a bypass control channel, the main control channel is used to connect to the first processor, and the bypass control channel is used to connect to the second processor; the first processor accesses the deserializer, the first optical sensor and the second optical sensor through the main control channel; when the first processor turns on the bypass control channel, the second processor can access the sensor and memory of the second optical sensor, and at this time the first processor cannot access the sensor and memory of the second optical sensor through the main control channel.

[0034] Technical features and beneficial effects of the present invention:

[0035] The present invention connects the two optical sensors of the binocular camera to the two data stream input ports of a deserializer respectively, and connects the two data stream output ports of the deserializer to two processor chips respectively. By setting the data stream working mode of the deserializer, the two input image data streams are processed and output through one or two data stream output ports of the deserializer chip. On the basis of a hardware technical solution, combined with software control, only one deserializer is used to achieve cooperation with one or two processors, which can achieve flexible configuration, be compatible with the selection of processor chips with different computing power and different application scenarios, and effectively reduce the overall hardware cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the structure in the prior art of using a deserializer chip to simultaneously connect two data streams of a binocular camera to a processor chip.

[0037] Figure 2 It is a structural diagram of using two deserializer chips in the prior art to connect two data streams of a binocular camera to two processor chips respectively.

[0038] Figure 3 It is a schematic diagram of the structure of using a deserializer chip in the prior art to simultaneously connect two data streams of a binocular camera to two processor chips.

[0039] Figure 4 It is a schematic diagram of the hardware connection structure of a binocular camera access system according to an embodiment of the present invention.

[0040] Figure 5 The present invention is a flowchart of initialization steps of a binocular camera access method according to an embodiment of the present invention.

[0041] Figure 6It is a flow chart of the optical sensor MIPI fault processing steps of the binocular camera access method according to the above embodiment of the present invention.

[0042] Figure 7 It is a flow chart of the steps for handling hot-swap faults of optical sensors in the binocular camera access method according to the above embodiment of the present invention.

[0043] Figure 8 Schematic diagram of the internal structure of the optical sensor in the binocular camera access system according to the above embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods.

[0045] The MIPI failure referred to in this article refers to the problem of mismatch between the start and end of the frame of image data due to EMC (Electromagnetic Compatibility) interference, ECC (Error Correcting Code) check error, CRC (Cyclic Redundancy Check) check error, etc.

[0046] The hot-swap failure referred to in this article refers to the problem of optical sensors being disconnected and then reconnected due to situations such as port connector disconnection, wiring harness disconnection, and abnormal sensor power supply.

[0047] Embodiment 1:

[0048] A binocular camera access system, such as Figure 4 As shown, the system includes a binocular camera module 101, a deserializer module 102, and a processor module 103. The binocular camera module includes a first optical sensor 101-1 and a second optical sensor 101-2, the deserializer module includes a deserializer chip, and the processor module includes a first processor 103-1 and a second processor 103-2, wherein the second processor can be implemented as an optional chip.

[0049] The first optical sensor and the second optical sensor both include a sensor, a serializer, a memory chip, and a power management chip. The internal structures and connection methods thereof are as follows: Figure 8 The sensor is used to convert the collected image information into a digital image signal of the MIPI (Mobile Industry Processor Interface) interface, and the serializer is used to convert the digital image signal of the MIPI interface into a high-speed serial signal for transmission.

[0050] The first optical sensor and the second optical sensor are connected to the data stream input ports linkA and linkB of the deserializer chip respectively. The hardware connection is realized through the LVDS (Low Voltage Differential Signaling) harness. The data stream is transmitted through the Serdes (SERializer / DESerializer) high-speed transmission protocol such as GMSL (Gigabit Multimedia Serial Links) or FPD-LINK (Flat Panel Display Link). The deserializer chip is used to parse the received high-speed serial signal and restore it to the digital image signal output of the MIPI interface. The data stream output ports portA (port A) and portB (port B) of the deserializer chip are connected to the first processor and the second processor respectively through the MIPI signal line. The MIPI signal line supports physical interface modes such as DPHY / CPHY / APHY. The first processor and the second processor are used to process the image data stream from the MIPI interface.

[0051] The first processor is a main processor, and is also used to access and set the deserializer, diagnose and process the faults of the first optical sensor and the second optical sensor, and initialize and synchronize the exposure of the first optical sensor and the second optical sensor. The high-speed serial signal transmitted between the serializer and the deserializer also includes a control signal of the processor to control the optical sensor.

[0052] The main control channel (Main Ctrl) and bypass control channel (Pass-through Ctrl) ports of the deserializer chip are connected to the first processor and the second processor respectively. The control channel between the processor and the deserializer adopts one of the serial communication protocols: I2C (Inter-Integrated Circuit) / UART (Universal Asynchronous Receiver / Transmitter) / SPI (Serial peripheral interface). The first processor can access and configure the deserializer chip, the first optical sensor, and the second optical sensor through the main control channel of the deserializer; when the first processor turns on the bypass control channel of the deserializer,

[0053] The second processor can access and configure the sensor and memory of the second optical sensor through the bypass control channel, while the first processor cannot access the sensor and memory of the second optical sensor through the main control channel.

[0054] The GPIO1, GPIO2, GPIO3, GPIO4, and GPIO5 pins of the deserializer chip are respectively connected to the TRIG1, TRIG2, ERRB1, ERRB2, and ERRB3 pins of the first processor.

[0055] The first processor and the second processor are connected via the GPIO11 and GPIO12 pins.

[0056] It is understandable that the above pin names and port names are only examples, and the present application does not limit the pin names and port names.

[0057] The first processor sets the data stream working mode of the deserializer through the main control channel of the deserializer, configures the first optical sensor data stream to be transmitted on the virtual channel 0 (VC0) of the deserializer, and configures the second optical sensor data stream to be transmitted on the virtual channel 1 (VC1) of the deserializer. The deserializer packs the VC field corresponding to the virtual channel into the corresponding image data packet. The first processor and the second processor determine the source of the received image data packet by judging the VC field. The following three application scenarios further illustrate the data stream working mode of the deserializer:

[0058] Application scenario 1: The binocular camera uses a deserializer chip to send two channels of image data to a high-computing processor chip through an output port. In this scenario example, the first processor sets the deserializer to bind VC0 and VC1 to the port A of the deserializer and outputs it to the first processor. Application scenario 1 is also called single processor mode.

[0059] Application scenario 2: The binocular camera uses a deserializer chip to send two channels of image data to two processor chips through two output ports. In this scenario example, the first processor sets the deserializer to bind VC0 to the portA port of the deserializer and outputs it to the first processor, and binds VC1 to the portB port of the deserializer and outputs it to the second processor. Application scenario 2 is also called dual processor mode.

[0060] Application scenario three: The binocular camera uses a deserializer chip to send two-way image data through one output port to a high-computing processor chip for perception algorithm processing, and at the same time sends them to another processor chip through another output port for synchronous display of the cockpit image. In this scenario example, the first processor sets the deserializer to bind VC0 and VC1 to the portA port of the deserializer and outputs it to the first processor, and binds VC0 and VC1 to the portB port of the deserializer and outputs it to the second processor, which displays the image on the cockpit vehicle display. Application scenario three is also called dual-processor dual-channel data bypass mode.

[0061] The technical solution of this embodiment, based on a hardware technical solution, combines software control and can work with one or two processors with only one deserializer, thereby achieving flexible configuration and being compatible with the selection of processor chips with different computing power and different application scenarios. In application scenarios two and three, it can also effectively reduce the overall hardware cost, and in application scenario three, it can also effectively reduce the delay of the vehicle display.

[0062] Embodiment 2:

[0063] A binocular camera access method, which realizes flexible access to a binocular camera by controlling various components of the binocular camera access system of Embodiment 1 of the present application, the method comprising:

[0064] Obtaining a processor configuration mode and setting a data stream working mode of the deserializer, the processor configuration mode including a single processor mode, a dual processor mode, and a dual processor dual-path data bypass mode;

[0065] The deserializer receives the data stream from the first optical sensor and the second optical sensor;

[0066] The deserializer processes the data stream according to the set data stream working mode and outputs it to the processor through one or two ports.

[0067] like Figure 5 As shown, the method also includes the following initialization steps:

[0068] The first processor initializes the deserializer, including setting the working mode of the deserializer data stream and mapping the serializer pins connected to the fault notification pins in the first optical sensor and the second optical sensor to GPIO3 and GPIO4 of the deserializer;

[0069] The first processor initializes the sensors and the string adder in the first optical sensor and the second optical sensor;

[0070] The first processor sends an exposure synchronization signal to the deserializer through the GPIO1 and GPIO2 pins, where the exposure synchronization signal is a square wave signal with an adjustable occupancy ratio and period, and is used to synchronize the input data streams of the first optical sensor and the second optical sensor;

[0071] The first processor determines whether to configure the dual processor mode, if yes, then goes to the single processor initialization step, if yes, then further determines whether to enable the dual-path data bypass mode, if yes, then goes to the dual processor initialization step, if yes, then goes to the bypass mode initialization step;

[0072] Single processor initialization steps: the first processor starts the first optical sensor data stream; the first processor starts the second optical sensor data stream; the first processor starts the deserializer data stream.

[0073] Dual processor initialization steps: the first processor starts the first optical sensor data stream; the first processor starts the bypass control channel; the first processor sends a notification signal to the second processor by pulling down the GPIO11 pin, indicating that the initialization setting is complete; after receiving the notification signal, the second processor starts the second optical sensor data stream; the second processor sends a notification signal to the first processor by pulling down the GPIO12 pin, indicating that the stream opening is complete; after receiving the notification signal, the first processor starts the deserializer data stream.

[0074] Bypass mode initialization steps: the first processor starts the first optical sensor data stream; the first processor starts the second optical sensor data stream; the first processor sends a notification signal to the second processor by pulling down the GPIO11 pin, indicating that the initialization setting is completed; the first processor starts the deserializer data stream.

[0075] like Figure 6 As shown, the binocular camera access method also includes the following first optical sensor MIPI fault processing steps:

[0076] MIPI fault monitoring step 1: The first processor obtains the level status of the deserializer GPIO3;

[0077] Determine whether the GPIO3 level is a low level, if yes, return to MIPI fault monitoring step 1, if yes, further determine whether it is configured as a dual processor mode, if yes, go to MIPI fault processing step 11, if yes, further determine whether it is a dual-path data bypass mode, if yes, go to MIPI fault processing step 11, if yes, go to MIPI fault processing step 12;

[0078] MIPI fault handling 11 steps: the first processor resets its own MIPI link; the first processor resets the first optical sensor data stream; the first processor resets the deserializer data stream.

[0079] MIPI fault handling steps 12: the first processor resets its own MIPI link; the first processor resets the first optical sensor data stream; the first processor sends a notification signal to the second processor by pulling down the GPIO11 pin; after receiving the notification signal, the second processor resets its own MIPI link; the second processor sends a notification signal to the first processor by pulling down the GPIO12 pin; after receiving the notification signal, the first processor resets the deserializer data stream.

[0080] Furthermore, the binocular camera access method also includes the following second optical sensor MIPI fault processing steps:

[0081] MIPI fault monitoring 2 steps: The first processor obtains the deserializer GPIO 4 level status;

[0082] Determine whether the GPIO 4 level is a low level, if not, return to MIPI fault monitoring step 2, if yes, further determine whether it is configured as a dual processor mode, if not, go to MIPI fault handling step 21, if yes, continue with the second optical sensor MIPI fault handling step;

[0083] The first processor sends a notification signal to the second processor by pulling down the GPIO11 pin;

[0084] After receiving the notification signal, the second processor resets its own MIPI link;

[0085] Determine whether to enable the dual-path data bypass mode, if yes, go to MIPI fault processing step 21, if no, go to MIPI fault processing step 22;

[0086] MIPI fault handling 21 steps: the first processor resets its own MIPI link; the first processor resets the second optical sensor data stream; the first processor resets the deserializer data stream.

[0087] MIPI fault handling 22 steps: the second processor resets the second optical sensor data stream; the second processor notifies the first processor by pulling down the GPIO12 pin; after receiving the notification signal, the first processor resets the deserializer data stream.

[0088] like Figure 7 As shown, the binocular camera access method also includes the following first optical sensor hot plug failure processing steps:

[0089] S100: The first processor reads the status of the status register LinkAlock corresponding to the Link A port of the deserializer;

[0090] S101: Determine whether the state changes from disconnected to connected, if not, return to step S100, if yes, continue with the first optical sensor hot-swap fault handling step;

[0091] S102: The first processor reinitializes the sensor and the serializer of the first optical sensor;

[0092] S103: The first processor resets its own MIPI link;

[0093] S104: The first processor resets the first optical sensor data stream;

[0094] S105: Determine whether to enable the dual-path data bypass mode. If yes, go to step S109.

[0095] If yes, continue with the first optical sensor hot-swap fault handling step;

[0096] S106: The first processor sends a notification signal to the second processor by pulling down the GPIO11 pin;

[0097] S107: After receiving the notification signal, the second processor resets its own MIPI link;

[0098] S108: The second processor sends a notification signal to the first processor by pulling down the GPIO12 pin;

[0099] S109: The first processor resets the deserializer data stream.

[0100] Furthermore, the binocular camera access method also includes the following second optical sensor hot plug failure processing steps:

[0101] Hot plug monitoring 2 steps: the first processor reads the status of the status register LinkBlock corresponding to the Link B port of the deserializer;

[0102] Determine whether the state changes from disconnected to connected, if not, return to hot-swap monitoring step 2, if yes, further determine whether it is configured as dual-processor mode, if not, go to the single-processor hot-swap processing step, if yes, further determine whether to enable dual-path data bypass mode, if not, go to the dual-processor hot-swap processing step, if yes, go to the bypass mode hot-swap processing step;

[0103] Single processor hot plug processing steps: the first processor reinitializes the sensor and the serializer of the second optical sensor; the first processor resets its own MIPI link; the first processor resets the second optical sensor data stream; the first processor resets the deserializer data stream.

[0104] Dual-processor hot-swap processing steps: the first processor reinitializes the sensor and serializer of the second optical sensor; the first processor opens the bypass control channel; the first processor notifies the second processor by pulling down the GPIO11 pin; after receiving the notification signal, the second processor resets its own MIPI link; the second processor resets the second optical sensor data stream; the second processor notifies the first processor by pulling down the GPIO12 pin; after receiving the notification signal, the first processor resets the deserializer data stream.

[0105] Bypass mode hot plug processing steps: the first processor reinitializes the sensor and serializer of the second optical sensor; the first processor resets its own MIPI link; the first processor resets the second optical sensor data stream; the first processor notifies the second processor by pulling down the GPIO11 pin; after receiving the notification signal, the second processor resets its own MIPI link; the second processor resets the second optical sensor data stream; the second processor notifies the first processor by pulling down the GPIO12 pin; after receiving the notification signal, the first processor resets the deserializer data stream.

[0106] The technical solution of this embodiment enables the binocular camera to be flexibly connected to a single processor or a dual processor through the interaction of software and hardware, is compatible with the needs of three different application scenarios, and supports the selection of processor chips with different computing power.

[0107] It should be understood that the above is only an exemplary embodiment of the present invention, and the protection scope of the present invention is not limited thereto. Those skilled in the art can easily conceive of modifications, replacements or adaptive changes without departing from the principle and essence of the present invention, which should all be covered within the protection scope of the present invention.

Claims

1. A binocular camera access method, applied to a binocular camera, wherein the binocular camera comprises a first optical sensor and a second optical sensor, characterized in that: The method comprises: Setting a data stream working mode of the deserializer, wherein the data stream working mode is determined according to a processor configuration mode; The deserializer receives the first optical sensor and the second optical sensor data streams; The deserializer processes the data stream according to the data stream working mode and outputs the data stream to the processor through one or two ports.

2. The binocular camera access method according to claim 1, characterized in that: The following initialization steps are also included: Initializing and configuring the deserializer, the first optical sensor, and the second optical sensor; Sending an exposure synchronization signal to the deserializer to synchronize input data streams of the first optical sensor and the second optical sensor; According to the processor configuration mode, the first optical sensor data stream is enabled, the second optical sensor data stream is enabled, and the deserializer data stream is enabled.

3. The binocular camera access method according to claim 1, characterized in that: The following first optical sensor MIPI fault handling steps are also included: receiving a notification signal indicating that the first optical sensor fails; According to the processor configuration mode, the MIPI link of the relevant processor input port is reset, the first optical sensor data stream is reset, and the deserializer data stream is reset.

4. The binocular camera access method according to claim 1, characterized in that: The following second optical sensor MIPI fault handling steps are also included: receiving a notification signal indicating that the second optical sensor fails; According to the processor configuration mode, the MIPI link of the relevant processor input port is reset, the second optical sensor data stream is reset, and the deserializer data stream is reset.

5. The binocular camera access method according to claim 1, characterized in that: The following steps are also included for handling the hot-swap failure of the first optical sensor: monitoring the connection state of the first optical sensor until a change in state from disconnected to connected is detected, and then reinitializing the first optical sensor; According to the processor configuration mode, the MIPI link of the relevant processor input port is reset, the first optical sensor data stream is reset, and the deserializer data stream is reset.

6. The binocular camera access method according to claim 1, characterized in that: The following second optical sensor hot-swap troubleshooting steps are also included: monitoring the connection state of the second optical sensor until a change in state from disconnected to connected is detected, and then reinitializing the second optical sensor; According to the processor configuration mode, the MIPI link of the relevant processor input port is reset, the second optical sensor data stream is reset, and the deserializer data stream is reset.

7. A binocular camera access system, characterized in that: include: a first optical sensor, a second optical sensor, a deserializer, a first processor, and an optional second processor; The first optical sensor and the second optical sensor are respectively connected to the two data stream input ports of the deserializer, and the two data stream output ports of the deserializer are respectively connected to the first processor and the second processor, and by setting the data stream working mode of the deserializer, the two input data streams of the input port are processed and output through one or two data stream output ports of the deserializer; The first processor is further used to configure the deserializer and control initialization and fault handling of the system.

8. The binocular camera access system as claimed in claim 7, characterized in that: The data stream working mode of the deserializer is set to process the input data streams of the first optical sensor and the second optical sensor and output them to the first processor and the second processor respectively.

9. The binocular camera access system according to claim 7, characterized in that: The data stream operation mode of the deserializer is set to process the input data streams of the first optical sensor and the second optical sensor and output them to the first processor and / or the second processor.

10. The binocular camera access system according to any one of claims 7 to 9, characterized in that: The first optical sensor and the second optical sensor each include a sensor, a serializer, and a memory; the deserializer further includes a main control channel and a bypass control channel, the main control channel is used to connect to the first processor, and the bypass control channel is used to connect to the second processor; The first processor accesses the deserializer, the first optical sensor and the second optical sensor through the main control channel; when the first processor turns on the bypass control channel, the second processor can access the sensor and memory of the second optical sensor, and at this time the first processor cannot access the sensor and memory of the second optical sensor through the main control channel.

Images