Trinocular camera access method based on high-speed MIPI switch and PWM control

Abstract

The invention relates to a trinocular camera access method based on a high-speed MIPI switch and PWM control, and belongs to the technical field of image acquisition. The method comprises the following steps: performing time division multiplexing on mobile industry processor interface hardware to obtain MIPI signals output by a first image sensor, a second image sensor and a third image sensor; mIPI signals of the first image sensor and the second image sensor are accessed to the input end of the second high-speed switch chip through the first high-speed switch chip; an MIPI signal output by the third image sensor is directly accessed to the input end of the second high-speed switch chip; the input end of the second high-speed switch chip combines the signals and then inputs the signals to the camera SOC chip, and the camera SOC chip carries out sequential control on the first high-speed switch chip and the second high-speed switch chip according to the PWM control signals. And the expansibility and flexibility of the system are enhanced, so that the chip can easily adapt to diversified hardware configuration and scene requirements.

Description

Technical Field

[0001] The present invention belongs to the technical field of image acquisition, and in particular relates to a three-eye camera access method based on high-speed MIPI switch and PWM control. Background Art

[0002] With the continuous advancement of security technology and the improvement of people's safety awareness, the security industry has developed rapidly. With the rapid development of digitalization, intelligence and Internet of Things technology, the security industry will usher in more innovative development opportunities. Multi-eye security cameras have a wider viewing angle and higher clarity, which can meet the security needs in more complex scenes. As an important branch of the security industry, its market prospects are very broad. Nowadays, consumers' expectations for cameras have gradually upgraded from "seeing" to "seeing clearly" and "seeing widely". It is expected that the camera can have a clear texture level, a wider field of view, and can capture a wider scene to meet the shooting needs in different scenes.

[0003] Multi-cameras are more expensive than traditional monocular cameras, which may limit their popularity and application in the mid- and low-end markets. How to reduce costs while maintaining performance is an important challenge facing the expansion of the multi-camera market. Summary of the invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a three-eye camera access method based on high-speed MIPI switch and MPW control;

[0005] The purpose of the present invention can be achieved through the following technical solutions:

[0006] A three-eye camera access method based on high-speed MIPI switch and PWM control, comprising:

[0007] The MIPI signals output by the first image sensor, the second image sensor and the third image sensor are obtained by time-division multiplexing the mobile industry processor interface hardware; the MIPI signals of the first image sensor and the second image sensor are connected to the input end of the second high-speed switch chip through the first high-speed switch chip; the MIPI signal output by the third image sensor is directly connected to the input end of the second high-speed switch chip; the input end of the second high-speed switch chip merges the signals and inputs them to the camera SOC chip, and the camera SOC chip performs timing control on the first high-speed switch chip and the second high-speed switch chip according to the PWM control signal to ensure that the MIPI signals of the three image sensors can be transmitted in a predetermined order and time interval.

[0008] Specifically, the camera SOC chip provides two PWM signals to control the states of the first high-speed switch chip and the second high-speed switch chip and the synchronization of the first image sensor, the second image sensor and the third image sensor; when the PWM signal of the first high-speed switch chip is at a high level, the signal of the first image sensor is output, and when the PWM signal of the first high-speed switch chip is at a low level, the signal of the second image sensor is output; when the PWM signal of the second high-speed switch chip is at a high level, the signal of the third image sensor is output, and when the PWM signal of the second high-speed switch chip is at a low level, the signal of the first high-speed switch chip is output.

[0009] Specifically, the first image sensor, the second image sensor and the third image sensor perform hardware frame synchronization according to the rising edge or falling edge of the MPW signal provided by the camera SOC chip; the timing of the image sensors is controlled by a clock provided externally.

[0010] Specifically, the first image sensor, the second image sensor and the third image sensor will output a new frame header after a rising edge or a falling edge of a given PWM signal is issued.

[0011] Specifically, the timing of the two signals output by the first image sensor and the second image sensor, including the complete data from the frame head to the frame tail, is completely wrapped in the blanking area of ​​the MIPI output of this sensor.

[0012] Specifically, an average buffer margin of 2 ms is set between the frame headers and frame tails of two paths of the first high-speed switch chip and the second high-speed switch chip.

[0013] Specifically, the output format heights of the first image sensor, the second image sensor, and the third image sensor are set to different heights, and the camera SOC chip distinguishes the image sensors according to the output format height statistics.

[0014] Specifically, the first image sensor, the second image sensor and the third image sensor are set to slave mode, and vertical synchronization signals and horizontal synchronization signals are input to corresponding pins of the sensors to control frame and row synchronization.

[0015] Specifically, the first high-speed switch chip and the second high-speed switch chip manage the timing signal and data transmission of the image sensor through the FPGA component, and expand a large-capacity RAM outside the FPGA component to cache image data.

[0016] Specifically, the camera SOC chip has a built-in image processing unit that can process and analyze the image data from the first image sensor, the second image sensor and the third image sensor in real time; the camera SOC chip further includes a high-speed data interface for transmitting the processed image data to an external device or a storage medium; the camera SOC chip is also configured with multiple GPIO pins, which can be used to control external devices or receive external signals to achieve more complex control logic and system integration.

[0017] The beneficial effects of the present invention are:

[0018] Through the coordinated work of high-speed switch chips and FPGA components, the image sensor data can be quickly read and processed, and the image data transmission efficiency can be improved; by expanding RAM outside the FPGA component, the reliance on expensive built-in cache resources is reduced, and the system cost is reduced. The built-in image processing unit and high-speed data interface of the SOC chip enable the system to adapt to a variety of different application scenarios, enhancing the flexibility and scalability of the system; using PWM control technology, the precise synchronization of the image sensor is ensured, thereby improving the imaging quality and achieving precise timing control. By integrating multiple functions into the SOC chip, the complexity of the external circuit is reduced, the design difficulty and production cost are reduced, and the hardware design is simplified.

[0019] The present invention allows a chip with a single MIPI design to access three MIPI signals through high-speed MIPI switches and PWM control. This lowers the threshold for using multi-cameras, enhances the scalability and flexibility of the system, saves design costs, optimizes resource utilization, and enables the chip to easily adapt to diverse hardware configurations and scenario requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0021] Figure 1 is a system flow chart of the present invention;

[0022] Figure 2 It is a schematic diagram of each node signal in the present invention;

[0023] Figure 3 This is a schematic diagram of output signals of the first high-speed switch chip in the present invention;

[0024] Figure 4 This is a schematic diagram of output signals of the second high-speed switch chip in the present invention;

[0025] Figure 5 It is a schematic diagram of the timing of the frame head and frame tail in the present invention. DETAILED DESCRIPTION

[0026] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0027] See also Figure 1-3 , a three-eye camera access method based on high-speed MIPI switch and PWM control, comprising:

[0028] The MIPI signals output by the first image sensor, the second image sensor and the third image sensor are obtained by time-division multiplexing the mobile industry processor interface hardware; the MIPI signals of the first image sensor and the second image sensor are connected to the input end of the second high-speed switch chip through the first high-speed switch chip; the MIPI signal output by the third image sensor is directly connected to the input end of the second high-speed switch chip; the input end of the second high-speed switch chip merges the signals and inputs them to the camera SOC chip, and the camera SOC chip performs timing control on the first high-speed switch chip and the second high-speed switch chip according to the PWM control signal to ensure that the MIPI signals of the three image sensors can be transmitted in a predetermined order and time interval.

[0029] In this embodiment, the basic idea is to time-division multiplex the MIPI (Mobile Industry Processor Interface) hardware, and three MIPI signals are respectively passed through two high-speed switch chips (Switch chips), wherein the MIPI outputs of two image sensors are first connected to one input of another high-speed switch chip through the high-speed switch chip, and then combined with the signal of the third image sensor and input to the camera SOC chip (Systemon Chip). The camera SOC chip provides two PWM (Pulse Width Modulation) signals, which respectively control the states of the two switch chips and the synchronization of the three image sensors in slave mode. Image sensor 0 and image sensor 1 are connected to the two inputs of Switch0 chip. When PWM is high, the data of image sensor 0 is output, and when PWM is low, the data of image sensor 1 is output. Similarly, the outputs of image sensor 2 and Switch0 chip are connected to the two inputs of Switch1 chip respectively. When PWM is high, the data of image sensor 2 is output, and when PWM is low, the data of Switch0 is output. Finally, the three timings are completely staggered to ensure that the subsequent SOC chips can be time-division multiplexed. The whole system process is as follows: Figure 1As shown. Specifically, the image sensor, camera SOC chip, etc. need to cooperate in the implementation process, and the image sensor needs to be configured as slave mode. Slave mode means that the image sensor performs hardware frame synchronization through the rising or falling edge of the PWM signal provided by the SOC chip, and the timing of the image sensor is controlled by the external clock, thereby organically combining the two. After configuring this mode, the image sensor will spit out the frame header of a new frame only after the given rising / falling edge is sent, thereby controlling the timing of the two image sensors through hardware. After configuring the PWM and sensor slave mode, the key signals of each node are as follows Figure 2 As shown in Figure 1, for a system with multiple image sensors, timing is the most important control. The image sensor timing effect that we ultimately want to achieve in actual testing is as follows: Figure 3 and Figure 4 As shown in the figure, the timing of the output of the other two sensors from the oscilloscope signal, including the complete data from the frame head to the frame tail, is completely wrapped in the blanking area of ​​the MIPI output of this sensor.

[0030] In order to ensure the robustness and stability of the entire system, an average buffer margin of 2ms is left between the frame headers and frame tails of the two channels. Figure 5 As shown, the system can calmly cope with short delays, interference or jitter that may be encountered on the signal transmission path, ensuring that the integrity and continuity of the frame structure are not affected, thereby maintaining high efficiency and accuracy of data transmission and enhancing the system's fault tolerance.

[0031] Similarly, for scenarios where the fluctuation is greater than the buffer margin, channel correction of the timing is also required. We recommend configuring the output format heights of image sensors 0, 1, and 2 to be slightly different, so that the subsequent camera SOC chip will distinguish the image sensors based on the statistics of the format height. This is because the input data enters through the same MIPI. For MIPI hardware, the two channels of data are consistent and cannot be truly matched. If the data of a sensor suddenly becomes abnormal, it may affect the entire subsequent timing. At this time, a software judgment mechanism is needed to help MIPI hardware to divide the data, and frame loss and channel flip correction will be performed for those that do not meet expectations.

[0032] Since the current frame height information is the statistical result of the previous MIPI frame in the hardware statistics, the channel correction of the SOC will only take effect in the next frame, which causes the channel flip correction to usually lag behind by two frames. After the channel flip correction, the image will return to normal timing.

[0033] Specifically, the camera SOC chip provides two PWM signals to control the states of the first high-speed switch chip and the second high-speed switch chip and the synchronization of the first image sensor, the second image sensor and the third image sensor; when the PWM signal of the first high-speed switch chip is at a high level, the signal of the first image sensor is output, and when the PWM signal of the first high-speed switch chip is at a low level, the signal of the second image sensor is output; when the PWM signal of the second high-speed switch chip is at a high level, the signal of the third image sensor is output, and when the PWM signal of the second high-speed switch chip is at a low level, the signal of the first high-speed switch chip is output.

[0034] Specifically, the first image sensor, the second image sensor and the third image sensor perform hardware frame synchronization according to the rising edge or falling edge of the MPW signal provided by the camera SOC chip; the timing of the image sensors is controlled by a clock provided externally.

[0035] Specifically, the first image sensor, the second image sensor and the third image sensor will output a new frame header after a rising edge or a falling edge of a given PWM signal is issued.

[0036] Specifically, the timing of the two signals output by the first image sensor and the second image sensor, including the complete data from the frame head to the frame tail, is completely wrapped in the blanking area of ​​the MIPI output of this sensor.

[0037] Specifically, an average buffer margin of 2 ms is set between the frame headers and frame tails of two paths of the first high-speed switch chip and the second high-speed switch chip.

[0038] Specifically, the output format heights of the first image sensor, the second image sensor, and the third image sensor are set to different heights, and the camera SOC chip distinguishes the image sensors according to the output format height statistics.

[0039] Specifically, the first image sensor, the second image sensor and the third image sensor are set to slave mode, and vertical synchronization signals and horizontal synchronization signals are input to corresponding pins of the sensors to control frame and row synchronization.

[0040] Specifically, the first high-speed switch chip and the second high-speed switch chip manage the timing signal and data transmission of the image sensor through the FPGA component, and expand a large-capacity RAM outside the FPGA component to cache image data.

[0041] Specifically, the camera SOC chip has a built-in image processing unit that can process and analyze the image data from the first image sensor, the second image sensor and the third image sensor in real time; the camera SOC chip further includes a high-speed data interface for transmitting the processed image data to an external device or a storage medium; the camera SOC chip is also configured with multiple GPIO pins, which can be used to control external devices or receive external signals to achieve more complex control logic and system integration.

[0042] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device.

[0043] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0044] The program code included on the computer readable medium can be transmitted with any appropriate medium, including but not limited to wireless, electric wire, optical cable, RF, etc., or any suitable combination of the above. The computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or their combinations, and the programming language includes object-oriented programming languages-such as Java, Smalltalk, C++, and also includes conventional procedural programming languages-such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0045] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A three-eye camera access method based on high-speed MIPI switch and PWM control, characterized in that: include: The MIPI signals output by the first image sensor, the second image sensor and the third image sensor are obtained by time-division multiplexing the mobile industry processor interface hardware; the MIPI signals of the first image sensor and the second image sensor are connected to the input end of the second high-speed switch chip through the first high-speed switch chip; the MIPI signal output by the third image sensor is directly connected to the input end of the second high-speed switch chip; the input end of the second high-speed switch chip merges the signals and inputs them to the camera SOC chip, and the camera SOC chip performs timing control on the first high-speed switch chip and the second high-speed switch chip according to the PWM control signal to ensure that the MIPI signals of the three image sensors can be transmitted in a predetermined order and time interval.

2. The method according to claim 1, characterized in that The camera SOC chip provides two PWM signals to control the states of the first high-speed switch chip and the second high-speed switch chip and the synchronization of the first image sensor, the second image sensor and the third image sensor; when the PWM signal of the first high-speed switch chip is at a high level, the signal of the first image sensor is output, and when the PWM signal of the first high-speed switch chip is at a low level, the signal of the second image sensor is output; when the PWM signal of the second high-speed switch chip is at a high level, the signal of the third image sensor is output, and when the PWM signal of the second high-speed switch chip is at a low level, the signal of the first high-speed switch chip is output.

3. The method according to claim 1, characterized in that The first image sensor, the second image sensor and the third image sensor perform hardware frame synchronization according to the rising edge or falling edge of the MPW signal provided by the camera SOC chip; the timing of the image sensor is controlled by a clock provided externally.

4. The method according to claim 3, characterized in that The first image sensor, the second image sensor and the third image sensor will output a new frame header after a rising edge or a falling edge of a given PWM signal is issued.

5. The method according to claim 1, characterized in that The timing of the two signals output by the first image sensor and the second image sensor, including the complete data from the frame head to the frame tail, is completely wrapped in the blanking area of ​​the sensor MIPI output.

6. The method according to claim 1, characterized in that An average buffer margin of 2 ms is set between the two frame headers and frame tails of the first high-speed switch chip and the second high-speed switch chip.

7. The method according to claim 1, characterized in that The output format heights of the first image sensor, the second image sensor and the third image sensor are set to different heights, and the camera SOC chip distinguishes the image sensors according to the statistics of the output format heights.

8. The method according to claim 1, characterized in that The first image sensor, the second image sensor and the third image sensor are set to slave mode, and the vertical synchronization signal and the horizontal synchronization signal are input to the corresponding pins of the sensor to control the synchronization of the frame and the line.

9. The method according to claim 1, characterized in that: The first high-speed switch chip and the second high-speed switch chip manage the timing signal and data transmission of the image sensor through the FPGA component, and expand a large-capacity RAM outside the FPGA component to cache image data.

10. The method according to claim 1, characterized in that The camera SOC chip has a built-in image processing unit, which is used to process and analyze the image data from the first image sensor, the second image sensor and the third image sensor in real time; the camera SOC chip further includes a high-speed data interface for transmitting the processed image data to an external device or a storage medium; the camera SOC chip is also configured with a plurality of GPIO pins, which are used to control external devices or receive external signals to realize control logic and system integration.

Images