An image sensor and camera

Abstract

The utility model relates to technical field of image acquisition especially relates to an image sensor and camera, including an image acquisition and processing component, image acquisition and processing component include: image sensor acquisition module, FPGA SOC module, DDR4 output cache module, image communication module, command transmission communication module, USB3.0 module, eMMC module, serial communication module and power management module. The image sensor and camera, high -speed MIPI interface technology is used between image sensor acquisition module and FPGA SOC module, improves output transmission bandwidth, reduces delay time. Meanwhile, MIPI signal adopts differential signal transmission, has very strong anti -interference ability and transmission rate etc. advantage, can satisfy the high -speed transmission demand of billion -pixel data.

Description

Technical Field

[0001] This utility model relates to the technical field of image acquisition, and in particular to an image sensor and camera. Background Technology

[0002] With the advancement of technology and the progress of the times, the requirements for resolution and frame rate of signal acquisition and processing are becoming increasingly stringent for image acquisition sensors, represented by CMOS CCDs. Traditional image acquisition sensor systems often face problems such as insufficient data transmission bandwidth and high power consumption when processing images at the gigapixel level, i.e., ultra-high resolution or ultra-high frame rate. Utility Model Content

[0003] In view of the problems of insufficient transmission bandwidth and high power consumption in the existing image acquisition methods, this utility model is proposed.

[0004] Therefore, one of the objectives of this invention is to provide an image sensor, which aims to provide a module for processing gigapixel-level pixels.

[0005] To solve the above-mentioned technical problems, this utility model provides the following technical solution: an image sensor, including an image acquisition and processing component, wherein the image acquisition and processing component includes: an image sensor acquisition module, an FPGA SOC module, a DDR output buffer module, an image communication module, a command transmission communication module, a USB module, an eMMC module, a serial communication module, and a power management module;

[0006] The image sensor acquisition module includes a multi-pixel two-dimensional array, which is divided into eight pixel sub-arrays of the same size. In terms of electrical architecture, the image sensor acquisition module is connected to the FPGA SOC module via a MIPI data link to transmit the acquired image signal to the FPGA SOC module.

[0007] The FPGA SOC module adopts a multi-core processor and programmable logic architecture. After receiving data from the image sensor acquisition module through the MIPI data link, it performs corresponding algorithm processing on the data, receives external control commands through the serial communication module or a command transmission communication module, stores the image in the eMMC module, and transmits the image data to an external terminal through an image communication module or a USB module communication interface.

[0008] The DDR output cache module is connected to the PL and PS terminals of the FPGA SOC module through a parallel interface to ensure that data can be read and written quickly.

[0009] The image communication module is connected to the GTH transceiver of the FPGA SOC module and is used to upload the acquired images to the server or remote terminal without compression after processing.

[0010] The command transmission communication module is connected to the FPGA SOC module through the PHY chip interface signal, and assigns various operation commands to the FPGA SOC module;

[0011] The USB module is connected to the FPGA SOC module in two parts. The first part is the low-speed part, which is connected to the PS terminal of the FPGA SOC module through the USB interface signal. The second part is the high-speed part, where the USB connector is directly connected to the PS terminal of the FPGA SOC module for data interaction.

[0012] The eMMC module is used to provide stored images to the FPGA SOC module. The FPGA SOC module is connected to the eMMC module chip through the PS terminal and uses the eMMC module protocol special processing unit to realize the parsing and management of the commands of the eMMC module chip.

[0013] The serial communication module provides serial communication functionality for the entire system, connecting to the PS terminal of the FPGA SOC module via a serial communication signal to exchange information with the outside world; and,

[0014] The power management module provides power supply and power consumption management for the entire system, adjusting the voltage and current of the power supply according to the system's operating status.

[0015] The advantages of this image sensor are: the high-speed MIPI interface technology between the image sensor acquisition module and the FPGA SOC module improves the output transmission bandwidth and reduces latency. Simultaneously, the MIPI signal uses differential signal transmission, which has strong anti-interference capabilities and high transmission rates, meeting the high-speed transmission requirements of billions of pixel data.

[0016] One of the objectives of this invention is to provide a camera that can handle cameras with resolutions of hundreds of millions of pixels.

[0017] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a camera, including the aforementioned image sensor.

[0018] The beneficial effects of this invention are: it can process images at the level of hundreds of millions of pixels, i.e., ultra-high resolution or ultra-high frame rate, at high speed. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this utility model, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this utility model. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 The system topology diagram of Embodiment 1 is shown;

[0021] Figure 2 The schematic diagram of the VPS4114A image sensor acquisition module is shown.

[0022] Figure 3 The diagram shows the topology of the interface signal module connected to the FPGA SOC module.

[0023] Figure 4 The schematic diagram of the PL terminal of the FPGA SOC module connecting to the DDR4 interface is shown.

[0024] Figure 5 The schematic diagram of the PS terminal of the FPGA SOC module connecting to the DDR4 interface is shown.

[0025] Figure 6 The schematic diagrams of two DDR output cache modules are shown.

[0026] Figure 7 The schematic diagram of the image communication module is shown;

[0027] Figure 8 The schematic diagram shows the command transmission communication module receiving the target signal and assigning the FPGA SOC module to perform various operation commands.

[0028] Figure 9 The schematic diagram shows the data interaction between the USB module and the FPGA SOC module;

[0029] Figure 10 The schematic diagram of the eMMC module is shown;

[0030] Figure 11 The schematic diagram of the serial communication module is shown.

[0031] Figure 12 A partial schematic diagram of the power management module is shown;

[0032] Figure 13 Another part of the schematic diagram of the power management module is shown;

[0033] Figure 14 A partial schematic diagram of the temperature sensor module is shown. Detailed Implementation

[0034] To enable those skilled in the art to better understand this utility model, the present utility model will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0035] The terminology used in this invention refers to those general terms currently widely used in the art in consideration of the functionality of this invention; however, these terms may vary according to the intent, precedent, or new technology of those skilled in the art. Furthermore, specific terms may be chosen by the applicant, and in such cases, their detailed meanings will be described in the detailed description of this invention. Therefore, the terminology used in this specification should not be construed as simple names, but rather based on the meaning of the terms and the overall description of this invention.

[0036] Example 1, referring to Figures 1-14 The first embodiment of this utility model provides an image sensor, including an image acquisition and processing component. The image acquisition and processing component includes: an image sensor acquisition module 100, an FPGA SOC module 200, a DDR output buffer module 300, an image communication module 400, a command transmission communication module 500, a USB module 600, an eMMC module 700, a serial communication module 800, and a power management module 900.

[0037] The image sensor acquisition module 100 includes a multi-pixel two-dimensional array, which is divided into eight pixel sub-arrays of the same size. In terms of electrical architecture, the image sensor acquisition module 100 is connected to the FPGA SOC module 200 via a MIPI data link to transmit the acquired image signal to the FPGA SOC module 200.

[0038] The preferred image sensor acquisition module 100 is model VPS4114A, a high-quality, high-performance image acquisition and processing chip capable of capturing 1 billion-pixel high-definition images. This chip features automatic power-on calibration. Shutter modes include global exposure and rolling exposure. Sampling modes include full sampling, subsampling, and windowing. The pixel pitch is 0.702um * 0.702um. The pixel array consists of 34048 rows, with 31200 pixels per row, for a total array size of 31200x34048. The entire pixel array is divided into eight subarrays, each 7800x17024 pixels in size. Each pixel outputs 10 bits of data, as shown in Figure 2.

[0039] The FPGA SOC module 200 employs at least one processor and a programmable logic architecture. After receiving data from the image sensor acquisition module 100 via a MIPI data link, it performs corresponding algorithmic processing on the data. It receives external control commands via a serial communication module 800 or a command transmission communication module 500, stores the image in the eMMC module 700, and transmits the image data to an external terminal via an image communication module 400 or a USB module 600 communication interface. The at least one processor can be a multi-core processor, a powerful single-core processor, or a combination of several single-core processors as known to those skilled in the art to form a suitable processor. The USB module can be, for example, a USB 3.0 module or other applicable USB modules under the USB standard architecture.

[0040] The image sensor acquisition module 100 and the FPGA SOC module 200 are primarily connected via MIPI signals. The acquired data is output to the FPGA SOC module 200 through a MIPI data link. After receiving the data, the FPGA SOC module 200 first decodes the data and then performs serial-to-parallel conversion. The data from the entire acquired signal pixel array is output through eight MIPI interfaces, each corresponding to one pixel subarray. Each MIPI interface has a width of 6Gbps, resulting in a total data bandwidth of 48Gbps for all eight interfaces.

[0041] The FPGA SOC module 200 uses a chip from Xilinx's Zynq UltraScale+MPSOC, which is a high-performance programmable system-on-a-chip (SOC) that integrates a multi-core processor and an FPGA architecture.

[0042] Core structure:

[0043] PS's ARM core processor;

[0044] Quad-core ARM Cortex-A53: 64-bit processor with a clock speed of up to 1.5GHz, supporting symmetric multiprocessing and virtualization;

[0045] PL's programmable logic;

[0046] Logic unit: 504K;

[0047] DSP Slices: 2520, supporting high-precision signal processing;

[0048] Block RAM: 32.1M, supporting distributed storage requirements;

[0049] High-speed GTXbank: Supports GTH transceivers up to 16.3Gbps.

[0050] After receiving data from the VPS4114A image sensor acquisition module 100 via the MIPI data link, the FPGA SOC module 200 performs a series of algorithmic processing steps on the data. It then receives control commands via the serial communication module 800 or the command transmission communication module 500, stores the image in the eMMC module 700, and simultaneously transmits the image data to the terminal via the image communication module 400 or the USB module 600. The FPGA SOC module 200 also performs power timing control on the VPS4114A image sensor acquisition module, controlling the power-on sequence and time, and configuring the sensor chip registers via I2C signals. The interface signal modules are connected to the FPGA SOC module topology as shown in the diagram. Figure 3 As shown.

[0051] Furthermore, the DDR output cache module 300 is connected to the PL and PS terminals of the FPGA SOC module 200 through a parallel interface to ensure that data can be read and written quickly. The main function of the DDR output cache module 300 is data caching.

[0052] The DDR output cache module 300 may be, for example, a DDR4 module, a DDR3 module, or a DDR5 module;

[0053] The preferred DDR output cache module 300 is Micron's MT40A1G16RC-062E IT, with a single chip capacity of 2GB, a 16-bit bus width, and a clock frequency of 2400MHz. The DDR output cache module 300 connects to the PL and PS banks of the FPGA SOC module 200 via a high-speed parallel interface, ensuring fast data read and write speeds. The hardware module design uses two DDR4 modules: a PS-side DDR4 module and a PL-side DDR4 module for the FPGA SOC module 200. Each DDR4 module has a 64-bit bus width and a capacity of 8GB. Figure 4 Connect the DDR4 interface to the PL terminal of the FPGA SOC module 200. Figure 5 Connect the DDR4 interface to the PS side of the FPGA SOC module 200. Figure 6 This is a partial schematic diagram of two DDR4 modules.

[0054] The image communication module 400 connects to the GTH transceiver of the FPGA SOC module 200. It is used to process and upload uncompressed images to a server or remote terminal. The image communication module 400 is an optical port supporting 10 Gigabit Ethernet. It enables high-speed network output transmission, facilitating the processing and uncompressed uploading of acquired images to a server or remote terminal. The 10 Gigabit Ethernet module integrates 10Gbps SerDes, supports SFP+, and can be directly connected to the FPGA's GTH transceiver.

[0055] The command transmission communication module 500 connects to the FPGA SOC module 200 via a PHY chip interface signal, assigning various operation commands to the FPGA SOC module 200. The command transmission communication module 500 supports Gigabit Ethernet; the Gigabit Ethernet chip integrates a MAC layer, enabling high-speed network data output transmission in existing network communication environments via the TCP / IP protocol. Currently, this module primarily uses Gigabit transmission control commands. After receiving the target signal, the Gigabit Ethernet connects to the FPGA SOC via the RGMII signal, assigning various operation commands to the FPGA SOC.

[0056] The USB module 600 connects to the FPGA SOC module 200 in two parts. The first part is the low-speed section, where the chip connects to the PS terminal of the FPGA SOC module 200 via a USB interface signal. The second part is the high-speed section, where the USB connector directly connects to the PS terminal of the FPGA SOC module 200 for data exchange. The USB module can be, for example, a USB 3.0 module or other applicable USB modules within the USB standard architecture.

[0057] The USB module 600 primarily outputs image data for the entire system. It connects to the FPGA SOC module 200 in two parts. The first part is the low-speed section, where the USB3320C chip connects to the PS terminal BANK502 of the FPGA SOC module 200 via the ULPI signal, with a communication bandwidth of 480MHz. The second part is the high-speed section, where the USB module 600's connector directly connects to the PS terminal BANK505, with a communication bandwidth of 5Gbps. The USB module 600 interacts with the FPGA SOC module 200 through this high-speed section.

[0058] The eMMC module 700 is used to provide stored images to the FPGA SOC module 200. The FPGA SOC module 200 is connected to the eMMC module 700 chip through the PS terminal and uses the dedicated processing unit of the eMMC module 700 protocol to realize the parsing and management of the commands of the eMMC module 700 chip.

[0059] The eMMC module 700 primarily provides image storage. The FPGA SOC module 200 connects to the eMMC module 700 chip via the PS-side BANK500 and uses a dedicated processing unit based on the eMMC 5.1 protocol to parse and manage commands from the eMMC module 700 chip.

[0060] The serial communication module 800 provides serial communication functionality for the entire system. It connects to the PS terminal of the FPGASOC module 200 via serial communication signals to communicate information with the outside world.

[0061] The serial communication module 800 mainly provides serial communication functionality for the entire system. Also known as the RS485 module, the serial communication module 800 is connected to the PS terminal of the FPGA SOC module 200 via a TTL signal. The FPGA SOC module 200 communicates with the outside world through the serial port signal.

[0062] The Power Management Module 900 provides power supply and power consumption management for the entire system, adjusting the voltage and current according to the system's operating status. The Module 900 primarily provides a stable power supply and power consumption management. The system mainly uses PMIC chips from TI and MPS, efficiently providing multiple outputs and precise voltage regulation, adjusting the voltage and current according to the system's operating status. This module has a 12V input power supply. The input terminals have two reverse-insertion protection diodes (D27 and D28), and a F1 fuse to prevent short circuits. Following the fuse is the EMI module for electromagnetic compatibility. After the EMI module is an NMOS transistor, and next to the PMOS transistor are two components (D25 and D15) for stabilizing the 12V power supply and for TVS (Transient Voltage Suppressor) operation, respectively.

[0063] The 12V power supply from the PMOS transistor is fed into the U3 module. After being stepped down, it provides a core voltage of 0.85V to the FPGA SOC module 200 and simultaneously outputs a power-good signal. This power-good signal is sent to the Enable pin of the U2 PMIC chip, thus enabling the U2 PMIC chip to operate. The U2 PMIC chip then outputs the required power voltages for the system, such as 0.85V / 1.2V / 0.9V / 1.8V / 1.2V / 3.3V / VTT / 2.5V, after stepping down the voltage. Once this PMIC module is up, the subsequent DC-DC and LDO modules will start up sequentially.

[0064] Furthermore, a temperature sensor module 1000 is also connected to the FPGA SOC module 200. This module is mainly used to monitor the temperature on the board. The temperature sensor module 1000 and the FPGA SOC module 200 communicate mainly through I2C. The FPGA SOC module 200 collects data from the temperature sensor chip through I2C to obtain the temperature information on the board.

[0065] The optimizations in this embodiment are as follows: 1. Data transmission optimization: High-speed serial interface technology is used between the image sensor acquisition module 100 and the FPGA SOC module 200 to improve the output transmission bandwidth and reduce the delay time. At the same time, the MIPI signal uses differential signal transmission, which has the advantages of strong anti-interference ability and high transmission rate, and can meet the high-speed transmission requirements of hundreds of millions of pixel data;

[0066] The output stream processing architecture is located within the FPGA SOC module 200. The PL (Parallel Processing) end of the FPGA SOC module 200 processes parallel data, while the PS (Power Sequence Processing) end processes the image sequentially. This avoids data congestion and waiting time. The data stream processing architecture decomposes the image processing algorithm into multiple stages, with each stage and part completing its own task. Through the parallel and pipelined operations of both the PL and PS of the FPGA SOC module 200, the requirements for ultra-high-speed image data throughput can be easily met.

[0067] 2. Processing Flow Optimization: Parallel computing optimizes the image processing algorithm, fully utilizing the parallel processing capabilities of the PL terminal of the FPGA SOC module 200. The image processing algorithm is decomposed into two parallel subtasks, which are distributed to different modules for accelerated processing. Parallel computing is employed to process multiple images or different regions of the same image simultaneously, thereby improving data processing speed.

[0068] Pipeline processing: Fully utilize the serial operation of the PS end of the FPGA SOC module 200 to process images, complete specific processing tasks, and pass the processing results to the next stage. At the same time, it works perfectly with the PL end of the FPGA SOC module 200 to realize continuous data processing, thereby improving the system's processing efficiency and throughput.

[0069] The beneficial effects of this embodiment are as follows: 1. Ultra-high resolution or ultra-high frame rate image acquisition and processing capability: This invention uses an image sensor with a resolution of hundreds of millions of pixels, combined with the powerful processing capability of the FPGA SOC module 200, which can realize the acquisition and processing of ultra-high resolution or ultra-high frame rate images, and meet various application scenarios with high image quality requirements.

[0070] 2. Low latency and high real-time performance: By optimizing the data transmission and processing flow, and adopting high-speed serial port interfaces and parallel processing technology, the data transmission latency and data processing time are reduced, improving the real-time performance of the system and enabling it to respond quickly to external events.

[0071] 3. Low power consumption and high efficiency: By adopting efficient power management technology and heat dissipation design, the power consumption of the system is reduced. At the same time, by optimizing the system architecture and algorithms, the utilization rate of resources is improved, achieving a balance between low power consumption and high efficiency.

[0072] Example 2, refer to Figures 1-5 This is the second embodiment of the present invention, which provides a camera including the image sensor in embodiment 1.

[0073] It should be noted that the above embodiments are only used to illustrate the technical solution of this utility model and are not intended to limit it. Although this utility model has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solution of this utility model without departing from the spirit and scope of the technical solution of this utility model, and all such modifications or substitutions should be covered within the scope of the claims of this utility model.

Claims

1. An image sensor, comprising an image acquisition and processing component, characterized in that, The image acquisition and processing components include: an image sensor acquisition module (100), an FPGA SOC module (200), a DDR output buffer module (300), an image communication module (400), a command transmission communication module (500), a USB module (600), an eMMC module (700), a serial communication module (800), and a power management module (900). The image sensor acquisition module (100) includes a multi-pixel two-dimensional array, which is divided into eight pixel sub-arrays of the same size. In terms of electrical architecture, the image sensor acquisition module (100) is connected to the FPGA SOC module (200) via a MIPI data link to transmit the acquired image signal to the FPGA SOC module (200). The FPGA SOC module (200) employs at least one processor and programmable logic architecture. After receiving data from the image sensor acquisition module (100) via the MIPI data link, it performs corresponding algorithm processing on the data, receives external control commands through the serial communication module (800) or a command transmission communication module (500), stores the image in the eMMC module (700), and transmits the image data to an external terminal through the communication interface of the image communication module (400) or a USB module (600). The DDR output cache module (300) is connected to the PL and PS terminals of the FPGA SOC module (200) through a parallel interface to ensure that data can be read and written quickly; The image communication module (400) is connected to the GTH transceiver of the FPGA SOC module (200) and is used to upload the acquired images to the server or remote terminal without compression after processing. The command transmission communication module (500) is connected to the FPGA SOC module (200) through the PHY chip interface signal and assigns operation commands to the FPGA SOC module (200); The USB module (600) is connected to the FPGA SOC module (200) in two parts. The first part is the low-speed part, which is connected to the PS terminal of the FPGA SOC module (200) through the USB interface signal. The second part is the high-speed part, in which the USB connector is directly connected to the PS terminal of the FPGA SOC module (200) to perform data interaction with the FPGA SOC module (200). The eMMC module (700) is used to provide stored images to the FPGA SOC module (200). The FPGA SOC module (200) is connected to the eMMC module (700) chip via the PS terminal. The eMMC module (700) protocol-specific processing unit is used to parse and manage the commands of the eMMC module (700) chip. The serial communication module (800) provides serial communication functionality for the entire system. It connects to the PS terminal of the FPGA SOC module (200) via a serial communication signal to communicate with the outside world; and... The power management module (900) provides power supply and power consumption management for the entire system, and adjusts the voltage and current of the power supply according to the working status of the system.

2. The image sensor according to claim 1, characterized in that: The FPGA SOC module (200) is connected to a temperature sensor module (1000). The temperature sensor module (1000) communicates with the FPGA SOC module (200) via I2C. The FPGA SOC module (200) collects data from the temperature sensor chip via I2C to obtain the temperature information on the board.

3. A camera, characterized in that, Includes the image sensor as described in claim 1 or 2.

Images