A device and method for real-time acquisition of original image data
Through the combination of image sensor, GMSL module and FPGA, the original data of the image sensor is acquired and saved in real time, which solves the problem that ISP cannot capture data in real time, and realizes efficient and stable data acquisition and automatic exposure, adapting to multiple scenarios.
Patent Information
- Application Number
- CN202210323674.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-29
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-03-29
AI Technical Summary
Existing ISPs cannot capture and save original image data from image sensors in real time when designing, resulting in low efficiency and poor stability. The FPGA provided by image sensor suppliers does not support automatic exposure and adapt to single scenes.
Using a device including an image sensor, a GMSL module and an FPGA, the original image data is transmitted and copied through the GMSL module, and transmitted to the FPGA and the image signal processing unit respectively. The image signal processing unit configures the exposure of the image sensor according to the original image data. Parameters, FPGA collects and saves the original image data in real time.
Real-time acquisition and storage of original data of image sensors, improve data acquisition efficiency and stability, support automatic exposure, and adapt to various scenario needs.
Smart Images

Figure CN114666515B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to image signal processing technology, and in particular to a real-time acquisition device and method for original image data. Background Art
[0002] In the current ISP (Image Signal Processing) field such as security, developers often need to obtain the original image data of the image sensor to perform lens shading correction, CCM correction, etc. Sometimes, the original image data is also needed to locate problems encountered in development. In addition, chip design companies also need to obtain the original image data for algorithm simulation, which all require the original image data of the image sensor.
[0003] At present, due to cost and other reasons, ISP does not have sufficient resources to support the function of real-time capture of raw image data during design, or it can only capture one frame of raw image data and then save it to the computer through serial port and other protocols. This is extremely inefficient and the stability is not good enough.
[0004] In order to support real-time capture of raw image data, one method is to open up enough cache and support high-speed transmission interfaces like USB when designing the ISP, which is unacceptable for cost-sensitive ISPs. Another method is to design the ISP to support capturing a frame of raw image data and then save it through the serial port. This method cannot capture in real time and has very limited application scenarios.
[0005] In addition, general image sensor suppliers have FPGAs to obtain raw image data from image sensors, but they do not support automatic exposure, are only adaptable to a limited number of scenarios, and cannot meet many situations. Summary of the invention
[0006] The object of the present invention is to provide a device and method for real-time acquisition of raw image data, so as to solve the problem that the ISP cannot capture and save the raw data from the image sensor in real time.
[0007] In order to solve the above problems, the present invention provides a real-time acquisition device for raw image data, characterized in that it includes an image sensor, a GMSL module, an FPGA and an image signal processing unit, the image sensor is used to expose and obtain raw image data; the GMSL module is used to transmit and copy the raw image data to obtain two identical copies of the raw image data, and transmit the two copies of the raw image data to the FPGA and the image signal processing unit respectively; the image signal processing unit is used to configure the exposure parameters of the image sensor according to the raw image data; the FPGA is used to collect and save the raw image data in real time.
[0008] Optionally, the image sensor transmits the raw image data to the GMSL module via a MIPI interface.
[0009] Optionally, the GMSL module includes a serializer, a deserializer and a controller connected to each other.
[0010] The serializer is used to receive the raw image data transmitted by the image sensor and perform serialization processing on the raw image data;
[0011] The deserializer receives the serialized raw image data transmitted by the deserializer, deserializes the raw image data, and copies the deserialized raw image data to obtain two identical copies of the raw image data, and then transmits the two identical copies of the raw image data to the FPGA and the image signal processing unit respectively; and
[0012] The controller is used to configure the serializer and the deserializer so that the serializer matches the image sensor, and the deserializer matches the FPGA and the image signal processing unit respectively.
[0013] Furthermore, the serializer transmits the original image data after serialization to the deserializer through a GMSL link.
[0014] Furthermore, the deserializer transmits two identical copies of the original image data to the FPGA and the image signal processing unit respectively through the MIPI interface.
[0015] Furthermore, the controller is connected to the serializer and deserializer via an I2C interface.
[0016] Optionally, the image signal processing unit is used to obtain the register parameters and exposure parameters required for the image sensor to maintain an exposure state according to the original image data transmitted by the deserializer, configure the register parameters to the image sensor, and automatically control the exposure parameters of the image sensor according to the exposure parameters.
[0017] Optionally, the image signal processing unit is connected to the image sensor via an I2C interface.
[0018] Optionally, an external storage device is also included, and the FPGA is connected to the external storage device, so that the FPGA is used to collect the original image data transmitted by the deserializer in real time and save the original image data in the external storage device.
[0019] The present invention also provides a method for real-time acquisition of original image data, using the above-mentioned real-time acquisition device for original image data, comprising the following steps:
[0020] Step S1: the image sensor performs exposure according to initial exposure parameters to obtain raw image data, and transmits the raw image data to the GMSL module;
[0021] Step S2: the GMSL module transmits and copies the original image data to obtain two identical copies of the original image data;
[0022] Step S3: the GMSL module transmits two identical copies of the original image data to the FPGA3 and the image signal processing unit respectively;
[0023] Step S4: the image signal processing unit obtains new exposure parameters of the image sensor according to the raw image data, the image sensor performs exposure according to the new exposure parameters to obtain raw image data, and transmits the raw image data to the GMSL module; at the same time, the FPGA3 collects and saves the raw image data in real time; and
[0024] Step S5: Execute steps S2 to S4 in a loop.
[0025] Compared with the prior art, the present invention has the following beneficial effects:
[0026] The present invention provides a device and method for real-time acquisition of raw image data. The real-time acquisition device comprises an image sensor, a GMSL module, an FPGA and an image signal processing unit. The image sensor is used for exposing to obtain raw image data; the GMSL module is used for transmitting and copying the raw image data to obtain two identical copies of the raw image data, and transmitting the two copies of the raw image data to the FPGA and the image signal processing unit respectively; the image signal processing unit is used for configuring the exposure parameters of the image sensor according to the raw image data; the FPGA is used for real-time acquisition and storage of the raw image data to solve the problem that the ISP cannot capture and store the raw data from the image sensor in real time. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 A structural block diagram of a device for real-time acquisition of raw image data provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0028] The following will further describe in detail a real-time acquisition device and method of raw image data of the present invention. The present invention will be described in more detail with reference to the accompanying drawings, in which preferred embodiments of the present invention are shown. It should be understood that those skilled in the art can modify the present invention described herein and still achieve the advantageous effects of the present invention. Therefore, the following description should be understood as being widely known to those skilled in the art and not as a limitation of the present invention.
[0029] For the sake of clarity, not all features of the actual embodiments are described. In the following description, well-known functions and structures are not described in detail because they would clutter the invention with unnecessary detail. It should be recognized that in the development of any actual embodiment, a large number of implementation details must be made to achieve the developer's specific goals, such as changing from one embodiment to another according to the limitations of the relevant system or the relevant business. In addition, it should be recognized that such development work may be complex and time-consuming, but it is just a routine task for those skilled in the art.
[0030] In order to make the purpose and features of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are further described below in conjunction with the accompanying drawings. It should be noted that the accompanying drawings are all in a very simplified form and use inaccurate ratios, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.
[0031] Figure 1 This is a structural block diagram of a device for real-time acquisition of raw image data provided by this embodiment. Figure 1 As shown, this embodiment provides a real-time acquisition device for raw image data, including an image sensor 1, a GMSL (Gigabit Multimedia Serial Links) module 2, an FPGA 3 and an image signal processing unit 4, wherein the image sensor 1 is used for exposing to obtain raw image data; the GMSL module 2 is used for receiving the raw image data provided by the image sensor 1, transmitting and copying the raw image data to obtain two identical copies of the raw image data, and transmitting the two copies of the raw image data to the FPGA 3 and the image signal processing unit 4 respectively; the image signal processing unit 4 configures the parameters of the image sensor 1 according to the raw image data, so that the raw image data obtained by the image sensor 1 can adapt to the current scene; the FPGA 3 is used for real-time acquisition and storage of the raw image data.
[0032] The image sensor 1 is, for example, a COMS image sensor, which is used to obtain raw image data through exposure. The raw image data includes image data information, transmission rate, len number, etc., wherein the data format of the raw image data is the raw image data format. In this embodiment, in order to simply and quickly transmit the raw image data, the image sensor 1 transmits the raw image data to the GMSL module 2 via the MIPI interface.
[0033] The GMSL module 2 includes a serializer 21, a deserializer 22 and a controller 23 connected to each other. The serializer 21 is used to receive the raw image data transmitted by the image sensor 1, and serialize the raw image data to organize the raw image data into packets, and transmit the packets in serial form to the deserializer 22. The image sensor 1 transmits the raw image data to the serializer 21 through the MIPI interface, and the serializer 21 transmits the packets to the deserializer 22 through the GMSL link.
[0034] The deserializer 22 receives the original image data serialized by the serializer 21, performs deserialization processing on the original image data, restores the original image data to its original appearance, and copies the original image data restored to its original appearance to obtain two identical original image data. The deserializer 22 transmits the two identical original image data to the FPGA 3 and the image signal processing unit 4 respectively. In detail, the deserializer 22 transmits the two identical original image data to the FPGA 3 and the image signal processing unit 4 respectively through the MIPI interface.
[0035] The controller 23 is, for example, an MCU (Microcontroller Unit), and the controller 23 is used to configure the serializer 21 and the deserializer 22, so that the serializer 21 can match the image sensor 1, and the deserializer 21 can match the FPGA 3 and the image signal processing unit 4 respectively, so that the GMSL module 2 can normally access the corresponding image sensor 1. In detail, the controller 23 configures the input mode and len number of the serializer 21, so that the input mode of the serializer 21 matches the output mode of the image sensor 1, and the len number of the serializer 21 matches the len number of the image sensor 1. The output mode of the deserializer 22 matches the input mode of the FPGA 3 and the input mode of the image signal processing unit 4; the len number of the deserializer 22 matches the len number of the FPGA 3 and the len number of the image signal processing unit 4; and ensures that the transmission rate of the deserializer 22 is greater than the transmission rate of the image sensor 1, so that the transmission can proceed normally. In this embodiment, the controller 23 is connected to the serializer 21 and the deserializer 22 via an I2C interface.
[0036] The image signal processing unit 4 is used to obtain the register parameters and exposure parameters required for the image sensor 1 to maintain a normal exposure state according to the raw image data transmitted by the deserializer 22, configure the register parameters to the image sensor 1, and automatically control the exposure parameters of the image sensor 1 according to the exposure parameters, so that the raw image data obtained by the image sensor 1 can adapt to the current scene. In this embodiment, the image signal processing unit 4 is connected to the image sensor 1 via an I2C interface.
[0037] The FPGA 3 is connected to an external storage device, such as a computer, and is used to collect the original image data transmitted by the deserializer 22 in real time and save the original image data to the external storage device.
[0038] This embodiment also provides a method for real-time acquisition of original image data, comprising the following steps:
[0039] Step S1, the image sensor 1 performs exposure according to initial exposure parameters to obtain raw image data, and transmits the raw image data to the GMSL module 2;
[0040] Step S2, the GMSL module 2 transmits and copies the original image data to obtain two identical copies of the original image data;
[0041] Step S3, the GMSL module 2 transmits two identical copies of the original image data to the FPGA 3 and the image signal processing unit 4 respectively;
[0042] Step S4, the image signal processing unit 4 obtains new exposure parameters of the image sensor 1 according to the raw image data, the image sensor is exposed according to the new exposure parameters to obtain raw image data, and the raw image data is transmitted to the GMSL module 2; at the same time, the FPGA 3 collects and saves the raw image data in real time;
[0043] Step S5, looping through steps S2 to S4 until the exposure of the image sensor 1 is completed.
[0044] Step S2 includes the following steps:
[0045] First, the image sensor 1 transmits the raw image data to the serializer 21 through the MIPI interface, and the serializer 21 transmits the packet to the deserializer through the GMSL link.
[0046] Next, the deserializer 22 receives the original image data serialized by the serializer 21, deserializes the original image data again to restore the original image data to its original appearance, and copies the original image data restored to its original appearance to obtain two identical copies of the original image data.
[0047] Next, the deserializer 22 sends two identical copies of the original image data to the FPGA 3 and the image signal processing unit 4 respectively through the MIPI interface.
[0048] Next, the controller 23 configures the serializer 21 and the deserializer 22 so that the serializer 21 can match the image sensor 1, and the deserializer 22 can match the FPGA 3 and the image signal processing unit 4 respectively, so that the GMSL module 2 can be normally connected to the corresponding image sensor 1.
[0049] In summary, the present invention provides a real-time acquisition device and method for raw image data. The present invention uses a GMSL link and an FPGA to acquire the raw image data of an image sensor, wherein the FPGA can capture the raw image data in real time, so that the present invention can capture the raw image data while supporting continuous acquisition of the raw image data of the image sensor to perform image algorithm simulation.
[0050] In addition, it should be noted that, unless otherwise specified or indicated, the terms "first" and "second" in the specification are only used to distinguish the various components, elements, steps, etc. in the specification, and are not used to indicate the logical relationship or sequential relationship between the various components, elements, steps, etc.
[0051] It is to be understood that, although the present invention has been disclosed as a preferred embodiment, the above embodiment is not intended to limit the present invention. For any person skilled in the art, without departing from the scope of the technical solution of the present invention, the technical content disclosed above can be used to make many possible changes and modifications to the technical solution of the present invention, or modified into equivalent embodiments of equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.
Claims
1. A device for acquiring raw image data in real time, characterized in that: Including image sensor, GMSL module, FPGA and image signal processing unit, The image sensor is used for exposure to obtain raw image data; The GMSL module is used to transmit and copy the original image data to obtain two identical copies of the original image data, and transmit the two copies of the original image data to the FPGA and the image signal processing unit respectively. The GMSL module includes a serializer, a deserializer and a controller connected to each other, the serializer is used to receive the raw image data transmitted by the image sensor, serialize the raw image data, and transmit the serialized raw image data to the deserializer through the GMSL link; The deserializer receives the serialized raw image data transmitted by the deserializer, deserializes the raw image data, and copies the deserialized raw image data to obtain two identical copies of the raw image data, and then transmits the two identical copies of the raw image data to the FPGA and the image signal processing unit respectively; and The controller is used to configure the serializer and the deserializer so that the serializer matches the image sensor, and the deserializer matches the FPGA and the image signal processing unit respectively; The image signal processing unit is used to configure the exposure parameters of the image sensor according to the original image data; and the FPGA is used to collect and save the original image data in real time.
2. The real-time acquisition device for raw image data according to claim 1, characterized in that: The image sensor transmits the raw image data to the GMSL module via a MIPI interface.
3. The real-time acquisition device for raw image data according to claim 1, characterized in that: The deserializer transmits two identical copies of the original image data to the FPGA and the image signal processing unit respectively through the MIPI interface.
4. The real-time acquisition device for raw image data according to claim 1, characterized in that: The controller is connected to the serializer and the deserializer via an I2C interface.
5. The real-time acquisition device for raw image data according to claim 1, characterized in that: The image signal processing unit is used to obtain the register parameters and exposure parameters required for the image sensor to maintain an exposure state according to the original image data transmitted by the deserializer, configure the register parameters to the image sensor, and automatically control the exposure parameters of the image sensor according to the exposure parameters.
6. The real-time acquisition device for raw image data according to claim 1, characterized in that: The image signal processing unit is connected to the image sensor via an I2C interface.
7. The real-time acquisition device for raw image data according to claim 1, characterized in that: It also includes an external storage device, and the FPGA is connected to the external storage device, so that the FPGA is used to collect the original image data transmitted by the deserializer in real time and save the original image data in the external storage device.
8. A method for real-time acquisition of raw image data, using the real-time acquisition device for raw image data according to any one of claims 1 to 7, characterized in that: The following steps are involved: Step S1: the image sensor performs exposure according to initial exposure parameters to obtain raw image data, and transmits the raw image data to the GMSL module; Step S2: the GMSL module transmits and copies the original image data to obtain two identical copies of the original image data, wherein the GMSL module includes a serializer, a deserializer, and a controller that are connected to each other, the serializer receives the original image data transmitted by the image sensor, serializes the original image data, and transmits the serialized original image data to the deserializer through the GMSL link; The deserializer receives the original image data after serialization processing transmitted by the deserializer, deserializes the original image data, and copies the original image data after deserialization processing to obtain two identical copies of the original image data; The controller configures the serializer and the deserializer so that the serializer matches the image sensor, and the deserializer matches the FPGA and the image signal processing unit respectively; Step S3: the deserializer of the GMSL module transmits two identical copies of the original image data to the FPGA and the image signal processing unit respectively; Step S4: the image signal processing unit obtains new exposure parameters of the image sensor according to the raw image data, the image sensor performs exposure according to the new exposure parameters to obtain raw image data, and transmits the raw image data to the GMSL module; at the same time, the FPGA collects and saves the raw image data in real time; and Step S5: Execute steps S2 to S4 in a loop.
