Processing method and FPGA chip for calibration information of camera module
By transferring the calibration information decoding and updating operations of the camera module to the back-end computing device, the problem of difficulty in development and waste of resources in calibration information processing of FPGA chips is solved, and the throughput capability and adaptability of the digital imaging system is improved.
Patent Information
- Application Number
- CN202011499602.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-18
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-12-18
AI Technical Summary
The existing camera module calibration information encoding and decoding schemes have problems of development difficulties and resource waste for FPGA chips, especially when floating-point operations and the compression encoding formats of image sensors of different manufacturers, it affects digital imaging. System throughput capability.
By transferring calibration information decoding and update operations to back-end computing devices (such as general-purpose chips of CPU/GPU architecture), the FPGA chip is used as a communication bridge, avoiding the construction of codec drivers on FPGAs, reducing logical resource usage, and improving image acquisition and Algorithm processing capability.
It realizes resource optimization of FPGA chips, improves the throughput capability of the digital imaging system, adapts to the calibration of various image sensor parameters, and shortens the development cycle.
Smart Images

Figure CN114648440B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image calibration, and more specifically, to a method for processing calibration information of a camera module and an FPGA chip of the camera module. Background Art
[0002] The calibration of the camera module is the key to the normal operation of the machine vision imaging system. The calibration results usually include many parameters for back-end processing such as image distortion correction, temperature drift correction, and spatial coordinate solution. These parameters are usually stored in a fixed format in the memory device of the camera module (for example, EEPROM or Flash) to form a calibration file.
[0003] In order to save storage space and take into account the needs of versatility, this fixed format is generally presented in the form of some data compression encoding that is easy to be recognized and processed by general-purpose chips (for example, CPU, GPU, ASIC). For example, Sony's camera module calibration files generally use MessagePack encoding technology, and the calibration data and sensor configuration information are stored in the EEPROM inside the camera module in a very concise manner.
[0004] Accordingly, when trying to read the calibration information, the read data must first be decompressed and translated according to a specific encoding format. Accordingly, if the calibration information needs to be updated, the calibration information to be written must also be arranged and compressed according to a specific encoding format.
[0005] For camera modules that use CPU / GPU architecture chips (for example, DSP, ARM, SOC) as front-end processors, this encoding and decoding usually only needs to call the corresponding encoding protocol dependency library to achieve. For example, the MessagePack mentioned above is a binary-based efficient object serialization library for cross-language communication. It supports the exchange of structured objects between many languages such as Python, Ruby, Java, C / C++, JavaScript, etc. Camera module manufacturers usually provide corresponding encoding protocol dependency libraries for users to develop and call.
[0006] In order to increase the rate of processing image data, more and more camera modules are trying to use FPGA chips (i.e., Field Programmable Gate Array) as the front-end processor of the camera module. Compared with CPU / GPU architecture chips, FPGA chips have powerful parallel processing capabilities. When used as the front-end processor of the imaging system, it can significantly improve the throughput of the digital imaging system.
[0007] However, the encoding protocol dependency libraries provided by existing camera module manufacturers rarely support the HDL development language commonly used in FPGA chips. If you write the encoding and decoding mechanism for calibration information yourself, it will increase the development difficulty on the one hand, and on the other hand, it will significantly consume the limited logic resources inside the FPGA chip.
[0008] For image sensors from different manufacturers, the compression and encoding formats of their calibration information are often different. For the same machine vision system, each image sensor with a different compression and encoding format needs to occupy FPGA logic resources to write the corresponding codec driver, which is obviously a waste of the limited logic resources inside the FPGA. It should be understood that if the limited logic resources in the FPGA chip are occupied, it will not be able to allocate enough logic resources for real-time image acquisition and algorithm processing, that is, it will seriously affect the throughput of the digital imaging system. At the same time, in the process of encoding and decoding the calibration information, floating-point data operations are often involved. However, existing FPGA chips only support fixed-point operations and have weak processing capabilities for floating-point operations.
[0009] Furthermore, in order to meet the requirements of real-time imaging acquisition and dynamic adjustment of image sensor parameters, FPGA chips often need to exclusively occupy the communication bus port of the camera module in terms of hardware electrical connection, which makes it impossible for the back-end chip of the CPU / GPU architecture, which has advantages in interpreting the calibration information encoding protocol and floating-point operations, to communicate directly with the calibration information memory at the hardware level.
[0010] In summary, an optimization solution for interpreting and updating calibration information is expected when the front-end chip of the camera module is an FPGA chip. Summary of the invention
[0011] One advantage of the present application is that it provides a method for processing calibration information of a camera module and an FPGA chip, wherein in the calibration information processing method, the FPGA chip as the front-end chip of the camera module acts as a communication bridge to transfer the calibration information decoding and update operations with high technical difficulty and high resource consumption to a back-end computing device that is relatively better at decoding and updating the calibration information, and the back-end computing device includes a general-purpose chip.
[0012] One advantage of the present application is that it provides a method for processing calibration information of a camera module and an FPGA chip, wherein, in the calibration information processing method, the FPGA chip as a front-end chip does not need to allocate logic resources to construct a decoding driver for the calibration information, so that the FPGA chip can fully allocate its logic resources for real-time image acquisition and algorithm processing, so as to improve the throughput of the FPGA chip.
[0013] One advantage of the present application is that it provides a method and FPGA chip for processing calibration information of a camera module, wherein, in the calibration information processing method, the FPGA chip as a front-end chip only needs to allocate relatively few logic resources to create a calibration information management unit for managing storage and transmission of calibration information, without allocating logic resources to construct a decoding driver for the calibration information.
[0014] Another advantage of the present application is that it provides a method for processing calibration information of a camera module and an FPGA chip, wherein the calibration information processing method has a relatively short development cycle and is suitable for calibration of parameters of various types of image sensors.
[0015] In order to achieve at least one of the above advantages or other advantages and purposes, according to one aspect of the present application, a method for processing calibration information of a camera module is provided, wherein the camera module includes an image sensor, a memory for storing calibration information, and an FPGA chip as a front-end chip, wherein the FPGA includes a calibration information storage area and an image processing unit, wherein the method for processing calibration information of the camera module includes:
[0016] In response to receiving a calibration information reading request from a back-end computing device, the FPGA chip reads the first-level calibration information from the memory storage of the camera module, wherein the first-level calibration information has an original compressed encoding format;
[0017] The FPGA chip transmits the primary calibration information to the back-end computing device, wherein the back-end computing device includes a universal chip, and the back-end computing device can decompress and convert the primary calibration information according to the original compression encoding format of the primary calibration information through the universal chip to obtain secondary calibration information, and the secondary calibration information is compatible with the file storage format of the calibration information storage area of the FPGA chip, so that the FPGA chip does not need to occupy its additional logic resources to write a codec driver compatible with the original compression encoding format; and
[0018] The FPGA chip receives the secondary calibration information from the back-end computing device and stores the secondary calibration information in the calibration information storage area.
[0019] In the method for processing calibration information for a camera module according to the present application, the first-level calibration information includes M segments;
[0020] The FPGA chip transmits the primary calibration information to the back-end computing device, including:
[0021] The FPGA chip transmits the Nth fragment in the first-level calibration information to the back-end computing device according to the general protocol, and the back-end computing device is used to store the Nth fragment in its calibration file storage area after receiving the Nth fragment in the first-level calibration information, and the back-end computing device is further used to determine whether N is greater than or equal to M, and if so, decompress and convert the obtained complete first-level calibration information, and if not, generate an instruction to obtain the next fragment; and
[0022] In response to receiving the instruction to obtain the next segment, the FPGA chip reads the segment with segment count number N+1 in the first-level calibration information from the memory storage of the camera module; and
[0023] The FPGA chip transmits the N+1th segment in the first-level calibration information to the back-end computing device according to the general protocol, and repeats this cycle until the back-end computing device obtains the complete first-level calibration information.
[0024] In the calibration information processing method for the camera module according to the present application, the FPGA chip receives the secondary calibration information and stores the secondary calibration information in the calibration information storage area, including: the FPGA chip receives a status query instruction from the back-end computing device; and the FPGA chip sends a calibration file write permission instruction to the back-end computing device, wherein the back-end computing device is used to transmit the secondary calibration information to the FPGA chip in response to receiving the calibration file write permission instruction.
[0025] In the method for processing calibration information of a camera module according to the present application, the method further includes: the FPGA chip receives updated first-level calibration information from the back-end computing device, the updated first-level calibration information is obtained by encoding the updated second-level calibration information in the original compression encoding format by the back-end computing device; and the FPGA chip writes the updated first-level calibration information into the memory storage of the camera module according to the camera module interface protocol.
[0026] In the calibration information processing method for the camera module according to the present application, the FPGA chip receives the updated first-level calibration information from the back-end computing device, including: the FPGA chip receives the Kth segment in the updated first-level calibration information from the back-end computing device; the FPGA chip writes the Kth segment in the updated first-level calibration information into the memory storage of the camera module according to the camera module interface protocol; and, the FPGA chip receives the K+1th segment in the updated first-level calibration information from the back-end computing device, wherein the back-end computing device is used to send the K+1th segment to the FPGA chip in response to detecting that the writing of the Kth segment is completed.
[0027] In the method for processing calibration information of a camera module according to the present application, the method further includes: the FPGA chip synchronously transmits the secondary calibration information stored in the calibration information storage area and the image data obtained from the image sensor to the image processing unit of the FPGA chip, wherein the image processing unit is used to correct the image data obtained from the image sensor based on the secondary calibration information; and the FPGA chip transmits the corrected image data to the image processing and display unit of the back-end computing device, wherein the image processing and display unit is used to display the corrected image data.
[0028] In the method for processing calibration information of a camera module according to the present application, the general-purpose chip of the back-end computing device includes: a CPU, a GPU and an ASIC.
[0029] In the method for processing calibration information of a camera module according to the present application, the first-level calibration information includes calibration information whose data format is floating-point data.
[0030] According to another aspect of the present application, there is provided an FPGA chip, comprising:
[0031] An information storage unit, comprising: an image buffer area and a calibration information storage area;
[0032] Image acquisition unit;
[0033] image processing unit, and
[0034] A calibration information management module, wherein the calibration information management unit is used to execute the calibration information processing method for the camera module as described above.
[0035] In the FPGA chip according to the present application, the image acquisition unit is used to receive image data from an image sensor and store the image data in the image buffer area.
[0036] Further objects and advantages of the present invention will be fully apparent from an understanding of the following description and the accompanying drawings.
[0037] These and other objects, features and advantages of the present invention will be more fully understood from the following detailed description, accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 The figure illustrates a flow chart of a method for processing calibration information of a camera module according to an embodiment of the present application.
[0039] Figure 2 The diagram illustrates a flow chart of the FPGA chip transmitting the first-level calibration information to the back-end computing device in the method for processing calibration information for the camera module according to an embodiment of the present application.
[0040] Figure 3 The diagram illustrates a flow chart of the FPGA chip receiving updated primary calibration information from the back-end computing device in the method for processing calibration information for a camera module according to an embodiment of the present application.
[0041] Figure 4 The figure illustrates a flow chart of decoding the calibration information stored in the memory storage of the camera module in the calibration information processing method for the camera module according to an embodiment of the present application.
[0042] Figure 5A The diagram illustrates one of the flow charts for updating calibration information in the method for processing calibration information for a camera module according to an embodiment of the present application.
[0043] Figure 5B The figure illustrates a second flowchart of updating the calibration information in the method for processing calibration information for a camera module according to an embodiment of the present application.
[0044] Figure 6 The diagram illustrates a block diagram of an imaging system for a method for processing calibration information of a camera module according to an embodiment of the present application. DETAILED DESCRIPTION
[0045] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The embodiments described below are only examples, and those skilled in the art may think of other obvious variations. The basic principles of the present invention defined in the following description may be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not deviate from the spirit and scope of the present invention.
[0046] Exemplary Methods
[0047] Accordingly, in an embodiment of the present application, a machine vision system (or imaging system) using the method for processing the calibration information includes: a camera module, an FPGA chip, and a back-end computing device. The camera module includes, but is not limited to, an RGB camera module, a TOF camera module, an IR camera module, a multi-camera camera module, etc., which includes an image sensor and a memory storage device (e.g., EEPROM, Flash, etc.) for storing calibration information. In addition, the calibration information is encoded with a certain data compression technology before being stored in the memory storage device to have a specific compression coding format. For ease of understanding and explanation, in an embodiment of the present application, the calibration information stored in the memory storage device of the camera module is defined as primary calibration information, which has an original compression coding format.
[0048] The FPGA is the front-end chip of the camera module. The FPGA chip has a strong parallel processing capability. When it is used as a front-end processor of an imaging system, it can significantly improve the throughput of the digital imaging system. In particular, in order to maximize the performance of the FPGA chip, the FPGA chip, as a front-end chip, exclusively occupies the communication bus port of the camera module.
[0049] The backend computing device uses a general-purpose chip of a GPU / CPU architecture. It should be understood that a general-purpose chip of a CPU / GPU architecture (eg, DSP, ARM, SOC) is good at encoding and decoding calibration information.
[0050] In an embodiment of the present application, a method for processing calibration information of a camera module includes three stages.
[0051] The first stage: using the FPGA chip as a relay bridge to transmit the primary calibration information in the camera module to the back-end computing device, wherein the back-end device can complete the interpretation processing of complex compression coding with a general-purpose chip to generate secondary calibration information;
[0052] The second stage: the back-end computing device writes the secondary calibration information into the calibration information storage area of the FPGA chip, so that it can participate in the image processing process, and optimizes the secondary calibration information in a closed loop according to the imaging effect until it reaches an ideal state;
[0053] The third stage: according to the user's choice, if the calibration information in the camera module needs to be updated, the back-end computing device will re-compress the modified secondary calibration information into primary calibration information and relay it back to the memory storage of the camera module through the FPGA chip to complete the calibration information update process.
[0054] Figure 1 FIG. 1 is a flowchart of a method for processing calibration information of a camera module according to an embodiment of the present application. Figure 1 As shown, according to the embodiment of the present application, the method for processing calibration information of the camera module includes: S110, in response to receiving a calibration information reading request from a back-end computing device, the FPGA chip reads the first-level calibration information from the memory storage of the camera module, and the first-level calibration information has an original compression coding format; S120, the FPGA chip transmits the first-level calibration information to the back-end computing device, wherein the back-end computing device includes a general-purpose chip, and the back-end computing device can decompress and convert the first-level calibration information according to the original compression coding format of the first-level calibration information through the general-purpose chip to obtain second-level calibration information, and the second-level calibration information is compatible with the file storage format of the calibration information storage area of the FPGA chip, so that the FPGA chip does not need to occupy its additional logic resources to write a codec driver compatible with the original compression coding format; and, S130, the FPGA chip receives the second-level calibration information from the back-end computing device and stores the second-level calibration information in the calibration information storage area.
[0055] In step S110, in response to receiving a calibration information reading request from a backend computing device, the FPGA chip reads the first-level calibration information from the memory storage of the camera module, and the first-level calibration information has an original compressed encoding format.
[0056] That is, in step S110, first, the backend computing device sends a calibration information reading request to the FPGA chip. Then, in response to receiving the calibration information reading request, the FPGA chip reads the first-level calibration information from the memory storage of the camera module, and the first-level calibration information has an original compressed encoding format.
[0057] In step S120, the FPGA chip transmits the first-level calibration information to the back-end computing device, wherein the back-end computing device includes a general-purpose chip, and the back-end computing device can decompress and convert the first-level calibration information according to the original compression encoding format of the first-level calibration information through the general-purpose chip to obtain second-level calibration information, and the second-level calibration information is compatible with the file storage format of the calibration information storage area of the FPGA chip, so that the FPGA chip does not need to occupy its additional logic resources to write a codec driver compatible with the original compression encoding format.
[0058] Here, taking the case where the first-level calibration information has M fragments as an example, the process of the FPGA chip reading the first-level calibration information from the memory storage of the camera module is explained.
[0059] First, the FPGA chip reads the Nth segment of the first calibration information from the memory storage of the camera module according to the camera module interface protocol and stores it in its calibration information storage area, where N is greater than or equal to 1 and less than or equal to M.
[0060] Next, the FPGA chip transmits the Nth fragment in the first-level calibration information to the back-end computing device according to the general protocol. After receiving the Nth fragment, the back-end computing stores it in its calibration file storage area, and further determines whether N is greater than or equal to M. If so, the obtained complete first-level calibration information is decompressed and converted. If not, an instruction to obtain the next fragment is generated.
[0061] That is, if N is less than M, the backend computing device will send an instruction to obtain the next segment to the FPGA chip. Accordingly, in response to receiving the instruction to obtain the next segment, the FPGA chip reads the segment with the segment count number N+1 in the first-level calibration information from the memory storage of the camera module;
[0062] Then, the FPGA chip transmits the N+1th segment in the first-level calibration information to the back-end computing device again according to the general protocol, and this cycle is repeated until the back-end computing device obtains the complete first-level calibration information.
[0063] Figure 2 The figure shows a flow chart of the FPGA chip transmitting the first-level calibration information to the back-end computing device in the method for processing calibration information for the camera module according to an embodiment of the present application. Figure 2 As shown, in an embodiment of the present application, the FPGA chip transmits the first-level calibration information to the back-end computing device, including: S210, the FPGA chip transmits the Nth fragment in the first-level calibration information to the back-end computing device according to the general protocol, and the back-end computing device is used to store the Nth fragment in its calibration file storage area after receiving the Nth fragment in the first-level calibration information, and the back-end computing device is further used to determine whether N is greater than or equal to M. If so, the acquired complete first-level calibration information is decompressed and converted, and if not, an instruction to obtain the next fragment is generated; S220, in response to receiving the instruction to obtain the next fragment, the FPGA chip reads the fragment with the fragment count number N+1 in the first-level calibration information from the memory storage of the camera module; and, S230, the FPGA chip transmits the N+1th fragment in the first-level calibration information to the back-end computing device according to the general protocol, and the cycle is repeated until the back-end computing device obtains the complete first-level calibration information.
[0064] In step S130, the FPGA chip receives the secondary calibration information from the back-end computing device and stores the secondary calibration information in the calibration information storage area.
[0065] Specifically, in step S130, first, the back-end computing device sends a status query instruction to the FPGA chip according to the general protocol. If the response status fed back by the FPGA chip is to allow the calibration file write instruction to be received, the back-end computing device sends the secondary calibration information to the FPGA chip according to the general protocol in data segments. The FPGA chip completes the storage of the secondary calibration information data segments in the interval of each general protocol communication until the file is completely transmitted and stored.
[0066] Combining steps S110 to S130, it can be found that in the first stage, the FPGA chip is used as a relay bridge to transmit the primary calibration information in the camera module to the back-end computing device, wherein the back-end device can complete the complex compression coding interpretation processing with a general chip to generate the secondary calibration information
[0067] Therefore, in the calibration information processing method for the camera module according to the present application, the FPGA chip acts as a communication bridge to transfer the calibration information decoding and updating operations with high technical difficulty and high resource consumption to the back-end computing device that is relatively better at decoding and updating the calibration information, so that the FPGA chip as the front-end chip does not need to allocate logic resources to construct a decoding driver for the calibration information, so that the FPGA chip can fully allocate its logic resources for real-time image acquisition and algorithm processing to improve the throughput of the FPGA chip. More specifically, in the calibration information processing method, the FPGA chip as the front-end chip only needs to allocate relatively few logic resources to create a calibration information management unit for managing storage and transmission of calibration information, without allocating logic resources to construct a decoding driver for calibration information.
[0068] Furthermore, after the FPGA chip receives the secondary calibration information, the FPGA chip can transmit the stored secondary calibration information to the image processing unit synchronously with the image data to start the FPGA image processing process. Furthermore, the FPGA chip completes the correction of the image data based on the secondary calibration data and outputs the result to the image processing display unit of the back-end computing device. That is, in the second stage, the back-end computing device writes the secondary calibration information to the calibration information storage area of the FPGA chip, so that it can participate in the image processing process, and optimizes the secondary calibration information in a closed loop according to the imaging effect until it reaches an ideal state.
[0069] Accordingly, in an embodiment of the present application, the method for processing calibration information for a camera module further includes the steps of: S140, the FPGA chip synchronously transmits the secondary calibration information stored in the calibration information storage area and the image data obtained from the image sensor to the image processing unit of the FPGA chip, wherein the image processing unit is used to correct the image data obtained from the image sensor based on the secondary calibration information; and, S150, the FPGA chip transmits the corrected image data to the image processing and display unit of the back-end computing device, wherein the image processing and display unit is used to display the corrected image data.
[0070] Furthermore, if the imaging effect processed by the back-end processing device or manually judged does not meet the preset requirements, if the calibration parameters need to be adjusted, the data parameters of the corresponding position in the secondary calibration information can be modified through the calibration information configuration interface of the back-end computing device; then, the back-end computing device re-compresses the modified secondary calibration information into primary calibration information and relays it back to the memory storage of the camera module through the FPGA chip to complete the calibration information update process.
[0071] Specifically, in an embodiment of the present application, the process in which the back-end computing device re-compresses the modified secondary calibration information into primary calibration information and relays it back to the memory storage of the camera module by the FPGA chip includes: first, the back-end computing device transmits the Kth fragment of the updated primary calibration information to the FPGA chip in accordance with a general protocol, and when the response signal is correct, the back-end computing device keeps monitoring the write completion flag register of the FPGA chip.
[0072] Then, after receiving the Kth segment in the updated first-level calibration information, the FPGA chip writes the Kth segment into the memory of the camera module according to the camera module interface protocol, and then updates the write completion flag register.
[0073] Next, after the back-end device detects that the Kth segment in the updated first-level calibration information has been written, if the writing of the updated first-level calibration information is not completed completely, the information segment number of the updated first-level calibration information is increased by 1, and the next segment is transmitted to the FPGA chip. This cycle is repeated until the updated first-level calibration information is completely written into the memory of the camera module.
[0074] Figure 3 The figure shows a flow chart of the FPGA chip receiving updated primary calibration information from the backend computing device in the method for processing calibration information for the camera module according to an embodiment of the present application. Figure 3As shown, in an embodiment of the present application, the FPGA chip receives updated first-level calibration information from the back-end computing device, including: S310, the FPGA chip receives the Kth segment in the updated first-level calibration information from the back-end computing device; S320, the FPGA chip writes the Kth segment in the updated first-level calibration information into the memory storage of the camera module according to the camera module interface protocol; and, S330, the FPGA chip receives the K+1th segment in the updated first-level calibration information from the back-end computing device, wherein the back-end computing device is used to send the K+1th segment to the FGPA chip in response to detecting that the writing of the Kth segment is completed.
[0075] In summary, a method for processing calibration information of a camera module based on an embodiment of the present application is explained, wherein in the method for processing calibration information, the FPGA chip serving as the front-end chip of the camera module serves as a communication bridge to transfer the calibration information decoding and update operations with high technical difficulty and high resource consumption to a back-end computing device that is relatively better at decoding and updating the calibration information. Thus, the FPGA chip does not need to allocate logic resources to construct a decoding driver for the calibration information, so that the FPGA chip can fully allocate its logic resources for real-time image acquisition and algorithm processing, thereby improving the throughput of the FPGA chip.
[0076] In order to more clearly understand the method for processing calibration information for a camera module according to an embodiment of the present application, the process of the method for processing calibration information for a camera module is described again in chronological order below.
[0077] Step 1: The backend computing device sends a calibration information query instruction to the FPGA chip according to the general protocol to obtain the space occupied value M of the calibration information in the calibration information memory of the camera module, where M is the maximum count number of the calibration information fragment.
[0078] Step 2: The back-end computing device sends a status query instruction to the FPGA chip according to the general protocol. If the response status fed back by the FPGA chip allows receiving the calibration management instruction, it proceeds to step 3. Otherwise, it times out and resends or performs exception processing.
[0079] Step 3: The back-end computing device sends an instruction to start the image sensor calibration information reading process (calibration information fragment count number N=0) to the FPGA chip according to the general protocol. When the response signal is correct, the system process enters step 4, otherwise it times out and resends or performs exception processing.
[0080] Step 4: After receiving the instruction to read the image sensor calibration information, the FPGA chip reads the first-level calibration information fragment N from the calibration information memory according to the camera module interface protocol and stores it in the FPGA register, and then proceeds to step 5. If a camera module interface communication error exception occurs, perform exception handling.
[0081] Step 5: The FPGA chip uploads the first-level calibration information fragment N to the back-end computing device according to the general protocol. If the response signal is correct, the system flow enters step 6, otherwise, exception processing is performed.
[0082] Step 6: The back-end computing device parses and obtains the fragment N of the primary calibration information according to the general protocol, and stores it in real time to the corresponding position in the calibration file 1, and then proceeds to step 7.
[0083] Step 7: Determine whether the calibration information has been completely acquired (N>=M). If completed, reset N=0 and go to step 9; otherwise, go to step 8.
[0084] Step 8: The backend computing device adds 1 to the calibration information fragment count number N, transmits information pointing to the next fragment, and sends a continuing image sensor calibration information reading process instruction containing the first-level calibration information fragment count number N to the FPGA chip according to the general protocol, and re-enters step 4.
[0085] Step 9: The back-end computing device decompresses and converts the stored recorded primary calibration information according to its original compression encoding format, generates secondary calibration information that matches the data storage format of the image storage unit in the FPGA chip, and proceeds to step 10.
[0086] Step 10: The back-end computing device sends a status query instruction to the FPGA chip according to the general protocol. If the response status fed back by the FPGA chip allows receiving the calibration file write instruction, proceed to step 11. Otherwise, it times out and resends or performs exception processing.
[0087] Here, the process from step 1 to step 10 is as follows Figure 4 shown.
[0088] Step 11: The back-end computing device sends secondary calibration information to the FPGA chip according to the general protocol and data segments. If there is no abnormality, it proceeds to step 12, otherwise it times out and resends or performs exception processing.
[0089] Step 12: The FPGA chip completes the storage of the secondary calibration information data fragments in the interval between each general protocol communication until the file is completely transmitted and stored, and then enters step 13.
[0090] Step 13: The FPGA outputs the stored calibration information along with the image information to the image processing unit synchronously, and then proceeds to step 14.
[0091] Step 14: Start the FPGA image processing process, synchronously transmit the secondary calibration information data and the original image to the image processing module to complete the processing, and enter step 15.
[0092] Step 15: After completing the image processing correction, the FPGA outputs the result to the image processing and display unit of the back-end computing device, and then proceeds to step 16.
[0093] Step 16: When the imaging effect is not satisfactory as determined by the post-processing of the back-end computing device or the manual judgment, if the calibration parameters need to be adjusted, proceed to step 17, otherwise proceed to step 18.
[0094] Step 17: After recalibration, the data parameters in the corresponding position of the secondary calibration information can be changed through the calibration information configuration interface of the back-end computing device, and then go to step 10 to retest the system imaging effect.
[0095] Step 18: When the user confirms that the original calibration information of the camera module needs to be updated, the user triggers the entry into step 19 in the interactive interface of the back-end computing device, otherwise the calibration information interpretation and update process ends.
[0096] Here, the process from step 11 to step 18 is as follows Figure 5A shown.
[0097] Step 19: The back-end computing device encapsulates the secondary calibration information in the original compression encoding format and reversely updates the primary calibration file data, and proceeds to step 20.
[0098] Step 20: The back-end computing device sends a status query instruction to the FPGA chip according to the general protocol to confirm whether the first-level calibration file writing process can be executed. If the FPGA chip responds with a status confirmation that it is allowed, it goes to step 21 (sets sequence number K=0), otherwise it times out and resends or performs exception processing.
[0099] Step 21: The back-end computing device transmits the first-level calibration file data segment K to the FPGA chip according to the general protocol. When the response signal is correct, the back-end computing device keeps monitoring the write completion flag register of the internal Sensor of the FPGA chip, so that the system process enters step 22, otherwise it times out and resends or performs exception processing.
[0100] Step 22: After receiving the first-level calibration file fragment K, the FPGA chip completes the writing process to the calibration information memory according to the camera module interface protocol, and then updates the Sensor write completion flag register and enters step 23. If the camera module interface communication error exception occurs, the exception handling is performed.
[0101] Step 23: After the computer detects that the first-level calibration file segment K has been written by the Sensor, if the first-level calibration file has not been completely updated, the calibration information segment count number K is increased by 1, the transmission information points to the next segment, and the step 21 is re-entered, otherwise the calibration information interpretation and update process is terminated.
[0102] Here, the process of steps 18 to 23 is as follows Figure 5B shown.
[0103] Schematic imaging system
[0104] Figure 6 FIG. 1 is a block diagram of an imaging system for a method for processing calibration information of a camera module according to an embodiment of the present application. Figure 6 As shown, the imaging system according to the embodiment of the present application includes: a camera module 1, an FPGA chip 6 and a back-end computing device 14, wherein the camera module 1 includes a memory storage 2 for storing calibration information. The FPGA chip includes a camera module management interface 3, a calibration information management unit 10, a general protocol interface 13, an image acquisition unit 5, an information storage unit 7, and an image processing unit 11, wherein the information storage unit 7 includes an image buffer area 8 and a calibration information storage area 9. The back-end computing device 14 includes: an image processing and display unit 15 and a storage and recording unit 16, wherein the storage and recording unit 16 includes: image data storage 17, calibration information storage 18, calibration information interpretation unit 19 and a configuration monitoring unit 20, and the configuration monitoring unit 20 includes a calibration information configuration 21.
[0105] Among them, the calibration information management unit 10 in the FPGA chip serves as a central bridge between the calibration information memory storage 2 in the camera module 1, the calibration information storage area 9 in the FPGA chip and the back-end computing device 14, and becomes the core execution module for controlling the calibration information data transmission and conversion process.
[0106] The calibration information configuration function embedded in the configuration monitoring unit in the back-end computing device 14 uses the conventional universal protocol interface 13 and human-computer interaction configuration monitoring technology to add control and monitoring functions to the calibration information interpretation and update process.
[0107] The calibration information interpretation unit in the back-end computing device 14 is responsible for parsing the original compression code of the calibration information, extracting the valid data and converting it into a data format that matches the FPGA calibration information storage area 9, forming a secondary calibration information segment and putting it into the calibration information storage unit 7.
[0108] The calibration information storage function embedded in the storage recording unit 16 in the back-end computing device 14 splices the calibration information fragments into a secondary calibration file.
[0109] The communication management interface of the camera module 1 connected to the FPGA chip, the specific form of which is usually determined by the model of the camera module 1, usually in the form of an I2C / SPI interface, which can realize the reading and writing of the internal calibration information and various configuration registers of the camera module 1.
[0110] The universal protocol interface 13 between the back-end computing device 14 and the FPGA chip is selected based on the basic principle of being able to meet the calibration information transmission bandwidth and functional requirements on the basis of a conventional visual imaging system, including but not limited to Ethernet, USB, I2C, UART, PCIE and other interface protocol forms. In principle, no additional hardware interconnection interface is required due to the need for calibration information interpretation and update.
[0111] In the FPGA chip, the image acquisition unit 5 in the FPGA chip obtains the original image data of the camera module 1 from the image input interface and puts it into the image buffer area 8. The information storage unit 7 containing the image buffer area 8 and the calibration information storage area 9 in the FPGA chip serves as a cache space for image acquisition and processing, so that the system operations on the image data and the calibration information can be output to the image processing unit 11 at the same time after being acquired and processed separately. The image processing unit 11 in the FPGA chip uses the parallel computing advantages of the FPGA to realize the rapid processing of the machine vision image data, so that the computing performance and throughput of the entire system are better than the machine vision imaging system architecture scheme of the pure back-end computing device 14. The image output interface in the FPGA chip transmits the data processed by the FPGA image to the image processing display unit 15 in the back-end computing device 14.
[0112] In the back-end computing device 14, the configuration monitoring unit 20 in the back-end computing device 14 includes a calibration information configuration module and a monitoring function module for other system status information. The storage recording unit 16 in the back-end computing device 14 stores image data and calibration data as needed according to human-computer interaction instructions. The image processing and display unit 15 in the back-end computing device 14 outputs machine vision imaging results.
[0113] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.
[0114] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.
[0115] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.
[0116] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
[0117] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.
Claims
1. A method for processing calibration information of a camera module, wherein the camera module comprises an image sensor, a memory for storing calibration information, and an FPGA chip as a front-end chip, wherein the FPGA comprises a calibration information storage area and an image processing unit, wherein: include: In response to receiving a calibration information reading request from a back-end computing device, the FPGA chip reads the first-level calibration information from the memory storage of the camera module, wherein the first-level calibration information has an original compressed encoding format; The FPGA chip transmits the primary calibration information to the back-end computing device, wherein the back-end computing device includes a universal chip, and the back-end computing device can decompress and convert the primary calibration information according to the original compression encoding format of the primary calibration information through the universal chip to obtain secondary calibration information, and the secondary calibration information is compatible with the file storage format of the calibration information storage area of the FPGA chip, so that the FPGA chip does not need to occupy its additional logic resources to write a codec driver compatible with the original compression encoding format; and The FPGA chip receives the secondary calibration information from the back-end computing device and stores the secondary calibration information in the calibration information storage area.
2. The method for processing calibration information of a camera module according to claim 1, wherein: The first-level calibration information includes M segments; The FPGA chip transmits the primary calibration information to the back-end computing device, including: The FPGA chip transmits the Nth fragment in the first-level calibration information to the back-end computing device according to the general protocol. The back-end computing device is used to store the Nth fragment in its calibration file storage area after receiving the Nth fragment in the first-level calibration information. The back-end computing device is further used to determine whether N is greater than or equal to M. If so, decompress and convert the obtained complete first-level calibration information. If not, generate an instruction to obtain the next fragment. In response to receiving the instruction to obtain the next segment, the FPGA chip reads the segment with segment count number N+1 in the first-level calibration information from the memory storage of the camera module; and The FPGA chip transmits the N+1th segment in the first-level calibration information to the back-end computing device according to the general protocol, and repeats this cycle until the back-end computing device obtains the complete first-level calibration information.
3. The method for processing calibration information of a camera module according to claim 1, wherein: The FPGA chip receives the secondary calibration information and stores the secondary calibration information in the calibration information storage area, including: The FPGA chip receives a status query instruction from the back-end computing device; and The FPGA chip sends a calibration file writing permission instruction to the back-end computing device, wherein the back-end computing device is used to transmit the secondary calibration information to the FPGA chip in response to receiving the calibration file writing permission instruction.
4. The method for processing calibration information for a camera module according to claim 1, further comprising: The FPGA chip receives updated primary calibration information from the back-end computing device, wherein the updated primary calibration information is obtained by encoding updated secondary calibration information in an original compression encoding format by the back-end computing device; and The FPGA chip writes the updated first-level calibration information into the memory storage of the camera module according to the camera module interface protocol.
5. The method for processing calibration information for a camera module according to claim 4, wherein: The FPGA chip receives updated primary calibration information from the back-end computing device, including: The FPGA chip receives the Kth segment of the updated primary calibration information from the back-end computing device; The FPGA chip writes the Kth segment in the updated first-level calibration information into the memory storage of the camera module according to the camera module interface protocol; and The FPGA chip receives the K+1th fragment in the updated first-level calibration information from the back-end computing device, wherein the back-end computing device is used to send the K+1th fragment to the FPGA chip in response to detecting that the writing of the Kth fragment is completed.
6. The method for processing calibration information for a camera module according to claim 1, further comprising: The FPGA chip synchronously transmits the secondary calibration information stored in the calibration information storage area and the image data obtained from the image sensor to the image processing unit of the FPGA chip, wherein the image processing unit is used to correct the image data obtained from the image sensor based on the secondary calibration information; and The FPGA chip transmits the corrected image data to the image processing and display unit of the back-end computing device, wherein the image processing and display unit is used to display the corrected image data.
7. The method for processing calibration information for a camera module according to claim 1, wherein: The general chips of the back-end computing device include: CPU, GPU and ASIC.
8. The method for processing calibration information for a camera module according to claim 1, wherein: The first-level calibration information includes calibration information whose data format is floating-point data.
9. An FPGA chip, characterized in that: include: An information storage unit, comprising: an image buffer area and a calibration information storage area; Image acquisition unit; Image processing unit, A calibration information management module, wherein the calibration information management unit is used to execute the calibration information processing method for a camera module as described in any one of claims 1 to 8.
10. The FPGA chip according to claim 9, wherein: The image acquisition unit is used for receiving image data from an image sensor and storing the image data in the image buffer area.
