Exposure center point synchronizing device and exposure center point synchronizing method

Abstract

The application provides an exposure center point synchronization device and method. An ISP on a YUV camera module can send an Nth frame exposure duration and an N+1th frame exposure duration to an SOC. The SOC can determine an output time of a frame synchronization signal of the N+1th frame based on a preset N+1th frame alignment reference time corresponding to an exposure center point of the RAW camera module in the N+1th frame, the N+1th frame exposure duration of the YUV camera module sent by the ISP, and the Nth frame exposure duration of the YUV camera module. After the image sensor of the YUV camera module starts exposure at the output time, the exposure center point of the YUV camera module in the N+1th frame is aligned with the N+1th frame alignment reference time, and exposure center point synchronization between the YUV camera module and the RAW camera module is achieved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an exposure center point synchronization device and an exposure center point synchronization method. Background Technology

[0002] In intelligent driving systems, intelligent robot vision systems, and the like, RAW camera modules and YUV camera modules are often deployed in combination. In order to achieve high-quality image fusion, object detection, BEV perception and other processing, it is usually required that the exposure center points (i.e., the center moment of the exposure time period) of these two types of camera modules be synchronized. That is, the exposure center points of the two camera modules are aligned to the same time reference.

[0003] In related technologies, the exposure center point of the camera module is usually adjusted by the SOC on the controller to achieve synchronization of the exposure center points between multiple camera modules.

[0004] However, since the exposure time of the YUV camera module is configured autonomously, the SOC cannot know the exposure time of the YUV camera module, and therefore cannot adjust the exposure center point of the YUV camera module, thus failing to achieve synchronization of the exposure center point between the YUV camera module and the RAW camera module. Summary of the Invention

[0005] This application provides an exposure center point synchronization device and method, which can realize the synchronization of exposure center points between YUV camera modules and RAW camera modules.

[0006] According to a first aspect of the embodiments of this application, an exposure center point synchronization device is provided, including: a RAW camera module, a YUV camera module and a SOC, wherein the YUV camera module has a built-in image sensor and an ISP, and the SOC is communicatively connected to the RAW camera module and the YUV camera module respectively. The ISP is used to determine the exposure duration of the YUV camera module in frame N+1 based on the image data collected by the image sensor in frame N, and to send the exposure duration of the YUV camera module in frame N and frame N+1 to the SOC, where N is a positive integer. The SOC is used to determine the output time of the frame synchronization signal of the N+1 frame based on the preset alignment reference time of the N+1 frame, the exposure time of the N+1 frame, and the exposure time of the N frame, and to send the frame synchronization signal to the image sensor at the output time. The alignment reference time of the N+1 frame corresponds to the exposure center point of the RAW camera module in the N+1 frame. An image sensor is used to respond to a frame synchronization signal and start exposure according to the exposure duration of the N+1th frame, so that the exposure center point of the N+1th frame of the YUV camera module is aligned with the alignment reference time of the N+1th frame.

[0007] In some embodiments of this application, the ISP is specifically used to obtain the actual brightness value corresponding to the image data; The exposure time adjustment is determined based on the difference between the actual brightness value and the preset desired brightness value. The exposure duration of the N+1th frame of the YUV camera module is determined based on the exposure duration adjustment and the exposure duration of the Nth frame.

[0008] Therefore, the ISP can determine the exposure time adjustment based on the difference between the actual brightness value of the image data and the preset expected brightness value, and adjust it based on the exposure time of the Nth frame to generate the exposure time of the N+1th frame. This realizes the closed-loop adjustment of the exposure time of the YUV camera module, so that the exposure time can be adaptively updated with changes in ambient brightness.

[0009] In some embodiments of this application, the SOC is specifically used to obtain the difference between the exposure duration of the (N+1)th frame and the exposure duration of the Nth frame; The sum of half the difference and the alignment reference time of the (N+1)th frame is used as the output time of the frame synchronization signal of the (N+1)th frame.

[0010] Therefore, by obtaining the difference between the exposure duration of the (N+1)th frame and the exposure duration of the Nth frame, and superimposing half of this difference with the alignment reference time of the (N+1)th frame, the output time of the frame synchronization signal of the (N+1)th frame can be accurately calculated. This ensures that after the image sensor starts exposure at this output time, its exposure center point is exactly aligned with the alignment reference time of the (N+1)th frame, thus achieving adaptive adjustment of the output time of the frame synchronization signal under the condition of exposure duration change.

[0011] In some embodiments of this application, the ISP is further configured to send a YUV data packet to the SOC after the exposure of the N+1th frame is completed. The YUV data packet includes the exposure duration of the N+1th frame and the image data of the N+1th frame acquired by the image sensor. The SOC is also used to determine the start time of the image sensor readout pixel array based on the time when the YUV data packet is received and the preset transmission delay; to obtain the row readout duration and the number of effective rows of the image sensor readout pixel array; and to determine the actual exposure center point of the YUV camera module in the N+1 frame based on the start time, row readout duration, number of effective rows and exposure duration of the N+1 frame. The actual exposure center point is verified based on the alignment reference time of frame N+1.

[0012] Therefore, the ISP can send YUV data packets containing exposure duration and image data to the SOC. The SOC can reconstruct the true start time of the image sensor readout pixel array based on the time of receiving the data packet and the preset transmission delay. It can also calculate the actual exposure center point of the YUV camera module in the N+1 frame by combining the row readout duration, the number of effective rows, and the actual exposure center point of the N+1 frame. Then, it compares and verifies the actual exposure center point with the alignment reference time, thereby verifying the alignment effect of the exposure center point of the YUV camera module and improving the reliability of the exposure center point synchronization.

[0013] In some embodiments of this application, the SOC is specifically used to use the difference between the time when the YUV data packet is received and the transmission delay as the start time of the image sensor readout pixel array.

[0014] Therefore, by subtracting the time when the SOC receives the YUV data packet from the preset transmission delay, the actual physical time when the image sensor begins to read out the pixel array can be determined, effectively eliminating the impact of the fixed delay introduced in the data readout, ISP pipeline processing and MIPI transmission on the subsequent exposure center point calculation.

[0015] In some embodiments of this application, the SOC is specifically used to determine the product of half the number of valid rows and the row readout time; Determine the difference between the product and half the exposure time of the (N+1)th frame; The sum of the starting time and the difference is used as the actual exposure center point of the YUV camera module in the N+1th frame.

[0016] Therefore, it can be seen that the SOC can use the start time of the pixel array readout by the image sensor, the row readout duration of the pixel array readout by the image sensor, the effective number of rows of the pixel array, and the exposure duration of the N+1th frame to accurately deduce the actual exposure center point of the YUV camera module in the N+1th frame.

[0017] In some embodiments of this application, the SOC is specifically used to determine that the actual exposure center point verification is passed if the absolute difference between the actual exposure center point and the alignment reference time of the N+1th frame is not greater than a preset threshold. If the absolute difference between the actual exposure center point and the alignment reference time of the (N+1)th frame is greater than the preset threshold, the actual exposure center point verification is determined to be unsuccessful. Based on the difference between the actual exposure center point and the alignment reference time of the (N+1)th frame, the compensation amount is determined. The output time of the frame synchronization signal of the (N+2)th frame is then corrected based on the compensation amount.

[0018] Therefore, by comparing the difference between the exposure center point of the YUV camera module in frame N+1 and the alignment reference time in frame N+1, and setting a preset threshold as the boundary for judging whether the deviation is acceptable, the quantitative evaluation and closed-loop correction of the exposure center point alignment effect of the YUV camera module can be achieved.

[0019] According to a second aspect of the embodiments of this application, an exposure center point synchronization method is provided, comprising: The ISP determines the exposure duration of the YUV camera module in frame N+1 based on the image data collected by the image sensor in frame N, and sends the exposure duration of the YUV camera module in frame N and frame N+1 to the SOC. N is a positive integer. The ISP and the image sensor are set in the YUV camera module, and the SOC communicates with the YUV camera module. The SOC determines the output time of the frame synchronization signal for the N+1 frame based on the preset alignment reference time, exposure duration of the N+1 frame, and exposure duration of the N frame, and sends the frame synchronization signal to the image sensor at the output time. The alignment reference time of the N+1 frame corresponds to the exposure center point of the RAW camera module in the N+1 frame, and the SOC communicates with the RAW camera module. The image sensor responds to the frame synchronization signal and begins exposure according to the exposure duration of the N+1th frame, so that the exposure center point of the N+1th frame of the YUV camera module is aligned with the alignment reference time of the N+1th frame.

[0020] According to a third aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory being used to store at least one program, the at least one program being loaded by the processor and executed by any of the above-described exposure center point synchronization methods.

[0021] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein at least one program is stored in the computer-readable storage medium, and the at least one program is loaded and executed by a processor to implement any of the above-described exposure center point synchronization methods.

[0022] Therefore, in the above-mentioned exposure center point synchronization device, the SOC is communicatively connected to the YUV camera module. The ISP on the YUV camera module can send the exposure duration of the Nth frame and the exposure duration of the N+1th frame to the SOC. Then, the SOC can determine the output time of the frame synchronization signal of the N+1th frame based on the preset alignment reference time of the N+1th frame corresponding to the exposure center point of the RAW camera module in the N+1th frame, the exposure duration of the N+1th frame of the YUV camera module sent by the ISP, and the exposure duration of the Nth frame of the YUV camera module. This ensures that after the image sensor of the YUV camera module starts exposure at this output time, the exposure center point of the YUV camera module in the N+1th frame is aligned with the alignment reference time of the N+1th frame, thus realizing the synchronization of the exposure center point between the YUV camera module and the RAW camera module. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the structure of an exposure center point synchronization device provided in an embodiment of this application; Figure 2 A flowchart illustrating an optical center point synchronization method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0027] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items that have essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or execution order. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various objects, these objects should not be limited by the terms.

[0028] These terms are simply used to distinguish one object from another. For example, without departing from the various examples, a first action can be called a second action, and similarly, a second action can be called a first action. Both the first and second actions can be actions, and in some cases, they can be separate and distinct actions.

[0029] "At least one" refers to one or more actions. For example, at least one action can be one action, two actions, three actions, or any integer number of actions greater than or equal to one. "Multiple" refers to two or more actions. For example, multiple actions can be two actions, three actions, or any integer number of actions greater than or equal to two.

[0030] In intelligent driving systems and intelligent robot vision systems, RAW camera modules and YUV camera modules are often deployed in combination. To achieve high-quality image fusion, object detection, and BEV perception, it is typically required that the exposure center points (i.e., the center moment of the exposure time) of these two types of camera modules be synchronized; that is, the exposure center points of each camera module must be aligned to the same time reference. In related technologies, the exposure center points of the camera modules are usually adjusted by a System-on-a-Chip (SOC) on the controller to achieve synchronization between multiple camera modules. However, since the exposure time of the YUV camera module is configured autonomously, the SOC cannot know the exposure time of the YUV camera module and therefore cannot adjust its exposure center point, thus preventing synchronization between the YUV and RAW camera modules.

[0031] To address the aforementioned technical problems, this application provides an exposure center point synchronization device that can synchronize the exposure center points between a YUV camera module and a RAW camera module.

[0032] Figure 1 This is a schematic diagram of an exposure center point synchronization device provided in an embodiment of this application. The following is in conjunction with... Figure 1This document provides a detailed description of the exposure center point synchronization device. This device can be applied in scenarios requiring multi-camera collaboration, such as intelligent driving systems and robot vision systems. It can be deployed on terminal devices such as vehicles, robots, drones, and aircraft. The device achieves exposure center point synchronization between the RAW and YUV camera modules in mixed deployment scenarios by aligning the exposure center point of the YUV camera module to the corresponding alignment reference time with the exposure center point of the RAW camera module.

[0033] Depend on Figure 1 It is known that the exposure center point synchronization device 100 may include a RAW camera module 101, a YUV camera module 102, and a SOC 103.

[0034] SOC103 can be the main control chip in the intelligent driving controller. The YUV camera module 102 has a built-in image sensor and ISP. The image sensor is used to acquire image data, and the ISP is used to perform image signal processing on the image data acquired by the image sensor, such as black level correction, depigmentation, color correction, gamma correction, RGB to YUV color space conversion, etc., to generate YUV format image data.

[0035] SOC103 communicates with both RAW camera module 101 and YUV camera module 102. For example, SOC103 can communicate with RAW camera module 101 via MIPI CSI-2 interface, and SOC103 can communicate with YUV camera module 102 via control buses such as I²C or SPI.

[0036] In practical applications, when the ambient brightness of the YUV camera module 102 changes (e.g., vehicles entering or exiting tunnels, sudden changes in lighting conditions), the brightness of the image captured by the image sensor will change accordingly. To ensure that the YUV camera module 102 can output images with stable brightness under different lighting conditions, the ISP can dynamically adjust the exposure time according to changes in ambient brightness. The exposure time refers to the cumulative time for the pixel array to receive light and perform photoelectric conversion during a single frame acquisition process. Since changes in ambient brightness are a continuous and gradual process, the image data acquired by the image sensor between adjacent frames has a high temporal correlation. The image brightness information of the current frame can effectively reflect the current ambient lighting state and can be used to predict the exposure requirements of the next frame.

[0037] Based on this, the ISP can determine the exposure duration of the (N+1)th frame of the YUV camera module 102 according to the image data acquired by the image sensor in the Nth frame. Here, N is a positive integer. Specifically, the ISP can continuously run the automatic exposure control logic on a frame-by-frame basis, predicting and determining the exposure duration of the next frame based on the image brightness information of the current frame within each frame period.

[0038] The Nth frame mentioned above is the current frame relative to the (N+1)th frame in terms of time sequence. For example, after the image data of the Nth frame is acquired and output by the image sensor, the ISP can calculate the exposure duration of the (N+1)th frame (i.e., the next frame) based on the image data of that frame.

[0039] The ISP can have an internal buffer register that stores the exposure duration of the current frame. Before the exposure of the Nth frame begins on the image sensor, the ISP can configure the exposure duration of the Nth frame on the image sensor of the YUV camera module 102 and store the exposure duration of the Nth frame in the buffer register.

[0040] After determining the exposure duration of the N+1th frame of the YUV camera module 102, the ISP can read the exposure duration of the Nth frame from the buffer register, update the exposure duration stored in the buffer register to the exposure duration of the N+1th frame, and send the exposure duration of the Nth frame and the exposure duration of the N+1th frame of the YUV camera module 102 to the SOC103.

[0041] Optionally, the ISP can include the exposure duration of the Nth frame and the exposure duration of the (N+1)th frame of the YUV camera module 102 in the frame-following data of the current frame, and send them together with the Nth frame image data acquired by the image sensor to the SOC 103. Alternatively, the ISP can also send the exposure duration of the Nth frame and the exposure duration of the (N+1)th frame to the SOC 103 via a separate control bus (such as I²C or SPI). This application does not limit the specific method of sending the above exposure duration.

[0042] Optionally, the ISP can also send the exposure durations for the next two or three frames to the SOC103. For example, the ISP can send the exposure durations for the (N+1)th and (N+2)th frames to the SOC103 together, allowing the SOC103 to anticipate the exposure duration trends for the next two frames. In this way, the SOC103 can calculate and configure the frame synchronization signal output time several frame cycles in advance when there are significant changes in exposure duration, further improving the predictability and robustness of synchronization control.

[0043] After receiving the exposure duration of the Nth frame and the exposure duration of the N+1th frame from the YUV camera module 102, the SOC103 can determine the output time of the frame synchronization signal of the N+1th frame based on the preset alignment reference time of the N+1th frame, the exposure duration of the N+1th frame, and the exposure duration of the Nth frame.

[0044] The frame synchronization signal is a periodic pulse signal output by the SOC103, used to trigger the image sensor to begin exposure of a frame. In this embodiment, the SOC103 can be directly connected to the frame synchronization signal receiving pin of the image sensor in the YUV camera module 102 via a separate frame synchronization signal line, and this frame synchronization signal line does not pass through the ISP. Therefore, the frame synchronization signal can be transmitted from the SOC103 to the image sensor via the frame synchronization signal line, avoiding the unknown delay introduced by ISP forwarding. Alternatively, the SOC103 can also be connected to the image sensor in the RAW camera module 101 via another frame synchronization signal line. By sending a frame synchronization signal to the RAW camera module 101, the image sensor in the RAW camera module 101 is triggered to begin exposure, achieving synchronization of the exposure center points between the YUV camera module 102 and the RAW camera module 101.

[0045] The alignment reference time for the (N+1)th frame is pre-set based on the exposure center point of the RAW camera module 101 in the (N+1)th frame, and corresponds to that exposure center point. For example, the alignment reference time for the (N+1)th frame can be equal to the exposure center point time of the RAW camera module 101 in the (N+1)th frame. Alternatively, considering differences in signal transmission paths or processing delays, a fixed offset can be added to or subtracted from the exposure center point time of the RAW camera module 101 as the alignment reference time for the (N+1)th frame. Yet another example is that the alignment reference time for the (N+1)th frame can also be independently generated by a high-precision timer within the SOC 103, setting a fixed phase point within each frame period as the alignment reference time, and the RAW camera module 101 itself needs to actively align to this alignment reference time.

[0046] After determining the output time of the frame synchronization signal for the N+1th frame, the SOC103 can send the frame synchronization signal to the image sensor of the YUV camera module 102 at that output time.

[0047] After receiving the frame synchronization signal, the image sensor of the YUV camera module 102 can start exposure according to the exposure duration of the N+1th frame configured on the image sensor, so that the exposure center point of the N+1th frame of the YUV camera module 102 is aligned with the alignment reference time of the N+1th frame, thereby realizing the synchronization of the exposure center point between the YUV camera module 102 and the RAW camera module 101.

[0048] Therefore, in the above-mentioned exposure center point synchronization device, SOC103 is communicatively connected to YUV camera module 102. The ISP on YUV camera module 102 can send the exposure duration of the Nth frame and the exposure duration of the N+1th frame to SOC103. Then, SOC103 can determine the output time of the frame synchronization signal of the N+1th frame based on the preset alignment reference time of the N+1th frame corresponding to the exposure center point of RAW camera module 101 in the N+1th frame, the exposure duration of the N+1th frame of YUV camera module 102 sent by ISP, and the exposure duration of the Nth frame of YUV camera module 102. This ensures that after the image sensor of YUV camera module 102 starts exposure at the output time, the exposure center point of YUV camera module 102 in the N+1th frame is aligned with the alignment reference time of the N+1th frame, thus realizing the exposure center point synchronization between YUV camera module 102 and RAW camera module 101.

[0049] In some embodiments of this application, the ISP can determine the exposure duration of the N+1th frame of the YUV camera module 102 based on the image data collected by the image sensor in the Nth frame.

[0050] First, the ISP can obtain the actual brightness value corresponding to the image data acquired by the image sensor in the Nth frame. For example, the ISP can calculate the average pixel brightness of the entire frame or a preset area (such as the central region, region of interest, etc.) in the image, and use the statistical result as the actual brightness value.

[0051] The ISP can then determine the exposure time adjustment based on the difference between the actual brightness value and the preset desired brightness value. The preset desired brightness value is the target brightness level that the ISP expects to achieve during automatic exposure control, and its specific value can be flexibly set according to the actual application scenario. For example, for a front-view camera for intelligent driving, the desired brightness value can be preset to a pixel grayscale value of 128 to keep the image brightness at a moderate level and avoid overexposure or underexposure.

[0052] When the actual brightness value is less than the preset desired brightness value, it indicates that the current image is too dark and the exposure time needs to be increased. The ISP can determine a positive exposure time adjustment amount. When the actual brightness value is greater than the preset desired brightness value, it indicates that the current image is too bright and the exposure time needs to be decreased. The ISP can determine a negative exposure time adjustment amount. The absolute value of the exposure time adjustment amount is positively correlated with the absolute value of the difference between the actual brightness value and the preset desired brightness value; that is, the larger the deviation, the larger the adjustment amount, and the smaller the deviation, the smaller the adjustment amount.

[0053] Finally, the ISP can determine the exposure duration of the (N+1)th frame of the YUV camera module 102 based on the exposure duration adjustment and the exposure duration of the Nth frame. Specifically, the ISP can add the exposure duration of the Nth frame to the exposure duration adjustment to obtain the exposure duration of the (N+1)th frame. For example, when the exposure duration adjustment is positive, the exposure duration of the (N+1)th frame increases relative to the exposure duration of the Nth frame. When the exposure duration adjustment is negative, the exposure duration of the (N+1)th frame decreases relative to the exposure duration of the Nth frame. When the actual brightness value matches the preset desired brightness value, the exposure duration adjustment is zero, and the exposure duration of the (N+1)th frame is the same as the exposure duration of the Nth frame.

[0054] Optionally, when determining the exposure duration of the (N+1)th frame, the ISP can further consider a preset exposure duration range limit. When the calculated exposure duration of the (N+1)th frame exceeds the minimum or maximum exposure duration supported by the image sensor, the ISP can correct the value to the nearest boundary value to ensure that the exposure duration falls within the valid range.

[0055] Optionally, the ISP can also set a maximum single adjustment step size. When the absolute value of the calculated exposure duration adjustment is greater than the maximum single adjustment step size, the adjustment amount is limited so that the change in exposure duration between frames does not exceed the preset maximum single adjustment step size, thereby avoiding sudden changes in image brightness and achieving a smooth transition in exposure brightness.

[0056] Therefore, the ISP can determine the exposure time adjustment based on the difference between the actual brightness value of the image data and the preset expected brightness value, and adjust it based on the exposure time of the Nth frame to generate the exposure time of the N+1th frame. This achieves closed-loop adjustment of the exposure time of the YUV camera module 102, enabling the exposure time to be adaptively updated according to changes in ambient brightness.

[0057] In some embodiments of this application, the SOC 103 can determine the output time of the frame synchronization signal of the (N+1)th frame of the YUV camera module 102. Specifically, the SOC 103 can first obtain the difference between the exposure duration of the (N+1)th frame and the exposure duration of the Nth frame. This difference can reflect the adjustment direction and adjustment magnitude of the exposure duration of the YUV camera module 102 from the Nth frame to the (N+1)th frame.

[0058] Then, the exposure center point is located in the middle of the exposure time period, that is, it is reached after half of the exposure time from the start of the exposure. Therefore, in order to align the exposure center point of the YUV camera module 102 in the N+1th frame to the alignment reference time of the N+1th frame, the SOC 103 can use half of the above difference and the sum of the alignment reference time of the N+1th frame as the output time of the frame synchronization signal of the N+1th frame.

[0059] Therefore, by obtaining the difference between the exposure duration of the (N+1)th frame and the exposure duration of the Nth frame, and superimposing half of this difference with the alignment reference time of the (N+1)th frame, the output time of the frame synchronization signal of the (N+1)th frame can be accurately calculated. This ensures that after the image sensor starts exposure at this output time, its exposure center point is exactly aligned with the alignment reference time of the (N+1)th frame, thus achieving adaptive adjustment of the output time of the frame synchronization signal under the condition of exposure duration change.

[0060] In some embodiments of this application, the above-described exposure center point synchronization device also incorporates a verification mechanism. Based on this, after the exposure of the N+1th frame is completed, the ISP can send a YUV data packet to the SOC103. The YUV data packet includes the exposure duration of the N+1th frame and the image data of the N+1th frame acquired by the image sensor.

[0061] Specifically, after the exposure of the N+1th frame is completed, the image sensor can send the acquired image data of the N+1th frame to the ISP. After the ISP processes the image data, it packages the processed image data of the N+1th frame and the actual exposure time used for that frame (i.e., the exposure time of the N+1th frame) into a YUV data packet and sends the YUV data packet to the SOC103.

[0062] When the SOC103 receives a YUV data packet, it latches the moment when the YUV data packet is received. Since this moment includes the transmission delay from when the image sensor starts reading out the pixel array to when the SOC103 receives the YUV data packet, the SOC103 needs to determine the start time of the image sensor reading out the pixel array based on the moment when the YUV data packet is received and the preset transmission delay.

[0063] The preset transmission delay refers to the fixed time difference between the moment the image sensor begins reading out the pixel array and the moment the SOC103 receives the YUV data packet. This time difference can be pre-calibrated. The SOC103 can subtract this transmission delay from the moment the YUV data packet is received to obtain the start time of the image sensor reading out the pixel array.

[0064] The SOC103 can then acquire the row readout time and the number of effective rows of the image sensor's readout pixel array. The row readout time refers to the time required for the image sensor to read out one row of pixel data, and the number of effective rows refers to the number of effective pixel rows contained in one frame of the image sensor's output. For example, for an image sensor with a resolution of 1920×1080, the number of effective rows is 1080. Both the row readout time and the number of effective rows are inherent parameters of the image sensor and can be pre-stored in the SOC103.

[0065] Furthermore, SOC103 can determine the actual exposure center point of YUV camera module 102 in the N+1 frame based on the start time, line readout duration, number of valid lines, and exposure duration of the N+1 frame.

[0066] Finally, the SOC103 can verify the actual exposure center point based on the alignment reference time of the (N+1)th frame. Specifically, the SOC103 can compare the determined actual exposure center point of the YUV camera module 102 in the (N+1)th frame with the alignment reference time of the (N+1)th frame to perform verification.

[0067] Therefore, the ISP can send YUV data packets containing exposure duration and image data to the SOC103. The SOC103 can reconstruct the true start time of the image sensor readout pixel array based on the time of receiving the data packet and the preset transmission delay. It can also calculate the actual exposure center point of the YUV camera module 102 in the N+1 frame by combining the row readout duration, the number of effective rows, and the actual exposure center point of the N+1 frame. The actual exposure center point is then compared and verified with the alignment reference time, thereby verifying the alignment effect of the exposure center point of the YUV camera module 102 and improving the reliability of the exposure center point synchronization.

[0068] In some embodiments of this application, since the moment when the YUV data packet is received is not the actual physical moment when the image sensor begins to read out the pixel array, after the image sensor begins to read out the pixel array, the data needs to go through multiple stages such as line-by-line readout, ISP pipeline processing (including black level correction, de-mosaic, color conversion, etc.), MIPI packetization and physical transmission before finally reaching the SOC103. The total time consumed by these stages is the transmission delay, which is pre-calibrated.

[0069] Therefore, the SOC103 can subtract a preset transmission delay from the moment the YUV data packet is received to obtain the start time of the image sensor's readout pixel array. For example, assuming the SOC103 receives the YUV data packet at 115000μs and the preset transmission delay is 500μs, the start time of the image sensor starting to readout the pixel array is 114500μs.

[0070] Therefore, by subtracting the time when the SOC103 receives the YUV data packet from the preset transmission delay, the actual physical time when the image sensor begins to read out the pixel array can be determined, effectively eliminating the impact of the fixed delay introduced in the data readout, ISP pipeline processing and MIPI transmission on the subsequent exposure center point calculation.

[0071] In some embodiments of this application, after determining the start time of the image sensor readout pixel array, the SOC103 also needs to combine the readout duration of the image sensor and the exposure duration of the N+1th frame to further calculate the actual exposure center point of the YUV camera module 102 in the N+1th frame.

[0072] Specifically, the SOC103 can first determine the product of half the number of valid rows and the row readout time. Here, half the number of valid rows of a frame represents the number of rows that need to be read from the first row to the middle row position.

[0073] Then, the SOC103 can determine the difference between the product of half the number of valid lines and the line readout duration and half the exposure duration of the (N+1)th frame.

[0074] Finally, the SOC103 can use the sum of the difference between the start time of the image sensor readout pixel array and the above value as the actual exposure center point of the YUV camera module 102 in the N+1 frame, that is, the precise position of the exposure center point of the image sensor on the time axis when the exposure actually occurs in the N+1 frame.

[0075] Therefore, SOC103 can accurately deduce the actual exposure center point of YUV camera module 102 in the N+1 frame by using the start time of the pixel array readout by the image sensor, the row readout duration of the pixel array readout by the image sensor, the effective number of rows of the pixel array, and the exposure duration of the N+1 frame.

[0076] In some embodiments of this application, the SOC103 can compare the actual exposure center point of the determined YUV camera module 102 in the N+1 frame with the alignment reference time of the N+1 frame to perform verification, so as to confirm whether the actual exposure center point of the YUV camera module 102 in the N+1 frame has been successfully aligned to the alignment reference time of the N+1 frame.

[0077] Specifically, the SOC103 can first determine whether the absolute difference between the actual exposure center point of the YUV camera module 102 in the N+1th frame and the alignment reference time in the N+1th frame is greater than a preset threshold. The preset threshold is a boundary for judging whether the synchronization error is acceptable, and its value can be flexibly set according to the synchronization accuracy requirements of the actual application scenario. For example, in an intelligent driving scenario, if the required synchronization accuracy of the exposure center point is better than ±5μs, the preset threshold can be set to 5μs; if the system has lower requirements for synchronization accuracy, the preset threshold can be set to 10μs or greater. It should be noted that the values ​​of 5μs and 10μs mentioned above are only illustrative examples, and this application does not limit the specific value of the preset threshold; it can be flexibly set according to the actual system requirements.

[0078] If the absolute difference between the actual exposure center point of the YUV camera module 102 in frame N+1 and the alignment reference time in frame N+1 is not greater than a preset threshold, it indicates that the deviation between the actual exposure center point and the alignment reference time is within an acceptable error range. The exposure center point of the YUV camera module 102 in frame N+1 has been basically aligned to the alignment reference time in frame N+1, and the synchronization accuracy meets the system requirements. Therefore, SOC 103 determines that the actual exposure center point verification has passed. At this point, SOC 103 does not need to perform any additional compensation correction, and the system continues to maintain its current synchronization control state, continuing to operate according to the normal frame period and exposure parameters.

[0079] If the absolute difference between the actual exposure center point of the YUV camera module 102 in frame N+1 and the alignment reference time of frame N+1 is greater than a preset threshold, it indicates that the deviation between the actual exposure center point and the alignment reference time of frame N+1 exceeds the acceptable error range. There is a large synchronization error between the exposure center point of the YUV camera module 102 in frame N+1 and the alignment reference time of frame N+1, and effective alignment has not been achieved. Therefore, SOC 103 determines that the actual exposure center point verification fails and can send an alarm signal to the upper-level processor.

[0080] In addition, the SOC103 can also activate a compensation mechanism to eliminate this synchronization error. Specifically, the SOC103 can determine the compensation amount based on the difference between the actual exposure center point and the alignment reference time of the (N+1)th frame, and then correct the output time of the frame synchronization signal of the (N+2)th frame based on the compensation amount.

[0081] The compensation amount is used to adjust the output time of the frame synchronization signal in frame N+2 to compensate for the synchronization deviation that occurred in frame N+1. If the difference between the actual exposure center point and the alignment reference time of frame N+1 is positive (i.e., the actual exposure center point is later than the alignment reference time of frame N+1), the corresponding compensation amount is this difference, and the output time of the frame synchronization signal in frame N+2 is advanced by this compensation amount accordingly. If the difference between the actual exposure center point and the alignment reference time of frame N+1 is negative (i.e., the actual exposure center point is earlier than the alignment reference time of frame N+1), the corresponding compensation amount is the opposite of this difference, and the output time of the frame synchronization signal in frame N+2 is delayed by this compensation amount accordingly.

[0082] As can be seen, by comparing the difference between the exposure center point of the YUV camera module 102 in the N+1th frame and the alignment reference time in the N+1th frame, and setting a preset threshold as the boundary for judging whether the deviation is acceptable, the quantitative evaluation and closed-loop correction of the exposure center point alignment effect of the YUV camera module 102 are realized.

[0083] Therefore, in the above-mentioned exposure center point synchronization device, SOC103 is communicatively connected to YUV camera module 102. The ISP on YUV camera module 102 can send the exposure duration of the Nth frame and the exposure duration of the N+1th frame to SOC103. Then, SOC103 can determine the output time of the frame synchronization signal of the N+1th frame based on the preset alignment reference time of the N+1th frame corresponding to the exposure center point of RAW camera module 101 in the N+1th frame, the exposure duration of the N+1th frame of YUV camera module 102 sent by ISP, and the exposure duration of the Nth frame of YUV camera module 102. This ensures that after the image sensor of YUV camera module 102 starts exposure at the output time, the exposure center point of YUV camera module 102 in the N+1th frame is aligned with the alignment reference time of the N+1th frame, thus realizing the exposure center point synchronization between YUV camera module 102 and RAW camera module 101.

[0084] This application also provides an exposure center point synchronization method. Figure 2 This is a flowchart illustrating an exposure center point synchronization method provided in an embodiment of this application. The following is a summary of the process. Figure 2 The method is described in detail below. This method can be applied to the above-mentioned exposure center point synchronization device. The method can include S201 to S203, as follows.

[0085] S201, the ISP determines the exposure duration of the N+1 frame of the YUV camera module based on the image data collected by the image sensor in the Nth frame, and sends the exposure duration of the Nth frame and the exposure duration of the N+1 frame of the YUV camera module to the SOC. N is a positive integer. The ISP and the image sensor are set in the YUV camera module, and the SOC is connected to the YUV camera module. S202, the SOC determines the output time of the frame synchronization signal of the N+1 frame based on the preset alignment reference time of the N+1 frame, the exposure time of the N+1 frame, and the exposure time of the N frame, and sends the frame synchronization signal to the image sensor at the output time. The alignment reference time of the N+1 frame corresponds to the exposure center point of the RAW camera module in the N+1 frame. The SOC and the RAW camera module are connected in communication. S203 responds to the frame synchronization signal via the image sensor and begins exposure according to the exposure duration of the N+1th frame, so that the exposure center point of the N+1th frame of the YUV camera module is aligned with the alignment reference time of the N+1th frame.

[0086] Embodiments of this application also provide a computer device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement any of the above-described exposure center point synchronization methods.

[0087] Taking computer devices as terminals as an example, Figure 3 A schematic diagram of the structure of a terminal provided in an embodiment of this application is shown below. Figure 3 In some embodiments of this application, terminal 300 may be: a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. Terminal 300 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0088] Typically, terminal 300 includes a processor 301 and a memory 302.

[0089] In some embodiments of this application, processor 301 may include one or more processing cores, such as a quad-core processor, a penta-core processor, etc. Processor 301 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 301 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 301 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 301 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0090] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 302 are used to store at least one program code, which is executed by the processor 301 to implement the process of terminal execution in the method embodiments of this application.

[0091] In some embodiments, the terminal 300 may also optionally include a peripheral device interface 303 and at least one peripheral device. The processor 301, memory 302, and peripheral device interface 303 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 303 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of a display screen 304, a camera assembly 305, an audio circuit 306, and a power supply 307.

[0092] The peripheral device interface 303 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 301 and the memory 302. In some embodiments, the processor 301, memory 302, and peripheral device interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 301, memory 302, and peripheral device interface 303 can be implemented on separate chips or circuit boards, and this application embodiment does not limit this.

[0093] Display screen 304 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 304 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 301 for processing. In this case, display screen 304 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 304, disposed on the front panel of terminal 300; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 300 or in a folded design; in other embodiments, display screen 304 may be a flexible display screen, disposed on a curved or folded surface of terminal 300. Furthermore, display screen 304 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. Display screen 304 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0094] The camera assembly 305 is used to acquire images or videos. In some embodiments, the camera assembly 305 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 305 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash is a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0095] The audio circuit 306 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals which are then input to the processor 301 for processing. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 300. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 301 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 306 may also include a headphone jack.

[0096] Power supply 307 is used to power the various components in terminal 300. Power supply 307 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 307 includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0097] Those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on terminal 300, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0098] Taking computer equipment as a server as an example, Figure 4 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 400 can vary significantly due to different configurations or performance. It may include one or more processors 401 (Central Processing Units, CPUs) and one or more memories 402. The one or more memories 402 store at least one computer program, which is loaded and executed by the one or more processors 401 to implement the aforementioned image noise reduction method. Of course, the server 400 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 400 may also include other components for implementing device functions, which will not be elaborated upon here.

[0099] Embodiments of this application also provide a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to perform the method described above. Optionally, the computer-readable storage medium may be read-only memory (ROM), random access memory (RAM), compact-disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0100] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0101] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An exposure center point synchronization device, characterized in that, include: The system comprises a RAW camera module, a YUV camera module, and a SOC. The YUV camera module has a built-in image sensor and an ISP. The SOC is communicatively connected to both the RAW camera module and the YUV camera module. The ISP is used to determine the exposure duration of the YUV camera module in the N+1th frame based on the image data collected by the image sensor in the Nth frame, and send the exposure duration of the YUV camera module in the Nth frame and the exposure duration of the N+1th frame to the SOC, where N is a positive integer; The SOC is used to determine the output time of the frame synchronization signal of the N+1 frame based on the preset alignment reference time of the N+1 frame, the exposure time of the N+1 frame, and the exposure time of the Nth frame, and to send the frame synchronization signal to the image sensor at the output time. The alignment reference time of the N+1 frame corresponds to the exposure center point of the RAW camera module in the N+1 frame. The image sensor is configured to respond to the frame synchronization signal and begin exposure according to the exposure duration of the (N+1)th frame, so that the exposure center point of the (N+1)th frame of the YUV camera module is aligned with the alignment reference time of the (N+1)th frame.

2. The apparatus according to claim 1, characterized in that, Specifically, the ISP is used to obtain the actual brightness value corresponding to the image data; The exposure time adjustment amount is determined based on the difference between the actual brightness value and the preset desired brightness value; The exposure duration of the N+1th frame of the YUV camera module is determined based on the exposure duration adjustment amount and the exposure duration of the Nth frame.

3. The apparatus according to claim 1, characterized in that, Specifically, the SOC is used to obtain the difference between the exposure duration of the (N+1)th frame and the exposure duration of the Nth frame; The sum of half of the difference and the alignment reference time of the (N+1)th frame is used as the output time of the frame synchronization signal of the (N+1)th frame.

4. The apparatus according to claim 1, characterized in that, The ISP is also used to send a YUV data packet to the SOC after the exposure of the N+1th frame is completed. The YUV data packet includes the exposure duration of the N+1th frame and the image data of the N+1th frame acquired by the image sensor. The SOC is further configured to: determine the start time of the image sensor readout pixel array based on the time when the YUV data packet is received and a preset transmission delay; obtain the row readout duration of the image sensor reading out the pixel array and the effective number of rows of the pixel array; and determine the actual exposure center point of the YUV camera module in the N+1 frame based on the start time, the row readout duration, the effective number of rows, and the exposure duration of the N+1th frame. The actual exposure center point is verified based on the alignment reference time of the (N+1)th frame.

5. The apparatus according to claim 4, characterized in that, Specifically, the SOC is used to take the difference between the time when the YUV data packet is received and the transmission delay as the start time of the image sensor readout pixel array.

6. The apparatus according to claim 4, characterized in that, Specifically, the SOC is used to determine the product of half of the number of valid rows and the row readout time; Determine the difference between the product and half the exposure duration of the (N+1)th frame; The sum of the starting time and the difference is taken as the actual exposure center point of the YUV camera module in the N+1th frame.

7. The apparatus according to claim 4, characterized in that, Specifically, the SOC is used to determine that the actual exposure center point verification is passed if the absolute difference between the actual exposure center point and the alignment reference time of the N+1th frame is not greater than a preset threshold. If the absolute difference between the actual exposure center point and the alignment reference time of the (N+1)th frame is greater than a preset threshold, it is determined that the actual exposure center point verification fails, and a compensation amount is determined based on the difference between the actual exposure center point and the alignment reference time of the (N+1)th frame; the output time of the frame synchronization signal of the (N+2)th frame is corrected based on the compensation amount.

8. A method for synchronizing the exposure center point, characterized in that, include: The ISP determines the exposure duration of the YUV camera module in frame N+1 based on the image data collected by the image sensor in frame N, and sends the exposure duration of the YUV camera module in frame N and frame N+1 to the SOC. N is a positive integer. The ISP and the image sensor are set in the YUV camera module, and the SOC is communicatively connected to the YUV camera module. The SOC determines the output time of the frame synchronization signal for the N+1 frame based on the preset alignment reference time of the N+1 frame, the exposure duration of the N+1 frame, and the exposure duration of the Nth frame, and sends the frame synchronization signal to the image sensor at the output time. The alignment reference time of the N+1 frame corresponds to the exposure center point of the RAW camera module in the N+1 frame. The SOC is communicatively connected to the RAW camera module. In response to the frame synchronization signal, the image sensor begins exposure according to the exposure duration of the (N+1)th frame, so that the exposure center point of the (N+1)th frame of the YUV camera module is aligned with the alignment reference time of the (N+1)th frame.

9. A computer device, characterized in that, The computer device includes a processor and a memory, the memory being used to store at least one program, the at least one program being loaded by the processor and executed as the exposure center point synchronization method as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one program, which is loaded and executed by a processor to implement the exposure center point synchronization method as described in claim 8.

Images