Multi-lens shooting system

Abstract

The embodiment of the invention discloses a multi-lens shooting system. The multi-lens shooting system comprises a control assembly, a double-rate synchronous dynamic random access memory DDR, a first processing assembly comprising n image front-end processing units IFPU, a second processing assembly comprising m high dynamic range imaging HDR sensors, and an HDR pipeline arranged on any IFPU, and n and m are integers greater than 0; and the control assembly is used for performing opening and closing control on a connection circuit between the IFPU and the corresponding HDR sensor, controlling the IFPU to output corresponding exposure data to at least one of the DDR pipeline and the HDR pipeline, and controlling the HDR pipeline to perform hardware HDR processing on at least one path of exposure data. According to the invention, the purpose of HDR multi-camera video recording is realized by designing and multiplexing a set of HDR hardware based on time division, and on the premise of ensuring a certain rate, the chip area is effectively reduced, and the construction cost is reduced.

Description

Technical Field

[0001] This disclosure generally relates to communication technologies, and more particularly to a multi-lens shooting system. Background Technology

[0002] With technological advancements and increasing user demands for mobile phone photography and higher resolution, mobile phones and other terminal devices supporting HDR (High Dynamic Range Imaging) dual-camera and multi-camera video recording have emerged. However, the construction cost of such devices remains high.

[0003] Therefore, how to reduce construction costs while ensuring a certain processing rate has become one of the important research directions. Summary of the Invention

[0004] This disclosure provides a multi-lens shooting system to address some of the shortcomings mentioned in the background art.

[0005] In a first aspect, embodiments of this disclosure provide a multi-lens shooting control system, the system comprising: a control component, a double-rate synchronous dynamic random access memory (DDR), a first processing component comprising n image front-end processing units (IFPUs), a second processing component comprising m high dynamic range imaging (HDR) sensors, and an HDR pipeline disposed on any IFPU, wherein n and m are both integers greater than 0; the control component is configured to control the opening and closing of the connection circuit between the IFPU and the corresponding HDR sensor, control the IFPU to output corresponding exposure data to at least one of the DDR and the HDR pipeline, and control the HDR pipeline to perform hardware HDR processing on at least one exposure data stream.

[0006] The embodiments provided in this disclosure have at least the following beneficial technical effects: According to an embodiment of the present disclosure, a transmission link control system can be designed to achieve HDR multi-camera recording by using a set of HDR hardware based on time-division multiplexing. While ensuring a certain data rate, the chip area is effectively reduced and the construction cost is reduced.

[0007] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a schematic diagram of a multi-lens shooting system; Figure 2 This is a schematic diagram of another multi-lens shooting system; Figure 3 This is a schematic diagram of another multi-lens shooting system; Figure 4 This is a schematic diagram of another multi-lens shooting system; Figure 5 This is a schematic diagram showing the connection relationships of some components in a camera system; Figure 6 This is a schematic diagram of another multi-lens shooting system; Figure 7 This is a schematic diagram of another multi-lens shooting system; Figure 8 This is a schematic diagram of another multi-lens shooting system; Figure 9 This is a schematic diagram of another multi-lens shooting system; Figure 10 It is a sequence diagram for collaborative work.

[0009] In the picture: 1000 - Multi-lens shooting system; 100 - Control component; 200 - DDR; 300 - First processing component (including first IFPU 301 and second IFPU 302); 400 - Second processing component (including first HDR sensor 401, second HDR sensor 402 and third HDR sensor 403); 500 - HDR pipeline; 600 - Preview data stream acquisition component; 700 - CSI controller. Detailed Implementation

[0010] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present disclosure and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings, not the entire structure.

[0011] With the widespread adoption of HDR technology, most mobile phones on the market now support HDR video recording. However, although software post-processing algorithms can achieve HDR video recording, due to time constraints, they can generally only reach a resolution of 1080P (1920 pixels wide, 1080 pixels high), and higher resolution recording functions such as 30fps (30 frames per second) remain difficult to achieve. To support higher resolution video recording, in recent years, more and more mobile phones and other terminal devices have gradually adopted HDR hardware pipelines to support HDR video recording.

[0012] Furthermore, due to the growing interest in short videos in recent years, more and more users have developed a greater demand for ultra-wide-angle and telephoto shooting capabilities. Consequently, an increasing number of mobile phones have begun to support HDR dual-camera and HDR multi-camera functions. However, in the hardware design of HDR, in addition to the area required by the HDR module itself, upstream and downstream modules also need to increase their area to support HDR, thus significantly increasing costs.

[0013] Therefore, this disclosure proposes a multi-lens shooting system that achieves HDR dual-camera and multi-camera operation by time-division multiplexing HDR hardware on a single core. Compared to two or more sets of HDR hardware, this system can effectively reduce costs while maintaining a certain speed.

[0014] Figure 1 This is a schematic diagram of a multi-lens shooting system according to this disclosure. Figure 1 As shown, the multi-lens shooting system 1000 includes: a control component 100, a double-rate synchronous dynamic random access memory (DDR) 200, a first processing component 300 including n image front processing units (IFPUs), a second processing component 400 including m HDR sensors, and an HDR pipeline 500 disposed on any of the IFPUs, wherein n and m are both integers greater than 0.

[0015] The control component 100 is used to control the opening and closing of the connection circuit between the IFPU and the corresponding HDR sensor, control the IFPU to output the corresponding exposure data to at least one of the DDR200 and HDR pipeline 500, and control the HDR pipeline to perform hardware HDR processing on at least one exposure data.

[0016] IFPU refers to the part of the ISP (Image Signal Processor) that primarily processes raw domain data.

[0017] According to an embodiment of the present disclosure, a multi-lens shooting system can be designed to achieve HDR multi-camera recording by using a set of HDR hardware based on time-division multiplexing. Under the premise of ensuring a certain speed, the chip area is effectively reduced and the construction cost is reduced.

[0018] The following is based on Figure 2The first processing component 300 shown includes a first IFPU (IFPU 0) 301 and a second IFPU (IFPU 1) 302. The second processing component 400 includes a first HDR sensor 401 (HDR wide-angle sensor), a second HDR sensor 402 (HDR ultra-wide-angle sensor) and a third HDR sensor 403 (HDR telephoto sensor). The HDR pipeline 500 is set in the first IFPU 301. Taking this as an example, the processing process of the multi-lens shooting system 1000 of this disclosure will be explained.

[0019] It should be noted that the multi-lens shooting system 1000 disclosed herein can switch between lenses based on a combination of software and hardware. In other words, the multi-lens switching can be divided into software mode and hardware mode. In software switching mode, the switching between the HDR wide-angle sensor, HDR ultra-wide-angle sensor, and HDR telephoto sensor can be achieved by configuring the first IFPU, the second IFPU, and the corresponding CSI controller through connection and disconnection operations.

[0020] In this embodiment of the disclosure, since there is only one set of HDR hardware, optical zoom is achieved by tangenting the HDR wide-angle sensor, the HDR ultra-wide-angle sensor and the HDR telephoto sensor.

[0021] In this case, the control component 100 is also used to control the first HDR sensor 401, the second HDR sensor 402 and the third HDR sensor 403 to switch in the target order.

[0022] In one possible implementation, after power-on, the control component 100 controls the first IFPU 301 to connect with the first HDR sensor 401; in response to detecting a trigger operation to switch to the second HDR sensor 402, it controls the second IFPU 302 to connect with the second HDR sensor 402, and in response to detecting that a first disconnection condition is met, it disconnects the connection between the first IFPU 301 and the first HDR sensor 401; in response to detecting a trigger operation to switch to the third HDR sensor 403, it controls the first IFPU 301 to connect with the third HDR sensor 403, and in response to detecting that a second disconnection condition is met, it disconnects the connection between the second IFPU 302 and the second HDR sensor 402.

[0023] It should be noted that this disclosure does not impose any restrictions on the specific conditions of the first disconnection condition and the second disconnection condition, which can be set according to the actual situation.

[0024] Optionally, the first disconnection condition can be set to the second HDR sensor 402 reaching the target stable state, and the second disconnection condition can be set to the third HDR sensor 403 reaching the target stable state. The target stable state can be a 3A stable state, namely, an autofocus (AF), auto exposure (AE), and auto white balance (AWB) stable state.

[0025] It should be noted that, in this embodiment of the disclosure, the control component 100 is also used to: control any IFPU connected to any HDR sensor to send exposure data to DDR 200 through the target virtual channel.

[0026] It should also be noted that, because in the exposure count of the first HDR sensor, the medium exposure data M and long exposure data L arrive earlier than the short exposure data S (e.g., hundreds or tens of lines earlier), and HDR combine processing requires 3-exposure alignment, directly using linebuf (line buffer) for alignment in this case would require a huge amount of memory. Therefore, if... Figure 3 As shown, the medium exposure data M and long exposure data L can be sent to DDR 200 first, and then from DDR 200 to HDR pipeline 500, while the short exposure data S can be sent directly to HDR pipeline 500, thus achieving the goal of trading bandwidth for area.

[0027] As one possible implementation, the control component 100 is also configured to: in response to detecting that only the first IFPU 301 is connected to the first HDR sensor 401 in the system, control the first IFPU 301 to output short exposure data of the first HDR sensor 401 to the HDR pipeline 500, control the HDR pipeline 500 to perform hardware HDR processing on the short exposure data, and control the first IFPU 301 to output medium exposure data and long exposure data of the first HDR sensor 401 to the DDR 200, and control the HDR pipeline 500 to perform hardware HDR processing on the medium exposure data and long exposure data in the DDR 200.

[0028] Furthermore, the control component 100 is also configured to: in response to detecting that the first IFPU 301 is connected to the first HDR sensor 401 and the second IFPU 302 is connected to the second HDR sensor 402 in the system, control the first IFPU 301 to output the exposure data of the first HDR sensor 401 to the DDR 200, control the second IFPU 302 to output the exposure data of the second HDR sensor 402 to the DDR 200, and control the HDR pipeline 500 to perform hardware HDR processing on the exposure data output by the first IFPU 301 and the exposure data output by the second IFPU 302 in the DDR 20.

[0029] Furthermore, the control component 100 is also configured to: in response to detecting that the second IFPU 302 in the system is connected to the second HDR sensor 402 and the first IFPU 301 is connected to the third HDR sensor 403, control the first IFPU 301 to output the exposure data of the third HDR sensor 403 to the DDR 200, and control the HDR pipeline 500 to perform hardware HDR processing on the exposure data output by the second IFPU 302 and the exposure data output by the first IFPU 301 in the DDR 200.

[0030] like Figure 4 As shown, the multi-lens shooting system 1000 disclosed herein also includes a preview data stream acquisition component 600.

[0031] The preview data stream acquisition component 600 is used to: in response to detecting that only the first IFPU 301 is connected to the first HDR sensor 401 in the system, acquire the preview data stream based on the exposure data output by the first IFPU 301 after hardware HDR processing; In response to the detection that the first IFPU 301 is connected to the first HDR sensor 401 and the second IFPU 302 is connected to the second HDR sensor 402 in the system, a preview data stream is acquired based on the exposure data output by the first IFPU 301 and the second IFPU 302 after hardware HDR processing. In response to the detection that only the second IFPU 302 is connected to the second HDR sensor 402 in the system, a preview data stream is acquired based on the exposure data output by the second IFPU 302 after hardware HDR processing; In response to the detection that the second IFPU 302 is connected to the second HDR sensor 402 and the first IFPU 301 is connected to the third HDR sensor 403 in the system, a preview data stream is acquired based on the exposure data output by the second IFPU 302 after hardware HDR processing; In response to the detection that only the first IFPU 301 and the third HDR sensor 403 are connected in the system, a preview data stream is acquired based on the exposure data output by the first IFPU 301 after hardware HDR processing.

[0032] It should be noted that, as Figure 5 As shown, in a camera system, the CSI (Camera Serial Interface) controller can connect to one or more IPs from IFPU 0, IFPU 1, IFPU Lite 0, and IFPU Lite 1. After receiving exposure data from the sensor via the PHY, the CSI controller can process a set of exposure data based on either the channel ID or the type ID. IFPU Lite 0 / 1 are two small cores within the IFPU IP, performing simple algorithmic processing on the sensor output image.

[0033] like Figure 6 As shown, the multi-lens shooting system 1000 disclosed herein further includes a camera serial interface (CSI) controller 700. In this case, the control component 100 is also configured to: in response to triggering the operation of configuring the target register, control the CSI controller 700 to disconnect the connection between the IFPU and the current HDR sensor, and establish a connection between the IFPU and a new HDR sensor. Furthermore, the control component 100 is also configured to: in response to triggering the operation of configuring a newly connected sensor, control the IFPU to pre-connect with the CSI controller 700.

[0034] To more clearly explain the processing procedure of the multi-lens shooting system 1000 disclosed herein, as follows: Figure 7-9 As shown below, the operation process of the device will be explained in three stages.

[0035] After power-on, IFPU 0 connects to the HDR wide-angle sensor, such as... Figure 7 As shown, IFPU 0 connects to an HDR wide-angle sensor. The HDR pipeline on IFPU 0 processes two- or three-exposure data and fuses them to output a fused image for previewing the data stream.

[0036] Switching to the HDR ultra-wide-angle sensor stage, such as Figure 8As shown, in response to the user switching to the HDR ultra-wide-angle sensor via zoom mode, IFPU 1 connects to the HDR ultra-wide-angle sensor and outputs exposure data to DDR through the virtual channel (vch) path; IFPU 0 no longer processes the exposure data, but instead uses vch to transmit the exposure data to DDR.

[0037] Among them, vch is used to transmit sensor data to DDR or send it to the ISP pipeline.

[0038] Furthermore, the HDR pipeline on IFPU 0 is time-division multiplexed to perform hardware HDR processing on the multi-channel exposure data output from the HDR ultra-wide-angle sensor and HDR wide-angle sensor in DDR, for use in the preview data stream.

[0039] In this case, after the 3A stabilization of the HDR ultra-wide-angle sensor is identified, the connection of the HDR wide-angle sensor is disconnected, and only the multi-exposure data output by IFPU 1 is used for the preview data stream after hardware HDR processing.

[0040] Switching to the HDR telephoto sensor stage, such as Figure 9 As shown, in response to the user switching to the HDR telephoto sensor via zoom mode, IFPU 0 connects to the HDR telephoto sensor and outputs exposure data to the DDR via the vch path.

[0041] Furthermore, the HDR pipeline on IFPU 0 is time-division multiplexed to perform hardware HDR processing on the exposure data output by the HDR telephoto sensor and HDR ultra-wide-angle sensor in DDR. At this time, the exposure data output by IFPU 1, after HDR processing, is used for the preview data stream.

[0042] In this case, after the 3A stabilization of the HDR telephoto sensor is identified, the connection of the HDR ultra-wide-angle sensor is disconnected, and only the exposure data output by IFPU 0 is processed by hardware HDR and used for the preview data stream.

[0043] It should be noted that the multi-lens shooting system 1000 disclosed herein can switch between lenses based on a combination of software and hardware.

[0044] In hardware switching mode, as one possible implementation, the CSI controller hardware will disconnect the IFPU from the current HDR sensor and establish a connection between the IFPU and the new HDR sensor by configuring the switch start register in software.

[0045] Optionally, the frame initiates the newly connected sensor via software configuration and pre-connects the IFPU and CSI controller, and Frame N configures the switch start register. Further, the current long exposure data, medium exposure data, and short exposure data are disconnected and the sensor is powered down after their respective blanks are filled, while the newly connected long exposure data, medium exposure data, and short exposure data are connected after the blank of Frame N+1. In this case, the timing diagram is shown in Figure 10.

[0046] Therefore, the multi-lens shooting system of this disclosure can achieve the purpose of HDR multi-camera recording by designing and implementing a set of HDR hardware based on time-division multiplexing. Under the premise of ensuring a certain speed, the chip area is effectively reduced and the construction cost is reduced.

[0047] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0048] The specific embodiments described herein do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A multi-lens shooting system, characterized in that, include: The system includes a control component, a double-rate synchronous dynamic random access memory (DDR), a first processing component comprising n image front-end processing units (IFPUs), a second processing component comprising m high dynamic range imaging (HDR) sensors, and an HDR pipeline disposed on any of the IFPUs, wherein n and m are both integers greater than 0. The control component is used to control the opening and closing of the connection circuit between the IFPU and the corresponding HDR sensor, control the IFPU to output the corresponding exposure data to at least one of the DDR and the HDR pipeline, and control the HDR pipeline to perform hardware HDR processing on at least one exposure data.

2. The multi-lens shooting system according to claim 1, characterized in that, The system includes: the first processing component includes a first IFPU and a second IFPU; the second processing component includes a first HDR sensor, a second HDR sensor and a third HDR sensor; the HDR pipeline is a device disposed on the first IFPU; and the control component is further configured to: control the first HDR sensor, the second HDR sensor and the third HDR sensor to switch in a target order.

3. The multi-lens shooting system according to claim 2, characterized in that, The control component is also used for: After power-on, control the first IFPU to connect to the first HDR sensor; In response to detecting an operation that triggers a switch to the second HDR sensor, the second IFPU is controlled to connect to the second HDR sensor, and in response to detecting that a first disconnection condition is met, the connection between the first IFPU and the first HDR sensor is disconnected; In response to detecting an operation that triggers a switch to the third HDR sensor, the first IFPU is controlled to connect to the third HDR sensor, and in response to detecting that a second disconnection condition is met, the connection between the second IFPU and the second HDR sensor is disconnected.

4. The multi-lens shooting system according to claim 3, characterized in that, The first disconnection condition is that the second HDR sensor reaches the target stable state, and the second disconnection condition is that the third HDR sensor reaches the target stable state.

5. The multi-lens shooting system according to claim 2, characterized in that, The control component is also used for: In response to detecting that only the first IFPU is connected to the first HDR sensor in the system, the first IFPU is controlled to output short exposure data of the first HDR sensor to the HDR pipeline, the HDR pipeline is controlled to perform hardware HDR processing on the short exposure data, and the first IFPU is controlled to output medium exposure data and long exposure data of the first HDR sensor to the DDR, the HDR pipeline is controlled to perform hardware HDR processing on the medium exposure data and the long exposure data in the DDR.

6. The multi-lens shooting system according to claim 2, characterized in that, The control component is also used for: In response to detecting that the first IFPU is connected to the first HDR sensor and the second IFPU is connected to the second HDR sensor in the system, the system controls the first IFPU to output the exposure data of the first HDR sensor to the DDR, controls the second IFPU to output the exposure data of the second HDR sensor to the DDR, and controls the HDR pipeline to perform hardware HDR processing on the exposure data output by the first IFPU and the exposure data output by the second IFPU in the DDR. In response to detecting that the second IFPU in the system is connected to the second HDR sensor and the first IFPU is connected to the third HDR sensor, the system controls the first IFPU to output the exposure data of the third HDR sensor to the DDR, and controls the HDR pipeline to perform hardware HDR processing on the exposure data output by the second IFPU and the exposure data output by the first IFPU in the DDR.

7. The multi-lens shooting system according to claim 2, characterized in that, The system also includes: a preview data stream acquisition component, used for: In response to the detection that only the first IFPU is connected to the first HDR sensor in the system, a preview data stream is obtained based on the exposure data output by the first IFPU after hardware HDR processing; In response to detecting that the first IFPU is connected to the first HDR sensor and the second IFPU is connected to the second HDR sensor in the system, the preview data stream is obtained based on the exposure data output by the first IFPU and the exposure data output by the second IFPU after hardware HDR processing; In response to the detection that only the second IFPU is connected to the second HDR sensor in the system, the preview data stream is obtained based on the exposure data output by the second IFPU after hardware HDR processing; In response to detecting that the second IFPU is connected to the second HDR sensor and the first IFPU is connected to the third HDR sensor in the system, the preview data stream is obtained based on the exposure data output by the second IFPU after hardware HDR processing; In response to the detection that only the first IFPU is connected to the third HDR sensor in the system, the preview data stream is obtained based on the exposure data output by the first IFPU after hardware HDR processing.

8. The multi-lens shooting system according to claim 2, characterized in that, The control component is also used for: Control any IFPU connected to any HDR sensor to send exposure data to the DDR via the target virtual channel.

9. The multi-lens shooting system according to claim 2, characterized in that, The system also includes a camera serial interface (CSI) controller, the control component being further configured to: In response to the operation of triggering the configuration target register, the CSI controller is controlled to disconnect the IFPU from the current HDR sensor and establish a connection between the IFPU and the new HDR sensor.

10. The multi-lens shooting system according to claim 9, characterized in that, The control component is also used for: In response to the operation of triggering the configuration to start a new connection of the sensor, the IFPU is controlled to pre-connect with the CSI controller.

Images