Electronic system, camera module and system on a chip

Abstract

An electronic system (200), a camera module (190, 210), a SoC (110, 220) and an electronic device (100), the camera module (210) comprises a first bidirectional interface (2101), the SoC (220) comprises a second bidirectional interface (2201), the second bidirectional interface (2201) is coupled to the first bidirectional interface (2101), and the first bidirectional interface (2101) and the second bidirectional interface (2201) support the same bidirectional data communication interface protocol, the camera module (190, 210) can acquire image data, perform first image processing on the image data to obtain a first image processing result, and the SoC (110, 220) can perform second image processing on the first image processing result to obtain a second image processing result. In this way, by using the bidirectional interface (2101, 2201) supporting the bidirectional data communication interface protocol to connect the camera module (210) and the SoC (220), the flexibility of the interface between the camera module (190, 210) and the SoC (110, 220) can be improved, and the camera module (190, 210) can be facilitated to flexibly process image data.

Description

Technical Field

[0001] This application relates to the field of consumer electronics, and more specifically, to an electronic system, a camera module, and a system on chip (SoC). Background Technology

[0002] Currently, with the development of image processing technology, electronic devices are typically equipped with one or more camera modules (often also called cameras). A camera module includes an optical lens and an image sensor, where the image sensor may include a photosensitive unit (e.g., a photodiode), an analog-to-digital converter (A / D converter), and an image signal processor (ISP). The working principle of a camera module is as follows: light reflected from the subject passes through the optical lens to generate an optical image, which is projected onto the photosensitive unit of the image sensor. The photosensitive unit converts the optical image into an electrical signal, i.e., an analog image signal. The analog image signal is then converted into a digital image signal by the A / D converter, and transmitted to the ISP for further processing. The image processing result from the ISP is then transmitted to a processing device within the electronic device, such as a System-on-a-Chip (SoC), where the SoC further processes the image to enable various image-based applications.

[0003] Currently, data transmission between camera modules and SoCs primarily utilizes the D-physical layer protocol (DPHY) or C-physical layer protocol (CPHY) defined by the Mobile Industry Processor Interface (MIPI) protocol organization. Generally, when designing intellectual property cores (IP), using DPHY or CPHY supports signal transmission from the camera module to the SoC, enabling the SoC to receive the image processing results already preliminarily processed by the camera module. However, existing interface designs lack flexibility, resulting in a relatively rigid approach to image data processing by the camera module. Summary of the Invention

[0004] This application provides an electronic system, a camera module, a SoC, and an electronic device, which can improve the flexibility of the interface between the camera module and the SoC, and facilitate the camera module to process image data flexibly.

[0005] In a first aspect, an electronic system is provided, including a camera module and a system-on-a-chip (SoC); the camera module includes a first bidirectional interface; the SoC includes a second bidirectional interface coupled to the first bidirectional interface, and the first bidirectional interface and the second bidirectional interface support the same bidirectional data communication interface protocol; the camera module is used to acquire image data and perform a first image processing on the image data to obtain a first image processing result; the SoC is used to perform a second image processing on the first image processing result to obtain a second image processing result.

[0006] The electronic system provided in this application differs from existing technologies. By coupling the camera module and the SoC through an interface that supports a bidirectional data communication protocol, the flexibility of the interface implementation can be improved, and the camera module can more flexibly process image data. For example, the camera module can receive and process image data from the SoC, offering greater flexibility.

[0007] In addition, since the first and second bidirectional interfaces support bidirectional data communication interface protocols, replacing the function of the MIPI interface in the existing technology, the camera module and SoC do not need to use other bus interfaces such as MIPI. This can reduce the number of reserved pins in the camera module and SoC and reduce the package area.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the bidirectional data communication interface protocol includes a serializer / deserializer (SerDes) interface protocol. By employing the SerDes interface protocol, high-speed bidirectional data communication can be achieved. Furthermore, the SerDes interface can embed a clock, thus eliminating the need to transmit clock signals, thereby enabling clock synchronization between the camera module and the SoC.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, each of the first bidirectional interface and the second bidirectional interface includes: a sending interface and a receiving interface; or, a transceiver interface multiplexed for sending and receiving.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the system further includes a memory connected to the SoC; the image data includes first image data; the camera module is further configured to send the first image data to the SoC via the first bidirectional interface before performing the first image processing on the first image data; the SoC is further configured to receive the first image data via the second bidirectional interface, store the first image data in the memory, read the first image data from the memory, and send the first image data to the camera module via the second bidirectional interface; the camera module is further configured to receive the first image data via the first bidirectional interface and perform the first image processing on the first image data.

[0011] In some scenarios, camera modules need to process large amounts of image data. However, since camera modules cannot process all image data in real time, more storage space is needed to cache the image data. This allows the image processor within the camera module to process the cached image data only after it has finished processing the currently processing image data. This achieves external caching and reloading of image data from the camera module. However, the storage space within a camera module is limited, and due to factors such as cost, area, packaging, and power consumption, it is impossible to integrate a larger capacity static random-access memory (SRAM) within the camera module. The solution provided in this application, through a first bidirectional interface and a second bidirectional interface, enables the camera module to access the memory via the SoC, thereby solving the above problems and expanding the application scenarios of the electronic system.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the memory includes double data rate (DDR) random-access memory (RAM).

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the image data includes second image data; the camera module is used to perform the first image processing on the second image data without needing to send the second image data through the first bidirectional interface. Therefore, in addition to the external caching and re-reading mode of image data, the camera module also has a direct processing mode, enriching the application solutions.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, the system further includes a memory connected to the SoC; the SoC is further configured to store a first image processing result corresponding to the second image data into the memory, and to read the first image processing result corresponding to the second image data from the memory, and send the first image processing result corresponding to the second image data to the camera module through a second bidirectional interface; the camera module is further configured to receive the first image processing result corresponding to the second image data through the first bidirectional interface, and to perform third image processing on the first image processing result corresponding to the second image data. It should be understood that the first image processing result corresponding to the second image data is the image processing result obtained after performing first image processing on the second image data. The above-mentioned solution provided in this application, through the first bidirectional interface and the second bidirectional interface, enables the camera module to access the memory via the SoC, expanding the application scenarios of the electronic system.

[0015] It should be understood that the third image processing and the first image processing can be the same or different, and this application does not limit them. For example, the third image processing can be at least one of noise reduction, calibration, or defect detection. This application does not specifically limit what kind of operation the third image processing is, and any reasonable operation can be considered as the third image processing.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, a storage controller is configured to store the first image data in the memory and read the first image data from the memory; or, to store the first image processing result corresponding to the second image data in the memory and read the first image processing result corresponding to the second image data from the memory.

[0017] Secondly, a camera module is provided, comprising: a first image processing device for acquiring image data and performing first image processing on the image data to obtain a first image processing result; a first bidirectional interface coupled to a second bidirectional interface of a system-on-a-chip (SoC), wherein the first bidirectional interface and the second bidirectional interface support the same bidirectional data communication interface protocol, and the first bidirectional interface is used to send the first image processing result to the SoC.

[0018] For example, the first image processing device may be an image sensor. The image sensor may include a photosensitive unit (e.g., a photodiode), an A / D converter, and an ISP. Light signals passing through the optical lens in the camera module can be projected onto the photosensitive unit of the image sensor. The photosensitive unit converts the light signal into an electrical signal, i.e., an analog image signal. The analog image signal is then converted into image data (i.e., a digital image signal) by the analog-to-digital converter. The ISP can perform a first image processing on the image data to obtain a first image processing result.

[0019] The camera module provided in this application connects to the SoC via a bidirectional interface that supports a bidirectional data communication interface protocol, enabling bidirectional data communication with the SoC and improving the flexibility of the interface for more flexible image data processing.

[0020] In conjunction with the second aspect, in some implementations of the second aspect, the bidirectional data communication interface protocol includes a serializer / deserializer SerDes interface protocol.

[0021] In conjunction with the second aspect, in some implementations of the second aspect, the first bidirectional interface includes: a sending interface and a receiving interface; or, a transceiver interface multiplexed for sending and receiving.

[0022] In conjunction with the second aspect, in some implementations of the second aspect, the image data includes first image data; the first bidirectional interface is further configured to send the first image data to the SoC, the first image data being further stored by the SoC in a memory connected to the SoC; and to receive the first image data stored in the memory from the SoC; the first image processing device is configured to perform the first image processing on the first image data.

[0023] In conjunction with the second aspect, in some implementations of the second aspect, the memory includes double-data-rate random access memory (DDR RAM).

[0024] In conjunction with the second aspect, in some implementations of the second aspect, the image data includes second image data; the first image processing device is further configured to perform the first image processing on the second image data without needing to send the second image data through the first bidirectional interface.

[0025] In conjunction with the second aspect, in some implementations of the second aspect, the first image processing result corresponding to the second image data is further stored by the SoC in a memory connected to the SoC; and the first image processing device receives the first image processing result corresponding to the second image data stored in the memory from the SoC; the first image processing device is further configured to perform third image processing on the first image processing result corresponding to the second image data.

[0026] Thirdly, a system-on-a-chip (SoC) is provided, comprising: a second bidirectional interface coupled to a first bidirectional interface of a camera module, wherein the first bidirectional interface and the second bidirectional interface support the same bidirectional data communication interface protocol, the second bidirectional interface being used to acquire a first image processing result from the camera module, the first image processing result being obtained by the camera module performing first image processing on image data; and a second image processing device being used to perform second image processing on the first image processing result to obtain a second image processing result.

[0027] For example, the second image processing device may be an ISP, a central processing unit (CPU) or other image processing unit, which can perform at least one subsequent processing such as 3A processing, image domain conversion or image post-processing on the first image data result.

[0028] In conjunction with the third aspect, in some implementations of the third aspect, the bidirectional data communication interface protocol includes a serializer / deserializer SerDes interface protocol.

[0029] In conjunction with the third aspect, in some implementations of the third aspect, the second bidirectional interface includes: a sending interface and a receiving interface; or a transceiver interface multiplexed for sending and receiving.

[0030] In conjunction with the third aspect, in some implementations of the third aspect, the SoC is connected to a memory, the image data includes first image data, and the second bidirectional interface is further used to acquire the first image data from the camera module; the SoC further includes: a storage controller, used to store the first image data in the memory and read the first image data from the memory; the second bidirectional interface is further used to send the first image data to the camera module, and the first image data is processed by the camera module using the first image processing.

[0031] In conjunction with the third aspect, in some implementations of the third aspect, the SoC is connected to a memory, and the SoC further includes: a memory controller, configured to store the first image processing result to the memory and read the first image processing result from the memory; the second bidirectional interface is further configured to send the first image processing result to the camera module, and the first image processing result is subjected to third image processing by the camera module.

[0032] In conjunction with the third aspect, in some implementations of the third aspect, the memory includes double-data-rate random access memory (DDR RAM).

[0033] Fourthly, an electronic device is provided, comprising the electronic system of the first aspect or any possible implementation thereof. Optionally, the electronic device is a portable electronic device, such as a mobile phone.

[0034] Fifthly, an image processing method is provided, comprising: a camera module acquiring first image data and sending the first image data to a SoC; the SoC receiving the first image data and storing the first image data in a memory; the SoC reading the first image data from the memory and sending the first image data to the camera module; the camera module receiving the first image data and performing first image processing on the first image data to obtain a first image processing result.

[0035] Optionally, the camera module sends the first image processing result to the SoC; the SoC performs second image processing on the first image processing result to obtain a second image processing result.

[0036] Optionally, the camera module includes a first bidirectional interface; the SoC includes a second bidirectional interface, the second bidirectional interface being coupled to the first bidirectional interface, and the first bidirectional interface and the second bidirectional interface supporting the same bidirectional data communication interface protocol.

[0037] Optionally, the bidirectional data communication interface protocol includes the SerDes interface protocol.

[0038] Optionally, each of the first bidirectional interface and the second bidirectional interface includes: a transmitting interface and a receiving interface; or, a transceiver interface multiplexed for transmitting and receiving.

[0039] Optionally, the memory includes DDR RAM.

[0040] In some scenarios, camera modules need to process large amounts of image data. However, since camera modules cannot process all image data in real time, more storage space is needed to cache the image data. This allows the image processor within the camera module to process the cached image data only after it has finished processing the currently processing image data. This achieves external caching and reloading of image data from the camera module. However, the storage space within a camera module is limited, and due to factors such as cost, area, packaging, and power consumption, it is impossible to integrate a larger capacity static random-access memory (SRAM) within the camera module. According to the image processing method provided in this application, the SoC enables the camera module to access memory, thereby solving the aforementioned problems.

[0041] A sixth aspect provides an image processing method, comprising: a camera module acquiring second image data and performing a first image processing on the second image data to obtain a first image processing result; sending the first image processing result to a SoC; the SoC receiving the first image processing result and storing the first image processing result in a memory; the SoC reading the first image processing result from the memory and sending the first image processing result to the camera module; the camera module receiving the first image processing result and performing a third image processing on the first image processing result to obtain a third image processing result.

[0042] Optionally, the method further includes: the camera module sending the third image processing result to the SoC; and the SoC performing a second image processing on the third image processing result to obtain a fourth image processing result.

[0043] Optionally, the camera module includes a first bidirectional interface; the SoC includes a second bidirectional interface, the second bidirectional interface being coupled to the first bidirectional interface, and the first bidirectional interface and the second bidirectional interface supporting the same bidirectional data communication interface protocol.

[0044] Optionally, the bidirectional data communication interface protocol includes the SerDes interface protocol.

[0045] Optionally, each of the first bidirectional interface and the second bidirectional interface includes: a transmitting interface and a receiving interface; or, a transceiver interface multiplexed for transmitting and receiving.

[0046] Optionally, the memory includes DDR RAM.

[0047] In some scenarios, camera modules need to process large amounts of image data. However, since camera modules cannot process all image data in real time, more storage space is needed to cache the image data. This allows the image processor within the camera module to process the cached image data only after it has finished processing the currently processing image data. This achieves external caching and reloading of image data from the camera module. However, the storage space within a camera module is limited, and due to factors such as cost, area, packaging, and power consumption, it is impossible to integrate a larger capacity static random-access memory (SRAM) within the camera module. According to the image processing method provided in this application, the SoC enables the camera module to access memory, thereby solving the aforementioned problems.

[0048] In a seventh aspect, an electronic system is provided, including a camera module and a system-on-a-chip (SoC) for performing the methods described in the fifth or sixth aspect above.

[0049] Eighthly, a camera module is provided, corresponding to the camera module in the electronic system of the seventh aspect.

[0050] In the ninth aspect, a SoC is provided, corresponding to the SoC in the electronic system of the seventh aspect. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the structure of an electronic device provided in this application.

[0052] Figure 2 This is a schematic diagram of an electronic system provided in this application.

[0053] Figure 3 This is a pin definition diagram of the SerDes interface.

[0054] Figure 4 This is a schematic diagram of a camera module provided in this application.

[0055] Figure 5 This is a schematic diagram of the structure of a SoC provided in this application.

[0056] Figure 6 This is a schematic flowchart of an image processing method provided in this application.

[0057] Figure 7 This is a schematic flowchart of another image processing method provided in this application. Detailed Implementation

[0058] The technical solutions of this application will now be described with reference to the accompanying drawings. The technical solutions provided in this application can be applied to electronic devices equipped with camera modules including one or more cameras, such as mobile phones, digital cameras, camcorders, tablet computers, media players, game consoles, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc.

[0059] This application provides a solution that improves interface flexibility by coupling a camera module and a System-on-a-Chip (SoC) in an electronic device through a bidirectional interface that supports a bidirectional data communication interface protocol. This allows the camera module to process image data more flexibly. For example, the camera module can receive and process image data from the SoC, rather than simply transmitting image data unidirectionally to the SoC, thus offering greater flexibility. The solution provided in this application is described below. It is understood that a bidirectional interface is capable of both sending and receiving data, not just unidirectional data transmission. For example, a bidirectional interface is a SerDes interface. This bidirectional interface differs from other interfaces that can only transmit control information to the camera module. By employing a bidirectional interface, the SoC can also transmit data to the camera module. This data may include image data from the image sensor within the camera module, or processing results obtained after processing the image data, etc., which will be described in detail later.

[0060] To facilitate understanding of this application, let's first use... Figure 1 Taking the electronic device shown as an example, we will briefly explain the possible structure of the electronic device. Figure 1 A schematic diagram of an electronic device is shown. The electronic device 100 may include a SoC 110 and one or more camera modules 190. The SoC 110 is a chip that includes logic operation circuitry and may optionally include analog circuitry, memory circuitry, or various interfaces.

[0061] The SoC 110 may include one or more processing units. For example, the SoC 110 may include one or more of the following units: a central processing unit (CPU), a neural network processing unit (NPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an ISP, a controller, a video codec, a digital signal processor (DSP), and a baseband processor. These different processing units may be independent devices or integrated into one or more processors.

[0062] The camera module 190 can be used to capture external videos or photos, and can be used to capture scenes at different distances. For example, the camera module can be used to capture distant scenes, close-up scenes, and macro scenes. The camera module 210 can also be used for selfies, but this application embodiment does not impose any limitations.

[0063] The camera module 190 may include an optical lens and an image sensor, wherein the image sensor may include a photosensitive unit (e.g., a photodiode), an A / D converter, and an image sensor (ISP). The optical lens is a device capable of receiving light signals and focusing them onto the photosensitive unit of the image sensor. The image sensor is used to acquire image data and can perform initial image processing on the image data. Specifically, the photosensitive unit can convert the light signal passing through the optical lens into an electrical signal, i.e., an analog image signal. The analog image signal is then converted into image data (i.e., a digital image signal) by the analog-to-digital converter, and the ISP can further process this image data. The processing result obtained by the ISP can be transmitted to the SoC 110 for further processing.

[0064] Optionally, the electronic device 100 may further include a display unit 170. The display unit 170 is used to display images, videos, etc. The display unit 170 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini LED, a micro LED, a micro-OLED, or a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or more display units 170. For example, the SoC 110 may further process the processing results output by the ISP in the camera module 190, and the resulting results may be provided to the display unit 170 for displaying the corresponding image.

[0065] Optionally, the electronic device 100 also includes a memory 120. The memory 120 can be used to store computer-executable program code, which includes instructions. The SoC 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the memory 120. The memory 120 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, phonebook, etc.). Furthermore, the memory 120 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0066] In some possible embodiments, memory can be provided in SoC 110. For example, the memory in SoC 110 is a cache memory. This memory can store instructions or data that SoC 110 has just used or is recurring. If SoC 110 needs to use the instruction or data again, it can directly retrieve it from this memory. This avoids repeated accesses, reduces the waiting time of SoC 110, and thus improves system efficiency. Of course, memory can also exist independently of SoC 110, as shown in memory 120 in the figure. This application does not limit this.

[0067] Optionally, the electronic device 100 also includes a transceiver 130. In addition, to further enhance the functionality of the electronic device 100, it may also include one or more of an antenna 140, an input unit 160, an audio circuit 180, and a sensor 101 (e.g., an accelerometer, a gravity sensor, etc.), wherein the audio circuit may also be coupled to a speaker 182, a microphone 184, etc. Optionally, the electronic device 100 may also include a power supply 150 for providing power to various devices or circuits in the terminal device. It should be understood that... Figure 1 For ease of understanding only, the various modules or units in an electronic device and the connection relationships between them are illustrated by way of example, but this should not constitute any limitation on this application. This application does not limit the specific modules or units included in the electronic device and the connection relationships between them.

[0068] It should also be understood that Figure 1 The electronic device 100 shown may include Figure 2 The electronic system 200 shown is... Figure 1 The camera module 190 in the electronic device 100 shown can be Figure 2The camera module 210 in the electronic system 200 shown, Figure 1 The SoC 110 in the illustrated electronic device 100 may be Figure 2 The SoC 220 in the electronic system 200 shown.

[0069] Figure 2 This is a schematic diagram of an electronic system provided in this application. See also... Figure 2 The electronic system 200 includes a camera module 210 and a SoC 220. The camera module 210 includes a first bidirectional interface 2101, and the SoC 220 includes a second bidirectional interface 2201. The second bidirectional interface 2201 is coupled to the first bidirectional interface 2101, and the first bidirectional interface 2101 and the second bidirectional interface 2201 support the same bidirectional data communication interface protocol.

[0070] The camera module 210 is used to acquire image data. For example, the SoC 220 can send a video or photo request to the camera module 210 through the second bidirectional interface 2201. When the camera module 210 receives the request through the first bidirectional interface 2101, it can acquire image data. Specifically, the camera module 210 may include an optical lens and an image sensor. The image sensor may include a photosensitive unit (e.g., a photodiode), an A / D converter, and an ISP. Specifically, the photosensitive unit can convert the light signal passing through the optical lens into an electrical signal, i.e., an analog image signal. The analog image signal is then converted into image data (i.e., a digital image signal) by the A / D converter.

[0071] In one method 1, the image data of the camera module 210 can be cached for subsequent processing. For example, after acquiring image data through the A / D converter, the ISP in the camera module 210 may not immediately perform first image processing on part or all of the acquired image data. For example, the image data that is not processed immediately may be a part of all the image data acquired by the camera module 210, corresponding to the first image data in the following text. The camera module 210 directly sends the acquired first image data to the second bidirectional interface 2201 via the first bidirectional interface 2101, so that the SoC 110 can receive the first image data and cache the first image data through the second bidirectional interface 2201. The specific method can be referred to the description of the following embodiments.

[0072] Unlike method 1, in method 2, the camera module 210 directly processes image data without buffering. For example, after acquiring image data through the A / D converter, the ISP in the camera module 210 can directly or immediately perform first image processing on the acquired image data, such as performing first image processing on the second image data from all acquired image data, to obtain a first image processing result. That is, without needing to send the second image data through the first bidirectional interface, the ISP performs the first image processing on the second image data to obtain a first image processing result.

[0073] In the above scheme, the first image processing may be at least one of the following: noise reduction, calibration, or defect detection. This application does not specifically limit the type of operation of the first image processing; any reasonable operation can be considered as the first image processing. The result of the first image processing can be sent to the second bidirectional interface 2201 via the first bidirectional interface 2101.

[0074] In summary, the electronic system provided in this application, by using a bidirectional interface that supports a bidirectional data communication interface protocol to connect the camera module and the SoC, can improve the flexibility of the interface so that the camera module and the SoC can transmit data information bidirectionally, enabling the camera module to process image data more flexibly.

[0075] In addition, since the first bidirectional interface and the second bidirectional interface support bidirectional data communication interface protocols, the first bidirectional interface and the second bidirectional interface can replace the function of the MIPI bus interface in the prior art. In this way, the camera module and the SoC do not need to use other bus interfaces such as MIPI, which can reduce the number of reserved pins of the camera module and the SoC and reduce the package area.

[0076] Optionally, in one approach, after receiving the first image processing result or the first image data through the second bidirectional interface 2201, the SoC 220 can perform second image processing on the first image processing result or the first image data to obtain a corresponding second image processing result. For example, the SoC 220 can perform at least one of the following processing on the received image data: 3A processing or data domain conversion, to obtain the corresponding image processing result. The 3A processing includes autofocus (AF), auto exposure (AE), and auto white balance (AWB). Furthermore, the SoC 220 can present the second image processing result to the user through a display screen; for example, the SoC 220 can transmit the second image processing result to a user. Figure 1 The display unit 170 shown is a similar display unit, which displays the second image processing result.

[0077] In some scenarios, the camera module 210 needs to process a large amount of image data. However, since the camera module 210 cannot process all image data in real time, more storage space is needed to cache the image data so that the ISP inside the camera module 210 can process other cached image data only after it has finished processing the currently processed image data. For example, the ISP inside the camera module 210 can also integrate image algorithms and AI algorithms, and the camera module 210 needs to cache multiple frames of image data when processing image data. Furthermore, functions such as face recognition, phone unlocking, environmental recognition, and monitoring require a always-on camera module 210 that is online in real time with lower power consumption. If all post-processing of image data is done by the SoC 220, the power consumption would be high. It is more reasonable for the camera module 210 to perform at least part of the post-processing, and when the camera module 210 performs post-processing, it requires approximately tens of MB of storage space to cache image data. For example, since the electronic system 200 stores personal privacy and asset information, data encryption in the camera module 210 is becoming increasingly urgent to prevent hackers from stealing data from the SoC 220 via the camera module 210 interface. Integrating the encryption algorithm into the camera module 210 requires at least tens of MB of storage space for data caching. Furthermore, the market offers a wide variety of camera manufacturers and models with significant differences in image data encoding and formats. Integrating a raw data domain image format classification algorithm within the camera module 210 requires approximately tens of MB of storage space for data caching. Additionally, to reduce the burden on the SoC 220, some low-resolution photography and raw data domain processing are performed within the camera module 210, requiring hundreds of MB of storage space for data caching. Finally, synchronizing data between multiple cameras within the camera module 210 via a high-speed bus for more efficient automatic image stabilization (AIS) also requires storage space.

[0078] As can be seen from the above application examples, the camera module 210 requires a large storage space when processing image data. However, the storage space within the camera module 210 is limited, and due to factors such as cost, area, packaging, and power consumption, it is impossible to integrate a larger capacity SRAM within the camera module 210. The solution provided in this application can solve the above problems. In one optional implementation, such as... Figure 2 As shown, the system 200 may further include a memory 230 connected to the SoC 220. Optionally, the memory 230 is used to support the scheme of caching image data of the camera module 210 mentioned in Method 1 above.

[0079] In one example corresponding to Method 1 above, after the camera module 210 sends the first image data to the SoC 220 via the first bidirectional interface 2101, the SoC 220, upon receiving the first image data via the second bidirectional interface, can store the first image data in the memory 230. After caching for a period of time, the SoC 220 can also reread the first image data from the memory 230 and send it back to the camera module 210 via the second bidirectional interface 2201. Accordingly, the camera module 210 can receive the first image data via the first bidirectional interface 2101 and begin performing first image processing on the received first image data.

[0080] Specifically, after receiving the first image data through the second bidirectional interface 2201, the SoC 220 can store the first image data in the memory 230 to achieve image data caching. When the camera module 210 needs to use the first image data, the cached data can be retransmitted to the camera module 210. That is, the storage path of the first image data is: first bidirectional interface 2101 → second bidirectional interface 2201 → memory 230. The reading path of the first image data is: memory 230 → second bidirectional interface 2201 → first bidirectional interface 2101.

[0081] In an example corresponding to Method 2 above, after the camera module 210 sends the first image processing result to the SoC 220 via the first bidirectional interface 2101, the SoC 220, upon receiving the first image processing result via the second bidirectional interface 2201, can store the first image processing result in the memory 230. After caching for a period of time, the SoC 220 can also reread the first image processing result from the memory 230 and send it back to the camera module 210 via the second bidirectional interface 2201. Accordingly, the camera module 210 can receive the first image processing result via the first bidirectional interface 2101 and begin performing third image processing on the received result. Specifically, after receiving the first image processing result via the second bidirectional interface 2201, the SoC 220 can store the first image processing result in the memory 230 to cache the image data. When the camera module 210 needs to use the first image processing result, the cached data can be retransmitted to the camera module 210. It is understood that the storage path and retrieval path of the first image processing result are the same as those of the first image data.

[0082] The third image processing described above may have the same or different functions as the first image processing, and this application does not limit this. For example, the third image processing may be at least one of noise reduction, calibration, or defect detection. This application does not specifically limit what kind of operation the third image processing is, and any reasonable operation can be classified as third image processing.

[0083] It should be understood that the result obtained after performing a first image processing on the first image data (i.e., the first image processing result) or the result obtained after performing a third image processing on the first image processing result (i.e., the third image processing result) can be further sent to the SoC 220 through the first bidirectional interface 2101. After receiving the first image processing result through the second bidirectional interface 2201, the SoC 220 can perform a second image processing on the first image processing result to obtain a second image processing result. Alternatively, after receiving the third image processing result through the second bidirectional interface 2201, the SoC 220 can perform a second image processing on the third image processing result to obtain a fourth image processing result. Furthermore, the SoC 220 can present the second image processing result or the fourth image processing result to the user through a display screen; for example, the SoC 220 can transmit the second image processing result or the fourth image processing result to a user. Figure 1 The display unit 170 shown is a similar display unit, which displays the second image processing result or the fourth image processing result.

[0084] Therefore, based on the above scheme, expressions such as "first," "third," and "fourth" are not used to limit the specific scheme of the embodiment. Various data, whether raw image data or image processing results processed by any ISP or other processing unit, can be transmitted between the camera module 210 and the SoC 220 through a bidirectional interface. This allows for flexible further image processing by any processing unit in the camera module 210 and SoC 220, and the processing results can be flexibly transmitted between the camera module 210 and SoC 220 via the bidirectional interface, improving the flexibility and processing performance of the scheme.

[0085] Based on the above solution, the camera module 210 can access the memory 230 through the first bidirectional interface and the second bidirectional interface. This access is relayed by the SoC 220, thereby solving the above problems and expanding the application scenarios of the electronic system.

[0086] Furthermore, such as Figure 2To achieve the aforementioned image data caching, the SoC 220 may further include a storage controller 2202, used to store the first image data (or the first image processing result) to the memory 230 and to read the first image data (or the first image processing result) from the memory 230. That is, the data relay function between the camera module 210 and the memory 230 can be implemented through the storage controller 2202 within the SoC 220. Since the camera module 210 does not possess the relevant capabilities of the storage controller 2202, data transfer and caching are indirectly achieved through the storage controller 2202 within the SoC 220, thus achieving the aforementioned data transmission function. Specifically, the storage path for the first image data (or the first image processing result) is: first bidirectional interface 2101 → second bidirectional interface 2201 → storage controller 2202 → memory 230. The reading path for the first image data (or the first image processing result) is: memory 230 → storage controller 2202 → second bidirectional interface 2201 → first bidirectional interface 2101.

[0087] Optionally, the memory may be DDR RAM, or other types of memory, which are not limited in this application.

[0088] Optionally, the first bidirectional interface may include a sending interface and a receiving interface, or a transceiver interface multiplexed for sending and receiving. That is, the first bidirectional interface may include two interfaces, one for sending and the other for receiving. Alternatively, the first bidirectional interface may be implemented by a single transceiver interface that can perform sending and receiving functions in a time-division multiplexing manner. Similarly, the second bidirectional interface may also include a sending interface and a receiving interface, or a transceiver interface multiplexed for sending and receiving.

[0089] Optionally, the aforementioned bidirectional data communication interface protocol can be the SerDes interface protocol. It should be understood that the SerDes interface protocol is a high-speed bidirectional interface protocol; for details regarding the SerDes interface protocol, please refer to existing technologies, which will not be elaborated upon here.

[0090] For example, Figure 3 A schematic diagram of a pin definition for the SerDes interface is shown. See also... Figure 3 The SerDes interface pins include two pairs of differential signal lines. One pair is for transmitting differential signals (TX-, TX+), and the other pair is for receiving differential signals (RX-, RX+). These two pairs of differential signal lines enable bidirectional, full-duplex, point-to-point data transmission.

[0091] Existing technologies using MIPI interfaces have a large number of pins; for example, a 3-lane CPHY requires 9 pins, resulting in high complexity. In contrast, the SerDes interface protocol provided in this application only requires 4 pins to achieve high-speed bidirectional transmission. Therefore, by using the SerDes interface to couple the camera module and the SoC, the number of reserved pins in both the camera module and the SoC can be reduced, thus decreasing the package area.

[0092] In addition, the SerDes interface can embed a clock, thus enabling the camera module and the SoC to have the same clock source without needing to transmit clock signals from the clock circuit inside the SoC to the first bidirectional interface 2101 and the second bidirectional interface 2201.

[0093] Optionally, when there is no data transmission between the camera module 210 and the SoC 220, the first bidirectional interface 2101 and the second bidirectional interface 2201 can enter a low-power mode. Correspondingly, the camera module 210 and the SoC 220 also enter a low-power mode. Thus, by enabling the first and second bidirectional interfaces to enter a low-power mode when there is no data transmission between the camera module 210 and the SoC 220, power consumption of the camera module 210 and the SoC 220 can be saved.

[0094] It should be noted that the electronic system 200 may also include one or more camera modules other than the aforementioned camera module 210. Each camera module may include a bidirectional interface that supports the bidirectional data communication interface protocol. For each camera module, the SoC includes a corresponding bidirectional interface, so that each camera module and the SoC can be coupled through a corresponding pair of bidirectional interfaces, thereby enabling point-to-point bidirectional data communication.

[0095] The following text will combine Figures 4 to 6 The camera module 210 and SoC 220 are described in detail. Figure 4 This is a schematic diagram of one structure of camera module 210. See also... Figure 4 The camera module 210 includes a first image processing device 2102 and a first bidirectional interface 2101. The first bidirectional interface 2101 is coupled to a second bidirectional interface of the system-on-chip (SoC) 220, and the first and second bidirectional interfaces support the same bidirectional data communication interface protocol.

[0096] The camera module 210 may also include an optical lens 2103. The optical lens 2103 affects image quality and imaging effect. It primarily utilizes the refraction principle of lenses to form an image; that is, light from a scene passes through the lens and forms a clear image on the focal plane, which is then recorded by a photosensitive material or sensor. The lens can be a system composed of different lenses (lenses), and its components can be lens structures, such as several lenses. The lenses can be plastic lenses or glass lenses, and can be spherical or aspherical. The lens can be a fixed-focal-length lens, a zoom lens, a standard lens, a short-focal-length lens, or a long-focal-length lens.

[0097] The first image processing device 2102 may be an image sensor, which may include a photosensitive unit (e.g., a photodiode) 2102a, an A / D conversion unit 2102b, and an image ISP 2102c. Specifically, the photosensitive unit 2102a can convert the light signal passing through the optical lens 2103 into an electrical signal, i.e., an analog image signal. The analog image signal is then converted into image data (i.e., a digital image signal) by the A / D conversion unit 2102b. The image ISP 2102c can further process the image data to obtain the corresponding processing result.

[0098] An image sensor is a semiconductor chip containing hundreds of thousands to millions of photodiodes on its surface. When illuminated, these photodiodes generate electrical charges, which are then converted into digital signals by an analog-to-digital converter (A / D) chip. Image sensors can be either charge-coupled devices (CCDs) or complementary metal-oxide-semiconductor (CMOS) devices. CCD image sensors use a highly sensitive semiconductor material to convert light into electrical charges, which are then converted into digital signals by an A / D converter. CMOS primarily utilizes silicon and germanium, allowing N-type (negative) and P-type (positive) semiconductors to coexist on the CMOS. The current generated by these complementary effects is recorded and interpreted by the processing chip as an image.

[0099] It should be understood that the functions and operations performed by the first image processing device 2102 and the first bidirectional interface 2101 can be found in the above description of system 200, and will not be repeated here.

[0100] It should also be understood that the camera module 210 may include other components besides the first image processing device 2102 and the first bidirectional interface 2101, such as at least one of a power module, random access memory (RAM) or read-only memory (ROM).

[0101] Figure 5 This is a schematic diagram of the SoC 220. See also... Figure 5 The SoC 220 includes a second bidirectional interface 2201 and a second image processing device 2203. The second bidirectional interface 2201 is coupled to the first bidirectional interface 2101 of the camera module 210, and the first bidirectional interface 2101 and the second bidirectional interface 2201 support the same bidirectional data communication interface protocol.

[0102] The second image processing device 2203 is used to perform second image processing on the first image processing result, first image data, or third image processing result. For example, the second image processing device 2203 may include one or more of the following units: an image signal processor (ISP) 2203a, a central processing unit (CPU) 2203b, a neural network processing unit (NPU) 2203c, and a digital signal processor (DSP) 2203d. The second image processing device 2203 may also include other image processing units, which are not limited in this application. The second image processing device 2203 can perform at least one subsequent processing on the first image data, first image processing result, or third image processing result, such as 3A processing, image domain conversion, or image post-processing.

[0103] It should also be understood that the SoC 220 may include other components besides the second bidirectional interface 2201 and the second image processing device 2203, such as a clock domain, power supply, and one or more of RAM and ROM.

[0104] Optionally, the SoC 220 is connected to the memory 230. The SoC 220 can store image data received through the second bidirectional interface 2201, such as first image data or first image processing result, into the memory 230. The SoC 220 can also read the first image data or first image processing result stored in the memory 230 from the memory 230, and then send it to the camera module 210 through the second bidirectional interface 2201.

[0105] Optionally, the SoC may further include a storage controller 2202. The storage controller 2202 is used to store the first image data or the first image processing result into the memory 230, and to read the first image data or the first image processing result from the memory 230.

[0106] It should be understood that the operations performed by the various units or modules in the SoC can be found in the previous description of System 200, and will not be repeated here.

[0107] Figure 6 This is a schematic flowchart of an image processing method 300 provided in this application. The method 300 can be applied to the aforementioned system 200, that is, the camera module, SoC and memory in the method 300 can respectively correspond to the camera module 210, SoC 220 and memory 230 in the system 200.

[0108] S310, the camera module acquires first image data and sends it to the SoC. Correspondingly, the SoC receives the first image data. S320, the SoC stores the first image data in its memory. S330, the SoC reads the first image data from its memory. When the camera module needs to use the first image data, the SoC can read it from its memory. S340, the SoC sends the first image data to the camera module. Correspondingly, the camera module receives the first image data. S350, the camera module performs first image processing on the first image data to obtain a first image processing result.

[0109] Optionally, the method may further include: S360, the camera module sends a first image processing result to the SoC; S370, the SoC performs second image processing on the received first image processing result to obtain a second image processing result. Specific details regarding this method 300 can be found in the preceding description of system 200, and will not be elaborated upon here.

[0110] According to the image processing method provided in this application, the camera module can access the memory through the SoC, thereby solving the problem that the camera module cannot process large amounts of data due to insufficient storage space.

[0111] Figure 7 This is a schematic flowchart of another image processing method 400 provided in this application. This method 400 can be applied to the aforementioned system 200, that is, the camera module, SoC and memory in method 400 can respectively correspond to the camera module 210, SoC 220 and memory 230 in system 200.

[0112] S410, the camera module acquires second image data and performs first image processing on the second image data to obtain a first image processing result. It should be noted that step S410 in method 400 is similar to step S350 in method 300. S420, the camera module sends the first image processing result to the SoC. Correspondingly, the SoC receives the first image processing result. S430, the SoC stores the first image processing result in its memory. S440, the SoC reads the first image processing result from its memory. The SoC can read the first image processing result from its memory when the camera module needs to use it. S450, the SoC sends the first image processing result to the camera module. Correspondingly, the camera module receives the first image processing result. S460, the camera module performs third image processing on the first image processing result to obtain a third image processing result.

[0113] Optionally, the method may further include: S470, the camera module sends a third image processing result to the SoC; S480, the SoC performs a second image processing on the received third image processing result to obtain a fourth image processing result. Specific details regarding method 400 can be found in the preceding description of system 200, and will not be elaborated upon here.

[0114] According to the image processing method provided in this application, the camera module can access the memory through the SoC, thereby solving the problem that the camera module cannot process large amounts of data due to insufficient storage space.

[0115] It is understood that the functions or algorithms implemented by any processing unit involved in the above embodiments, such as ISP, CPU, NPU, or DSP, can be fully or partially implemented through software, hardware, firmware, or any combination thereof. Firmware can be considered as a software implementation. When implemented using software or firmware, any processing unit can run the relevant software, which can be fully or partially implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the processing unit, the processes or functions described in the embodiments of this application are generated, fully or partially.

[0116] This application also provides a computer-readable storage medium storing a computer program. When the program is run on a processing unit or corresponding device, such as a System-on-a-Chip (SoC), the method flow of this application embodiment is implemented. This application also provides a computer program product. When the computer program product is run on the processing unit or corresponding device, the method flow of this application embodiment is implemented. The computer program product may include a large number of code instructions and be implemented by multiple software units; this embodiment does not limit this.

[0117] It is understood that any of the processing units can be selectively implemented in at least part of the hardware. For example, the ISP may include a large number of data logic operation circuits. When it is not necessary to run software, these circuits process the image signal in the manner of logic gate operations to achieve the corresponding functions or processes mentioned in the previous embodiments.

[0118] It is understandable that the functions or algorithms in the method flow of this embodiment are implemented more in a combination of software and hardware. For example, while performing logic gate operations, the ISP will also execute the necessary software to implement the corresponding functions or algorithms.

[0119] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An electronic system, characterized by The application relates to a camera module, a system on chip (SoC) and a memory connected to the SoC. The camera module comprises a first bidirectional interface. The SoC comprises a second bidirectional interface coupled to the first bidirectional interface, and the first bidirectional interface and the second bidirectional interface support the same bidirectional data communication interface protocol, which comprises a serializer / deserializer (SerDes) interface protocol. The camera module is configured to acquire image data, perform first image processing on the image data to obtain a first image processing result, and the image data comprises first image data. The SoC is configured to perform second image processing on the first image processing result to obtain a second image processing result. The camera module is further configured to send the first image data to the SoC through the first bidirectional interface before performing the first image processing on the first image data. The SoC is further configured to receive the first image data through the second bidirectional interface, store the first image data in the memory, read the first image data from the memory, and send the first image data to the camera module through the second bidirectional interface. The camera module is further configured to receive the first image data through the first bidirectional interface and perform the first image processing on the first image data. Each of the first bidirectional interface and the second bidirectional interface comprises:

2. The system of claim 1, wherein, a sending interface and a receiving interface; or a transceiving interface used for sending and receiving. The memory comprises a double data rate (DDR) random access memory (RAM).

3. The system of claim 1, wherein, The SoC comprises a storage controller configured to store the first image data in the memory and read the first image data from the memory.

4. The system of claim 1 or 3, wherein, The application relates to a camera module, a system on chip (SoC) and a memory connected to the SoC.

5. A camera module, comprising: The camera module comprises a first bidirectional interface. The SoC comprises a second bidirectional interface coupled to the first bidirectional interface, and the first bidirectional interface and the second bidirectional interface support the same bidirectional data communication interface protocol, which comprises a serializer / deserializer (SerDes) interface protocol. The camera module is configured to acquire image data, perform first image processing on the image data to obtain a first image processing result, and the image data comprises first image data. The SoC is configured to perform second image processing on the first image processing result to obtain a second image processing result. The camera module is further configured to send the first image data to the SoC through the first bidirectional interface before performing the first image processing on the first image data.

6. The camera module of claim 5, wherein, The SoC is further configured to receive the first image data through the second bidirectional interface, store the first image data in the memory, read the first image data from the memory, and send the first image data to the camera module through the second bidirectional interface. The camera module is further configured to receive the first image data through the first bidirectional interface and perform the first image processing on the first image data. The first bidirectional interface and the second bidirectional interface each comprise:

7. The camera module of claim 5, wherein, a sending interface and a receiving interface; or 8. A system on chip (SoC), comprising: a transceiving interface used for sending and receiving. The memory comprises a double data rate (DDR) random access memory (RAM). The SoC is connected to a memory, and the SoC comprises: a second bidirectional interface coupled to the first bidirectional interface of the camera module, and the first bidirectional interface and the second bidirectional interface support a same bidirectional data communication interface protocol, the second bidirectional interface configured to obtain a first image processing result from the camera module, the first image processing result being obtained by the camera module performing first image processing on image data, the image data including first image data, and the bidirectional data communication interface protocol including a serializer / deserializer (SerDes) interface protocol; a second image processing device configured to perform second image processing on the first image processing result to obtain a second image processing result; the second bidirectional interface is further configured to obtain the first image data from the camera module; the SoC further includes a memory controller configured to store the first image data to a memory and read the first image data from the memory; the second bidirectional interface is further configured to send the first image data to the camera module, and the first image data is processed by the camera module to perform the first image processing.

9. The SoC of claim 8, wherein, the second bidirectional interface includes: a sending interface and a receiving interface; or a transceiving interface that is reused for sending and receiving.

10. The SoC of claim 8, wherein, the memory includes a double data rate (DDR) random access memory (RAM).

Images