Image processing method, system, electronic device and storage medium

Abstract

The application provides an image processing method and system, electronic equipment and storage medium, and relates to the technical field of automobile electronics. A lightweight model and a low-latency cache for executing an AI algorithm are configured in an ISP system, a communication channel between the low-latency cache and each module in the ISP system is established through a high-speed read-write interface, in the process of image processing, each module reads corresponding data from the cache, and cooperates with the lightweight model to complete corresponding AI operation processing to complete image processing, and finally obtained image data is output. That is, an image processing system based on AI-ISP and low-latency cache is constructed, and in image processing, the AI algorithm executed by the lightweight model is embedded in any position in the image processing process to execute one or more corresponding AI algorithms. Not only can the purpose of adapting to complex scenes be achieved, but also the purposes of improving image processing efficiency and accuracy can be achieved.

Description

Technical Field

[0001] This application relates to the field of automotive electronics technology, and more particularly to an image processing method, system, electronic device, and storage medium. Background Technology

[0002] Current intelligent vehicle systems, such as intelligent cockpit systems, advanced driver assistance systems, electronic exterior rearview mirrors, and streaming media interior rearview mirrors, typically rely on multiple high-quality image sensors, including forward-facing, surround-view, side-facing, and electronic rearview mirrors, for environmental perception and image output. Based on this, the constructed visual imaging system primarily utilizes an ISP (Image Signal Processor) to process the raw data from the image sensors, thereby obtaining signals that conform to human or machine vision.

[0003] In current automotive vision imaging systems, the Information Service Provider (ISP) primarily relies on traditional algorithm IP cores for image processing. However, in today's dynamic driving scenarios, such as extreme conditions like extreme darkness, direct headlights, and high contrast, these traditional IP cores struggle to cope, resulting in poor image quality. Furthermore, traditional ISPs use a serial video stream, requiring read and write operations to external storage as it passes through various processing modules. This frequent external storage read and write operations also cause significant latency in the vision imaging system.

[0004] This shows that existing image processing methods suffer from problems such as inability to handle complex scenes, low image processing accuracy, and low processing efficiency. Summary of the Invention

[0005] In view of this, embodiments of this application provide an image processing method, system, electronic device, and storage medium to solve the problems of existing image processing methods being unable to cope with complex scenes, having low image processing accuracy, and having low processing efficiency.

[0006] To address the aforementioned issues, this application provides the following technical solutions:

[0007] The first aspect of this application discloses an image processing system, the image processing system comprising:

[0008] The system includes a cache, an image preprocessing module, an image processing module, a core processing module, and memory connected to the cache via a high-speed read / write interface for data transmission, as well as a display processing module connected to the memory.

[0009] The buffer is used to receive raw RAW image data acquired by the sensor in real time, as well as data generated by the image preprocessing module, image processing module and core processing module;

[0010] The image preprocessing module is used to read the original RAW image data from the cache, preprocess it, and transmit the preprocessed data to the cache and the image processing module.

[0011] The core processing module is configured with a lightweight model for executing AI algorithms. When the image preprocessing module and the image processing module perform corresponding processing, the model reads the corresponding data generated by the image preprocessing module and / or the image processing module from the cache, coordinates with the lightweight model to execute AI algorithms for image processing, and transmits the processed data to the cache.

[0012] The image processing module is used to read the preprocessed data and the processing data required for image processing from the cache, perform image processing, and transfer the obtained first image data to the memory based on the cache.

[0013] A second aspect of this application discloses an image processing method, applicable to the image processing system disclosed in the first aspect of this application, the image processing method comprising:

[0014] Utilize buffering to receive raw RAW image data acquired by the sensor in real time;

[0015] While the image preprocessing module and the image processing module are performing corresponding processing, the core processing module reads the corresponding data generated by the image preprocessing module and / or the image processing module from the cache, controls the lightweight model to execute the AI ​​algorithm to perform image processing, and transmits the processed data to the cache.

[0016] The display processing module reads the first image data from memory in real time, performs image display processing, and outputs the second image data.

[0017] The corresponding processing includes:

[0018] The image preprocessing module reads the original RAW image data from the cache, performs preprocessing, and transmits the preprocessed data to the cache and the image processing module.

[0019] The image processing module reads the preprocessed data and / or the processing data required for image processing from the cache, performs image processing, transmits the obtained YUV data to the cache, and transmits the obtained first image data to the memory based on the cache.

[0020] The third method of this application discloses an electronic device, which includes at least one processing core, a deserializer as input, and a serializer as output. The processing core is communicatively connected to the deserializer and the serializer. The processing core includes the image processing system disclosed in the first aspect of this application, and the image processing system is used to execute the image processing method disclosed in the second aspect of this application.

[0021] The fourth aspect of this application discloses a storage medium storing a program thereon, which, when executed by a controller, implements the image processing method disclosed in the second aspect of this application.

[0022] Based on the above-described embodiments of this application, an image processing method, system, electronic device, and storage medium are provided. The image processing system includes a cache and an ISP system configured with a lightweight model executing AI algorithms. The cache transmits data with various modules within the ISP system via a high-speed read / write interface. The cache is used to receive raw RAW image data acquired by the camera device in real time and to store data generated by various modules within the ISP system during image processing. During image processing, each module reads corresponding data from the cache and collaborates with the lightweight model to perform corresponding AI algorithm processing to complete the image processing, outputting the final image data. In this embodiment, by configuring a lightweight model executing AI algorithms and a low-latency cache within the ISP system, an image processing system based on AI-ISP and a low-latency cache is constructed. During image processing, the AI ​​algorithms executed by the lightweight model are embedded at any position in the image processing flow to execute one or more corresponding AI algorithms. This not only achieves the goal of adapting to complex scenes but also improves image processing efficiency and accuracy. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the structure of an image processing system disclosed in an embodiment of this application;

[0025] Figure 2 This is a schematic diagram of the structure of another image processing system disclosed in an embodiment of this application;

[0026] Figure 3 This is a schematic diagram of the structure of another image processing system disclosed in an embodiment of this application;

[0027] Figure 4 This is a schematic diagram illustrating a specific implementation of noise AI automatic adjustment in an image processing system disclosed in an embodiment of this application;

[0028] Figure 5 This is a schematic diagram illustrating a specific implementation of an image processing system for automatic adjustment of AI exposure parameters disclosed in an embodiment of this application;

[0029] Figure 6 This is an application example diagram of the lightweight model for executing AI algorithms disclosed in the embodiments of this application;

[0030] Figure 7 This is a schematic flowchart of an image processing method disclosed in an embodiment of this application;

[0031] Figure 8 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application;

[0032] Figure 9 This is a schematic diagram of the structure of another electronic device disclosed in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0035] The following are explanations of the English abbreviations that appear in this application:

[0036] ISP: Image Signal Processor;

[0037] AI: Artificial Intelligence;

[0038] CPU: Central Processing Unit;

[0039] NPU: Neural Processing Unit;

[0040] DSP: Digital Signal Processor;

[0041] SRAM: Static Random-Access Memory;

[0042] RRAM: Resistive Random-Access Memory.

[0043] As the background technology shows, existing ISPs process video streams serially during image processing. Each processing module in the ISP frequently needs to read and write to external storage to obtain the data to be processed. This not only leads to low processing efficiency but also easily causes delays, resulting in untimely output of image processing results. This, in turn, causes a chain reaction of delays in subsequent judgments based on the image processing results. Furthermore, the computing power of the traditional algorithm IP cores configured in existing ISPs is insufficient to handle complex, extreme, or highly variable scenarios.

[0044] Therefore, this application discloses an image processing system and its corresponding image processing method. This image processing system is built based on AI-ISP and a low-latency cache. It embeds AI algorithms executed by a lightweight model at any point in the image processing flow. When facing complex, extreme, or variable scenes, the lightweight model embedded at any point in the image processing flow executes one or more corresponding AI algorithms to collaboratively perform image processing such as noise reduction, exposure, and color calibration, thus achieving adaptability to complex scenes. Simultaneously, a low-latency cache is configured, and during image processing, data transfer between various modules is based on this low-latency cache, thereby improving image processing efficiency and accuracy.

[0045] like Figure 1 The diagram shown is an architectural schematic of an image processing system disclosed in an embodiment of this application. This architecture is built upon an AI-ISP and a low-latency cache. Specifically, a low-latency cache and a lightweight model for executing AI algorithms are configured in the ISP, thereby forming an image processing system capable of accurately and efficiently handling image processing in various scenarios.

[0046] The architecture of the image processing system based on AI-ISP and low-latency cache includes: cache 11, image preprocessing module 12, image processing module 13, core processing module 14 and memory 15 connected to cache 11 via high-speed read / write interface for data transmission, and display processing module 16 connected to memory 15.

[0047] The image preprocessing module 12 is a front-end module of the AI-ISP architecture, while the image processing module 13 and the display processing module 16 are back-end modules of the AI-ISP architecture. The image preprocessing module 12 is also connected to the image processing module 13, and the core processing module 14 is also connected to the memory 15. The core processing module 14 carries the logical operation portion of the AI-ISP architecture of this application.

[0048] The cache 11 is used to receive raw RAW image data acquired by the sensor in real time, as well as data generated by the image preprocessing module 12, image processing module 13 and core processing module 14 during image processing. Specifically, the data can be transferred between the image preprocessing module 12, image processing module 13 and core processing module 14, and memory 15 based on the cache 11.

[0049] The raw RAW image refers to the unprocessed raw image data acquired by the sensor.

[0050] In one embodiment of this application, the sensor can be a sensor configured in any in-vehicle device such as a smart cockpit system, an advanced driver assistance system, an electronic exterior rearview mirror, or a streaming media interior rearview mirror.

[0051] In this embodiment, the cache 11 is used to directly store the original RAW image input to the image processing system, as well as the data generated by the image preprocessing module 12, the image processing module 13, and the core processing module 14 during the image processing process. This allows each module in the subsequent AI-ISP architecture to transmit data through the cache 11 via a high-speed read / write interface, thereby avoiding frequent read / write operations by each module to external storage or memory 15 and improving image processing efficiency.

[0052] In one embodiment of this application, the cache 11 is a cache with the same low-latency function as the L1 / L2 cache; at the same time, the low-latency cache 11 has high reliability under extreme temperature and radiation environments, and can meet the stringent requirements of automotive-grade standards; it is also compatible with 28nm and below process nodes, supports multi-level storage, and has both high density and scalability. In specific applications, preferably, the cache 11 can be SRAM or RRAM. In this embodiment of the application, RRAM is preferred because it has extremely low standby power consumption and does not require data transfer. Its read and write speeds are close to those of SRAM, but its latency is much lower, 1000 times lower than that of traditional e-flash, meeting high throughput requirements and suitable for processing high real-time data.

[0053] The image preprocessing module 12 is used to read raw RAW image data from the cache 11, preprocess it, and then transmit the preprocessed data to the cache 11 and the image processing module 13. The image processing module 13 performs corresponding image processing based on the preprocessed data. It also enables other modules to read the preprocessed data from the cache 11 via a high-speed read / write interface.

[0054] The core processing module 14 is equipped with a lightweight model for executing AI algorithms. When the image preprocessing module 12 and the image processing module 13 are performing corresponding processing, the core processing module 14 reads the corresponding data generated by the image preprocessing module 12 and / or the image processing module 13 from the cache 11, coordinates with the lightweight model to execute AI algorithms for image processing, and transmits the obtained processed data to the cache 11 so that other modules can read the processed data from the cache 11 to continue image processing according to the needs of image processing.

[0055] In this embodiment, the AI ​​algorithm is configured in the core processing module 14 as a lightweight model. During the image processing of the image preprocessing module 12 and the image processing module 13, the core processing module 14 reads the intermediate data generated by the image preprocessing module 12 or the image processing module 13, or both, through the cache 11. The configured lightweight model is used to read the intermediate data and perform AI calculations, that is, the AI ​​algorithm is embedded into the image processing flow. The intermediate data generated during image processing is processed in real time, and the processing result is transmitted to the cache 11. The image preprocessing module 12 and / or the image processing module 13 continue to perform image processing based on the processing result, and finally obtains the image data adjusted in real time based on the AI ​​algorithm.

[0056] It should be noted that the core processing module 14, in conjunction with the cache 11, reads data from the cache 11 for processing and writes the processing results back to the cache 11. This process can occur at any position in the image processing flow, and is not limited to the beginning or end position. This process depends on the cache 11 and the lightweight model configured to execute the AI ​​algorithm.

[0057] The image processing module 13 is used to read preprocessed data and processing data required for image processing from the cache 11, and to transfer the obtained first image data to the memory 15 based on the cache 11.

[0058] In this embodiment of the application, the image processing module 13 reads the image processing data required for image processing from the cache 11, which is data processed by AI. The image processing that the image processing module 13 can perform includes, but is not limited to, brightness and contrast processing, color processing, sharpness processing, etc. When performing any image processing, the image processing module 13 can obtain the corresponding data processed by AI from the cache 11 based on the requirements.

[0059] It should be noted that the AI-processed data used in the image processing module 13 specifically originates from the intermediate data input to the cache generated by the image preprocessing module 12 and / or the image processing module 13 during image processing, and is obtained by the core processing module 14 through AI calculation.

[0060] The display processing module 16 is used to read the first image data from the memory 15 in real time for image display processing and output the second image data.

[0061] In this embodiment of the application, the image display processing performed by the display processing module 16 includes, but is not limited to, layer overlay, gamma correction, frame rate conversion, etc.

[0062] This application discloses an image processing system built on an AI-ISP and a low-latency cache. A data link is established between the cache and various modules in the AI-ISP based on a high-speed read / write interface, enabling data transmission between the AI-ISP modules. During image processing, the core processing module, configured with a lightweight model for performing AI calculations, can be embedded into the image processing flow executed by the image preprocessing module and the image processing module based on this cache. The cache reads intermediate data generated by the image preprocessing module and the image processing module in real time to execute AI algorithms for corresponding image processing. AI algorithms can be integrated into any position within the image processing flow, meaning that one or more AI algorithms can be executed collaboratively to process image data for different image processing needs.

[0063] It should be noted that, unlike using a standalone AI algorithm module, this embodiment embeds the AI ​​algorithm at any point in the image processing flow. This allows for the acquisition and use of all intermediate data to execute the AI ​​algorithm for corresponding image processing, making image processing more flexible and precise. Furthermore, the AI ​​possesses stronger adaptive learning capabilities for complex or extreme scenarios, enabling it to adapt to a wider range of complex or extreme situations.

[0064] Based on this, by configuring a lightweight model and a low-latency cache to execute AI algorithms in the ISP system, an image processing system based on AI-ISP and low-latency cache is constructed. During image processing, the AI ​​algorithms executed by the lightweight model are embedded at any position in the image processing flow to execute one or more corresponding AI algorithms. This not only achieves the goal of adapting to complex scenarios, but also improves the efficiency and accuracy of image processing.

[0065] In one embodiment of this application, Figure 1 The core processing module 14 shown includes an NPU and a CPU.

[0066] The NPU is equipped with a lightweight model, which is used to perform AI calculations.

[0067] The CPU is used to read statistical data generated by the image preprocessing module 12 from the cache 11 and control the NPU to perform AI algorithms for image processing based on the currently determined image processing requirements and in collaboration with a lightweight model.

[0068] In one embodiment of this application, the image processing requirement includes any one or more of exposure parameter adjustment, noise reduction, brightness processing, contrast processing, and color processing.

[0069] It should be noted that the statistical data generated by the image preprocessing module 12 is intermediate data in the image processing process. This statistical data mainly includes brightness statistics, color statistics, focus statistics, and noise statistics generated by the image preprocessing module 12 when it reads the original RAW image data from the cache 11 for preprocessing.

[0070] The NPU is used to read data generated by the image preprocessing module 12 and the image processing module 13 from the cache 11 during collaborative processing, execute AI algorithms based on the image processing requirements of the collaborative lightweight model, and transmit the first processing result to the cache 11.

[0071] In one embodiment of this application, combined with Figure 1 ,like Figure 2The diagram shown is a structural schematic of another image processing system disclosed in this application embodiment; in this image processing system, the core processing module 14 includes not only NPU and CPU, but also DSP.

[0072] The core of a DSP lies in its ability to rapidly process continuous digital signals and perform complex mathematical operations. In image processing, it possesses high-speed processing capabilities, enabling pixel-level operations within images. Based on its proprietary image processing instructions, it can efficiently perform image processing in parallel, increasing processing speed. Simultaneously, it exhibits low power consumption, allowing it to complete a large number of image processing tasks with relatively low power consumption. In this embodiment, the DSP, during collaborative processing, reads data generated by the image preprocessing module 12 and the image processing module 13 from the cache 11, performs DSP processing, and transmits the resulting second processing result back to the cache 11.

[0073] In some embodiments of this application described above, based on different image processing requirements, the image preprocessing module 12 generates corresponding intermediate data, such as statistical data, RGB image data converted from RAW image data, and YUV image data, when performing preprocessing and / or the image processing module 13 performs image processing. The image preprocessing module 12 and / or the image processing module 13 transfer the obtained intermediate data to the cache 11. The CPU controls the NPU, or the NPU and DSP, to read the corresponding intermediate data from the cache 11 and perform AI operations based on the currently determined image processing requirements. Then, the obtained first processing result, or the first processing result and the second processing result, are transferred to the cache 11 so that the image preprocessing module 12 and / or the image processing module 13 can read the corresponding first processing result, or the first processing result and the second processing result, from the cache 11 and continue image processing. That is, the image preprocessing module 12 and the image processing module 13 perform any one or more of the following adjustments based on the first processing result, or the first processing result and the second processing result: exposure parameter adjustment, noise reduction processing, brightness processing, contrast processing and color processing. This allows the AI ​​algorithm to be embedded in any position in the image processing flow, thereby improving the efficiency of image processing.

[0074] Combination Figure 1 and Figure 2 ,like Figure 3 The diagram shown is a structural schematic of another image processing system disclosed in this application embodiment; in this image processing system, the image preprocessing module 12 includes a RAW image processing submodule 121 and a noise reduction submodule 122.

[0075] The RAW image processing submodule 121 includes a RAW statistics unit and a RAW image processing unit; the noise reduction submodule 122 includes a noise statistics unit and a noise reduction unit.

[0076] The RAW image processing submodule 121 is used to read the original RAW image data from the buffer 11 and perform preprocessing, and transmit the obtained RAW image statistics and RAW image data to the buffer 11 and the noise reduction submodule 122.

[0077] In this embodiment, the RAW image processing submodule 121, based on the RAW statistics unit and the RAW image processing unit, obtains RAW image statistics data, including but not limited to brightness statistics data, color statistics data, and focus statistics data, as well as RAW image data after processing such as black level correction and lens correction, by preprocessing the original RAW image data read from the cache 11.

[0078] The noise reduction submodule 122 is used to divide the RAW image data into M×N regions, calculate and statistically analyze the noise intensity of each region, and transmit the obtained RAW noise statistics data to the buffer 11 and the image processing module 13.

[0079] Where M refers to the number of blocks (vertical divisions) of the RAW image data, and N refers to the number of blocks (horizontal divisions) of the RAW image data. That is, the image is divided into an M×N matrix, and subsequent image processing modules perform image processing based on this M×N matrix. The values ​​of M and N correspond to the smallest unit of AI processing.

[0080] It should be noted that the noise statistics mentioned above are the same as the RAW noise statistics.

[0081] In the image processing flow, the image processing module 13 is used to perform image processing based on the RAW noise statistics and transmit the obtained YUV image data to the buffer 11.

[0082] In one embodiment of this application, the image processing module 13 includes a global and local tone mapping submodule 131, a color processing submodule 132, and a visual processing submodule 133.

[0083] The global and local tone mapping submodule 131 is used to perform adaptive brightness and contrast enhancement processing for the image.

[0084] The color processing submodule 132 is used to perform color reconstruction and enhancement processing of the image.

[0085] The visual processing submodule 133 is used to perform processing such as sharpening, resolution scaling, and distortion correction.

[0086] Specifically, the RAW noise statistics are processed by the global and local tone mapping submodule 131 and the color processing submodule 132, and then the YUV image data is output by the color processing submodule 132.

[0087] It should be noted that, in the case of a multi-channel image source, the global and local tone mapping submodule 131 extracts the display content from the input multi-channel image data, calculates the average brightness information based on the display content, and performs weight equalization based on the average brightness information to obtain the adjustment coefficients for each channel in the multi-channel image. Based on these adjustment coefficients, the corresponding multi-channel image data is adaptively processed; that is, adaptive brightness and contrast enhancement processing is performed on the corresponding multi-channel image based on these adjustment coefficients. This achieves consistency in the brightness of the multi-channel image.

[0088] It should be noted that during the image processing process of the image preprocessing module 12 and the image processing module 13, intermediate data such as RAW image data, RAW image statistics data, noise statistics data, and YUV image data are all transmitted to the cache 11. Based on this, the core processing module 14 reads the intermediate data from the cache 11, coordinates the configured lightweight model to perform AI calculations, and inputs the resulting processing results into the cache 11. The image preprocessing module 12 and the image processing module 13 read the processing results. If the processing result is a parameter adjustment result, the image preprocessing module 12 and the image processing module 13 adjust their image processing parameters based on the parameter adjustment result. After completing the image processing, the image data is obtained after real-time AI adjustment. This adjustment includes, but is not limited to, automatic AI exposure parameter adjustment (exposure parameter adjustment), automatic AI noise adjustment (noise reduction processing), AI brightness and contrast adjustment, and AI color adjustment.

[0089] In this embodiment, low-latency caching is used to realize data transfer between sub-modules within each module during image processing. At the same time, each module performs corresponding image processing according to an M×N matrix. Based on this, not only can the data transmission latency between modules be effectively reduced and the image processing efficiency be improved, but also the configured lightweight model is used to perform AI calculation adjustments during image processing, thereby improving the efficiency and accuracy of image processing.

[0090] To illustrate in more detail how cache-based AI operations are embedded in any part of the image processing workflow, we will use noise reduction and exposure parameter adjustment as examples.

[0091] When the image processing requirement is noise reduction, combined with the above... Figures 1 to 3 ,like Figure 4 The diagram shown is a schematic representation of an image processing system disclosed in this application that implements automatic noise AI adjustment.

[0092] RAW image processing submodule 121 is used to read raw RAW image data from cache 11 and perform preprocessing, and then transfer the obtained RAW image data to cache 11 and noise reduction submodule 122.

[0093] The noise reduction submodule 122 is used to divide the RAW image data into M×N regions, calculate and statistically analyze the noise intensity of each region, and transmit the obtained RAW noise statistics data to the buffer 11 and the image processing module 13.

[0094] Image processing module 13 is used to perform image processing based on RAW noise statistics and transmit the obtained YUV image data to buffer 11. Specifically, color processing submodule 132 performs image processing based on RAW noise statistics transmitted by global and local tone mapping submodule 131 and transmits the obtained YUV image data to buffer 11.

[0095] The CPU is used to read RAW noise statistics from cache 11, divide the noise level of each region of the RAW noise statistics according to the preset noise level, and control the NPU and DSP to perform noise reduction collaborative processing.

[0096] Specifically, after the CPU reads the RAW noise statistics data from cache 11, it divides the M×N region of the RAW noise statistics data into four levels: high, medium-high, medium-low, and low, according to a preset noise level. In other words, the RAW image data indicated by the RAW noise statistics data is divided into corresponding noise levels according to the preset noise levels.

[0097] The NPU is used to read RAW image data and YUV image data from cache 11 when performing noise reduction collaborative processing. Based on the noise level and lightweight model, it matches the corresponding AI noise processing to each region divided by the RAW image data and transmits the resulting third processing data to cache 11. Different noise levels correspond to different AI noise processing for different regions. Figure 4 The third type of data processed in this context is AI-processed data.

[0098] The DSP is used to read RAW image data and YUV image data from the buffer 11 during noise reduction co-processing, perform DSP processing according to the noise level, and transfer the resulting fourth processed data to the buffer 11.

[0099] Specifically, the NPU and DSP perform corresponding noise reduction processing based on the noise level and the regions divided by the RAW image data. Low-level noise regions are not processed, while high, medium-high, and medium-low noise regions are processed using different methods. Figure 4 The fourth type of data to be processed is DSP data.

[0100] It should be noted that when the NPU performs AI noise processing, the smallest unit of AI processing matches the number of rows and columns (regions) of the RAW image data.

[0101] Image processing module 13 is used to perform image processing by reading third processing data and fourth processing data from cache 11.

[0102] Specifically, the global and local tone mapping submodule 131 is used to read the third and fourth processing data to continue image processing, and to continue to transmit the corresponding processed data to the color processing submodule 132 for processing, and to continue to transmit the corresponding processed data to the visual processing submodule 133 for processing. Finally, the first image data is transmitted to the memory 15 through the cache 11, and the display processing module 16 reads the first image data from the memory 15 in real time for image display processing and outputs the second image data.

[0103] It should be noted that in some embodiments of this application, automatic noise AI adjustment can be implemented without including a DSP. Specifically:

[0104] Image processing module 13 is used to read third processing data from cache 11 for image processing. Specifically, global and local tone mapping submodule 131 is used to read the third processing data to continue image processing, and then transmit the corresponding processed data to color processing submodule 132 for processing, and then transmit the corresponding processed data to visual processing submodule 133 for processing. Finally, the first image data is transmitted to memory 15 through cache 11, and the display processing module 16 reads the first image data from memory 15 in real time for image display processing and outputs the second image data.

[0105] In the embodiments of this application, during the noise reduction process involving the pre-processing and post-processing modules, the CPU determines and controls the NPU based on the read RAW noise statistics, or controls the NPU and DSP to intervene in the noise reduction process. This mainly involves controlling the lightweight model in the NPU to perform the corresponding AI noise reduction adjustment process, ensuring that the AI ​​noise reduction process is integrated into the noise reduction process. This allows the processing result after AI noise reduction to intervene before the image processing flow is completed, which can not only adapt to complex scenes, but also improve the accuracy of image processing.

[0106] When the image processing requirement is to adjust exposure parameters, combined with the above... Figures 1 to 3 ,like Figure 5 The diagram shown is a schematic representation of an image processing system disclosed in this application that implements automatic adjustment of AI exposure parameters.

[0107] RAW image processing submodule 121 is used to read raw RAW image data from cache 11 and perform preprocessing, and transmit the obtained RAW image statistics and RAW image data to cache 11 and noise reduction submodule 122.

[0108] The noise reduction submodule 122 is used to divide the RAW image data into M×N regions, calculate and statistically analyze the noise intensity of each region, and transmit the obtained RAW noise statistics data to the buffer 11 and the image processing module 13.

[0109] The color processing submodule 132 performs image processing based on the RAW noise statistics and transmits the resulting YUV image data to the buffer 11.

[0110] The CPU is used to read the brightness statistics from the RAW image statistics in cache 11 and control the NPU and lightweight model to perform collaborative processing of exposure parameter adjustments.

[0111] The NPU is used to read RAW image data and YUV image data from cache 11 when performing exposure parameter adjustment collaborative processing. It performs AI scene recognition based on brightness statistics and a lightweight model to obtain scene classification. The exposure adjustment module combines the brightness statistics and the scene classification to generate new exposure parameters and transmits the obtained new exposure parameters to cache 11.

[0112] Among them, such as Figure 6 As shown, this lightweight model executing the AI ​​algorithm is used to acquire RGB images converted from RAW images, and the corresponding exposure parameters of the RGB images for scene recognition and scene classification. These exposure parameters include exposure time, analog gain, and digital gain. The scene classifications include daytime normal scenes, nighttime normal scenes, nighttime direct headlight scenes, nighttime flickering scenes, extremely dark scenes, and tunnel scenes. Each scene classification corresponds to a different adjustment mode, and each adjustment mode includes a description of the scene, i.e., scene information and adjustment strategy. For example, daytime normal scenes correspond to adjustment mode one, nighttime normal scenes correspond to adjustment mode two, nighttime direct headlight scenes correspond to adjustment mode three, nighttime flickering scenes correspond to adjustment mode four, extremely dark scenes correspond to adjustment mode five, and tunnel scenes correspond to adjustment mode six. The RAW image is then converted to an RGB image using an NPU or CPU.

[0113] This exposure adjustment module is used to determine the target exposure parameters based on the actual brightness Y_cur in the brightness statistics data and the target brightness Y_tar. It adjusts the exposure parameters according to the adjustment mode corresponding to different scene classifications, compares the adjusted target exposure parameters with the exposure parameters, and determines the new exposure parameters based on the comparison results. The target exposure parameters include the target exposure time Texp_tar1, the target analog gain Again_tar1, and the target digital gain Dgain_tar1.

[0114] The RAW image processing submodule 121 is also used to feed back the new exposure parameters to the sensor, so that the sensor can adjust the exposure parameters based on the new exposure parameters and continue to acquire image data.

[0115] In one embodiment of this application, if the scene is classified as a nighttime flickering scene, the exposure adjustment module is used to obtain the flickering frequency fre, adjust the exposure parameters based on the flickering frequency fre, the scene information corresponding to the nighttime flickering scene and the target exposure parameters, and obtain the adjusted target exposure parameters. Based on the comparison result between the adjusted target exposure parameters and the exposure parameters, the single-frame adjustment exposure parameters are calculated to obtain the new exposure parameters.

[0116] The flicker frequency fre is calculated and determined using an NPU or other hardware module based on adjustment mode four corresponding to the night flicker scene. The adjusted target exposure parameters include the secondary adjusted target exposure time Texp_tar2, the secondary adjusted target analog gain Again_tar2, and the secondary adjusted target digital gain Dgain_tar2.

[0117] A separate temporal smoothing filter module can be configured to perform single-frame adjustment of the exposure parameters based on the comparison results between the adjusted target exposure parameters and the exposure parameters, thereby obtaining new exposure parameters.

[0118] The RAW image processing submodule 121 is also used to feed back new exposure parameters to the sensor.

[0119] The sensor captures a RAW image with the new exposure parameters, and then proceeds to the next image processing cycle.

[0120] In this embodiment, during the exposure parameter adjustment process of the pre-processor and post-processor modules, the CPU reads data related to exposure parameter adjustment from the cache, namely brightness statistics and RAW noise statistics. Based on the read RAW noise statistics, it controls the NPU to intervene in the exposure parameter adjustment process. This mainly involves controlling the NPU to perform corresponding AI scene automatic recognition processing based on the brightness statistics and a lightweight model. According to the automatically recognized scene, such as a complex scene with dim lighting determined by AI scene recognition, new exposure parameters are generated and input into the RAW image processing subsystem through the cache. This allows the RAW image processing subsystem to feed back these new exposure parameters to the sensor, enabling the sensor to adjust its exposure parameters based on the new parameters and continue image data acquisition. This ensures that the current image processing system can adapt to complex scenes, thus achieving automatic exposure adjustment based on different scenes.

[0121] In this embodiment, in addition to embedding AI computation in the image processing flow of noise reduction and exposure parameter adjustment, when each module performs corresponding image processing, the CPU can control the NPU to perform image processing based on the cache used to store the intermediate data of each module, thereby embedding the AI ​​algorithm into the intermediate flow of each image processing.

[0122] In one embodiment of this application, when the current global and local tone mapping submodule 131 performs image processing, it can also embed AI operations to perform AI adjustment of brightness and contrast. The processing logic is the same: the RGB image processed by the global and local tone mapping submodule 131 is output to the buffer 11, the CPU controls the NPU to perform AI brightness and contrast processing based on the RGB image, and the corresponding brightness and contrast parameter adjustment result is output to the buffer 11. The global and local tone mapping submodule 13 adjusts the parameters used when processing the image based on the brightness and contrast parameter adjustment result.

[0123] In this embodiment, the original RAW image data is directly obtained based on the cache, and a communication channel is established with each module through a high-speed read / write interface. During image processing, the intermediate data generated by each module is transmitted based on the cache, and the AI ​​algorithm executed by the lightweight model is embedded at any position in the image processing flow. One or more corresponding AI algorithms are executed based on the intermediate data to complete the corresponding AI adjustments. This not only achieves the purpose of adapting to complex scenes, but also improves the efficiency and accuracy of image processing.

[0124] Based on the image processing system disclosed in the above embodiments of this application, this application also discloses an image processing method applicable to the above-disclosed image processing system, such as... Figure 7 As shown, the image processing method includes:

[0125] S701: Utilizes a buffer to receive raw RAW image data captured by a camera device in real time.

[0126] S702: When the image preprocessing module and the image processing module are performing corresponding processing, the core processing module reads the corresponding data generated by the image preprocessing module and / or the image processing module from the cache, controls the lightweight model to execute the AI ​​algorithm to perform image processing, and transmits the processed data to the cache.

[0127] S703: Based on the display processing module, the first image data is read from memory in real time for image display processing, and the second image data is output.

[0128] The corresponding processing performed in S703 includes:

[0129] The image processing module reads the raw RAW image data from the cache, performs preprocessing, and then transmits the preprocessed data to the cache and the image processing module.

[0130] The image processing module reads preprocessed data and / or processing data required for image processing from the cache, performs image processing, transmits the obtained YUV data to the cache, and transmits the obtained first image data to memory based on the cache.

[0131] In one embodiment of this application, in the case of a multi-channel image source, the global and local tone mapping submodules in the image processing module extract the display content in the input multi-channel image data, calculate the average brightness information based on the display content, and perform weight equalization based on the average brightness information to obtain the adjustment coefficients of each channel in the multi-channel, and adaptively process the corresponding multi-channel image data based on the adjustment coefficients.

[0132] In one embodiment of this application, the process of executing the above-described S702 core processing module to read the corresponding data generated by the image preprocessing module and / or image processing module from the cache, controlling the lightweight model to perform AI operations for image processing, and transmitting the obtained processed data to the cache includes:

[0133] The CPU determines the current image processing requirements, reads the statistical data generated by the image preprocessing module from the cache, and controls the NPU to perform the corresponding operations; the image processing requirements include any one or more of exposure parameter adjustment, noise reduction, brightness processing, contrast processing, and color processing.

[0134] In one embodiment of this application, controlling the NPU to perform corresponding operations includes: the CPU controlling the NPU to read data generated by the image preprocessing module and the image processing module from the cache, executing AI algorithms based on the image processing requirements in collaboration with a lightweight model to perform image processing, and transmitting the obtained first processing result to the cache.

[0135] In one embodiment of this application, if a DSP is present in the core processing module, the process of executing the above-described S702 core processing module reading the corresponding data generated by the image preprocessing module and / or image processing module from the cache, controlling the lightweight model to perform AI operations for image processing, and transmitting the obtained processed data to the cache includes:

[0136] The CPU determines the current image processing requirements, reads the statistical data generated by the image preprocessing module from the cache, and controls the NPU and DSP to perform corresponding operations; the image processing requirements include any one or more of exposure parameter adjustment, noise reduction, brightness processing, contrast processing, and color processing.

[0137] Controlling the NPU and DSP to perform corresponding operations, including:

[0138] The CPU controls the NPU to read data generated by the image preprocessing module and the image processing module from the cache, and performs AI algorithms in collaboration with the lightweight model to process the image based on the image processing requirements, and then transmits the first processing result to the cache.

[0139] The CPU controls the DSP to read data generated by the image preprocessing module and the image processing module from the cache, process the data, and then transmit the resulting second processing result back to the cache.

[0140] In the above-described corresponding processing, if the image processing module has the same structure as the disclosed image processing system, including a RAW image processing submodule and a noise reduction submodule, the process of the image processing module reading the original RAW image data from the cache for preprocessing and transmitting the preprocessed data to the cache and the image processing module specifically includes:

[0141] S801: The RAW image processing submodule reads the original RAW image data from the buffer and performs preprocessing, then transfers the resulting RAW image data to the buffer and noise reduction submodule.

[0142] S802: The noise reduction submodule divides the RAW image data into M×N regions, calculates and statistically analyzes the noise intensity of each region, and transmits the obtained RAW noise statistics data to the buffer and image processing module.

[0143] Here, M refers to the number of vertical divisions in the RAW image data, and N refers to the number of horizontal divisions in the RAW image data. The values ​​of M and N correspond to the smallest unit of AI processing.

[0144] Accordingly, the image processing module reads the preprocessed data from the cache, performs image processing, and transfers the resulting YUV data back to the cache, including:

[0145] The image processing module performs image processing based on RAW noise statistics and transmits the resulting YUV image data to the buffer.

[0146] In one embodiment of this application, in conjunction with the above description, when the image processing requirement is noise reduction, the noise reduction process includes:

[0147] S901: The RAW image processing submodule reads the raw RAW image data from the buffer and performs preprocessing, then transfers the resulting RAW image data to the buffer and noise reduction submodule.

[0148] S902: The noise reduction submodule 122 divides the RAW image data into M×N regions, calculates and statistically analyzes the noise intensity of each region, and transmits the obtained RAW noise statistics data to the buffer and image processing module.

[0149] S903: The image processing module performs image processing based on RAW noise statistics and transmits the resulting YUV image data to the buffer.

[0150] S904: The CPU reads RAW noise statistics from the cache, divides the noise level of each region of the RAW noise statistics according to the preset noise level, and controls the NPU and DSP to perform noise reduction collaborative processing.

[0151] S905: When performing noise reduction collaborative processing, the NPU reads RAW image data and YUV image data from the cache. Based on the noise level and lightweight model, it matches the corresponding AI noise processing to each region divided by the RAW image data and transmits the resulting third-processed data to the cache. Different noise levels correspond to different AI noise processing for different regions.

[0152] S906: When the DSP performs noise reduction co-processing, it reads RAW image data and YUV image data from the buffer, performs DSP processing according to the noise level, and transfers the resulting fourth processed data to the buffer.

[0153] S907: The image processing module reads the third and fourth processing data from the cache to perform image processing, and then transfers the resulting first image data to memory through the cache.

[0154] S909: The display processing module reads the first image data from memory in real time, performs image display processing, and outputs the second image data.

[0155] It should be noted that, in one embodiment of this application, when executing S904, if the system does not have a DSP, or even if it does, only the NPU can be controlled to perform noise reduction processing. Correspondingly, when executing S907, the image processing module reads the third processing data from the cache for image processing, and transfers the obtained first image data to memory through the cache.

[0156] During the noise reduction process using the aforementioned collaborative AI algorithm, exposure parameter adjustments can also be performed simultaneously. Different execution steps can be embedded after the same steps. It should be noted that exposure parameter adjustments can also be performed independently. The process of adjusting exposure parameters includes:

[0157] S1001: The RAW image processing submodule reads the raw RAW image data from the cache and performs preprocessing. It then transmits the obtained RAW image statistics and RAW image data to the cache and noise reduction submodule.

[0158] S1002: The noise reduction submodule divides the RAW image data into M×N regions, calculates and statistically analyzes the noise intensity of each region, and transmits the obtained RAW noise statistics data to the buffer and image processing module.

[0159] S1003: The color processing submodule in the image processing module performs image processing based on the RAW noise statistics and transmits the resulting YUV image data to the buffer.

[0160] S1004: The CPU reads the brightness statistics from the RAW image statistics from the cache and controls the NPU to perform collaborative processing to adjust the exposure parameters.

[0161] S1005: When the NPU performs collaborative processing for adjusting exposure parameters, it reads RAW image data and YUV image data from the cache, performs AI scene recognition based on brightness statistics and a lightweight model to obtain scene classification, and the exposure adjustment module combines the brightness statistics and the scene classification to generate new exposure parameters, and then transmits the obtained new exposure parameters to the cache.

[0162] In the specific execution of S1005, the lightweight model obtains the RGB image converted from the RAW image and the corresponding exposure parameters of the RGB image for scene recognition and scene classification. The exposure parameters include exposure time, analog gain and digital gain. The scene classification includes daytime normal scene, nighttime normal scene, nighttime direct headlight scene, nighttime flickering scene, extremely dark scene and tunnel scene.

[0163] The exposure adjustment module determines the target exposure parameters based on the actual brightness in the brightness statistics and the target brightness. It adjusts the exposure parameters according to the adjustment mode corresponding to different scene categories, compares the adjusted target exposure parameters with the original exposure parameters, and determines new exposure parameters based on the comparison results. The target exposure parameters include target exposure time, target analog gain, and target digital gain.

[0164] In one embodiment of this application, if the scene is classified as a nighttime flickering scene, the exposure adjustment module obtains the flickering frequency, adjusts the exposure parameters based on the flickering frequency, the scene information corresponding to the nighttime flickering scene, and the target exposure parameters, and obtains the adjusted target exposure parameters. Based on the comparison result between the adjusted target exposure parameters and the exposure parameters, the single-frame adjustment exposure parameters are calculated to obtain the new exposure parameters.

[0165] S1006: The RAW image processing submodule is used to feed back the new exposure parameters to the sensor, so that the sensor can adjust the exposure parameters based on the new exposure parameters and continue to acquire image data.

[0166] It should be noted that the principle of each module performing the corresponding processing is the same as the execution principle of the corresponding module in the image processing system disclosed in the above embodiments of this application, and will not be repeated here.

[0167] In this embodiment, the original RAW image data is directly obtained based on the cache, and a communication channel is established with each module through a high-speed read / write interface. During image processing, the intermediate data generated by each module is transmitted based on the cache, and the AI ​​algorithm executed by the lightweight model is embedded at any position in the image processing flow. One or more corresponding AI algorithms are executed based on the intermediate data to complete the corresponding AI adjustments. This not only achieves the purpose of adapting to complex scenes, but also improves the efficiency and accuracy of image processing.

[0168] Based on the image processing system and corresponding image processing method disclosed in the above embodiments of this application, this application also discloses an electronic device, such as... Figure 8 As shown, the electronic device includes at least one processing core 100, a deserializer 101 as input, and a serializer 102 as output. The processing core 100 is communicatively connected to the deserializer 101 and the serializer 102. The processing core 100 includes the image processing system disclosed in the aforementioned embodiments of this application. The image processing system is used to execute the image processing method disclosed in the aforementioned embodiments of this application.

[0169] In one embodiment of this application, the electronic device may be a controller, such as... Figure 9As shown, its input deserializer 101 is communicatively connected to the serializer of the camera device containing sensor 103, and its output serializer 102 is communicatively connected to the deserializer containing TFT display module 104.

[0170] Preferably, the electronic device is a controller that includes an electronic exterior rearview mirror CMS system that enables low-latency image transmission and display.

[0171] This application also discloses a storage medium storing a program that, when executed by a controller, implements the image processing method described above.

[0172] In the embodiments of this application, the storage medium may specifically be a computer storage medium. Computer storage media includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0173] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0174] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0175] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing system, characterized in that, The image processing system includes: The system includes a cache, an image preprocessing module, an image processing module, a core processing module, and memory connected to the cache via a high-speed read / write interface for data transmission, as well as a display processing module connected to the memory. The buffer is used to receive raw RAW image data acquired by the sensor in real time, as well as data generated by the image preprocessing module, image processing module and core processing module; The image preprocessing module is used to read the original RAW image data from the cache, preprocess it, and then transmit the preprocessed data to the cache. The core processing module is configured with a lightweight model for executing AI algorithms. When the image preprocessing module and the image processing module perform corresponding processing, the model reads the corresponding data generated by the image preprocessing module and / or the image processing module from the cache, coordinates with the lightweight model to execute AI algorithms for image processing, and transmits the processed data to the cache. The image processing module is used to read the preprocessed data and the processing data required for image processing from the cache, perform image processing, and transfer the obtained first image data to the memory based on the cache.

2. The system according to claim 1, characterized in that, The core processing module includes an NPU and a CPU; the lightweight model is configured in the NPU. The CPU is used to read statistical data generated by the image preprocessing module from the cache, and control the NPU to perform AI algorithms for image processing in conjunction with the lightweight model based on the currently determined image processing requirements; the image processing requirements include any one or more of exposure parameter adjustment, noise reduction, brightness processing, contrast processing, and color processing; The NPU is used to read data generated by the image preprocessing module and the image processing module from the cache during collaborative processing, and to perform AI algorithms in collaboration with the lightweight model to process the image based on the image processing requirements, and then transmit the first processing result to the cache.

3. The system according to claim 2, characterized in that, The image preprocessing module includes a RAW image processing submodule and a noise reduction submodule; The RAW image processing submodule is used to read the original RAW image data from the cache and perform preprocessing, and transmit the obtained RAW image statistics data and RAW image data to the cache and the noise reduction submodule. The noise reduction submodule is used to divide the RAW image data into M×N regions, calculate and statistically analyze the noise intensity of each region, and transmit the obtained RAW noise statistics data to the cache and the image processing module. M refers to the number of RAW image data segments in the vertical direction, and N refers to the number of RAW image data segments in the horizontal direction; the values ​​of M and N are matched with the smallest unit of AI processing.

4. The system according to claim 3, characterized in that, When the image processing requirement is noise reduction: The image processing module is used to perform image processing based on the RAW noise statistics and transmit the obtained YUV image data to the cache; The CPU is used to read the RAW noise statistics from the cache, classify the noise level of the RAW noise statistics according to the preset noise level, and control the NPU and the lightweight model to perform noise reduction collaborative processing. The NPU is used to read the RAW image data and the YUV image data from the cache when performing the noise reduction collaborative processing, and to match the corresponding AI noise processing for each region divided by the RAW image data based on the noise level and the lightweight model, and to transmit the obtained third processing data to the cache; wherein, regions with different noise levels correspond to different AI noise processing.

5. The system according to claim 2, characterized in that, The image processing requirement is for adjusting exposure parameters: The CPU is used to read the brightness statistics from the statistics in the cache and control the NPU to perform exposure parameter adjustment collaborative processing. The NPU is used to read the RAW image data and YUV image data from the cache when performing the exposure parameter adjustment collaborative processing, perform AI scene recognition based on the lightweight model to obtain scene classification, and generate new exposure parameters by the exposure adjustment module in combination with the brightness statistics and the scene classification, and transmit the obtained new exposure parameters to the cache. The RAW image processing submodule is also used to feed the new exposure parameters back to the sensor, so that the sensor adjusts the exposure parameters based on the new exposure parameters and continues to acquire image data.

6. The system according to claim 5, characterized in that, The lightweight model that executes the AI ​​algorithm is used to acquire the RGB image converted from the RAW image, the exposure parameters corresponding to the RGB image, and to perform scene recognition and determine scene classification. The exposure parameters include exposure time, analog gain, and digital gain. The scene classification includes daytime normal scene, nighttime normal scene, nighttime direct headlight scene, nighttime flickering scene, extremely dark scene, and tunnel scene. The exposure adjustment module is used to determine the target exposure parameters based on the actual brightness in the brightness statistics and the target brightness, adjust the exposure parameters according to the adjustment mode corresponding to different scene classifications, compare the adjusted target exposure parameters with the exposure parameters, and determine new exposure parameters based on the comparison result; the target exposure parameters include target exposure time, target analog gain, and target digital gain; If the scene is classified as a nighttime flickering scene, the exposure adjustment module is used to obtain the flickering frequency, adjust the exposure parameters based on the flickering frequency, the scene information corresponding to the nighttime flickering scene, and the target exposure parameters to obtain the adjusted target exposure parameters, and calculate the single-frame adjustment exposure parameters based on the comparison result between the adjusted target exposure parameters and the exposure parameters to obtain the new exposure parameters.

7. The system according to any one of claims 1 to 6, characterized in that, The image processing module includes global and local tone mapping sub-modules: The global and local tone mapping submodule is used to extract the display content from the input multi-channel image data in the case of a multi-channel image source, calculate the average brightness information based on the display content, perform weight equalization based on the average brightness information to obtain the adjustment coefficient of each channel in the multi-channel, and adaptively process the corresponding multi-channel image data based on the adjustment coefficient.

8. The system according to any one of claims 1 to 6, characterized in that, The cache includes static random access memory (SRAM) or resistive random access memory (RRAM).

9. An image processing method, characterized in that, The image processing system applicable to any one of claims 1 to 8, the image processing method comprising: Utilize buffering to receive raw RAW image data acquired by the sensor in real time; While the image preprocessing module and the image processing module are performing corresponding processing, the core processing module reads the corresponding data generated by the image preprocessing module and / or the image processing module from the cache, controls the lightweight model to execute the AI ​​algorithm to perform image processing, and transmits the processed data to the cache. The display processing module reads the first image data from memory in real time, performs image display processing, and outputs the second image data. The corresponding processing includes: The image preprocessing module reads the original RAW image data from the cache, performs preprocessing, and transmits the preprocessed data to the cache and the image processing module. The image processing module reads the preprocessed data and / or the processing data required for image processing from the cache, performs image processing, transmits the obtained YUV data to the cache, and transmits the obtained first image data to the memory based on the cache.

10. The method according to claim 9, characterized in that, The core processing module reads the corresponding data generated by the image preprocessing module and / or the image processing module from the cache, controls the lightweight model to execute AI algorithms for image processing, and transmits the processed data to the cache, including: The CPU determines the current image processing requirements, reads the statistical data generated by the image preprocessing module from the cache, and controls the NPU to perform corresponding operations; the image processing requirements include any one or more of exposure parameter adjustment, noise reduction, brightness processing, contrast processing, and color processing. The control of the NPU to perform corresponding operations includes: The CPU controls the NPU to read the data generated by the image preprocessing module and the image processing module from the cache, and, based on the image processing requirements, coordinates with the lightweight model to execute AI algorithms to perform image processing, and transmits the first processing result to the cache.

11. The method according to claim 10, characterized in that, If the image processing requirement is exposure parameter adjustment, the CPU reads the statistical data generated by the image preprocessing module from the cache, and controls the NPU and the lightweight model to execute AI algorithms for image processing, including: The CPU reads the RAW image data from the cache and obtains the brightness statistics data in the RAW image data; The CPU controls the NPU to read the RAW image data and the YUV image data from the cache, performs AI scene recognition based on the brightness statistics and the lightweight model to obtain scene classification, and the exposure adjustment module combines the brightness statistics and the scene classification to generate new exposure parameters, and transmits the obtained new exposure parameters to the cache; the RAW image processing submodule feeds the new exposure parameters back to the sensor, so that the sensor adjusts the exposure parameters based on the new exposure parameters and continues to acquire image data.

12. The method according to claim 11, characterized in that, The AI ​​scene recognition based on the brightness statistics and the lightweight model yields a scene classification. The exposure adjustment module then combines the brightness statistics and the scene classification to generate new exposure parameters, including: The lightweight model acquires the RGB image converted from the RAW image and the exposure parameters corresponding to the RGB image to perform scene recognition and determine the scene category; the exposure parameters include exposure time, analog gain and digital gain; the scene categories include daytime normal scene, nighttime normal scene, nighttime direct headlight scene, nighttime flickering scene, extremely dark scene and tunnel scene. The exposure adjustment module determines the target exposure parameters based on the actual brightness in the brightness statistics and the target brightness. It adjusts the exposure parameters according to the adjustment mode corresponding to different scene categories, compares the adjusted target exposure parameters with the original exposure parameters, and determines new exposure parameters based on the comparison results. The target exposure parameters include target exposure time, target analog gain, and target digital gain. If the scene is classified as a nighttime flickering scene, the exposure adjustment module obtains the flickering frequency, adjusts the exposure parameters based on the flickering frequency, the scene information corresponding to the nighttime flickering scene, and the target exposure parameters to obtain the adjusted target exposure parameters, and calculates the single-frame adjustment exposure parameters based on the comparison result between the adjusted target exposure parameters and the exposure parameters to obtain the new exposure parameters.

13. The method according to any one of claims 9 to 12, characterized in that, In the case of a multi-channel image source, the global and local tone mapping sub-modules in the image processing module extract the display content from the input multi-channel image data, calculate the average brightness information based on the display content, and perform weight equalization based on the average brightness information to obtain the adjustment coefficients of each channel in the multi-channel image. Based on the adjustment coefficients, the corresponding multi-channel image data is adaptively processed.

14. An electronic device, characterized in that, The electronic device includes at least one processing core, a deserializer as input, and a serializer as output, the processing core being communicatively connected to the deserializer and the serializer, the processing core including the image processing system of any one of claims 1 to 8, the image processing system being used to execute the image processing method of any one of claims 9 to 13.

15. A storage medium, characterized in that, It stores a program that, when executed by the controller, implements the image processing method as described in any one of claims 9 to 13.

Images