Graphics processor, graphics processor operation method and electronic device

Abstract

This application discloses a graphics processor, a method for operating the graphics processor, and an electronic device, relating to the field of computers. This application utilizes a DMA module to directly transfer data between SRAM and the configuration registers / memory of the ISP, eliminating the need for frequent CPU intervention, simplifying the ISP configuration process, and improving configuration efficiency. The SRAM maps a region of the same size as the ISP configuration registers / memory, and through direct read / write operations of DMA, it achieves rapid updates and retrieval of ISP configuration and information. Compared to directly using multiple registers and memories, this mapping method reduces the consumption of hardware resources, making the area of ​​the entire SRAM smaller than the total area of ​​using registers and memories individually.

Description

Technical Field

[0001] This application relates to the field of computers, and more particularly to a graphics processor, a method for operating the graphics processor, and an electronic device. Background Technology

[0002] In current image processing technology, the configuration of the Image Signal Processor (ISP) is achieved through a mechanism called the Shadow Register / MEM. The core of this mechanism lies in the precise correspondence between the Shadow Register / MEM and the configuration / information registers and configuration / information MEM within the ISP. The specific operation process is as follows: Before the ISP officially starts, the software system uses the APB bus to perform detailed configuration of the Shadow Register / MEM. Once the configuration is successfully completed, the ISP is activated. At this point, the ISP automatically reads the configuration information from the Shadow Register / MEM and loads it into the Configuration Register / MEM, thus beginning the image processing task.

[0003] After the ISP successfully processes the current frame's image, it exports the processing results and related information from the Information Register / MEM and writes them back to the Shadow Register / MEM for subsequent reading and analysis by the software system. Simultaneously, the ISP prepares for the processing of the next frame's image; it reads the configuration information for the next frame's image from the Shadow Register / MEM and loads it into the Configuration Register / MEM. It's worth noting that in the ISP's workflow, the Configuration Register / MEM plays a write-only (wr) role, while the Information Register / MEM is responsible for read-only (rd) tasks.

[0004] However, this existing technology also has certain limitations. First, from a hardware resource perspective, due to the large number of ISP configuration and information registers / MEMs, adopting a shadow register / MEM configuration method will inevitably lead to a significant increase in hardware overhead. Specifically, the introduction of shadow registers / MEMs requires twice the hardware resources to support them. Second, from a CPU resource utilization perspective, since the shadow registers / MEMs need to be reconfigured via the APB bus for each frame of image, this will lead to frequent CPU resource consumption and potential performance bottlenecks.

[0005] In conclusion, while the current shadow register / MEM configuration method has played a role in the ISP configuration process, its drawbacks, such as high hardware overhead and high CPU resource consumption, cannot be ignored. Therefore, it is necessary to further optimize and improve this configuration method to enhance the efficiency and performance of the ISP. Summary of the Invention

[0006] This application provides a graphics processor, a method for operating the graphics processor, and an electronic device, which can solve the problems of high hardware overhead and low efficiency of graphics processors in the prior art. The technical solution is as follows:

[0007] In a first aspect, embodiments of this application provide a graphics processor, including:

[0008] Selector, DMA module, FLASH memory, and algorithm module;

[0009] The FLASH memory includes a configuration register, an information register, a configuration memory, and an information memory. The configuration register and the configuration memory are set to write-only mode, and the information register and the information memory are set to read-only mode.

[0010] The image processor is externally equipped with SRAM, which contains a first storage area, a second storage area, a third storage area, and a fourth storage area.

[0011] The first input terminal of the selector is connected to the APB bus outside the image processor, the second input terminal of the selector is connected to the DMA module, the output terminal of the selector is connected to the first terminal of the FLASH memory, the second terminal of the FLASH memory is connected to the ISP algorithm module, and the control terminal of the selector is connected to the controller in the graphics processor.

[0012] The operation of the graphics processor includes:

[0013] The controller instructs the selector to turn on the first input and output terminals through the control terminal of the selector, then receives configuration information from the external APB bus, and uses the received configuration information to configure the configuration register, enabling the DMA module and configuring the base address of the DMA module.

[0014] When the controller receives an external vertical synchronization signal, it instructs the selector to turn on the second input and output terminals, instructs the DMA module to write the information in the first storage area into the configuration register, read the data in the information register into the second storage area, write the data in the third storage area into the configuration memory, and write the data in the information memory into the fourth storage area.

[0015] After the controller detects that the DMA module has completed the read / write operation, it instructs the selector to turn on the first input and output terminals.

[0016] Secondly, embodiments of this application provide a method for operating an image processor, including:

[0017] The controller instructs the selector to turn on the first input and output terminals through the control terminal of the selector, then receives configuration information from the external APB bus, and uses the received configuration information to configure the configuration register, enabling the DMA module and configuring the base address of the DMA module.

[0018] When the controller receives an external vertical synchronization signal, it instructs the selector to turn on the second input and output terminals, instructs the DMA module to write the information in the first storage area into the configuration register, read the data in the information register into the second storage area, write the data in the third storage area into the configuration memory, and write the data in the information memory into the fourth storage area.

[0019] After the controller detects that the DMA module has completed the read / write operation, it instructs the selector to turn on the first input and output terminals.

[0020] The beneficial effects of the technical solutions provided in some embodiments of this application include at least the following:

[0021] By utilizing a DMA module to directly transfer data between SRAM and the ISP's configuration registers / memory, the configuration process of the ISP is simplified and its efficiency improved, eliminating the need for frequent CPU intervention. The SRAM maps a region of the same size as the ISP configuration registers / memory, enabling rapid updates and reads of ISP configuration and information through direct DMA read / write operations. Compared to using multiple registers and memories, this mapping method reduces hardware resource consumption, resulting in a smaller SRAM area compared to using registers and memory individually. The SRAM is connected to the graphics processor via the AHB bus, which offers higher data transfer efficiency and bandwidth than the APB bus, thus accelerating the read / write speed of ISP configuration information and data. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the structure of an image processor in the prior art;

[0024] Figure 2 This is a schematic diagram of the structure of the image processor provided in an embodiment of this application;

[0025] Figure 3 This is a mapping diagram of SRAM and FLASH memory in the image processor provided in the embodiments of this application;

[0026] Figure 4 This is a flowchart illustrating the operation method of the image processor provided in an embodiment of this application;

[0027] Figure 5 This is a timing diagram of a graphics processor operating according to the present application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0029] See Figure 1 and Figure 2 This is a schematic diagram of a graphics processor provided in an embodiment of this application, including: a selector, a DMA module, a FLASH memory, and an algorithm module;

[0030] The FLASH memory includes a configuration register, an information register, a configuration memory, and an information memory. The configuration register and configuration memory are set to write-only mode, while the information register and information memory are set to read-only mode. An external SRAM is provided for the image processor. The SRAM has a first storage area sram_0, a second storage area sram_1, a third storage area sram_2, and a fourth storage area 3. The first input terminal of the selector is connected to the APB bus outside the image processor, the second input terminal of the selector is connected to the DMA module, the output terminal of the selector is connected to the first terminal of the FLASH memory, the second terminal of the FLASH memory is connected to the ISP algorithm module, and the control terminal of the selector is connected to the controller in the graphics processor.

[0031] The selector is responsible for selecting the signal path. Based on the controller's instructions, the selector can switch different input sources to the output. The first input is connected to the APB (Advanced Peripheral Bus) to receive configuration information from external sources. The second input is connected to the DMA module to receive data transferred by the DMA module. The output is connected to the first end of the FLASH memory to send the selected data or configuration information to the FLASH memory. The control terminal is connected to the controller and receives instructions from the controller to switch input sources.

[0032] The DMA module enables direct data transfer between memory and peripherals without CPU intervention, thus improving data transfer efficiency. It reads data from SRAM and writes it to FLASH memory or different memory areas within SRAM. Configuration is handled by the controller via configuration registers, including enabling the DMA module and setting its base address.

[0033] FLASH memory is a non-volatile memory used to store configuration information, image data, etc.

[0034] The configuration register is set to write-only mode to store the DMA module's configuration information and other necessary configuration parameters.

[0035] The information register is set to read-only mode to store information about the FLASH memory or other components, such as version information and status information.

[0036] Configure the memory to write-only mode to store the image processor's operating configuration or parameters.

[0037] The information storage device is set to read-only mode to store image data or processing results, etc.

[0038] The ISP algorithm module is used to process image data, such as denoising, sharpening, and color correction. It receives the configuration information and image data required for processing from the FLASH memory, as well as the processed image data, which can be stored back to the FLASH memory or transferred to other components.

[0039] In the embodiments of this application, the ISP algorithm module includes the following algorithm sub-modules: TPG test generator, BLC black level correction module, DPC bad pixel correction module, LSC lens shading correction module, BNRbayer noise reduction module, AWB automatic white balance module, AE automatic exposure module, CFA color interpolation module, GAMMA correction module, CCM color correction module, CSC color space conversion module, and SHARP&CE sharpening and color enhancement module.

[0040] The TPG test generator is used to generate test images to verify the performance and accuracy of the ISP algorithm module. It is an indispensable tool during the research, development, debugging, and testing phases of the ISP algorithm module.

[0041] The BLC black level correction module shifts the overall signal output from the image sensor vertically to eliminate the influence of dark current on the image signal. By subtracting a fixed black level value, the image data is made closer to the true brightness distribution.

[0042] The DPC (Device Processing and Correction) module detects and corrects bad pixels in image sensors to improve image quality. It identifies bad pixels by comparing them to their surrounding pixels and then replaces or repairs them.

[0043] The LSC lens shading correction module corrects image brightness unevenness caused by lens optical characteristics. It adjusts the overall or local brightness of the image by calculating brightness compensation values ​​for different areas.

[0044] The BNR Bayer noise reduction module reduces noise in images and improves image clarity. It uses low-pass filters (such as mean filtering, Gaussian filtering, etc.) to smooth the image, thereby eliminating noise.

[0045] The AWB (Auto White Balance) module automatically adjusts image colors to ensure that white objects appear white under any lighting conditions. It estimates the color temperature of the light source by analyzing the color distribution in the image, particularly in gray and white areas, and adjusts the intensity of the red, green, and blue channels accordingly.

[0046] The AE (Auto Exposure) module automatically determines the correct exposure to control image brightness. By measuring the light distribution in the image, it calculates the ideal exposure settings (such as aperture size, shutter speed, and ISO value).

[0047] The CFA color interpolation module converts raw Bayer format image data into a full-color image. It calculates the color value of each pixel using interpolation algorithms such as bilinear interpolation and bicubic interpolation.

[0048] The GAMMA correction module performs non-linear adjustments to the brightness of an image to conform to the visual characteristics of the human eye. It maps input values ​​within different brightness ranges to corresponding output values ​​using lookup tables or formulas.

[0049] The CCM color correction module corrects color errors caused by color bleeding between color blocks at the filter plate. It compares a standard image with the image obtained from the sensor to derive a correction matrix, which is then applied to correct the image's color.

[0050] The CSC color space conversion module converts images from one color space to another. It processes image data using color space conversion formulas (such as RGB to YUV conversion).

[0051] SHARP&CE's sharpening and color enhancement modules improve image clarity and detail while enhancing color saturation and contrast. Sharpening and color enhancement are achieved through algorithms that amplify edge or directional information with brightness differences between adjacent areas and adjust the intensity of color channels.

[0052] SRAM stands for Static Random Access Memory, used to temporarily store data processed by an image processor. It includes a first storage area, a second storage area, a third storage area, and a fourth storage area, used for different data transfer and storage tasks.

[0053] See Figure 4 The diagram showing the operation flow of the image processor includes:

[0054] S401, the controller instructs the selector to turn on the first input and output terminals through the control terminal of the selector, then receives configuration information from the external APB bus, and uses the received configuration information to configure the configuration register, enable the DMA module and configure the base address of the DMA module.

[0055] The controller connects the APB bus to the configuration register of the FLASH memory via a selector. Configuration information from the APB bus is transferred to the configuration register. The controller uses this configuration information to configure the DMA module, including enabling the DMA module and setting its base address.

[0056] S402. When the controller receives an external vertical synchronization signal, it instructs the selector to turn on the second input and output terminals, instructs the DMA module to write the information in the first storage area into the configuration register, read the data in the information register into the second storage area, write the data in the third storage area into the configuration memory, and write the data in the information memory into the fourth storage area.

[0057] The controller receives a vertical synchronization signal, marking the start of a new frame of image data. The controller connects the DMA module to the FLASH memory via a selector.

[0058] The DMA module begins operation, performing the following steps: It writes data (e.g., new configuration information) from the first SRAM memory area to the FLASH memory configuration register. It reads data from the FLASH memory information register and stores it in the second SRAM memory area for subsequent processing or transfer. It writes data (e.g., image processor operating configuration or parameters) from the third SRAM memory area to the FLASH memory configuration memory. Finally, it writes data (e.g., image data or processing results) from the FLASH memory information memory to the fourth SRAM memory area as a backup or data source for subsequent processing.

[0059] In the embodiments of this application, the configuration register is used to store: enable information of each algorithm submodule, correction threshold of BLC, correction matrix configuration of CCM, and sharpening degree / color saturation / color contrast of SHARP&CE. The information register is used to store indications of whether each algorithm submodule has completed its operation, indications of whether there are errors, the number of white points in AWB, and the average brightness of the area in AE. The configuration memory is used to store: location information of bad pixels in DPC and correction coefficients for each partition of LSC. The information memory is used to store: histogram information and the number of pixels with y-values ​​ranging from 0 to 255.

[0060] In an image processor, registers are crucial components for storing configuration information, status information, and processing results. Based on your provided technical solution, the configuration register, information register, configuration memory, and information memory each perform different functions. The following is a detailed breakdown and explanation of these register functions:

[0061] The configuration register is primarily used to store configuration information for each algorithm submodule of the image processor. This configuration information determines how the image processor processes image data. Specifically, it includes:

[0062] Enable information for each algorithm submodule: This determines which algorithm submodules (such as denoising, sharpening, color correction, etc.) will be enabled. By configuring this information, you can flexibly select the image processing algorithms to apply to meet different image processing needs.

[0063] Black Level Correction (BLC) correction threshold: Black Level Correction is used to adjust the black level of image data to eliminate black offsets in the image. The correction threshold defines which pixel values ​​should be considered black and corrected accordingly.

[0064] The CCM (Color Correction Matrix) configuration: The color correction matrix is ​​used to adjust the colors of an image, making them more accurate and realistic. The configuration of the correction matrix defines the mapping relationship from the original color space to the target color space.

[0065] SHARP & CE (Sharpening & Color Enhancement) sharpening level / color saturation / color contrast: Used to adjust the sharpness, color saturation, and color contrast of an image to improve its visual quality and clarity.

[0066] The information register is primarily used to store the status information of various modules in the image processor. This status information reflects the current working status and processing results of the image processor. Specifically, it includes:

[0067] Indicator information indicating whether each algorithm submodule has completed its operation: This indicates whether each algorithm submodule has finished processing the image data. By reading this information, one can understand the processing progress and status of the image processor.

[0068] Error message: This indicates whether any errors or anomalies occurred during processing. By reading this information, errors can be detected and addressed promptly, ensuring the normal operation of the image processor.

[0069] AWB (Automatic White Balance) white point count: Automatic white balance is used to adjust the white balance of an image to make it more natural and realistic. The white point count indicates the number of white points detected during the automatic white balance process, which helps determine the degree of white balance adjustment.

[0070] AE (Automatic Exposure) Area Average Brightness: Automatic exposure is used to adjust the exposure of an image, making it brighter and clearer. Area average brightness represents the average brightness value of each area during the automatic exposure process, helping to determine the degree of exposure adjustment.

[0071] Configuration memory is primarily used to store long-term configuration information for the image processor, which remains valid even after the image processor restarts. Specifically, it includes:

[0072] Dead Pixel Correction (DPC) bad pixel location information: Dead pixel correction is used to eliminate bad pixels in an image and improve image quality. Bad pixel location information records the location of bad pixels in the image so that correction can be performed during image processing.

[0073] The correction factor for each zone in LSC (Lens Shading Correction): Lens shading correction is used to eliminate uneven image brightness caused by lens shading effects. The correction factor for each zone defines the degree of brightness correction required for different areas.

[0074] The information storage device is mainly used to store temporary state information and processing results of the image processor. This information is read and used during or after image processing. Specifically, it includes:

[0075] Histogram information: Histograms are used to represent the brightness or color distribution of image data. By reading histogram information, we can understand the statistical characteristics of the image's brightness and color distribution, which is helpful for further image processing and analysis.

[0076] The number of pixels with y-values ​​ranging from 0 to 255: This indicates the number of pixels corresponding to each brightness value (0 to 255) in the image. By reading this information, one can understand the brightness distribution of the image and the statistical characteristics of the pixel values.

[0077] S403. After the controller detects that the DMA module has completed the read / write operation, it instructs the selector to turn on the first input and output terminals.

[0078] After completing the read / write operation, the DMA module sends a completion signal to the controller. Upon detecting this signal, the controller reconnects the APB bus to the FLASH memory's configuration register via a selector. The system then returns to its initial state, awaiting the arrival of the next frame of image data and the reception of new configuration information.

[0079] Please see Figure 5 , Figure 5 This is a timing diagram of the image processor. A high level for VSYNC (vertical synchronization signal) indicates a valid image frame, and a low level indicates an invalid image frame. A high level for HREF indicates a valid image line, and a low level indicates an invalid image line. Time T1 represents the time interval from the valid image frame to the valid first image line of image frame 1, and time T2 represents the time interval from the valid first image line of image frame 1 to the invalidation of the image frame. Time T3 represents the image frame interval. Time T4 is the same as time T1, and time T4 is the same as time T2, belonging to the next frame, frame 2. Generally, time T1 is relatively long, on the order of milliseconds. The DMA module is triggered when the rising edge of VSYNC arrives, and then performs related operations. After all operations are completed, a completion interrupt is generated. The software can then read and write sram_1 and sram_3 through the controller in the graphics processor within time T2 to perform related calculations and prepare the configuration information for the next frame, such as white balance, exposure, and gamma.

[0080] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0081] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A graphics processor, characterized in that, include: Selector, DMA module, FLASH memory, and algorithm module; The FLASH memory includes a configuration register, an information register, a configuration memory, and an information memory. The configuration register and the configuration memory are set to write-only mode, and the information register and the information memory are set to read-only mode. The graphics processor is externally equipped with SRAM, which contains a first storage area, a second storage area, a third storage area, and a fourth storage area. The first input terminal of the selector is connected to the APB bus outside the graphics processor, the second input terminal of the selector is connected to the DMA module, the output terminal of the selector is connected to the first terminal of the FLASH memory, the second terminal of the FLASH memory is connected to the ISP algorithm module, and the control terminal of the selector is connected to the controller in the graphics processor. The operation of the graphics processor includes: The controller instructs the selector to turn on the first input and output terminals through the control terminal of the selector, then receives configuration information from the external APB bus, and uses the received configuration information to configure the configuration register, enabling the DMA module and configuring the base address of the DMA module. When the controller receives an external vertical synchronization signal, it instructs the selector to turn on the second input and output terminals, instructs the DMA module to write the information in the first storage area into the configuration register, read the data in the information register into the second storage area, write the data in the third storage area into the configuration memory, and write the data in the information memory into the fourth storage area. After the controller detects that the DMA module has completed the read / write operation, it instructs the selector to turn on the first input and output terminals.

2. The graphics processor according to claim 1, characterized in that, The ISP algorithm module includes the following algorithm sub-modules: TPG test generator, BLC black level correction module, DPC bad pixel correction module, LSC lens shading correction module, BNR bayer noise reduction module, AWB automatic white balance module, AE automatic exposure module, CFA color interpolation module, GAMMA correction module, CCM color correction module, CSC color space conversion module, and SHARP&CE sharpening and color enhancement module.

3. The graphics processor according to claim 1 or 2, characterized in that, The DMA module transmits data via the AHB bus.

4. The graphics processor according to claim 3, characterized in that, Two memory locations that have a mapping relationship have the same address.

5. The graphics processor according to claim 1, 2, or 4, characterized in that, The configuration register is used to store: the enable information of each algorithm submodule, the correction threshold of BLC, the correction matrix configuration of CCM, and the sharpening degree / color saturation / color contrast of SHARP&CE. The information register is used to store: indication information of whether each algorithm submodule has completed its operation, indication information of whether there is an error, the number of white dots in AWB, and the average brightness of the AE area; The configuration memory is used to store: the location information of DPC bad pixels and the correction coefficients of each LSC partition; The information storage device is used to store: histogram information and the number of pixels with y-values ​​ranging from 0 to 255.

6. A method for operating a graphics processor, characterized in that, The operating method is applied to the graphics processor as described in claim 1, and the operating method includes: The controller instructs the selector to turn on the first input and output terminals through the control terminal of the selector, then receives configuration information from the external APB bus, and uses the received configuration information to configure the configuration register, enabling the DMA module and configuring the base address of the DMA module. When the controller receives an external vertical synchronization signal, it instructs the selector to turn on the second input and output terminals, instructs the DMA module to write the information in the first storage area into the configuration register, read the data in the information register into the second storage area, write the data in the third storage area into the configuration memory, and write the data in the information memory into the fourth storage area. After the controller detects that the DMA module has completed the read / write operation, it instructs the selector to turn on the first input and output terminals.

7. An electronic device, characterized in that, include: SRAM and the graphics processor as described in any one of claims 1 to 5.

Images