Lens distortion correction method, electronic device and electronic equipment

Abstract

The invention discloses a lens distortion correction method, an electronic device and electronic equipment, relates to the technical field of image processing, and is used for improving the processing efficiency of a chip in an image distortion correction process. In the embodiment of the invention, the component for executing the lens distortion correction processing and the component of the ISP are decoupled, the first processing unit (namely the component for executing the lens distortion correction processing) performs the lens distortion correction processing on the image of the RAW domain output by the camera device, and the corrected image is not written into the memory any more. However, the second processing unit directly acquires the corrected target image of the RAW domain from the first processing unit 110, so that the requirements of ISP processing and lens distortion correction processing on processing resources such as memory bandwidth and the like can be effectively reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a lens distortion correction method, an electronic device and an electronic device. Background Art

[0002] Currently, wide-angle lenses or fisheye lenses have been widely used in computing visual perception systems such as terminals, video security, and autonomous driving. Due to the inevitable optical aberration of the lens, the image collected by the lens will be distorted in different ways.

[0003] Lens distortion correction (LDC) is a technical means to calibrate and correct the distortion of the optical lens itself by using image processing methods, thereby eliminating image deformation and distortion. In current image signal processing (ISP) systems, the commonly used distortion correction method is to perform distortion correction processing on images in the RGB domain or YUV domain. Before performing lens distortion correction processing, the RGB domain or YUV domain image after ISP pre-processing needs to be cached in DDR (double data rate) memory, resulting in the processing efficiency of the system on chip (SoC) being limited by the access bandwidth of DDR. Summary of the invention

[0004] The embodiments of the present application provide a lens distortion correction method, an electronic device, and an electronic device, which are used to improve the processing efficiency of a chip during the distortion correction of an image.

[0005] In a first aspect, an embodiment of the present application provides an electronic device, which includes a first processing unit and a second processing unit. The first processing unit is used to obtain a RAW source image to be processed, and perform lens distortion correction processing on the RAW source image to obtain a RAW target image; the RAW source image is a distorted image, and the RAW target image is a non-distorted image; the second processing unit is used to obtain the RAW target image from the first processing unit, and perform image signal processing ISP on the RAW target image.

[0006] In the solution provided in the embodiment of the present application, distortion processing is performed on the RAW image, and lens distortion correction processing is no longer performed on the RGB image or the YUV image, that is, distortion processing is not performed on the image during the ISP processing, so that the first processing unit may no longer cache the RAW image after the distortion correction processing in a memory outside the processing unit, such as DDR, that is, there is no need to cache the processed data in the DDR during the lens distortion correction processing and the ISP processing, so that the processing efficiency of the ISP+lens distortion correction processing is no longer limited by the access bandwidth of the memory, and the processing efficiency can be improved.

[0007] In a possible implementation, the electronic device further includes a memory; the memory is used to store the RAW source image to be processed; and the first processing unit is specifically used to read the RAW source image from the memory.

[0008] In a possible implementation, the memory, the first processing unit, and the second processing unit are integrated into a same system-on-chip (SoC) in the electronic device. Exemplarily, the memory may be DDR.

[0009] In a possible implementation, the first processing unit includes an output buffer, and the first processing unit stores the RAW target image obtained by lens distortion correction processing in the output buffer. The second processing unit reads the RAW target image from the output buffer of the first processing unit.

[0010] In a possible implementation, the first processing unit and the second processing unit may be integrated into the same logic device or into different logic devices. For example, the first processing unit may be a separate component or integrated into other logic devices, and the logic devices include but are not limited to: CPU, GPU, DSP, NPU. The second processing unit may also be a separate component or integrated into other logic devices. The logic devices include but are not limited to: dedicated neural processors such as CPU, GPU, DSP, NPU. For example, the first processing unit, the second processing unit and the memory are integrated into the SoC, the first processing unit may be understood as a subsystem in the SoC, and the second processing unit may be understood as another subsystem in the SoC. Exemplarily, the CPU, GPU, DSP, and NPU may all be processors in the system on chip. In some embodiments, the first processing unit and the second processing unit are integrated into the same logic device, and the first processing unit and the second processing unit share the same cache in the logic device, wherein a space of the same cache is used as an output cache. The first processing unit stores the processed RAW target image (or the target sub-block of the RAW target image) in the output cache. The second processing unit reads the RAW target image from the output cache for subsequent ISP.

[0011] In a possible implementation, the first processing unit is specifically used to divide the RAW source image into blocks to obtain multiple RAW source sub-blocks, and perform lens distortion correction processing on each of the multiple RAW source sub-blocks to obtain multiple RAW target sub-blocks included in the RAW target image; the second processing unit is specifically used to perform image signal processing ISP on the multiple RAW target sub-blocks.

[0012] In the above implementation of the present application, when the RAW source image is distorted, it is performed in blocks, which can reduce the processing delay caused by the processing of the entire image.

[0013] In a possible implementation, when the first processing unit performs lens distortion correction processing on the RAW source image, inter-channel interpolation processing may be performed on multiple color channels included in the RAW source image in the horizontal direction and the vertical direction.

[0014] In a possible implementation, the first processing unit, when performing lens distortion correction processing on each of the multiple RAW source sub-blocks, is specifically used to: perform inter-channel interpolation processing on multiple color channels included in the first RAW source sub-block in the horizontal direction and the vertical direction; wherein the first RAW source sub-block is any RAW source sub-block among the multiple RAW source sub-blocks.

[0015] Through the above implementation, the present application performs inter-channel interpolation processing on the RAW source sub-block in the horizontal direction and the vertical direction, which can improve the processing effect of lens distortion and achieve or even exceed the processing effect of performing lens distortion correction in the RGB domain.

[0016] In a possible implementation, the memory further stores a coordinate mapping table, wherein the coordinate mapping table records a mapping relationship between pixel coordinates of the RAW source image and pixel coordinates of the RAW target image; the first processing unit is specifically used to: for a first RAW target sub-block in the RAW target image, determine, according to the coordinate mapping table, an area of ​​a first RAW source sub-block mapped by the first RAW target sub-block in the RAW source image, wherein the first RAW target sub-block is any RAW target sub-block in the RAW target image.

[0017] In the above implementation of the present application, a predetermined coordinate mapping table is used to determine the blocks of the RAW source image, and there is no need to determine the relationship between the RAW source image and the RAW target image in real time, which can improve processing efficiency and reduce processing delay.

[0018] In a possible implementation manner, the first processing unit is specifically configured to:

[0019] According to the arrangement order of the multiple RAW target sub-blocks, the area of ​​the i-th RAW source sub-block mapped by the i-th RAW target sub-block in the RAW source image is determined according to the coordinate mapping table; and according to the area of ​​the i-th RAW source sub-block in the RAW source image, the i-th RAW source sub-block is read from the memory to perform lens distortion correction processing on the i-th RAW source sub-block; or,

[0020] According to the arrangement order of the multiple RAW target sub-blocks, determine the area of ​​the i-th RAW source sub-block mapped by the i-th RAW target sub-block in the RAW source image according to the lookup table; and read the i-th RAW source sub-block from the memory according to the area of ​​the i-th RAW source sub-block in the RAW source image, so as to perform lens distortion correction processing on the i-th RAW source sub-block; the lookup table includes the starting point and size of the RAW source sub-blocks corresponding to the multiple RAW target sub-blocks in the RAW source image; the lookup table is generated according to the coordinate mapping table; or,

[0021] According to the coordinate mapping table, the area of ​​the RAW source sub-block to which each RAW target sub-block in the multiple RAW target sub-blocks is respectively mapped in the RAW source image is determined, and in the arrangement order of the multiple RAW target sub-blocks, the i-th RAW source sub-block is read from the memory according to the area of ​​the i-th RAW source sub-block to which the i-th RAW target sub-block is mapped in the RAW source image, so as to perform lens distortion correction processing on the i-th RAW source sub-block.

[0022] In a possible implementation, the first processing unit includes a ping-pong cache consisting of a first cache and a second cache; the storage rates of the first cache and the second cache are both higher than the memory; the ping-pong cache is used to sequentially store the i-th RAW source sub-block and the i+1-th RAW source sub-block read from the memory by the first processing unit in a ping-pong storage manner.

[0023] In the above implementation of the present application, the ping-pong cache method is adopted to solve the memory limitation between sub-blocks. The storage rate of the first cache and the second cache is higher than that of the memory, which can improve the processing efficiency.

[0024] In some embodiments, the size of the first cache and the second cache is related to the size of the blocks of the RAW source image. For example, when the RAW source image is divided into blocks, it can be reasonably divided according to the size of the first cache or the second cache to improve the utilization of the cache.

[0025] In a possible implementation manner, the first processing unit is further configured to:

[0026] According to the arrangement order of the multiple RAW source sub-blocks, lens distortion correction processing is sequentially performed on the multiple RAW source sub-blocks, and after the lens distortion correction processing of N RAW source sub-blocks is completed to obtain N RAW target sub-blocks, the N RAW target sub-blocks are reorganized into a first RAW target block;

[0027] The second processing unit is specifically configured to perform ISP on the first RAW target block.

[0028] In the above implementation of the present application, a block-based approach is used for distortion correction processing, so that when performing ISP processing, there is no need to wait for the entire image to be processed before executing ISP, which can improve processing efficiency and reduce processing delay.

[0029] In a possible implementation, the first processing unit includes a logic circuit and an output buffer;

[0030] The logic circuit is used for:

[0031] performing lens distortion correction processing on the multiple RAW source sub-blocks in sequence according to the arrangement order of the multiple RAW source sub-blocks, and after completing the lens distortion correction processing on the N RAW source sub-blocks to obtain the N RAW target sub-blocks, reorganizing the N RAW target sub-blocks into a first RAW target block, and storing the first RAW target block in the output buffer; the N second RAW sub-blocks are located in the same row or the same column in the second RAW image;

[0032] The second processing unit is further configured to read the first RAW target block from the output buffer and perform ISP on the first RAW target block.

[0033] In a possible implementation manner, the output buffer is a line buffer.

[0034] In a second aspect, an embodiment of the present application provides an electronic device, comprising the electronic device described in the first aspect or any implementation manner of the first aspect.

[0035] In a possible implementation, the electronic device is a mobile terminal, a vehicle-mounted terminal, a pan-tilt camera, a vehicle-mounted camera, or a video surveillance device.

[0036] In a third aspect, an embodiment of the present application provides a lens distortion correction method, which is applied to an electronic device, wherein the electronic device includes a first processing unit and a second processing unit, and the method includes:

[0037] The RAW source image to be processed is obtained through the first processing unit, and the lens distortion correction processing is performed on the RAW source image to obtain a RAW target image; and the RAW target image is obtained from the first processing unit through the second processing unit, and image signal processing ISP is performed on the RAW target image; the RAW source image is a distorted image, and the RAW target image is a non-distorted image.

[0038] In a possible implementation manner, performing lens distortion correction processing on the RAW source image by the first processing unit to obtain a RAW target image includes:

[0039] The RAW source image is divided into multiple RAW source sub-blocks by the first processing unit, and lens distortion correction processing is performed on each of the multiple RAW source sub-blocks to obtain multiple RAW target sub-blocks included in the RAW target image.

[0040] In a possible implementation manner, performing lens distortion correction processing on each of the multiple RAW source sub-blocks respectively includes:

[0041] Performing inter-channel interpolation processing on multiple color channels included in the first RAW source sub-block in the horizontal direction and the vertical direction;

[0042] The first RAW source sub-block is any RAW source sub-block among the multiple RAW source sub-blocks.

[0043] In a possible implementation, the electronic device further includes a memory, the memory storing a coordinate mapping table, wherein the coordinate mapping table records a mapping relationship between pixel coordinates of a RAW source image and pixel coordinates of the RAW target image;

[0044] The step of dividing the RAW source image into blocks to obtain a plurality of RAW source sub-blocks by the first processing unit includes:

[0045] The first processing unit determines, according to the coordinate mapping table, regions of the RAW source sub-blocks to which the plurality of RAW target sub-blocks in the RAW target image are respectively mapped in the RAW source image.

[0046] In a possible implementation manner, determining, according to the coordinate mapping table, areas of the RAW source sub-blocks to which the multiple RAW target sub-blocks in the RAW target image are respectively mapped in the RAW source image includes:

[0047] According to the arrangement order of the multiple RAW target sub-blocks, the region of the i-th RAW source sub-block mapped by the i-th RAW target sub-block in the RAW source image is determined in sequence according to the coordinate mapping table; or,

[0048] According to the arrangement order of the multiple RAW target sub-blocks, the area of ​​the i-th RAW source sub-block mapped by the i-th RAW target sub-block in the RAW source image is determined in sequence according to the lookup table; the lookup table includes the starting point and size of the RAW source sub-blocks corresponding to the multiple RAW target sub-blocks in the RAW source image; the lookup table is generated according to the coordinate mapping table.

[0049] In a possible implementation manner, performing lens distortion correction processing on each of the multiple RAW source sub-blocks respectively includes:

[0050] The i-th RAW source sub-block is read from the memory according to the area of ​​the i-th RAW source sub-block in the RAW source image, and lens distortion correction processing is performed on the i-th RAW source sub-block.

[0051] In a possible implementation manner, the i-th RAW source sub-block is read from the memory according to the area of ​​the i-th RAW source sub-block in the RAW source image;

[0052] According to the arrangement order of the multiple RAW target sub-blocks, the i-th RAW source sub-block is read from the memory in sequence according to the area of ​​the i-th RAW source sub-block mapped by the i-th RAW target sub-block in the RAW source image.

[0053] In a possible implementation, the method further includes:

[0054] According to the arrangement order of the multiple RAW source sub-blocks, after completing the lens distortion correction process of N RAW source sub-blocks to obtain N RAW target sub-blocks, reorganize the N RAW target sub-blocks into a first RAW target block;

[0055] Performing image signal processing ISP on the RAW target image, including:

[0056] ISP is performed on a first RAW target block in the RAW target image.

[0057] In a possible implementation manner, the N second RAW sub-blocks are located in the same row or the same column in the second RAW image.

[0058] In a fourth aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a memory and at least one processor, the memory being used to store a computer program, the at least one processor being configured to call all or part of the computer program stored in the memory to execute the method described in the third aspect above. The at least one processor comprises the first processing unit and the second processing unit. Optionally, the electronic device further comprises an image sensor.

[0059] In a fifth aspect, an embodiment of the present application provides a system on chip, the system on chip comprising at least one processor, a memory and an interface circuit, the interface circuit being used to obtain a computer program from outside the chip system; the computer program is used to implement the method described in the third aspect when executed by the at least one processor. The at least one processor comprises the first processing unit and the second processing unit.

[0060] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the computer program is used to implement the method according to the third aspect. The at least one processor includes the first processing unit and the second processing unit.

[0061] In a seventh aspect, an embodiment of the present application provides a computer program product, which is used to implement the method described in the third aspect when the computer program product is executed by at least one processor. The at least one processor includes the first processing unit and the second processing unit.

[0062] It should be understood that the second to seventh aspects of the present application are consistent with the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here.

[0063] Based on the implementations provided in the above aspects, the present application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application;

[0065] Figure 2 A schematic diagram of image segmentation provided in an embodiment of the present application;

[0066] Figure 3 A schematic diagram of the forward mapping and backward mapping methods provided in the embodiment of the present application;

[0067] Figure 4 A schematic diagram of reverse mapping provided in an embodiment of the present application;

[0068] Figure 5A A schematic diagram of a block division method provided in an embodiment of the present application;

[0069] Figure 5B A schematic diagram of another block division method provided in an embodiment of the present application;

[0070] Fig. 6A A schematic diagram of an RGB arrangement provided in an embodiment of the present application;

[0071] Figure 6B A schematic diagram of an RGBG arrangement provided in an embodiment of the present application;

[0072] Figure 6C A schematic diagram of the RGBW arrangement provided in an embodiment of the present application;

[0073] Fig.6D A schematic diagram of a channel interpolation method provided in an embodiment of the present application;

[0074] Figure 7 A schematic diagram of sub-block merging provided in an embodiment of the present application;

[0075] Figure 8 A schematic diagram of the structure of a first processing unit provided in an embodiment of the present application;

[0076] Fig. 9 A schematic diagram of a logical processing method of a first processing unit provided in an embodiment of the present application;

[0077] Fig.10 A schematic diagram of data skipping provided in an embodiment of the present application;

[0078] Fig.11 A schematic diagram of the structure of another electronic device 100 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0080] Among them, in the description of this application, unless otherwise specified, "multiple" means two or more than two. In addition, " / " indicates that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; "and / or" in this application is only a kind of association relationship describing the associated objects, indicating that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different. It should also be noted that, unless otherwise specified, the specific description of some technical features in one embodiment can also be used to explain the corresponding technical features mentioned in other embodiments.

[0081] In the embodiments of the present application, words such as "coupling" are not limited to direct physical or mechanical connections, but may include electrical connections, whether direct or indirect, which are equivalent to connectivity in a broad sense. In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0082] The present application can be applied to the fields of computational imaging and computational perception that require image distortion correction, such as assisted driving, automatic driving, and terminals. The electronic device provided in the embodiment of the present application may be a module, chip, chipset, circuit board or component integrated in an electronic device. For example, the electronic device may be a system on chip (SoC) applied to an electronic device. The electronic device may be a user equipment (UE), such as various types of devices such as a mobile phone, a tablet computer, a smart screen or an image capture device, or an on-board imaging system (or on-board camera) in assisted driving or automatic driving, or a pan-tilt camera or video surveillance device applied in video security, etc. The electronic device is provided with a camera device, which may also be referred to as a lens or an image sensor, for collecting unprocessed (RAW) image data. For example, the electronic device may also be installed with various software applications such as camera applications, video call applications or online video capture applications for driving the camera device to capture images, and the user may start the above-mentioned various applications to use the camera device to capture images or videos. In the embodiment of the present application, the image captured by the camera device is distorted, for example, it may be a fisheye lens, a wide-angle lens, or a pinhole lens, etc. The embodiment of the present application does not specifically limit the type of lens.

[0083] As an example, when the present application is applied to application scenarios such as video security or autonomous driving, electronic devices, such as PTZ cameras or vehicle-mounted cameras, etc., first undergo lens distortion correction and ISP processing (i.e., image enhancement processing) for the collected RAW images, and then perform subsequent processing tasks, such as 2D, 3D target detection, lane line detection, scene semantic segmentation, etc. A collected image with distortion is corrected by an electronic device to an ideal (undistorted) image before undergoing back-end processing.

[0084] As another example, when the present application is applied to application scenarios such as computational imaging of a terminal or computational perception of autonomous driving, after the terminal or a device such as a vehicle-mounted camera captures an image, the image is first corrected for lens distortion by an electronic device, and then ISP is executed, and then the computational perception task is performed.

[0085] See also Figure 1As shown, it is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. The electronic device may be, for example, a chip or a chipset or a circuit board equipped with a chip or a chipset. The chip or chipset or the circuit board equipped with a chip or a chipset may work under necessary software drive. The electronic device 100 includes a plurality of processing units. The plurality of processing units may be integrated in one or more chips. The plurality of chips may be regarded as a chipset. When a plurality of processing units are integrated in the same chip, the chip may also be referred to as a system on chip. In addition to the plurality of processing units, the electronic device 100 also includes one or more other necessary components, such as a memory 130. The first processing unit 110, the second processing unit 120 and the memory 130 may be integrated in one chip, which is a system on chip. For example, the memory 130 may include an on-chip random access memory (RAM, Random Access Memory) or include a DDR memory, etc.

[0086] The first processing unit 110 can be used as a separate component or integrated into other logic devices, which include but are not limited to: central processing unit (CPU), graphics processing unit (GPU), signal processor (digital signal processing, DSP), neural network processor (neural network processing unit, NPU) and other processors. Dedicated neural processors may include but are not limited to convolutional neural network processors, tensor processors or neural processing engines. Taking the integration of the first processing unit 110, the second processing unit 120 and the memory 130 in the SoC as an example, the first processing unit 110 can be understood as a subsystem in the SoC, that is, the first processing unit 110 can also be referred to as the first SoC subsystem. Exemplarily, the CPU, GPU, DSP, and NPU can all be processors within the system on chip.

[0087] The second processing unit 120 can also be used as a separate component or integrated in other logic devices. The logic devices include but are not limited to processors such as CPU, GPU, DSP, application specific integrated circuits (ASIC), NPU, etc. Taking the integration of the first processing unit 110, the second processing unit 120 and the memory 130 in the SoC as an example, the second processing unit 120 can also be understood as a subsystem in the SoC, that is, the second processing unit 120 can also be called a second SoC subsystem. Exemplarily, the CPU, GPU, DSP, and NPU can all be processors in the system on chip.

[0088] In some embodiments, the second processing unit 120 may be an ISP system, and the ISP system may use an ASIC ISP or an artificial intelligence (AI) ISP. In other embodiments, the first processing unit 110 and the second processing unit 120 may be combined into an ISP system. The ISP system may use an ASIC ISP or an AI ISP.

[0089] In some embodiments, the first processing unit 110 and the second processing unit 120 may be integrated as two components in the same logic device, or may be integrated in different logic devices.

[0090] The memory 130 is used to store the RAW source image to be processed. The RAW source image to be processed can be collected by a camera device, such as a fisheye lens, a wide-angle lens or a pinhole lens, etc. It should be noted that the RAW source image is a two-dimensional image. The RAW source image to be processed is a distorted image. The first processing unit 110 reads the RAW source image from the memory 130. The first processing unit 110 is used to perform lens distortion correction processing on the RAW source image to obtain a RAW target image. The RAW target image can be understood as a distortion-free image. Furthermore, the second processing unit 120 is used to obtain the RAW target image from the first processing unit 110 and perform image signal processing (ISP) on the RAW target image. ISP may include, but is not limited to: black level correction (BLC), auto white balance (AWB), auto exposure (AE), auto focus (AF), bad point correction (BPC), noise elimination, shadow correction, demosaicing, color correction matrix (CCM) or gamma correction, etc. ISP may also include converting RAW domain images into RGB images or YUV domain images.

[0091] In the embodiment of the present application, the components for performing the lens distortion correction processing and the components of the ISP are decoupled, and the first processing unit 110 (i.e., the component performing the lens distortion correction processing) performs the lens distortion correction processing on the image in the RAW domain output by the camera device. The corrected image is no longer written into the memory, but the second processing unit 120 directly obtains the corrected target image in the RAW domain from the first processing unit 110, which can effectively reduce the demand for processing resources such as memory bandwidth by the ISP processing + lens distortion correction processing.

[0092] In some possible implementation scenarios, the ISP system is composed of the first processing unit 110 and the second processing unit 120. The first processing unit 110 can be understood as an ISP front-end module, and the second processing unit 120 can be understood as an ISP back-end module. For example, the first processing unit 110 can be understood as a lens distortion correction (LDC) module, and the second processing unit 120 can be understood as an ISP back-end module. Here, the LDC module can be interpreted as an ISP front-end module.

[0093] In a possible implementation, the lens distortion correction process may be performed by interpolation. For example, when the first processing unit performs lens distortion correction on the RAW source image, it may perform inter-channel interpolation processing on multiple color channels included in the RAW source image in the horizontal direction and the vertical direction.

[0094] In a possible implementation manner, the first processing unit 110 may perform lens distortion correction processing on the RAW source image after dividing the RAW source image into blocks. The lens distortion correction processing of the RAW source image by the first processing unit 110 may include image division and image interpolation. The first processing unit 110 may divide the RAW source image into blocks to obtain multiple RAW source sub-blocks, and then when performing lens distortion correction processing, lens distortion correction processing may be performed on each of the multiple RAW source sub-blocks to obtain RAW target sub-blocks corresponding to the multiple RAW source sub-blocks. That is, multiple RAW target sub-blocks constitute a RAW target image. When performing ISP, the second processing unit may perform ISP on multiple RAW target sub-blocks separately, or may perform ISP on multiple RAW target sub-blocks uniformly, such as performing ISP after merging them into one image, or may perform ISP on each row or column of the RAW target sub-blocks in the multiple RAW target sub-blocks.

[0095] Image segmentation:

[0096] The following is an explanation of the image block division method. In some embodiments, a coordinate mapping table can be pre-established, and the coordinate mapping table is used to record the mapping relationship between the pixel coordinates of the RAW source image and the pixel coordinates of the RAW target image. Since the RAW target image is a normal image without distortion, the RAW target image can be divided into blocks, and then the RAW source sub-block corresponding to each RAW target sub-block is determined by the coordinate mapping table. Exemplarily, the block division can be uniformly divided. For example, see Figure 2 As shown, the block division of the RAW target image after lens distortion correction can be seen in Figure 2 As shown in (b). Figure 2(b) in FIG. 8 takes 8 RAW target sub-blocks as an example. The 8 RAW source sub-blocks mapped by the 8 RAW target sub-blocks can be determined based on the coordinate mapping table. Figure 2 The pre-established coordinate mapping table may be pre-stored in the memory 130 .

[0097] The process of establishing a coordinate mapping table is described below. A coordinate mapping table between a distorted image and an undistorted image can be established based on the intrinsic parameters and distortion coefficients of the lens. In some embodiments, the intrinsic parameters and distortion coefficients of the lens can be calibrated. For example, calibration is performed using a calibration pattern. The calibration pattern can be, for example, a calibration checkerboard, or other calibration patterns, such as a QR code, etc. The calibration pattern is photographed in different directions through the lens, and the corresponding RGB image is obtained after ISP. Then, based on the RGB images obtained by shooting in multiple directions, the intrinsic parameters and distortion coefficients are obtained through an optimization algorithm. When solving the intrinsic parameters, different imaging models can be used for calculation for different lens types. For example, for wide-angle lenses, a pinhole camera imaging model can be used to solve the intrinsic parameters. For another example, for fisheye lenses, a fisheye camera imaging model can be used to solve the intrinsic parameters.

[0098] After calibrating the lens's intrinsic parameters and distortion coefficients, the distortion coordinates can be projected onto the pixel plane through the intrinsic parameter matrix to obtain the corresponding relationship between the coordinates of any pixel point on the distorted image and the coordinates of the pixel point on the image before distortion. However, the entire projection process is relatively complicated. If a coordinate transformation is performed on each pixel point on each frame of the image, it will cause unnecessary calculation delays. Since the lens's intrinsic parameter matrix and distortion coefficients are fixed, that is, the coordinate mapping between the distorted image and the original image is fixed, the mapping relationship of the pixel coordinates is calculated offline in the implementation of this application and saved in the coordinate mapping table. In actual use, it is only necessary to find the corresponding relationship between each pair of pixel points by looking up the table.

[0099] As an example, the present application may use a forward mapping method or a reverse mapping method when constructing a coordinate mapping table. Forward mapping may also be referred to as forward mapping. Reverse mapping may also be referred to as backward mapping or backward mapping. Figure 3 (a) shows the forward mapping method, which means calculating the corresponding pixel coordinates of the mapped target image based on the pixel coordinates of the source image. Figure 3 (b) shows the backward mapping method. Backward mapping refers to calculating the corresponding pixel coordinates in the source image based on the pixel coordinates of the target image.

[0100] As an example, the embodiment of the present application can adopt a backward mapping method. Since the integer coordinates of each pixel on the Raw source image will be converted into floating-point coordinates after being projected by the mapping function T, the floating-point coordinates are generally rounded when the target image is output. This processing will introduce many holes and misaligned pixels in the output image. Therefore, the coordinate mapping table constructed by the backward mapping method can avoid pixel holes and pixel misalignment in the target image. Figure 4 shown.

[0101] Furthermore, after establishing the coordinate mapping table, the area of ​​the RAW source sub-block corresponding to each RAW target sub-block in the RAW target image can be determined according to the coordinate mapping table. Since the degree of distortion of different areas of the entire image may be different, the size of the RAW target sub-block is the same, but after mapping, the size of the RAW source sub-block corresponding to the RAW source image is not the same, and there may be overlap between different RAW source sub-blocks. For example, see Figure 2 shown.

[0102] In some possible implementations, the mapping relationship between each RAW target sub-block and the area of ​​the RAW source sub-block in the RAW source image can be determined in advance based on the coordinate mapping table. The area corresponding to each sub-block in each RAW source image can be determined in advance and recorded, so as to avoid block calculation during the distortion processing. The use of pre-calculation can improve processing efficiency and processing resource utilization. Based on this, a lookup table can be stored in the memory 130, and the lookup table records the area of ​​each RAW source sub-block in the RAW source image, for example, the starting point and size (i.e., length and width) of each RAW source sub-block can be recorded. The lookup table is generated based on the coordinate mapping table.

[0103] In the embodiment of the present application, the execution order of the lens distortion correction processing for the RAW source sub-blocks can be determined according to the arrangement order of the RAW target sub-blocks, so that the first processing unit 110 can perform the lens distortion correction processing for each RAW source sub-block according to the execution order.

[0104] In the embodiment of the present application, the coordinate mapping table is stored in the memory 130. In some possible scenarios, in order to reduce the load of the memory 130 (such as DDR), the coordinate mapping table can be a coordinate mapping grid with a lower resolution obtained after downsampling, or a new data representation obtained through other encoding or compression. The embodiment of the present application does not specifically limit this.

[0105] Image interpolation:

[0106] In a first possible implementation, the RAW source sub-block mapped by each RAW target sub-block may be determined in sequence according to the arrangement order of the multiple RAW target sub-blocks, and image interpolation processing may be performed on each RAW source sub-block after each RAW source sub-block is determined. For example, according to the arrangement order of the multiple RAW target sub-blocks, the region of the i-th RAW source sub-block mapped by the i-th RAW target sub-block in the RAW source image may be determined according to the coordinate mapping table; and the i-th RAW source sub-block may be read from the memory 130 according to the region of the i-th RAW source sub-block in the RAW source image, so as to perform image interpolation processing on the i-th RAW source sub-block. Figure 5A shown.

[0107] In a second possible implementation, the RAW source sub-blocks corresponding to the multiple RAW target sub-blocks may be determined first, and then image interpolation processing may be performed on each RAW source sub-block in sequence. For example, the region of the RAW source sub-block to which each RAW target sub-block in the multiple RAW target sub-blocks is mapped in the RAW source image may be determined respectively according to the coordinate mapping table, and the i-th RAW source sub-block may be read from the memory according to the region of the i-th RAW source sub-block to which the i-th RAW target sub-block is mapped in the RAW source image in the order in which the multiple RAW target sub-blocks are arranged, so as to perform image interpolation processing on the i-th RAW source sub-block. Figure 5B shown.

[0108] In a third possible implementation, a lookup table is stored in the memory 130, and the lookup table records the starting point and size of each RAW source sub-block. The first processing unit 110 determines the area of ​​the i-th RAW source sub-block in the RAW source image according to the arrangement order of the multiple RAW target sub-blocks and according to the lookup table; and reads the i-th RAW source sub-block from the memory 130 according to the area of ​​the i-th RAW source sub-block in the RAW source image to perform image interpolation processing on the i-th RAW source sub-block.

[0109] In the embodiment of the present application, when performing image interpolation processing on each RAW source sub-block, an inter-channel interpolation algorithm can be used. The first processing unit performs inter-channel interpolation processing on the multiple color channels included in each RAW source sub-block in the horizontal direction (X direction) and the vertical direction (Y direction). The interpolation order can be X direction first and then Y direction, or Y direction first and then X direction, or X direction and Y direction can be interpolated simultaneously. Since the RAW image is a cross arrangement of multiple channels of red, green and blue. For example, the RAW image adopts RGB arrangement, see Fig. 6A As shown, or RGBG arrangement can be used, see Figure 6B As shown, or RGBW arrangement, see Figure 6C As shown, etc. Figure 6BThe RAW shown uses RGBG arrangement as an example. For the pixels of each channel in the RAW image, the pixels of its neighboring channels belong to different color channels, and the interval of the same color channel is two pixels. The embodiment of the present application adopts an inter-channel interpolation algorithm to avoid serious visual effects such as blurring and aliasing.

[0110] The following is an example of the inter-channel interpolation in two-dimensional directions (horizontal and vertical directions), but does not specifically limit the inter-channel interpolation processing methods in the horizontal direction and the vertical direction. Fig.6D As shown in the figure, taking the RGBG arrangement of the Raw image as an example, assuming that a pixel point on the RAW target image corresponds to point P = (x, y) on the source image, assuming that the neighborhood of point P is known (x1, y1), (x2, y2), ..., (x k ,y k ) point. Taking the x direction first and then the y direction as an example, interpolation is first performed in the x direction. In order not to be limited to a certain interpolation method, the interpolation method is f(x) = a0 + a1x + a2x 2 +..+a k x k Indicates that the polynomial coefficients a0, a1…a k It is calculated by substituting the neighboring pixel values. For example, f(x,y1) is calculated by f(x1,y1)…f(x k ,y1) into the calculation. And so on, we can get the neighbors in the x direction f(x,y1),f(x,y2)…f(x,y k ) After interpolating the pixels, the same interpolation process is performed in the y direction to obtain the final f(x,y). It should be noted that if the interpolation is performed in the y direction first and then in the x direction, the result is the same as the result of interpolation in the above order.

[0111] In a possible implementation manner, the lens distortion correction process of the RAW source image by the first processing unit 110 may include image reconstruction in addition to image segmentation and image interpolation.

[0112] In a first possible reorganization manner, the multiple RAW target sub-blocks after the block LDC processing may be reorganized into a RAW target image, so that the second processing unit 120 can perform ISP on the RAW target image.

[0113] In the second possible reorganization method, the above-mentioned LDC may be performed according to the arrangement order of the multiple RAW target sub-blocks, for example, the RAW source sub-blocks corresponding to the multiple RAW target sub-blocks belonging to the same row are sequentially LDCed according to the arrangement order of the multiple RAW target sub-blocks belonging to the same row (for example, from left to right or from right to left, etc.). Then, the multiple RAW source sub-blocks in the row are reorganized into a RAW target block, so that the second processing unit 120 performs ISP on the RAW target block. For example, see Figure 7 As shown, after the RAW target sub-blocks with serial numbers 1-4 are obtained through LDC processing, the RAW target sub-blocks with serial numbers 1-4 are merged. Therefore, the second processing unit 120 does not need to wait until all the RAW source sub-blocks included in the entire RAW source image complete LDC before starting ISP, which can reduce the calculation delay of ISP, that is, the delay is reduced from one image to one RAW target block. In some embodiments, after the RAW target sub-blocks belonging to multiple rows are obtained through processing, the second processing unit 120 can perform ISP on the RAW target block after the RAW sub-blocks of multiple rows are merged into one RAW target block.

[0114] In a third possible reorganization method, the above-mentioned LDC may be performed according to the arrangement order of the multiple RAW target sub-blocks, which is based on the RAW sub-block columns. For example, the RAW source sub-blocks corresponding to the multiple RAW target sub-blocks belonging to the same column respectively perform LDC in sequence according to the arrangement order of the multiple RAW target sub-blocks belonging to the same column (for example, from top to bottom or from bottom to top, etc.). Then, the multiple RAW source sub-blocks in the column are reorganized into a RAW target block, so that the second processing unit 120 can perform ISP on the RAW target block. For example, see Figure 7 As shown, after the RAW target sub-blocks with serial numbers 1, 5, 9, 13, 17, 21, 24, and 29 are obtained through LDC processing, the RAW target sub-blocks with serial numbers 1, 5, 9, 13, 17, 21, 24, and 29 are merged. Therefore, the second processing unit 120 does not need to wait until all the RAW source sub-blocks included in the entire RAW source image have completed LDC before starting ISP, which can reduce the calculation delay of ISP, that is, the delay is reduced from one image to one RAW target block. In some embodiments, after the RAW target sub-blocks belonging to multiple columns are obtained through processing, the second processing unit 120 can perform ISP on the RAW target block after merging the RAW sub-blocks of multiple columns into one RAW target block.

[0115] The embodiment of the present application is described below in conjunction with the specific structures of the first processing unit 110 and the second processing unit 120 in the electronic device.

[0116] See also Figure 8As shown, the first processing unit 110 includes a cache and a logic circuit, and the second processing unit 120 also includes a cache and a logic circuit. The cache of the first processing unit 110 is used to cache the RAW image (or RAW source sub-block) read from the memory 130 and the processed RAW image (or RAW target sub-block). The logic circuit of the first processing unit 110 is responsible for executing the read and write scheduling of the RAW source image (including multiple RAW source sub-blocks), such as reading the coordinate mapping table from the specified storage area of ​​the memory 130, and executing high-intensity operations such as image segmentation and image interpolation.

[0117] In some possible scenarios, when the first processing unit 110 and the second processing unit 120 are integrated in a SoC system, the first processing unit 110 can be understood as a subsystem in the SoC system, and the first processing unit 110 can also be understood as a lens distortion correction module in the SoC system. The second processing unit 120 can be understood as another subsystem in the SoC system. The second processing unit 120 can also be understood as an ISP module in the SoC system. The logic circuit in the first processing unit 110 can include two parts: firmware and hardware. The firmware in the first processing unit 110 is responsible for performing the scheduling work of RAW image reading and writing, such as reading the RAW image or the coordinate mapping table from the specified memory area in the memory, etc., and can also be used to modify the status bits of each register, etc. The hardware in the first processing unit 110 is responsible for performing high-intensity operations such as image interpolation. The logic circuit in the second processing unit 120 can include two parts: firmware and hardware. The firmware part in the second processing unit 120 is responsible for performing some scheduling work required for the image processing algorithm, such as controlling the switch state and execution order of each module in the second processing unit 120, modifying the status bits of each register, etc. Some low-intensity logic operations can also be performed. The hardware part in the second processing unit 120 is responsible for performing high-intensity operations in the image processing algorithm, and the calculation logic of the hardware part can be controlled by the state in the register.

[0118] The cache of the first processing unit 110 may include an input cache. When the first processing unit 110 divides the RAW source image into blocks, it may divide the blocks reasonably according to the size of the input cache of the first processing unit 110. In order to solve the memory bound between sub-blocks, the input cache in the first processing unit 110 may include a ping-pong cache composed of two cache areas. For example, the first processing unit 110 includes a first cache and a second cache. The size of the first cache and the size of the second cache may be the same. When dividing the RAW source image into blocks, it may be reasonably divided according to the size of the first cache or the second cache to improve the utilization of the cache. The first cache and the second cache constitute a ping-pong cache. The storage rates of the first cache and the second cache are both higher than the memory. The ping-pong cache composed of the first cache and the second cache is used to sequentially store two RAW source sub-blocks that the first processing unit 110 reads continuously from the memory 130, such as the i-th RAW source sub-block and the i+1-th RAW source sub-block, in a ping-pong storage manner. For example, the first processing unit 110 reads the first RAW source sub-block from the memory 130 to the first cache, and reads the second RAW source sub-block from the memory 130 to the second cache. After the first RAW source sub-block is read from the first cache and the image interpolation processing is performed, the third RAW source sub-block is continued to be read from the memory 130 to the first cache. After the first RAW source sub-block is read from the first cache and the image interpolation processing is performed, the second RAW source sub-block is continued to be read from the second cache and the image interpolation processing is performed, and so on, until the image interpolation processing is completed for all RAW source sub-blocks included in the RAW source image. In some possible scenarios, for video stream processing, after the current RAW source image is processed, the next frame of the RAW source image is processed.

[0119] The cache of the first processing unit 110 may also include an output cache. The logic circuit of the first processing unit 110 reads the i-th RAW source sub-block in the first cache. After image interpolation processing is performed on the i-th RAW source sub-block, the i-th RAW target sub-block obtained after processing may be stored in the output cache of the first processing unit 110. In some embodiments, the output cache may also adopt a ping-pong cache storage method, and the output cache includes a third cache and a fourth cache, and the third cache and the fourth cache constitute a ping-pong cache. In one possible manner, when the first processing unit 110 does not perform merging of the RAW target sub-blocks, the sizes of the third cache and the fourth cache may be the same as the sizes of the first cache and the second cache. In another possible manner, the first processing unit 110 merges the RAW target sub-blocks, such as merging a row of RAW target sub-blocks or multiple rows of RAW target sub-blocks to obtain a RAW target block. The sizes of the third cache and the fourth cache may be related to the size of the RAW target block.

[0120] The LDC processing flow is described below in conjunction with the structure of the first processing unit 110 .

[0121] For example, see Fig. 9 As shown, the first processing unit 110 reads the RAW source sub-block 1 corresponding to the RAW target sub-block 1 from the memory 130 for the RAW target sub-block 1 of the output size H5*W5. For example, the RAW source sub-block 1 is a sub-block of size H0*W0. Since the read and write latency of the memory 130 is larger than that of the cache of the first processing unit 110, the first processing unit 110 reads the RAW source sub-block from the memory 130 to the input cache of the first processing unit 110. In order to further optimize the read and write conflicts within the block, the logic circuit in the first processing unit 110 can further divide the RAW source sub-block 1 of H0*W0 into blocks for image interpolation processing. For example, the first processing unit 110 can write the image data of the sub-block of size H2*W2 in the RAW source sub-block 1 of H0*W0 into the register as needed to perform inter-channel interpolation processing to obtain an interpolation unit of size H4*W4. When a number of consecutive interpolation units of size H4*W4 are combined into a sub-block of size H5*W5, namely, RAW target sub-block 1. The RAW target sub-block 1 is stored in the output buffer of the first processing unit 110. A number of consecutive sub-blocks of size H5*W5 are combined to obtain a RAW target block consisting of one or several rows of RAW target sub-blocks of the RAW target image. Further, the second processing unit 110 can read the RAW target block from the output buffer to perform ISP.

[0122] In the embodiment of the present application, when performing inter-channel interpolation processing, the logic circuit adopts a register solution to read and write the pixels included in the minimum interpolation interval required for each pixel to be interpolated in the sub-block, so as to solve the problem that the cache in the first processing unit 110 does not support data skipping. Fig.10 shown.

[0123] As an example, the first processing unit 110 and the second processing unit 120 are integrated into the same logic device, and the first processing unit 110 and the second processing unit 120 can use shared memory. Shared memory, for example, can use SRAM. The memory 130 can use DDR. The cache of the first processing unit 110 is a partial area of ​​the shared memory, and the cache of the second processing unit 120 is a partial area of ​​the shared memory. The cache of the first processing unit 110 and the cache of the second processing unit 120 do not overlap in the shared memory. Of course, the first processing unit 110 and the second processing unit 120 are not integrated into the same logic device, and shared memory can also be used. The read and write rate of the shared memory needs to be higher than the read and write rate of the memory 130. In some scenarios, the first processing unit 110 and the second processing unit 120 may not use the shared memory method, and have their own caches independently.

[0124] See also Fig.11 As shown, it is a schematic diagram of the structure of another electronic device 100 provided in an embodiment of the present application. The first processing unit 110 in the electronic device 100 includes a plurality of cascaded processing modules, namely a pre-processing module, an interpolation module and a post-processing module. The pre-processing module is used to obtain the distorted image captured by the image sensor, that is, the RAW source image, and perform block processing on the RAW source image according to the coordinate mapping table obtained by offline calculation. The interpolation module is responsible for traversing the block image, finding the neighboring pixels according to the coordinate mapping table, and performing two-dimensional interpolation processing using the correlation between the multi-channels of the image in the Raw domain to realize the distortion correction processing of the image within the block. The post-processing module is mainly responsible for splicing the block correction image into a target block according to the set timing, and sending the spliced ​​target block to the second processing unit 120. The second processing unit 120 is responsible for converting the corrected Raw image into an RGB image and performing ISP. For example, the second processing unit 120 may include a plurality of cascaded processing modules, such as a 3A module, a de-mosaic module, a tone mapping module, and the like.

[0125] In the embodiment of the present application, online LDC is performed on the Raw image through the blocking and reorganization strategy, which can greatly alleviate the frame delay problem caused by ISP processing. In the embodiment of the present application, a small amount of dedicated memory is added to the first processing unit 110 during the hardware design, but it will almost invade the remaining hardware modules and underlying software design of the ISP. In the embodiment of the present application, the RAW image or block after the distortion correction processing will no longer be written to the DDR. Compared with RGB LDC, it can reduce the reading and writing of image data from the memory once, which can greatly alleviate the memory bandwidth occupancy problem of the ISP system.

[0126] Based on the above embodiments, the present application also provides a lens distortion correction method, which is applied to an electronic device. The electronic device includes a memory, a first processing unit and a second processing unit, and the method includes:

[0127] The first processing unit reads the RAW source image to be processed from the memory, and performs lens distortion correction processing on the RAW source image to obtain a RAW target image; and the second processing unit performs image signal processing ISP on the RAW target image; the RAW source image is a distorted image, and the RAW target image is a non-distorted image. The specific workflow of the first processing unit and the second processing unit can be referred to the previous description, which will not be repeated here.

[0128] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) that contain computer-usable program code.

[0129] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0130] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An electronic device, characterized in that: The electronic device comprises a first processing unit and a second processing unit; The first processing unit is used to obtain a RAW source image to be processed, and perform lens distortion correction processing on the RAW source image to obtain a RAW target image; the RAW source image is a distorted image, and the RAW target image is a non-distorted image; The second processing unit is used to obtain the RAW target image from the first processing unit and perform image signal processing ISP on the RAW target image.

2. The electronic device according to claim 1, wherein: Also includes a memory; the memory, the first processing unit and the second processing unit are integrated into the same system-on-chip SoC in the electronic device; The memory is used to store the RAW source image; The first processing unit is specifically configured to read the RAW source image from the memory.

3. The electronic device according to claim 1 or 2, characterized in that: The first processing unit and the second processing unit are integrated in a same logic device.

4. The electronic device according to any one of claims 1 to 3, characterized in that: The first processing unit is specifically configured to divide the RAW source image into blocks to obtain a plurality of RAW source sub-blocks, and perform lens distortion correction processing on each of the plurality of RAW source sub-blocks to obtain a plurality of RAW target sub-blocks included in the RAW target image; The second processing unit is specifically configured to perform image signal processing (ISP) on the multiple RAW target sub-blocks.

5. The electronic device as claimed in claim 4, characterized in that: The first processing unit, when performing lens distortion correction processing on each RAW source sub-block of the plurality of RAW source sub-blocks, is specifically configured to: Performing inter-channel interpolation processing on multiple color channels included in the first RAW source sub-block in the horizontal direction and the vertical direction; The first RAW source sub-block is any RAW source sub-block among the multiple RAW source sub-blocks.

6. The electronic device according to claim 4 or 5, characterized in that: The memory in the electronic device further stores a coordinate mapping table, wherein the coordinate mapping table records the mapping relationship between the pixel coordinates of the RAW source image and the pixel coordinates of the RAW target image; The first processing unit is specifically configured to: For a first RAW target subblock in the RAW target image, an area of ​​a first RAW source subblock mapped by the first RAW target subblock in the RAW source image is determined according to the coordinate mapping table, wherein the first RAW target subblock is any RAW target subblock in the RAW target image.

7. The electronic device according to claim 6, wherein: The first processing unit is specifically configured to: Determining, according to the arrangement order of the plurality of RAW target sub-blocks and based on the coordinate mapping table, a region of the i-th RAW source sub-block to which the i-th RAW target sub-block is mapped in the RAW source image; and reading the i-th RAW source sub-block from the memory according to the area of ​​the i-th RAW source sub-block in the RAW source image, so as to perform lens distortion correction processing on the i-th RAW source sub-block; or, According to the arrangement order of the multiple RAW target sub-blocks, determining, according to a lookup table, a region of the i-th RAW source sub-block to which the i-th RAW target sub-block is mapped in the RAW source image; and reading the i-th RAW source sub-block from the memory according to the area of ​​the i-th RAW source sub-block in the RAW source image, so as to perform lens distortion correction processing on the i-th RAW source sub-block; the lookup table includes the starting points and sizes of the RAW source sub-blocks corresponding to the multiple RAW target sub-blocks in the RAW source image; The lookup table is generated according to the coordinate mapping table; or, According to the coordinate mapping table, the area of ​​the RAW source sub-block to which each RAW target sub-block in the multiple RAW target sub-blocks is respectively mapped in the RAW source image is determined, and in the arrangement order of the multiple RAW target sub-blocks, the i-th RAW source sub-block is read from the memory according to the area of ​​the i-th RAW source sub-block to which the i-th RAW target sub-block is mapped in the RAW source image, so as to perform lens distortion correction processing on the i-th RAW source sub-block.

8. The electronic device as claimed in claim 7, characterized in that: The first processing unit includes a ping-pong cache composed of a first cache and a second cache; the storage rates of the first cache and the second cache are both higher than the memory; The ping-pong cache is used to sequentially store the i-th RAW source sub-block and the (i+1)-th RAW source sub-block read by the first processing unit from the memory in a ping-pong storage manner.

9. The electronic device according to claim 7 or 8, characterized in that: The first processing unit is further configured to: According to the arrangement order of the multiple RAW source sub-blocks, lens distortion correction processing is sequentially performed on the multiple RAW source sub-blocks, and after the lens distortion correction processing of N RAW source sub-blocks is completed to obtain N RAW target sub-blocks, the N RAW target sub-blocks are reorganized into a first RAW target block; The second processing unit is specifically configured to perform ISP on the first RAW target block.

10. The electronic device according to claim 9, wherein: The first processing unit includes a logic circuit and an output buffer; The logic circuit is used for: According to the arrangement order of the multiple RAW source sub-blocks, sequentially perform lens distortion correction processing on the multiple RAW source sub-blocks, after completing the lens distortion correction processing of N RAW source sub-blocks to obtain N RAW target sub-blocks, reorganize the N RAW target sub-blocks into a first RAW target block, and store the first RAW target block into the output buffer; The N second RAW sub-blocks are located in the same row or the same column in the second RAW image; The second processing unit is further configured to read the first RAW target block from the output buffer and perform ISP on the first RAW target block.

11. The electronic device according to claim 10, wherein: The output buffer is a line buffer.

12. An electronic device, characterized in that: Comprising the electronic device as described in any one of claims 1-11.

13. The electronic device according to claim 12, characterized in that: The electronic device is a mobile terminal, a vehicle-mounted terminal, a pan-tilt camera, a vehicle-mounted camera or a video surveillance device.

14. A lens distortion correction method, characterized in that: Applied to an electronic device, the electronic device includes a first processing unit and a second processing unit, and the method includes: The RAW source image to be processed is obtained through the first processing unit, and the lens distortion correction processing is performed on the RAW source image to obtain a RAW target image; and the RAW target image is obtained from the first processing unit through the second processing unit, and image signal processing ISP is performed on the RAW target image; the RAW source image is a distorted image, and the RAW target image is a non-distorted image.

15. The method according to claim 14, characterized in that The performing lens distortion correction processing on the RAW source image by the first processing unit to obtain the RAW target image includes: The RAW source image is divided into multiple RAW source sub-blocks by the first processing unit, and lens distortion correction processing is performed on each of the multiple RAW source sub-blocks to obtain multiple RAW target sub-blocks included in the RAW target image.

16. The method according to claim 15, characterized in that The performing lens distortion correction processing on each RAW source sub-block of the plurality of RAW source sub-blocks respectively comprises: Performing inter-channel interpolation processing on multiple color channels included in the first RAW source sub-block in the horizontal direction and the vertical direction; The first RAW source sub-block is any RAW source sub-block among the multiple RAW source sub-blocks.

17. The method according to claim 15 or 16, characterized in that The electronic device further comprises a memory, wherein the memory stores a coordinate mapping table, wherein the coordinate mapping table records a mapping relationship between pixel coordinates of a RAW source image and pixel coordinates of a RAW target image; The step of dividing the RAW source image into blocks to obtain a plurality of RAW source sub-blocks by the first processing unit includes: The first processing unit determines, according to the coordinate mapping table, regions of the RAW source sub-blocks to which the plurality of RAW target sub-blocks in the RAW target image are respectively mapped in the RAW source image.

18. The method according to claim 17, characterized in that The determining, according to the coordinate mapping table, areas of the RAW source sub-blocks to which the multiple RAW target sub-blocks in the RAW target image are respectively mapped in the RAW source image comprises: According to the arrangement order of the multiple RAW target sub-blocks, the region of the i-th RAW source sub-block mapped by the i-th RAW target sub-block in the RAW source image is determined in sequence according to the coordinate mapping table; or, According to the arrangement order of the multiple RAW target sub-blocks, the area of ​​the i-th RAW source sub-block mapped by the i-th RAW target sub-block in the RAW source image is determined in sequence according to the lookup table; the lookup table includes the starting point and size of the RAW source sub-blocks corresponding to the multiple RAW target sub-blocks in the RAW source image; the lookup table is generated according to the coordinate mapping table.

19. The method according to claim 17 or 18, characterized in that: The performing lens distortion correction processing on each RAW source sub-block of the plurality of RAW source sub-blocks respectively comprises: The i-th RAW source sub-block is read from the memory according to the area of ​​the i-th RAW source sub-block in the RAW source image, and lens distortion correction processing is performed on the i-th RAW source sub-block.

20. The method of claim 19, wherein: reading the i-th RAW source sub-block from the memory according to the area of ​​the i-th RAW source sub-block in the RAW source image; According to the arrangement order of the multiple RAW target sub-blocks, the i-th RAW source sub-block is read from the memory in sequence according to the area of ​​the i-th RAW source sub-block mapped by the i-th RAW target sub-block in the RAW source image.

21. The method according to any one of claims 18 to 20, characterized in that: The method further comprises: According to the arrangement order of the plurality of RAW source sub-blocks, after completing lens distortion correction processing of N RAW source sub-blocks to obtain N RAW target sub-blocks, reorganize the N RAW target sub-blocks into a first RAW target block; Performing image signal processing ISP on the RAW target image, including: ISP is performed on a first RAW target block in the RAW target image.

22. The method according to claim 21, characterized in that The N second RAW sub-blocks are located in the same row or the same column in the second RAW image.

Images