Image processing method and related device

Abstract

The invention discloses an image processing method and a related device, and relates to the field of information processing, and the method comprises the steps: determining at least two groups of pixel coordinates in a to-be-processed image, each group of pixel coordinates comprises at least three pixel coordinates, the to-be-processed image is a distorted image, and each group of pixel coordinates corresponds to a distortion line of the to-be-processed image; according to any two adjacent groups of pixel coordinates, coordinate ranges of at least two areas are determined in the to-be-processed image in sequence, and each area comprises a plurality of pixels; and correcting the image data of each pixel in the coordinate range of the area every time the coordinate range of the area is determined, so as to obtain a target sub-image corresponding to the area.

Description

Technical Field

[0001] The present application relates to the field of information processing, and in particular to an image processing method and related devices. Background Art

[0002] Since the manufacturing precision of the lens and the deviation of the assembly process will introduce distortion, resulting in distortion of the original image, the camera system needs to correct the distortion caused by the lens to obtain a normal image.

[0003] Currently, the algorithm for image distortion processing is that the firmware calculates the coordinate mapping (x, y) of each output coordinate point in the original distorted image based on the lens design parameters and lens calibration parameters, and sparses the points of the output image, and stores the coordinate data of the sampled sparse coordinates (map lut) in DDR SDRAM (Double Data Rate Synchronous Dynamic Random Access Memory, hereinafter referred to as DDR).

[0004] The algorithm obtains the corrected output image through the following two calculation processes:

[0005] 1. Each output coordinate that is sparsely distributed is obtained by bilinear interpolation.

[0006] 2. Each output coordinate corresponds to the corrected pixel data, which is obtained by bicubic interpolation using the neighborhood of the original distorted image.

[0007] Since the storage of the input image in DDR is related to the distortion, various factors need to be weighed when reading the DDR image, which makes it difficult to process the sparsely processed coordinates for application in the above interpolation process.

[0008] In the prior art, in order to implement the image distortion processing algorithm, two large-space SRAM (Static Random Access Memory) are used to ping-pong preload the image. The original image is written into the DDR in a scan mode, and two large SRAM spaces are opened in the DDR. Many rows of original data are pre-fetched from the DDR in advance, and the two SRAMs are stored in advance according to the map lut sparse coordinates pre-calculated by SW. The neighborhood data that needs to be interpolated is reorganized from the SRAM to complete the bicubic pixel interpolation. If a block of SRAM has space, multiple rows of data are pre-fetched again to prepare for the next round of interpolation calculations.

[0009] However, since each row of image data needs to be stored in the same SRAM, it is necessary to ensure that there is enough storage space in each SRAM. When the solution has large distortion and rotation scene applications, it is necessary to ensure that the same row of image data in SRAM is stored in the same SRAM, which has a large area cost. Therefore, a larger SRAM space needs to be opened up, which places high requirements on the SRAM space. Summary of the invention

[0010] The first aspect of the present application provides an image processing method, comprising:

[0011] Determine at least two groups of pixel coordinates in the image to be processed, each group of pixel coordinates includes at least three pixel coordinates, the image to be processed is a distorted image, and each group of pixel coordinates corresponds to a distortion line of the image to be processed;

[0012] According to any two adjacent groups of pixel coordinates, coordinate ranges of at least two regions are determined in the image to be processed in sequence, each region including a number of pixels;

[0013] Each time the coordinate range of a region is determined, the image data of each pixel in the coordinate range of the region is corrected to obtain a target sub-image corresponding to the region.

[0014] In a possible implementation, determining at least two groups of pixel coordinates in the image to be processed includes:

[0015] Determining a distortion function of the image to be processed;

[0016] Determine at least two groups of reference pixel coordinates based on a pixel matrix corresponding to the image to be processed, wherein the reference pixel coordinates are pixel coordinates extracted at set intervals in a reference line corresponding to the image to be processed;

[0017] At least two groups of pixel coordinates corresponding to the at least two groups of reference pixel coordinates are respectively determined according to the distortion function.

[0018] In a possible implementation, each time the coordinate range of a region is determined, the image data of each pixel in the coordinate range of the region is corrected to obtain a target sub-image corresponding to the region, including:

[0019] According to the coordinate range of any area, obtaining target image data corresponding to any area in the image to be processed;

[0020] Sending the target image data to a first storage space;

[0021] Based on the first storage space having remaining storage locations, storing the target image data in the remaining storage locations;

[0022] Based on the fact that there is no remaining storage position in the first storage space, a target storage position is determined, and the target image data is stored in the target storage position, wherein the image data stored in the target storage position has been corrected;

[0023] Correction processing is performed according to the target image data in the first storage space to obtain a target sub-image corresponding to any one of the regions.

[0024] In a possible implementation, each time the coordinate range of a region is determined, the image data of each pixel in the coordinate range of the region is corrected to obtain a target sub-image corresponding to the region, including:

[0025] When the coordinate range of any area is determined, image data corresponding to each pixel within the coordinate range of any area is obtained from a second storage space, and the second storage space stores image data of all pixels in the image to be processed;

[0026] storing the image data in a target storage location in the first storage space, wherein the target storage location is a storage location allocated for any one of the regions;

[0027] Each time image data of any region is stored in the first storage space, correction processing is performed according to the image data corresponding to the any region to obtain a target sub-image corresponding to the any region.

[0028] In a possible implementation, storing the image data in a target storage location in the first storage space includes:

[0029] storing first image data corresponding to the first region in a first storage location in a first storage space, where the first storage location is a storage location allocated for the first region;

[0030] If the coordinate range of the first area and the coordinate range of the second area have an overlapping range, storing the first image data corresponding to the overlapping range in a second storage position in the first storage space, where the second storage position is a storage position allocated for the second area;

[0031] Obtaining second image data corresponding to each pixel outside the overlapping range in the second area from the second storage space;

[0032] The second image data is stored in the second storage location.

[0033] In a possible implementation, obtaining, from the second storage space, image data corresponding to each pixel within the coordinate range of any region includes:

[0034] Based on the coordinate range of any one of the regions, determine a set of image data blocks corresponding to any one of the regions, each set of image data blocks comprising at least two image data blocks, each image data block comprising pixel values ​​of each pixel in a pixel matrix of a preset size;

[0035] generating a first request according to the image data block set, wherein the first request is used to request each image data block in the image data block set from a second storage space;

[0036] Receive each image data block in the image data block set fed back by the second storage space.

[0037] In a possible implementation, obtaining, from the second storage space, second image data corresponding to each pixel outside the overlapping range in the second area includes:

[0038] Determining, according to the coordinate range of the second area, a set of initial image data blocks corresponding to the second area;

[0039] Determine a set of overlapping image data blocks corresponding to the overlapping range;

[0040] generating a second request according to the initial image data block set and the overlapping image data block set, wherein the second request is used to request a target image data block set from a second storage space, wherein the target image data block set and the overlapping image data block set constitute the initial image data block set;

[0041] Receive each image data block in the target image data block set fed back by the second storage space.

[0042] In a possible implementation, storing the image data in a target storage location in the first storage space includes:

[0043] Dividing each image data block in any one of the regions into a specific number of image data sub-blocks, each image data sub-block consisting of a pixel matrix of a set size;

[0044] Determine, according to a preset sequence, an arrangement order of each image data sub-block in any region and an offset address corresponding to each image data sub-block, wherein the ordering sequence number of any image data sub-block is different from that of an adjacent image data sub-block, and the offset address represents an offset amount of the image data sub-block from a specific position in any region;

[0045] According to the arrangement order of each image data sub-block in any one area and the offset address, each image data sub-block is stored in a target storage location.

[0046] In a possible implementation, performing correction processing according to the image data corresponding to any one of the regions to obtain a target sub-image corresponding to any one of the regions includes:

[0047] According to any two adjacent sets of pixel coordinates, interpolate to obtain the coordinates of each pixel in the corresponding area;

[0048] Determine any pixel coordinate in any area, and determine a specific number of pixel coordinates corresponding to an interpolation range of the any pixel according to a preset interpolation rule;

[0049] At least according to the offset address corresponding to the image data sub-block and the coordinates of each pixel within the interpolation range of the any pixel, acquiring at least part of the data of the image data sub-block corresponding to each pixel within the interpolation range from the corresponding storage position of the first storage space;

[0050] Interpolate at least part of the image data sub-block corresponding to each pixel within the interpolation range to obtain the corrected image data corresponding to any pixel, until each pixel in any area obtains the corresponding corrected image data, thereby obtaining a target sub-image corresponding to any area.

[0051] A second aspect of the present application provides an image processing device, comprising:

[0052] A determination structure is used to determine at least two groups of pixel coordinates in the image to be processed, each group of pixel coordinates includes at least three pixel coordinates, the image to be processed is a distorted image, and each group of pixel coordinates corresponds to a distortion line of the image to be processed; based on any two adjacent groups of pixel coordinates, coordinate ranges of at least two regions in the image to be processed are determined in sequence, each region includes at least two pixels;

[0053] The correction structure is used to determine the coordinate range of an area each time, and to perform correction processing on the image data of each pixel in the coordinate range of the area to obtain a target image.

[0054] In a possible implementation, the determining structure includes:

[0055] A coordinate acquisition module, used to acquire at least two groups of pixel coordinates in the image to be processed, wherein the image to be processed includes at least two groups of pixel coordinates, each group of pixel coordinates includes at least three pixel coordinates, the image to be processed is a distorted image, and each group of pixel coordinates corresponds to a distortion line of the image to be processed;

[0056] An image data acquisition module, used for determining the coordinate ranges of at least two regions in the image to be processed in turn according to any two adjacent groups of pixel coordinates, and storing the image data of the corresponding coordinate range in the image to be processed to the corresponding storage position in the first storage space according to the coordinate range of each of the regions;

[0057] The image acquisition module obtains pixel coordinates from the coordinate acquisition module each time, and pre-establishes a connection channel required for transmitting the pixel coordinates this time.

[0058] In a possible implementation, the method further includes:

[0059] A cache module is used to divide the first storage space into a plurality of storage locations, each storage location stores image data of an area, each storage location is divided into a specific number of storage units, each storage unit is used to store an image data sub-block, each image data sub-block is composed of a pixel matrix of a set size, and records the sorting sequence number and offset address of the image data sub-block stored in each storage unit, wherein the offset address represents the offset between the image data sub-block and a specific position in the area to which it belongs.

[0060] In a possible implementation, the correction structure includes:

[0061] A correction module, configured to interpolate coordinates of each pixel in a corresponding area based on any two adjacent sets of pixel coordinates; determine any pixel coordinate in any area, and determine a specific number of pixel coordinates corresponding to an interpolation range of the any pixel based on a preset interpolation rule, wherein the interpolation range includes a specific number of pixels, and the any pixel is located in a central area of ​​the interpolation range; determine a storage position corresponding to each pixel in the interpolation range in a first storage space based at least on an offset address corresponding to the image data sub-block and the coordinates of each pixel in the interpolation range of the any pixel, wherein the storage position stores the image data sub-block; obtain at least part of the image data sub-block corresponding to each pixel in the interpolation range from the first storage space; interpolate based on at least part of the image data sub-block corresponding to each pixel in the interpolation range to obtain corrected image data corresponding to the any pixel, until each pixel in the image to be processed obtains corresponding corrected image data, thereby obtaining a target image;

[0062] The correction module obtains pixel coordinates from the coordinate acquisition module each time and pre-establishes a connection channel required for transmitting the pixel coordinates this time.

[0063] A third aspect of the present application provides a computer program product, comprising computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the image processing method of the first aspect or any implementation of the first aspect.

[0064] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0065] The memory is used to store computer programs;

[0066] The processor is used to execute the computer program so that the electronic device can implement the image processing method of the first aspect or any implementation manner of the first aspect.

[0067] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the image processing method of the first aspect or any implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0069] Figure 1 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0070] Figure 2 is a schematic diagram of the coordinate range of the region in the image to be processed provided in an embodiment of the present application;

[0071] Figure 3 is a schematic diagram of a process for determining at least two groups of pixel coordinates in an image to be processed provided by an embodiment of the present application;

[0072] Figure 4 is a distortion schematic diagram provided in an embodiment of the present application;

[0073] Figure 5 It is a schematic diagram of the comparison between a normal image and a distorted image provided by an embodiment of the present application;

[0074] Figure 6 It is a schematic diagram of the SRAM storage address provided in an embodiment of the present application;

[0075] Figure 7 It is a schematic flow chart of determining the coordinate range of a region each time, performing correction processing on the image data of each pixel in the coordinate range of the region, and obtaining a target sub-image corresponding to the region, provided by an embodiment of the present application;

[0076] Figure 8 is a schematic diagram of a first storage space provided in an embodiment of the present application;

[0077] Fig. 9 It is a schematic flow chart of determining the coordinate range of a region each time, performing correction processing on the image data of each pixel in the coordinate range of the region, and obtaining a target sub-image corresponding to the region, provided by an embodiment of the present application;

[0078] Fig.10 is a schematic diagram of a process for storing the image data into a target storage location in the first storage space provided by an embodiment of the present application;

[0079] Fig.11 is a schematic diagram of the overlapping range provided in the embodiment of the present application;

[0080] Fig.12 is a schematic diagram of a process of obtaining image data corresponding to each pixel within a coordinate range of any area from a second storage space provided by an embodiment of the present application;

[0081] Fig.13 is a schematic diagram of a process of obtaining second image data corresponding to each pixel outside the overlapping range in the second area from the second storage space provided by an embodiment of the present application;

[0082] Fig.14 is a schematic diagram of a process for storing the image data into a target storage location in the first storage space provided by an embodiment of the present application;

[0083] Fig.15 It is a schematic diagram of the correspondence between the serial number and the offset address of each bank in a single area provided by an embodiment of the present application;

[0084] Fig.16 It is a schematic diagram of dividing a single area into several bank partial areas provided in an embodiment of the present application;

[0085] Fig.17 It is a schematic diagram of a flow chart of performing correction processing according to the image data corresponding to any area to obtain a target sub-image corresponding to any area provided by an embodiment of the present application;

[0086] Fig.18 is a schematic diagram of pixel reference addresses corresponding to each storage location in the first storage space provided in an embodiment of the present application;

[0087] Fig.19 is a schematic diagram of the relationship between the image data sub-block and the interpolation range provided in an embodiment of the present application;

[0088] Fig. 20 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0089] Fig.21 is a system block diagram of an image processing method provided by an embodiment of the present application;

[0090] Fig. 22 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0091] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0092] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0093] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0094] Figure 1 is a flowchart of an image processing method provided by an embodiment of the present application, such as Figure 1 As shown, an image processing method provided in an embodiment of the present application may include steps 101 to 103, and these steps are described in detail below.

[0095] 101. Determine at least two groups of pixel coordinates in an image to be processed, each group of pixel coordinates includes at least three pixel coordinates, the image to be processed is a distorted image, and each group of pixel coordinates corresponds to a distortion line of the image to be processed;

[0096] The image to be processed is a distorted image. The method in this embodiment is to process the distorted image to eliminate the distortion and obtain a normal image.

[0097] The image to be processed is an image that has been sparsely processed, and the present application performs processing based on pixel coordinates in the sparsely processed image.

[0098] Wherein, a plurality of distortion lines are determined in the image to be processed, the distortion lines are determined by a baseline corresponding to the image to be processed, a plurality of pixel coordinates are determined in sequence in the distortion lines, and the intervals of each pixel coordinate in the baseline are the same.

[0099] The pixel coordinates contained in each pixel coordinate are determined according to the horizontal and vertical reference lines corresponding to the image to be processed.

[0100] In a possible implementation, two adjacent distortion lines may be determined first, and a pixel coordinate may be obtained in each distortion line in turn to form a pair of pixel coordinates. Since the four pixel coordinates included in the two pairs of pixel coordinates may form an area, the subsequent step 102 may be directly executed after the two pairs of pixel coordinates are determined to continue to obtain a third pair of pixel coordinates. The third pair of pixel coordinates and the second pair of pixel coordinates may form another area, and so on.

[0101] In a possible implementation, the determined at least two sets of pixel coordinates are extracted from the distorted image, and may be selected based on the baseline corresponding to the image to be processed and the corresponding position coordinates in the image to be processed. Figure 3 The process of determining at least two sets of pixel coordinates is described in detail.

[0102] 102. According to any two adjacent groups of pixel coordinates, determine the coordinate ranges of at least two regions in the image to be processed in sequence, each region including a number of pixels;

[0103] Among them, since the comparison of four pixel coordinates can respectively determine the maximum and minimum values ​​of the horizontal and vertical coordinates, a region (mesh) can be determined based on the four pixel coordinates. Correspondingly, multiple pixel coordinates can be determined along the determined multiple distortion lines in the image to be processed. Any two adjacent coordinates on any two distortion lines can form a region, which contains a number of pixels.

[0104] In a possible implementation, two adjacent distortion lines may be determined first, and a pixel coordinate may be obtained in each distortion line in turn to form a pair of pixel coordinates. Since the four pixel coordinates included in the two pairs of pixel coordinates may form an area, the subsequent step 103 may be directly executed after the two pairs of pixel coordinates are determined, and the third pair of pixel coordinates may be obtained. The third pair of pixel coordinates and the second pair of pixel coordinates may form an area. The subsequent step 103 may be executed based on the area until the two adjacent distortion lines are completed, and then another adjacent distortion line may be determined based on one of the distortion lines to form two new adjacent distortion lines. Then, multiple pairs of pixel coordinates may be determined from the two adjacent distortion lines, and so on, until all areas in the image to be processed are corrected, and the image processing process is completed.

[0105] Among them, according to any two adjacent distortion lines and any two pixel coordinates adjacent thereto, another area can be determined, and the two adjacent distortion lines can be used as a reference to determine multiple areas involved in the two distortion lines in sequence. Subsequently, correction processing can be performed on the areas involved in the two distortion lines first, and then processing can be performed on the areas involved in subsequent distortion lines.

[0106] Figure 2It is a schematic diagram of the coordinate range of the region in the image to be processed provided in an embodiment of the present application, and the schematic diagram includes a distortion line A and a distortion line B. Two pixel coordinates a1 and a2 are determined in the distortion line A, and two pixel coordinates b1 and b2 are determined in the distortion line B. The minimum / maximum value of the horizontal coordinate among the four coordinates a1, a2, b1 and b2 is used as the minimum / maximum value of the horizontal coordinate of the corresponding region, and the minimum / maximum value of the vertical coordinate among the four coordinates is used as the minimum / maximum value of the vertical coordinate of the corresponding region, and a region 201 including a1, a2, b1 and b2 is determined.

[0107] 103. Whenever the coordinate range of a region is determined, the image data of each pixel in the coordinate range of the region is corrected to obtain a target sub-image corresponding to the region.

[0108] Wherein, each time the coordinate range of a region in the image to be processed is determined, a correction process is performed on the region to obtain a target sub-image corresponding to the region.

[0109] Among them, the correction processing can be performed based on the image data of each pixel in the coordinate range of the area. The image data is the image data of each pixel in the coordinate range of the image to be processed, and the target sub-image corresponding to the area is corrected.

[0110] The image data involved in the embodiments of the present application may refer to the pixel value of a pixel.

[0111] Among them, each area in the image to be processed is corrected to obtain a plurality of target sub-images, and the plurality of target sub-images are spliced ​​and fused together to obtain a target image corresponding to the image to be processed.

[0112] In a possible implementation, after each region is determined, the image data of the region can be obtained from the external second storage space and stored locally, so that the correction processing can be performed later using the locally stored data. Figure 7 This process is described in detail in .

[0113] In a possible implementation, after acquiring the image data corresponding to any area, the image data is stored in the first local storage space, and the correction processing can be directly performed using the locally stored data. Fig. 9 This process is described in detail in .

[0114] In order to implement the image distortion processing algorithm, the cache implementation method is adopted. The original image is written into DDR in block form. According to the map lut sparse coordinates pre-calculated by SW, data is obtained from DDR with block as the cacheline size, and cache management is performed in the module. The neighborhood data that needs to be interpolated is reorganized from the cache to complete the bicubic pixel interpolation.

[0115] In this embodiment, in the distorted image to be processed, multiple groups of pixel coordinates corresponding to the distortion lines are determined, and each group of pixel coordinates is at least three pixel coordinates determined on the corresponding distortion line; the coordinate ranges of at least two regions in the image to be processed are determined in turn using any two adjacent groups of coordinate pixels; each time the coordinate range of a region is determined, correction processing is performed on the image data of each pixel within the coordinate range of the region to obtain a target sub-image corresponding to the region. In this process, based on the distortion line, two adjacent pixel coordinates are respectively determined on any two adjacent distortion lines to obtain four pixel coordinates, and a region is determined based on the four pixel coordinates. After a region is determined, correction processing can be performed on the image data of each pixel in the region. There is no need to store a row of image data in the same position. Only a storage position needs to be set for each region, and the image data of each determined region can start to be corrected. There is no need to store a row of image data completely before correction processing can be performed, thereby reducing the requirement for the size of the storage space.

[0116] Figure 3 It is a schematic diagram of a process of determining at least two groups of pixel coordinates in an image to be processed provided by an embodiment of the present application, which may include steps 301 to 303, and these steps are described in detail below.

[0117] 301. Determine a distortion function of the image to be processed;

[0118] The distortion of the image to be processed may be caused by the use of a wide-angle lens or a telephoto lens for shooting, which forms a barrel distortion (Barrel) or a pincushion distortion (Pincushion) accordingly.

[0119] Figure 4 4 is a schematic diagram of distortion provided by an embodiment of the present application, which includes barrel distortion 401 and pincushion distortion 402. The barrel distortion 401 is that the straight lines in the picture are sunken toward the center, forming a barrel-shaped effect, which is generally due to the use of a wide-angle lens, especially when shooting buildings or landscapes; the pincushion distortion 402 is that the straight lines in the picture are convex toward the center, forming a pillow-shaped effect, which is generally due to the use of a telephoto lens, especially when shooting distant objects.

[0120] In one possible implementation, a distortion test card can be used to test the lens process. The lens is used to take a picture of the distortion test card to determine the degree and direction of curvature in the picture. The degree and direction of curvature are used to calculate the distortion function of the picture. The distortion function is the distortion function of the lens. All pictures taken by the lens use the distortion function. The distortion function of the camera can be determined as the distortion function of the image to be processed.

[0121] 302. Determine at least two groups of reference pixel coordinates based on a pixel matrix corresponding to the image to be processed, where the reference pixel coordinates are pixel coordinates extracted at set intervals in a reference line corresponding to the image to be processed;

[0122] The image to be processed includes a pixel matrix composed of a number of pixels, and the number of pixels included in the rows and columns of the pixel matrix is ​​preset.

[0123] As an example, the image to be processed contains a 1908×1920 pixel matrix.

[0124] The reference line may be a line set along any one of two intersecting edges of a normal image (without distortion) in a normal image. The line may be set along an edge in the up-down direction or in the left-right direction.

[0125] Among them, multiple baselines can be determined according to the distortion function, and different baselines correspond to different distortion lines, so as to perform subsequent correction processing according to the distortion of each distortion line.

[0126] refer to Figure 4 ,Should Figure 4 In the two distortion cases shown in , each line is used as a distortion line, and each distortion line can be obtained by distortion based on a reference line in a normal image.

[0127] In a possible implementation, multiple pixel coordinates may be extracted at set intervals for a reference line in a normal image range, and the pixel coordinates have corresponding coordinates in the image to be processed, and the two are associated based on the distortion situation.

[0128] In a possible implementation, a normal image may be predetermined to determine a plurality of reference lines in both horizontal and vertical directions according to a set distance, and the intersection position of each reference line is used as a reference pixel coordinate.

[0129] 303. Determine at least two groups of pixel coordinates corresponding to the at least two groups of reference pixel coordinates respectively according to the distortion function.

[0130] The distortion function is used to perform sparse processing on the image to be processed, and then the correction process is implemented by interpolation using at least two groups of determined pixel coordinates.

[0131] After the coordinates of each reference pixel are determined, the pixel coordinates corresponding to each reference pixel coordinate in the image to be processed are determined according to the distortion function.

[0132] In a possible implementation, after the corresponding pixel coordinates are determined for all the reference pixel coordinates, the at least two groups of pixel coordinates are used to perform a subsequent process of determining the coordinate range of each region.

[0133] In a possible implementation, one side of a normal image can be used as a reference, and two adjacent reference lines can be used as the reference in sequence to determine the coordinates of each reference pixel on the two adjacent reference lines. After the reference pixel coordinates are determined, the pixel coordinates corresponding to the reference pixel coordinates in the image to be processed are determined. Once a pair of pixel coordinates is determined, the pair of pixel coordinates is directly used to perform the subsequent process of determining the coordinate range of each area.

[0134] Among them, multiple reference pixels can be determined on each reference line from left to right, the pixel coordinates corresponding to each reference pixel in the image to be processed can be determined, and the pixel coordinates corresponding to each reference pixel on each reference line in the image to be processed can be determined in order from top to bottom.

[0135] As an example, in a normal image, a pixel coordinate is determined for every 15 pixels in each row / column, and multiple reference pixel coordinates are determined. For example, (0, 0), (0, 15), and (0, 30) are determined as reference pixel coordinates, and the pixel coordinates corresponding to the reference pixel coordinates in the distorted image are determined using the distortion function. The pixel coordinates corresponding to the reference pixel coordinates in the distorted image are used as the basis for subsequent processing. The reference pixel coordinates are regularly distributed in the image, and the reference pixel coordinates are used to determine the corresponding pixel coordinates in the distorted image. The pixel coordinates are then used for correction processing to ensure the accuracy of the processing.

[0136] Figure 5 It is a schematic diagram of the comparison between a normal image and a distorted image provided by an embodiment of the present application. The normal image 501 corresponds to the distorted image 502. Black dots are used to represent reference pixel coordinates in the normal image. Four reference pixel points 1-4 are listed in the figure. There are corresponding pixel coordinates in the distorted image. The figure has corresponding four pixels 1-4.

[0137] In a possible implementation, multiple pixel coordinates corresponding to a distortion line may be taken as a group, pixel coordinates of a specific group may be predetermined, and stored in groups.

[0138] When storing the pixel coordinates of the specific group, the storage space SRAM address can be divided into three parts, each part storing a group of pixel coordinates. Of course, the number of parts into which the storage space is divided can be set according to actual conditions, as long as each part can store a group of pixel coordinates.

[0139] As an example, the image to be processed is 1000×1000 pixels, and 100×100 pixel coordinates are determined therefrom to achieve sparseness. The 100 pixel coordinates of the first three rows (three groups) are stored in order in respective parts of the SRAM. When the 100 pixel coordinates of the fourth row (fourth group) are subsequently determined, they can be stored in the part corresponding to the pixel coordinates of the first row (first group), thereby realizing the recycling of storage space.

[0140] Figure 6 : is a schematic diagram of the SRAM storage address provided in an embodiment of the present application. The storage address of the SRAM includes 0~959 bits (bit), wherein the storage address of the SRAM is divided into three parts, each with 320 bits, namely 0~319, 320~639, and 640~959, and each part stores a set of pixel coordinates.

[0141] Subsequently, the stored pixel coordinates of each row may be used to sequentially determine an area so as to perform correction on the area.

[0142] In this embodiment, the distortion function of the image to be processed is determined; based on the pixel matrix corresponding to the image to be processed, at least two groups of reference pixel coordinates are determined, and the reference pixel coordinates are pixel coordinates extracted at set intervals in the reference line corresponding to the image to be processed; according to the distortion function, at least two groups of pixel coordinates corresponding to the at least two groups of reference pixel coordinates are respectively determined, thereby realizing the process of determining at least two groups of pixel coordinates in the image to be processed by using the distortion function. Since the pixel coordinates are determined based on the reference line corresponding to the image to be processed and the pixel coordinates extracted at set intervals, the pixels corresponding to each pixel in the normal image are regularly arranged, which provides a basis for subsequent correction processing and improves the accuracy of correction.

[0143] Figure 7 The embodiment of the present application provides a flow chart of determining the coordinate range of a region, correcting the image data of each pixel in the coordinate range of the region, and obtaining a target sub-image corresponding to the region, which may include steps 701 to 705. These steps are described in detail below.

[0144] 701. Obtain target image data corresponding to any area in the image to be processed according to the coordinate range of any area;

[0145] After the coordinate range of any region in the image to be processed is determined, the target image data corresponding to the any region can be obtained from the storage space storing the image data to be processed.

[0146] Among them, when storing the image data to be processed, the image data of each pixel is stored in combination with the coordinate position of each pixel. Accordingly, based on the coordinate range in the area, the image data of each pixel within the coordinate range can be obtained in the storage space of the image data to be processed.

[0147] As an example, the coordinate range is [0, 10] on the x-axis and [0, 15] on the y-axis, and image data of each pixel covered by the coordinate range is obtained in the storage space.

[0148] 702. Send the target image data to a first storage space;

[0149] The first storage space is used to cache image data to be corrected.

[0150] In a possible implementation, the first storage space is divided into a plurality of storage locations, each storage location can store image data of a region, and the image data in the region can be stored corresponding to pixels in the image to be processed.

[0151] After the target image data corresponding to any area is obtained, it is sent to the first storage space for storage.

[0152] The first storage space is divided into a plurality of storage locations, but the number of the storage locations is less than the total number of regions in the image to be processed, and the storage locations need to be allocated to reduce the demand for the storage space.

[0153] 703. Based on the fact that there are remaining storage locations in the first storage space, store the target image data in the remaining storage locations;

[0154] When there are remaining storage locations in the first storage space, the image data can be stored in the remaining storage locations in sequence.

[0155] As an example, the first storage space has 10 storage locations, and currently the 1st to 8th storage locations are occupied. After the image data of a certain area is obtained, it is stored in the 9th storage location.

[0156] It should be noted that, since the area determined in the image to be processed is continuous, when storing the image data of the image to be processed, the storage location is also determined in a continuous order for storage, so as to facilitate querying the image data in the subsequent correction process.

[0157] 704. Based on the fact that there is no remaining storage position in the first storage space, determine a target storage position, store the target image data in the target storage position, and the image data stored in the target storage position has been corrected;

[0158] When there are no remaining storage locations in the first storage space, one of the occupied storage locations is determined as a target storage location, the image data stored in the target storage location has been corrected, and the target image data corresponding to any area is stored in the target storage location to achieve reuse of the storage location.

[0159] As an example, the first storage space has 10 storage locations, and currently all 10 storage locations are occupied. After obtaining the image data of a certain area, it is determined that the storage location of the image data of the previous area is the 5th storage location, and the image data of the current area is stored in the next storage location (the 6th storage location). Since the process of storing image data and correcting is executed simultaneously, as long as the image data in the storage location has completed the correction processing, it can be reused.

[0160] 705. Perform correction processing according to the target image data in the first storage space to obtain a target sub-image corresponding to any area.

[0161] The correction process is performed in turn according to the image data stored in the storage locations in the first storage space to obtain the target sub-image corresponding to each area.

[0162] After the image data of the corresponding area is stored in the first storage position in the first storage space, the correction process can be started, and the storage position corresponding to the image data after the correction process can be reused by the image data of other areas obtained subsequently.

[0163] Figure 8 This is a schematic diagram of the first storage space provided by an embodiment of the present application, in which the first storage space is divided into n storage locations, which are named storage location 0, storage location 2, ..., storage location n-2, and storage location n-1. After the first region is determined in the image to be processed, the image data of each pixel corresponding to the coordinate range of the first region in the image data to be processed is obtained, and stored in storage location 0; after the second region is determined in the image to be processed, the image data of each pixel corresponding to the coordinate range of the second region in the image data to be processed is obtained, and stored in storage location 1; and so on, the image data of each pixel corresponding to the coordinate range of n regions is obtained, and stored in storage location n; when the n+1th region is determined in the image to be processed, the image data corresponding to the n+1th region is obtained, and stored in storage location 0, and the n storage locations are recycled.

[0164] In a possible implementation, the storage locations are divided for the first storage space by analyzing factors such as throughput, latency of returned data, and outstanding cache capacity of the first storage space to determine the number of storage locations divided in the first storage space.

[0165] By storing image data separately in regional meshes, it is possible to replace the space of the mesh region size in the image to be processed, which can simplify the implementation of back pressure between the spatial replacement and interpolation calculation of each region, and can accurately design the delay of the bus return data.

[0166] The image data of each area can be stored according to the number of PPC (Pixel Per Clock, the number of pixels transmitted in each clock cycle). By increasing the number of storage locations and reducing the width of each storage location, the overall space is ensured not to increase. At the same time, the throughput parameters can be configured.

[0167] In this embodiment, based on the coordinate range of any area, target image data corresponding to any area in the image to be processed is obtained; the target image data is sent to the first storage space; based on the fact that there are remaining storage locations in the first storage space, the target image data is stored in the remaining storage locations; based on the fact that there are no remaining storage locations in the first storage space, the target storage location is determined, and the target image data is stored in the target storage location, and the image data stored in the target storage location has been corrected; correction processing is performed based on the image data in the first storage space to obtain a target sub-image corresponding to any area, and the first storage space is divided into multiple storage locations, each storage location stores image data corresponding to an area, and multiple storage locations can be reused, thereby improving the utilization rate of the storage space and reducing the restrictions on the size of the storage space.

[0168] Fig. 9 The embodiment of the present application provides a flow chart of determining the coordinate range of an area each time, correcting the image data of each pixel in the coordinate range of the area to obtain a target sub-image corresponding to the area, which may include steps 901 to 903, and these steps are described in detail below.

[0169] 901. When the coordinate range of any area is determined, image data corresponding to each pixel within the coordinate range of any area is obtained from a second storage space, and the second storage space stores image data of all pixels in the image to be processed;

[0170] Among them, the second storage space can be a DDR in an electronic device, which stores image data of all pixels in the image to be processed. Moreover, the second storage space is stored based on the coordinates of the pixel points, and the coordinates of the pixel points can be used to query the image data corresponding to the pixel.

[0171] After the coordinate range of the determined area is determined, image data corresponding to each pixel within the coordinate range of the area is obtained from the second storage space.

[0172] In a possible implementation, the coordinates of each pixel within the coordinate range are determined according to the coordinate range of the region, and then the image data of each pixel is obtained from the second storage space using the coordinates of each pixel.

[0173] 902. Store the image data in a target storage location in the first storage space, where the target storage location is a storage location allocated to the any area;

[0174] The image data corresponding to each pixel within the coordinate range of a region is stored in a target storage position in the first storage space.

[0175] The method for determining the target storage location can refer to the aforementioned Figure 7 It is explained in detail in this embodiment and will not be described in detail in this embodiment.

[0176] After the image data corresponding to each pixel in a determined area is stored in the target storage location, the correction process can be started.

[0177] In one possible implementation, since the image data corresponding to each pixel in each area is stored in each storage location in sequence, generally, the areas are determined in sequence, and correction processing is performed on the corresponding area using the image data in each storage location in the order in which the image data is stored.

[0178] In a possible implementation, when the coordinate ranges of two adjacent regions overlap, the image data of the overlapping ranges can be acquired once and stored multiple times, and the image data of the overlapping ranges can be stored in the storage locations of the two regions respectively. Fig.10 The process of storing the acquired image data into the target storage location is described in detail.

[0179] 903. When image data of any region is stored in the first storage space, correction processing is performed according to the image data corresponding to the region to obtain a target sub-image corresponding to the region.

[0180] Wherein, each time image data of a region is stored in the first storage space, a correction process for the region is started, specifically, correction is performed using the image data corresponding to the region to obtain a target sub-image corresponding to the region.

[0181] In a possible implementation, in the process of correcting the image data corresponding to any area, it is necessary to interpolate the pixel coordinates determined above to obtain the pixel coordinates in the area, and then obtain the image data corresponding to the pixel coordinates obtained by the interpolation from the corresponding storage position of the first storage space. This process is subsequently performed. Fig.17 Detailed description in.

[0182] In this embodiment, when the coordinate range of any area is determined, the image data corresponding to each pixel within the coordinate range of any area is obtained from the second storage space storing the image data of all pixels in the image to be processed; the image data is stored in the target storage position in the first storage space, and the target storage position is the storage position allocated for the any area; each time the image data of any area is stored in the first storage space, correction processing is performed based on the image data corresponding to the any area to obtain a target sub-image corresponding to the any area, so that each time the coordinate range of a region is determined, the image data corresponding to each pixel within the coordinate range is obtained and the image data is stored in the first storage space, and each time the image data of a region is stored in the first storage space, the image data of the region can be used for correction processing, so that a pipeline process of obtaining image data and using the image data for correction processing is realized, and there is no need to set a large first storage space, thereby reducing the size requirement of the first storage space for storing image data.

[0183] Fig.10 It is a flowchart of storing the image data into the target storage location in the first storage space provided by an embodiment of the present application, which may include steps 1001 to 1004, and these steps are described in detail below.

[0184] 1001. Store first image data corresponding to a first region in a first storage location in a first storage space, where the first storage location is a storage location allocated for the first region;

[0185] Among them, in the image to be processed, after determining a pixel coordinate on two adjacent distortion lines respectively, they are used as a pair of pixel coordinates. After determining the two pairs of pixel coordinates, an area can be determined based on the two pairs of pixel coordinates, and then the image data acquisition and correction process can be performed on the area.

[0186] After the three pairs of pixel coordinates are determined, an area can be determined based on any two adjacent pairs of pixel coordinates, and then two areas can be determined to determine the possible overlapping range of the two areas.

[0187] After the first area is determined, first image data corresponding to the first area is acquired from the second storage space, and the first image data is stored in the first storage position of the first storage space.

[0188] The first storage location is used to store the first area and the corresponding first image data.

[0189] After the second area is determined, it can be determined whether the second area overlaps with the previous area. If so, the subsequent step 1002 is performed; if not, the second image data corresponding to the second area is directly obtained from the second storage space.

[0190] Since the pixel coordinates are set using distortion lines in this embodiment, there may be overlaps between the various regions.

[0191] 1002. If the coordinate range of the first region and the coordinate range of the second region have an overlapping range, store the first image data corresponding to the overlapping range in a second storage location in the first storage space, where the second storage location is a storage location allocated for the second region;

[0192] The first storage space is divided into a plurality of storage locations, the first storage location is used to store the first image data corresponding to the first area, and the second storage location is used to store the second image data corresponding to the second area.

[0193] If the first area and the second area overlap, the overlapping range of the coordinate range of the first area and the coordinate range of the second area is determined.

[0194] The overlapping range may be determined by using the upper and lower limits of the first area coordinate range and the upper and lower limits of the second area coordinate range.

[0195] The first area includes the overlapping range, and after obtaining the first image data corresponding to the first area, the first image data of the overlapping range is stored in the second storage space.

[0196] Among them, since the image data in the overlapping range is the same, it can be acquired from the second storage space only once and stored in multiple places without repeated acquisition.

[0197] 1003. Obtain second image data corresponding to each pixel outside the overlapping range in the second area from the second storage space;

[0198] When acquiring the second image data corresponding to the second area from the second storage space, the acquired part in the overlapping range is removed, and only the second image data corresponding to each pixel outside the overlapping range in the second area is acquired.

[0199] Among them, the image data corresponding to each pixel in the overlapping range is stored as both the first image data and the second image data in a first storage location corresponding to the first area and a second storage location corresponding to the second area, respectively. It is acquired once and stored multiple times without repeated acquisition of the same image data, thereby reducing the bandwidth occupied by the acquisition and improving the acquisition speed.

[0200] 1004. Store the second image data into the second storage location.

[0201] Among them, the acquired second image data is stored in a second storage location corresponding to the second area, and the second storage location stores image data corresponding to the overlapping range and image data corresponding to the non-overlapping range in the second area. The two parts of data are combined to obtain image data of all pixels corresponding to the second area.

[0202] In one possible implementation, when it is determined that two or three adjacent areas have overlapping ranges, the current area division is more appropriate. If four or more adjacent areas have the same overlapping range, it indicates that the area division is inappropriate and the position of the pixel points needs to be adjusted. The set interval can be widened.

[0203] Fig.11 It is a schematic diagram of the overlapping range provided in an embodiment of the present application, which includes four areas, namely areas 1101-1104. Lines of different colors are used in the figure to represent the ranges of different areas. The overlapping range of any two areas is determined based on the maximum and minimum values ​​of the pixel coordinates in each area.

[0204] Among them, region 1101 and region 1102 have overlapping range 11012, region 1101 and region 1103 have overlapping range 11013; region 1102 and region 1103 also have overlapping range 11023, region 1102 and region 1104 have overlapping range 11024; region 1103 and region 1104 also have overlapping range 11043. Region 1101 corresponds to storage position 1 of the first storage space, region 1102 corresponds to storage position 2 of the first storage space, region 1103 corresponds to storage position 3 of the first storage space, and region 1104 corresponds to storage position 4 of the first storage space.

[0205] Correspondingly, when obtaining the first image data corresponding to area 1101, the first image data corresponding to area 1101 is stored in storage location 1, which is used to store the image data corresponding to area 1101; the first image data of the overlapping range 11012 is stored in storage location 2, and the first image data of the overlapping range 11013 is stored in storage location 3.

[0206] When obtaining the second image value corresponding to the area 1102, only the data of the area 1102 except the overlapping range 11012 is obtained and stored in the storage location 2; and the data of the overlapping range 11023 except the overlapping range 11013 is stored in the storage location 3, and the data of the overlapping range 11024 is stored in the storage location 4;

[0207] Among them, when obtaining the third image data corresponding to area 1103, only the image data of area 1103 except the overlapping range 11023 and the overlapping range 11013 are obtained and stored in storage location 3; and, the image data of the overlapping range 11034 except the overlapping range 11024 is stored in storage location 4.

[0208] When acquiring the fourth image data corresponding to the area 1104 , only the image data of the area 1104 excluding the overlapping range 11024 and the overlapping range 11034 is acquired and stored in the storage location 4 .

[0209] In this embodiment, the first image data corresponding to the first area is stored in a first storage position in the first storage space, and the first storage position is a storage position allocated for the first area; if the coordinate range of the first area and the coordinate range of the second area have an overlapping range, the first image data corresponding to the overlapping range is stored in a second storage position in the first storage space, and the second storage position is a storage position allocated for the second area; the second image data corresponding to each pixel outside the overlapping range in the second area is obtained from the second storage space; the second image data is stored in the second storage position. The overlapping design between adjacent areas can achieve the purpose of reducing bandwidth, and can also ensure the integrity of the image data of the entire area, which is convenient for replacement.

[0210] Fig.12 It is a flowchart of obtaining image data corresponding to each pixel within the coordinate range of any area from the second storage space provided by an embodiment of the present application, which may include steps 1201 to 1204, and these steps are described in detail below.

[0211] 1201. Determine, based on the coordinate range of any area, an image data block set corresponding to any area, each image data block set includes at least two image data blocks, and each image data block includes pixel values ​​of each pixel in a pixel matrix of a preset size;

[0212] After the coordinate range of a region is determined based on four adjacent pixel coordinates, a set of image data blocks corresponding to the region is further determined.

[0213] In a possible implementation, the image to be processed is divided into a plurality of image data blocks according to a pixel matrix of a preset size, and the image data blocks may be used as transmission units for transmission in the second storage space.

[0214] The image data block may include pixel values ​​of each pixel in a 16×8 block, and the preset size pixel matrix is ​​16×8.

[0215] Among them, a region is a rectangle or a square, and its coordinate range may include multiple image data blocks. One region corresponds to an image data block set, and the image data block set includes multiple image data blocks.

[0216] 1202. Generate a first request according to the image data block set, where the first request is used to request each image data block in the image data block set from a second storage space;

[0217] After the image data block set corresponding to the coordinate range of any area is determined, a first request is generated according to the image data block set to request each image data block in the image data block set from the second storage space.

[0218] After receiving the first request, the second storage space sequentially sends each image data block requested by the first request to the first storage space for storage.

[0219] In a possible implementation, the image to be processed is divided into several blocks with identifiers, such as numbers. After the coordinate range of the region is determined, the identifiers of the multiple blocks corresponding to the coordinate range can be determined, and the first request is generated using the identifiers of the multiple blocks. Correspondingly, the identifiers of each block are also recorded in the second storage space, and the image data of the corresponding block is obtained using the identifiers.

[0220] 1203. Receive each image data block in the image data block set fed back by the second storage space.

[0221] The second storage space is received and fed back to the image data block, and the image data block is stored in the corresponding storage position in sequence.

[0222] In a possible implementation, a system level cache (SLC) is set between the second storage space and the first storage space, or a second level general SLC is added separately to implement data transfer between DDR and SRAM, thereby reducing DDR bandwidth consumption.

[0223] In this embodiment, after the coordinate range of any area is determined, a set of image data blocks corresponding to the any area is determined, each image data block set includes at least two image data blocks, and each image data block includes pixel values ​​of each pixel in a pixel matrix of a preset size; a first request is generated based on the image data block set, and the first request is used to request each image data block in the image data block set from the second storage space; each image data block in the image data block set fed back by the second storage space is received, and a plurality of image data blocks included in the coordinate range of any area are determined using image data blocks composed of pixel values ​​of each pixel in a pixel matrix of a preset size, and a first request corresponding to the plurality of image data blocks is generated to request the second storage space to provide the plurality of image data blocks, thereby realizing a process of obtaining image data of any area from the second storage space.

[0224] Fig.13 It is a flowchart of obtaining the second image data corresponding to each pixel outside the overlapping range in the second area from the second storage space provided by an embodiment of the present application, which may include steps 1301 to 1304, and these steps are described in detail below.

[0225] 1301. Determine an initial image data block set corresponding to the second area according to a coordinate range of the second area;

[0226] After the coordinate range of the second area is determined, a plurality of sets of image data blocks included in the second area are determined based on the coordinate range.

[0227] Among them, the explanation of the determination process can refer to the aforementioned Fig.12 Step 1201 is not described in detail here.

[0228] 1302. Determine a set of overlapping image data blocks corresponding to the overlapping range;

[0229] After determining the overlapping range of the first area and the second area, a set consisting of image data blocks corresponding to the overlapping range is determined, and the data blocks corresponding to the overlapping range constitute an overlapping image data block set.

[0230] As an example, the first area corresponds to an image data block set, including 16 image data blocks, and the second area corresponds to an image data block set also including 16 image data blocks. There are 4 overlapping image data blocks between the two. The 16 image data blocks of the first area have been obtained from the second storage space and are all stored in the storage position corresponding to the first area, and the 4 overlapping image data blocks have been stored in the storage position corresponding to the second area.

[0231] At this time, the coordinate range of the second area is determined, and then the image data of each pixel in the second area is obtained.

[0232] 1303. Generate a second request according to the initial image data block set and the overlapping image data block set, wherein the second request is used to request a target image data block set from the second storage space, and the target image data block set and the overlapping image data block set constitute the initial image data block set;

[0233] Among them, a second request is generated based on the initial image data block set and the overlapping image data block set, the initial image data block set corresponds to the data blocks contained in the second area as a whole, and the overlapping image data block set corresponds to the overlapping range, and the difference between the two is the image data blocks that have not yet been obtained in the second area.

[0234] Correspondingly, the second request is a request to obtain the image data block that has not yet been obtained in the second area.

[0235] In a possible implementation, an initial request can be first generated based on the initial image data block set, and the initial request is used to request each image data block in the initial image data block set from the second storage space; after determining the overlapping image data block set corresponding to the overlapping range, the initial request is adjusted based on the overlapping image data block set to obtain the second request.

[0236] In a possible implementation, a target data block set may be determined based on the initial image data block set and the overlapping image data block set. The target data block set is image data blocks that have not yet been obtained in the second area. Then, a second request is generated based on the target data block set.

[0237] 1304. Receive each image data block in the target image data block set fed back by the second storage space.

[0238] The second storage space is received and fed back to the image data block, and the image data block is sequentially stored in the storage position corresponding to the second area.

[0239] The storage location corresponding to the second region also stores an overlapping image data block set, and the target image data block set is combined with the overlapping image data block set to obtain the image data block set of the second region.

[0240] In this embodiment, an initial image data block set corresponding to the second area is determined according to the coordinate range of the second area; an overlapping image data block set corresponding to the overlapping range is determined; a second request is generated based on the initial image data block set and the overlapping image data block set, and the second request is used to request a target image data block set from the second storage space, and the target image data block set and the overlapping image data block set constitute the initial image data block set; each image data block in the target image data block set fed back by the second storage space is received, so that image data in the remaining range of the overlapping area is obtained, thereby saving the bandwidth required for obtaining image data.

[0241] Fig.14 It is a flowchart of storing the image data into the target storage location in the first storage space provided by an embodiment of the present application, which may include steps 1401 to 1403, and these steps are described in detail below.

[0242] 1401. Divide each image data block in any area into a specific number of image data sub-blocks, each image data sub-block consisting of a pixel matrix of a set size;

[0243] After a region is determined, each image data block in the region is divided into a plurality of image data sub-blocks.

[0244] The image data sub-block may adopt a 4×4 pixel matrix, and the image data sub-block includes the pixel value of each pixel in the pixel matrix of the set size.

[0245] Among them, an image data block is a block containing a 16×8 pixel matrix, and an image data sub-block is a bank containing a 4×4 pixel matrix. Accordingly, a block is divided into 8 banks.

[0246] Correspondingly, each image data block in an area is divided into a specific number of image data sub-blocks bank.

[0247] 1402. Determine, in accordance with a preset sequence, an arrangement sequence of each image data sub-block in the any region and an offset address corresponding to each image data sub-block, wherein the sequence number of any image data sub-block is different from that of an adjacent image data sub-block, and the offset address represents an offset between the image data sub-block and a specific position in the any region;

[0248] After the image data block is divided into a plurality of image data sub-blocks, the arrangement order of each image data sub-block and the offset of the image data sub-block for a specific position in the region to which it belongs are determined according to a preset order.

[0249] Among them, when each image data sub-block is arranged according to a preset order, the sorting sequence number of any image data sub-block is different from that of any adjacent image data sub-block, thereby achieving decentralized storage of each image data sub-block and facilitating the protection of image data during subsequent interpolation processing.

[0250] The offset address may be an offset address of the image data sub-block relative to the minimum coordinate point in the corresponding area.

[0251] Among them, data reorganization is required from DDR data to SRAM data, converting the bit width data of DDR into the bit width data of SRAM.

[0252] The preset order corresponds to the rule of reorganization.

[0253] Fig.15 This is a schematic diagram of the correspondence between the serial number and offset address of each bank in a single area provided by an embodiment of the present application, wherein in the schematic diagram, a mesh is divided into 96 banks, of which 4 banks form a group, and their offset addresses relative to the minimum coordinate point in the area are the same. a_b in a represents the bank number, and b represents the offset address. The image data of each pixel in the area obtained from the DDR is stored in the corresponding bank in sequence.

[0254] Fig.16 1 is a schematic diagram of a single area divided into several banks provided in an embodiment of the present application, wherein the schematic diagram shows 32 banks obtained by dividing two block ranges in one area, which are 8 groups of banks 0-3. Fig.16 The 8 groups of bank0-3 in the above Fig.15 The two blocks in correspondence are the block in the first row and the first column and the block in the second row and the second column.

[0255] 1403. Store each image data sub-block in a target storage location according to the arrangement order of each image data sub-block in any area and the offset address.

[0256] Among them, according to the arrangement order of the image data sub-blocks determined in the aforementioned step 1402 and the corresponding offset addresses, the pixel positions in the image to be processed can be determined and stored in the data storage positions allocated to each area in the first storage space.

[0257] Among them, since the data storage location of the blocks in each area is regular, the complexity of interpolation calculation to access SRAM data can be simplified; and bank division is done well to ensure the throughput requirements of the configuration in any scenario.

[0258] Moreover, since blocks are divided into banks, in the subsequent interpolation process, processing requests for non-entire blocks can be initiated, which can further reduce the bandwidth required for the requests.

[0259] In this embodiment, each image data block in any region is divided into a specific number of image data sub-blocks, each image data sub-block is composed of a pixel matrix of a set size; the arrangement order of each image data sub-block in any region and the offset address corresponding to each image data sub-block are determined according to a preset order, and the sorting sequence number of any image data sub-block is different from that of the adjacent image data sub-block, and the offset address represents the offset between the image data sub-block and a specific position in any region; according to the arrangement order and offset address of each image data sub-block in any region, each image data sub-block is stored in a target storage location. By dividing the image data block block into smaller image data sub-block banks, and determining the arrangement order and offset address of each image data sub-block in the region according to a preset order, the image data sub-blocks in the region are stored in a dispersed manner, and the relationship between the image data and the pixel position of each sub-block is recorded by using the arrangement order of the sub-blocks and the corresponding offset address, so as to realize the data reorganization of the data obtained from the second storage space to the local first storage space.

[0260] Fig.17 The embodiment of the present application provides a flow chart of performing correction processing based on the image data corresponding to any area to obtain a target sub-image corresponding to the any area, which may include steps 1701 to 1704, and these steps are described in detail below.

[0261] 1701. According to any two adjacent sets of pixel coordinates, interpolate to obtain the coordinates of each pixel in the corresponding area;

[0262] Wherein, based on the at least two groups of pixel coordinates determined in the image to be processed, interpolation is performed respectively to obtain the coordinates of each pixel in the corresponding area.

[0263] The two groups of arbitrary adjacent pixel coordinates are pixel coordinates after sparse processing. In order to achieve correction for the image to be processed, the sparse pixel coordinates need to be interpolated to obtain the coordinates of all pixels of the image to be processed.

[0264] Among them, two adjacent pairs of pixel coordinates (four pixel coordinates) can be obtained from two adjacent groups of pixel coordinates. The two adjacent pairs of pixel coordinates can determine an area, which is used as the area to be interpolated. For the area to be interpolated, the aforementioned four pixel coordinates are used to perform interpolation calculation to obtain the coordinates of each pixel in the area to be interpolated.

[0265] In a possible implementation, the area to be interpolated may include 256 pixels. Using the two pairs of pixel coordinates mentioned above, each interpolated pixel coordinate is obtained in a 16×16 order until all pixel coordinates in the area are obtained.

[0266] 1702. Determine any pixel coordinate in any area, and determine a specific number of pixel coordinates corresponding to an interpolation range of the any pixel according to a preset interpolation rule;

[0267] The interpolation range is determined according to a preset interpolation rule.

[0268] In a possible implementation, the interpolation method may adopt bilinear interpolation.

[0269] Wherein, within the interpolation range of the arbitrary pixel, the arbitrary pixel is located in the central area of ​​the interpolation range.

[0270] As an example, the interpolation range can be 4×4, and an interpolation range including a certain pixel coordinate is determined based on the pixel coordinate. Specifically, it can be a combination of the left two rows / one row and the right one row / two rows of the pixel coordinate, as well as the upper two rows / one row and the lower two rows / two rows. The present application does not limit the specific relationship between the pixel coordinate used as the reference and the interpolation range. It is only necessary to ensure that the same interpolation rule is used in one image processing.

[0271] The interpolation range may be 4×4, and accordingly, the specific number is 15. By using the preset interpolation rule, 15 pixel coordinates around any pixel are interpolated in sequence, and 16 pixels within the interpolation range are obtained by combining them.

[0272] In one possible implementation, after obtaining the four pixel coordinates, only the minimum value of the area (the minimum value of the horizontal and vertical coordinates) can be determined as the subsequent reading reference line, and the data in the image data sub-block can be obtained from the first storage space based on the reading reference line to provide an image data basis for interpolation.

[0273] 1703. Obtain at least part of the image data sub-block corresponding to each pixel within the interpolation range from a corresponding storage position in the first storage space according to at least the offset address corresponding to the image data sub-block and the coordinates of each pixel within the interpolation range of the any pixel;

[0274] The above-mentioned embodiment records that, in the storage position of the first storage space, the storage rule of each data sub-block of the corresponding area is stored.

[0275] When obtaining corresponding image data from the first storage space for each pixel in the interpolation area, the image data sub-block to which the pixel belongs is considered, so as to obtain the image data sub-block to which it belongs from the first storage space according to its belonging relationship.

[0276] As an example, image data interpolation is performed based on a 4×4 pixel matrix. After a certain pixel is determined, image data of 16 pixels involved in the image data interpolation range are acquired.

[0277] In one possible implementation, since the coordinates of each pixel are known, the base address of the image data corresponding to each area recorded in the first storage space can be used to determine the area to which the pixel belongs; and then combined with the aforementioned offset address, the specific bank of the area to which the pixel belongs can be determined.

[0278] Fig.18 This is a schematic diagram of the pixel reference address corresponding to each storage position in the first storage space provided in an embodiment of the present application, wherein, in this schematic diagram, the first storage space includes 10 storage positions, corresponding to mesh0-9 respectively, wherein each storage position contains 24 image data sub-block banks, mesh0 includes bank0-bank23, mesh1 includes bank 24-bank47, and so on, mesh9 includes bank 216-bank239.

[0279] Among them, according to Fig.18 In the YC channel, the size of a single bank in the Y channel is 240×160 bits, there are 10 meshes, and each mesh has 24 banks, so 240 (24×10) is obtained. The 240 in the Y channel also represents the bit width of the SRAM.

[0280] The offset address is based on a vertex in bank0 of mesh0 in the image to be processed, and the vertex is the minimum coordinate pixel in bank0.

[0281] Among them, the coordinates of the pixel can be compared with the reference address in the reference address schematic diagram to determine the area to which the pixel belongs. Since general interpolation is performed using image data in the same mesh, it is sufficient to use only the coordinates of the pixel to be interpolated to determine the area to which it belongs, and then determine the image data sub-block to which the coordinates of each pixel within the interpolation range of the pixel to be interpolated belong in the area.

[0282] The offset between the coordinates of each pixel within the interpolation range and the specific position of the region to which it belongs is determined in turn, and the image data sub-block corresponding to the pixel can be determined by comparing the offset with the offset address corresponding to each image data sub-block recorded in the aforementioned method.

[0283] In a possible implementation, the specific position of the area may be the minimum coordinate point in the area.

[0284] Since the interpolation range includes multiple pixels, each pixel in the interpolation range may fall in the same image data sub-block, or may fall in two adjacent image data sub-blocks, or even in four adjacent image data sub-blocks.

[0285] Fig.19 1 is a schematic diagram of the relationship between the image data sub-block and the interpolation range provided by the embodiment of the present application. The schematic diagram shows an area involved in 16 image data sub-blocks (a total of 4 groups of bank0-4), each bank is a 4×4 pixel matrix, and the interpolation range is also 4×4. Interpolation range 1901 is located in bank1 in the first row and second column; interpolation range 1902 falls between bank3 in the second row and second column and bank1 and bank3 in the third row and second column, part of which is in bank1 and part of which is in bank3; interpolation range 1903 falls between bank2 in the second row and third column and bank3 in the second row and fourth column, part of which is in bank2 and part of which is in bank3; interpolation range 1904 falls between bank0 in the third row and third column, bank1 in the third row and fourth column, bank2 in the fourth row and third column, and bank3 in the fourth row and fourth column, and part of which is in bank0-3. The interpolation range is represented by a dotted box in the figure.

[0286] The image data sub-block uses a specific data storage format to store the image data of 16 (4×4) pixels in sequence.

[0287] Among them, if some pixels in the interpolation range belong to different image data sub-blocks, the above method can be used to obtain partial data from each image data sub-block; if all pixels in the interpolation range belong to the same image data sub-block, all data of the image data sub-block can be directly obtained from the data storage location.

[0288] In a possible implementation, since values ​​in two directions, H (horizontal) and V (vertical), are used during interpolation, corresponding interpolation coefficients in the H and V directions may be obtained according to the coordinate phases of pixels in the interpolation range.

[0289] If the pixels in the interpolation range belong to two adjacent image data sub-blocks, and the two image data sub-blocks are arranged in the left-right direction (such as Fig.19 1903), according to the interpolation coefficients in the H and V directions of the coordinate phase of the pixel, the specific position of the pixel in each image data sub-block is determined, and then the image data corresponding to the specific position is obtained.

[0290] If the pixels in the interpolation range belong to two adjacent image data sub-blocks, and the two image data sub-blocks are arranged in the vertical direction (such as Fig.19 1902), according to the interpolation coefficient in the V direction of the coordinate phase of the pixel, the specific position of the pixel in each image data sub-block is determined, and then the image data corresponding to the specific position is obtained.

[0291] If the pixels in the interpolation range belong to four adjacent image data sub-blocks (such as Fig.19 1904), according to the interpolation coefficients in the H and V directions of the coordinate phase of the pixel, the specific position of the pixel in each image data sub-block is determined, and then the image data corresponding to the specific position is obtained.

[0292] As an example, the image data sub-block storage space has 160 bits (0-159), where the (mn)th bit stores the sth pixel, and the (mn)th bit is the specific position of the pixel in the image data sub-block, where s represents any one of the 16 pixels, and the values ​​of m and n are set according to actual conditions and are not limited in this application. After determining that the sth pixel belongs to a certain interpolation range, the (mn) bit data is obtained from the image data sub-block storage space.

[0293] 1704. Interpolate at least part of the image data sub-block corresponding to each pixel in the interpolation range to obtain corrected image data corresponding to any pixel, until all pixels in any area obtain corresponding corrected image data, thereby obtaining a target sub-image corresponding to any area.

[0294] After obtaining the data corresponding to each pixel within the interpolation range, the data is used to perform interpolation to obtain the corresponding corrected image data.

[0295] The interpolation method may adopt bicubic interpolation.

[0296] In a possible implementation, the interpolation coefficients in the H and V directions obtained in the above steps are used to perform interpolation according to the bicubic algorithm to obtain the dedistorted corrected image data.

[0297] The above interpolation processing is performed on each pixel in the interpolation area to obtain corresponding corrected image data, and the corrected image data of each pixel are combined to obtain the target sub-image of the area.

[0298] In this embodiment, based on the multiple groups of pixel coordinates obtained after sparseness, any two adjacent groups of pixel coordinates are determined to interpolate to obtain the coordinates of each pixel in the corresponding area; using the pixel coordinates of any area, according to a preset interpolation rule, a specific number of pixel coordinates corresponding to the interpolation range of any pixel are determined; any pixel coordinate is determined in any area, at least based on the offset address corresponding to the image data sub-block and the coordinates of each pixel within the interpolation range of the any pixel, at least part of the data of the image data sub-block corresponding to each pixel within the interpolation range is obtained from the corresponding storage position of the first storage space; based on at least part of the data of the image data sub-block corresponding to each pixel within the interpolation range, interpolation is performed to obtain the corrected image data corresponding to the any pixel, until each pixel in the any area obtains the corresponding corrected image data, and the target sub-image corresponding to the any area is obtained, thereby realizing the process of interpolation correction for each area.

[0299] An image processing method provided by an embodiment of the present application is introduced above, and a device for executing the above image processing method will be introduced below.

[0300] See also Fig. 20 , Fig. 20 Schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Fig. 20 As shown, the image processing device 2000 includes:

[0301] A determination structure 2001 is used to determine at least two groups of pixel coordinates in the image to be processed, each group of pixel coordinates includes at least three pixel coordinates, the image to be processed is a distorted image, and each group of pixel coordinates corresponds to a distortion line of the image to be processed; based on any two adjacent groups of pixel coordinates, coordinate ranges of at least two regions in the image to be processed are determined in sequence, each region includes at least two pixels;

[0302] The correction structure 2002 is used to perform correction processing on the image data of each pixel in the coordinate range of each region each time the coordinate range of the region is determined to obtain a target image.

[0303] In a possible implementation, the determination structure 2001 includes:

[0304] A coordinate acquisition module, used to acquire at least two groups of pixel coordinates in the image to be processed, wherein the image to be processed includes at least two groups of pixel coordinates, each group of pixel coordinates includes at least three pixel coordinates, the image to be processed is a distorted image, and each group of pixel coordinates corresponds to a distortion line of the image to be processed;

[0305] An image data acquisition module is used to determine the coordinate ranges of at least two regions in the image to be processed in turn according to any two adjacent sets of pixel coordinates, and store the image data of the corresponding coordinate range in the image to be processed to the corresponding storage position in the first storage space according to the coordinate range of each of the regions;

[0306] The image acquisition module obtains pixel coordinates from the coordinate acquisition module each time, and pre-establishes a connection channel required for transmitting the pixel coordinates this time.

[0307] In a possible implementation, before the image acquisition module acquires the pixel coordinates from the coordinate acquisition module, a handshake protocol may be used to establish a connection channel between the two.

[0308] In a possible implementation, the method further includes:

[0309] A cache module is used to divide the first storage space into a plurality of storage locations, each storage location stores image data of an area, each storage location is divided into a specific number of storage units, each storage unit is used to store an image data sub-block, each image data sub-block is composed of a pixel matrix of a set size, and records the sorting sequence number and offset address of the image data sub-block stored in each storage unit, the offset address represents the offset between the image data sub-block and a specific position in the area to which it belongs.

[0310] In a possible implementation, the correction structure specifically includes:

[0311] A correction module is used to interpolate the coordinates of each pixel in the corresponding area based on any two adjacent sets of pixel coordinates; determine any pixel coordinate in any area, and determine a specific number of pixel coordinates corresponding to the interpolation range of the any pixel based on a preset interpolation rule, wherein the interpolation range includes a specific number of pixels, and the any pixel is located in the central area of ​​the interpolation range; determine the storage position corresponding to each pixel in the interpolation range in a first storage space based at least on an offset address corresponding to the image data sub-block and the coordinates of each pixel in the interpolation range of the any pixel, wherein the storage position stores the image data sub-block; obtain at least part of the image data sub-block corresponding to each pixel in the interpolation range from the first storage space; interpolate based on at least part of the image data sub-block corresponding to each pixel in the interpolation range to obtain the corrected image data corresponding to the any pixel, until each pixel in the image to be processed obtains the corresponding corrected image data, thereby obtaining the target image;

[0312] The correction module obtains pixel coordinates from the coordinate acquisition module each time and pre-establishes a connection channel required for transmitting the pixel coordinates this time.

[0313] In a possible implementation, before the correction module obtains the pixel coordinates from the coordinate acquisition module, a handshake protocol may be used to establish a connection channel between the two.

[0314] Among them, for the functional explanation of each component structure in an image processing device provided in this embodiment, please refer to the explanation in the aforementioned method embodiment, and it will not be repeated in this embodiment.

[0315] In this embodiment, in the distorted image to be processed, multiple groups of pixel coordinates corresponding to the distortion lines are determined, and each group of pixel coordinates is at least three pixel coordinates determined on the corresponding distortion line; the coordinate ranges of at least two regions in the image to be processed are determined in turn using any two adjacent groups of coordinate pixels; each time the coordinate range of a region is determined, correction processing is performed on the image data of each pixel within the coordinate range of the region to obtain a target sub-image corresponding to the region. In this process, based on the distortion line, two adjacent pixel coordinates are respectively determined on any two adjacent distortion lines to obtain four pixel coordinates, and a region is determined based on the four pixel coordinates. After a region is determined, correction processing can be performed on the image data of each pixel in the region. There is no need to store a row of image data in the same position. Only a storage position needs to be set for each region, and the image data of each determined region can start to be corrected. There is no need to store a row of image data completely before correction processing can be performed, thereby reducing the requirement for the size of the storage space.

[0316] An image processing method and apparatus provided in an embodiment of the present application are introduced above, and a system for executing the above image processing method will be introduced below.

[0317] See also Fig.21 , Fig.21 21 is a system block diagram of an image processing method provided in an embodiment of the present application. The system includes: an image processing device 2101 provided in an embodiment of the present application and other structures.

[0318] The image processing device includes map (coordinate acquisition module), prefetch (image data acquisition module), BNAK (cache module) and interp (correction module).

[0319] Among them, the other structures include r2b (raster (dot matrix image) to block conversion module), b2r (blockto raster conversion module), RDMA (read DMA module), WDMA (write DMA module), pre-processing pipeline core (FEpipeline), SLC, DDR control module (DDR), etc.

[0320] The FE pipeline performs pre-processing on the image to be processed, controls the r2b module to perform line scanning, and processes the image data of the image to be processed into 16×8 image data blocks. The image to be processed divided into blocks is written to DDR through WDMA.

[0321] The DDR stores complete image data and sparse coordinate data of the image to be processed. The complete image data includes the coordinates and pixel value of each pixel, and the sparse coordinates include multiple coordinates on the distortion line of the image to be processed.

[0322] The map module moves the sparse coordinates of the image to be processed from DDR to the local memory through RDMA. Specifically, the coordinates of the first three lines of distortion lines are stored in the preset address segments 0, 1, and 2 of SRAM (see Figure 6 ). After the coordinates of the current two rows are put into SRAM, they are used starting from the first coordinate of the first row, specifically by providing them to the prefetch module.

[0323] Among them, the map module and the prefetch module use the vld signal of the vld and rdy protocols to perform handshake interaction. The handshake can ensure the smooth connection channel between the two modules and provide a basis for the subsequent transmission of transmission coordinates.

[0324] Among them, after the map module completes putting the coordinates in an address segment, it pulls up the vld signal to prompt the prefetch module to obtain it; each time the prefetch module obtains a pair of coordinates (one coordinate for each of two adjacent rows) from the map module, a handshake is required.

[0325] The prefetch module obtains two pairs of coordinates from the map module, calculates the coordinate range (maximum and minimum range) of the area (mesh) composed of the two pairs of coordinates, and uses the coordinate range of each mesh to obtain the corresponding image data from DDR.

[0326] The prefetch module obtains the coordinate range of the overlapping area between any two adjacent meshes after obtaining the three adjacent meshes. When obtaining the image data of the first mesh, it initiates image data acquisition from DDR in the order from left to right and from top to bottom according to the coordinate range of the first mesh, and determines the number of copies of each image data block. If it overlaps with a mesh, it adds one copy. When requesting the image data of the second mesh from DDR through RDMA, the data of the overlapping area is deleted, and only the image data of the non-overlapping area is requested.

[0327] The RDMA converts the request into an AXI (Advanced eXtensible Interface) communication protocol and sends it to the DDR.

[0328] Among them, the data from DDR to SRAM needs to be reorganized. The prefetch module also converts the bit width data of DDR into the bit width data of SRAM. The rules of reorganization refer to the above Fig.15 According to the above process, the image data of each row of mesh is obtained. After the acquisition of the mesh image data of one row is completed, the next row is processed according to the rule of the above row, and finally the processing of the whole image is completed.

[0329] Among them, the BANK module is the design of SRAM. This module is used to divide the SRAM into banks and allocate addresses to each bank. It interacts with the prefetch module to write to the SRAM, and interacts with the subsequent interp module to read the SRAM.

[0330] Specifically, determine the bit width, depth and number of the SRAM, refer to the above Fig.18 The specifications of SRAM are as follows: the number of Y channel banks is 8, the number of C channel banks is 4, the size of a single Y channel bank is 240×160 bits, and the size of a single C channel bank is 240×80 bits. The read and write control timing of SRAM is the control timing of the standard simple dual-port SRAM.

[0331] Among them, the interp module obtains the sparse coordinates from the aforementioned prefetch module and uses the coordinates for interpolation.

[0332] Specifically, each time the interp module obtains a pair of coordinates (one coordinate for each of two adjacent rows) from the prefetch module, it needs to perform a vld and rdy protocol handshake. This handshake can ensure that the connection channel between the two modules is unobstructed and provide a basis for the subsequent transmission of transmission coordinates.

[0333] The interp module obtains coordinates in units of lines. When the second pair of coordinate points of each row of mesh is obtained, the min value of the mesh is calculated as the reference line for reading. The minimum coordinate value is calculated once each set of mesh coordinates is obtained. When the second pair of coordinate points is obtained, the two pairs of coordinate points form the first mesh. The four vertex coordinates are interpolated by bilinear coordinates, and the coordinates of each interpolation position are obtained in the order of 16×16 mesh. When each interpolation position coordinate is obtained, the data containing bicubic interpolation is read from the BANK module. The control timing of reading SRAM refers to the above. Fig.19 , and obtain the 4×4 neighborhood data required for interpolation calculation. At the same time, the corresponding interpolation coefficients in the H and V directions are obtained according to the coordinate phase of the interpolation position, and the bicubic interpolation calculation is completed, and finally the dedistorted data is obtained.

[0334] According to the above process, interpolation processing is performed on each row of mesh. After the interpolation processing of one row of mesh is completed, the interpolation processing of the next row is performed according to the rule of the above row. Finally, the interpolation processing result of the entire image is completed to obtain the corrected image.

[0335] Among them, in the process of the prefetch module obtaining image data from DDR, after the obtained image data is stored in the storage location of each mesh in BANK, the interp module can start the interpolation process and perform pipeline processing on the image data in the BANK module.

[0336] In order to adapt the data format of the raster scan of the subsequent module, the dedistorted result is converted into the corresponding format through the b2r module, and then written to DDR offline through WDMA, and also output to the post-processing module through the stream interface.

[0337] Should Fig.21 In the figure, src pixel data is used to represent the image data in the distorted image to be processed (including the pixel value and coordinates of each pixel), map lut is used to represent the sparse coordinates, dst pixel data is used to represent the image data in the dedistorted image, and sync ctrl is used to represent the interaction between the interp module and the prefetch module.

[0338] The present application also provides an electronic device in an embodiment. Fig. 22 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Fig. 22 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0339] like Fig. 22As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 2201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 2202 or a program loaded from a storage device 2208 to a random access memory (RAM) 2203. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 2203. The processing device 2201, the ROM 2202, and the RAM 2203 are connected to each other via a bus 2204. An input / output (I / O) interface 2205 is also connected to the bus 2204.

[0340] Typically, the following devices may be connected to the I / O interface 2205: input devices 2206 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 2207 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 2208 including, for example, a memory card, a hard disk, etc.; and communication devices 2209. The communication device 2209 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 22 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0341] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any image processing method provided in the embodiment of the present application.

[0342] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any image processing method provided in the embodiment of the present application.

[0343] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0344] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0345] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0346] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. An image processing method, comprising: Determine at least two groups of pixel coordinates in the image to be processed, each group of pixel coordinates includes at least three pixel coordinates, the image to be processed is a distorted image, and each group of pixel coordinates corresponds to a distortion line of the image to be processed; According to any two adjacent groups of pixel coordinates, coordinate ranges of at least two regions are determined in the image to be processed in sequence, each region including a number of pixels; Each time the coordinate range of a region is determined, the image data of each pixel in the coordinate range of the region is corrected to obtain a target sub-image corresponding to the region.

2. The image processing method according to claim 1, wherein determining at least two sets of pixel coordinates in the image to be processed comprises: Determining a distortion function of the image to be processed; Determine at least two groups of reference pixel coordinates based on a pixel matrix corresponding to the image to be processed, wherein the reference pixel coordinates are pixel coordinates extracted at set intervals in a reference line corresponding to the image to be processed; At least two groups of pixel coordinates corresponding to the at least two groups of reference pixel coordinates are respectively determined according to the distortion function.

3. The image processing method according to claim 1, wherein each time the coordinate range of a region is determined, the image data of each pixel in the coordinate range of the region is corrected to obtain a target sub-image corresponding to the region, comprising: According to the coordinate range of any area, obtaining target image data corresponding to any area in the image to be processed; Sending the target image data to a first storage space; Based on the first storage space having remaining storage locations, storing the target image data in the remaining storage locations; Based on the fact that there is no remaining storage position in the first storage space, a target storage position is determined, and the target image data is stored in the target storage position, wherein the image data stored in the target storage position has been corrected; Correction processing is performed according to the target image data in the first storage space to obtain a target sub-image corresponding to any one of the regions.

4. The image processing method according to claim 1, wherein each time the coordinate range of a region is determined, the image data of each pixel in the coordinate range of the region is corrected to obtain a target sub-image corresponding to the region, comprising: When the coordinate range of any area is determined, image data corresponding to each pixel within the coordinate range of any area is obtained from a second storage space, and the second storage space stores image data of all pixels in the image to be processed; storing the image data in a target storage location in the first storage space, wherein the target storage location is a storage location allocated for any one of the regions; Each time image data of any region is stored in the first storage space, correction processing is performed according to the image data corresponding to the any region to obtain a target sub-image corresponding to the any region.

5. The image processing method according to claim 4, wherein storing the image data to a target storage location in the first storage space comprises: storing first image data corresponding to the first region in a first storage location in a first storage space, where the first storage location is a storage location allocated for the first region; If the coordinate range of the first area and the coordinate range of the second area have an overlapping range, storing the first image data corresponding to the overlapping range in a second storage position in the first storage space, where the second storage position is a storage position allocated for the second area; Obtaining second image data corresponding to each pixel outside the overlapping range in the second area from the second storage space; The second image data is stored in the second storage location.

6. The image processing method according to claim 4, wherein obtaining the image data corresponding to each pixel within the coordinate range of any area from the second storage space comprises: Based on the coordinate range of any one of the regions, determine a set of image data blocks corresponding to any one of the regions, each set of image data blocks comprising at least two image data blocks, each image data block comprising pixel values ​​of each pixel in a pixel matrix of a preset size; generating a first request according to the image data block set, wherein the first request is used to request each image data block in the image data block set from a second storage space; Receive each image data block in the image data block set fed back by the second storage space.

7. The image processing method according to claim 5, wherein obtaining the second image data corresponding to each pixel outside the overlapping range in the second area from the second storage space comprises: Determining, according to the coordinate range of the second area, a set of initial image data blocks corresponding to the second area; Determine a set of overlapping image data blocks corresponding to the overlapping range; generating a second request according to the initial image data block set and the overlapping image data block set, wherein the second request is used to request a target image data block set from a second storage space, wherein the target image data block set and the overlapping image data block set constitute the initial image data block set; Receive each image data block in the target image data block set fed back by the second storage space.

8. The image processing method according to claim 4, wherein storing the image data to a target storage location in the first storage space comprises: Dividing each image data block in any one of the regions into a specific number of image data sub-blocks, each image data sub-block consisting of a pixel matrix of a set size; Determine, according to a preset sequence, an arrangement order of each image data sub-block in any region and an offset address corresponding to each image data sub-block, wherein the ordering sequence number of any image data sub-block is different from that of an adjacent image data sub-block, and the offset address represents an offset amount of the image data sub-block from a specific position in any region; According to the arrangement order of each image data sub-block in any one area and the offset address, each image data sub-block is stored in a target storage location.

9. The image processing method according to claim 8, performing correction processing according to the image data corresponding to any one of the regions to obtain a target sub-image corresponding to any one of the regions, comprising: According to any two adjacent sets of pixel coordinates, interpolate to obtain the coordinates of each pixel in the corresponding area; Determine any pixel coordinate in any area, and determine a specific number of pixel coordinates corresponding to an interpolation range of the any pixel according to a preset interpolation rule; At least according to the offset address corresponding to the image data sub-block and the coordinates of each pixel within the interpolation range of the any pixel, acquiring at least part of the data of the image data sub-block corresponding to each pixel within the interpolation range from the corresponding storage position of the first storage space; Interpolate at least part of the image data sub-block corresponding to each pixel within the interpolation range to obtain the corrected image data corresponding to any pixel, until each pixel in any area obtains the corresponding corrected image data, thereby obtaining a target sub-image corresponding to any area.

10. An image processing device, comprising: A determination structure is used to determine at least two groups of pixel coordinates in the image to be processed, each group of pixel coordinates includes at least three pixel coordinates, the image to be processed is a distorted image, and each group of pixel coordinates corresponds to a distortion line of the image to be processed; based on any two adjacent groups of pixel coordinates, coordinate ranges of at least two regions in the image to be processed are determined in sequence, each region includes at least two pixels; The correction structure is used to determine the coordinate range of an area each time, and to perform correction processing on the image data of each pixel in the coordinate range of the area to obtain a target image.

11. The image processing apparatus according to claim 10, wherein the determining structure comprises: A coordinate acquisition module, used to acquire at least two groups of pixel coordinates in the image to be processed, wherein the image to be processed includes at least two groups of pixel coordinates, each group of pixel coordinates includes at least three pixel coordinates, the image to be processed is a distorted image, and each group of pixel coordinates corresponds to a distortion line of the image to be processed; An image data acquisition module, used for determining the coordinate ranges of at least two regions in the image to be processed in turn according to any two adjacent groups of pixel coordinates, and storing the image data of the corresponding coordinate range in the image to be processed to the corresponding storage position in the first storage space according to the coordinate range of each of the regions; The image acquisition module obtains pixel coordinates from the coordinate acquisition module each time, and pre-establishes a connection channel required for transmitting the pixel coordinates this time.

12. The image processing apparatus according to claim 11, further comprising: A cache module is used to divide the first storage space into a plurality of storage locations, each storage location stores image data of an area, each storage location is divided into a specific number of storage units, each storage unit is used to store an image data sub-block, each image data sub-block is composed of a pixel matrix of a set size, and records the sorting sequence number and offset address of the image data sub-block stored in each storage unit, wherein the offset address represents the offset between the image data sub-block and a specific position in the area to which it belongs.

13. The image processing device according to claim 12, wherein the correction structure comprises: A correction module, used for interpolating the coordinates of each pixel in the corresponding area based on any two adjacent sets of pixel coordinates; Determine any pixel coordinate in any area, determine a specific number of pixel coordinates corresponding to an interpolation range of the any pixel according to a preset interpolation rule, the interpolation range includes a specific number of pixels, and the any pixel is located in the central area of ​​the interpolation range; determine a storage position corresponding to each pixel in the interpolation range in a first storage space according to at least an offset address corresponding to the image data sub-block and the coordinates of each pixel in the interpolation range of the any pixel, the storage position stores the image data sub-block; obtain at least part of the image data sub-block corresponding to each pixel in the interpolation range from the first storage space; perform interpolation according to at least part of the image data sub-block corresponding to each pixel in the interpolation range to obtain corrected image data corresponding to the any pixel, until each pixel in the image to be processed obtains the corresponding corrected image data, and obtains a target image; The correction module obtains pixel coordinates from the coordinate acquisition module each time and pre-establishes a connection channel required for transmitting the pixel coordinates this time.

Images