An image remapping method and an image processing apparatus

Abstract

Embodiments of the present application disclose an image remapping method and an image processing device, which can decompose image remapping into two single-direction mappings when determining a target image (i.e., a pre-distorted image) based on an original image. Specifically, the image processing device first performs a row transformation on each row of pixels in the original image, and then performs a column transformation on each column of pixels based on the row transformation result (i.e., the first image) to obtain the target image. That is to say, when calculating a row of pixels in the first image, the image processing device only needs to sequentially read a row of pixels in the original image, rather than reading them in a disorderly manner across rows; similarly, when calculating a column of pixels in the target image, the image processing device only needs to sequentially read a column of pixels in the first image, rather than reading them in a disorderly manner across columns. Therefore, the image processing device can read the original image in an orderly manner when processing it, thereby improving the efficiency of the image processing device in processing the original image.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of image processing, and in particular, to an image remapping method and an image processing device. Background Art

[0002] Distortion is an aberration commonly existing in imaging systems. For a projection imaging system, if distortion occurs during the projection of the original image, the image presented on the imaging plane will be a distorted image (i.e., a distorted image) compared to the original image, rather than the aforementioned original image.

[0003] In traditional technologies, the pre-distortion method is often used to convert the original image into a pre-distorted image that can offset the distortion. Specifically, when the processor calculates the pixel value of each pixel in the pre-distorted image based on the original image, the processor needs to find the point A' corresponding to pixel A in the pre-distorted image from the original image, and assign the pixel value of the point A' to pixel A in the pre-distorted image. However, since the pre-distorted image has different degrees of distortion in both the row and column directions compared to the original image, the horizontal and vertical coordinates of the point A' are not necessarily integers. Therefore, the processor needs to use an interpolation algorithm to calculate the pixel value of the point A' based on multiple pixels near the point A'. This process is repeated until the processor calculates the pixel value of each pixel in the pre-distorted image. Then, after the processor inputs the pre-distorted image into the display device of the projection imaging system, the user can observe the original image obtained by distorting the aforementioned pre-distorted image through the imaging plane.

[0004] Since the display driver in traditional technologies requires the image to be input into the driver in row order, and the pixels of each row in the pre-distorted image generally correspond to the pixels of the original image in a disordered manner, a large number of disordered accesses to the content of the original image are involved in the process of calculating the pre-distorted image, which will affect the processing efficiency of the processor. Therefore, there is an urgent need for a solution that can improve the efficiency of processing the original image. Summary of the Invention

[0005] The embodiments of the present application provide an image remapping method and an image processing device for improving the efficiency of generating a pre-distorted image based on an original image.

[0006] In a first aspect, the present application provides an image remapping method, which can be applied to an imaging scenario based on a projection imaging system. In this method, the image processing device first performs a row transformation process and then a column transformation process. Specifically, the image processing device acquires each row of pixels of the original image, and acquires the row transformation parameters corresponding to each row of pixels. The row transformation parameters are used to indicate the correspondence between the abscissa of the pixels in the original image and the abscissa of the pixels in the first image. Then, the image processing device determines the pixel value of each pixel in the first image according to each row of pixels in the original image and the row transformation parameters corresponding to each row of pixels. At this time, the image processing device obtains an image with pre-distortion in the row direction and no pre-distortion in the column direction (i.e., the aforementioned first image). Then, the image processing device acquires each column of pixels of the first image, and acquires the column transformation parameters corresponding to each column of pixels. The column transformation parameters are used to indicate the correspondence between the ordinate of the pixels in the first image and the ordinate of the pixels in the target image. Then, the image processing device determines the pixel value of each pixel in the target image according to each column of pixels in the first image and the column transformation parameters corresponding to each column of pixels. At this time, the image processing device obtains an image with pre-distortion in both the row direction and the column direction (i.e., the target image), and the target image is used to cancel the imaging distortion of the imaging device to display the original image.

[0007] It should be understood that when the image processing device acquires each row of pixels of the original image, specifically, the image processing device reads the pixel value of each pixel in each row of pixels and the coordinates of the pixel in the original image. Among them, the coordinates of the pixel in the original image include the abscissa and the ordinate. Every time the image processing device acquires a row of pixels from the original image, correspondingly, the image processing device acquires the row transformation parameters corresponding to this row of pixels. Then, the image processing device can calculate a row of pixels in the first image based on the foregoing row of pixels and the row transformation parameters corresponding to this row of pixels. By analogy, the image processing device will successively calculate each row of pixels in the first image. The process from the original image to the first image can be called the row transformation process. Similarly, when the image processing device acquires each column of pixels of the first image, specifically, the image processing device reads the pixel value of each pixel in each column of pixels and the coordinates of the pixel in the first image. Among them, the coordinates of the pixel in the first image include the abscissa and the ordinate. Every time the image processing device acquires a column of pixels from the first image, correspondingly, the image processing device acquires the column transformation parameters corresponding to this column of pixels. Then, the image processing device can calculate a column of pixels in the target image based on the foregoing column of pixels and the column transformation parameters corresponding to this column of pixels. By analogy, the image processing device will successively calculate each column of pixels in the target image. The process from the first image to the target image can be called the column transformation process. At this time, the image processing device has performed pre-distortion processing (i.e., the foregoing row transformation and column transformation) on the original image in the row direction and the column direction respectively, and the target image is an image that can offset the imaging distortion of the projection imaging system.

[0008] In this application, when the image processing device determines the target image (i.e., the pre-distorted image) based on the original image, the image remapping is decomposed into two single-direction mappings. Specifically, the image processing device first performs a row transformation on each row of pixels in the original image, and then performs a column transformation on each column of pixels in the result of the row transformation (i.e., the first image) to obtain the target image. That is to say, when the image processing device calculates a row of pixels in the first image, it only needs to sequentially read a row of pixels in the original image, rather than randomly read across rows; similarly, when the image processing device calculates a column of pixels in the target image, it also only needs to sequentially read a column of pixels in the first image, rather than randomly read across columns. Therefore, the image processing device can read in an orderly manner when processing the original image, thereby improving the efficiency of the image processing device in processing the original image.

[0009] In an alternative embodiment, the row transformation parameter includes the abscissa of the first pixel value point corresponding to the first pixel in the original image. The first pixel is any pixel in the first image, and the first pixel value point is used to determine the pixel value of the first pixel. The ordinate of the first pixel value point in the original image is the same as the ordinate of the first pixel in the first image.

[0010] In this embodiment, it is proposed that the row transformation parameter can be represented by the abscissa of the first pixel value point in the original image. The first pixel value point is the point corresponding to the first pixel in the first image to be obtained. It can be understood that after the distortion occurs in the row direction, the first pixel value point falls to the position corresponding to the aforementioned first pixel. Therefore, when calculating the pixel value of the first pixel, it is necessary to find the pixel value of the position of the first pixel value point in the original image. Since there is only distortion in the row direction between the first pixel and the first pixel value point, the ordinate of the first pixel value point in the original image is the same as the ordinate of the first pixel in the first image. Therefore, when the aforementioned image processing device calculates the first pixels in a row of the first image, it only needs to sequentially obtain a row of pixels in the original image.

[0011] In an alternative embodiment, the column transformation parameter includes the ordinate of the second pixel value point corresponding to the second pixel in the first image. The second pixel is any pixel in the target image, and the second pixel value point is used to determine the pixel value of the second pixel. The abscissa of the second pixel value point in the first image is the same as the abscissa of the second pixel in the target image.

[0012] In this embodiment, it is proposed that the column transformation parameter can be represented by the ordinate of the second pixel value point in the first image. The second pixel value point is the point corresponding to the second pixel in the target image to be obtained. It can be understood that after the distortion occurs in the column direction, the second pixel value point falls to the position corresponding to the aforementioned second pixel. Therefore, when calculating the pixel value of the second pixel, it is necessary to find the pixel value of the position of the second pixel value point in the first image. Since there is only distortion in the column direction between the second pixel and the second pixel value point, the abscissa of the second pixel value point in the first image is the same as the abscissa of the second pixel in the target image. Therefore, when the aforementioned image processing device calculates the second pixels in a column of the target image, it only needs to sequentially obtain a column of pixels in the first image.

[0013] Compared with traditional technologies, the image processor in traditional technologies only involves mapping parameters from the original image to the target image during image processing, and requires simultaneous remapping of both the horizontal and vertical directions in a single mapping process. Therefore, when calculating a pixel in the target image, it is required that the image processor can at least read four pixels distributed in a rectangle from the original image. Also, since the distortion degree in the row direction is different from that in the column direction, when the image processor calculates each pixel value in the target image one by one, it needs to obtain pixels in a disordered manner from the original image. In contrast, the solution of the present application only needs to obtain pixels row by row from the original image and column by column from the first image. Therefore, the solution proposed in the present application can improve the image processing efficiency of the image processing device.

[0014] In an alternative embodiment, the foregoing row transformation parameter is stored in a row mapping relationship table, and the position of the row transformation parameter in the row mapping relationship table is used to indicate the coordinates of a first pixel, where the first pixel is the pixel corresponding to the foregoing row transformation parameter. It can also be understood that the position of the row transformation parameter in the row mapping relationship table is used to indicate the coordinates of the first pixel determined based on the row transformation parameter.

[0015] In addition, the column transformation parameter is stored in a column mapping relationship table, and the position of the column transformation parameter in the column mapping relationship table is used to indicate the coordinates of a second pixel, where the second pixel is the pixel corresponding to the foregoing column transformation parameter. It can also be understood that the position of the column transformation parameter in the column mapping relationship table is used to represent the coordinates of the second pixel determined based on the column transformation parameter.

[0016] In an alternative embodiment, the image processing device determines the pixel value of each pixel in the first image according to each row of pixels in the original image and the row transformation parameter corresponding to each row of pixels, including: determining the first pixel value point corresponding to the first pixel according to the row transformation parameter; determining at least one third pixel according to the abscissa of the first pixel value point, where the third pixel and the first pixel value point are in the same row; determining the pixel value of the first pixel according to the pixel values of the at least one third pixel. The image processing device determines the pixel value of each pixel in the target image according to each column of pixels in the first image and the column transformation parameter corresponding to each column of pixels, including: determining the second pixel value point corresponding to the second pixel according to the column transformation parameter; determining at least one fourth pixel according to the ordinate of the second pixel value point, where the fourth pixel and the second pixel value point are in the same column; determining the pixel value of the second pixel according to the pixel values of the at least one fourth pixel.

[0017] In an alternative embodiment, the abscissa of the first pixel value point is an integer. Determining at least one third pixel according to the abscissa of the first pixel value point includes: determining the first pixel value point as the third pixel. Determining the pixel value of the first pixel according to the pixel values of the at least one third pixel includes: assigning the pixel value of the third pixel to the first pixel to obtain the pixel value of the first pixel.

[0018] In an alternative embodiment, the abscissa of the first pixel value point is a non-integer. Determining at least one third pixel according to the abscissa of the first pixel value point includes: determining at least one pixel adjacent to the first pixel value point as the third pixel, where the pixel adjacent to the first pixel value point is in the same row as the first pixel value point. Determining the pixel value of the first pixel according to the pixel values of the at least one third pixel includes: using an interpolation algorithm to calculate a first weighted average of the pixel values of the at least one third pixel based on the abscissa of the first pixel value point and the abscissas of the at least one third pixel; assigning the first weighted average to the first pixel to obtain the pixel value of the first pixel.

[0019] Among them, the aforementioned pixel adjacent to the first pixel value point can be understood as a pixel one to the left and / or one to the right of the first pixel value point; it can also be understood as the second pixel to the left and / or the second pixel to the right; or even as two pixels to the left and / or two pixels to the right. There is no specific limitation here.

[0020] In an alternative embodiment, the ordinate of the second pixel value point is an integer. Determining at least one fourth pixel according to the ordinate of the second pixel value point includes: determining the second pixel value point as the fourth pixel. Determining the pixel value of the second pixel according to the pixel values of the at least one fourth pixel includes: assigning the pixel value of the fourth pixel to the second pixel to obtain the pixel value of the second pixel.

[0021] In an alternative embodiment, the ordinate of the second pixel value point is a non-integer. Determining at least one fourth pixel according to the ordinate of the second pixel value point includes: determining at least one pixel adjacent to the second pixel value point as the fourth pixel, where the pixel adjacent to the second pixel value point is in the same column as the second pixel value point. Determining the pixel value of the second pixel according to the pixel values of the at least one fourth pixel includes: using an interpolation algorithm to calculate a second weighted average of the pixel values of the at least one fourth pixel based on the ordinate of the second pixel value point and the ordinates of the at least one fourth pixel; assigning the second weighted average to the second pixel to obtain the pixel value of the second pixel.

[0022] In an alternative embodiment, the image processing device obtains the row transformation parameters corresponding to each row of pixels, including: reading the row transformation parameters corresponding to each row of pixels in sequence in the row direction from a memory, where a row mapping relationship table is stored in the memory, and the row mapping relationship table includes the row transformation parameters corresponding to each row of pixels in the original image. The image processing device obtains the column transformation parameters corresponding to each column of pixels, including: reading the column transformation parameters corresponding to each column of pixels in sequence in the column direction from a memory, where a column mapping relationship table is stored in the memory, and the column mapping relationship table includes the column transformation parameters corresponding to each column of pixels in the original image.

[0023] In this embodiment, two mapping relationship tables are stored in the memory, including a row mapping relationship table from the original image to the first image (i.e., the image with only row transformation), and a column mapping relationship table from the first image to the target image. In the solution of the traditional technology, a mapping relationship table from the original image to the target image is used. Therefore, in the solution of the traditional technology, when calculating the pixel value of each pixel, interpolation needs to be performed by referring to multiple pixels in the original image that are not in the same row and not in the same column.

[0024] In an alternative embodiment, after performing row transformation processing on each row of pixels in the original image and the row transformation parameters corresponding to each row of pixels to obtain the first image, the method further includes: the image processing device stores the first pixel of each row of the first image in the memory. The process of the image processing device obtaining each column of pixels of the first image can specifically be: the image processing device sequentially reads each column of pixels of the first image from the memory. In the solution of the traditional technology, the foregoing first image does not appear in the image processing process, and there are only the original image and the pre-distorted image (i.e., the target image).

[0025] In an alternative embodiment, after the image processing device generates the target image, the image processing device may send the target image to the imaging device, and the imaging device is configured to perform imaging based on the foregoing target image so that the original image is projected on the imaging plane based on the foregoing target image. Among them, the imaging device may be a projection imaging system.

[0026] In a second aspect, the present application provides an image processing device, which may be an image processing chip in a projection imaging system or a processor with image processing capabilities. The image processing device includes: an acquisition module and a calculation module. Among them, the acquisition module is configured to acquire each row of pixels of the original image and acquire the row transformation parameters corresponding to each row of pixels. Among them, the row transformation parameters are used to indicate the corresponding relationship between the abscissa of the pixels in the original image and the abscissa of the pixels in the first image, and the first image is an image with pre-distortion in the row direction and no pre-distortion in the column direction.

[0027] A calculation module is configured to determine the pixel value of each pixel in the first image according to each row of pixels in the original image and the row transformation parameter corresponding to each row of pixels respectively. In addition, the acquisition module is further configured to acquire each column of pixels in the first image and the column transformation parameter corresponding to each column of pixels. The column transformation parameter is used to indicate the corresponding relationship between the ordinate of the pixel in the first image and the ordinate of the pixel in the target image, and the target image is an image with pre-distortion in the row direction and pre-distortion in the column direction. In addition, the calculation module is further configured to determine the pixel value of each pixel in the target image according to each column of pixels in the first image and the column transformation parameter corresponding to each column of pixels respectively, and the target image is used to offset the imaging distortion of the imaging device to display the original image.

[0028] In an alternative embodiment, the row transformation parameter includes the abscissa of the first pixel value point corresponding to the first pixel in the original image, the first pixel is any pixel in the first image, the first pixel value point is used to determine the pixel value of the first pixel, and the ordinate of the first pixel value point in the original image is the same as the ordinate of the first pixel in the first image. The column transformation parameter includes the ordinate of the second pixel value point corresponding to the second pixel in the first image, the second pixel is any pixel in the target image, the second pixel value point is used to determine the pixel value of the second pixel, and the abscissa of the second pixel value point in the first image is the same as the abscissa of the second pixel in the target image.

[0029] In an alternative embodiment, the row transformation parameter is stored in a row mapping relationship table, and the position of the row transformation parameter in the row mapping relationship table is used to represent the coordinates of the first pixel determined based on the row transformation parameter. It can also be understood that the position of the row transformation parameter in the row mapping relationship table is used to indicate the coordinates of the first pixel, where the first pixel is the pixel corresponding to the foregoing row transformation parameter. The column transformation parameter is stored in a column mapping relationship table, and the position of the column transformation parameter in the column mapping relationship table is used to indicate the coordinates of the second pixel, where the second pixel is the pixel corresponding to the foregoing column transformation parameter. The position of the column transformation parameter in the column mapping relationship table is used to represent the coordinates of the second pixel determined based on the column transformation parameter.

[0030] In an alternative embodiment, the calculation module is specifically configured to: determine the first pixel value point corresponding to the first pixel according to the row transformation parameter; determine at least one third pixel according to the abscissa of the first pixel value point, where the third pixel and the first pixel value point are in the same row; determine the pixel value of the first pixel according to the pixel values of the at least one third pixel; determine the second pixel value point corresponding to the second pixel according to the column transformation parameter; determine at least one fourth pixel according to the ordinate of the second pixel value point, where the fourth pixel and the second pixel value point are in the same column; determine the pixel value of the second pixel according to the pixel values of the at least one fourth pixel.

[0031] In an alternative embodiment, the image processing apparatus further includes a storage module. The storage module is configured to store the first pixels of each row of the first image in a memory.

[0032] In an alternative embodiment, the row mapping table and the column mapping table are stored in a memory. The row mapping table includes the row transformation parameters corresponding to each row of pixels in the original image, and the column mapping table includes the column transformation parameters corresponding to each column of pixels in the original image. The obtaining module is specifically configured to: sequentially read the row transformation parameters corresponding to each row of pixels from the memory in the row direction; sequentially read the column transformation parameters corresponding to each column of pixels from the memory in the column direction.

[0033] In an alternative embodiment, the image processing apparatus further includes a storage module. The sending module is configured to send the target image to the imaging device, and the imaging device is configured to perform imaging based on the foregoing target image so that the original image is projected on the imaging plane based on the foregoing target image. Wherein, the imaging device may be a projection imaging system.

[0034] In a third aspect, the present application provides an image remapping method, which can be applied to an imaging scenario based on a projection imaging system. In this method, the image processing device first performs a row transformation process and then a column transformation process. Specifically, the image processing device acquires each column of pixels of the original image, and acquires the column transformation parameters corresponding to each column of pixels. The column transformation parameters are used to indicate the correspondence between the ordinate of the pixels in the original image and the ordinate of the pixels in the second image. Then, the image processing device determines the pixel values of each pixel in the second image according to each column of pixels in the original image and the column transformation parameters corresponding to each column of pixels. At this time, the image processing device obtains an image with pre-distortion in the column direction but without pre-distortion in the row direction (i.e., the aforementioned second image). Then, the image processing device acquires each row of pixels of the second image, and acquires the row transformation parameters corresponding to each row of pixels. The row transformation parameters are used to indicate the correspondence between the abscissa of the pixels in the second image and the abscissa of the pixels in the target image. Then, the image processing device determines the pixel values of each pixel in the target image according to each row of pixels in the second image and the row transformation parameters corresponding to each row of pixels. At this time, the image processing device obtains an image with pre-distortion in both the row direction and the column direction (i.e., the target image), and the target image is used to offset the imaging distortion of the imaging device to display the original image.

[0035] It should be understood that when the image processing device acquires each column of pixels of the original image, specifically, the image processing device reads the pixel value of each pixel in each column of pixels and the coordinates of the pixel in the original image. Among them, the coordinates of the pixel in the original image include the abscissa and the ordinate. Each time the image processing device acquires a column of pixels from the original image, correspondingly, the image processing device acquires the column transformation parameters corresponding to this column of pixels. Then, the image processing device can calculate a column of pixels in the second image based on the aforementioned column of pixels and the column transformation parameters corresponding to this column of pixels. By analogy, the image processing device will calculate each column of pixels in the second image in sequence. The process from the original image to the second image can be called the column transformation process. Similarly, when the image processing device acquires each row of pixels of the second image, specifically, the image processing device reads the pixel value of each pixel in each row of pixels and the coordinates of the pixel in the second image. Among them, the coordinates of the pixel in the second image include the abscissa and the ordinate. Each time the image processing device acquires a row of pixels from the second image, correspondingly, the image processing device acquires the row transformation parameters corresponding to this row of pixels. Then, the image processing device can calculate a row of pixels in the target image based on the aforementioned row of pixels and the row transformation parameters corresponding to this row of pixels. By analogy, the image processing device will calculate each row of pixels in the target image in sequence. The process from the second image to the target image can be called the row transformation process. At this time, the image processing device has performed pre-distortion processing (i.e., the aforementioned row transformation and column transformation) on the original image in the row direction and the column direction respectively, and the target image is an image that can offset the imaging distortion of the projection imaging system.

[0036] In this application, when the image processing device determines the target image (i.e., the pre-distorted image) based on the original image, the image remapping is decomposed into two single-direction mappings. Specifically, the image processing device first performs a column transformation on each column of pixels in the original image, and then performs a row transformation on each row of pixels based on the column transformation result (i.e., the second image) to obtain the target image. That is to say, when the image processing device calculates a column of pixels in the second image, it only needs to sequentially read a column of pixels in the original image, rather than randomly read across columns; similarly, when the image processing device calculates a row of pixels in the target image, it also only needs to sequentially read a row of pixels in the second image, rather than randomly read across rows. Therefore, the image processing device can read the original image in an orderly manner when processing the original image, thereby improving the efficiency of the image processing device in processing the original image.

[0037] In an alternative embodiment, the column transformation parameter includes the ordinate of the fifth pixel value point corresponding to the fifth pixel in the original image, where the fifth pixel is any pixel in the second image, the fifth pixel value point is used to determine the pixel value of the fifth pixel, and the abscissa of the fifth pixel value point in the original image is the same as the abscissa of the fifth pixel in the second image.

[0038] In this embodiment, it is proposed that the column transformation parameter can be represented by the ordinate of the fifth pixel value point in the original image. The fifth pixel value point is the point corresponding to the fifth pixel in the second image to be obtained. It can be understood that after the distortion occurs in the column direction, the fifth pixel value point falls to the position corresponding to the aforementioned fifth pixel. Therefore, when calculating the pixel value of the fifth pixel, it is necessary to find the pixel value at the position of the fifth pixel value point in the original image. Since there is only distortion in the column direction between the fifth pixel and the fifth pixel value point, the abscissa of the fifth pixel value point in the original image is the same as the abscissa of the fifth pixel in the second image. Therefore, when calculating a column of fifth pixels in the second image, the aforementioned image processing device only needs to obtain a column of pixels in the original image column by column.

[0039] In an alternative embodiment, the row transformation parameter includes the abscissa of the sixth pixel value point corresponding to the sixth pixel in the second image, where the sixth pixel is any pixel in the target image, the sixth pixel value point is used to determine the pixel value of the sixth pixel, and the ordinate of the sixth pixel value point in the second image is the same as the ordinate of the sixth pixel in the target image.

[0040] In this embodiment, it is proposed that the row transformation parameter can be represented by the abscissa of the sixth pixel value point in the second image. The sixth pixel value point is the point corresponding to the sixth pixel in the target image to be obtained. It can be understood that after the distortion occurs in the row direction, the sixth pixel value point falls to the position corresponding to the aforementioned sixth pixel. Therefore, when calculating the pixel value of the sixth pixel, it is necessary to find the pixel value at the position of the sixth pixel value point in the second image. Since there is only distortion in the row direction between the sixth pixel and the sixth pixel value point, the ordinate of the sixth pixel value point in the second image is the same as the ordinate of the sixth pixel in the target image. Therefore, when calculating a row of sixth pixels in the target image, the aforementioned image processing device only needs to obtain a row of pixels in the second image row by row.

[0041] In an alternative embodiment, the foregoing column transformation parameters are stored in a column mapping relation table, and the position of the column transformation parameter in the column mapping relation table is used to represent the coordinates of a fifth pixel determined based on the column transformation parameter. It can also be understood that the position of the column transformation parameter in the column mapping relation table is used to indicate the coordinates of the fifth pixel, where the fifth pixel is the pixel corresponding to the foregoing column transformation parameter. In addition, the row transformation parameter is stored in a row mapping relation table, and the position of the row transformation parameter in the row mapping relation table is used to represent the coordinates of a sixth pixel determined based on the row transformation parameter. It can also be understood that the position of the row transformation parameter in the row mapping relation table is used to indicate the coordinates of the sixth pixel, where the sixth pixel is the pixel corresponding to the foregoing row transformation parameter.

[0042] In an alternative embodiment, the image processing device determines the pixel value of each pixel in the second image according to each column of pixels in the original image and the column transformation parameter corresponding to each column of pixels, including: determining a fifth pixel value point corresponding to the fifth pixel according to the column transformation parameter; determining at least one seventh pixel according to the ordinate of the fifth pixel value point, where the seventh pixel and the fifth pixel value point are in the same column; and determining the pixel value of the fifth pixel according to the pixel values of the at least one seventh pixel. The image processing device determines the pixel value of each pixel in the target image according to each row of pixels in the second image and the row transformation parameter corresponding to each row of pixels, including: determining a sixth pixel value point corresponding to the sixth pixel according to the row transformation parameter; determining at least one eighth pixel according to the abscissa of the sixth pixel value point, where the eighth pixel and the sixth pixel value point are in the same row; and determining the pixel value of the sixth pixel according to the pixel values of the at least one eighth pixel.

[0043] In an alternative embodiment, the ordinate of the fifth pixel value point is an integer. Determining at least one seventh pixel according to the ordinate of the fifth pixel value point includes: determining that the fifth pixel value point is the seventh pixel. Determining the pixel value of the fifth pixel according to the pixel values of the at least one seventh pixel includes: assigning the pixel value of the seventh pixel to the fifth pixel to obtain the pixel value of the fifth pixel.

[0044] In an alternative embodiment, the ordinate of the fifth pixel value point is a non-integer. Determining at least one seventh pixel according to the ordinate of the fifth pixel value point includes: determining at least one pixel adjacent to the fifth pixel value point as the seventh pixel, and the pixel adjacent to the fifth pixel value point and the fifth pixel value point are in the same column. Determining the pixel value of the fifth pixel according to the pixel values of the at least one seventh pixel includes: using an interpolation algorithm to calculate a first weighted average of the pixel values of the at least one seventh pixel according to the ordinate of the fifth pixel value point and the ordinates of the at least one seventh pixel; assigning the first weighted average to the fifth pixel to obtain the pixel value of the fifth pixel.

[0045] In an alternative embodiment, the abscissa of the sixth pixel value point is an integer. Determining at least one eighth pixel according to the abscissa of the sixth pixel value point includes: determining the sixth pixel value point as the eighth pixel. Determining the pixel value of the sixth pixel according to the pixel values of the at least one eighth pixel includes: assigning the pixel value of the eighth pixel to the sixth pixel to obtain the pixel value of the sixth pixel.

[0046] In an alternative embodiment, the abscissa of the sixth pixel value point is a non-integer. Determining at least one eighth pixel according to the abscissa of the sixth pixel value point includes: determining at least one pixel adjacent to the sixth pixel value point as the eighth pixel, and the pixel adjacent to the sixth pixel value point and the sixth pixel value point are in the same row. Determining the pixel value of the sixth pixel according to the pixel values of the at least one eighth pixel includes: using an interpolation algorithm to calculate a second weighted average of the pixel values of the at least one eighth pixel according to the abscissa of the sixth pixel value point and the abscissas of the at least one eighth pixel; assigning the second weighted average to the sixth pixel to obtain the pixel value of the sixth pixel.

[0047] In an alternative embodiment, the image processing device obtains the column transformation parameters corresponding to each column of pixels, including: sequentially reading the column transformation parameters corresponding to each column of pixels from the memory in the column direction, and a column mapping relationship table is stored in the memory, and the column mapping relationship table includes the column transformation parameters corresponding to each column of pixels in the original image. The image processing device obtains the row transformation parameters corresponding to each row of pixels, including: sequentially reading the row transformation parameters corresponding to each row of pixels from the memory in the row direction, and a row mapping relationship table is stored in the memory, and the row mapping relationship table includes the row transformation parameters corresponding to each row of pixels in the original image.

[0048] In this embodiment, two mapping tables are stored in the memory, including a column mapping table from the original image to the second image (i.e., the image with only column transformation), and a row mapping table from the second image to the target image. In the solution of the traditional technology, a mapping table from the original image to the target image is used. Therefore, in the solution of the traditional technology, when calculating the pixel value of each pixel, interpolation needs to be performed by referring to multiple pixels in the original image that are not in the same row and not in the same column.

[0049] In an alternative embodiment, after performing column transformation processing according to each column of pixels in the original image and the column transformation parameters corresponding to each column of pixels to obtain the second image, the method further includes: The image processing device stores the fifth pixel of each column of the second image in the memory. The process of the image processing device obtaining each row of pixels of the second image can specifically be: The image processing device sequentially reads each row of pixels of the second image from the memory. In the solution of the traditional technology, the aforementioned second image does not appear in the image processing process, and there are only the original image and the pre-distorted image (i.e., the target image).

[0050] In an alternative embodiment, after the image processing device generates the target image, the image processing device may send the target image to the imaging device, and the imaging device is used to perform imaging based on the aforementioned target image so that the original image is projected on the imaging plane based on the aforementioned target image. Among them, the imaging device may be a projection imaging system.

[0051] Fourthly, the present application provides an image processing device, which may be an image processing chip in a projection imaging system or a processor with image processing functions. The image processing device includes: an acquisition module and a calculation module. Among them, the acquisition module is used to acquire each column of pixels of the original image and acquire the column transformation parameters corresponding to each column of pixels. Among them, the column transformation parameters are used to indicate the corresponding relationship between the ordinate of the pixels in the original image and the ordinate of the pixels in the second image. The calculation module is used to determine the pixel value of each pixel in the second image according to each column of pixels in the original image and the column transformation parameters corresponding to each column of pixels respectively. In addition, the acquisition module is further used to acquire each row of pixels of the second image and acquire the row transformation parameters corresponding to each row of pixels. Among them, the row transformation parameters are used to indicate the corresponding relationship between the abscissa of the pixels in the second image and the abscissa of the pixels in the target image. The calculation module is further used to determine the pixel value of each pixel in the target image according to each row of pixels in the second image and the row transformation parameters corresponding to each row of pixels respectively. At this time, the image processing device obtains an image with pre-distortion in the row direction and pre-distortion in the column direction (i.e., the target image), and the target image is used to offset the imaging distortion of the imaging device to display the original image.

[0052] In an alternative embodiment, the column transformation parameter includes the ordinate of the fifth pixel value point corresponding to the fifth pixel in the original image, where the fifth pixel value point is used to determine the pixel value of the fifth pixel in the second image, and the abscissa of the fifth pixel value point in the original image is the same as the abscissa of the fifth pixel in the second image.

[0053] In an alternative embodiment, the row transformation parameter includes the abscissa of the sixth pixel value point corresponding to the sixth pixel in the second image, where the sixth pixel value point is used to determine the pixel value of the sixth pixel in the target image, and the ordinate of the sixth pixel value point in the second image is the same as the ordinate of the sixth pixel in the target image.

[0054] In an alternative embodiment, the foregoing column transformation parameter is stored in a column mapping relationship table, and the position of the column transformation parameter in the column mapping relationship table is used to represent the coordinates of the fifth pixel determined based on the column transformation parameter. The row transformation parameter is stored in a row mapping relationship table, and the position of the row transformation parameter in the row mapping relationship table is used to represent the coordinates of the sixth pixel determined based on the row transformation parameter.

[0055] In an alternative embodiment, the calculation module is specifically configured to: determine the fifth pixel value point corresponding to the fifth pixel according to the column transformation parameter; determine at least one seventh pixel according to the ordinate of the fifth pixel value point, where the seventh pixel and the fifth pixel value point are in the same column; and determine the pixel value of the fifth pixel according to the pixel values of the at least one seventh pixel. The calculation module is specifically configured to: determine the sixth pixel value point corresponding to the sixth pixel according to the row transformation parameter; determine at least one eighth pixel according to the abscissa of the sixth pixel value point, where the eighth pixel and the sixth pixel value point are in the same row; and determine the pixel value of the sixth pixel according to the pixel values of the at least one eighth pixel.

[0056] In an alternative embodiment, the ordinate of the fifth pixel value point is an integer. The calculation module is specifically configured to: determine that the fifth pixel value point is the seventh pixel; and assign the pixel value of the seventh pixel to the fifth pixel to obtain the pixel value of the fifth pixel.

[0057] In an alternative embodiment, the ordinate of the fifth pixel value point is a non-integer. The calculation module is specifically configured to: determine at least one pixel adjacent to the fifth pixel value point as the seventh pixel, where the pixel adjacent to the fifth pixel value point and the fifth pixel value point are in the same column. The calculation module is specifically configured to: use an interpolation algorithm to calculate a first weighted average of the pixel values of the at least one seventh pixel according to the ordinate of the fifth pixel value point and the ordinates of the at least one seventh pixel; assign the first weighted average to the fifth pixel to obtain the pixel value of the fifth pixel.

[0058] In an alternative embodiment, the abscissa of the sixth pixel value point is an integer. The calculation module is specifically configured to: determine the sixth pixel value point as the eighth pixel; assign the pixel value of the eighth pixel to the sixth pixel to obtain the pixel value of the sixth pixel.

[0059] In an alternative embodiment, the abscissa of the sixth pixel value point is a non-integer. The calculation module is specifically configured to: determine at least one pixel adjacent to the sixth pixel value point as the eighth pixel, where the pixel adjacent to the sixth pixel value point and the sixth pixel value point are in the same row; use an interpolation algorithm to calculate a second weighted average of the pixel values of the at least one eighth pixel according to the abscissa of the sixth pixel value point and the abscissas of the at least one eighth pixel; assign the second weighted average to the sixth pixel to obtain the pixel value of the sixth pixel.

[0060] In an alternative embodiment, the image processing device further includes a storage module. The storage module is configured to store the fifth pixel of each column of the second image in a memory.

[0061] In an alternative embodiment, the image processing device obtains the column transformation parameters corresponding to each column of pixels, including: sequentially reading the column transformation parameters corresponding to each column of pixels from the memory in a column direction, where the memory stores a column mapping relationship table, and the column mapping relationship table includes the column transformation parameters corresponding to each column of pixels in the original image. The obtaining module is specifically configured to: sequentially read the row transformation parameters corresponding to each row of pixels from the memory in a row direction; sequentially read the column transformation parameters corresponding to each column of pixels from the memory in a column direction.

[0062] In an alternative embodiment, the image processing device further includes a storage module. The sending module is configured to send the target image to the imaging device, and the imaging device is configured to perform imaging based on the foregoing target image so that the original image is projected onto an imaging plane based on the foregoing target image. Wherein, the imaging device may be a projection imaging system.

[0063] Fifth aspect, the present application provides a method for constructing a mapping relationship, which is used to calculate the row mapping relationship table and the column mapping relationship table introduced in the foregoing implementation method. This method can be executed in the foregoing image processing device; it can also be executed by other computing devices, and the calculated row mapping relationship table and column mapping relationship table are stored in the foregoing image processing device (or an external memory connected to the image processing device). In this method, the computing device obtains the coordinates of each calibration point in the calibration image; obtains the coordinates of each distorted point in the distorted image, where the distorted image is an image obtained through the imaging distortion of the imaging device, and the distorted points in the distorted image correspond one-to-one with the calibration points in the calibration image. Then, the computing device determines the row mapping relationship table and the column mapping relationship table respectively according to the coordinates of the distorted point and the coordinates of the calibration point.

[0064] Optionally, the computing device stores the foregoing row mapping relationship table and the foregoing column mapping relationship table in a memory.

[0065] In an optional implementation manner, the computing device determines the row mapping relationship table and the column mapping relationship table respectively according to the coordinates of the distorted point and the coordinates of the calibration point, including: S1. Fix the ordinate as the ordinate of the distorted point, and construct a row transformation model with the abscissa of the calibration point as the independent variable and the abscissa of the corresponding distorted point as the dependent variable. S2. Calculate the abscissa of each pixel in the third image on the original image according to the row transformation model to obtain the row mapping relationship table, where the coordinates of each pixel in the third image are integers. S3. Fix the abscissa as the abscissa of the calibration point, and construct a column transformation model with the ordinate of the calibration point as the independent variable and the ordinate of the corresponding distorted point as the dependent variable. S4. Calculate the ordinate of each pixel in the fourth image on the third image according to the column transformation model to obtain the column mapping relationship table, where the coordinates of each pixel in the fourth image are integers, and the fourth image is a pre-distorted image of the calibration image.

[0066] In an optional implementation manner, the computing device determines the row mapping relationship table and the column mapping relationship table respectively according to the coordinates of the distorted point and the coordinates of the calibration point, including: T1. Fix the ordinate as the ordinate of the distorted point, and construct a column transformation model with the ordinate of the calibration point as the independent variable and the ordinate of the corresponding distorted point as the dependent variable. T2. Calculate the ordinate of each pixel in the fifth image on the original image according to the column transformation model to obtain the column mapping relationship table, where the coordinates of each pixel in the fifth image are integers. T3. Fix the abscissa as the abscissa of the calibration point, and construct a row transformation model with the abscissa of the calibration point as the independent variable and the abscissa of the corresponding distorted point as the dependent variable. T4. Calculate the abscissa of each pixel in the sixth image on the fifth image according to the row transformation model to obtain the row mapping relationship table, where the coordinates of each pixel in the sixth image are integers, and the sixth image is a pre-distorted image of the calibration image.

[0067] In an alternative embodiment, determining the row mapping relationship table and the column mapping relationship table respectively according to the coordinates of the distortion points and the coordinates of the calibration points includes: constructing a distortion model with the abscissa and ordinate of the distortion points as independent variables and the abscissa and ordinate of the calibration points as dependent variables; calculating the abscissa and ordinate of each pixel in the seventh image on the calibration image according to the distortion model to obtain a distortion mapping relationship table, where the abscissa and ordinate of each pixel in the seventh image are both integers; splitting the coordinates in the distortion mapping relationship table along the horizontal direction to obtain the row mapping relationship table; splitting the coordinates in the distortion mapping relationship table along the vertical direction to obtain the column mapping relationship table.

[0068] In a sixth aspect, the present application provides a computing device for calculating the row mapping relationship table and the column mapping relationship table introduced in the foregoing implementation method. The computing device includes: an acquisition module and a determination module. Among them, the acquisition module is used to acquire the coordinates of each calibration point in the calibration image and the coordinates of each distortion point in the distortion image. Among them, the distortion image is an image obtained through the imaging distortion of the imaging device, and the distortion points in the distortion image correspond one-to-one to the calibration points in the calibration image. The determination module is used to determine the row mapping relationship table and the column mapping relationship table respectively according to the coordinates of the distortion points and the coordinates of the calibration points.

[0069] Optionally, the computing device stores the foregoing row mapping relationship table and the foregoing column mapping relationship table in a memory.

[0070] In an alternative embodiment, the determination module is specifically configured to: fix the ordinate as the ordinate of the distortion point, and construct a row transformation model with the abscissa of the calibration point as the independent variable and the abscissa of the corresponding distortion point as the dependent variable. Calculate the abscissa of each pixel in the third image on the original image according to the row transformation model to obtain the row mapping relationship table, where the coordinates of each pixel in the third image are integers. Fix the abscissa as the abscissa of the calibration point, and construct a column transformation model with the ordinate of the calibration point as the independent variable and the ordinate of the corresponding distortion point as the dependent variable. Calculate the ordinate of each pixel in the fourth image on the third image according to the column transformation model to obtain the column mapping relationship table, where the coordinates of each pixel in the fourth image are all integers, and the fourth image is a pre-distortion image of the calibration image.

[0071] In an alternative embodiment, the determining module is specifically configured to: fix the ordinate of the distortion point, use the ordinate of the calibration point as the independent variable, and use the ordinate of the corresponding distortion point as the dependent variable to construct a column transformation model. Calculate the ordinate of each pixel in the fifth image on the original image according to the column transformation model to obtain the column mapping relationship table, where the coordinates of each pixel in the fifth image are integers. Fix the abscissa as the abscissa of the calibration point, use the abscissa of the calibration point as the independent variable, and use the abscissa of the corresponding distortion point as the dependent variable to construct a row transformation model. Calculate the abscissa of each pixel in the sixth image on the fifth image according to the row transformation model to obtain the row mapping relationship table, where the coordinates of each pixel in the sixth image are integers, and the sixth image is a pre-distorted image of the calibration image.

[0072] In an alternative embodiment, the determining module is specifically configured to: use the abscissa and ordinate of the distortion point as independent variables, and the abscissa and ordinate of the calibration point as dependent variables to construct a distortion model; calculate the abscissa and ordinate of each pixel in the seventh image on the calibration image according to the distortion model to obtain a distortion mapping relationship table, where the abscissa and ordinate of each pixel in the seventh image are integers; split the coordinates in the distortion mapping relationship table along the horizontal direction to obtain the row mapping relationship table; split the coordinates in the distortion mapping relationship table along the vertical direction to obtain the column mapping relationship table.

[0073] In a seventh aspect, the present application further provides an image processing apparatus, which includes a processor and an internal memory, and the processor is coupled to the internal memory; the processor is configured to read a row mapping parameter table and a column mapping parameter table from an external memory into the internal memory; the internal memory further stores a program, and when the program instructions stored in the internal memory are executed by the processor, the image processing apparatus implements the method described in any one of the foregoing first aspect or third aspect embodiments.

[0074] In an eighth aspect, the present application further provides an imaging apparatus, including an image processing apparatus and a display device. Among them, the image processing apparatus is configured to generate a target image according to the original image according to the method described in any one of the foregoing first aspect or third aspect embodiments, and transmit the target image to the display device so that the imaging apparatus projects the original image. Optionally, the imaging apparatus is a projection imaging system.

[0075] In a ninth aspect, the present application further provides a computer-readable storage medium, including a computer program, and the computer program is executed by a processor to implement the method described in any one of the foregoing first aspect or third aspect embodiments.

[0076] In a tenth aspect, the present application also provides a computer program product including instructions. The computer program product includes computer program code which, when running on a computer, causes the computer to execute the method described in any one of the implementation manners of the foregoing first aspect or third aspect.

[0077] As can be seen from the above technical solutions, the embodiments of the present application have the following advantages:

[0078] In the embodiments of the present application, when the image processing device determines a target image (i.e., a pre-distorted image) based on an original image, the image remapping is decomposed into two single-direction mappings. Specifically, the image processing device first performs a row transformation on each row of pixels in the original image, and then performs a column transformation on each column of pixels based on the row transformation result (i.e., the first image) to obtain the target image; or, first performs a column transformation on each column of pixels in the original image, and then performs a row transformation on each row of pixels based on the column transformation result to obtain the target image. That is to say, when the image processing device calculates a row of pixels in the first image, it only needs to read one row of pixels in the original image and does not need to read across rows; similarly, when the image processing device calculates a column of pixels in the target image, it only needs to read one column of pixels in the first image and does not need to read across columns. Therefore, the image processing device can read the original image in an orderly manner, thereby improving the efficiency of the image processing device in processing the original image. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application.

[0080] Figure 1A A simple structural diagram of a projection imaging system;

[0081] Figure 1B An example diagram of an original image in the present application;

[0082] Figure 1C An example diagram of a distorted image in the present application;

[0083] Figure 2 A flowchart of an image remapping method in the present application;

[0084] Figure 3A An example diagram of a row transformation process in the present application;

[0085] Figure 3B An example diagram of a column transformation process in the present application;

[0086] Figure 3CThis is an example diagram of the row transformation and column transformation processes in this application;

[0087] Figure 4 This is a flowchart of the mapping relationship construction method in this application;

[0088] Figure 5A This is an example diagram of the calibrated image in this application;

[0089] Figure 5B This is a schematic diagram of an embodiment of the mapping relationship construction method in this application;

[0090] Figure 6A This is another flowchart of the image remapping method in this application;

[0091] Figure 6B This is another example diagram of the row transformation and column transformation processes in this application;

[0092] Figure 7 This is a schematic diagram of an embodiment of the image processing apparatus in this application;

[0093] Figure 8 This is another schematic diagram of an embodiment of the image processing apparatus in this application. Detailed implementation manners

[0094] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments.

[0095] Terms such as "first", "second", "third", "fourth", etc. (if any) in the specification, claims and above-mentioned accompanying drawings of this application are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments described here can be implemented in an order different from that shown or described here. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or devices.

[0096] The embodiments of this application provide an image remapping method and an image processing apparatus for improving the efficiency of generating a pre-distorted image based on an original image.

[0097] The image remapping method of the present application is mainly applied to offset the distortion generated by an imaging device based on a projection imaging system. The principle of distortion generation and the principle of offsetting distortion will be introduced below in combination with the structure of the projection imaging system:

[0098] As Figure 1A shown, it is a schematic diagram of the simple structure of the projection imaging system 10. Among them, the projection imaging system 10 includes an optical module 101 and a display device 102. Among them, the display device 102 is used to generate an image that needs to be projected and imaged. The light emitted by the display device 102 reaches the imaging plane 11 after passing through the optical module 101, and the image is presented on the imaging plane 11.

[0099] In practical applications, the aforementioned display device 102 may be a liquid crystal display (LCD), digital light processing (DLP), or liquid crystal on silicon (LCOS), etc. The aforementioned optical module 101 may include one or more lenses, free-form mirrors, etc.

[0100] However, since the projection imaging system 10 will inevitably have aberrations during the imaging process, the image presented on the aforementioned imaging plane 11 is not the image displayed by the display device 102 (hereinafter referred to as the original image), but the distorted image (hereinafter referred to as the distorted image) generated based on the aforementioned image. In order to offset the aforementioned distortion, the original image needs to be processed into a pre-distorted image with the opposite distortion direction and the same distortion degree as the distorted image. Then, the aforementioned pre-distorted image is generated on the display device 102, so that when the light emitted by the display device 102 passes through the optical module 101 and reaches the imaging plane 11, the image presented on the imaging plane 11 is the original image obtained by distorting the pre-distorted image. Thus, the image distortion caused by optical aberrations can be offset. Exemplarily, if the display device 102 generates a normal image as Figure 1B shown, then the image projected by the aforementioned projection imaging system 10 onto the imaging plane 11 may be any one of the distorted images as Figure 1C shown. Exemplarily, if the original image displayed by the display device 102 is Figure 1B shown as a normal image and the distorted image presented on the imaging plane 11 is the pincushion distortion image shown in (a) of Figure 1C . At this time, the aforementioned original image (i.e., the normal image shown in Figure 1B ) can be processed into a barrel distortion image shown in (b) of Figure 1C , and the aforementioned barrel distortion image is generated by the display device 102. Then, the imaging plane 11 can present an image as Figure 1BThe normal image shown, which is obtained by distorting the barrel distortion image (i.e., the pre-distortion image).

[0101] In this regard, the traditional technique often uses the method of bilinear interpolation. One pixel in the pre-distortion image is calculated based on four pixels arranged in a rectangle in the original image, and the pixel values of each pixel in the pre-distortion image are obtained by cycling in this way. However, in this process, at least four pixels that are not in the same row and not in the same column need to be read from the original image for each pixel calculated in the pre-distortion image. Therefore, a large number of out-of-order accesses to the original image need to be performed, which will affect the processing efficiency of the processor.

[0102] In this regard, the image remapping method proposed in the embodiments of the present application can not only be applied to correct the distortion caused by an imaging device based on a projection imaging system, but also establish a mapping relationship between the original image and the pre-distortion image in both the row and column directions. Through two single-direction mapping processes, the aforementioned original image is processed into an image after row transformation (or an image after column transformation) and an image after row transformation and column transformation (or an image after column transformation and row transformation) to obtain the pre-distortion image. Since there is no need to access data across rows or columns, the processing efficiency of the processor can be improved.

[0103] It should be understood that the form of distortion that the image remapping method in the embodiments of the present application can correct can be pincushion distortion (as shown in (a) in Figure 1C ), barrel distortion (as shown in (b) in Figure 1C ), trapezoidal distortion (as shown in (c) in Figure 1C ), fan distortion (as shown in (d) in Figure 1C ), and parallelogram distortion (as shown in (e) in Figure 1C ), or any combination or superposition of any at least two of the aforementioned distortion forms, or a distortion without a specific shape, etc. Specifically, it is not limited here.

[0104] It should be noted that the imaging device based on the aforementioned projection imaging system may be a head up display (HUD) (also known as a head-up display), or may be a head mounted display (HMD) (also known as a head-mounted display). Among them, the aforementioned HUD can be a display in an aircraft cockpit system, installed above the central instrument panel of the cockpit, serving as a window for information exchange between the pilot and the aircraft; the aforementioned HUD can also be applied in the vehicle to everything (V2X) field, serving as an in-vehicle display to provide driving information for the driver. The aforementioned HMD is often used in augmented reality (AR) technology and virtual reality (VR) technology. For example, AR glasses (also known as AR displays) and VR glasses (also known as VR displays). When the imaging device adopts any of the aforementioned implementation methods, the imaging plane for receiving the image projected by the imaging device is the user's retina. In addition, the imaging device can also be an ordinary projector. At this time, the imaging plane for receiving the image projected by the projector is an image receiving device such as a screen.

[0105] It should also be noted that the image processing device for performing the aforementioned image remapping method may include an image processing chip located in the aforementioned HUD or HMD. For example, a field programmable gate array (FPGA), a digital signal process (DSP), a central processing unit (CPU), or a graphics processing unit (GPU) and other chips with image processing functions.

[0106] In addition, the aforementioned display device 102 can be integrated with the aforementioned image processing device. For example, both the display device 102 and the image processing device are integrated in the aforementioned HUD (or HMD). Each pre-distorted image calculated by the image processing device can be immediately input into the aforementioned display device 102 for display. The aforementioned display device 102 can also be distributed with the aforementioned image processing device. For example, the image processing device is externally placed in a computer connected to the aforementioned HUD (or HMD). The computer can transmit the calculated pre-distorted image to the display device 102 in the aforementioned HUD (or HMD) for display in a wired or wireless manner.

[0107] The image remapping method proposed in this application will be introduced based on the foregoing scenario. In this method, as Figure 2 shown, if the image processing device first performs a row transformation on the original image and then performs a column transformation on the image after the row transformation, the original image will perform the following steps:

[0108] Step 201: Obtain each row of pixels of the original image.

[0109] Among them, the original image is a non-distorted image that is expected to be presented on the imaging plane. The solution proposed in this application needs to perform pre-distortion processing on the original image to obtain a target image (i.e., a pre-distorted image). By inputting the foregoing target image into a projection imaging system for projection, the foregoing original image can be presented on the imaging plane.

[0110] In this embodiment, the image processing device can read the foregoing original image from an external memory through a data transmission interface or other communication interfaces, or receive the foregoing original image from other external devices. The image processing device can temporarily store the obtained original image in the internal memory of the image processing device, and read one row of pixels from the internal memory every time one row of pixels is calculated. Specifically, it is not limited here. However, no matter which of the foregoing methods is used to obtain the original image, the image processing device will sequentially read each row of pixels of the original image, and in the subsequent processing process, each row of pixels will also be processed sequentially.

[0111] In an alternative embodiment, if the image processing device obtains each row of pixels of the original image from a memory (for example, double data rate synchronous dynamic random access memory (DDR SDRAM) (abbreviation: DDR)), then this process can specifically be that the image processing device reads each row of pixels in sequence in the row direction from the starting address storing the original image. Specifically, depending on the number of pixels read each time according to the bit width of the memory, the number of read operations required for the image processing device to read one row of pixels is different. For example, if the bit width is 256bit and a three-channel pixel is 8×3 = 24bit, then 10 pixels can be read at a time. If the original image is a 100×100 image, then the image processing device only needs 10 read operations to read a whole row of pixels of the original image.

[0112] It should be understood that the image processing device obtaining each row of pixels of the original image can be specifically understood as that the image processing device reads the pixel value of each pixel in each row of pixels and the coordinates of the pixel in the original image. Among them, the coordinates of the pixel in the original image include the abscissa and the ordinate.

[0113] In addition, if the image processing device obtains each row of pixels of the original image from the memory, the process of the image processing device obtaining each row of pixels in the original image can be specifically as follows: The image processing device reads each row of pixels in the original image from the memory row by row, that is, reads the pixels with consecutive abscissas in a certain row. Optionally, the image processing device can read in the order of the row ordinal numbers. For example, after reading the pixels of the first row, then read the pixels of the second row, and then read the pixels of the third row; it can also be read in the reverse order of the row ordinal numbers. For example, after reading the pixels of the second row, then read the pixels of the first row. In addition, the image processing device can start reading from the first pixel of a certain row or start reading from the Nth pixel of a certain row. Specifically, no limitation is made here.

[0114] Step 202: Obtain the row transformation parameter corresponding to each row of pixels.

[0115] In this embodiment, there is no limitation on the time sequence between step 201 and step 202. That is to say, the image processing device can execute step 201 first and then step 202, or execute step 202 first and then step 201, or can also execute the foregoing steps 201 and 202 simultaneously. Specifically, no limitation is made here. In the subsequent embodiments, only the example that the image processing device can first obtain a row of pixels from the original image and then obtain the row transformation parameter corresponding to the obtained row of pixels will be introduced.

[0116] The row transformation parameter is a parameter for performing row transformation processing on a certain row of pixels in the original image. The row transformation processing refers to moving the position of the pixels in the row direction. It can also be understood that the abscissa of the pixels changes while the ordinate does not change. For the convenience of introduction, the image obtained only by performing row transformation processing on the original image is called the first image. Then, the row transformation parameter is used to indicate the corresponding relationship between the abscissa of the pixels in the original image and the abscissa of the pixels in the first image. It should also be understood that the change in the position of the pixels (i.e., the movement of the pixels) is essentially the migration of the pixel values. Therefore, the row transformation parameter can be used to determine from which pixel in the original image the pixel value of a certain pixel in the first image is taken. For the convenience of introduction, any pixel in the first image is called the first pixel, and the point corresponding to the first pixel in the original image is called the first pixel value-taking point. Then, the row transformation parameter includes the abscissa of the first pixel value-taking point corresponding to the first pixel in the original image. The first pixel value-taking point is used to determine the pixel value of the first pixel in the first image, and the ordinate of the first pixel value-taking point in the original image is the same as the ordinate of the first pixel in the first image.

[0117] Optionally, the line transformation parameters can be stored in a memory in the form of a line mapping relationship table, and the line mapping relationship table includes a plurality of line transformation parameters.

[0118] In an alternative embodiment, the number of line transformation parameters in the line mapping relationship table is equal to the number of pixels in the first image described above, and each line transformation parameter in the line mapping relationship table corresponds to each pixel in the first image in a one-to-one correspondence according to coordinates. The position of the line transformation parameter in the line mapping relationship table is used to represent the coordinates of the first pixel in the first image determined based on the line transformation parameter. Exemplarily, if the first image is an image of 1080×1080, the corresponding line mapping table of the original image is also 1080×1080. If the line transformation parameter a1 is located in the second row and the third column of the line mapping relationship table, then the line transformation parameter a1 is used to determine the pixel value of the pixel with coordinates (3, 2) in the first image.

[0119] In another alternative embodiment, the line mapping relationship table can be stored in the memory in a compressed manner, that is, the compressed line mapping relationship table is stored in the memory. The number of line transformation parameters in the compressed line mapping relationship table is less than the number of pixels in the first image described above. At this time, when the image processing device obtains the line transformation parameter corresponding to each row of pixels, it can be understood that the compressed line mapping relationship table is read and the compressed line mapping relationship table is restored to a complete line mapping relationship table, and then, a row of line transformation parameters corresponding to the first image is read from the complete line mapping relationship table; it can also be understood that several line transformation parameters in the compressed line mapping relationship table are read, and then, the several line transformation parameters read from the compressed line mapping relationship table are restored to a row of line transformation parameters. The above restoration process can adopt an interpolation algorithm, which is not specifically limited here.

[0120] It should be noted that the aforementioned pixel values are the values assigned by a computer when the image is digitized, representing the average brightness information of a certain pixel in the image. The aforementioned original image can be a single-channel image, a three-channel image, or a four-channel image. Among them, a single-channel image is also called a grayscale image, and each pixel is represented by a pixel value. If the pixel value of a single-channel image is represented by 8 bits, the value range of the pixel value of this single-channel image is: 0 (black) to 255 (white). Among them, a three-channel image generally refers to an RGB image, which can present colors or represent a black-and-white image. If the pixel value of a three-channel image is represented by 8 bits, each pixel value is represented by three channel values, that is, the pixel value is represented by the superposition of red (0-255), green (0-255), and blue (0-255). A four-channel image is based on a three-channel image and adds a brightness channel to represent the degree of transparency. If the original image is a single-channel image, the first image and the target image calculated by this image processing device are also single-channel images; if the original image is a three-channel image, the first image and the target image calculated by this image processing device are also three-channel images; specifically, it is not limited here. If the original image is a three-channel image, the pixel value of a pixel is composed of three channel values, and the same calculation needs to be performed on each channel value during the calculation process.

[0121] In this embodiment, the process by which the image processing device obtains the row transformation parameters corresponding to each row of pixels can specifically be that the image processing device reads the row transformation parameters corresponding to each row of pixels row by row from the memory, that is, reads the row transformation parameters of consecutive abscissas in a certain row. Optionally, the image processing device can read in the order of the row ordinal numbers. For example, after reading the row transformation parameters of the first row, then read the row transformation parameters of the second row, and then read the row transformation parameters of the third row; it can also be read in reverse order of the row ordinal numbers. For example, after reading the row transformation parameters of the second row, then read the row transformation parameters of the first row. In addition, the image processing device can start reading from the first row transformation parameter of a certain row or start reading from the Nth row transformation parameter of a certain row. Specifically, it is not limited here.

[0122] In this embodiment, the image processing device can read the row mapping table by reading the original image. That is to say, the order in which the image processing device reads the pixels in the original image is the same as the order in which the image processing device reads the row transformation parameters in the row mapping table. Exemplarily, if the image processing device starts reading pixels from the Nth row and Mth column in the original image, then the image processing device should also start reading the row transformation parameters from the Nth row and Mth column in the row transformation parameter table.

[0123] Step 203: Determine the pixel value of each pixel in the first image according to each row of pixels in the original image and the row transformation parameters corresponding to each row of pixels respectively.

[0124] The first image is an image with pre-distortion in the row direction but not in the column direction, that is, the first image is the result of row transformation processing of the original image to be obtained.

[0125] In this embodiment, the process of calculating the first image by the image processing device can be understood as the process of determining the pixel value of each pixel in the first image by the image processing device. Since the position (i.e., coordinates) of each pixel in the first image is known, i.e., the horizontal coordinate and the vertical coordinate of each pixel in the first image are continuous integers. Therefore, the row transformation parameters of the first image can be queried according to the horizontal coordinate and the vertical coordinate of the first pixel, i.e., the row transformation parameters with the same horizontal coordinate and vertical coordinate can be queried according to the horizontal coordinate and the vertical coordinate of the first pixel.

[0126] For ease of understanding, Figure 3A Take as an example to introduce. Among them, the original image 31 is a 4×4 image (that is, an image with four rows and four columns), and the row mapping relationship table corresponding to the original image 31 also has four rows and four columns. In the original image 31, each pixel has a certain pixel value, which is represented by s(i, j), where i represents the horizontal coordinate and j represents the vertical coordinate. For example, s(1, 1) represents the pixel value of a pixel with a horizontal coordinate of 1 and a vertical coordinate of 1. In the first image 33, the pixel value of each pixel needs to be calculated by the image processing device according to the original image 31 and the row mapping relationship table 32. Taking the pixel 331 in the first row and the first column of the first image 33 as an example, the horizontal coordinate of the pixel 331 is 1 and the vertical coordinate is 1, that is, the pixel 331 is located in the first row and the first column of the first image. Then, the row transformation parameter corresponding to the pixel 331 is the row transformation parameter 321 corresponding to the first row and the first column in the row mapping relationship table 32.

[0127] Then, the image processing device determines the first pixel value point corresponding to the first pixel according to the row transformation parameter, and determines at least one third pixel according to the horizontal coordinate of the first pixel value point. Then, the image processing device determines the pixel value of the first pixel according to the pixel value of the at least one third pixel. The third pixel and the first pixel value point are located in the same row.

[0128] The horizontal coordinate of the first pixel value point may be an integer or a non-integer.

[0129] In an optional embodiment, if the horizontal coordinate of the first pixel value point is an integer, then the image processing device can directly determine that the first pixel value point is the third pixel, and then assign the pixel value of the third pixel to the first pixel to obtain the pixel value of the first pixel.

[0130] Still with the above Figure 3AFor example, if the first pixel is pixel 332, then the row transformation parameter corresponding to this first pixel is row transformation parameter 322. At this time, the value of this row transformation parameter 322 is 4, and this row transformation parameter 332 is located in the second row of the row mapping relationship table 32. Then, the pixel value of this pixel 332 is determined by the pixel value of the fourth pixel in the second row of the original image (that is, the pixel value with the abscissa of 4 and the ordinate of 2). In this example, 4 is the abscissa of the first pixel value point, and this first pixel value point is the fourth pixel in the second row of the original image (that is, the pixel with the abscissa of 4 and the ordinate of 2); this third pixel is directly determined by the first pixel value point, that is, this third pixel is this first pixel value point.

[0131] In another alternative implementation, if the abscissa of the first pixel value point is a non-integer, then the image processing device needs to determine one or more third pixels near this non-integer. Specifically, the image processing device determines at least one pixel adjacent to the first pixel value point as this third pixel, and this pixel adjacent to the first pixel value point is in the same row as the first pixel value point. Then, the image processing device uses an interpolation algorithm to calculate the first weighted average of the pixel values of this at least one third pixel according to the abscissa of the first pixel value point and the abscissas of this at least one third pixel, and assigns this first weighted average to this first pixel to obtain the pixel value of this first pixel.

[0132] Still taking the above Figure 3A as an example, if the first pixel is pixel 331, then the row transformation parameter corresponding to this first pixel is row transformation parameter 321. At this time, the value of this row transformation parameter 321 is 1.6, and this row transformation parameter 331 is located in the first row of the row mapping relationship table 32. Therefore, this first pixel value point is the point with coordinates (1.6, 1) in the original image. At this time, the image processing device needs to determine the pixels adjacent to this first pixel value point and in the same row as this first pixel value point as the third pixels. It can be seen from the original image 31 that the point (1.6, 1) is between pixel 311 (with the abscissa of 1 and the ordinate of 1) and pixel 312 (with the abscissa of 2 and the ordinate of 1). Then, the image processing device can determine pixel 311 and pixel 312 as the third pixels. If the pixel value of pixel 311 is 100 and the pixel value of pixel 312 is 120, using the interpolation algorithm, the weighted average can be determined as = 112. In this example, 1.6 is the abscissa of the first pixel value point, this first pixel value point is the point (1.6, 1), and the third pixels are pixel 311 and pixel 312.

[0133] In practical applications, the image processing device may determine that the two pixels to the left of the first pixel value point and the two pixels to the right of the first pixel value point are the third pixel values, and specific details are not limited here. Additionally, the common one-dimensional linear interpolation is listed in the foregoing example. In practical applications, the image processing device may also adopt interpolation methods such as bilinear interpolation in the row direction and multiple non-linear interpolations, and specific details are not limited here.

[0134] By analogy, after the image processing device calculates each pixel in the first image, the foregoing first image can be obtained.

[0135] In this embodiment, when calculating the pixel value of a certain pixel in the first image, the image processing device only needs one or more third pixels in the same row as the pixel in the original image, rather than the pixels in other rows. In the traditional technology, if it is necessary to calculate the pixel value of a pixel in the first image, at least four pixels in two rows of the original image are required. Therefore, the solution of the present application is beneficial to improving the efficiency of the image processing device in calculating the pre-distorted image. In addition, since the image processing device only needs one row of pixels instead of two rows of pixels during calculation, when the image processing device reads the pixels or row transformation parameters in the memory, it also does not need to read two rows of pixels or two rows of row transformation parameters, avoiding out-of-order access to the memory, reducing the number of times of activating the memory, solving the random read and write problem in complex remapping, and reducing the hardware implementation difficulty. When calculating the target image in parallel, the original image pixels required by multiple parallel target pixels are also continuous, and there is no need to read redundant original images, reducing the size of the internal memory required for implementing high parallel computing.

[0136] In this embodiment, after the image processing device executes step 203 and before executing step 204, the image processing device may also store the foregoing first image in the memory. It should be understood that the image processing device may write the pixel values of each row of pixels in the first image into the memory as soon as they are calculated. The image processing device may also store the calculated row of pixels in the cache, and after calculating multiple rows of pixels, the image processing device writes the foregoing multiple rows of pixels into the memory in sequence. Specific details are not limited here.

[0137] In addition, the image processing device may directly write the pixel values of each row of pixels into the memory row by row, or may perform a transpose process on the pixel values of each row of pixels and write each row of pixels into the memory column by column. Specific details are not limited here.

[0138] Step 204: Obtain each column of pixels of the first image.

[0139] Among them, the first image is an image obtained by only performing a row transformation on the original image. For specific details, please refer to the relevant introduction in the foregoing step 202, which will not be elaborated here.

[0140] Specifically, the image processing device sequentially reads each column of pixels of the foregoing first image from the memory. If the pixel values of each row of pixels of the image processing device are written into the memory row by row, the image processing device will read each column of pixels of the first image in the memory column by column. If the image processing device transposes the pixel values of each row of pixels and writes each row of pixels into the memory column by column, the image processing device will read each column of pixels of the first image in the memory row by row.

[0141] Step 205: Obtain the column transformation parameter corresponding to each column of pixels.

[0142] In this embodiment, there is no limitation on the time sequence between step 204 and step 205. That is to say, the image processing device can execute step 204 first and then step 205, or execute step 205 first and then step 204, or execute the foregoing step 204 and step 205 simultaneously. Specifically, it is not limited here. In the subsequent embodiments, only the example in which the image processing device can first obtain a column of pixels from the first image and then obtain the column transformation parameter corresponding to the column of pixels will be introduced.

[0143] The column transformation parameter is a parameter used to perform a column transformation on a certain column of pixels in the first image. The column transformation process refers to the movement of pixels in the column direction. It can also be understood that the ordinate of the pixels changes while the abscissa does not change. The image obtained by only performing a column transformation on the first image is called the target image, that is, the pre-distorted image based on the original image. Inputting the target image into the projection imaging system, the original image obtained by distorting the pre-distorted image will be displayed on the imaging plane. The column transformation parameter is used to indicate the corresponding relationship between the ordinate of the pixels in the first image and the ordinate of the pixels in the target image. It should also be understood that the change in the position of the pixels (i.e., the movement of the pixels) is essentially the migration of the pixel values. Therefore, the column transformation parameter can be used to determine from which pixel in the first image the pixel value of a certain pixel in the target image is taken. For the convenience of introduction, any pixel in the target image is called the second pixel, and the point corresponding to the second pixel in the first image is called the second pixel value-taking point. Thus, the column transformation parameter includes the ordinate of the second pixel value-taking point corresponding to the second pixel in the first image. The second pixel value-taking point is used to determine the pixel value of the second pixel in the target image, and the abscissa of the second pixel value-taking point in the first image is the same as the abscissa of the second pixel in the target image.

[0144] Optionally, the column transformation parameters may be stored in a memory in the form of a column mapping relation table, and the column mapping relation table includes a plurality of column transformation parameters.

[0145] In an alternative embodiment, the number of column transformation parameters in the column mapping relation table is equal to the number of pixels in the target image, and each column transformation parameter in the column mapping relation table corresponds to each pixel in the target image in a one-to-one correspondence according to coordinates. The position of the column transformation parameter in the column mapping relation table is used to represent the coordinates of the second pixel in the target image determined based on the column transformation parameter. Exemplarily, if the target image is an image of 1080×1080, the corresponding column mapping table of the target image is also 1080×1080. If the column transformation parameter b1 is located in the second row and the third column of the column mapping relation table, then the column transformation parameter b1 is used to determine the pixel value of the pixel with coordinates (3, 2) in the target image.

[0146] In another alternative embodiment, the column mapping relation table may be stored in a memory in a compressed manner, that is, what is stored in the memory is a compressed column mapping relation table. The number of column transformation parameters in the compressed column mapping relation table is less than the number of pixels in the foregoing target image. At this time, when the foregoing image processing device obtains the column transformation parameters corresponding to each column of pixels, it can be understood that the compressed column mapping relation table is read and the compressed column mapping relation table is restored to a complete column mapping relation table, and then, a column of column transformation parameters corresponding to the target image is read from the foregoing complete column mapping relation table; it can also be understood that several column transformation parameters in the compressed column mapping relation table are read, and then, the several column transformation parameters read from the compressed column mapping relation table are restored to a column of column transformation parameters. The foregoing restoration process may adopt an interpolation algorithm, and specific details are not limited herein.

[0147] In this embodiment, the process of the image processing device obtaining the column transformation parameters corresponding to each column of pixels may specifically be that the image processing device reads from the memory the column transformation parameters corresponding to each column of pixels by column, that is, reads the column transformation parameters of consecutive ordinates in a certain column. Optionally, the image processing device may read in the order of the column ordinates. For example, after reading the row transformation parameters of the first column, then read the column transformation parameters of the second column, and then read the column transformation parameters of the third column; it may also be read in the reverse order of the column ordinates. For example, after reading the column transformation parameters of the second column, then read the column transformation parameters of the first column. In addition, the image processing device may start reading from the first column transformation parameter of a certain column, or may start reading from the Nth column transformation parameter of a certain column. Specific details are not limited herein.

[0148] In this embodiment, the image processing device may read the column mapping relationship table by reading the first image. That is to say, the order in which the image processing device reads the first pixels in the first image is the same as the order in which the image processing device reads the column transformation parameters in the column mapping relationship table. Exemplarily, if the image processing device starts reading the first pixel from the Nth row and Mth column of the first image, the image processing device should also start reading the column transformation parameters from the Nth row and Mth column of the column transformation parameter table.

[0149] Step 206: Determine the pixel value of each pixel in the target image according to each column of pixels in the first image and the column transformation parameters corresponding to each column of pixels.

[0150] Among them, the target image is a pre-distortion image used to offset the imaging distortion of the imaging device. Inputting the target image into the projection imaging system, the original image obtained by distorting the pre-distortion image will be displayed on the imaging plane.

[0151] In this embodiment, the process of the image processing device calculating the target image can be understood as the process of the image processing device determining the pixel value of each pixel in the target image. Since the position (i.e., coordinates) of each pixel in the target image is known, that is, the abscissa and ordinate of each pixel in the target image are consecutive integers. Therefore, the column transformation parameters of the target image can be queried according to the abscissa and ordinate of the second pixel, that is, query the column transformation parameters with the same abscissa and ordinate according to the abscissa and ordinate of the second pixel.

[0152] For ease of understanding, take Figure 3B as an example for introduction. Among them, the first image 33 is a 4×4 image, and the column mapping relationship table corresponding to the first image 33 is also four rows and four columns. In the first image 33, the pixel value of each pixel has been calculated by the image processing device. The pixel values of each pixel in the target image 35 need to be calculated by the image processing device according to the first image 33 and the column mapping relationship table 34. Taking the first pixel 351 in the first row and the first column of the target image 35 as an example, the abscissa of the pixel 351 is 1 and the ordinate is 1, that is, the pixel 351 is located in the first row and the first column of the target image. Then, the column transformation parameter corresponding to the pixel 351 is the column transformation parameter 341 corresponding to the first row and the first column in the column mapping relationship table 34.

[0153] Then, the image processing device will determine the second pixel value point corresponding to the target pixel according to the column transformation parameter, and determine at least one fourth pixel according to the ordinate of the second pixel value point. Then, the image processing device determines the pixel value of the target pixel according to the pixel values of the at least one fourth pixel. Among them, the fourth pixel and the second pixel value point are in the same row.

[0154] Among them, the abscissa of the second pixel value point may be an integer or a non-integer.

[0155] In an alternative embodiment, if the abscissa of the second pixel value point is an integer, then the image processing device can directly determine that the second pixel value point is the fourth pixel, and then assign the pixel value of the fourth pixel to the second pixel to obtain the pixel value of the second pixel.

[0156] Still taking the above Figure 3B as an example, if the second pixel is pixel 352, then the column transformation parameter corresponding to the second pixel is column transformation parameter 342. At this time, the value of the column transformation parameter 342 is 3, and the column transformation parameter 342 is located in the fourth column of the column mapping table 34. Then the pixel value of the pixel 342 is determined by the pixel value of the third pixel in the fourth column of the first image 33 (that is, the pixel value of the pixel with abscissa 4 and ordinate 3). In this example, 3 is the abscissa and ordinate of the second pixel value point, and the second pixel value point is the third pixel in the fourth column of the first image 33 (that is, the pixel with abscissa 4 and ordinate 3); the fourth pixel is directly determined by the second pixel value point, that is, the fourth pixel is the second pixel value point.

[0157] In another alternative embodiment, if the abscissa of the second pixel value point is a non-integer, then the image processing device needs to determine one or more fourth pixels near the non-integer. Specifically, the image processing device determines at least one pixel adjacent to the second pixel value point as the fourth pixel, and the pixel adjacent to the second pixel value point is in the same row as the second pixel value point. Then, the image processing device uses an interpolation algorithm to calculate the second weighted average of the pixel values of the at least one fourth pixel according to the abscissa of the second pixel value point and the abscissas of the at least one fourth pixel, and assigns the second weighted average to the second pixel to obtain the pixel value of the second pixel.

[0158] Still taking the above Figure 3BFor example, if the second pixel is pixel 351, then the column transformation parameter corresponding to this second pixel is column transformation parameter 341. At this time, the value of this column transformation parameter 341 is 1.5, and this column transformation parameter 341 is located in the first column of the column mapping relationship table 34. Therefore, the value point of this second pixel is the point with coordinates (1, 1.5) in the first image 33. At this time, this image processing device needs to determine that the pixels adjacent to this second pixel value point and in the same column as this second pixel value point are the fourth pixels. From the first image 33, it can be seen that the point (1, 1.5) is between pixel 331 (abscissa is 1 and ordinate is 1) and pixel 334 (abscissa is 1 and ordinate is 2), then this image processing device can determine that pixel 331 and pixel 334 are the fourth pixels. If the pixel value of pixel 331 is 110 and the pixel value of pixel 334 is 120, using the interpolation algorithm, the weighted average value = 115 can be determined. In this example, 1.5 is the ordinate of the second pixel value point, this second pixel value point is the point (1, 1.5), and the fourth pixels are pixel 331 and pixel 334.

[0159] In practical applications, this image processing device can determine that the two pixels above the second pixel value point and the two pixels below the second pixel value point are the fourth pixel values, and specifically it is not limited here. In addition, the one-dimensional linear interpolation listed in the foregoing example is a common one. In practical applications, this image processing device can also adopt interpolation methods such as bilinear interpolation in the row direction and multiple non-linear interpolations, and specifically it is not limited here.

[0160] And so on, this image processing device calculates each pixel in the target image.

[0161] Step 207: Send the foregoing target image to the imaging device.

[0162] Wherein, this imaging device is used to perform imaging based on the foregoing target image so that the foregoing original image is projected on the imaging plane based on the foregoing target image.

[0163] In this embodiment, step 207 is an optional step.

[0164] In this embodiment, when the image processing device determines the target image (i.e., the pre-distortion image) based on the original image, the image remapping is decomposed into two single-direction mappings. As Figure 3C shown, this image processing device first performs a row transformation on each row of pixels in the original image (i.e., Figure 3C step ① in Figure 3CIn step ②), the target image is obtained. That is to say, when the image processing device calculates a row of pixels in the first image, it only needs to read a row of pixels in the original image, rather than reading across rows; similarly, when the image processing device calculates a column of pixels in the target image, it also only needs to read a column of pixels in the first image, rather than reading across columns. Therefore, the image processing device can read the original image in an orderly manner, thereby improving the efficiency of the image processing device in processing the original image.

[0165] Before adopting the image remapping method proposed in the embodiment of the present application, it is necessary to first determine the row mapping relationship table and the column mapping relationship table. As Figure 4 shown, the computing device can determine the row mapping relationship table and the column mapping relationship table in the following manner:

[0166] Step 401a: Obtain the coordinates of each calibration point in the calibration image.

[0167] Among them, the calibration image is an image used to measure the imaging distortion of an imaging device (for example, the aforementioned head-up display HUD, head-mounted display HMD, and projector, etc.). The calibration image contains multiple calibration points with determined coordinates. Generally, the multiple calibration points in the calibration image are arranged in a certain geometric shape in the calibration image.

[0168] Exemplarily, the calibration image can be an image with only calibration points as shown on the Figure 5A left side, or an image containing a standard grid as shown on the Figure 5A right side. Specifically, the present application does not make any limitations. If the calibration image is an image containing a standard grid, then the vertices of the grid (i.e., the intersection points between adjacent grids) are the calibration points.

[0169] Step 401b: Obtain the coordinates of each distorted point in the distorted image.

[0170] Among them, the distorted image is an image obtained through the imaging distortion of the imaging device. The distorted points in the distorted image correspond one-to-one to the calibration points in the calibration image. That is to say, a certain calibration point in the calibration image becomes a certain distorted point in the distorted image after distortion. Exemplarily, as Figure 5BAs shown, if the calibration image is 501, when the distortion caused by the imaging device is fan-shaped distortion, the distorted image corresponding to the calibration image 501 can be as shown in the distorted image 502. The calibration points in the calibration image 501 correspond one-to-one with the distorted points in the distorted image 502. For example, after distortion, the point a1 with coordinates (x, y) becomes the point a2 with coordinates (u, v). Calculating the pre-distortion model requires calculating the mapping from 501 to 503, but in fact only 501 and 502 are observable, and 503 is unknown. Since the mapping between 502 and 501 and the mapping between 501 and 503 are equivalent mappings, in actual calculation, only the mapping from 502 to 501 (i.e., the mapping from the distorted image to the calibration image) needs to be calculated to obtain the mapping from 501 to 503, which is the pre-distortion mapping required by the parameter table (for example, the row mapping relationship table or the column mapping relationship table).

[0171] Step 402: Determine the row mapping relationship table and the column mapping relationship table respectively according to the coordinates of the distorted point and the coordinates of the calibration point.

[0172] If the image processing device determines the target image in the order of rows first and then columns, the steps for the computing device to determine the row mapping relationship table and the column mapping relationship table are as follows:

[0173] S1: Fix the ordinate as the ordinate of the distorted point, and construct a row transformation model with the abscissa of the calibration point as the independent variable and the abscissa of the corresponding distorted point as the dependent variable.

[0174] S2: Calculate the abscissa of each pixel in the third image on the original image according to the row transformation model to obtain the row mapping relationship table, and the coordinates of each pixel in the third image are integers.

[0175] S3: Fix the abscissa as the abscissa of the calibration point, and construct a column transformation model with the ordinate of the calibration point as the independent variable and the ordinate of the corresponding distorted point as the dependent variable.

[0176] S4: Calculate the ordinate of each pixel in the fourth image on the third image according to the column transformation model to obtain the column mapping relationship table, and the coordinates of each pixel in the fourth image are integers, and the fourth image is the pre-distorted image of the calibration image.

[0177] Exemplarily, use (x i , y i ) to represent the coordinates of the calibration point on the original image, and use (u i , v i ) to represent the coordinates of the distorted point on the distorted image. The computing device first calculates the distortion model that only performs row transformation on the calibration point, that is, according to (x i , v i ) to (u i , vi ) The mapping of the model F under the row transformation is calculated α1 (x i , y i ). Optionally, the computing device resamples the area within (x i , v i ) to obtain a set of uniform calibration points (x i , y' i ) within the valid range of the row-transformed image, and calculates the coordinates (u' α1 , y' i ) corresponding to the calibration points (x i , y' i ) in the original image according to the aforementioned model F i )(x, y) = (F α1 (x i , y' i ), y' i ). Then, the computing device calculates the model F i (x, y) under the row transformation according to the mapping from (x i , y' i ) to (u' i , y' α2 ). Then, the computing device calculates the distortion model for column transformation only for the calibration points, that is, calculates the model F i (x, y) under the column transformation according to the mapping from (x i , y i ) to (x i , v β1 ). Then, the computing device uses the distortion model to calculate the coordinates of each point on the target image in the original image, and obtains the row mapping relation table and the column mapping relation table:

[0178] M X (i, j) = F α2 (i, j), M Y (i, j) = F β1 (i, j), 0 ≤ i < width, 0 ≤ i < height;

[0179] Among them, i represents the abscissa of the pixel in the target image; j represents the ordinate of the pixel in the target image; width represents the width of the target image, which can be understood as the maximum value of the abscissa of the pixels in the target image; height represents the height of the target image, which can be understood as the maximum value of the ordinate of the pixels in the target image.

[0180] In another alternative implementation, the computing device can also approximately obtain the aforementioned row mapping relation table and column mapping relation table by performing column transformation on the mapping relation table from the original image to the target image. Specifically, the following formula can be used for splitting:

[0181] M X (i, j) = M(i, jj, 0), M Y (i, j) = M(i, j, 1), 0 ≤ i < width, 0 ≤ i < height;

[0182] Among them, M(i, j, 0) represents the abscissa of the pixel in the original image corresponding to the j-th row and i-th column of the target image; M(i, j, 1) represents the ordinate of the pixel in the original image corresponding to the j-th row and i-th column of the target image; jj = argmin|M(i, k, 1) - j|, where jj represents the value of k that makes M(i, k, 1) closest to j, that is, the parameter in the i-th column and j-th row of the decomposed row transformation parameter table (only including row transformation parameters) is the row transformation parameter of the parameter pair (including both row transformation parameters and column transformation parameters) in the i-th column of the original parameter table whose column transformation parameter is closest to j, that is, fixing the ordinate and taking the abscissa.

[0183] It should be noted that in some methods for implementing image remapping, the mapping relation table will be compressed, that is, the complete point-to-point mapping relation is not stored. In this case, the decomposed and compressed row mapping relation table and column mapping relation table can still be obtained through the solution of the present application.

[0184] Compared with the traditional technology that directly calculates the distortion mapping from (x i , y i ) to (u i , v i ), the present application first calculates the mapping from (x i , v i ) to (u i , v i ), and then calculates the mapping from (x i , y i ) to (x i , v i ). Therefore, the present application can obtain the row mapping relation table and the column mapping relation table.

[0185] If the image processing device determines the target image in the column-first and then row-first manner, the steps for determining the row mapping relation table and the column mapping relation table are as follows:

[0186] T1. Fix the ordinate as the ordinate of the distortion point, and construct a column transformation model with the ordinate of the calibration point as the independent variable and the ordinate of the corresponding distortion point as the dependent variable.

[0187] T2. Calculate the ordinate of each pixel in the fifth image on the original image according to this column transformation model to obtain this column mapping relation table, and the coordinates of each pixel in the fifth image are integers.

[0188] T3. Fix the abscissa of the horizontal coordinate as the abscissa of the calibration point, and construct a row transformation model with the abscissa of the calibration point as the independent variable and the abscissa of the corresponding distorted point as the dependent variable.

[0189] T4. Calculate the abscissa of each pixel in the sixth image on the fifth image according to this row transformation model to obtain a row mapping relationship table. The coordinates of each pixel in the sixth image are all integers, and the sixth image is the pre-distorted image of the calibration image.

[0190] Exemplarily, use (x i , y i ) to represent the coordinates of the calibration point on the original image, and use (u i , v i ) to represent the coordinates of the distorted point on the distorted image. The computing device first calculates the distortion model that only performs column transformation on the calibration point, that is, calculates the model F i , y i ) to (u i , v i ) to calculate the model F β2 (x, y) under column transformation. Optionally, the computing device resamples the area within (u i , y i ) to obtain a set of uniform calibration points (x' i , y i ) within the effective range of the column-transformed image, and calculates the coordinates (x' β2 corresponding to the calibration point (x' i , y i ) in the original image according to the aforementioned model F i , v' i ) = (x' i , F β2 (x' i , y' i ). Then, the computing device calculates the model F i , y i ) to (x' i , v' i ) to calculate the model F β3 (x, y) under column transformation. Then, the computing device calculates the distortion model that only performs row transformation on the calibration point, that is, calculates the model F i , y i ) to (u i , y i ) to calculate the model F α3 (x, y) under row transformation. Then, the computing device uses the distortion model to calculate the coordinates of each point on the target image in the original image to obtain a row mapping relationship table and a column mapping relationship table:

[0191] M Y(i, j) = F β3 (i, j), M X (i, j) = F α3 (i, j), 0 ≤ i < width, 0 ≤ i < height;

[0192] Wherein, i represents the abscissa of the pixel in the target image; j represents the ordinate of the pixel in the target image; width represents the width of the target image, which can be understood as the maximum value of the abscissa of the pixels in the target image; height represents the height of the target image, which can be understood as the maximum value of the ordinate of the pixels in the target image.

[0193] In another alternative embodiment, the computing device can also transform the mapping relationship table from the original image to the target image so as to approximately obtain the foregoing row mapping relationship table and column mapping relationship table. Specifically, the following formula can be used for splitting: M X (i, j) = M(ii, j, 0), M Y (i, j) = M(i, j, 1), 0 ≤ i < width, 0 ≤ i < height;

[0194] Wherein, ii = argmin|M(k, j, 1) - i|, where ii represents the value of k that makes M(k, j, 1) closest to i, that is, the parameter in the i-th column and j-th row of the decomposed column transformation parameter table (only including column transformation parameters) is the column coordinate parameter of the parameter pair (including both row transformation parameters and column transformation parameters) in the j-th row of the original parameter table whose row transformation parameter is closest to i, that is, fixing the abscissa and taking the ordinate.

[0195] Step 403: Store the row mapping relationship table and the column mapping relationship table in the memory.

[0196] In this embodiment, after the computing device calculates the foregoing row mapping relationship table and column mapping relationship table, the computing device stores the foregoing row mapping relationship table and column mapping relationship table in the memory. The memory can be the memory in the projection imaging system. The memory is an external memory relative to the foregoing image processing device, and the internal memory in the foregoing image processing device can obtain the foregoing row mapping relationship table and column mapping relationship table from the internal memory storing the row mapping relationship table and column mapping relationship table.

[0197] Next, the image remapping method proposed in this application will be introduced based on the foregoing scenario. In this method, as Figure 6A shown, if the image processing device first performs column transformation on the original image and then performs row transformation on the image after column transformation processing, the original image will perform the following steps:

[0198] Step 601: Obtain each column of pixels of the original image.

[0199] Among them, the definition of the original image can specifically refer to the introduction in the foregoing step 201.

[0200] It should be understood that when the image processing device obtains each column of pixels of the original image, it can be specifically understood that the image processing device reads the pixel value of each pixel in each column of pixels and the coordinates of the pixel in the original image. Among them, the coordinates of the pixel in the original image include the abscissa and the ordinate.

[0201] Step 602: Obtain the column transformation parameters corresponding to each column of pixels.

[0202] In this embodiment, there is no limitation on the time sequence between step 601 and step 602. That is to say, the image processing device can execute step 601 first and then step 602, or execute step 602 first and then step 601, or execute the foregoing steps 601 and 602 simultaneously. Specifically, it is not limited here. In the subsequent embodiments, only an example in which the image processing device can first obtain a column of pixels from the original image and then obtain the column transformation parameters corresponding to the column of pixels will be introduced.

[0203] Among them, the column transformation parameters are used to indicate the corresponding relationship between the ordinate of the pixels in the original image and the ordinate of the pixels in the second image (i.e., the column transformation result). The second image is an image with pre-distortion in the column direction and no pre-distortion in the row direction, and the ordinate of the second image is an integer.

[0204] Optionally, the column transformation parameters include the ordinate of the fifth pixel value point corresponding to the fifth pixel in the original image. The fifth pixel value point is used to determine the pixel value of the fifth pixel in the second image, and the abscissa of the fifth pixel value point in the original image is the same as the abscissa of the fifth pixel in the second image.

[0205] Optionally, the foregoing column transformation parameters are stored in a column mapping relationship table, and the position of the column transformation parameters in the column mapping relationship table is used to represent the coordinates of the fifth pixel determined based on the column transformation parameters.

[0206] In this embodiment, the process by which the image processing device obtains the column transformation parameters corresponding to each column of pixels can specifically be that the image processing device reads the column transformation parameters corresponding to each column of pixels in sequence in the column direction from the memory. Specifically, depending on the bit width of the memory, the number of column transformation parameters read each time is different, and the number of times the memory needs to be activated to read a column of column transformation parameters is different. In this embodiment, the image processing device can read the column mapping relationship table in the same way as reading the original image. Specifically, please refer to the example in step 201.

[0207] Step 603: Determine the pixel values of each pixel in the second image according to each column of pixels in the original image and the column transformation parameters corresponding to each column of pixels.

[0208] Wherein, the second image is an image with pre-distortion in the column direction and no pre-distortion in the row direction. That is to say, the second image is the result obtained by subjecting the original image to column transformation and to be obtained.

[0209] In this embodiment, the process by which the image processing device calculates the second image can be understood as the process by which the image processing device determines the pixel values of each pixel in the second image. Since the position (i.e., coordinates) of each pixel in the second image is known, that is, the abscissa and ordinate of each pixel in the second image are consecutive integers. Therefore, according to the abscissa and ordinate of the fifth pixel, the column transformation parameters of the second image can be queried, that is, query the column transformation parameters with the same abscissa and ordinate according to the abscissa and ordinate of the fifth pixel.

[0210] Specifically, the image processing device first determines the fifth pixel value point corresponding to the fifth pixel according to the column transformation parameters; then, determines at least one seventh pixel according to the ordinate of the fifth pixel value point, and the seventh pixel and the fifth pixel value point are in the same column; then, determines the pixel value of the fifth pixel according to the pixel values of the at least one seventh pixel.

[0211] In an alternative embodiment, if the ordinate of the foregoing fifth pixel value point is an integer, the image processing device directly determines the fifth pixel value point as the seventh pixel, and then assigns the pixel value of the seventh pixel to the fifth pixel to obtain the pixel value of the fifth pixel.

[0212] In an alternative embodiment, if the ordinate of the foregoing fifth pixel value point is a non-integer, the image processing device determines at least one pixel adjacent to the fifth pixel value point as the seventh pixel, and the pixel adjacent to the fifth pixel value point and the fifth pixel value point are in the same column; then, uses an interpolation algorithm to calculate the first weighted average of the pixel values of the at least one seventh pixel according to the ordinate of the fifth pixel value point and the ordinate of the at least one seventh pixel; assigns the first weighted average to the fifth pixel to obtain the pixel value of the fifth pixel.

[0213] Specifically, it is similar to the content of the foregoing step 203. Please refer to the introduction in the foregoing step 203.

[0214] Step 604: Obtain each row of pixels of the second image.

[0215] Among them, the second image is an image with pre-distortion in the column direction and no pre-distortion in the row direction. That is to say, the second image is the result obtained by subjecting the original image to column transformation and is the one to be obtained.

[0216] Specifically, the image processing device sequentially reads each row of pixels of the aforementioned second image from the memory. If the pixel values of each column of pixels of the image processing device are written into the memory column by column, then the image processing device will read each row of pixels of the second image in the memory row by row. If the image processing device transposes the pixel values of each column of pixels and writes each column of pixels into the memory row by row, then the image processing device will read each row of pixels of the second image in the memory column by column.

[0217] Step 605: Obtain the row transformation parameters corresponding to each row of pixels.

[0218] In this embodiment, there is no limitation on the time sequence between step 604 and step 605. That is to say, the image processing device can execute step 604 first and then step 605, or execute step 605 first and then step 604, or execute the aforementioned steps 604 and 605 simultaneously. Specifically, no limitation is made here. In subsequent embodiments, only the example where the image processing device can first obtain a row of pixels from the second image and then obtain the row transformation parameters corresponding to that row of pixels will be introduced.

[0219] Among them, the row transformation parameter is used to indicate the corresponding relationship between the ordinate of the pixel in the second image and the ordinate of the pixel in the target image.

[0220] Optionally, the row transformation parameter includes the abscissa of the sixth pixel value point corresponding to the sixth pixel in the second image. The sixth pixel value point is used to determine the pixel value of the sixth pixel in the target image, and the ordinate of the sixth pixel value point in the second image is the same as the ordinate of the sixth pixel in the target image.

[0221] Optionally, the row transformation parameter is stored in a row mapping relationship table, and the position of the row transformation parameter in the row mapping relationship table is used to represent the coordinates of the sixth pixel determined based on the row transformation parameter.

[0222] Specifically, please refer to the introduction corresponding to the aforementioned step 205.

[0223] Step 606: Determine the pixel value of each target image according to each row of pixels in the second image and the row transformation parameters corresponding to each row of pixels respectively.

[0224] Among them, the target image is the pre-distorted image used to offset the imaging distortion of the imaging device.

[0225] Specifically, the image processing device determines the sixth pixel value point corresponding to the sixth pixel according to the row transformation parameter; determines at least one eighth pixel according to the horizontal coordinate of the sixth pixel value point, and the eighth pixel and the sixth pixel value point are located in the same row; determines the pixel value of the sixth pixel according to the pixel value of the at least one eighth pixel.

[0226] In an optional embodiment, if the horizontal coordinate of the aforementioned sixth pixel value point is an integer, the image processing device determines that the sixth pixel value point is the eighth pixel, and then assigns the pixel value of the eighth pixel to the sixth pixel to obtain the pixel value of the sixth pixel.

[0227] In an optional embodiment, if the horizontal coordinate of the aforementioned sixth pixel value point is a non-integer, the image processing device determines at least one pixel adjacent to the sixth pixel value point as the eighth pixel, and the pixel adjacent to the sixth pixel value point is located at the same horizontal position as the sixth pixel value point; then, an interpolation algorithm is used to calculate a second weighted average of the pixel values ​​of the at least one eighth pixel based on the horizontal coordinate of the sixth pixel value point and the horizontal coordinate of the at least one eighth pixel; the second weighted average is assigned to the sixth pixel to obtain the pixel value of the sixth pixel.

[0228] For details, please refer to the corresponding introduction of the aforementioned step 206.

[0229] Step 607: Send the target image to an imaging device.

[0230] The imaging device is used to perform imaging based on the aforementioned target image, so that the aforementioned original image is projected on an imaging plane based on the aforementioned target image.

[0231] In this embodiment, step 607 is an optional step.

[0232] In this embodiment, when the image processing device determines the target image (i.e., the pre-distorted image) based on the original image, the image remapping is decomposed into two unidirectional mappings. Figure 6B As shown, the image processing device first performs a column transformation on each column of pixels in the original image (i.e. Figure 6B In step ①), based on the column transformation result (i.e., the second image), each row of pixels is transformed (i.e., Figure 6B In step ②), the target image is obtained. That is, when the image processing device calculates a column of pixels in the second image, it only needs to read a column of pixels in the original image, without reading across columns; similarly, when the image processing device calculates a row of pixels in the target image, it only needs to read a row of pixels in the second image, without reading across rows. Therefore, the image processing device can read in order when processing the original image, thereby improving the efficiency of the image processing device in processing the original image.

[0233] As Figure 7 shown, it is a schematic structural diagram of an image processing device 70 provided by an embodiment of the present application. The image processing device 70 may be located in a head-up display (HUD, also known as a head-up display), a head-mounted display (HMD), or a projector. The image processing device may be an image processing chip located in the aforementioned head-up display (HUD), head-mounted display (HMD), or projector. The Figure 2 and Figure 6A corresponding steps in the embodiment may be executed by the image processing device 70.

[0234] The image processing device 70 includes at least one processor 701 and at least one memory 702. The aforementioned processor 701 and the aforementioned memory 702 are interconnected by a line. It should be understood that Figure 7 only one processor 701 and one memory 702 are shown.

[0235] Among them, the processor 701 may be a field programmable gate array (FPGA), a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), or other functional units or functional modules with image processing functions.

[0236] In addition, the aforementioned processor 701 may be a single-core processor; it may also be a multi-core processor. For example, the processor 701 may be composed of multiple FPGAs or multiple DSPs. At this time, the processor 701 can process multiple images simultaneously. For example, if the processor 701 includes two FPGAs, namely FPGA1 and FPGA2, after FPGA1 performs a row transformation on image 1, it sends the row transformation result to FPGA2, and FPGA2 performs a column transformation on the row transformation result of image 1. While FPGA2 is performing a column transformation on the row transformation result of the aforementioned image 1, FPGA1 can continue to perform a row transformation on the next image (for example, image 2), and then send the row transformation result of image 2 to FPGA2, and FPGA2 performs a column transformation on the row transformation result of image 2.

[0237] In addition, the processor 701 may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). The processor 701 may be a single semiconductor chip or may be integrated with other circuits into a semiconductor chip. For example, it may form a system-on-a-chip (SoC) with other circuits (such as codec circuits, hardware acceleration circuits, or various bus and interface circuits), or may be integrated as a built-in processor of an application specific integrated circuit (ASIC) within the ASIC. The ASIC integrated with the processor may be separately packaged or may be packaged together with other circuits.

[0238] In addition, the aforementioned memory 702 may be a read-only memory (ROM), or may be other types of static storage devices that can store static information and instructions, or may be a random access memory (RAM), or may be other types of dynamic storage devices that can store information and instructions, or may also be an electrically erasable programmable read-only memory (EEPROM), and specific details are not limited here. Exemplarily, the aforementioned memory 702 may be a double data rate synchronous dynamic random access memory (DDR SDRAM) (abbreviated as DDR). The memory 702 may exist independently but is connected to the aforementioned processor 701. Optionally, the memory 702 may also be integrated with the aforementioned processor 701. For example, it may be integrated within one or more chips.

[0239] In addition, the memory 702 may be understood as the internal memory of the image processing device 70. The memory 702 is used to store the row mapping relationship table, column mapping relationship table, and the original image read from the external memory. The memory 702 is also used to store the intermediate images generated during the calculation of the target image. Exemplarily, the row transformation result generated based on the original image (for example, the aforementioned first image), or the column transformation result generated based on the original image (for example, the aforementioned second image). The memory 702 is also used to store the program code for implementing the technical solution of the embodiments of the present application. The aforementioned program code may be controlled and executed by the processor 701, and various types of computer program codes being executed may also be regarded as the driver programs of the processor 701.

[0240] Optionally, the image processing device 70 further includes a communication interface 703 for communicating with an external memory or other devices. The image processing device 70 can receive instructions or data from other devices through the communication interface 703. Exemplarily, the image processing device 70 can read the row mapping relation table, column mapping relation table, and the original image from the external memory through the foregoing communication interface 703.

[0241] Exemplarily, the foregoing processor 701 can obtain each row of pixels of the original image and the row transformation parameters corresponding to each row of pixels from an external memory or an internal memory. Then, the processor 701 determines the pixel value of each pixel in the first image respectively according to each row of pixels in the original image and the row transformation parameters corresponding to each row of pixels, thereby obtaining the first image. Then, the processor 701 stores the foregoing first image in the internal memory. Then, the processor obtains each column of pixels of the first image from the foregoing internal memory, and obtains the column transformation parameters corresponding to each column of pixels from an external memory or an internal memory. Then, the processor 701 determines the pixel value of each pixel in the target image respectively according to each column of pixels in the first image and the column transformation parameters corresponding to each column of pixels, thereby obtaining the target image. Optionally, the processor 701 stores the foregoing target image in the external memory.

[0242] Optionally, the processor 701 determines the first pixel value point corresponding to the first pixel according to the row transformation parameter; then, determines at least one third pixel according to the abscissa of the first pixel value point, where the third pixel and the first pixel value point are in the same row; then, determines the pixel value of the first pixel according to the pixel values of the at least one third pixel.

[0243] Optionally, the processor 701 determines the second pixel value point corresponding to the second pixel according to the column transformation parameter; then, determines at least one fourth pixel according to the ordinate of the second pixel value point, where the fourth pixel and the second pixel value point are in the same column; then, determines the pixel value of the second pixel according to the pixel values of the at least one fourth pixel.

[0244] Exemplarily, the foregoing processor 701 may obtain each column of pixels of the original image and the column transformation parameters corresponding to each column of pixels from an external memory or an internal memory. Then, the processor 701 determines the pixel values of each pixel in the second image respectively according to each column of pixels in the original image and the column transformation parameters corresponding to each column of pixels, thereby obtaining the second image. Then, the processor 701 stores the foregoing second image in the internal memory. Then, the processor obtains each row of pixels of the second image from the foregoing internal memory, and obtains the row transformation parameters corresponding to each row of pixels from an external memory or an internal memory. Then, the processor 701 determines the pixel values of each pixel in the target image respectively according to each row of pixels in the second image and the row transformation parameters corresponding to each row of pixels, thereby obtaining the target image. Optionally, the processor 701 stores the foregoing target image in the external memory.

[0245] Optionally, the processor 701 determines the fifth pixel value point corresponding to the fifth pixel according to the column transformation parameter; determines at least one seventh pixel according to the ordinate of the fifth pixel value point, where the seventh pixel and the fifth pixel value point are in the same column; and determines the pixel value of the fifth pixel according to the pixel values of the at least one seventh pixel.

[0246] Optionally, the processor 701 determines the sixth pixel value point corresponding to the sixth pixel according to the row transformation parameter; determines at least one eighth pixel according to the abscissa of the sixth pixel value point, where the eighth pixel and the sixth pixel value point are in the same row; and determines the pixel value of the sixth pixel according to the pixel values of the at least one eighth pixel.

[0247] In this embodiment, when the processor 701 determines the target image (i.e., the pre-distorted image) based on the original image, the image remapping is decomposed into two single-direction mappings. Specifically, the processor 701 first performs a row transformation on each row of pixels in the original image, and then performs a column transformation on each column of pixels in the result of the row transformation (i.e., the first image) to obtain the target image; or, first performs a column transformation on each column of pixels in the original image, and then performs a row transformation on each row of pixels based on the result of the column transformation to obtain the target image. That is to say, when the processor 701 calculates a row of pixels in the first image, it only needs to read a row of pixels in the original image and does not need to read across rows; similarly, when the processor 701 calculates a column of pixels in the target image, it only needs to read a column of pixels in the first image and does not need to read across columns. Therefore, the image processing device can read the original image in an orderly manner when processing the original image, thereby improving the efficiency of the image processing device in processing the original image.

[0248] In addition, when the processor 701 reads the row mapping relationship table, the column mapping relationship table, the original image, the first image, and the second image from the memory 702, the processor 701 does not need to read out of order across rows, reducing the number of times the storage units in the memory 702 are activated and improving the service life of the memory 702.

[0249] For the rest, reference can be made to Figure 2 and Figure 6A the corresponding embodiments of the image remapping method, which will not be elaborated here.

[0250] An embodiment of the present application also provides a computer-readable storage medium, which stores a row mapping relationship table, a column mapping relationship table, and a program for generating a target image. When the foregoing computer program runs on a computer, the computer can be caused to execute as described above Figure 2 or Figure 6A the method introduced in the illustrated embodiments.

[0251] In a possible implementation manner, the foregoing Figure 7 illustrated image processing device 70 is a chip, which is also referred to as a digital processing chip. The chip includes a processing unit and a communication unit. Among them, the processing unit obtains program instructions through the communication unit, and the program instructions are executed by the processing unit so that the processing unit executes the foregoing Figure 2 or Figure 6A method steps introduced in the corresponding embodiments. Specifically, the processing unit is a circuit integrating the foregoing processor 701 or used to implement the functions of the foregoing processor 701, and the communication unit is a circuit or interface integrating the foregoing communication interface 703 or used to implement the functions of the foregoing communication interface 703.

[0252] Optionally, when a storage unit is integrated in the chip, the storage unit may be a storage device such as a memory. At this time, the processing unit in the chip can call program code from the storage unit to implement the foregoing Figure 2 or Figure 6A method steps introduced in the corresponding embodiments. When the chip does not integrate a storage unit, the chip can be connected to an external storage device such as a memory through the foregoing communication unit so as to obtain program code from the foregoing external storage device to implement the foregoing Figure 2 or Figure 6A method steps introduced in the corresponding embodiments.

[0253] As Figure 8 shown, a schematic structural diagram of an image processing device 80 provided by an embodiment of the present application is shown. The foregoing Figure 2 and Figure 6A corresponding method embodiments can both be based on Figure 8 the structure of the illustrated image processing device 80.

[0254] The image processing device 80 includes a plurality of functional modules. Each of the foregoing functional modules may be integrated in a processing unit, may exist physically separately for each module, or two or more modules may be integrated in one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0255] Specifically, the image processing device 80 may be located in a head-up display (HUD, also known as a head-up display) or a head-mounted display (HMD). The image processing device may be an image processing chip located in the foregoing head-up display (HUD) or head-mounted display (HMD). The foregoing Figure 2 and Figure 6A The steps in the corresponding embodiments may be executed by the image processing device 80. Specifically, the image processing device 80 includes an acquisition module 801 and a calculation module 802.

[0256] Exemplarily, the acquisition module 801 is configured to acquire each row of pixels of the original image, and acquire the row transformation parameter corresponding to each row of pixels. Wherein, the row transformation parameter is used to indicate the corresponding relationship between the abscissa of the pixels in the original image and the abscissa of the pixels in the first image, and the first image is an image with pre-distortion in the row direction and no pre-distortion in the column direction.

[0257] Exemplarily, the calculation module 802 is configured to determine the pixel value of each pixel in the first image according to each row of pixels in the original image and the row transformation parameter corresponding to each row of pixels, respectively.

[0258] Exemplarily, the acquisition module 801 is further configured to acquire each column of pixels of the first image, and acquire the column transformation parameter corresponding to each column of pixels. The column transformation parameter is used to indicate the corresponding relationship between the ordinate of the pixels in the first image and the ordinate of the pixels in the target image, and the target image is an image with pre-distortion in both the row direction and the column direction.

[0259] Exemplarily, the calculation module 802 is further configured to determine the pixel value of each pixel in the target image according to each column of pixels in the first image and the column transformation parameter corresponding to each column of pixels, respectively. The target image is used to offset the imaging distortion of the imaging device to display the original image.

[0260] In an alternative embodiment, the calculation module 802 is specifically configured to: determine the first pixel value point corresponding to the first pixel according to the row transformation parameter; determine at least one third pixel according to the abscissa of the first pixel value point, and the third pixel and the first pixel value point are in the same row; determine the pixel value of the first pixel according to the pixel values of the at least one third pixel.

[0261] In an alternative embodiment, a second pixel value point corresponding to the second pixel is determined according to the column transformation parameter; at least one fourth pixel is determined according to the ordinate of the second pixel value point, and the fourth pixel and the second pixel value point are located in the same column; the pixel value of the second pixel is determined according to the pixel values of the at least one fourth pixel.

[0262] In an alternative embodiment, the image processing apparatus 80 further includes a storage module 803;

[0263] The storage module is configured to store the first pixels of each row of the first image in a memory.

[0264] In an alternative embodiment, the obtaining module 801 is specifically configured to:

[0265] Read the row transformation parameters corresponding to each row of pixels from the memory in the row direction sequence;

[0266] Read the column transformation parameters corresponding to each column of pixels from the memory in the column direction sequence.

[0267] In an alternative embodiment, the image processing apparatus 80 further includes a sending module 804;

[0268] The sending module 804 is configured to send the target image to the imaging device, and the imaging device is configured to perform imaging based on the foregoing target image, so that the foregoing original image is projected on the imaging plane based on the foregoing target image. Wherein, the imaging device may be a projection imaging system.

[0269] The rest can refer to Figure 2 or Figure 6A the method of the image processing apparatus in the corresponding embodiment, which will not be elaborated here.

[0270] In the implementation process, each step of the foregoing method may be completed by an integrated logic circuit in hardware in the processor or an instruction in software form. The steps of the method disclosed in combination with the embodiments of the present application may be directly embodied as being executed and completed by a hardware processor, or executed and completed by a combination of hardware and software modules in the processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read only memory, a programmable read only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the foregoing method. To avoid repetition, it will not be described in detail here. It should also be understood that the first, second, third, fourth, and various digital numbers involved herein are only for the convenience of description for distinction, and do not limit the scope of the embodiments of the present application.

[0271] It should be understood that the term "and / or" in this text is merely a description of the association relationship between associated objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. Additionally, the character " / " in this text generally represents an "or" relationship between the associated objects before and after.

[0272] It should be understood that in various embodiments of the present application, the magnitudes of the serial numbers of the above processes do not imply the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0273] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0274] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image remapping method, characterized in that, it includes: Obtain each row of pixels of the original image; Obtain the row transformation parameters corresponding to each row of pixels, where the row transformation parameters are used to indicate the correspondence between the abscissa of the pixels in the original image and the abscissa of the pixels in the first image, and the first image is an image with pre-distortion in the row direction and no pre-distortion in the column direction; Determine the pixel values of each pixel in the first image respectively according to each row of pixels in the original image and the row transformation parameters corresponding to each row of pixels; Obtain each column of pixels of the first image; Obtain the column transformation parameters corresponding to each column of pixels, where the column transformation parameters are used to indicate the correspondence between the ordinate of the pixels in the first image and the ordinate of the pixels in the target image, and the target image is an image with pre-distortion in both the row direction and the column direction; Determine the pixel values of each pixel in the target image respectively according to each column of pixels in the first image and the column transformation parameters corresponding to each column of pixels, and the target image is used to offset the imaging distortion of the imaging device to display the original image.

2. The method according to claim 1, characterized in that, The row transformation parameter includes the abscissa of the first pixel value point corresponding to the first pixel in the original image, the first pixel is any pixel in the first image, the first pixel value point is used to determine the pixel value of the first pixel, and the ordinate of the first pixel value point in the original image is the same as the ordinate of the first pixel in the first image; The column transformation parameter includes the ordinate of the second pixel value point corresponding to the second pixel in the first image, the second pixel is any pixel in the target image, the second pixel value point is used to determine the pixel value of the second pixel, and the abscissa of the second pixel value point in the first image is the same as the abscissa of the second pixel in the target image.

3. The method according to claim 2, characterized in that, The row transformation parameters are stored in a row mapping relationship table, and the position of the row transformation parameters in the row mapping relationship table is used to indicate the coordinates of the first pixel; The column transformation parameters are stored in a column mapping relationship table, and the position of the column transformation parameters in the column mapping relationship table is used to indicate the coordinates of the second pixel.

4. The method according to claim 2 or 3, characterized in that, The step of determining the pixel values of each pixel in the first image respectively according to each row of pixels in the original image and the row transformation parameters corresponding to each row of pixels includes: Determine the first pixel value point corresponding to the first pixel according to the row transformation parameter; Determine at least one third pixel according to the abscissa of the first pixel value point, and the third pixel is in the same row as the first pixel value point; Determine the pixel value of the first pixel according to the pixel values of the at least one third pixel; Determining the pixel value of each pixel in the target image respectively according to each column of pixels in the first image and the column transformation parameter corresponding to each column of pixels includes: Determining the second pixel value point corresponding to the second pixel according to the column transformation parameter; Determining at least one fourth pixel according to the ordinate of the second pixel value point, the fourth pixel and the second pixel value point being in the same column; Determining the pixel value of the second pixel according to the pixel values of the at least one fourth pixel.

5. The method according to claim 2 or 3, wherein, after determining the pixel value of each pixel in the first image respectively according to each row of pixels in the original image and the row transformation parameter corresponding to each row of pixels, the method further includes: Storing the first pixel of each row of the first image.

6. The method according to any one of claims 1 to 3, wherein, the method further includes: Sending the target image to the imaging device, the imaging device being configured to perform imaging based on the target image so that the original image is projected onto the imaging plane based on the target image.

7. An image processing device, wherein, comprising: An acquisition module, configured to acquire each row of pixels of the original image; The acquisition module is further configured to acquire the row transformation parameter corresponding to each row of pixels, the row transformation parameter being used to indicate the correspondence between the abscissa of the pixels in the original image and the abscissa of the pixels in the first image, the first image being an image with pre-distortion in the row direction and no pre-distortion in the column direction; A calculation module, configured to determine the pixel value of each pixel in the first image respectively according to each row of pixels in the original image and the row transformation parameter corresponding to each row of pixels; The acquisition module is further configured to acquire each column of pixels of the first image; The acquisition module is further configured to acquire the column transformation parameter corresponding to each column of pixels, the column transformation parameter being used to indicate the correspondence between the ordinate of the pixels in the first image and the ordinate of the pixels in the target image, the target image being an image with pre-distortion in both the row direction and the column direction; The calculation module is further configured to determine the pixel value of each pixel in the target image respectively according to each column of pixels in the first image and the column transformation parameter corresponding to each column of pixels, the target image being used to cancel the imaging distortion of the imaging device to display the original image.

8. The image processing device according to claim 7, wherein, the row transformation parameter includes the abscissa of the first pixel value point corresponding to the first pixel in the original image, the first pixel being any pixel in the first image, the first pixel value point being used to determine the pixel value of the first pixel, and the ordinate of the first pixel value point in the original image being the same as the ordinate of the first pixel in the first image; The column transformation parameter includes the ordinate of the second pixel value point corresponding to the second pixel in the first image, where the second pixel is any pixel in the target image, the second pixel value point is used to determine the pixel value of the second pixel, and the abscissa of the second pixel value point in the first image is the same as the abscissa of the second pixel in the target image.

9. The image processing apparatus according to claim 8, wherein, the row transformation parameter is stored in a row mapping relationship table, and the position of the row transformation parameter in the row mapping relationship table is used to indicate the coordinates of the first pixel; the column transformation parameter is stored in a column mapping relationship table, and the position of the column transformation parameter in the column mapping relationship table is used to indicate the coordinates of the second pixel.

10. The image processing apparatus according to claim 8 or 9, wherein, the calculation module is specifically configured to: determine the first pixel value point corresponding to the first pixel according to the row transformation parameter; determine at least one third pixel according to the abscissa of the first pixel value point, where the third pixel and the first pixel value point are in the same row; determine the pixel value of the first pixel according to the pixel values of the at least one third pixel; determine the second pixel value point corresponding to the second pixel according to the column transformation parameter; determine at least one fourth pixel according to the ordinate of the second pixel value point, where the fourth pixel and the second pixel value point are in the same column; determine the pixel value of the second pixel according to the pixel values of the at least one fourth pixel.

11. The image processing apparatus according to claim 8 or 9, wherein, the image processing apparatus further includes a storage module; the storage module is used to store the first pixel of each row of the first image into a memory.

12. The image processing apparatus according to claim 9, wherein, the row mapping relationship table and the column mapping relationship table are stored in a memory, the row mapping relationship table includes the row transformation parameters corresponding to each row of pixels in the original image, and the column mapping relationship table includes the column transformation parameters corresponding to each column of pixels in the original image; the obtaining module is specifically configured to: read the row transformation parameters corresponding to each row of pixels from the memory in a row direction sequence; read the column transformation parameters corresponding to each column of pixels from the memory in a column direction sequence.

13. The image processing apparatus according to any one of claims 7 to 9, wherein, the image processing apparatus further includes a sending module; the sending module is used to send the target image to the imaging device, and the imaging device is used to perform imaging based on the target image so that the original image is projected on an imaging plane based on the target image.

14. An image processing apparatus, wherein, comprising: a processor and an internal memory, the processor and the internal memory are coupled; the processor is used to read a row mapping parameter table and a column mapping parameter table from an external memory into the internal memory; The internal memory also stores a program, and when the program instructions stored in the internal memory are executed by the processor, the image processing apparatus implements the method according to any one of claims 1 to 6.

15. An imaging device, characterized in that it comprises: an image processing apparatus and a display device; The image processing apparatus is configured to generate a target image according to the original image according to the method according to any one of claims 1 to 6, and send the target image to the display device so that the imaging device projects the original image.

16. A computer-readable storage medium, comprising a computer program, where the computer program is executed by a processor to implement the method according to any one of claims 1 to 6.

Images