A hardware acceleration implementation method for immersive projection mapping based on CACHE architecture

Abstract

The present invention discloses a method for hardware acceleration of immersive projection mapping based on a CACHE architecture, and belongs to the field of image processing. The method includes six parts: target graphic scanning, source image coordinate calculation, on-chip buffering, coordinate conversion, interpolation processing, and image processing. A group-associated CACHE structure is used to optimize the problem of low efficiency of accessing SDR / DDR SDRAM due to discontinuous reading of source image pixels during graphic image projection transformation. Through a flexible CACHE structure and replacement strategy, it is ensured that better efficiency is achieved at different deformation angles. At the same time, by caching the source image coordinates to the on-chip cache, the main processing modules can be executed in parallel, further improving the processing speed. This method can achieve real-time processing of three-dimensional transformation of images with a resolution above full HD, avoiding the waste of storage bandwidth caused by writing to the memory area again.

Description

Technical Field

[0001] The present invention relates to image processing, and in particular to a method for realizing hardware acceleration of immersive projection mapping based on a CACHE architecture. Background Art

[0002] One characteristic of 3D conversion projection is that the coordinates of points in the source image corresponding to adjacent points in the target area vary greatly, especially when the projection angle is large. This causes the position of the data required for processing to vary greatly in the memory. For commonly used bus structures, the burst transmission characteristics cannot be fully utilized. For the more commonly used large-capacity high-speed memory SDRAM, discontinuous addresses will also cause low access efficiency.

[0003] Usually, image accelerators process images in blocks based on macroblocks or triangles, or use line cache to improve SDRAM access efficiency. However, current designs cannot guarantee the real-time deformation of images with resolutions above full HD when the image deformation angle is large, and all need to be written back to SDRAM, resulting in further waste of system bandwidth. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide an immersive projection mapping hardware acceleration implementation method based on the CACHE architecture to solve the problem of insufficient system bandwidth during image processing.

[0005] Technical solution: A method for implementing immersive projection mapping hardware acceleration based on CACHE architecture, comprising the following steps: S1. Target graphic scanning: first describe the outline of the target graphic, then scan the target graphic area line by line and point by point until the scanning is completed; S2. Source image coordinate calculation: The target area and the source image are projected; therefore, the relationship between the two is expressed by a constant coefficient matrix: ,and ; After solving the above matrix, the coordinates (u, v) on the corresponding source image can be calculated from any point (x, y) on the target area. The calculation of the source image coordinates is performed simultaneously with the scanning in step S1. S3, on-chip buffer: store the coordinates calculated in step S2 into the on-chip memory, so that steps S1 and S2 can continue to calculate the source image coordinates of the next target pixel; S4, coordinate conversion: after the temporarily stored source image coordinates are obtained in step S2, the coordinates are converted into the physical address of the source image, and then the data of the source image is obtained from the CACHE buffer; S5, interpolation processing: the data taken out from the CACHE buffer in step S4 is interpolated according to the decimal part of the coordinates calculated in step S2; S6, Image Processing: Automatically calculate the display coordinates of the output pixels according to the output image resolution; output a configurable background color when the current display coordinates are outside the target graphic; output the interpolated pixels calculated in the fifth part when the current display coordinates are inside the target graphic.

[0006] Preferably, in step S1, when describing the outline of the target figure, since the target figure is a quadrilateral composed of four vertices, when the coordinates of the four vertices are known, the coordinates of the points on any side of the quadrilateral can be calculated by the two-point straight line equation, thereby obtaining the coordinate outline of the quadrilateral and dividing it into a left outline and a right outline.

[0007] Preferably, in step S1, when scanning the target image area line by line, starting from the starting point, the y direction is fixed, and the x direction is increased. x0+Δx , x0+2Δx , x0+3Δx ,......, x0+kΔx , until it encounters the right contour of the image and ends the scan of this line, then jumps to the next line and starts scanning, scanning in the direction of increasing x until it encounters the right contour of the graphic and ends, repeating the above scanning operation to scan each line until the last line of the graphic ends.

[0008] Preferably, the method for obtaining the matrix elements in step S2 is: substituting the coordinates of the four points on the target area and the four points on the source image to establish an equation group for calculation, and obtaining the perspective projection transformation formula is: ; Starting from the starting point of the target area, the change rule of the source image coordinates is: Each increase in the x coordinate of the target area Δx , the value of the numerator in the above formula is accumulated m11*Δx , the denominator value is accumulated m31*Δx ; Each increase in the y coordinate of the target area Δy , the value of the numerator in the above formula is accumulated m12*Δy , the denominator value is accumulated m32*Δy ; The coordinates in the corresponding source image can be calculated by the above formula. The calculated coordinates will consist of an integer part and a decimal part, wherein the integer part is used for point selection in step S3, and the decimal part is used for interpolation in step S4.

[0009] Preferably, in step S4, if the required pixel data exists in the current CACHE, the data is directly returned from the CACHE; if the required pixel data does not exist in the current CACHE, an entire CACHE row is updated; according to actual application requirements, the CACHE can be configured as one or more channels, each channel corresponding to a TAG table to record the starting physical address of the source image corresponding to each CACHE row in this channel.

[0010] Preferably, the length of the cache is configured to store a fixed number of pixels according to the tilt angle of the image, and the number of pixels that can be stored in each cache line is the same. The size K of the on-chip memory and the number n of cache ways, the length m of the cache line, and the depth t of each cache way must satisfy the following formula: .

[0011] Preferably, when the tilt angle of the target image exceeds a certain threshold and the number of CACHE misses exceeds another threshold, the TAG table is automatically reversed so that the CACHE row with the strongest spatial correlation is retained.

[0012] Beneficial effects: The method of the present invention uses a group-associative CACHE structure to optimize the problem of low access efficiency of SDR / DDR SDRAM access caused by discontinuous reading of source image pixels during graphic image projection transformation. Through a flexible CACHE structure and replacement strategy, it is guaranteed that better efficiency improvement is achieved at different deformation angles. At the same time, by caching the source image coordinates to the on-chip cache, the main processing modules can be executed in parallel, further improving the processing speed. This method can achieve real-time processing of three-dimensional transformation of images with resolutions above full HD, avoiding the waste of storage bandwidth caused by writing to the memory area again. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the image B on the target area D obtained by 3D deformation processing of the source image A; Figure 2 It is a schematic diagram of image projection transformation; Figure 3 It is a schematic diagram of image scanning rules; Figure 4 is a schematic diagram of the method flow chart; Figure 5 It is a schematic diagram of scanning the target image area line by line and point by point; Figure 6 This is a schematic diagram of the CACHE structure; Figure 7 This is a schematic diagram of the TAG table reversal. DETAILED DESCRIPTION

[0014] In order to make the technical solution of the present invention clearer, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. Example

[0015] like Figure 2 As shown, Figure 2 The screen (a) shown in the figure is projected onto the screen at a certain angle. Figure 2 The figure BB is displayed on the screen (b) in the figure. Through the projection transformation of AA and BB, the conversion function F(x,y) from AA to BB is calculated. If the projection angle remains unchanged, if the BB image is to be displayed as a regular square or rectangle, the BB image needs to be transformed into a CC figure. If the projection angle remains unchanged, if the screen image is to be projected onto the screen and becomes a CC figure, the original AA figure needs to be transformed. At this time, the CC figure is calculated according to the inverse transformation of the F(x,y) function to obtain the DD figure. In other words, in order to realize the CC figure, the original AA figure displayed on the screen needs to be converted into a DD figure display, so that the figure projected at the F(x,y) angle will become a regular rectangle or square CC. According to the perspective projection transformation, the AA figure is first projected onto the screen to calculate the information of the BB figure, that is, the projection rule function F(x,y) is calculated based on the four vertex coordinates of the AA figure and the four vertex coordinates of the BB figure on the screen. Then, according to the desired figure CC displayed on the screen, the four vertex coordinates of the DD figure are calculated according to the inverse mapping image of F(x,y). Finally, the transformation function G(x,y) is calculated according to the four vertex coordinates of the DD figure and the four vertex coordinates of the source image AA. At this point, the DD figure and the AA figure have established a mapping relationship. Therefore, it can be seen that the target area and the source image are projected transformations.

[0016] Hardware image processing module according to Figure 3 According to the scanning rules, the graphics are scanned from left to right and from top to bottom starting from the first row as shown by the horizontal line. According to the mapping relationship G(x,y) obtained above, the coordinates corresponding to the source image are found from the first row and first column according to the mapping relationship, and the corresponding pixels are filled to the first row and first column, and the pixels to be displayed are found and filled from left to right and from top to bottom.

[0017] like Figure 4 As shown, the immersive projection mapping hardware acceleration implementation method based on the CACHE architecture disclosed in this embodiment includes the following six parts: Part 1: First describe the outline of the target figure, such as Figure 5As shown in the figure, the target figure is a quadrilateral composed of four vertices b0 b1 b2b3. Therefore, if the coordinates of the four vertices are known, the coordinates of any point on the edge of the quadrilateral can be calculated by the two-point straight line equation, so that the coordinate outline of the quadrilateral can be obtained. Figure 5 The two edges b0b2b3 in are called the left contour, and the two edges b0b1b3 are called the right contour.

[0018] Next, the target graphic area B is scanned line by line, point by point. Figure 5 For example, take b0 as the starting point (x0, y0), the y direction does not move, and the x direction increases x0+Δx , x0+2Δx , x0+3Δx ,......, x0+kΔx Scan until it touches the right contour of the image and ends the scan of this line. Then jump to the next line and start scanning. (The starting coordinates of the next line are selected from the left contour of the image, and the coordinates are ( x0+kΔy,y0+Δy ) where (x0, y0) is the starting coordinate of the previous line, the distance between the two lines is Δy, and k is the slope of the left contour), scan in the direction of increasing x until it reaches the right contour of the figure. Repeat scanning each line until the last line of the figure ends.

[0019] Part 2: Before performing the calculation, the following preparations are required. Figure 1 For example: Since the target area D and the source image A are perspective projection transformations, the relationship between them can be expressed by a constant coefficient matrix, as follows: ,and ; After solving this matrix, the coordinates (u, v) on the corresponding source image A can be calculated from any point (x, y) on the target area D. The matrix elements can be calculated by substituting the coordinates of the four points b0 b1 b2 b3 on D and the four points a0 a1 a2a3 on A to establish an equation group. The perspective projection transformation formula is: ; The calculation of the source image coordinates of the second part is performed simultaneously with the scanning of the first part. Starting from the point (x0, y0) on D, the change pattern of the source image coordinates is as follows: Each increase in the x coordinate of the target area Δx , the value of the numerator in the above formula is accumulated m11*Δx , the denominator value is accumulated m31*Δx .

[0020] Each increase in the y coordinate of the target area Δy , the value of the numerator in the above formula is accumulated m12*Δy, the denominator value is accumulated m32*Δy .

[0021] The coordinates in the corresponding source image can be calculated by the above formula. The calculated coordinates will consist of an integer part and a decimal part, where the integer part is used for point selection in the third part and the decimal part is used for interpolation in the fourth part.

[0022] Part 3: Part 3 temporarily stores the coordinates calculated by Part 2 to the on-chip memory, so that Parts 1 and 2 can continue to calculate the source image coordinates of the next target pixel without waiting for Part 4 to complete its work. Since the external memory is shared by the system and is slow, if the CACHE needs to be updated, it usually takes a long time for Part 4 to receive the next set of coordinates. The role of Part 3 is to enable Parts 1 and 2 and Parts 4 and 5 to work in parallel, thereby improving efficiency.

[0023] Part 4: The fourth part starts working after obtaining the temporarily stored source image coordinates from the third part. The work of this part is to convert this coordinate into the physical address of the source image and obtain the data of image A from the CACHE buffer. The CACHE buffer is used to cache the source image data obtained from the memory. If the required pixel data exists in the current CACHE, the data is returned directly from the CACHE. If the required pixel data does not exist in the current CACHE, an entire CACHE line is updated. According to actual application requirements, CACHE can be configured as one or more channels, and each channel corresponds to a TAG table to record the starting physical address of the source image corresponding to each CACHE line in this channel. The length of the CACHE can be configured to store a fixed number of pixels according to the tilt angle of the image. The number of pixels that can be stored in each CACHE line is the same. The size K of the on-chip memory, the number n of CACHE channels, the length m of the CACHE line, and the depth t of each CACHE channel must satisfy the following formula: .

[0024] The CACHE structure is as follows Figure 6 As shown. Because the target image is scanned point by point horizontally, the reading of the corresponding source image has a certain local correlation. When the target image has a small tilt angle, the corresponding original image has a strong horizontal correlation in reading, and a longer CACHE row length can improve efficiency. When the target image has a large tilt angle, the corresponding original image has a strong vertical correlation in reading, and usually SDRAM can only read horizontally by row, so a shorter CACHE row length can improve efficiency. This design can reasonably configure the number of CACHE ways n, the length of the CACHE row m, and the depth t of each CACHE way according to the tilt angle of the target image to ensure full utilization of the on-chip memory.

[0025] In this embodiment, the replacement of CACHE rows can use an enhanced LRU (Least Recent Use) update strategy. In most cases, LRU can provide a better CACHE hit rate. However, because the image is stored by row, each time the row is changed, the local correlation of the pixels circulates from the end of the row to the beginning of the row. If the target image is tilted at a too large angle, it cannot be guaranteed that the data at the head of the row is still stored in the CACHE every time the end of the row is scanned. At this time, the LRU strategy becomes the worst strategy. Because at this time, the data replaced by the CACHE is the point with the largest time interval with the current scanning point, and these points are the ones with the strongest spatial correlation with the current point. This design takes this situation into consideration. When the tilt angle of the target image exceeds a certain threshold and the number of CACHE misses exceeds another threshold, the TAG table is automatically reversed, so that the CACHE row with the strongest spatial correlation is retained, such as Figure 7 shown.

[0026] In this embodiment, replacement strategies such as FRU / FIFO / RANDOM are also retained, and system developers can also use these strategies according to actual needs.

[0027] Part 5: Use the data retrieved from the CACHE buffer in Part 4 to perform interpolation processing based on the decimal part of the coordinates calculated in Part 2. The interpolation method can be selected according to application needs.

[0028] Part 6: Automatically calculate the display coordinates of the current output pixel according to the output image resolution. When the current display coordinates are outside the target graphic B, output a configurable background color. When the current display coordinates are within the target graphic B, output the interpolated pixels calculated in Part 5.

[0029] This embodiment optimizes image scanning, calculates the area range of each row within the display area where the target image DD falls, and uses an operator to locate its edge. When the scan falls outside the DD area, the display engine can directly display the defined background color without calculating coordinates and searching for positions. When the edge position is scanned, the operator will tell the scanner to start calculating according to the G(x,y) function to find the position of the original image. When scanning each row, only the positions within the operator area are calculated and searched, and the areas outside the operator are not calculated and searched. Compared with the traditional method of displaying images, which starts from the first point of each row and calculates the source image coordinates until the last point of the last row, this embodiment changes the scanning method and greatly reduces the number of calculations.

[0030] This embodiment also optimizes the cache extraction of the source image. The original image AA is normally extracted from the hard disk and the entire image is stored in the DDR. There is a 64-byte cache in the image extraction module. When the scanner locates the original image coordinates, the data at the original image coordinate position will be taken out, and the 64-byte data behind the original image in the DDR row will also be extracted. Figure 3 From the slope scan line of the AA area, it can be seen that the extraction rule of the real image data is carried out according to the inclined oblique line. If the 64 bytes of data behind the current coordinates are extracted according to the standard cache rule, the hit efficiency is low. This embodiment implements an algorithm that calculates the extraction trajectory of the source image while scanning the image in each line. When the scanner encounters the area defined by each line operator, it triggers the change of the extraction trajectory, so that the data near the best trajectory that hits the original image position is correctly extracted into the cache line, thereby greatly improving the hit rate of the image cache and reducing the size of the cache.

[0031] This embodiment also optimizes the CACHE configuration, because the image changes from AA to DD at different projection angles, and the projected DD image is deformed differently. In this way, the length of the search track is different. This embodiment implements a structural design of variable cache line length and variable cache line depth, which can configure different line depths and line lengths according to different projection angles, so that the cache misses are less and the hit efficiency is higher. In addition, when the cache misses, the miss statistics are also performed, and the cache TAG is flipped to allow users to hit the cache more efficiently.

[0032] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for hardware acceleration of immersive projection mapping based on CACHE architecture, characterized in that: The following steps are involved: S1. Target graphic scanning: first describe the outline of the target graphic, then scan the target graphic area line by line and point by point until the scanning is completed; S2. Source image coordinate calculation: The target area and the source image are projected; therefore, the relationship between the two is expressed by a constant coefficient matrix: ,and ; After solving the above matrix, the coordinates (u, v) on the corresponding source image can be calculated from any point (x, y) on the target area. The calculation of the source image coordinates is performed simultaneously with the scanning in step S1. S3, on-chip buffer: store the coordinates calculated in step S2 into the on-chip memory, so that steps S1 and S2 can continue to calculate the source image coordinates of the next target pixel; S4, coordinate conversion: after the temporarily stored source image coordinates are obtained in step S2, the coordinates are converted into the physical address of the source image, and then the data of the source image is obtained from the CACHE buffer; S5, interpolation processing: the data taken out from the CACHE buffer in step S4 is interpolated according to the decimal part of the coordinates calculated in step S2; S6, Image processing: Automatically calculate the display coordinates of the output pixels according to the output image resolution; output a configurable background color when the current display coordinates are outside the target graphic; When the current display coordinates are within the target graphic, the interpolated pixels calculated in the fifth part are output.

2. The method for implementing immersive projection mapping hardware acceleration based on CACHE architecture according to claim 1, characterized in that: In step S1, when describing the outline of the target figure, since the target figure is a quadrilateral composed of four vertices, when the coordinates of the four vertices are known, the coordinates of the points on any side of the quadrilateral can be calculated by the two-point straight line equation, thereby obtaining the coordinate outline of the quadrilateral and dividing it into a left outline and a right outline.

3. The method for implementing immersive projection mapping hardware acceleration based on CACHE architecture according to claim 2, characterized in that: In step S1, when scanning the target image area line by line, starting from the starting point, the y direction is fixed and the x direction is increased. x0+Δx , x0+2Δx , x0+3Δx ,......, x0+kΔx , until it encounters the right contour of the image and ends the scan of this line, then jumps to the next line and starts scanning, scanning in the direction of increasing x until it encounters the right contour of the graphic and ends, repeating the above scanning operation to scan each line until the last line of the graphic ends.

4. The method for implementing immersive projection mapping hardware acceleration based on CACHE architecture according to claim 1, characterized in that: The method for obtaining the matrix elements in step S2 is: substitute the coordinates of the four points on the target area and the four points on the source image to establish an equation group for calculation, and obtain the perspective projection transformation formula: ； Starting from the starting point of the target area, the change rule of the source image coordinates is: Each increase in the x coordinate of the target area Δx , the value of the numerator in the above formula is accumulated m11*Δx , the denominator value is accumulated m31*Δx ; Each increase in the y coordinate of the target area Δy , the value of the numerator in the above formula is accumulated m12*Δy , the denominator value is accumulated m32*Δy ; The coordinates in the corresponding source image can be calculated by the above formula. The calculated coordinates will consist of an integer part and a decimal part, wherein the integer part is used for point selection in step S3, and the decimal part is used for interpolation in step S4.

5. The method for implementing immersive projection mapping hardware acceleration based on CACHE architecture according to claim 1, characterized in that: In step S4, if the required pixel data exists in the current CACHE, the data is directly returned from the CACHE; if the required pixel data does not exist in the current CACHE, an entire CACHE row is updated; according to actual application requirements, the CACHE can be configured as one or more channels, each channel corresponding to a TAG table to record the starting physical address of the source image corresponding to each CACHE row in this channel.

6. The method for implementing immersive projection mapping hardware acceleration based on CACHE architecture according to claim 5, characterized in that: The length of the cache is configured to store a fixed number of pixels according to the tilt angle of the image. The number of pixels that can be stored in each cache line is the same. The size K of the on-chip memory and the number n of cache ways, the length m of the cache line, and the depth t of each cache way must satisfy the following formula: 。 7. The method for implementing immersive projection mapping hardware acceleration based on CACHE architecture according to claim 6, characterized in that: When the target image tilt angle exceeds a certain threshold and the number of cache misses exceeds another threshold, the TAG table is automatically reversed so that the cache row with the strongest spatial correlation is retained.

Images