A frame buffer merging method based on dirty rectangle area calculation

Abstract

The present invention provides a frame buffer merging method based on dirty rectangular area calculation, comprising: S1. after a frame buffer device driver applies for reserved frame buffer, the number of reserved frame buffers is set to MAX_NUM; S2. for synchronization between frame buffers of MAX_NUM, the formula is satisfied: Si=S(i-1)+Ri; i=(i+1)%MAX_NUM; S3. assuming MAX_NUM=3, the dirty rectangular area R1 currently to be drawn can be updated to the frame buffer S1, that is, S1=R1; S4. the dirty rectangular area R1 currently to be drawn can be updated to the frame buffer S1, that is, S1=R1; 2. Update R2 to frame buffer S2, S2 = R1 x R2. S5. Now draw R3, update R3 to frame buffer S3, S3 = R1 x R2 x R3. S6. When there are no new dirty rectangles, for frame buffer devices with triple framebuffer vertical synchronization enabled, update the recorded rectangle to frame buffer S1, S4 = S1 + R2 x R3. S7. When updating frame buffer S2, S5 = S2 + R3. S8. When updating frame buffer S3, S6 = S3. This method can be used to merge images between multiple frame buffers using vertical synchronization based on dirty rectangle calculations.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a frame buffer merging method based on dirty rectangular area calculation. Background Art

[0002] As the requirements for graphics and images in the embedded field continue to increase, an increasingly rich set of graphics controls are being used to draw software application interfaces, placing higher demands on the frame rate of graphics displays and image quality. Existing graphical user interface development frameworks typically use a dirty rectangle mechanism when interacting with frame buffer devices for image data, reducing resource consumption by reducing the amount of data updates. However, the dirty rectangle mechanism cannot solve the problem of screen tearing. In scenarios where images are frequently updated, we can easily see abnormalities such as misalignment and discontinuity in the image displayed on the screen. Therefore, we introduced a vertical synchronization mechanism in the graphical user interface to prevent the frame buffer currently being read by the hardware from being overwritten by the graphical user interface simultaneously. This mechanism also reduces display latency by providing other operable frame buffers to the graphical user interface through multiple buffers.

[0003] However, there is no combination between the existing dirty rectangle technology and the vertical synchronization technology. When applying dirty rectangles, a typical graphical user interface only updates data on a fixed frame buffer and does not address the image merging problem between frame buffers.

[0004] In addition, technical terms commonly used in this field include:

[0005] Dirty rectangles: A key concept in graphics performance optimization, these refer to the local areas that need to be redrawn when an image is updated. Using dirty rectangles can reduce the amount of data required for image updates, lowering resource consumption and enhancing display performance.

[0006] Frame buffer device: A frame buffer device is an abstraction of graphics hardware that represents the frame buffer area of ​​some video hardware and allows software applications to access the graphics hardware through a well-defined interface.

[0007] Vertical synchronization: When the display device screen reads the frame buffer line by line, the image is prohibited from updating the frame buffer currently used by the hardware. The image is allowed to overwrite the current frame buffer only after the screen display is completed and the current frame buffer is no longer read, so as to avoid screen tearing problems, but it will cause display delays.

[0008] Screen tearing: When the display device screen reads the image in the frame buffer line by line for display, the program is drawing the screen and overwrites the read frame buffer, resulting in abnormal phenomena such as image misalignment and discontinuity on the screen.

[0009] Multi-buffering: When a display device has vertical synchronization enabled, the number of frame buffers can be expanded to reduce display latency, providing more buffers for image updates. This allows the image to be overwritten on other frame buffers while one frame buffer is disabled, and when the display device screen finishes reading a frame buffer, it can immediately switch to another frame buffer that has been updated for reading. Summary of the Invention

[0010] In order to solve the above problems, the purpose of this application is to provide a method that can perform image merging between multiple frame buffers based on dirty rectangle area calculation and correspondingly using vertical synchronization technology.

[0011] Specifically, the present invention provides a frame buffer merging method based on dirty rectangular area calculation, the method comprising the following steps:

[0012] Step S1. After the frame buffer device driver applies for reserved frame buffer, the number of reserved frame buffers is set to MAX_NUM;

[0013] Step S2: For MAX_NUM frame buffer synchronization, the following formula is satisfied:

[0014] Si=S(i-1)+Ri;

[0015] i=(i+1)%MAX_NUM;

[0016] Where i represents the subscript value corresponding to the frame buffer of the current operation;

[0017] Si represents the graphics data value of the i-th frame buffer;

[0018] S(i-1) represents the graphics data value of the i-1th frame buffer;

[0019] Ri represents the graphic data that needs to be drawn currently;

[0020] After each drawing of a graphic data and synchronization between the frame buffers, the current i value is incremented by one, and the incremented value is modulo MAX_NUM to ensure that the i value is within the frame buffer subscript range;

[0021] Step S3. When the frame buffer MAX_NUM=N, SN=S(N-1)+RN; if MAX_NUM=3, that is, 3 frame buffers are reserved, then when the graphical user interface obtains the dirty rectangular area R1 to be drawn, R1 can be updated to the frame buffer S1. At this time, i=1, that is,

[0022] S1=R1;

[0023] Step S4. When the graphical user interface obtains the dirty rectangular area R2 to be drawn, R2 can be updated to the frame buffer S2, that is, the area after the R1 and R2 matrices are merged and deduplicated is updated, and R1xR2 is used to represent the matrix merge and deduplication. At this time

[0024] S2 = R1xR2;

[0025] Step S5. When the graphical user interface obtains the dirty rectangular area R3 that needs to be drawn, R3 can be updated to the frame buffer S3.

[0026] S3 = R1xR2xR3;

[0027] Step S6. When there is no new dirty rectangular area in the graphical user interface, that is, when the user no longer draws a new image, there is no new dirty rectangular area input at this time. For the frame buffer device with the triple frame buffer vertical synchronization mechanism enabled, that is, the case of MAX_NUM=3 and the case of MAX_NUM>1, it is still necessary to update the rectangular area in the record to the frame buffer S1. Here, the selection of updating frame buffer S1, frame buffer S2, and frame buffer S3 is performed in sequence. After updating frame buffer S3, frame buffer S1 is updated again, and the cycle continues. The recorded rectangular area needs to comply with the validity number not exceeding MAX_NUM, so the R1 area is discarded from the record. Since MAX_NUM is set to 3, there are only three frame buffers in total. Therefore, when a rectangular area, here is the R1 area, is synchronized 3 times between frame buffers, all frame buffers have updated the R1 area. The R1 area no longer needs to be synchronized and can be discarded. This is recorded as

[0028] S4=S1+R2xR3;

[0029] Step S7. For the frame buffer device with triple frame buffer vertical synchronization mechanism enabled when there is no new dirty rectangular area, that is, the case of MAX_NUM=3, update the frame buffer S2 in sequence.

[0030] S5=S2+R3;

[0031] Here, step S7 corresponds to the case where there is no new dirty rectangular area input. If there is a dirty rectangular area input, the process proceeds normally according to step S4.

[0032] Step S8. For the frame buffer device with triple frame buffer vertical synchronization mechanism enabled when there is no new dirty rectangular area, that is, the case of MAX_NUM=3, update the frame buffer S3 in sequence.

[0033] S6=S3;

[0034] Here, step S8 corresponds to the case where no new dirty rectangular area is input. If a dirty rectangular area is input, the process proceeds normally according to step S5.

[0035] In step S1, the method for the frame buffer device driver to apply for reserved frame buffer is to reserve frame buffer shared memory in the kernel display driver, with continuous physical addresses and hardware alignment; the function for reserving memory is the system function dma_alloc_coherent, and the parameter is the memory size.

[0036] In step S2, the synchronization between frame buffers is to avoid the waiting time of frame drawing by switching between multiple buffers during the graphics frame drawing process, so as to make data processing smoother. At this time, when switching between multiple buffers, the drawing process of the previous frame buffer needs to be copied to avoid the loss of drawn data. This process is the synchronization between frame buffers.

[0037] The graphical user interface, that is, when the user needs to display an image on the frame buffer, the user will specify the size and position of the display image and prepare the graphic data; after the dirty rectangular area to be drawn is obtained based on the display image size and position specified by the user, the width and height of the dirty rectangular area are obtained from the image size, and the x and y values ​​of the image's two-dimensional space are obtained from the image display position. The corresponding coordinate system origin here is the starting address of the frame buffer.

[0038] In step S4, the specific steps of updating the area after the R1 and R2 matrices are merged and deduplicated are as follows:

[0039] Each rectangle structure contains four parameters, namely the horizontal coordinate x, vertical coordinate y of the upper left corner, and the width w and height h of the rectangle;

[0040] First, determine whether the two rectangles intersect; this can be determined by comparing the positions of the right, left, top, and bottom boundaries of the two rectangles;

[0041] If the right border of any rectangle is smaller than the left border of the other rectangle, or the left border is larger than the right border of the other rectangle, or the lower border is smaller than the upper border of the other rectangle, or the upper border is larger than the lower border of the other rectangle, then the two rectangles do not intersect and the original image before merging is directly returned;

[0042] If two rectangles intersect, the combined image can be represented by finding the union rectangle of the two rectangles; the x-coordinate of the new rectangle's upper left corner is the minimum of the x-coordinates of the two rectangles' upper left corners, the y-coordinate of the new rectangle's upper left corner is the minimum of the y-coordinates of the two rectangles' upper left corners, the width of the rectangle is the maximum of the widths of the two rectangles, and the height of the rectangle is the maximum of the heights of the two rectangles;

[0043] Then, the overlapping area between the two rectangles needs to be removed. The deduplicated rectangle can be calculated by determining the positions of the boundaries of the two rectangles. The right boundary of the deduplicated rectangle is the minimum of the right boundaries of the two rectangles, the left boundary is the maximum of the left boundaries of the two rectangles, the bottom boundary is the minimum of the bottom boundaries of the two rectangles, and the top boundary is the maximum of the top boundaries of the two rectangles.

[0044] Finally, the deduplicated rectangles are output as the merged deduplicated image.

[0045] Therefore, the advantage of the present application is that the solution of the present application can meet the requirements of multiple buffers for image merging between frame buffers while using the dirty rectangle mechanism of the graphic user interface, thereby enhancing the graphic display effect of the frame buffer device. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0047] Figure 1 is a schematic diagram of an embodiment of the present method.

[0048] Figure 2 is a flow chart of this method. DETAILED DESCRIPTION

[0049] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail with reference to the accompanying drawings.

[0050] This application provides a frame buffer merging method based on dirty rectangular area calculation, refer to Figure 1 As shown, the core method logic of this application is described as follows: S1 to S6 in the figure here refer to the graphics memory operated by the corresponding steps, R1, R2, and R3 refer to the dirty rectangular areas input by the user, which represent the part of the graphics area corresponding to S1, S2, and S3 that needs to be updated. Further, as Figure 2 As shown:

[0051] Step S1. After the frame buffer device driver applies for reserved frame buffer, the frame buffer device driver applies for reserved frame buffer by reserving frame buffer shared memory in the kernel display driver, with continuous physical addresses and hardware alignment; the memory reservation function is the system function dma_alloc_coherent, the parameter of which is the memory size; and the number of reserved frame buffers is set to MAX_NUM;

[0052] Step S2: For synchronization between MAX_NUM frame buffers, the following formula must be satisfied:

[0053] Si=S(i-1)+Ri;

[0054] i=(i+1)%MAX_NUM;

[0055] Where i represents the subscript value corresponding to the frame buffer of the current operation;

[0056] Si represents the graphics data value of the i-th frame buffer;

[0057] S(i-1) represents the graphics data value of the i-1th frame buffer;

[0058] Ri represents the graphic data that needs to be drawn currently;

[0059] After each drawing of a graphic data and synchronization between the frame buffers, the current i value is incremented by one, and the incremented value is modulo MAX_NUM to ensure that the i value is within the frame buffer subscript range;

[0060] The inter-frame buffer synchronization is to avoid the waiting time of frame drawing by switching between multiple buffers during the graphics frame drawing process, making data processing smoother. At this time, when switching between multiple buffers, the drawing process of the previous frame buffer needs to be copied to avoid the loss of drawing data. This process is inter-frame buffer synchronization;

[0061] Step S3. When the frame buffer MAX_NUM=N, SN=S(N-1)+RN; let MAX_NUM=3, that is, 3 frame buffers are reserved; then when the graphical user interface, that is, the user needs to display an image on the frame buffer, the user will specify the size and position of the display image, and the user will prepare the graphic data. After the dirty rectangular area R1 to be drawn is obtained according to the display image size and position specified by the user, the width and height of the dirty rectangular area are obtained from the image size, and the x and y values ​​of the image two-dimensional space are obtained from the image display position. The corresponding coordinate system origin here is the starting address of the frame buffer; R1 can be updated to the frame buffer S1, and at this time i=1, that is

[0062] S1=R1;

[0063] Step S4. When the graphical user interface obtains the dirty rectangular area R2 to be drawn, it can be updated to the frame buffer S2, that is, the area after the R1 and R2 matrices are merged and deduplicated is updated, and the matrix merge and deduplication is represented by R1xR2. At this time

[0064] S2 = R1xR2;

[0065] The specific steps are:

[0066] Each rectangle structure contains four parameters, namely the horizontal coordinate x, vertical coordinate y of the upper left corner, and the width w and height h of the rectangle.

[0067] First, determine whether the two rectangles intersect. This can be done by comparing the positions of the right, left, top, and bottom borders of the two rectangles. If the right border of any rectangle is smaller than the left border of the other rectangle, or the left border is larger than the right border of the other rectangle, or the bottom border is smaller than the top border of the other rectangle, or the top border is larger than the bottom border of the other rectangle, then the two rectangles do not intersect, and the original image before the merge is returned directly.

[0068] If two rectangles intersect, the combined image can be represented by finding the union rectangle of the two rectangles. The x-coordinate of the new rectangle's upper left corner is the minimum of the x-coordinates of the two rectangles' upper left corners, the y-coordinate of the new rectangle's upper left corner is the minimum of the y-coordinates of the two rectangles' upper left corners, the width of the new rectangle is the maximum of the widths of the two rectangles, and the height of the new rectangle is the maximum of the heights of the two rectangles.

[0069] Next, we need to remove the overlapping areas between the two rectangles. We can calculate the deduplicated rectangles by determining the positions of the boundaries of the two rectangles. The right boundary of the deduplicated rectangle is the minimum of the right boundaries of the two rectangles, the left boundary is the maximum of the left boundaries of the two rectangles, the bottom boundary is the minimum of the bottom boundaries of the two rectangles, and the top boundary is the maximum of the top boundaries of the two rectangles.

[0070] Finally, the deduplicated rectangles are output as the merged deduplicated image.

[0071] Step S5. When the graphical user interface obtains the dirty rectangular area R3 to be drawn, it can be updated to the frame buffer S3.

[0072] S3 = R1xR2xR3;

[0073] Step S6. When there is no new dirty rectangular area in the graphical user interface, that is, when the user no longer draws a new image, there is no new dirty rectangular area input at this time. For the frame buffer device with the triple frame buffer vertical synchronization mechanism enabled, that is, the case of MAX_NUM=3 and the case of MAX_NUM>1 mentioned above, it is still necessary to update the rectangular area in the record to the frame buffer S1. Here, the selection of updating frame buffer S1, frame buffer S2, and frame buffer S3 is performed in sequence. After updating frame buffer S3, frame buffer S1 is updated again, and the cycle continues; however, the recorded rectangular area needs to comply with the valid number not exceeding MAX_NUM, so the R1 area is discarded from the record. Since MAX_NUM is set to 3, there are only three frame buffers in total. Therefore, when a rectangular area, such as the R1 area, has been synchronized 3 times between the frame buffers, all frame buffers have updated the R1 area at this time, and the R1 area no longer needs to be synchronized and can be discarded. This is recorded as

[0074] S4=S1+R2xR3;

[0075] Step S7. For the frame buffer device with triple frame buffer vertical synchronization mechanism enabled when there is no new dirty rectangular area, that is, the case of MAX_NUM=3, update the frame buffer S2 in sequence. Here, step S7 corresponds to the case where there is no new dirty rectangular area input. If there is a dirty rectangular area input, proceed normally according to step S4.

[0076] S5=S2+R3;

[0077] Step S8. For the frame buffer device with triple frame buffer vertical synchronization mechanism enabled when there is no new dirty rectangular area, that is, the case of MAX_NUM=3, update the frame buffer S3 in sequence. Step S8 here corresponds to the case where there is no new dirty rectangular area input. If there is a dirty rectangular area input, proceed normally according to step S5.

[0078] S6=S3.

[0079] In summary, the key points of the method of the present application are the relevant steps of the frame buffer image merging method.

[0080] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A frame buffer merging method based on dirty rectangular area calculation, characterized in that: The method comprises the following steps: Step S1. After the frame buffer device driver applies for reserved frame buffer, the number of reserved frame buffers is set to MAX_NUM; Step S2: For MAX_NUM frame buffer synchronization, the following formula is satisfied: Si = S(i-1) + Ri; i = (i+1)%MAX_NUM; Where i represents the subscript value corresponding to the frame buffer of the current operation; Si represents the graphics data value of the i-th frame buffer; S(i-1) represents the graphics data value of the i-1th frame buffer; Ri represents the graphic data that needs to be drawn currently; After each drawing of a graphic data and synchronization between the frame buffers, the current i value is incremented by one, and the incremented value is modulo MAX_NUM to ensure that the i value is within the frame buffer subscript range; Step S3. When the frame buffer MAX_NUM=N, SN = S(N-1) + RN; if MAX_NUM=3, that is, 3 frame buffers are reserved, then when the graphical user interface obtains the dirty rectangular area R1 to be drawn, R1 is updated to the frame buffer S1. At this time, i=1, that is, S1 = R1; Step S4. When the graphical user interface obtains the dirty rectangular area R2 to be drawn, R2 is updated to the frame buffer S2, that is, the area after the R1 and R2 matrices are merged and deduplicated is updated, and R1xR2 is used to represent the matrix merge and deduplication. At this time S2 = R1xR2; Step S5. When the graphical user interface obtains the dirty rectangular area R3 to be drawn, R3 is updated to the frame buffer S3. S3 = R1xR2xR3; Step S6. When there is no new dirty rectangular area in the graphical user interface, that is, when the user no longer draws a new image, there is no new dirty rectangular area input at this time. For the frame buffer device with the triple frame buffer vertical synchronization mechanism enabled, that is, the case of MAX_NUM=3 and the case of MAX_NUM>1, it is still necessary to update the rectangular area in the record to the frame buffer S1. Here, the selection of updating frame buffer S1, frame buffer S2, and frame buffer S3 is performed in sequence. After updating frame buffer S3, frame buffer S1 is updated again, and the cycle continues. The recorded rectangular area needs to comply with the valid number not exceeding MAX_NUM, so the R1 area is discarded from the record. Since MAX_NUM is set to 3, there are only three frame buffers in total. Therefore, when a rectangular area, here is the R1 area, is synchronized 3 times between frame buffers, all frame buffers have updated the R1 area at this time, and the R1 area no longer needs to be synchronized, and the R1 area is discarded; this is recorded as S4 = S1 + R2xR3; Step S7. For the frame buffer device with triple frame buffer vertical synchronization mechanism enabled when there is no new dirty rectangular area, that is, the case of MAX_NUM=3, update the frame buffer S2 in sequence. S5 = S2 + R3; Here, step S7 corresponds to the case where there is no new dirty rectangular area input. If there is a dirty rectangular area input, the process proceeds normally according to step S4. Step S8. For the frame buffer device with triple frame buffer vertical synchronization mechanism enabled when there is no new dirty rectangular area, that is, the case of MAX_NUM=3, update the frame buffer S3 in sequence. S6 = S3; Here, step S8 corresponds to the case where no new dirty rectangular area is input. If a dirty rectangular area is input, the process proceeds normally according to step S5.

2. The frame buffer merging method based on dirty rectangular area calculation according to claim 1, characterized in that: In step S1, the method for the frame buffer device driver to apply for reserved frame buffer is to reserve frame buffer shared memory in the kernel display driver, with continuous physical addresses and hardware alignment; the function for reserving memory is the system function dma_alloc_coherent, and the parameter is the memory size.

3. The frame buffer merging method based on dirty rectangular area calculation according to claim 1, characterized in that: In step S2, the synchronization between frame buffers is to avoid the waiting time of frame drawing by switching between multiple buffers during the graphics frame drawing process, so as to make data processing smoother. At this time, when switching between multiple buffers, the drawing process of the previous frame buffer needs to be copied to avoid the loss of drawn data. This process is the synchronization between frame buffers.

4. The frame buffer merging method based on dirty rectangular area calculation according to claim 1, characterized in that: The graphical user interface, that is, when the user needs to display an image on the frame buffer, the user will specify the size and position of the display image and prepare the graphic data; after the dirty rectangular area to be drawn is obtained based on the display image size and position specified by the user, the width and height of the dirty rectangular area are obtained from the image size, and the x and y values ​​of the image's two-dimensional space are obtained from the image display position. The corresponding coordinate system origin here is the starting address of the frame buffer.

5. The frame buffer merging method based on dirty rectangular area calculation according to claim 1, characterized in that: In step S4, the specific steps of updating the area after the R1 and R2 matrices are merged and deduplicated are as follows: Each rectangle structure contains four parameters, namely the horizontal coordinate x, vertical coordinate y of the upper left corner, and the width w and height h of the rectangle; First, determine whether the two rectangles intersect; this is done by comparing the positions of the right, left, top, and bottom boundaries of the two rectangles; If the right border of any rectangle is smaller than the left border of the other rectangle, or the left border is larger than the right border of the other rectangle, or the lower border is smaller than the upper border of the other rectangle, or the upper border is larger than the lower border of the other rectangle, then the two rectangles do not intersect and the original image before merging is directly returned; If the two rectangles intersect, the union rectangle of the two rectangles is used to represent the merged image; the x-coordinate of the upper left corner of the new rectangle is the minimum of the x-coordinates of the upper left corners of the two rectangles, the y-coordinate of the upper left corner of the new rectangle is the minimum of the y-coordinates of the upper left corners of the two rectangles, the width of the rectangle is the maximum of the widths of the two rectangles, and the height of the rectangle is the maximum of the heights of the two rectangles; Then, the overlapping area between the two rectangles needs to be removed; the deduplicated rectangle is calculated by determining the positions of the boundaries of the two rectangles; the right boundary of the deduplicated rectangle is the minimum of the right boundaries of the two rectangles, the left boundary is the maximum of the left boundaries of the two rectangles, the bottom boundary is the minimum of the bottom boundaries of the two rectangles, and the top boundary is the maximum of the top boundaries of the two rectangles; Finally, the deduplicated rectangles are output as the merged deduplicated image.

Images