Image rendering method, display equipment, device, storage medium and program product

Abstract

The embodiment of the invention provides an image rendering method and device, display equipment, a storage medium and a program product, and relates to the technical field of machine vision, the method is applied to a CPU, and the method comprises the following steps: responding to an adjustment operation for an image currently displayed in a display area; determining the position of an image area needing to be displayed in the display area in the original image after the adjustment operation as a reference position; the image currently displayed in the display area is obtained based on the original image; generating an image which is the same as the display area in size and consistent with the picture content at the reference position in the original image to obtain a first image; and sending the first image to the GPU, so that the GPU renders the first image in the display area. Thus, it can be avoided that much video memory space is occupied in the image rendering process, especially under the condition that the resolution of the original image is high, occupation of the video memory space in the image rendering process can be reduced, and storage resources are saved.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to an image rendering method, display device, apparatus, storage medium and program product. Background Art

[0002] In the field of machine vision technology, it is often necessary to display images on a display device. For example, when an image needs to be displayed, the GPU (Graphics Processing Unit) needs to obtain the entire image and scale the entire image according to the size of the display area in the display device to render the scaled image in the display area. Subsequently, the user can adjust the screen content displayed in the display area to display different image areas in the image. In response to the adjustment operation triggered by the user, the GPU determines the image area that currently needs to be displayed from the entire image, and renders the determined image area in the display area.

[0003] It can be seen that during the entire process of rendering an image, the GPU needs to store the entire image in the video memory, and the resolution of the entire image is often high, so the video memory occupancy is also high. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image rendering method, display device, apparatus, storage medium and program product to reduce the occupation of video memory space during image rendering and save storage resources. The specific technical solution is as follows:

[0005] In a first aspect of an embodiment of the present application, an image rendering method is provided, which is applied to a CPU (Central Processing Unit), and the method includes:

[0006] In response to an adjustment operation on an image currently displayed in a display area, determining a position of an image area in an original image that needs to be displayed in the display area after the adjustment operation as a reference position; wherein the image currently displayed in the display area is obtained based on the original image;

[0007] Generate an image having the same size as the display area and consistent with the picture content at the reference position in the original image to obtain a first image;

[0008] The first image is sent to a GPU so that the GPU renders the first image in the display area.

[0009] Optionally, generating an image having the same size as the display area and consistent with the picture content at the reference position in the original image to obtain a first image includes:

[0010] If a first size of the image area at the reference position is larger than a second size of the display area, sampling the image area at the reference position according to a ratio of the first size to the second size to obtain an image of the second size as the first image;

[0011] If the first size is equal to the second size, extracting the image area at the reference position from the original image as the first image;

[0012] If the first size is smaller than the second size, the image area at the reference position is expanded in the original image to obtain an image area to be used; wherein the pixel coordinates of the vertices of the image area to be used in the original image are integers; the image area to be used is interpolated according to the ratio of the second size to the first size to be used; and the image area of ​​the second size is extracted from the interpolation result obtained as the first image.

[0013] Optionally, expanding the image area at the reference position in the original image to obtain the image area to be used includes:

[0014] In the original image, the pixel coordinates of the upper left corner vertex of the image area at the reference position are rounded toward the upper left corner, and the pixel coordinates of the lower right corner vertex of the image area at the reference position are rounded toward the lower right corner, to obtain the image area to be used;

[0015] The step of extracting the image region of the second size from the obtained interpolation result as the first image includes:

[0016] Calculate the product of the pixel coordinates of the top left corner of the image area at the reference position in the original image and the ratio to be used to obtain the starting pixel coordinates;

[0017] The starting pixel coordinates are used as the pixel coordinates of the upper left corner, and the image area of ​​the second size is extracted from the obtained interpolation result to obtain the first image.

[0018] Optionally, before determining, in response to the adjustment operation on the image currently displayed in the display area, the position of the image area in the original image that needs to be displayed in the display area after the adjustment operation as the reference position, the method further includes:

[0019] When receiving a display instruction for the original image, scaling the original image according to the size of the display area to obtain a second image;

[0020] The second image is sent to the GPU so that the GPU renders the second image in the display area.

[0021] Optionally, the method further includes:

[0022] Acquire color information of a graphic marked on the original image, and a corresponding position of the graphic in the original image;

[0023] Based on the relationship between the position corresponding to the graphic in the original image and the reference position, and the zoom ratio of the first image relative to the original image, determine a graphic region to be rendered in the first image, and determine the position of the graphic region to be rendered in the first image;

[0024] The position of the graphics area to be rendered in the first image and the color information of the graphics area to be rendered are sent to the GPU, so that the GPU renders the graphics area to be rendered on an upper layer of the first image in the display area.

[0025] A second aspect of the embodiment of the present application further provides an image rendering method, which is applied to a GPU, and the method includes:

[0026] Receive a first image sent by the CPU; wherein the first image is: an image generated by the CPU having the same size as the display area and having the same screen content as that at a reference position in the original image; the reference position is: a position of an image area in the original image that needs to be displayed in the display area after the adjustment operation determined by the CPU in response to an adjustment operation on the image currently displayed in the display area; the image currently displayed in the display area is obtained based on the original image;

[0027] The first image is rendered in the display area.

[0028] Optionally, before receiving the first image sent by the CPU, the method further includes:

[0029] receiving a second image sent by the CPU; wherein the second image is obtained by scaling the original image by the CPU according to the size of the display area when receiving a display instruction for the original image;

[0030] The second image is rendered in the display area.

[0031] Optionally, the method further includes:

[0032] receiving the position of the graphics area to be rendered in the first image and the color information of the graphics area to be rendered sent by the CPU; wherein the graphics area to be rendered is determined by the CPU based on the relationship between the position corresponding to the graphics marked in the original image and the reference position in the original image, and the zoom ratio of the first image relative to the original image;

[0033] The to-be-rendered graphic area is rendered on an upper layer of the first image in the display area according to the received position and color information.

[0034] Optionally, rendering the first image in the display area includes:

[0035] Loading an underlying canvas in the display area, and rendering the first image in the underlying canvas;

[0036] Rendering the graphics area to be rendered on an upper layer of the first image in the display area according to the received position and color information includes:

[0037] A top-level canvas is loaded on the upper layer of the bottom-level canvas in the display area, and the graphics area to be rendered is rendered in the top-level canvas according to the received position and color information.

[0038] Optionally, after loading an underlying canvas in the display area and rendering the first image in the underlying canvas, the method further includes:

[0039] storing the bottom canvas on which the first image has been rendered in a video memory;

[0040] When the position of the new graphics area to be rendered in the first image and the color information of the new graphics area to be rendered sent by the CPU are received, the bottom canvas on which the first image has been rendered and recorded in the video memory is loaded, and the new graphics area to be rendered is rendered in the top canvas according to the received new position and color information.

[0041] According to a third aspect of the embodiments of the present application, a display device is provided, including:

[0042] A CPU, used to execute any of the above-mentioned image rendering methods applied to a CPU;

[0043] A GPU is used to execute any of the above-mentioned image rendering methods applied to a GPU.

[0044] According to a fourth aspect of the embodiments of the present application, there is further provided an image rendering device, which is applied to a CPU, and the device comprises:

[0045] A reference position determination module, configured to, in response to an adjustment operation on an image currently displayed in a display area, determine a position of an image area in an original image that needs to be displayed in the display area after the adjustment operation as a reference position; wherein the image currently displayed in the display area is obtained based on the original image;

[0046] An image generation module, used to generate an image having the same size as the display area and consistent with the picture content at the reference position in the original image, to obtain a first image;

[0047] The first image sending module is configured to send the first image to the GPU, so that the GPU renders the first image in the display area.

[0048] Optionally, the image generation module includes:

[0049] A first image generation submodule is configured to, if a first size of the image area at the reference position is larger than a second size of the display area, sample the image area at the reference position according to a ratio of the first size to the second size to obtain an image of the second size as the first image;

[0050] A second image generating submodule, configured to extract the image region at the reference position from the original image as a first image if the first size is equal to the second size;

[0051] The third image generation submodule is used to expand the image area at the reference position in the original image to obtain the image area to be used if the first size is smaller than the second size; wherein the pixel coordinates of the vertices of the image area to be used in the original image are integers; interpolate the image area to be used according to the ratio of the second size to the first size to be used; and extract the image area of ​​the second size from the interpolation result obtained as the first image.

[0052] Optionally, the third image generation submodule is specifically used to, in the original image, round the pixel coordinates of the upper left corner vertex of the image area at the reference position toward the upper left corner, and round the pixel coordinates of the lower right corner vertex of the image area at the reference position toward the lower right corner, to obtain the image area to be used; calculate the product of the pixel coordinates of the upper left corner vertex of the image area at the reference position in the original image and the ratio to be used, to obtain the starting pixel coordinates; use the starting pixel coordinates as the pixel coordinates of the upper left corner vertex, extract the image area of ​​the second size from the obtained interpolation result, and obtain the first image.

[0053] Optionally, the device further comprises:

[0054] an image scaling module, configured to, in response to the adjustment operation on the image currently displayed in the display area, determine the position of the image area in the original image that needs to be displayed in the display area after the adjustment operation as a reference position, and when receiving a display instruction for the original image, scale the original image according to the size of the display area to obtain a second image;

[0055] The second image sending module is used to send the second image to the GPU, so that the GPU renders the second image in the display area.

[0056] Optionally, the device further comprises:

[0057] A graphic information acquisition module, used to acquire color information of a graphic marked on the original image, and a corresponding position of the graphic in the original image;

[0058] a graphics position determination module, configured to determine a graphics region to be rendered in the first image, and determine a position of the graphics region to be rendered in the first image, based on a relationship between a position corresponding to the graphics in the original image and the reference position, and a zoom ratio of the first image relative to the original image;

[0059] A graphics information sending module is used to send the position of the graphics area to be rendered in the first image and the color information of the graphics area to be rendered to the GPU, so that the GPU renders the graphics area to be rendered on the upper layer of the first image in the display area.

[0060] According to a fifth aspect of the embodiments of the present application, there is further provided an image rendering device, which is applied to a GPU, and the device includes:

[0061] A first image receiving module is used to receive a first image sent by the CPU; wherein the first image is: an image generated by the CPU having the same size as the display area and consistent with the screen content at a reference position in the original image; the reference position is: a position of an image area in the original image that needs to be displayed in the display area after the adjustment operation determined by the CPU in response to an adjustment operation on the image currently displayed in the display area; the image currently displayed in the display area is obtained based on the original image;

[0062] A first image rendering module is configured to render the first image in the display area.

[0063] Optionally, the device further comprises:

[0064] A second image receiving module is used to receive a second image sent by the CPU before receiving the first image sent by the CPU; wherein the second image is obtained by scaling the original image according to the size of the display area by the CPU when receiving a display instruction for the original image;

[0065] A second image rendering module is configured to render the second image in the display area.

[0066] Optionally, the device further comprises:

[0067] a graphics information receiving module, configured to receive the position of the graphics area to be rendered in the first image and the color information of the graphics area to be rendered sent by the CPU; wherein the graphics area to be rendered is determined by the CPU based on the relationship between the position corresponding to the graphics marked in the original image and the reference position in the original image, and the zoom ratio of the first image relative to the original image;

[0068] The graphics area rendering module is used to render the graphics area to be rendered on the upper layer of the first image in the display area according to the received position and color information.

[0069] Optionally, the first image rendering module is specifically configured to load an underlying canvas in the display area and render the first image in the underlying canvas;

[0070] The graphic area rendering module is specifically used to load a top-level canvas on the upper layer of the bottom-level canvas in the display area, and render the graphic area to be rendered in the top-level canvas according to the received position and color information.

[0071] Optionally, the device further comprises:

[0072] A canvas storage module, configured to load an underlying canvas in the display area, and after rendering the first image in the underlying canvas, store the underlying canvas on which the first image has been rendered in a video memory;

[0073] A canvas loading module is used to load the bottom canvas on which the first image has been rendered recorded in the video memory upon receiving the position of a new graphics area to be rendered in the first image and the color information of the new graphics area to be rendered sent by the CPU, and render the new graphics area to be rendered in the top canvas according to the received new position and color information.

[0074] In another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned image rendering methods applied to a CPU or any of the above-mentioned image rendering methods applied to a GPU is implemented.

[0075] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-described image rendering methods applied to a CPU or any of the above-described image rendering methods applied to a GPU.

[0076] Beneficial effects of the embodiments of the present application:

[0077] An image rendering method provided by an embodiment of the present application is applied to a CPU, and the CPU can respond to an adjustment operation on an image currently displayed in a display area, determine the position of an image area in an original image that needs to be displayed in the display area after the adjustment operation, as a reference position; wherein the image currently displayed in the display area is obtained based on the original image; generate an image with the same size as the display area and consistent with the picture content at the reference position in the original image to obtain a first image; send the first image to a GPU, so that the GPU renders the first image in the display area.

[0078] Based on the above processing, the image currently displayed in the display area can be obtained based on the original image. The user can trigger an adjustment operation for the image currently displayed in the display area to adjust the screen content displayed in the display area. The CPU can determine the position of the image area currently required to be displayed from the original image in response to the adjustment operation, and generate a first image. Then, the CPU can send the first image to the GPU, and accordingly, the GPU can render the received first image in the display area. In this way, during the entire process of image rendering, the CPU processes the original image to obtain the image area currently required to be displayed. The GPU only needs to render the received first image without processing the entire original image, and there is no need to store the entire original image in the video memory. It can avoid occupying more video memory space during the image rendering process, especially when the resolution of the original image is high, and it can also reduce the occupation of video memory space during the image rendering process, saving storage resources.

[0079] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0081] Figure 1 An interactive schematic diagram of an image rendering method provided in an embodiment of the present application;

[0082] Figure 2 A schematic diagram of determining a first image in an image rendering method provided in an embodiment of the present application;

[0083] Figure 3 A schematic diagram of a process of determining a first image by a CPU in an image rendering method provided in an embodiment of the present application;

[0084] Figure 4a A schematic diagram of an original image provided in an embodiment of the present application;

[0085] Figure 4b A schematic diagram of a grid-like interpolation result provided in an embodiment of the present application;

[0086] Figure 4c A schematic diagram of a grid form of a first image provided in an embodiment of the present application;

[0087] Figure 5 An interactive schematic diagram of another image rendering method provided in an embodiment of the present application;

[0088] Figure 6 A schematic diagram of a flow chart of an image rendering method provided in an embodiment of the present application;

[0089] Figure 7 A flowchart of another image rendering method provided in an embodiment of the present application;

[0090] Figure 8 A flowchart of another image rendering method provided in an embodiment of the present application;

[0091] Fig. 9 A schematic diagram of the structure of an image rendering device provided in an embodiment of the present application;

[0092] Fig.10 A schematic diagram of the structure of another image rendering device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0093] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0094] In the field of machine vision technology, it is often necessary to display images on a display device. For example, when an image needs to be displayed, the GPU needs to obtain the entire image and scale the entire image according to the size of the display area in the display device to render the scaled image in the display area. Subsequently, the user can adjust the screen content displayed in the display area to display different image areas in the image. In response to the adjustment operation triggered by the user, the GPU determines the image area that currently needs to be displayed from the entire image, and renders the determined image area in the display area. It can be seen that during the entire process of rendering the image, the GPU needs to store the entire image in the video memory, and the resolution of the entire image is often high, and the video memory occupancy is also high.

[0095] In order to reduce the usage of video memory during image rendering and save computing resources, an embodiment of the present application provides an image rendering method, which is applied to a display device, and the display device includes: a CPU and a GPU. Figure 1 , Figure 1 An interactive schematic diagram of an image rendering method provided in an embodiment of the present application. The image rendering method includes:

[0096] Step S101: In response to an adjustment operation on an image currently displayed in a display area, the CPU determines a position of an image area in an original image that needs to be displayed in the display area after the adjustment operation as a reference position.

[0097] The image currently displayed in the display area is obtained based on the original image.

[0098] Step S102: The CPU generates an image having the same size as the display area and consistent with the screen content at the reference position in the original image, thereby obtaining a first image.

[0099] Step S103: the CPU sends the first image to the GPU.

[0100] Step S104: the GPU receives the first image sent by the CPU.

[0101] Step S105: the GPU renders the first image in the display area.

[0102] Based on the above processing, the image currently displayed in the display area can be obtained based on the original image. The user can trigger an adjustment operation for the image currently displayed in the display area to adjust the screen content displayed in the display area. The CPU can determine the position of the image area currently required to be displayed from the original image in response to the adjustment operation, and generate a first image. Then, the CPU can send the first image to the GPU, and accordingly, the GPU can render the received first image in the display area. In this way, during the entire process of image rendering, the CPU processes the original image to obtain the image area currently required to be displayed. The GPU only needs to render the received first image without processing the entire original image, and there is no need to store the entire original image in the video memory. It can avoid occupying more video memory space during the image rendering process, especially when the resolution of the original image is high, and it can also reduce the occupation of video memory space during the image rendering process, saving storage resources.

[0103] For step S101, the image to which the screen content currently needs to be displayed in the display area belongs can be called the original image. For example, the scene involved in the present application can be a visual inspection scene, and an image acquisition device (such as a camera) can be used to acquire images of products on a production line, and the acquired images can be processed by an image processing algorithm to obtain an original image. For example, the acquired image contains a large amount of background, and the image processing algorithm can be used to determine the image area where the product is located, and crop the image area where the product is located from the acquired image to obtain the original image.

[0104] The image currently displayed in the display area may be obtained based on the original image. For example, the image currently displayed in the display area may be consistent with the picture content of the original image. Before step S101, the method further includes:

[0105] Step 1: When receiving a display instruction for an original image, the CPU scales the original image according to the size of the display area to obtain a second image.

[0106] Step 2: The CPU sends the second image to the GPU.

[0107] Step 3: The GPU receives the second image sent by the CPU.

[0108] Step 4: The GPU renders the second image in the display area.

[0109] When receiving a display instruction for the original image, that is, when the original image needs to be displayed, the CPU can scale the original image according to the size of the display area (i.e., the second size in the subsequent embodiments) to obtain a second image. For example, the CPU can calculate the ratio of the second size to the size of the original image to obtain a scaling factor of the second image relative to the original image, and scale the original image according to the obtained scaling factor to obtain the second image. The second image has the same screen content as the original image.

[0110] For example, if the size of the original image is larger than the second size, the scaling ratio of the second image calculated by the CPU relative to the original image is less than 1, and the CPU can sample the original image according to the calculated scaling ratio to obtain a second image of the second size. For example, if the calculated ratio is 0.2, one pixel is sampled for every five adjacent pixels in the original image to obtain a second image of the second size. If the size of the original image is equal to the second size, the scaling ratio of the second image calculated by the CPU relative to the original image is equal to 1, and the original image is the second image. If the size of the original image is smaller than the second size, the scaling ratio of the second image calculated by the CPU relative to the original image is greater than 1, and the CPU can interpolate the original image according to the calculated scaling ratio to obtain a second image of the second size. For example, if the calculated ratio is 2, every two adjacent pixels in the original image can be interpolated to obtain a second image of the second size.

[0111] And the CPU can send the second image to the GPU, and accordingly, the GPU can render the received second image in the display area, that is, the second image consistent with the screen content of the original image can be displayed in the display area. When receiving a display instruction for a new original image, the CPU can display an image consistent with the screen content of the new original image according to the above steps 1 to 4. That is, the display area can display the screen content of multiple different original images, which also involves multi-image switching. Based on the solution provided by the present application, for each original image, when it is necessary to display the screen content of the original image in the display area, the CPU can scale the original image. The GPU only needs to render the received image without processing the original image, and there is no need to store the original image in the video memory, that is, there is no need to load the original image into the video memory. Accordingly, in the process of switching to display the screen content of different original images, it is possible to avoid loading the original image to increase the time consumption and affect the efficiency of image switching, especially when the resolution of the original image is high. In this way, the efficiency of image switching can be improved, the delay of image switching can be reduced, and high frame rate and low delay of multi-image switching can be achieved.

[0112] When it is necessary to adjust the image content displayed in the display area to display different local image content in the same original image, the user can trigger an adjustment operation for the image currently displayed in the display area. For example, the adjustment operation may include a drag operation, a zoom-in operation, or a zoom-out operation. Accordingly, the CPU can determine, in response to the adjustment operation, the position of the image area (which may be referred to as the image area to be displayed) in the original image that needs to be displayed in the display area after the adjustment operation as a reference position. For example, the image area to be displayed may be a rectangular image area, and the CPU may determine the positions of the four vertices of the image area to be displayed as the reference position.

[0113] For steps S102 to S105, the CPU can extract the image area to be displayed from the original image according to the reference position, and can process the image area to be displayed to generate an image of a second size that is consistent with the screen content at the reference position in the original image (i.e., the screen content of the image area to be displayed), thereby obtaining the first image. The specific manner in which the CPU processes the image area to be displayed can refer to the relevant descriptions of steps 1 to 3 in the subsequent embodiments. The CPU can send the first image to the GPU, and accordingly, the GPU can render the first image in the display area.

[0114] In one embodiment, step S102 includes:

[0115] Step 1: If the first size of the image area at the reference position is larger than the second size of the display area, the image area at the reference position is sampled according to the ratio of the first size to the second size to obtain an image of the second size as the first image;

[0116] Step 2: If the first size is equal to the second size, extract the image area at the reference position from the original image as the first image;

[0117] Step 3: If the first size is smaller than the second size, the image area at the reference position in the original image is expanded to obtain the image area to be used; wherein the pixel coordinates of the vertices of the image area to be used in the original image are integers; the image area to be used is interpolated according to the ratio of the second size to the first size to be used; and the image area of ​​the second size is extracted from the interpolation result obtained as the first image.

[0118] In an embodiment of the present application, the size of the image area at the reference position in the original image (i.e., the image area to be displayed in the above embodiment) can be referred to as the first size. The size of the display area can be referred to as the second size. If the first size is larger than the second size, the CPU can sample the image area to be displayed according to the ratio of the first size to the second size to obtain an image of the second size as the first image. For example, if the ratio of the first size to the second size is 3, the CPU can sample every three adjacent pixels in the image area to be displayed to obtain one pixel, and obtain the first image of the second size.

[0119] If the first size is equal to the second size, the CPU can extract the image area to be displayed from the original image, that is, the image area to be displayed can be cut out from the original image as the first image.

[0120] If the first size is smaller than the second size, a pixel width in the image area to be displayed may correspond to multiple pixel widths in the display area, that is, a pixel width in the original image corresponding to a pixel width in the display area is not an integer, and a pixel width represents the interval between two pixels. Since the adjustment operation is for the image currently displayed in the display area, the screen content displayed in the display area moves a pixel width, and the corresponding pixel width moved in the original image is not an integer. Therefore, the pixel coordinates of the vertices of the image area to be displayed may not be integers in the original image, that is, the reference position is at the sub-pixel level in the original image. The CPU can only crop the image area at the physical pixel level from the original image. Therefore, the CPU needs to process the image area to be displayed to support sub-pixel level cropping.

[0121] The CPU can expand the image area to be displayed in the original image so that the pixel coordinates of the vertices of the expanded image area in the original image are integers, thereby obtaining the image area to be used. For example, the CPU can round the pixel coordinates of the upper left corner vertex of the image area to be displayed in the original image to the upper left corner, and round the pixel coordinates of the lower right corner vertex of the image area to be displayed in the lower right corner, thereby obtaining the image area to be used. For example, the pixel coordinates of the upper left corner vertex can be taken as the nearest integer pixel coordinates in the direction of the upper left corner, and the pixel coordinates of the lower right corner vertex can be taken as the nearest integer pixel coordinates in the direction of the lower right corner. In this way, it can be ensured that the expanded image area to be used includes the image area to be displayed, and further, it can be ensured that the position corresponding to the reference position can be determined from the interpolation result of the image area to be used later, and it can also be ensured that the first image can be determined from the interpolation result.

[0122] After obtaining the image area to be used, the CPU can interpolate the image area to be used according to the ratio of the second size to the first size (which can be called the ratio to be used) so that a pixel width in the obtained interpolation result can correspond to a pixel width in the display area. Then, the CPU can extract the image area of ​​the second size from the obtained interpolation result as the first image. For example, the CPU can calculate the product of the pixel coordinates of the upper left corner vertex of the image area to be displayed in the original image and the ratio to be used to obtain the starting pixel coordinates. Correspondingly, the starting pixel coordinates can represent the pixel coordinates corresponding to the upper left corner vertex of the display area in the obtained interpolation result. With the starting pixel coordinates as the pixel coordinates of the upper left corner vertex, the CPU can extract the image area of ​​the second size from the obtained interpolation result to obtain the first image. Alternatively, the CPU can also calculate the product of the pixel coordinates of the upper right corner vertex of the image area to be displayed in the original image and the ratio to be used to obtain the starting pixel coordinates, and with the starting pixel coordinates as the pixel coordinates of the upper right corner vertex, extract the image area of ​​the second size from the obtained interpolation result to obtain the first image. In this way, the CPU can determine the position corresponding to the reference position from the interpolation result of the image area to be used, ensuring that the first image can be determined from the interpolation result.

[0123] For example, the first size is 50×50, the second size is 100×100, and one pixel width in the display area corresponds to 0.5 pixel width in the image area to be displayed. In the original image, the pixel coordinates of the upper left corner vertex of the image area to be displayed can be (1.5, 1.5), and the pixel coordinates of the lower right corner vertex can be (51.5, 51.5). Then, after rounding, the pixel coordinates of the upper left corner vertex of the image area to be used can be (1, 1), and the pixel coordinates of the lower right corner vertex can be (52, 52). The ratio to be used is 2, and the image area to be used is interpolated according to the ratio to be used, such as inserting a new pixel point between every two adjacent pixels in the image area to be used. The starting pixel coordinates can be the product of (1.5, 1.5) and 2, that is, (3, 3). Take (3, 3) as the pixel coordinates of the upper left corner vertex, extract the 100×100 image area from the interpolation result, and obtain the first image.

[0124] See also Figure 2 , Figure 2 A schematic diagram of determining a first image in an image rendering method provided in an embodiment of the present application. The CPU can determine an image area 211 to be displayed from an original image 21, and process the image area 211 to be displayed to obtain a first image 22 of a second size. For the specific processing process, please refer to the relevant description of step three.

[0125] Based on the above processing, the CPU can generate the first image based on the size relationship between the first size and the second size, based on the original image and the image area to be displayed. In this way, it is further ensured that the CPU can generate the first image. Accordingly, the GPU only needs to render the received first image without processing the entire original image, and there is no need to store the entire original image in the video memory. It can avoid occupying more video memory space during the image rendering process, especially when the resolution of the original image is high, which can reduce the occupation of video memory space during the image rendering process and save storage resources.

[0126] In one embodiment, see Figure 3 , Figure 3 A schematic diagram of a flow chart of a CPU determining a first image in an image rendering method provided in an embodiment of the present application. The flow chart includes:

[0127] Step S301: Determine the original image and result display size. That is, the CPU determines the first size and the second size.

[0128] Step S302: Determine the zoom ratio and zoom center. That is, the CPU determines the ratio to be used in the above embodiment and the pixel coordinates of the top left corner of the image area to be displayed.

[0129] Step S303: the zoom center is shifted toward the upper left corner. That is, if the first size is smaller than the second size, the CPU may round the pixel coordinates of the upper left corner vertex of the image area to be displayed in the original image toward the upper left corner.

[0130] Step S304: Calculate the intermediate image. That is, the CPU interpolates the image area to be used according to the ratio to be used.

[0131] Step S305: Calculate the offset difference according to the zoom center floating point and integer. That is, the CPU calculates the starting pixel coordinates in the above embodiment.

[0132] Step S306: offset the result image. That is, the CPU uses the starting pixel coordinates as the pixel coordinates of the upper left corner, extracts the image area of ​​the second size from the interpolation result, and obtains the first image.

[0133] In one embodiment, see Figures 4a to 4c , Figure 4a A schematic diagram of an original image provided in an embodiment of the present application. The original image is an image of the surface of the product to be inspected in the visual inspection scene. The image currently displayed in the display area is obtained based on the original image. Through step S101 and step 3 in the above embodiment, the CPU can determine Figure 4a The reference positions in the original image are shown and can be Figure 4aThe image area at the reference position in the original image shown in FIG. 4 is expanded to obtain the image area to be used 401. Then, the CPU can interpolate the image area to be used 401 according to the ratio to be used. In order to facilitate the description of the pixel points included in the interpolation result, the interpolation result can be represented in the form of a grid, such as Figure 4b shown.

[0134] Figure 4b A schematic diagram of a grid-like interpolation result provided in an embodiment of the present application. Figure 4b Contains 6×4, a total of 24 grids. Each grid represents a pixel point in the interpolation result. The CPU can extract the image area of ​​the second size from the interpolation result as the first image. The first image can also be represented in the form of a grid, such as Figure 4c shown.

[0135] Figure 4c A schematic diagram of a grid format of a first image provided in an embodiment of the present application. Figure 4c It contains 6×3, a total of 18 complete grids, and 22 incomplete grids on the edges. The complete grids represent physical pixels, and the incomplete grids represent sub-pixels. That is, Figure 4c The pixel coordinates of the vertices of the represented first image are not integers but are at a sub-pixel level. It can be seen that based on the solution provided by the present application, the CPU can support sub-pixel level cropping.

[0136] In one embodiment, see Figure 5 , Figure 5 An interactive schematic diagram of another image rendering method provided in an embodiment of the present application, the method further includes:

[0137] Step S501: The CPU obtains the color information of the graphic marked on the original image, and the corresponding position of the graphic in the original image.

[0138] Step S502: The CPU determines the graphic area to be rendered in the first image and the position of the graphic area to be rendered in the first image based on the relationship between the corresponding position of the graphic in the original image and the reference position, and the zoom ratio of the first image relative to the original image.

[0139] Step S503: the CPU sends the position of the graphics area to be rendered in the first image and the color information of the graphics area to be rendered to the GPU.

[0140] Step S504: the GPU receives the position of the graphics area to be rendered in the first image and the color information of the graphics area to be rendered sent by the CPU.

[0141] Step S505: The GPU renders the graphics area to be rendered in the upper layer of the first image in the display area according to the received position and color information.

[0142] In an embodiment of the present application, the original image may be marked in advance to obtain a graphic corresponding to the original image, and the color information of the graphic and the corresponding position of the graphic in the original image, that is, the position of the graphic relative to the original image, may be stored in a preset storage location (e.g., memory). The position of the graphic relative to the original image may be represented by the pixel coordinates of the graphic in the original image, and the pixel coordinates of the graphic in the original image may not be integers. For example, the original image may be processed in advance using a 2D (two-dimensional) image detection algorithm to obtain a graphic marked for the original image. For example, the 2D image detection algorithm may be a target detection algorithm, and accordingly, a detection box of a specified object in the original image may be obtained as a graphic marked for the original image; or, the 2D image detection algorithm may also be an edge detection algorithm, and accordingly, an edge contour of an object (e.g., a preset workpiece) contained in the original image may be obtained as a graphic marked for the original image.

[0143] The display area can display the image content in the original image, and can display the graphics corresponding to the original image. The CPU can obtain the color information of the graphics marked for the original image, and the corresponding position of the graphics in the original image, which are stored in advance. Correspondingly, the CPU can determine the graphic area (which can be called the graphics area to be processed) that belongs to the image area to be displayed at the corresponding position based on the relationship between the corresponding position of the graphics in the original image and the reference position, that is, the relative relationship between the corresponding position of the graphics in the original image and the position of the image area to be displayed in the original image. And the scaling ratio of the graphics area to be rendered relative to the original image area can be determined according to the scaling ratio of the first image relative to the original image, so as to scale the graphics area to be processed according to the determined scaling ratio to obtain the graphics area to be rendered. And combined with the position of the corresponding position of the graphics area to be processed in the image area to be displayed, the position of the graphics area to be rendered in the first image can be determined.

[0144] The graphics area to be rendered may be obtained by scaling the complete graphics marked for the original image, or may be obtained by scaling a part of the graphics marked for the original image. And the CPU may determine the color information of the graphics area to be rendered from the color information of the graphics marked for the original image obtained. After determining the position of the graphics area to be rendered in the first image, the CPU may send the position of the graphics area to be rendered in the first image and the color information of the graphics area to be rendered to the GPU. Accordingly, when receiving the position of the graphics area to be rendered in the first image and the color information of the graphics area to be rendered sent by the CPU, the GPU may render the graphics area to be rendered on the upper layer of the first image in the display area.

[0145] Based on the above processing, it is possible to display the image content of the original image in the display area while displaying the graphics marked for the original image, thereby ensuring the display effect of the display area for the original image and improving the user experience of browsing the displayed content in the display area.

[0146] In one embodiment, see Figure 6 , Figure 6 A flowchart of an image rendering method provided in an embodiment of the present application.

[0147] Step S601: Start.

[0148] Step S602: original image. That is, the CPU obtains the original image.

[0149] Step S603: scaling the image in its entirety, i.e., steps 1 to 4 in the above embodiment.

[0150] Step S604: drawing graphics. That is, the CPU obtains the color information of the graphics marked for the original image and the corresponding position of the graphics in the original image, and determines the position of the graphics displayed in the display area according to the ratio of the size of the original image to the size of the display area. The CPU sends the determined position and color information of the graphics to the GPU. The GPU renders the graphics on the upper layer of the image displayed in the display area according to the received position and color information.

[0151] Step S605: moving the image, that is, steps S101 to S104 in the above embodiment.

[0152] Step S606: interface display, that is, step S105 in the above embodiment and steps S502 to S505 in the above embodiment.

[0153] Step S607: End.

[0154] In one embodiment, step S105 includes: the GPU loads the underlying canvas in the display area, and renders the first image in the underlying canvas;

[0155] Step S505 includes: the GPU loads a top-level canvas on the upper layer of the bottom-level canvas in the display area, and renders the graphics area to be rendered in the top-level canvas according to the received position and color information.

[0156] In an embodiment of the present application, the GPU may load a bottom canvas in the display area and render the first image in the bottom canvas. The GPU may load a top canvas on the upper layer of the bottom canvas in the display area and render the graphics area to be rendered in the top canvas according to the received position and color information.

[0157] Based on the above processing, the images and graphics to be displayed can be displayed on the bottom canvas and top canvas of the display area respectively, further ensuring that the image content of the original image can be displayed in the display area while displaying the graphics marked for the original image, so as to ensure the display effect of the display area for the original image and improve the user's experience of browsing the display content in the display area. It can also avoid rendering images and graphics in the same canvas, resulting in the rendered graphics polluting the canvas.

[0158] In one embodiment, see Figure 7 , Figure 7 A flowchart of another image rendering method provided in an embodiment of the present application. The method includes:

[0159] Step S701: original image. That is, the CPU obtains the original image.

[0160] Step S702: cropping. That is, the CPU extracts the image area that needs to be displayed in the display area from the original image.

[0161] Step S703: Integer scaling. That is, the CPU generates an image that is the same size as the display area and consistent with the screen content at the reference position in the original image, obtains the first image, and sends the first image to the GPU.

[0162] Step S704: bottom canvas. That is, when receiving the first image, the GPU loads the bottom canvas in the display area and renders the first image in the bottom canvas.

[0163] Step S705: Shifting the image. That is, if the user triggers an adjustment operation for the image currently displayed in the display area, steps S101 to S105 are executed.

[0164] Step S706: top canvas. That is, step S501 to step S504, and the GPU loads the top canvas on the upper layer of the bottom canvas in the display area, and renders the graphics area to be rendered in the top canvas according to the received position and color information.

[0165] Step S707: Displaying the image. That is, the image and graphics currently required to be displayed are displayed in the display area.

[0166] In one embodiment, the image rendering method further includes: the GPU stores the bottom canvas of the rendered first image in the video memory. When receiving the position of the new graphics area to be rendered in the first image and the color information of the new graphics area to be rendered sent by the CPU, the GPU loads the bottom canvas of the rendered first image recorded in the video memory, and renders the new graphics area to be rendered in the top canvas according to the received new position and color information.

[0167] In an embodiment of the present application, the GPU may store the underlying canvas of the rendered first image in the video memory. When the graphics in the original image need to be redrawn, for example, when the detection box in the original image needs to be re-marked, the CPU may obtain the color information of the new graphics marked for the original image, and the corresponding position of the new graphics in the original image.

[0168] If the image content to be displayed in the display area changes, the CPU needs to re-determine the new first image according to steps S101 to S103 in the above embodiment, and send the new first image to the GPU. Accordingly, the GPU can render the new first image in the bottom canvas. And the CPU needs to redetermine the graphics area to be rendered in the new first image based on the relationship between the corresponding position of the new graphics in the original image and the reference position, and the zoom ratio of the new first image relative to the original image, and redetermine the position of the graphics area to be rendered in the new first image. And the position of the new graphics area to be rendered in the first image and the color information of the new graphics area to be rendered can be sent to the GPU. Accordingly, the GPU can render the new graphics area to be rendered in the top canvas according to the received new position and color information.

[0169] If the image content to be displayed in the display area has not changed, that is, the first image remains unchanged, the CPU only needs to redetermine the new graphic area to be rendered, and redetermine the position of the new graphic area to be rendered in the first image. And the position of the new graphic area to be rendered in the first image, as well as the color information of the new graphic area to be rendered can be sent to the GPU. Correspondingly, the GPU can load the underlying canvas of the rendered first image recorded in the video memory, and render the new graphic area to be rendered in the top canvas according to the received new position and color information. In this way, the CPU does not need to recalculate the first image, and the GPU does not need to re-render the first image in the underlying canvas, that is, there is no need to redo the entire process of drawing the display area. The consumption of computing resources and the time spent on rendering the content of the picture displayed in the display area can be reduced, and the rendering efficiency of the display area can be improved.

[0170] In one embodiment, see Figure 8 , Figure 8 A flowchart of another image rendering method provided in an embodiment of the present application. The method includes:

[0171] Step S801: Start.

[0172] Step S802: whether to move or scale the image. If yes, execute step S803; if no, execute step S804. That is, whether the image content to be displayed in the display area changes.

[0173] Step S803: Recalculate. That is, if the image content to be displayed in the display area changes, the CPU needs to re-determine the new first image according to steps S101 to S103 in the above embodiment, and send the new first image to the GPU. Accordingly, the GPU can render the new first image in the bottom canvas. And the CPU needs to re-determine the position of the new graphics area to be rendered in the new first image, and send the position of the new graphics area to be rendered in the first image and the color information of the new graphics area to be rendered to the GPU. Accordingly, the GPU can render the new graphics area to be rendered in the top canvas according to the received new position and color information.

[0174] Step S804: copy the intermediate image. That is, if the image content to be displayed in the display area does not change, that is, the first image does not change, the GPU can load the underlying canvas of the rendered first image recorded in the video memory.

[0175] Step S805: Draw graphics. That is, if the image content to be displayed in the display area has not changed, that is, the first image remains unchanged, the CPU only needs to redetermine the new graphics area to be rendered and redetermine the position of the new graphics area to be rendered in the first image. The position of the new graphics area to be rendered in the first image and the color information of the new graphics area to be rendered can be sent to the GPU. Accordingly, the GPU can render the new graphics area to be rendered in the top-level canvas according to the received new position and color information.

[0176] Step S806: interface display, that is, completing the display of the images and graphics currently required to be displayed in the display area.

[0177] Step S807: End.

[0178] In the embodiment of the present application, an intermediate buffer layer is added, that is, the underlying canvas of the rendered first image is stored in the video memory. In this way, in the scenario of the visual application platform, for the situation where the graphics position often moves but the first image does not change, it is only necessary to re-render the new graphics area to be rendered, without frequently performing the entire rendering process to redraw the graphics, which greatly improves the redrawing efficiency and the rendering efficiency of the display area.

[0179] The subsequent image rendering method applied to the CPU and the image rendering method applied to the GPU may refer to the relevant description of the image rendering method applied to the display device in the above embodiment.

[0180] Based on the same inventive concept, the embodiment of the present application further provides an image rendering method, which is applied to a CPU, and the method includes:

[0181] In response to an adjustment operation on an image currently displayed in a display area, determining a position of an image area in an original image that needs to be displayed in the display area after the adjustment operation as a reference position; wherein the image currently displayed in the display area is obtained based on the original image;

[0182] Generate an image having the same size as the display area and consistent with the picture content at the reference position in the original image to obtain a first image;

[0183] The first image is sent to a GPU so that the GPU renders the first image in the display area.

[0184] Based on the above processing, the image currently displayed in the display area can be obtained based on the original image. The user can trigger an adjustment operation for the image currently displayed in the display area to adjust the screen content displayed in the display area. The CPU can determine the position of the image area currently required to be displayed from the original image in response to the adjustment operation, and generate a first image. Then, the CPU can send the first image to the GPU, and accordingly, the GPU can render the received first image in the display area. In this way, during the entire process of image rendering, the CPU processes the original image to obtain the image area currently required to be displayed. The GPU only needs to render the received first image without processing the entire original image, and there is no need to store the entire original image in the video memory. It can avoid occupying more video memory space during the image rendering process, especially when the resolution of the original image is high, and it can also reduce the occupation of video memory space during the image rendering process, saving storage resources.

[0185] In one embodiment, generating an image having the same size as the display area and consistent with the screen content at the reference position in the original image to obtain a first image includes:

[0186] If a first size of the image area at the reference position is larger than a second size of the display area, sampling the image area at the reference position according to a ratio of the first size to the second size to obtain an image of the second size as the first image;

[0187] If the first size is equal to the second size, extracting the image area at the reference position from the original image as the first image;

[0188] If the first size is smaller than the second size, the image area at the reference position is expanded in the original image to obtain an image area to be used; wherein the pixel coordinates of the vertices of the image area to be used in the original image are integers; the image area to be used is interpolated according to the ratio of the second size to the first size to be used; and the image area of ​​the second size is extracted from the interpolation result obtained as the first image.

[0189] In one embodiment, the image area at the reference position is expanded in the original image to obtain the image area to be used, including:

[0190] In the original image, the pixel coordinates of the upper left corner vertex of the image area at the reference position are rounded toward the upper left corner, and the pixel coordinates of the lower right corner vertex of the image area at the reference position are rounded toward the lower right corner, to obtain the image area to be used;

[0191] The step of extracting the image region of the second size from the obtained interpolation result as the first image includes:

[0192] Calculate the product of the pixel coordinates of the top left corner of the image area at the reference position in the original image and the ratio to be used to obtain the starting pixel coordinates;

[0193] The starting pixel coordinates are used as the pixel coordinates of the upper left corner, and the image area of ​​the second size is extracted from the obtained interpolation result to obtain the first image.

[0194] In one embodiment, before determining, in response to the adjustment operation on the image currently displayed in the display area, the position of the image area in the original image that needs to be displayed in the display area after the adjustment operation as the reference position, the method further includes:

[0195] When receiving a display instruction for the original image, scaling the original image according to the size of the display area to obtain a second image;

[0196] The second image is sent to the GPU so that the GPU renders the second image in the display area.

[0197] In one embodiment, the method further comprises:

[0198] Acquire color information of a graphic marked on the original image, and a corresponding position of the graphic in the original image;

[0199] Based on the relationship between the position corresponding to the graphic in the original image and the reference position, and the zoom ratio of the first image relative to the original image, determine a graphic region to be rendered in the first image, and determine the position of the graphic region to be rendered in the first image;

[0200] The position of the graphics area to be rendered in the first image and the color information of the graphics area to be rendered are sent to the GPU, so that the GPU renders the graphics area to be rendered on an upper layer of the first image in the display area.

[0201] Based on the same inventive concept, the embodiment of the present application further provides an image rendering method, which is applied to a GPU, and the method includes:

[0202] Receive a first image sent by the CPU; wherein the first image is: an image generated by the CPU having the same size as the display area and having the same screen content as that at a reference position in the original image; the reference position is: a position of an image area in the original image that needs to be displayed in the display area after the adjustment operation determined by the CPU in response to an adjustment operation on the image currently displayed in the display area; the image currently displayed in the display area is obtained based on the original image;

[0203] The first image is rendered in the display area.

[0204] Based on the above processing, the image currently displayed in the display area can be obtained based on the original image. The user can trigger an adjustment operation for the image currently displayed in the display area to adjust the screen content displayed in the display area. The CPU can determine the position of the image area currently required to be displayed from the original image in response to the adjustment operation, and generate a first image. Then, the CPU can send the first image to the GPU, and accordingly, the GPU can render the received first image in the display area. In this way, during the entire process of image rendering, the CPU processes the original image to obtain the image area currently required to be displayed. The GPU only needs to render the received first image without processing the entire original image, and there is no need to store the entire original image in the video memory. It can avoid occupying more video memory space during the image rendering process, especially when the resolution of the original image is high, and it can also reduce the occupation of video memory space during the image rendering process, saving storage resources.

[0205] In one embodiment, before receiving the first image sent by the CPU, the method further includes:

[0206] receiving a second image sent by the CPU; wherein the second image is obtained by scaling the original image by the CPU according to the size of the display area when receiving a display instruction for the original image;

[0207] The second image is rendered in the display area.

[0208] In one embodiment, the method further comprises:

[0209] receiving the position of the graphics area to be rendered in the first image and the color information of the graphics area to be rendered sent by the CPU; wherein the graphics area to be rendered is determined by the CPU based on the relationship between the position corresponding to the graphics marked in the original image and the reference position in the original image, and the zoom ratio of the first image relative to the original image;

[0210] The to-be-rendered graphic area is rendered on an upper layer of the first image in the display area according to the received position and color information.

[0211] In one embodiment, rendering the first image in the display area includes:

[0212] Loading an underlying canvas in the display area, and rendering the first image in the underlying canvas;

[0213] Rendering the graphics area to be rendered on an upper layer of the first image in the display area according to the received position and color information includes:

[0214] A top-level canvas is loaded on the upper layer of the bottom-level canvas in the display area, and the graphics area to be rendered is rendered in the top-level canvas according to the received position and color information.

[0215] In one embodiment, after loading the underlying canvas in the display area and rendering the first image in the underlying canvas, the method further includes:

[0216] storing the bottom canvas on which the first image has been rendered in a video memory;

[0217] When the position of the new graphics area to be rendered in the first image and the color information of the new graphics area to be rendered sent by the CPU are received, the bottom canvas on which the first image has been rendered and recorded in the video memory is loaded, and the new graphics area to be rendered is rendered in the top canvas according to the received new position and color information.

[0218] Based on the same inventive concept, an embodiment of the present application further provides a display device, including:

[0219] A CPU, used to execute any one of the image rendering methods applied to a CPU in the above embodiments;

[0220] The GPU is used to execute any image rendering method applied to the GPU in the above embodiments.

[0221] Based on the display device provided by the embodiment of the present application, the image currently displayed in the display area can be obtained based on the original image. The user can trigger an adjustment operation for the image currently displayed in the display area to adjust the screen content displayed in the display area. The CPU can determine the position of the image area currently required to be displayed from the original image in response to the adjustment operation, and generate a first image. Then, the CPU can send the first image to the GPU, and accordingly, the GPU can render the received first image in the display area. In this way, during the entire process of image rendering, the CPU processes the original image to obtain the image area currently required to be displayed. The GPU only needs to render the received first image without processing the entire original image, and there is no need to store the entire original image in the video memory. It can avoid occupying more video memory space during the image rendering process, especially when the resolution of the original image is high, it can also reduce the occupation of the video memory space during the image rendering process, saving storage resources.

[0222] Based on the same inventive concept, the present application also provides an image rendering device, which is applied to a CPU, see Fig. 9 , Fig. 9 A schematic diagram of the structure of an image rendering device provided in an embodiment of the present application, the device comprising:

[0223] A reference position determination module 901 is used to determine, in response to an adjustment operation on an image currently displayed in a display area, a position of an image area in an original image that needs to be displayed in the display area after the adjustment operation as a reference position; wherein the image currently displayed in the display area is obtained based on the original image;

[0224] An image generation module 902 is used to generate an image having the same size as the display area and consistent with the picture content at the reference position in the original image, to obtain a first image;

[0225] The first image sending module 903 is configured to send the first image to the GPU, so that the GPU renders the first image in the display area.

[0226] Based on the image rendering device provided by the embodiment of the present application, the image currently displayed in the display area can be obtained based on the original image. The user can trigger an adjustment operation for the image currently displayed in the display area to adjust the screen content displayed in the display area. The CPU can determine the position of the image area currently required to be displayed from the original image in response to the adjustment operation, and generate a first image. Then, the CPU can send the first image to the GPU, and accordingly, the GPU can render the received first image in the display area. In this way, during the entire process of image rendering, the CPU processes the original image to obtain the image area currently required to be displayed. The GPU only needs to render the received first image without processing the entire original image, and there is no need to store the entire original image in the video memory. It can avoid occupying more video memory space during the image rendering process, especially when the resolution of the original image is high, it can also reduce the occupation of video memory space during the image rendering process, saving storage resources.

[0227] In one embodiment, the image generation module 902 includes:

[0228] A first image generation submodule is configured to, if a first size of the image area at the reference position is larger than a second size of the display area, sample the image area at the reference position according to a ratio of the first size to the second size to obtain an image of the second size as the first image;

[0229] A second image generating submodule, configured to extract the image region at the reference position from the original image as a first image if the first size is equal to the second size;

[0230] The third image generation submodule is used to expand the image area at the reference position in the original image to obtain the image area to be used if the first size is smaller than the second size; wherein the pixel coordinates of the vertices of the image area to be used in the original image are integers; interpolate the image area to be used according to the ratio of the second size to the first size to be used; and extract the image area of ​​the second size from the interpolation result obtained as the first image.

[0231] In one embodiment, the third image generation submodule is specifically used to, in the original image, round the pixel coordinates of the upper left corner vertex of the image area at the reference position toward the upper left corner, and round the pixel coordinates of the lower right corner vertex of the image area at the reference position toward the lower right corner, to obtain the image area to be used; calculate the product of the pixel coordinates of the upper left corner vertex of the image area at the reference position in the original image and the ratio to be used to obtain the starting pixel coordinates; use the starting pixel coordinates as the pixel coordinates of the upper left corner vertex, extract the image area of ​​the second size from the obtained interpolation result, and obtain the first image.

[0232] In one embodiment, the apparatus further comprises:

[0233] an image scaling module, configured to, in response to the adjustment operation on the image currently displayed in the display area, determine the position of the image area in the original image that needs to be displayed in the display area after the adjustment operation as a reference position, and when receiving a display instruction for the original image, scale the original image according to the size of the display area to obtain a second image;

[0234] The second image sending module is used to send the second image to the GPU, so that the GPU renders the second image in the display area.

[0235] In one embodiment, the apparatus further comprises:

[0236] A graphic information acquisition module, used to acquire color information of a graphic marked on the original image, and a corresponding position of the graphic in the original image;

[0237] a graphics position determination module, configured to determine a graphics region to be rendered in the first image, and determine a position of the graphics region to be rendered in the first image, based on a relationship between a position corresponding to the graphics in the original image and the reference position, and a zoom ratio of the first image relative to the original image;

[0238] A graphics information sending module is used to send the position of the graphics area to be rendered in the first image and the color information of the graphics area to be rendered to the GPU, so that the GPU renders the graphics area to be rendered on the upper layer of the first image in the display area.

[0239] Based on the same inventive concept, the present application also provides an image rendering device, which is applied to a GPU. Fig.10 , Fig.10 A schematic diagram of the structure of another image rendering device provided in an embodiment of the present application, the device comprising:

[0240] The first image receiving module 1001 is used to receive a first image sent by the CPU; wherein the first image is: an image generated by the CPU having the same size as the display area and having the same screen content as the reference position in the original image; the reference position is: the position of the image area in the original image that needs to be displayed in the display area after the adjustment operation determined by the CPU in response to the adjustment operation on the image currently displayed in the display area; the image currently displayed in the display area is obtained based on the original image;

[0241] The first image rendering module 1002 is configured to render the first image in the display area.

[0242] Based on the image rendering device provided by the embodiment of the present application, the image currently displayed in the display area can be obtained based on the original image. The user can trigger an adjustment operation for the image currently displayed in the display area to adjust the screen content displayed in the display area. The CPU can determine the position of the image area currently required to be displayed from the original image in response to the adjustment operation, and generate a first image. Then, the CPU can send the first image to the GPU, and accordingly, the GPU can render the received first image in the display area. In this way, during the entire process of image rendering, the CPU processes the original image to obtain the image area currently required to be displayed. The GPU only needs to render the received first image without processing the entire original image, and there is no need to store the entire original image in the video memory. It can avoid occupying more video memory space during the image rendering process, especially when the resolution of the original image is high, it can also reduce the occupation of video memory space during the image rendering process, saving storage resources.

[0243] In one embodiment, the apparatus further comprises:

[0244] A second image receiving module is used to receive a second image sent by the CPU before receiving the first image sent by the CPU; wherein the second image is obtained by scaling the original image according to the size of the display area by the CPU when receiving a display instruction for the original image;

[0245] A second image rendering module is configured to render the second image in the display area.

[0246] In one embodiment, the apparatus further comprises:

[0247] a graphics information receiving module, configured to receive the position of the graphics area to be rendered in the first image and the color information of the graphics area to be rendered sent by the CPU; wherein the graphics area to be rendered is determined by the CPU based on the relationship between the position corresponding to the graphics marked in the original image and the reference position in the original image, and the zoom ratio of the first image relative to the original image;

[0248] The graphics area rendering module is used to render the graphics area to be rendered on the upper layer of the first image in the display area according to the received position and color information.

[0249] In one embodiment, the first image rendering module 1002 is specifically configured to load an underlying canvas in the display area and render the first image in the underlying canvas;

[0250] The graphic area rendering module is specifically used to load a top-level canvas on the upper layer of the bottom-level canvas in the display area, and render the graphic area to be rendered in the top-level canvas according to the received position and color information.

[0251] In one embodiment, the apparatus further comprises:

[0252] A canvas storage module, configured to load an underlying canvas in the display area, and after rendering the first image in the underlying canvas, store the underlying canvas on which the first image has been rendered in a video memory;

[0253] A canvas loading module is used to load the bottom canvas on which the first image has been rendered recorded in the video memory upon receiving the position of a new graphics area to be rendered in the first image and the color information of the new graphics area to be rendered sent by the CPU, and render the new graphics area to be rendered in the top canvas according to the received new position and color information.

[0254] In another embodiment provided in the present application, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned image rendering methods applied to a CPU or any of the above-mentioned image rendering methods applied to a GPU are implemented.

[0255] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any image rendering method applied to a CPU or any image rendering method applied to a GPU in the above embodiments.

[0256] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0257] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0258] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the image rendering method applied to the CPU, the image rendering method applied to the GPU, the display device, the apparatus, the computer-readable storage medium, and the computer program product embodiment, since they are basically similar to the image rendering method embodiment applied to the display device, the description is relatively simple, and the relevant parts can be referred to the partial description of the image rendering method embodiment applied to the display device.

[0259] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. An image rendering method, characterized in that: Applied to a central processing unit (CPU), the method comprises: In response to an adjustment operation on an image currently displayed in a display area, determining a position of an image area in an original image that needs to be displayed in the display area after the adjustment operation as a reference position; wherein the image currently displayed in the display area is obtained based on the original image; Generate an image having the same size as the display area and consistent with the picture content at the reference position in the original image to obtain a first image; The first image is sent to a graphics processor (GPU), so that the GPU renders the first image in the display area.

2. The method according to claim 1, characterized in that Generating an image having the same size as the display area and having the same content as the picture at the reference position in the original image to obtain a first image, comprising: If a first size of the image area at the reference position is larger than a second size of the display area, sampling the image area at the reference position according to a ratio of the first size to the second size to obtain an image of the second size as the first image; If the first size is equal to the second size, extracting the image area at the reference position from the original image as the first image; If the first size is smaller than the second size, the image area at the reference position is expanded in the original image to obtain an image area to be used; wherein the pixel coordinates of the vertices of the image area to be used in the original image are integers; the image area to be used is interpolated according to the ratio of the second size to the first size to be used; and the image area of ​​the second size is extracted from the interpolation result obtained as the first image.

3. The method according to claim 2, characterized in that Expanding the image area at the reference position in the original image to obtain the image area to be used includes: In the original image, the pixel coordinates of the upper left corner vertex of the image area at the reference position are rounded toward the upper left corner, and the pixel coordinates of the lower right corner vertex of the image area at the reference position are rounded toward the lower right corner, to obtain the image area to be used; The step of extracting the image region of the second size from the obtained interpolation result as the first image includes: Calculate the product of the pixel coordinates of the top left corner of the image area at the reference position in the original image and the ratio to be used to obtain the starting pixel coordinates; The starting pixel coordinates are used as the pixel coordinates of the upper left corner, and the image area of ​​the second size is extracted from the obtained interpolation result to obtain the first image.

4. The method according to claim 1, characterized in that: Before determining, in response to the adjustment operation on the image currently displayed in the display area, the position of the image area in the original image that needs to be displayed in the display area after the adjustment operation as the reference position, the method further includes: When receiving a display instruction for the original image, scaling the original image according to the size of the display area to obtain a second image; The second image is sent to the GPU so that the GPU renders the second image in the display area.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Acquire color information of a graphic marked on the original image, and a corresponding position of the graphic in the original image; Based on the relationship between the position corresponding to the graphic in the original image and the reference position, and the zoom ratio of the first image relative to the original image, determine a graphic region to be rendered in the first image, and determine the position of the graphic region to be rendered in the first image; The position of the graphics area to be rendered in the first image and the color information of the graphics area to be rendered are sent to the GPU, so that the GPU renders the graphics area to be rendered on an upper layer of the first image in the display area.

6. An image rendering method, characterized in that: Applied to a GPU, the method comprises: Receive a first image sent by the CPU; wherein the first image is: an image generated by the CPU having the same size as the display area and having the same screen content as that at a reference position in the original image; the reference position is: a position of an image area in the original image that needs to be displayed in the display area after the adjustment operation determined by the CPU in response to an adjustment operation on the image currently displayed in the display area; the image currently displayed in the display area is obtained based on the original image; The first image is rendered in the display area.

7. The method according to claim 6, characterized in that Before receiving the first image sent by the CPU, the method further includes: receiving a second image sent by the CPU; wherein the second image is obtained by scaling the original image by the CPU according to the size of the display area when receiving a display instruction for the original image; The second image is rendered in the display area.

8. The method according to claim 6 or 7, characterized in that: The method further comprises: receiving the position of the graphics area to be rendered in the first image and the color information of the graphics area to be rendered sent by the CPU; wherein the graphics area to be rendered is determined by the CPU based on the relationship between the position corresponding to the graphics marked in the original image and the reference position in the original image, and the zoom ratio of the first image relative to the original image; The to-be-rendered graphic area is rendered on an upper layer of the first image in the display area according to the received position and color information.

9. The method according to claim 8, characterized in that Rendering the first image in the display area includes: Loading an underlying canvas in the display area, and rendering the first image in the underlying canvas; Rendering the graphics area to be rendered on an upper layer of the first image in the display area according to the received position and color information includes: A top-level canvas is loaded on the upper layer of the bottom-level canvas in the display area, and the graphics area to be rendered is rendered in the top-level canvas according to the received position and color information.

10. The method according to claim 9, characterized in that After loading the underlying canvas in the display area and rendering the first image in the underlying canvas, the method further includes: storing the bottom canvas on which the first image has been rendered in a video memory; When the position of the new graphics area to be rendered in the first image and the color information of the new graphics area to be rendered sent by the CPU are received, the bottom canvas on which the first image has been rendered and recorded in the video memory is loaded, and the new graphics area to be rendered is rendered in the top canvas according to the received new position and color information.

11. A display device, characterized in that: include: A CPU, configured to execute the method according to any one of claims 1 to 5; GPU, used to execute any one of the methods described in claims 6-10.

12. An image rendering device, characterized in that: Applied to a CPU, the device comprises: A reference position determination module, configured to, in response to an adjustment operation on an image currently displayed in a display area, determine a position of an image area in an original image that needs to be displayed in the display area after the adjustment operation as a reference position; wherein the image currently displayed in the display area is obtained based on the original image; An image generation module, used to generate an image having the same size as the display area and consistent with the picture content at the reference position in the original image, to obtain a first image; The first image sending module is configured to send the first image to the GPU, so that the GPU renders the first image in the display area.

13. The device according to claim 12, characterized in that The image generation module comprises: A first image generation submodule is configured to, if a first size of the image area at the reference position is larger than a second size of the display area, sample the image area at the reference position according to a ratio of the first size to the second size to obtain an image of the second size as the first image; A second image generating submodule, configured to extract an image region at the reference position from the original image as a first image if the first size is equal to the second size; A third image generation submodule is configured to, if the first size is smaller than the second size, expand the image area at the reference position in the original image to obtain an image area to be used; wherein the pixel coordinates of the vertices of the image area to be used in the original image are integers; interpolate the image area to be used according to the ratio of the second size to the first size to be used; and extract the image area of ​​the second size from the interpolation result obtained as the first image; and / or, The third image generation submodule is specifically used to, in the original image, round the pixel coordinates of the upper left corner vertex of the image area at the reference position toward the upper left corner, and round the pixel coordinates of the lower right corner vertex of the image area at the reference position toward the lower right corner, to obtain the image area to be used; calculate the product of the pixel coordinates of the upper left corner vertex of the image area at the reference position in the original image and the ratio to be used to obtain the starting pixel coordinates; take the starting pixel coordinates as the pixel coordinates of the upper left corner vertex, extract the image area of ​​the second size from the obtained interpolation result, and obtain the first image; and / or, The device also includes: an image scaling module, configured to, in response to the adjustment operation on the image currently displayed in the display area, determine the position of the image area in the original image that needs to be displayed in the display area after the adjustment operation as a reference position, and when receiving a display instruction for the original image, scale the original image according to the size of the display area to obtain a second image; A second image sending module, configured to send the second image to the GPU, so that the GPU renders the second image in the display area; and / or, The device also includes: A graphic information acquisition module, used to acquire color information of a graphic marked on the original image, and a corresponding position of the graphic in the original image; a graphics position determination module, configured to determine a graphics region to be rendered in the first image, and determine a position of the graphics region to be rendered in the first image, based on a relationship between a position corresponding to the graphics in the original image and the reference position, and a zoom ratio of the first image relative to the original image; A graphics information sending module is used to send the position of the graphics area to be rendered in the first image and the color information of the graphics area to be rendered to the GPU, so that the GPU renders the graphics area to be rendered on the upper layer of the first image in the display area.

14. An image rendering device, characterized in that: Applied to a GPU, the device comprises: A first image receiving module is used to receive a first image sent by the CPU; wherein the first image is: an image generated by the CPU having the same size as the display area and consistent with the screen content at a reference position in the original image; the reference position is: a position of an image area in the original image that needs to be displayed in the display area after the adjustment operation determined by the CPU in response to an adjustment operation on the image currently displayed in the display area; the image currently displayed in the display area is obtained based on the original image; A first image rendering module is configured to render the first image in the display area.

15. The device according to claim 14, characterized in that The device also includes: A second image receiving module is used to receive a second image sent by the CPU before receiving the first image sent by the CPU; wherein the second image is obtained by scaling the original image according to the size of the display area by the CPU when receiving a display instruction for the original image; A second image rendering module, used for rendering the second image in the display area; and / or, The device also includes: a graphics information receiving module, configured to receive the position of the graphics area to be rendered in the first image and the color information of the graphics area to be rendered sent by the CPU; wherein the graphics area to be rendered is determined by the CPU based on the relationship between the position corresponding to the graphics marked in the original image and the reference position in the original image, and the zoom ratio of the first image relative to the original image; A graphics area rendering module, used for rendering the graphics area to be rendered on an upper layer of the first image in the display area according to the received position and color information; and / or, The first image rendering module is specifically configured to load an underlying canvas in the display area and render the first image in the underlying canvas; The graphic area rendering module is specifically used to load a top-level canvas on the upper layer of the bottom-level canvas in the display area, and render the graphic area to be rendered in the top-level canvas according to the received position and color information; and / or, The device also includes: A canvas storage module, configured to load an underlying canvas in the display area, and after rendering the first image in the underlying canvas, store the underlying canvas on which the first image has been rendered in a video memory; A canvas loading module is used to load the bottom canvas on which the first image has been rendered recorded in the video memory upon receiving the position of a new graphics area to be rendered in the first image and the color information of the new graphics area to be rendered sent by the CPU, and render the new graphics area to be rendered in the top canvas according to the received new position and color information.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 5 or any one of claims 6 to 10 is implemented.

17. A computer program product, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 5 or any one of claims 6 to 10.

Images