An image processing method based on system cache and related device

Abstract

The application discloses an image processing method based on system cache. After receiving an input of displaying a specified image, an electronic device reads compressed image data of the specified image from an external memory, decompresses and decodes the compressed image data of the specified image, and obtains image data of the specified image in a first data format. The electronic device processes the image data of the specified image in the first data format, and obtains image data of the specified image in a second data format. The image data in the second data format occupies a smaller storage space than the image data in the first data format. In this way, the electronic device can directly use the system cache to transmit the image data of the specified image because the image data in the second data format occupies a smaller storage space. Moreover, the CPU and the GPU do not need to convert the data format of the image data because each hardware module of the electronic device supports processing the image data in the second data format.

Description

TECHNICAL FIELD

[0001] The present application relates to the terminal field, and particularly to an image processing method based on system cache and related apparatus. BACKGROUND

[0002] When receiving a user input and displaying an image, multiple hardware modules of the electronic device need to process the image data cooperatively, so as to display the image on the display screen. Since the data volume of the image data is large and the storage space occupied by the image data is large, the image data needs to be transferred through the memory when being transferred between the hardware modules, which results in low image data transmission efficiency and long transmission time. SUMMARY

[0003] The present application provides an image processing method based on system cache and related apparatus, which realizes reading compressed image data of a specified image from an external memory after the electronic device receives an input of displaying the specified image, decompressing and decoding the compressed image data of the specified image, and obtaining image data of the specified image in a first data format. The electronic device processes the image data of the specified image in the first data format, and obtains image data of the specified image in a second data format. The storage space occupied by the image data in the second data format is smaller than the storage space occupied by the image data in the first data format. In this way, since the storage space occupied by the image data in the second data format is small, the electronic device can directly use the system cache to transmit the image data of the specified image, thereby improving the efficiency of the electronic device in transmitting the image data. Moreover, since all the hardware modules of the electronic device support processing the image data in the second data format, the CPU and the GPU do not need to convert the data format of the image data.

[0004] In a first aspect, the present application provides a system cache-based image processing method applied to an electronic device, the electronic device comprising a central processing unit (CPU), a graphics processing unit (GPU) and a system cache (SC). The method comprises: receiving, by the electronic device, an input of displaying a first image; in response to the input of displaying the first image, obtaining, by the electronic device, first data of the first image based on a first data format; splitting, by the electronic device, the first data into a plurality of data pieces through the CPU, each data piece comprising data of x*y pixel points in the first data; compressing, by the electronic device, the plurality of data pieces to obtain second data in a second data format; transmitting, by the electronic device, the second data from the CPU to the GPU through the SC; rendering, by the electronic device, the first image based on the second data through the GPU; and displaying, by the electronic device, the first image. In this way, the electronic device represents image data of a specified image in the second data format, thereby reducing the data volume of the image data, and enabling the electronic device to directly transmit the image data of the specified image using the system cache, thereby improving the efficiency of the electronic device in transmitting the image data. Moreover, since each hardware module of the electronic device supports processing of image data in the second data format, the CPU and the GPU do not need to convert the data format of the image data.

[0005] In a possible implementation, the electronic device obtains the first data in the first data format, specifically comprising: in response to the first input, obtaining, by the electronic device, original data from a read-only memory (ROM) of the electronic device through the CPU, the original data being compressed data of the first image. Decompressing and decoding, by the electronic device, the original data through the CPU to obtain the first data.

[0006] In a possible implementation, the original data is obtained from the ROM through the CPU, specifically comprising: reading, by the electronic device, the original data from the ROM to an internal memory of the electronic device through the CPU, the internal memory also being used to store the first data. In this way, the electronic device reads the original data from the ROM to the internal memory, thereby facilitating the electronic device to process the original data to obtain the first data.

[0007] In a possible implementation, the second data is transmitted from the CPU to the GPU through the SC, specifically comprising: determining whether the data volume of the second data is less than or equal to a data volume threshold value of the SC. In response to the data volume of the second data being less than or equal to the data volume threshold value of the SC, the SC transmits the second data to the GPU. In this way, the electronic device can determine whether the system cache supports transmission of the second data based on the data volume of the second data and the data volume threshold value of the system cache, and the electronic device transmits the second data through the system cache only when the system cache supports transmission of the second data, thereby avoiding a data overflow situation.

[0008] In a possible implementation, the data piece includes a first data block and a second data block, the first data block includes values of Y components of the pixel points, and the second data block includes values of U components and V components of the pixel points. In this way, the electronic device separately arranges different components, so as to facilitate the electronic device to reduce the data amount of the U components and the V components.

[0009] In a possible implementation, the electronic device divides the first data into a plurality of data pieces through the CPU, and specifically includes: the electronic device obtains a U value based on values of U components of each 4 adjacent pixel points in the first data, and takes the U value as the values of the U components of the 4 pixel points. The electronic device obtains a V value based on values of V components of each 4 adjacent pixel points in the first image, and takes the V value as the values of the V components of the 4 pixel points. The electronic device divides data of each x*y pixel points in the first data to obtain a data piece. In this way, the electronic device reduces the data amount of the U components and the V components to 1 / 4 of the original, and reduces the storage space occupied by the image data.

[0010] In a possible implementation, the first image corresponding to the original data has a first size, and the method further includes: before the electronic device transmits the second data from the CPU to the GPU through the SC, the electronic device processes the second data, and adjusts the size of the first image from the first size to a second size. The electronic device displays the first image, and specifically includes: the electronic device displays the first image of the second size. In this way, when the electronic device displays the image of the first size as the first size, the image processing method based on the system cache can quickly decode and render the image of the changed size, without caching the first image of the second size, and the time for the electronic device to display the first image is reduced.

[0011] In a possible implementation, the encoding format of the first data is YUV. The encoding format of the plurality of data pieces is YUV420. In this way, the data amount of the plurality of data pieces is less than the data amount of the first data.

[0012] In a possible implementation, x is greater than 100, and y is greater than 100.

[0013] In a possible implementation, x is equal to 512, and y is equal to 512. In this way, because the values of x and y are large, the electronic device can change the first data into the second data with a small data amount and high accuracy. The second data can be a frame buffer, or the second data is texture data that can be recognized by the GPU. The GPU can directly render a surface based on the second data and display the surface on the display screen.

[0014] In a second aspect, the present application provides an electronic device, comprising a processor and a memory. The memory is coupled to the processor, and the memory is configured to store computer program codes, the computer program codes comprising computer instructions, which, when executed by the processor, cause the electronic device to perform the system cache-based image processing method in any possible implementation manner of the first aspect.

[0015] In a third aspect, the present application provides a computer storage medium, comprising computer instructions, which, when executed on an electronic device, cause the electronic device to perform the system cache-based image processing method in any possible implementation manner of the first aspect.

[0016] In a fourth aspect, the present application provides a computer program product, which, when executed on a computer, causes the computer to perform the system cache-based image processing method in any possible implementation manner of the first aspect.

[0017] In a fifth aspect, the present application provides a chip, comprising a processing circuit and an interface circuit, the interface circuit being configured to receive code instructions and transmit the code instructions to the processing circuit, and the processing circuit being configured to execute the code instructions to perform the system cache-based image processing method in any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A flowchart of an image data transmission process of each hardware module of an electronic device according to an embodiment of the present application is shown in FIG. 4;

[0019] Figure 2 A flowchart of a system cache-based image processing method according to an embodiment of the present application is shown in FIG. 5;

[0020] Figure 3 An image compression diagram according to an embodiment of the present application is shown in FIG. 6;

[0021] Figure 4 A flowchart of an image data processing process of an electronic device according to an embodiment of the present application is shown in FIG. 7;

[0022] Figure 5 A flowchart of an image data transmission process of each hardware module of another electronic device according to an embodiment of the present application is shown in FIG. 8;

[0023] Figure 6 A hardware structure diagram of an electronic device according to an embodiment of the present application is shown in FIG. 9. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B; the "and / or" in the text only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist together, and B exists alone.

[0025] Hereinafter, the terms "first" and "second" are only used for description purposes, and cannot be understood as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" is two or more.

[0026] In a possible implementation, after receiving the input of the user displaying the specified image, the electronic device can read the image data of the specified image from the read only memory (ROM) or other non-volatile memory to the internal memory. Wherein, the internal memory can be a random access memory (RAM), for example, the internal memory can be a double data rate synchronous dynamic random access memory (DDR SDRAM / DDR). Then, the central processing unit (CPU) of the electronic device can read the data of the specified image from the memory and process the data of the specified image. The electronic device can transmit the processed data of the specified image to the graphics processing unit (GPU). The GPU can render the specified image based on the processed data of the specified image. Wherein, the data of the specified image stored in the read only memory of the electronic device is the data of the specified image processed after compression using an image compression algorithm (for example, JPEG compression algorithm). Wherein, the processing of the image data of the electronic device can include but not limited to decompressing the image, decoding the image, changing the size of the image, etc.

[0027] For example, the process of the data of the specified image circulating between the various hardware modules of the electronic device is as follows Figure 1The CPU of the electronic device reads the compressed image data of the specified image from the read-only memory to the memory. The CPU of the electronic device can include an image processing module, an image scaling module, and the like for processing the image data. The image processing module can be used to decompress the compressed image data. The image scaling module can be used to adjust the size of the image. Here, the CPU of the electronic device can transmit the compressed image data of the specified image to the image processing module through the system cache (SC) and the DDR. The image processing module can decompress and decode the compressed image data of the specified image to obtain the decompressed and decoded image data of the specified image. The image scaling module can obtain the image data of the specified image with the adjusted size based on the decompressed and decoded image data provided by the image processing module. In some examples, the image processing module can be a joint photographic experts group intellectual property core (JPG IP). The image scaling module can be a scaler intellectual property core (SCALER IP). It should be noted that the electronic device including the image processing module and the image scaling module is only an example, and the electronic device can include more or fewer modules to achieve the function of processing image data.

[0028] After that, the CPU of the electronic device can send the processed image data of the specified image to the GPU through the memory. The GPU can render the specified image based on the processed image data. The electronic device can display the specified image on the display screen after rendering the specified image.

[0029] In some examples, the electronic device will generally represent the decompressed and decoded image data in the RGBA data format. The image data in the RGBA data format can include information of all pixel points constituting the image. The value of the RGB channel of the pixel point is used to represent the information of the pixel point in the RGB color space, and the value of the alpha channel of the pixel point is used to represent the transparency of the pixel point. The image data of the specified image includes the RGBA information of all pixel points constituting the specified image. The electronic device can store the image data of the specified image in a linear storage manner. For example, the electronic device can store the data of the specified image according to the table shown in Table 1.

[0030] Table 1: Image data in RGBA encoding format

[0031] pixel_0_0_RGBA pixel_1_0_RGBA pixel_2_0_RGBA pixel_3_0_RGBA …… pixel_0_1_RGBA pixel_1_1_RGBA pixel_2_1_RGBA pixel_3_1_RGBA …… pixel_0_2_RGBA pixel_1_2_RGBA pixel_2_2_RGBA pixel_3_2_RGBA …… pixel_0_3_RGBA pixel_1_3_RGBA pixel_2_3_RGBA pixel_3_3_RGBA …… pixel_0_4_RGBA pixel_1_4_RGBA pixel_2_4_RGBA pixel_3_4_RGBA …… …… …… …… …… ……

[0032] As shown in Table 1 above, each cell in Table 1 represents a pixel in a specified image. Each table includes the values ​​for the four RGBA channels of the pixel. In this way, electronic devices can generate and display a specified image based on the information of each pixel in Table 1.

[0033] In summary, electronic devices use the RGBA data format to represent image data, resulting in excessive storage space consumption. Furthermore, the processing of images generates new data, requiring even more storage space. Consequently, when image data is transferred between different hardware modules of the electronic device, the sheer volume of image data necessitates each module copying the received data into memory, leading to increased processing time. Figure 1 Taking the scenario shown as an example, when an electronic device processes image data for a specified image, the image data can start from the CPU, pass through DDR and SC, and reach the image processing module. Afterward, the image data will again pass through DDR to reach the image scaling module, and then through DDR to reach the GPU. Throughout this process, each IP core copies the image data to its own storage space, resulting in multiple redundant copies of data, meaning that the data for a single image is copied at least three times. Furthermore, because the image data representation formats in the CPU and GPU are different, the GPU must convert the image data format after receiving it from the CPU before it can process the image data. For example, when an electronic device loads a 2048*2048 image, it will take approximately 270ms and require approximately 32MB of storage space. Specifically, the electronic device needs 44ms to decode the image, 17ms to resize the image, 28ms for GPU image processing, and 181ms for GPU data format conversion.

[0034] Based on the above example, the embodiment of the present application provides an image processing method based on system cache. After receiving the input of displaying the specified image, the electronic device can read the compressed image data of the specified image from the external storage. The electronic device can decompress and decode the compressed image data of the specified image to obtain the image data of the specified image in the first data format. Then, the electronic device can process the image data of the specified image in the first data format to obtain the image data of the specified image in the second data format. The second data format is different from the first data format, and the storage space occupied by the image data in the second data format is smaller than the storage space occupied by the image data in the first data format. The electronic device can directly transmit the image data in the second data format through the system cache. Each hardware module of the electronic device supports processing the image data in the second data format. In this way, the electronic device represents the image data of the specified image in the second data format, reduces the data amount of the image data, so that the electronic device can directly transmit the image data of the specified image using the system cache, and improves the efficiency of the electronic device in transmitting the image data. Moreover, since each hardware module of the electronic device supports processing the image data in the second data format, the CPU and the GPU do not need to convert the data format of the image data.

[0035] Specifically, as shown in the following, Figure 2 The specific steps of the image processing method based on system cache are as follows:

[0036] S201. The electronic device receives the input of opening the first image.

[0037] The input can be not limited to single click, double click, long press, sliding, voice instruction input, etc.

[0038] S202. The CPU of the electronic device reads the original data from the read-only memory, and the original data is the data of the compressed first image.

[0039] The CPU of the electronic device can read the original data from the read-only memory (for example, hard disk). The electronic device can read the original data into the internal storage (for example, DDR).

[0040] In some examples, the electronic device can read a part (for example, 70%) of the original data into the system cache, and read another part (for example, 30%) of the original data into the internal storage. In this way, when the electronic device delivers the original data to the image processing module, the electronic device can transmit 70% of the original data to the image processing module through the system cache, and transmit 30% of the original data to the image processing module through the internal storage, thereby improving the data transmission rate.

[0041] S203. The electronic device decompresses and decodes the original data to obtain first data, the first data being decompressed and decoded original data, and the data format of the first data being a first data format.

[0042] The electronic device can use an image processing module (e.g., JPEG IP) to decompress and decode the original data to obtain the first data.

[0043] S204. The electronic device processes the first data to obtain second data in a second data format, the second data format being different from the first data format, and the second data being stored in a system cache.

[0044] Specifically, the electronic device first performs a slicing operation on the first data, and then compresses the sliced first data to obtain the second data in the second data format.

[0045] In this embodiment of the present application, the electronic device performs a slicing operation on the first data to divide the first data into one or more data tiles of a specified size. Each of the one or more data tiles includes information of a specified number of pixel points of the first image. The specified size can be used to indicate the number of pixel points, i.e., the specified number, included in each data tile. In this embodiment of the present application, the data tile of the specified size can be referred to as a large tile. In some examples, the specified number is x*y, x is greater than 100, and y is greater than 100.

[0046] In this embodiment of the present application, the specified size is greater than the size of a data tile in an RGBA encoding format, e.g., the size of a data tile based on the RGBA encoding format is 16px*16px or 32px*8px, etc. The data tile of the RGBA encoding format can be referred to as a small tile. For example, the specified size can be 512px*512px. In this way, since the specified size of the data tile of the first data is large and includes more information of pixel points, the redundant information is more, and it is convenient for the electronic device to compress the large tile.

[0047] Optionally, before or after slicing the first data, the electronic device may also compress some information of the pixel points. For example, if the first data represents the information of each pixel point using the YUV data encoding format. Among them, the value of Y is used to represent the brightness of the pixel point, and the values of U and V are used to represent the chromaticity of the pixel point. During the slicing process, the electronic device can retain the Y component of each pixel point and compress the U component and V component of every 4 adjacent pixel points. Among them, the electronic device can obtain a U value based on the U component values of every adjacent 4 pixel points and use this U value as the U component value of these 4 pixel points. Similarly, the electronic device can obtain a V value based on the V component values of every adjacent 4 pixel points and use this V value as the V component value of these 4 pixel points. It should be noted that the calculated value (for example, U value, V value) can be an average value, a maximum value, a minimum value, a median value, etc., and the embodiments of the present application do not limit this.

[0048] It should also be noted that the 4 adjacent pixel points can be 4 pixel points arranged horizontally, or 4 pixel points arranged vertically, or 4 pixel points arranged in a "field" shape, and the embodiments of the present application do not limit this. In this way, 1 pixel point in the first data corresponds to 1 Y component, the U component values of every 4 adjacent pixel points in the first data are the same, and the V component values are also the same. The electronic device only needs to retain the U component value of one pixel point among these 4 pixel points and the V component value of one pixel point among these 4 pixel points. This makes every 4 pixel points in the first data jointly correspond to 1 U component, and every 4 pixel points in the first data jointly correspond to 1 V component. The Y component of each pixel point occupies 1 channel, and the UV components jointly occupy 0.5 channels. The Y component:U component:V component in the first data = 4:1:1, reducing the data volume of the first data of the electronic device to 1 / 2 of the original.

[0049] After that, the electronic device can perform compression processing on the sliced first data to obtain the second data in the second data format. For example, the electronic device can process the sliced first data through data compression technology to obtain the second data. For example, the data compression technology can be the ARM frame buffer compression technology (arm frame buffer compression, AFBC). Among them, the electronic device can use the image processing module to process the first data to obtain the second data.

[0050] For example, taking a specified size of 512*512 as an example, the steps of the electronic device processing the first data to obtain the second data in the second data format are described. The first data format is YUV data format. The first data in the first data format includes the YUV information of all pixels of the first image. The electronic device can perform slicing processing on the first data, and the sliced ​​first data is shown in Table 2.

[0051] Table 2: First data after slicing

[0052]

[0053] As shown in Table 2, the first data after slicing includes two data blocks, plane0 and plane1. Plane0 stores the Y component, and plane1 stores the U and V components. The electronic device includes multiple tiles, each of which can include tile1 and tile2. Each tile includes information for 512*512 pixels. It should be noted that Table 2 is merely an example; the actual first data obtained by the electronic device after slicing can be represented in other forms (e.g., arrays), and this embodiment does not limit this representation.

[0054] The electronic device can then compress the sliced ​​first data to obtain second data in a second data format. For example, Figure 3 In the diagram, (a) represents plane0 in the first sliced ​​data, and the gray box represents the Y component data. Electronic devices... Figure 3 After compressing the data shown in (a), we can obtain Figure 3 The data shown in (b) is as follows. Figure 3 In the diagram, (b) represents plane0 in the second set of data. The striped boxes represent valid data, and the white boxes represent invalid data. From... Figure 3 It can be seen that after the electronic device compresses the first data after slicing, it can concentrate the effective data in the Y component, U component and V component together, further reducing the amount of image data in the first image.

[0055] In some examples, valid data is a non-zero value, and invalid data is a zero value. It is understood that this example uses only plane0; the electronic device will also compress the data in plane1.

[0056] In this way, because the electronic device compresses and slices the first data, it concentrates the effective data of the Y, U, and V components in the first data together, while the invalid data is also concentrated together. When transmitting image data, the electronic device can transmit only the effective data portion, reducing the amount of image data and improving image processing efficiency.

[0057] Compared with the processing method of dividing the first data into small slices and performing compression processing based on the small slices, the embodiment of the present application adopts large slices for compression, and the same information between different large slices is less, so that the redundant information in the compressed image data is less, the storage space occupied by the compressed image data is smaller, and the compression rate is high.

[0058] The compressed image data obtained by the electronic device based on the large slices not only can improve the compression rate of the image and reduce the storage space while ensuring the relative integrity of the information, but also is easier to process because the CPU and GPU can directly process the image data of the large slices. When the electronic device displays the image corresponding to the compressed image data, the CPU can directly transmit the compressed image data to the image processing module through the system cache, thereby accelerating the image transmission rate, because the data amount of the compressed image data is small.

[0059] It should be noted that the size of the data slice can determine the size of the storage space occupied by the compressed data and the image processing time required by the GPU to process the image. Among them, the smaller the data slice, the lower the compression rate, the larger the storage space occupied by the compressed data, the longer the data transmission time, and the shorter the image processing time required by the GPU to process the image. The larger the data slice, the higher the compression rate, the smaller the storage space occupied by the compressed data, the shorter the data transmission time, and the longer the image processing time required by the GPU to process the image. Therefore, the optimal size of the data slice can be determined by the image processing time and the data transmission time, and when the data slice is the optimal size, the sum of the image processing time and the data transmission time is the smallest.

[0060] In some examples, it can be obtained through experiments that the specified size is 512px*512px, the image processing time and the data transmission time are both short, and 512px*512px is taken as the size of the large slice.

[0061] In other examples, the size of the large slice can also be 256px*256px or 1024px*1024px.

[0062] In this way, since the data format of the second data is the second data format, the data amount of the first image is reduced, and the electronic device does not need to transmit the data in the manner of DDR copying. In addition, since the data amount of the second data is small, when the second data is transmitted between different hardware modules (for example, between different IP cores, between an IP core / CPU and a GPU) of the electronic device, the system cache can be used for transmission of the second data, and the time consumed for data transmission is shortened. For example, the electronic device uses the second data format to represent the second data, so that the data amount of the second data is less than a data amount threshold. The data amount threshold can be queried according to the model of the processor. In some application scenarios, the data amount threshold can be 8 MB.

[0063] In some examples, since the second data format is universal, each IP core, GPU, and system on chip (SoC) unit can recognize (or read) image data in the second data format. After receiving the image data in the second data format, these hardware modules directly process the image data.

[0064] In some examples, since each hardware module of the electronic device can process data in the second data format, the electronic device can perform hardware-accelerated decoding (hardware decoding for short) on the image data, and implement image processing operations through a dedicated decoding circuit fixed in hardware (for example, a JPG IP or a SCALER IP). In this way, compared with software-accelerated decoding (that is, decoding of the image by the CPU through software programming), the processing efficiency is improved, and optimization of scenarios such as cross-IP transmission of image data and delivery of video frames is also achieved.

[0065] In some examples, after obtaining the second data, the electronic device can also process the second data to change the size, resolution, and the like of the first image. For example, the electronic device can change the size of the first image through an image scaling module. The electronic device then transmits the processed second data to the GPU.

[0066] Optionally, in the process of processing the first data, the electronic device can also improve the first data in other dimensions (for example, data storage form, and the like) to reduce the data amount of the second data.

[0067] S205. The GPU of the electronic device reads the second data from the system cache and renders the first image based on the second data.

[0068] The CPU / image processing module / image scaling module of the electronic device can transmit the second data to the GPU. In this process, since the data amount of the second data is small, the second data can directly reach the GPU through the system cache, without passing through the DDR. After receiving the second data, the GPU can render the first image based on the second data.

[0069] It should be noted that, under the premise of large slice compression, the electronic device can change the first data into second data with small data volume and high accuracy. The second data can be a frame buffer, or the second data is texture data that can be recognized by the GPU. The GPU can directly render a surface based on the second data and display the surface on the display screen.

[0070] In this way, the electronic device uses the large slice to realize information synchronization between various hardware modules. After the CPU processes the tile, the GPU can continue to process the tile. The CPU and the GPU can perform calculations at the same time. In this way, an efficient pipeline is formed, the time for the electronic device to process image data is saved, the CPU and the GPU do not need to wait for each other when processing, and the efficiency of the chip is improved.

[0071] In a possible implementation, before the CPU / image processing module / image scaling module of the electronic device transmits the second data to the GPU, the electronic device can determine whether the data volume of the second data is less than a data volume threshold (for example, 8 MB) of system cache transmission. When the electronic device determines that the data volume of the second data is less than or equal to the data volume threshold of system cache transmission, the CPU / image processing module / image scaling module of the electronic device can transmit the second data to the GPU through the system cache. When the electronic device determines that the data volume of the second data is greater than the data volume threshold of system cache transmission, the CPU / image processing module / image scaling module of the electronic device transmits the second data to the GPU through the internal storage. In this way, the electronic device can determine whether the system cache supports transmission of the second data based on the data volume of the second data and the data volume threshold of system cache transmission. The electronic device transmits the second data through the system cache only when the system cache supports transmission of the second data, so as to avoid data overflow.

[0072] S206. The electronic device displays the first image.

[0073] After the GPU generates the first image, the first image can be transmitted to the display screen for display.

[0074] It should be noted that, the above is not limited to the scenario of displaying the first image. In other scenarios involving images or videos, for example, video encoding and decoding, video frame transmission, sharing images, downloading images, saving images, etc., the second data format and / or the image processing method provided in the embodiments of the present application can be used to improve the processing efficiency of the electronic device.

[0075] For example, Figure 4As shown, the electronic device stores the first image in the form of a joint photographic experts group (JPG / JPEG) file. The first image has a size of 1920*1080 and is compressed in a JPEG format. During the process of displaying the first image, the electronic device can first read the compressed data of the first image, i.e., the original data, from a file named pic1.jpg in the external storage. The image processing module of the electronic device can decompress and decode the original data to obtain the first data in a first data format. The electronic device can process the first data to obtain second data in a second data format.

[0076] If the first data format is an RGBA8888 data format, the size of the first data obtained by the electronic device is 1920*1080*4 byte = 8.2944 MB. The electronic device can perform large slice compression on the first data to obtain the second data. After the large slice compression, the second data can be compressed to 1 / 4 to 1 / 8 of the first data. In the embodiments of the present application, the second data format can be referred to as an -RGB data format.

[0077] If the first data format is a YUV data format, the size of the first data obtained by the electronic device is 1920*1080*3 byte = 6.22 MB. The electronic device can perform large slice compression on the first data to obtain the second data. After the large slice compression, the second data can be compressed to 1 / 4 to 1 / 8 of the first data.

[0078] After that, the electronic device can transmit the second data to the GPU through the system cache. The GPU can render the first image based on the second data. The electronic device can display the first image on the display screen.

[0079] Next, the image processing method based on the system cache provided by the embodiments of the present application will be introduced in combination with specific scenarios.

[0080] As shown, the data of the first image flows among the various hardware modules of the electronic device. Figure 5 As shown, the electronic device includes a CPU, a GPU, a system cache (SC), and a DDR, etc. The CPU includes one or more modules, such as an image processing module and an image scaling module. The description of the various hardware modules can be referred to in the Figure 1In the illustrated embodiment, the electronic device reads the compressed image data of the specified image from the read-only memory to the DDR via the CPU. The CPU can send the compressed image data of the specified image to the image processing module via the SC and the DDR. In some examples, the electronic device can read a portion (e.g., 70%) of the compressed image data into the system cache and read another portion (e.g., 30%) of the compressed image data into the DDR. The electronic device can transmit 70% of the raw data to the image processing module via the system cache and transmit 30% of the raw data to the image processing module via the DDR when passing the raw data to the image processing module, thereby improving the data transmission rate.

[0081] The image processing module can decompress and decode the compressed image data of the specified image to obtain decompressed and decoded image data of the specified image, and process the decompressed and decoded image data to obtain image data in a second data format. For a description of how the electronic device obtains image data in the second data format, refer to Figure 2 In the illustrated embodiment, the electronic device can then send the image data in the second data format to the image scaling module via the system cache. The image scaling module can obtain image data of the specified image in a resized size based on the data provided by the image processing module. The resized image data is also in the second data format described above.

[0082] The image scaling module of the electronic device can then send the resized image data to the GPU via the system cache. The GPU can render the specified image based on the processed image data. The electronic device can display the specified image.

[0083] As can be seen from the comparison between Figure 1 and Figure 3 , compared with the scenario shown in Figure 1 , the image processing method provided in the embodiments of the present application can enable the electronic device to display the specified image more quickly. For example, taking an image with an image size of 1080*1920 as an example, the difference between the method shown in Figure 1 and the image processing method provided in the embodiments of the present application when processing image data is described.

[0084] Table 3: Image data corresponding to different image processing methods

[0085]

[0086] As shown in Table 3, the size of the original image is 1080*1920. When the image is transmitted in the RGBA encoding format, the number of data channels is 4, and the slice size is 16*16. After the electronic device performs slicing and other processes on the original image, the size of the image is 7.9 MB, the compression rate of the image is 1, and the traffic memory is 15.8 MB. The traffic memory represents the size of the storage space occupied in the system cache when the image data is transmitted. Since the compression rate of the image is 1, the electronic device will still transmit the data of the 4 channels when transmitting the image data in the RGBA encoding format, and the storage space occupied in the system cache is 7.9*2=15.8 MB.

[0087] When the image is transmitted in the second data format, the number of data channels is 1.5, and the slice size is 512*512. After the electronic device performs slicing and other processes on the original image, the size of the image is 2.9 MB, the compression rate of the image is 0.5, and the traffic memory is 1.8 MB. Since the compression rate of the image is 0.5, and since the arrangement of the image data in the second data format is also different from the arrangement of the image data in the RGBA encoding format, the amount of image data will be further reduced. Therefore, the image data in the second data format can be transmitted through the system cache. When the image data enters the system cache, the data size may become 1 / 3 of the original size, i.e., 2.9 / 3≈0.96 MB of memory is occupied. The system cache needs to allocate 2.9 / 3*2≈1.8 MB of memory space for the image data.

[0088] In summary, the image processing method based on the system cache provided in the embodiments of the present application occupies only 1 / 8 of the memory space of the system cache shown in Figure 1 In combination with Figure 5 and Figure 1 It can be seen that when the various hardware modules transmit image data, based on the method provided in the embodiments of the present application, for some images with a large amount of data, the system cache can still be used for transfer, without occupying the space of the DDR, thereby improving the data transmission rate.

[0089] In some application scenarios, when the electronic device displays an image with a size of 900px*900px, a display size of 2048px*2048px, and an image size of 1.1 MB, the time consumed by using different image processing methods is shown in Table 4.

[0090] Table 4: Time required for the electronic device to process image data in different data formats

[0091]

[0092] As shown in Table 4, in the process of image display, when the encoding format of the image data of the specified image of the electronic device is RGBA, the electronic device parses the image data in a soft-decoding manner, decodes the image, scales the image, and renders the image, all of which consume a lot of time. Moreover, due to the large amount of data of the image, data transmission through DDR is required, which consumes a long time. Therefore, the electronic device needs to consume more time to display the image. When the encoding format of the image data of the specified image of the electronic device is the second data format, the electronic device can parse the image data in a hard-decoding manner. Moreover, the amount of data of the image data is small, and data transmission through the system cache is required, which consumes a short time. Therefore, the electronic device consumes less time to decode the image, scale the image, and render the image using the GPU.

[0093] Moreover, due to the long time consumed by the electronic device to decode and render the image in the RGBA data encoding form, in order to display the image more quickly when the user opens the image again, the electronic device can convert the obtained 900px*900px image into a PNG image format and save it in the external storage. In this way, the electronic device needs to consume 181 ms to cache the data of the 900px*900px image. However, the image processing method based on the system cache provided in this application can quickly decode and render the image with the changed size without caching. Therefore, when the electronic device displays an image with different sizes for the first time, the electronic device displays the image in the manner shown in Table 4, and the total time consumed to process the image data is 270 ms. When the electronic device displays the image by using the image processing method based on the system cache provided in this embodiment, the total time consumed to process the image data is 10 ms, which reduces the time consumed by the electronic device to process the image data. Figure 1

[0094] The electronic device provided in the embodiments of this application is described below.

[0095] The electronic device can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device, and / or a smart city device, and the specific type of the electronic device is not specially limited in the embodiments of this application.

[0096] Figure 6 ​A structural schematic diagram of the electronic device 100 is shown.

[0097] The electronic device can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0098] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0099] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices, or can be integrated in one or more processors.

[0100] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to instruction operation codes and timing signals to complete the control of fetching instructions and executing instructions.

[0101] The processor 110 can also have internal memory that is used to store instructions and data. In some embodiments, the internal memory of the processor 110 is a cache memory. The cache memory can hold instructions and data that the processor 110 has recently used or that are likely to be used. If the processor 110 needs to use the instructions or data again, it can be retrieved directly from the cache memory. This avoids repetition and reduces the latency of the processor 110, thus improving the efficiency of the system.

[0102] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0103] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the electronic device 100. In another embodiment of the present application, the electronic device 100 can also use different interface connection modes or a combination of multiple interface connection modes in the above embodiments.

[0104] The charging management module 140 is configured to receive charging input from a charger. The charging management module 140 can charge the battery 142 and also supply power to the electronic device through the power management module 141. The power management module 141 is configured to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to supply power to the processor 110, the internal memory 121, the display 194, the camera 193 and the wireless communication module 160, etc.

[0105] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc. The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas.

[0106] The mobile communication module 150 can provide a solution including 2G / 3G / 4G / 5G wireless communication applied to the electronic device 100.

[0107] The modem processor can include a modulator and a demodulator. The modulator is used to modulate a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal processed by the baseband processor is transmitted to the application processor. The application processor outputs the sound signal through the audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays the image or video through the display screen 194.

[0108] The wireless communication module 160 can provide a solution including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. wireless communication solutions applied to the electronic device 100.

[0109] In some embodiments, the antenna 1 of the electronic device 100 is coupled with the mobile communication module 150, and the antenna 2 is coupled with the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0110] The electronic device 100 realizes the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0111] The display screen 194 is configured to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diodes (QLED), or the like. In some embodiments, the electronic device 100 can include one or N display screens 194, where N is a positive integer greater than 1.

[0112] The electronic device 100 can implement the photographing function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor.

[0113] The ISP is configured to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing to convert it into an image visible to the naked eye. The ISP can also algorithmically optimize the noise and brightness of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be disposed in the camera 193.

[0114] The camera 193 is configured to capture still images or videos. An object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In some embodiments, the electronic device 100 can include one or N cameras 193, where N is a positive integer greater than 1.

[0115] The digital signal processor is used to process digital signals, in addition to being able to process digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0116] The video codec is used to compress or decompress digital video. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0117] The NPU is a neural-network (NN) computing processor, which can quickly process input information by drawing on the structure of a biological neural network, such as drawing on the transmission mode between human brain neurons, and can also continuously self-learn. Through the NPU, the electronic device 100 can realize intelligent cognition applications such as image recognition, face recognition, voice recognition, text understanding, etc.

[0118] The external memory interface 120 can be used to connect an external non-volatile memory to expand the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to realize data storage functions. For example, files such as music and videos are saved in the external non-volatile memory.

[0119] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various function applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc. The data storage area can store data created during use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0120] The electronic device 100 can implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, and an application processor, etc. For example, music playback, recording, etc. The audio module 170 is used to convert digital audio information into analog audio signals for output, and is also used to convert analog audio input into digital audio signals. The speaker 170A, also known as a "loudspeaker", is used to convert audio electrical signals into sound signals. The receiver 170B, also known as a "earpiece", is used to convert audio electrical signals into sound signals. The microphone 170C, also known as a "microphone", "sound transducer", is used to convert sound signals into electrical signals.

[0121] The pressure sensor 180A is used to sense a pressure signal, and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. The barometric pressure sensor 180C is used to measure air pressure. The magnetic sensor 180D includes a Hall sensor, and the magnetic sensor 180D can be used to detect the opening and closing of a flip leather cover. The acceleration sensor 180E can detect the magnitude of acceleration of the electronic device 100 in various directions (generally three axes). The distance sensor 180F is used to measure distance. The proximity light sensor 180G can also be used for automatic unlocking and locking in a cover mode and a pocket mode. The ambient light sensor 180L is used to sense ambient light brightness. The fingerprint sensor 180H is used to collect fingerprints. The temperature sensor 180J is used to detect temperature. The touch sensor 180K, also known as a "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also known as a "touch panel". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the touch event type. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, which is different from the position where the display screen 194 is located. The bone conduction sensor 180M can obtain vibration signals. The keys 190 include a power-on key, a volume key, etc. The motor 191 can generate a vibration prompt. The indicator 192 can be an indicator light, which can be used to indicate the charging state, the power change, and can also be used to indicate messages, missed calls, notifications, etc. The SIM card interface 195 is used to connect a SIM card.

[0122] The above-described embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A system cache-based image processing method applied to an electronic device, comprising: The electronic device comprises a central processing unit (CPU), a graphics processing unit (GPU) and a system cache (SC); the method comprises: The electronic device receives an input of displaying a first image; In response to the input of displaying the first image, the electronic device acquires first data of the first image based on a first data format; The electronic device divides the first data into a plurality of data pieces by the CPU, and each data piece comprises data of x*y pixel points in the first data; The electronic device compresses the plurality of data pieces to obtain second data in a second data format; The electronic device transmits the second data from the CPU to the GPU by the SC; The electronic device renders the first image based on the second data by the GPU; The electronic device displays the first image.

2. The method of claim 1, wherein, The electronic device acquires first data in a first data format, specifically comprising: In response to the input of displaying the first image, the electronic device acquires original data from a read-only memory (ROM) of the electronic device by the CPU, and the original data is data of the first image after compression; The electronic device decompresses and decodes the original data by the CPU to obtain the first data.

3. The method of claim 2, wherein, The CPU acquires the original data from the ROM, specifically comprising: The electronic device reads the original data from the ROM to an internal memory of the electronic device by the CPU, and the internal memory is also used to store the first data.

4. The method of claim 1, wherein, The electronic device transmits the second data from the CPU to the GPU by the SC, specifically comprising: Judging whether the data amount of the second data is less than or equal to a data amount threshold value of the SC transmission; In response to the data amount of the second data being less than or equal to the data amount threshold value of the SC transmission, the SC transmits the second data to the GPU.

5. The method of claim 1, wherein, The data piece comprises a first data block and a second data block, the first data block comprises a value of a Y component of a pixel point, and the second data block comprises a value of a U component and a value of a V component of the pixel point.

6. The method of claim 1, wherein, The CPU divides the first data into a plurality of data pieces, specifically comprising: The electronic device obtains a U value based on the value of the U component of each adjacent 4 pixel points in the first data, and takes the U value as the value of the U component of the 4 pixel points; The electronic device obtains a V value based on the value of the V component of each adjacent 4 pixel points in the first image, and takes the V value as the value of the V component of the 4 pixel points; The electronic device divides the data of each x*y pixel points in the first data to obtain a data piece.

7. The method of claim 2 or 3, wherein, The size of the first image corresponding to the original data is a first size; the method further comprises: Before the electronic device transmits the second data from the CPU to the GPU by the SC, the electronic device processes the second data to adjust the size of the first image from the first size to a second size; The electronic device displays the first image, specifically comprising: The electronic device displays the first image of the second size.

8. The method of claim 1, wherein, The first data is in a YUV format; and the plurality of data pieces is in a YUV420 format.

9. The method of claim 1, wherein, The x is greater than 100, and the y is greater than 100.

10. The method of claim 9, wherein, The x is equal to 512, and the y is equal to 512.

11. An electronic device, comprising: An electronic device comprising a processor and a memory; wherein the memory is coupled to the processor, and the memory is configured to store computer program codes comprising computer instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 1-10.

12. A computer-readable storage medium comprising instructions, wherein: The instructions, when executed on an electronic device, cause the electronic device to perform the method of any one of claims 1-10.

13. A chip, characterized by An apparatus comprising processing circuitry and interface circuitry, the interface circuitry configured to receive code instructions and transmit the code instructions to the processing circuitry, the processing circuitry configured to execute the code instructions to perform the method of any one of claims 1-10.

Images