An image processing method and apparatus

Abstract

The present application provides an image processing method and apparatus, which relate to the field of terminal system optimization, and solve the problems of memory waste in the process of image display in the prior art and the lag problem caused by the large time consumption of converting bitmap data into texture data and uploading it to the memory of the GPU. The method includes: displaying an interaction interface of an application, and when a preset operation acting on the interaction interface by a user is detected, decoding the image data to be displayed by the application into bitmap data, and encapsulating the bitmap data into texture data; storing the texture data in a memory partition accessible by a graphics processing unit (GPU); triggering the GPU to read the texture data to perform rendering processing to obtain rendered data; triggering the display to display an image according to the rendered data.

Description

Technical Field

[0001] This application relates to the technical field of terminals, and in particular, to an image processing method and apparatus. Background Art

[0002] Before a mobile device displays a picture on the interface of an application (APP), the central processing unit (CPU) of the mobile device needs to first decode the picture into bitmap data, and save the bitmap data in a memory partition belonging to the CPU. Then, the CPU converts the bitmap data in the CPU memory partition into texture data, and transfers the texture data to a memory partition belonging to the graphics processing unit (GPU). Finally, the GPU draws the texture data and displays it on the display screen of the mobile device.

[0003] It can be understood that the process of displaying pictures on the APP requires a block of memory of the CPU to store the bitmap data generated during data processing during decoding, and also requires a block of memory of the GPU to store the texture data after converting the bitmap data. It can be seen that the above technical solution requires allocating a block of memory to the CPU and a block of memory to the GPU, resulting in a waste of memory occupation. Moreover, converting the bitmap data into texture data and uploading it to the memory of the GPU takes a long time, and it is easy to experience lags during the picture display process, making it not smooth for users to view pictures, resulting in a poor user experience. Summary of the Invention

[0004] This application provides an image processing method and apparatus, which solves the problems of waste of memory occupation during the processing of picture display in the prior art and the lag caused by the long time taken to convert bitmap data into texture data and upload it to the memory of the GPU.

[0005] To achieve the above object, this application adopts the following technical solutions:

[0006] In a first aspect, an image processing method is provided. The method can be applied to an electronic device. The method may include: The electronic device displays an interaction interface of an application. When a preset operation on the interaction interface by the user is detected, and at the same time the application starts to obtain picture data to be displayed based on an input event, the picture data to be displayed by the application is decoded into bitmap data, and the bitmap data is encapsulated into texture data; the texture data is stored in a memory partition accessible by the graphics processing unit (GPU); the GPU is triggered to read the texture data and perform a drawing process to obtain rendered data; the display is triggered to display the picture according to the rendered data. It should be understood that the preset operation includes, but is not limited to, sliding, clicking, double-clicking, long-pressing, force-pressing, air gestures, moving the line of sight focus, etc.

[0007] In the prior art, the memory allocator first applies for a memory partition for the CPU to store bitmap data accessible to the CPU, and then applies for a memory partition for the GPU to store texture data accessible to the GPU. In the embodiment of the present application, the memory allocator defined by the electronic device is used to apply for a memory partition accessible to the graphics processing unit (GPU) for storing texture data. When decoding picture data, the texture data generated after decoding the picture data is stored in this memory partition. In this way, there is no need to apply for a memory partition accessible to the CPU, which can solve the problem of waste of memory occupation in the process of picture display in the prior art; moreover, in the process of decoding the picture data in the present application, it is not necessary to convert the bitmap data into texture data and then upload the data from the memory of the CPU to the memory of the GPU, which can solve the problem of jamming caused by the large time consumption of data copying and improve the user experience.

[0008] In a possible design, before decoding the picture data to be displayed by the application into bitmap data, the method further includes: creating a software decoder; defining a memory allocator, where the memory allocator is used to apply for a memory partition accessible to the GPU, and the memory partition accessible to the GPU is used to store texture data. In the above possible implementation, by improving the memory allocator and defining a memory partition accessible to the GPU, and this memory partition is used to store the texture data generated by decoding, there is no need to apply for a memory accessible to the CPU, reducing the waste of memory occupation.

[0009] In a possible design, the memory allocator is used to apply for a memory partition accessible to the GPU, including: the memory allocator is used to call a first interface to apply for a memory partition accessible to the GPU from the internal memory, and the first interface is a standard interface for the GPU to apply for a memory partition. The memory partition accessible to the GPU includes the physical address range accessible to the GPU and the size of the memory partition accessible to the GPU. In the above possible implementation, by calling the standard interface for the GPU to apply for a memory partition to apply for memory, it avoids applying for the memory of the CPU and reduces the waste of memory occupation.

[0010] In a possible design, decoding the picture data to be displayed of an application into bitmap data and encapsulating the bitmap data as texture data includes: decoding the picture data of the first row of the picture data to be displayed to generate the bitmap data of the picture data of the first row, performing data conversion on the bitmap data of the picture data of the first row to generate the texture data of the picture data of the first row; then performing the above processing on the second row data of the picture data to be displayed until the picture data of the last row of the picture data to be displayed is processed. In the above possible implementation, by decoding the picture data and performing parallel conversion processing on the generated bitmap data, the texture data is directly obtained, thus avoiding the process of converting the bitmap data stored in the memory partition of the CPU into texture data and then uploading it to the memory partition of the GPU, thereby avoiding the lag problem caused by the large time consumption of data upload.

[0011] In a possible design, decoding the picture data of the first row of the picture data to be displayed to generate the bitmap data of the picture data of the first row, performing data conversion on the bitmap data of the picture data of the first row to generate the texture data of the picture data of the first row; then performing the above processing on the second row data of the picture data to be displayed until the picture data of the last row of the picture data to be displayed is processed, including: calling a decoding function to decode the first row of picture data to generate the bitmap data corresponding to the first row of picture data; calling a texture conversion dynamic library to perform data type conversion processing on the bitmap data corresponding to the first row of picture data to generate the texture data corresponding to the first row of picture data, and the texture conversion dynamic library includes a conversion function for converting bitmap data into texture data; then performing the above processing on the second row of picture data of the picture data to be displayed until the picture data of the last row of the picture data to be displayed is processed. In the above possible implementation, by performing parallel acceleration processing on the bitmap data, the texture data is directly generated, thus avoiding the process of converting the bitmap data stored in the memory partition of the CPU into texture data and then uploading it to the memory partition of the GPU, thereby avoiding the lag problem caused by the large time consumption of data upload.

[0012] In a second aspect, an electronic device is provided, which includes: a memory and one or more processors; the memory and the processor are coupled; the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method described in the first aspect and any of its possible design manners.

[0013] A third aspect provides a chip system, which can be applied to an electronic device; the system includes one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected by lines; the interface circuits are configured to receive signals from the memory of the electronic device and send the signals to the processors, and the signals include computer instructions stored in the memory; when the processors execute the computer instructions, the electronic device executes the method described in the first aspect and any of its possible design manners.

[0014] A fourth aspect provides a readable storage medium, in which instructions are stored. When the readable storage medium runs on an electronic device, the electronic device is caused to execute the method described in the first aspect and any of its possible design manners.

[0015] A fifth aspect provides a computer program product, characterized in that when the computer program product runs on a computer, the computer is caused to execute the method described in the first aspect and any of its possible design manners.

[0016] It can be understood that any of the above-provided electronic devices, chip systems, readable storage media, and computer program products can be implemented according to the corresponding picture display method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the picture display method provided above, which will not be elaborated here. Description of the Drawings

[0017] Figure 1 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

[0018] Figure 2 It is a schematic structural diagram of a processor and a memory of an electronic device provided by an embodiment of the present application;

[0019] Figure 3 It is a software structure diagram of an electronic device provided by an embodiment of the present application;

[0020] Figure 4 It is a processing schematic diagram of a picture display method provided by an embodiment of the present application;

[0021] Figure 5 It is a processing schematic diagram of another picture display method provided by an embodiment of the present application;

[0022] Figure 6 It is a processing flow schematic diagram of a picture display method provided by an embodiment of the present application;

[0023] Figure 7 It is a software processing flow diagram of a picture display method provided by an embodiment of the present application;

[0024] Figure 8It is a software processing flowchart of another picture display method provided by an embodiment of this application;

[0025] Figure 9 It is a schematic structural diagram of a chip system provided by an embodiment of this application. Detailed implementation manners

[0026] Before introducing the embodiments of this application, relevant terms designed in the technical solution will be briefly introduced first.

[0027] A bitmap, also known as a dot matrix picture or a raster picture, is composed of individual points called pixels (picture elements). These pixel points can be arranged and dyed differently to form a picture.

[0028] Picture data: It can be a local picture file to be displayed by an electronic device or a picture data stream downloaded. The format of the picture data can be Portable Network Graphics (PNG), JPEG format (Joint Photographic Experts Group), or a stream media file (Stream), etc.

[0029] Bitmap data: The picture data in bitmap format generated after decoding the picture data can be called bitmap data.

[0030] Texture data: A format of picture data, which is bitmap data that can represent the surface details of an object; specifically, it can be picture data representing the colored plane patterns or uneven grooves of a picture. Texture data can be data that can be recognized and processed by a GPU for drawing.

[0031] Rendered data: The data generated during the pixel rendering and pixel filling processes by an image processor GPU according to texture data and drawing instructions is called rendered data. The display processing module of an electronic device can perform image display processing based on the rendered data.

[0032] Lag: It is a phenomenon that appears in electronic devices such as mobile phones and laptops. The specific situation can be the frame lag phenomenon of various electronic devices during operation, such as when playing games or displaying pictures.

[0033] Application Programming Interface (API): It is a set of predefined functions, aiming to provide applications and developers with the ability to access a set of routines based on a certain software or hardware, without the need to access the source code or understand the details of the internal working mechanism.

[0034] The electronic devices involved in this application can be mobile phones, tablet computers, desktop computers, laptops, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, as well as devices with touch screens such as cellular phones, personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices, etc.

[0035] As Figure 1 shown is a schematic structural diagram of the electronic device 100.

[0036] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0037] It can be understood that the structure schematically shown in the embodiments of this application does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0038] The processor 110 may include one or more processing units. It should be understood that the processor 110 may include a Central Processing Unit (CPU), an application processor (AP), a modem, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors. By way of example, Figure 1 only the central processor and the graphics processor are shown in the processor 110.

[0039] Among them, the controller may be the nerve center and command center of the electronic device 100. The controller may generate operation control signals according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0040] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0041] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0042] It can be understood that the interface connection relationships among the modules illustrated in the embodiments of the present application are only illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection manners in the above embodiments, or a combination of multiple interface connection manners.

[0043] The electronic device 100 realizes the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. The GPU is used to execute mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0044] The display screen 194 is used to display pictures, videos, etc. The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0045] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external memory card.

[0046] The internal memory 121 can be used to store computer-executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, a picture playback function, etc.). The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.). In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. It should be noted that in the embodiments of the present application, the internal memory 121 has the same meaning as the memory described in the embodiments of the present application, and the data storage area in the internal memory 121 may include a memory partition accessible by the CPU and a memory partition accessible by the GPU.

[0047] Next, the hardware connection relationship between the central processing unit CPU, the graphics processing unit GPU, and the memory included in the electronic device 100, as well as the respective functions of the three, will be introduced. As Figure 2As shown, the CPU can be a large-scale integrated circuit and is the computing core and control center of a computer. Its main function is to interpret computer instructions and process data in computer software.

[0048] The CPU mainly includes an arithmetic unit (Arithmetic Logic Unit, ALU), a cache, and buses for data, control, and status that connect them. It, together with the internal memory and input / output (I / O) devices, is collectively referred to as the three core components of an electronic computer.

[0049] The GPU, also known as the display core, visual processor, and display chip, is a microprocessor specialized for processing image operations. The purpose of the GPU is to convert and drive the display information required by the electronic device and provide a line scan signal to the display to control the correct display of the display. It is an important component connecting the display and the CPU. As Figure 2 shown, the GPU can include: a work management module for managing the GPU to execute the rendering instructions issued by the CPU; an Advanced Peripheral Bus (APB) module, a shader module, rendering materials, a memory management unit (MMU), and an L2 cache module. The CPU sends a command to the GPU, and this command can specifically be an instruction to render an image; the GPU can interact with the CPU through sending interrupts for data.

[0050] The internal memory, simply referred to as memory, is a device in an electronic device for storing data and programs and is a bridge for communication between the CPU and the GPU. As Figure 2 shown, the memory can be a Double Data Rate (DDR) memory, simply referred to as DDR memory. All programs in the electronic device run in the memory. Therefore, the performance of the memory has a very great impact on the electronic device. The storage data area in the memory can be used to temporarily store the operation data of the processor CPU or GPU, as well as data exchanged with external memories such as hard disks. As long as the electronic device is running, the CPU or GPU will transfer the data that needs to be operated to the memory for operation, and then transfer the result out of the memory after the operation is completed. Therefore, as Figure 2 shown, the DDR memory includes a memory partition accessible by the GPU and a memory partition accessible by the CPU.

[0051] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In the embodiments of this application, the Android system with a layered architecture is taken as an example to exemplarily illustrate the software structure of the electronic device 100.

[0052] Figure 3 It is a software structure block diagram of the electronic device 100 in the application embodiment.

[0053] The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely the application layer, the application framework layer, the Android runtime, the system libraries, and the kernel layer.

[0054] The application layer may include a series of application packages.

[0055] Such as Figure 3 shown, the application packages may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0056] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0057] Such as Figure 3 shown, the application framework layer may include a window manager, a content provider, a view system, a telephone manager, a resource manager, a notification manager, etc.

[0058] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0059] The content provider is used to store and obtain data, and make this data accessible to applications. The data may include videos, pictures, audio, dialed and answered calls, browsing history and bookmarks, phone book, etc.

[0060] The view system includes visible controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a short message notification icon may include a view for displaying text and a view for displaying pictures.

[0061] The telephone manager is used to provide the communication function of the electronic device 100. For example, the management of call states (including connection, disconnection, etc.).

[0062] The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc.

[0063] The notification manager enables an application to display notification information in the status bar. It can be used to convey messages of the notification type, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform that a download is complete, a message reminder, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or a scrolling text, such as a notification of a background running application, or a notification that appears on the screen in the form of a dialogue window. For example, it can prompt text information in the status bar, emit a prompt tone, vibrate the electronic device, blink the indicator light, etc.

[0064] The Android Runtime includes core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0065] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0066] The system libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0067] The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

[0068] The media libraries support the playback and recording of multiple common audio and video formats, as well as static picture files, etc. The media libraries can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0069] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc.

[0070] The 2D graphics engine is a drawing engine for 2D drawing.

[0071] The kernel layer is the layer between the hardware and the software. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver.

[0072] Next, in combination with the scenario of capturing the trigger for picture display, the working processes of the software and hardware of the electronic device 100 will be exemplarily described.

[0073] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including information such as touch coordinates and the timestamp of the touch operation). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking the touch operation as a touch click operation and the control corresponding to the click operation as the control of the gallery application icon as an example, the gallery application calls the interface of the application framework layer to start the gallery application, and then starts the display driver by calling the kernel layer to present the picture through the display screen 194.

[0074] An embodiment of the present application provides a picture processing method, such as Figure 4 , which may include several processing processes such as picture loading, picture decoding, and GPU rendering.

[0075] First, perform a picture loading process on the picture to be displayed. When the mobile phone displays the interactive interface of an application and detects a preset operation on the interactive interface by the user, for example, the preset operation may include swiping, clicking, double-clicking, long-pressing, hard pressing, air gesture, line-of-sight focus movement, etc. At this time, it is first necessary to perform picture loading processing on the picture to be displayed to obtain the picture data to be displayed. Among them, the picture data to be displayed may be a local picture file of the mobile phone or a picture data stream downloaded. The format of the picture data may be in formats such as PNG, JPEG, or Stream. Among them, the local picture file may be a picture file read by the system from the local storage of the mobile phone through the input / output port, and the picture data stream may be picture data downloaded from the network through a Uniform Resource Locator (URL).

[0076] Next, perform picture decoding processing. This part of the processing process may be a process of an application in the mobile phone displaying a picture as shown in Figure 5 . When the application needs to display a picture, the bitmap data obtained by the decoder through decoding processing is not stored in the memory partition of the CPU, but directly accelerated in parallel through the texture management module. The bitmap data is encapsulated into texture data, and the generated texture data is stored in the memory partition of the GPU. Thus, it saves the memory space of the CPU occupied by bitmap decoding in the prior art for picture display, and eliminates the time-consuming problem caused by uploading texture data, making the user view the picture smoothly and improving the user experience.

[0077] Finally, the GPU rendering operation may specifically be that the GPU receives the rendering instruction sent by the CPU, performs rendering processing on the texture data in the GPU memory partition, and the generated rendered data is stored in the memory of the GPU for the next display processing.

[0078] Based on understanding the principles of this application, the technical solutions of this application will be further elaborated below.

[0079] As Figure 6 shown, the picture processing method may include the following steps 601 - 603:

[0080] 601: The electronic device displays an interactive interface of an application. When a preset operation on the interactive interface by the user is detected, obtain the picture data of the picture to be displayed, create a decoder, and define a memory allocator. This memory allocator is used to apply for a memory partition accessible by the Graphics Processing Unit (GPU), and this GPU - accessible memory partition is used to store texture data.

[0081] It should be noted that the decoder includes a software decoder and a hardware decoder. This embodiment takes the software decoder as an example for illustration, which does not constitute a limitation to this application.

[0082] This application embodiment is described by taking the electronic device as a mobile phone as an example.

[0083] Exemplarily, when a certain application is running, the mobile phone displays the interactive interface of the application. If the mobile phone detects a sliding operation on the interactive interface by the user, the picture content displayed on the screen needs to be refreshed. As Figure 7 shown, after the touch hardware in the hardware layer of the mobile phone receives the user's trigger operation, it sends the input event to the touch input in the kernel layer. Then, this input event goes through the event reading and event distribution of the system library. The event reading is responsible for reading the input event from the touch input, and the event distribution is responsible for distributing the input event to the window management in the application framework layer. Window management is mainly used to distribute the input event to different processing modules. For example, the picture display event of this application will be distributed by the window management to the interface management for processing. The interface management passes the event to the corresponding interface display area through event transmission. The interface management receives the event of this picture display and executes the above - mentioned Figure 5 shown processing process. At this time, the window management in the application framework layer will receive the input event and obtain the picture data. It can be understood that the window management receives the input event of the trigger operation for picture display, and this input event is used to trigger the acquisition of the picture data to be displayed.

[0084] Among them, the picture data to be displayed includes the picture files stored locally on the mobile phone or the picture data stream downloaded through the URL. For example, when the user clicks on a certain application "Gallery" on the mobile phone, and the mobile phone needs to display the thumbnail of the pictures in the gallery, it triggers the display scenario of the locally stored pictures. Another example is when the application is WeChat, and the user slides their finger on the screen, causing the picture content displayed in the WeChat display interface to change. This picture content can download the picture data stream through the URL to display the picture after the sliding operation.

[0085] When creating a software decoder, such as Figure 7 , specifically, when a trigger operation for picture display is obtained, the application will call the decoding module in the system library through the decoding processing interface to complete the creation of the software decoder. When calling the decoding module, the system will simultaneously define a memory allocator by calling the memory management, implementing the definition of the memory allocator belonging to the decoding module. That is to say, when triggering the decoding of the software decoder, the memory allocation attributes of the data generated after decoding are defined. In the embodiment of the present application, the data obtained after the decoding module decodes the picture data is texture data. Therefore, the memory partition allocated by the memory allocator defined in the present application is a memory partition that can be accessed by the GPU and stores the texture data.

[0086] In the embodiment of the present application, two new classes are added to the Android runtime and the system library of the software structure, or two new functions are added, including a first function and a second function, to implement the process of memory allocation and management in picture decoding processing. Exemplarily, the two new functions can be TextureImageAllocator and TextureImage. Among them, TextureImageAllocator can be used to implement the application for allocating a memory partition accessible by the GPU; TextureImage can be used to indicate that this memory partition is used to store texture data.

[0087] Exemplarily, the specific execution process of the relevant software can be as follows Figure 8 As shown, the CPU calls the decoding interface through the entry implemented by the decoder bottom layer, selects to call the memory allocator TextureImageAllocator of the texture data type, and then calls the class TextureImage that stores the texture data to start applying for a memory partition from the internal memory. The CPU then calls the interface for starting to allocate the memory partition through the class that stores the texture data, and points the memory allocator of the texture data type to the corresponding memory allocation interface to start allocating the memory partition. After the internal memory allocates the memory partition, the CPU obtains the physical address of the allocated memory partition and points the physical address of this memory partition to the memory partition that stores the decoded data.

[0088] 602: The electronic device allocates a memory partition accessible by the GPU.

[0089] The electronic device allocates a memory partition. The memory allocator is used to call the first interface to apply for a memory partition accessible by the GPU from the internal memory. The first interface can be a standard interface for the GPU to apply for a memory partition. The memory partition accessible by the GPU includes the physical address range accessible by the GPU and the size of the memory partition accessible by the GPU.

[0090] The specific process of allocating memory partitions can be that the system calls the memory application interface of the GPU. This interface is defined and encapsulated by the system and can be used to apply for and respond to memory allocation to achieve the allocation of GPU memory. What is obtained after the application is the DDR address space authorized for GPU access, and the response data can include pointer data for storing memory addresses. This application avoids the problem of memory waste caused by allocating two pieces of memory in the prior art.

[0091] For example, the memory management calls the standard memory application interface of the GPU to apply for a memory partition from the DDR memory. The DDR memory feeds back the size of the allocated memory partition and the memory pointer indicating the physical address of the memory partition to the GPU. The data storage type of this memory partition is texture data, etc.

[0092] 603: The electronic device decodes the picture data to be displayed into bitmap data, encapsulates the bitmap data into texture data, and stores the texture data in the memory partition accessible by the GPU.

[0093] The picture data is decoded line by line to generate bitmap data. The system directly performs data type conversion processing on the generated bitmap data line by line to generate texture data, which is stored in the memory partition accessible by the GPU applied for in step 602.

[0094] In the embodiment of this application, a library for parallel converting texture data is newly added at the Android running and system library layers, denoted as the texture conversion libtexture dynamic library. This dynamic library can convert bitmap data into texture data. The libtexture dynamic library in the embodiment of this application can dynamically generate a conversion function according to the data types between the generated bitmap data and texture data, and implement this conversion function through the libtexture dynamic library. By adopting the method of parallel processing and acceleration library, texture data is directly generated.

[0095] The specific execution process of the relevant software can be as follows Figure 8 As shown, the CPU calls the decoding function (for example, the libjpeg decoding function) and starts to scan and decode line by line. After the picture data is decoded line by line to generate bitmap data; then the CPU calls texture conversion (for example, the libtexture dynamic library), and converts the bitmap data into texture data through parallel acceleration processing line by line, and stores the generated texture data in the applied memory space. Then the CPU continues to execute the processing of the next line according to the above process.

[0096] Exemplarily, the CPU calls the decoding function libjpeg to decode the first row of picture data and generate bitmap data corresponding to the first row of picture data; the CPU calls the texture conversion dynamic library libtexture to perform data type conversion processing on the bitmap data corresponding to the first row of picture data and generate texture data corresponding to the first row of picture data. The texture conversion dynamic library includes a conversion function for converting bitmap data into texture data; then the CPU performs the above processing on the second row of picture data of the picture data to be displayed until the last row of picture data of the picture data to be displayed is processed.

[0097] In the embodiment of the present application, by utilizing the basic functions of the existing decoder, the functions and algorithms for encapsulating textures are extended. Specifically, on the basis of the libjpeg decoding function and based on the existing decoding capabilities, the libtexture dynamic library is added. The data generated by libjpeg decoding is subjected to parallel decoding processing and data type conversion to generate texture data, thereby solving the problem of time-consuming texture conversion and uploading processes in the prior art and the resulting stuttering phenomenon, and improving the user experience.

[0098] 604: The electronic device triggers the GPU to read the texture data and perform rendering processing to obtain the rendered data; stores the rendered data in the memory partition of the display, and triggers the display to display the picture according to the rendered data.

[0099] Further, the GPU can perform rendering processing according to the texture data and the rendering instruction to obtain the rendered data. As Figure 7 shown, the rendered data saves the rendered data in the memory of the GPU. After the data composition (SurfaceFlinger) for display processing obtains this memory of the GPU, it is stored in the memory of the liquid crystal display (LCD) through the compositor hardware abstraction layer and the display driver, and is displayed by the liquid crystal display LCD.

[0100] In the embodiment of the present application, by adding and improving in the application framework layer and system library of the system, the direct parallel processing of picture data after decoding is realized, texture data is generated, and the texture data is stored in the memory partition allocated for the GPU, so that the GPU can perform rendering processing according to the texture data, solving the problem of waste of applying two pieces of memory for the CPU and GPU in the prior art, and at the same time solving the problem of time-consuming data copying from the memory of the CPU to the memory of the GPU during the texture uploading process in the prior art, improving the stuttering phenomenon and enhancing the user experience.

[0101] Some other embodiments of the present application provide an electronic device, which may include: a memory and one or more processors, and the memory is coupled to the processors. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processors execute the computer instructions, the electronic device can perform each function or step in the above method embodiments. The structure of the electronic device may refer to Figure 1 the structure of the electronic device 100 shown.

[0102] Embodiments of the present application further provide a chip system, which can be applied to the electronic device in the above embodiments, such as Figure 9 shown. The chip system includes at least one processor 901 and at least one interface circuit 902. The processor 901 and the interface circuit 902 can be interconnected through a line. For example, the interface circuit 902 can be used to receive signals from other devices (such as the memory of the electronic device). For another example, the interface circuit 902 can be used to send signals to other devices (such as the processor 901). Exemplarily, the interface circuit 902 can read the instructions stored in the memory and send the instructions to the processor 901. When the instructions are executed by the processor 901, the electronic device can perform each function or step performed by the electronic device in the above embodiments. Of course, the chip system may further include other discrete devices, and the embodiments of the present application do not make specific limitations thereto.

[0103] Embodiments of the present application further provide a computer storage medium, which includes computer instructions. When the computer instructions run on the above electronic device, the electronic device performs each function or step performed by the mobile phone in the above method embodiments.

[0104] Embodiments of the present application further provide a computer program product. When the computer program product runs on a computer, the computer performs each function or step performed by the mobile phone in the above method embodiments.

[0105] Through the description of the above embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0106] In several embodiments provided by this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections between each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0107] The units described as separate components may or may not be physically separated. The components displayed as units may be one physical unit or multiple physical units, that is, they can be located in one place, or they can be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0108] In addition, in each embodiment of this application, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0109] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in the embodiments of this application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks or optical discs that can store program codes.

[0110] Finally, it should be noted that: the above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. A method for image processing, characterized in that, Applied to an electronic device, the method includes: Display an interactive interface of an application. When a preset operation by the user on the interactive interface is detected, decode the picture data to be displayed by the application into bitmap data, and encapsulate the bitmap data as texture data through a conversion function that converts the bitmap data into texture data; Store the texture data in a memory partition accessible by a Graphics Processing Unit (GPU); Trigger the GPU to read the texture data and perform rendering processing to obtain the rendered data; Trigger the display to display a picture according to the rendered data; The step of decoding the picture data to be displayed by the application into bitmap data and encapsulating the bitmap data as texture data through a conversion function that converts the bitmap data into texture data includes: Decode the picture data of the first row of the picture data to be displayed to generate bitmap data of the first row of the picture data, perform data conversion on the bitmap data of the first row of the picture data to generate texture data of the first row of the picture data; then perform the above processing on the second row data of the picture data to be displayed until the picture data of the last row of the picture data to be displayed is processed.

2. The method according to claim 1, wherein Before decoding the picture data to be displayed by the application into bitmap data, the method further includes: Create a software decoder; Define a memory allocator that is used to apply for a memory partition accessible by the GPU, and the memory partition accessible by the GPU is used to store the texture data.

3. The method according to claim 2, wherein The memory allocator is used to apply for a memory partition accessible by the GPU includes: The memory allocator is used to call a first interface to apply for the memory partition accessible by the GPU from the internal memory. The first interface is a standard interface for the GPU to apply for a memory partition, and the memory partition accessible by the GPU includes the physical address range accessible by the GPU and the size of the memory partition accessible by the GPU.

4. The method according to claim 1, characterized in that, Decode the picture data of the first row of the picture data to be displayed to generate bitmap data of the first row of the picture data, perform data conversion on the bitmap data of the first row of the picture data to generate texture data of the first row of the picture data; Then performing the above processing on the second row data of the picture data to be displayed until the picture data of the last row of the picture data to be displayed is processed includes: Call a decoding function to perform decoding processing on the first row of picture data to generate bitmap data corresponding to the first row of picture data; Call a texture conversion dynamic library to perform data type conversion processing on the bitmap data corresponding to the first row of picture data to generate texture data corresponding to the first row of picture data. The texture conversion dynamic library includes a conversion function that converts bitmap data into texture data; Then perform the above processing on the second row of picture data of the picture data to be displayed until the picture data of the last row of the picture data to be displayed is processed.

5. An electronic device, characterized in that, The electronic device includes: a memory and one or more processors; the memory is coupled to the processor; the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device performs the following operations: Display an interaction interface of an application. When a preset operation on the interaction interface by the user is detected, decode the picture data to be displayed by the application into bitmap data, and encapsulate the bitmap data into texture data through a conversion function that converts the bitmap data into texture data; Store the texture data in a memory partition accessible by a graphics processing unit (GPU); Trigger the GPU to read the texture data and perform rendering processing to obtain rendered data; Trigger the display to display a picture according to the rendered data; The decoding the picture data to be displayed by the application into bitmap data, and encapsulating the bitmap data into texture data through a conversion function that converts the bitmap data into texture data includes: Decode the picture data of the first row of the picture data to be displayed to generate bitmap data of the first row of the picture data, perform data conversion on the bitmap data of the first row of the picture data to generate texture data of the first row of the picture data; then perform the above processing on the second row data of the picture data to be displayed until the picture data of the last row of the picture data to be displayed is processed.

6. The electronic device according to claim 5, wherein The electronic device is further used to perform the following operations: Create a software decoder; Define a memory allocator, and the memory allocator is used to apply for a memory partition accessible by the GPU, and the memory partition accessible by the GPU is used to store the texture data.

7. The electronic device according to claim 6, wherein The memory allocator is used to apply for the memory partition accessible by the GPU, specifically including: The memory allocator is used to call a first interface to apply for the memory partition accessible by the GPU from the internal memory. The first interface is a standard interface for the GPU to apply for a memory partition, and the memory partition accessible by the GPU includes the physical address range accessible by the GPU and the size of the memory partition accessible by the GPU.

8. The electronic device according to claim 5, wherein Decode the picture data of the first row of the picture data to be displayed to generate bitmap data of the first row of the picture data, perform data conversion on the bitmap data of the first row of the picture data to generate texture data of the first row of the picture data; Then perform the above processing on the second row data of the picture data to be displayed until the picture data of the last row of the picture data to be displayed is processed, including: Call a decoding function to perform decoding processing on the first row of picture data to generate bitmap data corresponding to the first row of picture data; Call a texture conversion dynamic library to perform data type conversion processing on the bitmap data corresponding to the first row of picture data to generate texture data corresponding to the first row of picture data. The texture conversion dynamic library includes a conversion function that converts bitmap data into texture data; Then, the above processing is performed on the second row of picture data of the picture data to be displayed until the picture data of the last row of the picture data to be displayed is processed.

9. A chip system, characterized in that, The chip system is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected by lines; the interface circuits are used to receive signals from the memory of the electronic device and send the signals to the processors, and the signals include computer instructions stored in the memory; when the processors execute the computer instructions, the electronic device executes the picture processing method according to any one of claims 1-4.

10. A readable storage medium, characterized in that, Instructions are stored in the readable storage medium, and when the readable storage medium runs on an electronic device, the electronic device is caused to execute the picture processing method according to any one of claims 1-4.

11. A computer program product, characterized in that, When the computer program product runs on a computer, the computer is caused to execute the picture processing method according to any one of claims 1-4.

Images