Rendering method and electronic device

Abstract

The application provides a rendering method and an electronic device, relates to the technical field of computer graphics, and can effectively reduce the single-frame rendering time length, guarantee the screen refresh rate, and improve the user experience by means of parallel rendering. The method comprises the following steps: an electronic device acquires a rendering object to be displayed, the rendering object comprising a control and a layout. A control tree comprising the rendering object to be displayed is acquired, and the control tree is traversed from a root node of the control tree: different control tree levels are set as different rendering levels, and intersecting rendering objects of the same control tree level are set as different rendering levels. Then, the rendering objects of the same rendering level are rendered in parallel, and the rendering objects of different rendering levels are rendered in series.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer graphics, and particularly relate to a rendering method and an electronic device. BACKGROUND

[0002] With the development of terminal technology, electronic devices (such as mobile phones, large-screen devices, etc.) can provide better display experience for users. For example, some electronic devices are now configured with a display with a refresh rate of 144 hertz (Hz) per second. Compared with a general display with a refresh rate of 60 Hz, a higher refresh rate can make the interface operation more smooth.

[0003] Generally, in order to ensure the refresh rate, the single-frame rendering duration of the display picture needs to be reduced, otherwise it is difficult to ensure the full-frame display effect of the display with a high refresh rate, and the phenomenon of frame loss and lagging is likely to occur, affecting the user experience. For a display with a refresh rate of 144 Hz, the single-frame rendering duration needs to be less than 6 ms, and in the future, there may be a demand for a higher refresh rate, which requires a shorter single-frame rendering duration. However, at present, it is already difficult to complete single-frame rendering within 6 ms for a display picture in a complex scene. SUMMARY

[0004] The rendering method and the electronic device provided by the embodiments of the present application can effectively reduce the single-frame rendering duration by means of parallel rendering, ensure the screen refresh rate, and improve the user experience.

[0005] To achieve the above-mentioned purpose, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, the embodiments of the present application provide a rendering method applied to an electronic device, which can include: the electronic device acquires a rendering object to be displayed, the rendering object including a control and a layout, and acquires a control tree including the rendering object to be displayed. Starting from a root node of the control tree, the control tree is traversed: different control tree levels are set as different rendering levels, and intersecting rendering objects of the same control tree level are set as different rendering levels; the rendering objects of the same rendering level are rendered in parallel, and the rendering objects of different rendering levels are rendered in series.

[0007] In some embodiments, the control tree is used to represent the controls and layouts organized in a tree structure. The control tree level is used to represent the tree node relationship of the controls and layouts in the control tree after the control tree is traversed. Each node in the control tree corresponds to a rendering object (i.e., a control or a layout), and each control tree level contains one or more rendering objects (i.e., including one or more controls and one or more layouts). Therefore, after the control tree level is acquired, it is necessary to determine whether the rendering objects (i.e., controls and layouts) of the same control tree level in the control tree intersect, so as to divide the rendering levels according to the intersection.

[0008] In some embodiments, the serial rendering comprises sequentially rendering the rendering objects according to the rendering level order by calling threads; and the parallel rendering comprises calling multiple rendering threads to simultaneously perform rendering tasks of the same rendering level.

[0009] In this way, the method provided by the embodiments of the present application can effectively reduce the rendering time and avoid the phenomenon of frame loss in the displayed image. That is, the parallel rendering method can reduce the possibility of frame loss in the display of the electronic device, ensure the smoothness of the display screen, and improve the user's visual experience.

[0010] In a possible implementation, the parallel rendering of the rendering objects of the same rendering level comprises: parallel rendering of constituent element data in the same instruction group in the rendering objects of the same rendering level, and serial rendering of constituent element data in different instruction groups in the rendering objects of the same rendering level.

[0011] In a possible implementation, the constituent element data comprises one or more of the following: foreground layer constituent element data, content layer constituent element data, and background layer constituent element data.

[0012] The background layer constituent element data corresponds to the background of the control or layout after rendering, the content layer constituent element data corresponds to the display content of the control or layout after rendering, and the foreground layer constituent element data corresponds to the foreground of the control or layout after rendering. For example, if the control is a text control, the content layer constituent element data of the control comprises corresponding text data.

[0013] In some embodiments, after the electronic device completes the division of the rendering levels, the data of different rendering levels can be rendered according to the rendering level order, but the rendering order of the constituent element data in the same rendering level needs to be further determined. Optionally, the constituent element data in the same rendering level is grouped into instruction groups, and the control and layout are rendered according to the grouping result of the instruction groups.

[0014] In this way, the data of the rendering objects is further classified according to the constituent element data, which can further optimize the parallel rendering tasks performed in the same rendering level. The data of the rendering objects comprises the following cases: if the rendering object is a control, the data of the rendering object is control data; and if the rendering object is a layout, the data of the rendering object is layout data.

[0015] In a possible implementation, the data categories of the constituent element data in the same instruction group are the same, and the data categories of the constituent element data in different instruction groups are different; or, the instruction execution orders of the constituent element data in the same instruction group in the corresponding rendering object are the same, and the instruction execution orders of the constituent element data in different instruction groups in the corresponding rendering object are different.

[0016] In some embodiments, the background layer constituent element data in the same rendering hierarchy is divided into one instruction group, the content layer constituent element data is divided into one instruction group, and the foreground layer constituent element data is divided into one instruction group. For example, the background data in the same rendering hierarchy is divided into one instruction group, and the text data is divided into one instruction group. Optionally, the instruction group grouping result is marked in the corresponding control data or layout data, and then in the rendering process, the background layer constituent element data instruction group, the content layer constituent element data instruction group, and the foreground layer constituent element data instruction group in the same rendering hierarchy can be rendered in sequence according to the marking.

[0017] In this way, the rendering object data is quickly divided according to the category of the constituent element data, the data of the instruction group that can be rendered in parallel is determined, and the rendering efficiency is improved.

[0018] In a possible implementation, the intersecting rendering objects in the same control tree hierarchy are set as different rendering hierarchies, including: obtaining the minimum rectangular region corresponding to the rendering objects in the same control tree hierarchy; and setting the rendering objects intersecting the minimum rectangular region in the same control tree hierarchy as different rendering hierarchies.

[0019] In some embodiments, the instruction execution order includes, for example, the original rendering order of the constituent element data of the control data and the layout data. Specifically, the original rendering order of the constituent element data of the control data or the layout data is generally to render the background layer constituent element data first, then render the content layer constituent element data, and finally render the foreground layer constituent element data. If the control data or the layout data does not include the constituent element data of a layer or several layers, the data of the next layer can be directly rendered by skipping the data of the layer. Then, according to the instruction execution order, the data with the same instruction execution order in the same rendering hierarchy is divided into the same instruction group, and the data with different instruction execution orders is divided into different instruction groups. Optionally, the instruction group grouping result is marked in the corresponding control data or layout data, and then in the rendering process, the rendering is performed according to the marked instruction group grouping result and the instruction group execution order.

[0020] In this way, the problem of rendering thread suspension and waiting in the case where the data of the rendering object does not include the constituent element data of a certain category is avoided, and the unnecessary rendering waiting time is reduced.

[0021] In a possible implementation, the minimum rectangle area corresponding to the rendering object is a rectangular area covering the rendering object and having a minimum area.

[0022] In some embodiments, during the drawing of the controls and the layout, the controls and the layout have corresponding drawing areas, and if the drawing areas partially or wholly intersect, it indicates that the corresponding controls and the layout intersect. Optionally, the corresponding drawing areas can be obtained according to the control data and the layout data, and the drawing area of a control or a layout can be represented by a minimum rectangular area, for example, a minimum rectangular area that can cover the corresponding control or layout and has a minimum area.

[0023] In this way, the electronic device determines the intersection of the rendering objects according to the intersection of the minimum rectangular areas.

[0024] In a possible implementation, before rendering the rendering objects of the same rendering level in parallel, the method further includes: obtaining a resource condition. According to the resource condition, the maximum number of rendering threads allowed to be called in parallel is determined.

[0025] In some embodiments, the electronic device allocates rendering threads to perform data rendering according to the resource condition of the CPU and the GPU. For example, the UI thread generates rendering instructions to trigger the rendering threads to perform data rendering according to the rendering level division result and the instruction group grouping result, and the rendering threads perform data rendering after receiving the rendering instructions.

[0026] In a possible implementation, rendering the rendering objects of the same rendering level in parallel includes: if the first number of rendering threads required for rendering the rendering objects of the same rendering level in parallel is less than or equal to the maximum number, calling the first number of rendering threads to render the rendering objects of the same rendering level in parallel. If the first number is greater than the maximum number, calling the maximum number of rendering threads to render the rendering objects of the same rendering level in parallel.

[0027] In some embodiments, after the electronic device completes the instruction group grouping of the data to be rendered, the electronic device determines whether the number of rendering threads allowed to be called in parallel meets the rendering requirement of the instruction group according to the resource condition of the CPU and the GPU. If the number of threads allowed to be called in parallel meets the rendering requirement of the instruction group, the electronic device calls the corresponding number of rendering threads to render the data in the instruction group in parallel. If the number of threads allowed to be called in parallel does not meet the rendering requirement of the instruction group, the electronic device calls the maximum number of rendering threads to render part of the data in the instruction group in parallel, and renders all the data in the instruction group in batches.

[0028] In this way, the electronic device determines the maximum number of rendering threads allowed to be called according to the resource condition before performing the rendering task, ensures the smooth execution of the parallel rendering task, and avoids rendering exceptions.

[0029] In a second aspect, an electronic device is provided. The electronic device includes a processor and a memory. The memory is coupled to the processor and is configured to store computer program codes. The computer program codes include computer instructions. When the processor reads the computer instructions from the memory, the electronic device is caused to perform the following operations. A rendering object to be displayed is obtained. The rendering object includes controls and layouts. A control tree including the rendering object to be displayed is obtained. The control tree is traversed from a root node of the control tree. Different control tree levels are set as different rendering levels. Intersecting rendering objects in a same control tree level are set as different rendering levels. Rendering objects in a same rendering level are rendered in parallel. Rendering objects in different rendering levels are rendered in series.

[0030] In a possible implementation, the rendering objects in a same rendering level are rendered in parallel. The rendering objects in a same rendering level are rendered in parallel includes rendering constituent element data in a same instruction group in a same rendering level in parallel and rendering constituent element data in different instruction groups in a same rendering level in series.

[0031] In a possible implementation, the constituent element data includes one or more of the following: foreground layer constituent element data, content layer constituent element data, and background layer constituent element data.

[0032] In a possible implementation, the constituent element data in a same instruction group is of a same data category, and the constituent element data in different instruction groups is of different data categories. Alternatively, the constituent element data in a same instruction group is of a same execution order of corresponding instructions in a corresponding rendering object, and the constituent element data in different instruction groups is of different execution orders of corresponding instructions in a corresponding rendering object.

[0033] In a possible implementation, the intersecting rendering objects in a same control tree level are set as different rendering levels includes obtaining a minimum rectangular region corresponding to the rendering objects in a same control tree level, and setting the rendering objects in a same control tree level that intersect the minimum rectangular region as different rendering levels.

[0034] In a possible implementation, the minimum rectangular region corresponding to the rendering objects is a rectangular region that covers the rendering objects and has a minimum area.

[0035] In a possible implementation, when the processor reads the computer instructions from the memory, the electronic device is further caused to perform the following operations. A resource condition is obtained. According to the resource condition, a maximum number of rendering threads allowed to be called in parallel is determined.

[0036] In a possible implementation, rendering the rendering objects of the same rendering level in parallel includes: if a first number of rendering threads required for rendering the rendering objects of the same rendering level in parallel is less than or equal to a maximum number, invoking the first number of rendering threads to render the rendering objects of the same rendering level in parallel; or if the first number is greater than the maximum number, invoking a maximum number of rendering threads to render the rendering objects of the same rendering level in parallel.

[0037] In addition, the technical effects of the electronic device in the second aspect can refer to the technical effects of the rendering method in the first aspect, which will not be described here.

[0038] In a third aspect, an embodiment of the present application provides an electronic device having a function of implementing the rendering method in the first aspect and any possible implementation manner thereof. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium including computer instructions, when the computer instructions are executed on an electronic device, the electronic device executes the rendering method in the first aspect and any possible implementation manner thereof.

[0040] In a fifth aspect, an embodiment of the present application provides a computer program product, when the computer program product is executed on an electronic device, the electronic device executes the rendering method in the first aspect and any possible implementation manner thereof.

[0041] In a sixth aspect, an embodiment of the present application provides circuitry including processing circuitry configured to perform the rendering method in the first aspect and any possible implementation manner thereof.

[0042] In a seventh aspect, an embodiment of the present application provides a chip system including at least one processor and at least one interface circuit, the at least one interface circuit is configured to perform a transceiving function and send instructions to the at least one processor, when the at least one processor executes the instructions, the at least one processor executes the rendering method in the first aspect and any possible implementation manner thereof. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 The interface schematic diagram provided by the embodiment of the present application;

[0044] Figure 2 The morphological schematic diagram of the electronic device provided by the embodiment of the present application;

[0045] Figure 3Schematic diagram of the structure of the electronic device provided in the embodiments of this application Figure 1 ;

[0046] Figure 4 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0047] Figure 5 A flowchart of the rendering method provided in the embodiments of this application;

[0048] Figure 6a A schematic diagram of the control tree structure provided in the embodiments of this application. Figure 1 ;

[0049] Figure 6b A schematic diagram of the control tree structure provided in the embodiments of this application. Figure 2 ;

[0050] Figure 7 This is a schematic diagram of a layered scene with rendering hierarchy provided in an embodiment of this application;

[0051] Figure 8 This application provides a schematic diagram of instruction grouping scenarios in its embodiments. Figure 1 ;

[0052] Figure 9 This application provides a schematic diagram of instruction grouping scenarios in its embodiments. Figure 2 ;

[0053] Figure 10a A schematic diagram of the image frame refresh cycle provided in the embodiments of this application. Figure 1 ;

[0054] Figure 10b A schematic diagram of the image frame refresh cycle provided in the embodiments of this application. Figure 2 ;

[0055] Figure 11 Schematic diagram of the structure of the electronic device provided in the embodiments of this application Figure 2 ;

[0056] Figure 12 Schematic diagram of the structure of the electronic device provided in the embodiments of this application Figure 3 . Detailed Implementation

[0057] The rendering method and electronic device provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0058] The terms "comprising" and "having," and any variations thereof, used in the description of the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0059] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0060] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone.

[0061] First, for ease of understanding, the relevant terms and concepts involved in the embodiments of this application will be introduced below.

[0062] (1) Controls and control data

[0063] A view, also known as a widget in some implementations, is a user interface element that serves as a human-computer interaction interface, receiving user input on an electronic device. For example, an electronic device receives user-input commands by detecting user interactions with views. Views can include text views, buttons, image views, progress bars, etc.

[0064] For example, such as Figure 1 As shown in (a), the main interface 101 displays application icons. These application icons, as controls, can be used to launch applications. Here, the application icons are equivalent to the controls of a launcher application. A launcher is a system application that displays the user's phone desktop, including displaying and managing application icons and various desktop controls. For example, after the phone detects the user clicking control 1011 (taking the "Settings" application as an example), it displays... Figure 1The interface 102 shown in (b) is an interface of the electronic device. As shown in the interface 102, the controls can include a picture control 1021, a text control 1022, a button 1023, and the like.

[0065] The control data is generated by an application including the control. For example, the application icon control of the launcher application described above is a control used by the launcher to display and facilitate the user to open the installed other application. Therefore, the control data is obtained and generated by the launcher according to the installed other application, and the data of the control can also be changed in the launcher, for example, the user can adjust the position and size of the control, and the like. The electronic device performs drawing of the control according to the control data to display the control visible to the user.

[0066] (2) Layout and layout data

[0067] The layout is used to indicate the position and size of a display area, and the display position effect of the control in the display area. The types of the layout include linear layout, relative layout, absolute layout, and the like. Among them, the linear layout is used to represent that the controls are displayed in a linear arrangement according to a horizontal or vertical direction or the like. The relative layout is used to represent the display position of the control relative to other controls. The absolute layout is used to represent the display position of the control in the display interface. For example, the layout of the control A includes that the center pixel point of the control A coincides with the center point of the display screen, and the size is 200 pixels, so the layout of the control A is the absolute layout. The layout of the control B includes that the center pixel point of the control B is located at the left side of the center pixel point of the control A by N pixel points, and the size of the control B is the same as that of the control A, so the layout of the control B is the relative layout.

[0068] For example, as shown in the interface 102 of (b), the display position of each setting option such as Bluetooth, mobile network, desktop, and wallpaper corresponds to the type of the layout, which includes the linear layout. Figure 1

[0069] The layout data includes the information of the position and size of a display area, the control contained in the display area, and the type of the layout corresponding to the control, and the like.

[0070] In some embodiments, the process of the electronic device displaying the image includes at least a user interface (UI) thread and a rendering thread. Among them, the UI thread is used to control the display, update, and control interaction of the UI interface, and can generate graphics data and rendering instructions. The rendering thread completes the rendering work of the control data, the layout data, and the like according to the graphics data and the rendering instructions, and generates an image frame.

[0071] ​It should be noted that the number of UI threads and rendering threads called by the electronic device in the graphic rendering process is one or more. The UI thread in the embodiment of the present application can also be described as a UI process, a UI task, etc. in different operating systems. The rendering thread in the embodiment of the present application can also be described as a rendering process, a rendering task, etc. in different operating systems. The following will not be described.

[0072] In some embodiments, the display frame rate represents the smoothness of the electronic device displaying images. The higher the display frame rate, the smoother the display of images, and the more realistic the display effect can be achieved. At present, users have higher and higher requirements for the display effect of electronic devices. In order to ensure the display effect, it is necessary to improve the display frame rate, which requires improving the image generation rate. Among them, the rendering time of control data, layout data and the like has a great influence on the image generation rate.

[0073] For example, if the electronic device needs to achieve a high display frame rate of 144HZ, it needs to ensure that the single-frame image rendering time is less than 6ms. However, in a complex scenario, such as when there are many controls included in a single-frame image, it is difficult to ensure that the single-frame image rendering time is less than 6ms. The electronic device cannot complete the rendering of the current image frame within the rendering period, which causes the image frame to be unable to be synthesized, resulting in a phenomenon of frame freezing (i.e. frame loss) in the display of the electronic device.

[0074] Based on this, the embodiment of the present application provides a rendering method, which effectively reduces the rendering time by layering and parallel rendering of control data and layout data to be rendered, meets the requirements of high display frame rate on image generation rate, and avoids the phenomenon of frame freezing in the display of the electronic device.

[0075] The rendering method provided by the embodiment of the present application can be applied to an electronic device. For example, as shown in the figure, Figure 2 The electronic device can be a large-screen display device 21, a mobile phone 22, a notebook computer 23, a tablet computer 24, a vehicle-mounted device 25, a wearable device (such as a smart watch 26), an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), an artificial intelligence device, etc. The specific type of the electronic device is not limited in the embodiment of the present application.

[0076] For example, Figure 3 The structure of the electronic device 100 is shown.

[0077] The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, and the like.

[0078] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than illustrated, or combine certain components, or split certain components, or different arrangement of components. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0079] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), and the like. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0080] The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching instructions and executing instructions.

[0081] The memory in the processor 110 can also be provided for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can save instructions or data that the processor 110 has just used or repeatedly uses. If the processor 110 needs to use the instructions or data again, it can be directly called from the memory. Avoiding repeated access, reducing the waiting time of the processor 110, thus improving the efficiency of the system.

[0082] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0083] The I2C interface is a bidirectional synchronous serial bus including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 can include multiple sets of I2C bus. The processor 110 can be coupled to a touch sensor, a charger, a flash, a camera 193, etc. through different I2C bus interfaces respectively. For example, the processor 110 can be coupled to a touch sensor through an I2C interface, so that the processor 110 and the touch sensor communicate through the I2C bus interface to realize the touch function of the electronic device 100.

[0084] The MIPI interface can be used to connect the processor 110 and peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate through the CSI interface to realize the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate through the DSI interface to realize the display function of the electronic device 100.

[0085] The USB interface 130 is an interface conforming to the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transmit data between the electronic device 100 and a peripheral device. It can also be used to connect a headset to play audio through the headset. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0086] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the electronic device 100. In some other embodiments of the present application, the electronic device 100 can also use different interface connection methods or combinations of multiple interface connection methods.

[0087] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. The charging management module 140 can also supply power to the electronic device while charging the battery 142 through the power management module 141.

[0088] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to supply power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160, etc.

[0089] The wireless communication function of the electronic device 100 can be realized through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc.

[0090] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0091] The mobile communication module 150 can provide a solution for wireless communication including 2G / 3G / 4G / 5G, etc. applied to the electronic device 100. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive an electromagnetic wave by the antenna 1, and perform filtering, amplification, etc. on the received electromagnetic wave, and transfer the processed signal to the modem processor to be demodulated. The mobile communication module 150 can also amplify a signal modulated by the modem processor, and radiate the signal as an electromagnetic wave through the antenna 1. In some embodiments, at least part of the function modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least part of the function modules of the mobile communication module 150 can be disposed in the same device as at least part of the modules of the processor 110.

[0092] The wireless communication module 160 can provide a solution for wireless communication including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) network), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrated with at least one communication processing module. The wireless communication module 160 receives an electromagnetic wave via the antenna 2, performs frequency modulation and filtering on the electromagnetic wave signal, and transmits the processed signal to the processor 110. The wireless communication module 160 can also receive a signal to be transmitted from the processor 110, perform frequency modulation and amplification on the signal, and radiate the signal as an electromagnetic wave through the antenna 2.

[0093] In some embodiments, the antenna 1 and the mobile communication module 150 of the electronic device 100 are coupled, and the antenna 2 and the wireless communication module 160 are coupled, so that the electronic device 100 can communicate with a network and other devices through wireless communication technology. The wireless communication technology can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include a global positioning system (GPS), a global navigation satellite system (GLONASS), a beidu navigation satellite system (BDS), a quasi-zenith satellite system (QZSS), and / or a satellite based augmentation systems (SBAS).

[0094] The electronic device 100 implements a display function through a GPU, a display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs, which execute program instructions to generate or change display information.

[0095] The display screen 194 is configured to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diode (QLED), or the like. In some embodiments, the electronic device 100 can include one or N display screens 194, where N is a positive integer greater than 1.

[0096] The sensor module 180 can include a pressure sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0097] In some embodiments, the display screen 194 can be a touch screen. That is, the display screen 194 includes a touch sensor. The touch sensor can also be referred to as a “touch panel”. That is, the display screen 194 can include a display panel and a touch panel, and the touch sensor and the display screen 194 form a touch screen, also referred to as a “touch screen”. The touch sensor is configured to detect a touch operation applied thereto or in the vicinity thereof. After the touch operation is detected by the touch sensor, the touch operation can be transmitted to an upper layer by a kernel layer driver (e.g., a TP driver) to determine a touch event type. The display screen 194 can provide visual output related to the touch operation. In other embodiments, the touch sensor can be disposed on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0098] For example, after the electronic device 100 detects a user operation of long-pressing and dragging an application icon, the position of the application icon (i.e., the control) can change in each frame of image during the dragging process, and the rendering of the control data and the layout data is required.

[0099] The electronic device 100 can implement a photographing function through an ISP, a camera 193, a video codec, a GPU, a display 194, and an application processor.

[0100] The camera 193 is configured to capture still images or videos. An object projects an optical image through a lens to a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transmitted to an ISP to be converted into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into an image signal in a standard format, such as RGB, YUV, or the like. In some embodiments, the electronic device 100 can include one or N cameras 193, where N is a positive integer greater than 1.

[0101] The external memory interface 120 can be configured to connect an external memory card, such as a Micro SD card, to extend the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, music, video, and other files can be stored in the external memory card.

[0102] The internal memory 121 can be configured to store computer-executable program code including instructions. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required for a function (such as a sound playing function, an image playing function, or the like), and the like. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phonebook, or the like), and the like. The processor 110 executes various function applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121 and / or the instructions stored in the memory disposed in the processor.

[0103] The audio module 170 is configured to convert digital audio information into an analog audio signal for output, and to convert an analog audio input into a digital audio signal. The audio module 170 can also be configured to encode and decode audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or some of the functional modules of the audio module 170 can be disposed in the processor 110. The electronic device 100 can implement an audio function through the audio module 170 and an application processor, for example, music playing, voice recording, and the like. The audio module can include a speaker, a receiver, a microphone, and a headphone interface.

[0104] The keys 190 include a power key, a volume key, and the like. The keys 190 can be mechanical keys. Alternatively, the keys 190 can be touch keys. The electronic device 100 can receive a key input and generate a key signal input related to user settings and function control of the electronic device 100.

[0105] The motor 191 can generate a vibration cue. The motor 191 can be used for a call vibration cue or a touch vibration feedback. For example, a touch operation for different applications (e.g., a photograph, audio playback, and the like) can correspond to different vibration feedback effects. A touch operation for different regions of the display 194 can correspond to different vibration feedback effects. Different application scenarios (e.g., a time reminder, a received message, an alarm, a game, and the like) can correspond to different vibration feedback effects. The touch vibration feedback effects can also be customizable.

[0106] The indicator 192 can be an indicator light and can be used to indicate a charging state, a power change, a message, a missed call, a notification, and the like.

[0107] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1.

[0108] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiment of the present application takes an Android system with a layered architecture as an example to exemplarily illustrate the software structure of the electronic device 100.

[0109] Figure 4 is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0110] The layered architecture divides software into several layers, each of which has a clear role and division of labor. The layers communicate with each other through a software interface. In some embodiments, the Android system is divided into four layers, from top to bottom, an application layer, an application framework layer, a system library of an Android runtime, and a kernel layer.

[0111] The application layer can include a series of application packages.

[0112] As shown in Figure 4 , the application packages can include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, and the like.

[0113] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes some pre-defined functions.

[0114] As shown in Figure 4 the application framework layer can include a window manager, a content provider, a view system, a resource manager, a notification manager, etc.

[0115] The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, and intercept the screen, etc.

[0116] The activity manager is responsible for managing activities, starting, switching, scheduling components in the system, and managing and scheduling applications, etc.

[0117] The content provider is used to store and obtain data, and make the data accessible to applications. The data can include videos, images, audios, dialed and received calls, browsing history and bookmarks, phone books, etc.

[0118] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures.

[0119] The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc.

[0120] The notification manager enables applications to display notification information in the status bar, which can be used to convey notification type messages that can automatically disappear after a short stay without user interaction.

[0121] The Android runtime includes a core library and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0122] The core library includes two parts: one part is the function function called by the java language, and the other part is the core library of Android.

[0123] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the java files of the application layer and the application framework layer into binary files. The virtual machine is used to manage the life cycle of objects, stack management, thread management, security and exception management, and garbage collection, etc.

[0124] The system library can include a plurality of functional modules. For example, a surface manager, media libraries, a three-dimensional graphics processing library (for example, OpenGL ES), a two-dimensional graphics engine (for example, SGL), and the like.

[0125] The surface manager is used to manage a display subsystem and provides fusion of 2D and 3D layers for a plurality of application programs.

[0126] The media libraries support playback and recording of a plurality of commonly used audio, video formats, and static image files. The media libraries can support a plurality of audio and video encoding formats, for example, MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, and the like.

[0127] The three-dimensional graphics processing library is used to implement three-dimensional graphics drawing, image rendering, synthesis, and layer processing, and the like.

[0128] The two-dimensional graphics engine is a drawing engine for 2D drawing. Common graphics engines include Vulkan, Canvas, and the like.

[0129] The kernel layer is a layer between hardware and software. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver.

[0130] The user performs an input operation on the electronic device (for example, an operation of triggering the electronic device to start an application program). The kernel layer can generate a corresponding input event (for example, an application program starting event) according to the input operation and report the event to the application framework layer. The view system of the application framework layer determines control data and layout data corresponding to the application program and determines flow logic of rendering of the rendering engine, and sends the flow logic to the two-dimensional graphics engine of the system library for processing. After processing, the processed data is sent to the display driver of the kernel layer, and the display driver displays a corresponding application interface on the display screen.

[0131] Figure 5 A rendering method provided by an embodiment of the present application is shown in the figure. The method can be applied to an electronic device, for example, a mobile phone. Figure 5 As shown in the figure, the method can include S501-S503.

[0132] S501, control data and layout data in a control tree to be drawn are acquired, and control tree levels of controls and layouts in the control tree are determined.

[0133] In some embodiments, after the electronic device detects the event triggering the rendering, the electronic device obtains the rendering object to be displayed and the control tree of the rendering object to be displayed (i.e., the control tree to be drawn), and then determines the control tree level of the rendering object in the control tree. The rendering object includes a control and a layout. The data of the rendering object includes the following cases: if the rendering object is a control, the data of the rendering object is control data; if the rendering object is a layout, the data of the rendering object is layout data.

[0134] In some embodiments, the event triggering the rendering is, for example, a touch operation of a user on the display screen detected by the electronic device, such as an operation of clicking a certain interactive control, or a sliding (e.g., up and down, left and right, etc.) operation of the user on the display screen; or can be a voice input of the user detected by the electronic device; or can be an event of automatic switching of a picture of a foreground application of the electronic device, etc., which is not limited here. The foreground application is an application corresponding to an interface currently displayed on the display screen of the electronic device.

[0135] It should be noted that the picture content of the foreground application can not only include a picture visible to the user, but also include content without a user interface, content of a transparent layer, or content hidden from the user by other application interfaces, and the electronic device needs to obtain all the control data to be drawn and layout data corresponding to the picture content.

[0136] In some embodiments, the control tree (view tree) is used to represent the controls and layouts organized in a tree structure. The determination of the control tree level is to determine the tree node relationship of the controls and layouts in the control tree by traversing the control tree, and fill the determined control tree level of the controls and layouts into the corresponding control data and layout data, so as to divide the rendering levels according to the control tree level in the subsequent step S502.

[0137] It should be noted that “,” is used to represent the object association relationship of “and / or”, such as control data and layout data, which represents three cases of existence of control data alone, existence of layout data alone, and existence of both control data and layout data, which will not be described again hereinafter.

[0138] For example, it is assumed that the controls and layouts in the control tree include absolute layout 0, absolute layout 1, absolute layout 2, absolute layout 3, relative layout 1, button 0, button 1, button 2, button 3, text 0, text 1, text 2, and text 3. For example, Figure 6aAs shown, a schematic diagram showing the positional relationship of the controls and layouts in the control tree is shown. Taking an electronic device configured with an Android system as an example, after detecting a touch operation of a user on the display screen, the electronic device starts the Activity corresponding to the window, and creates the corresponding control tree. Subsequently, the control data and layout data in the control tree are obtained, and according to the node depth of the control tree node in the control tree, the corresponding control tree depth level is divided, and the corresponding control tree level mark can be filled in the corresponding control data and layout data according to the level, so as to facilitate the subsequent step S502 of dividing the rendering level.

[0139] Specifically, in the control tree creation process, the root view is first determined, and then the setContentView() method is called to complete the creation of other nodes above the root view. Then, the node relationship in the control tree can be determined according to the Z-axis value, which is related to the creation order of the controls and layouts. For example, Figure 6a As shown, the root view in the current control tree is absolute layout 0, and then Figure 6b As shown, it is determined that the level of absolute layout 0 in the control tree is control tree level 0, and the control tree level 0 mark is filled in the layout data corresponding to the absolute layout 0.

[0140] Subsequently, the absolute layout 0 is taken as a parent node (parentCount), and according to the positional relationship as shown in Figure 6a The child nodes (childrenCount) of the absolute layout 0 include absolute layout 1, absolute layout 2 and relative layout 1. Therefore, as shown in Figure 6b It is determined that the levels of absolute layout 1, absolute layout 2 and relative layout 1 in the control tree are control tree level 1, and the control tree level 1 mark is recorded in the corresponding layout data.

[0141] Subsequently, the absolute layout 1, absolute layout 2 and relative layout 1 are taken as parent nodes respectively, and according to the positional relationship as shown in Figure 6a The child nodes corresponding to each parent node are determined, for example, the child nodes of the absolute layout 1 include button 0 and button 1, the child nodes of the absolute layout 2 include text 0 and text 1, and the child nodes of the relative layout 1 include button 2, button 3 and absolute layout 3, and then as shown in Figure 6b It is determined that the levels of each child node in the control tree are control tree level 2, and the control tree level 2 mark is recorded in the corresponding control data and layout data.

[0142] Finally, according to the positional relationship as shown in Figure 6a The absolute layout 3 is taken as a parent node, and it is determined that its child nodes include text 2 and text 3, as shown in Figure 6bThe determination of the hierarchy of the absolute layout 3 in the control tree is control tree hierarchy 3, and the control tree hierarchy 3 is recorded in the corresponding control data.

[0143] Thus, the control tree hierarchy is determined according to the above steps, and the data structure of the control tree is obtained as shown in Figure 6b It can be understood that other traversal algorithms can also be used to determine the control tree hierarchy of the control tree, which will not be described here.

[0144] It should be noted that one or more steps in S501 can be completed by calling the UI thread of the electronic device. The present application does not limit the creation order of the nodes in the control tree, such as user-defined determination of the node creation order. After obtaining the control tree, the control tree hierarchy can be determined according to the node relationship in the control tree.

[0145] In addition, after the control tree hierarchy is determined by traversing the control tree, if the control data and layout data included in the subsequent control tree do not change, the control tree only needs to be maintained, and the control tree does not need to be traversed before rendering. Only the previous data can be inherited. Alternatively, if the control data and layout data included in the control tree change, the control tree hierarchy and its child control tree hierarchy of the changed part need to be determined again.

[0146] It can be understood that the control tree hierarchy 0, the control tree hierarchy 1, the control tree hierarchy 2, and the control tree hierarchy 3 are used to represent the control tree hierarchy order, which is only an example of illustration. For example, the control tree hierarchy can also be represented as control tree hierarchy 1-control tree hierarchy 4, control tree hierarchy A-control tree hierarchy D, etc.

[0147] S502, according to the intersection, divide the rendering hierarchy.

[0148] In some embodiments, each node in the control tree corresponds to a rendering object (i.e., a control or a layout), and each control tree hierarchy includes one or more rendering objects (i.e., including one or more controls, one or more layouts). Then, after obtaining the control tree hierarchy, it is necessary to determine whether the rendering objects (i.e., controls, layouts) in the same control tree hierarchy in the control tree intersect, so as to divide the rendering hierarchy according to the intersection.

[0149] For example, according to the control tree level, the controls and layouts are preliminarily divided into corresponding rendering levels (i.e., different control tree levels are set as different rendering levels) first, and then on this basis, the controls and layouts intersecting in the same control tree level are divided into different rendering levels (i.e., the intersecting rendering objects in the same control tree level are set as different rendering levels), and the controls and layouts not intersecting are divided into the same rendering level, to complete the division of the rendering levels. Optionally, the rendering levels of the determined controls and layouts are marked in the corresponding control data and layout data, so as to facilitate the rendering of the controls and layouts according to the rendering levels in step S503.

[0150] In some embodiments, in the process of drawing the controls and layouts, the controls and layouts have corresponding drawing areas, and if the drawing areas partially or wholly intersect, it indicates that the corresponding controls and layouts intersect. Optionally, the corresponding drawing areas can be obtained according to the control data and layout data, and the drawing area of a control or layout can be represented by a minimum rectangular area, for example, which includes an area that can cover the corresponding control or layout and has the minimum area.

[0151] For example, as shown in FIG. 61, the area in the rectangular frame indicated by reference numeral 61 can completely cover the drawing area of text 2 and has the minimum area, and thus is the minimum rectangular area corresponding to text 2. The area in the rectangular frame indicated by reference numeral 62 can completely cover the drawing area of text 3 and has the minimum area, and thus is the minimum rectangular area corresponding to text 3. Further, as shown in FIG. 62, text 2 and text 3 are both set as control tree level 3 (i.e., the same control tree level), and thus it is necessary to determine whether the two rendering objects (i.e., text 2 and text 3) intersect, to determine the rendering levels. Figure 6a Figure 6b Figure 6a

[0152] It should be noted that, in the schematic diagram of the positional relationship of the controls and layouts in the control tree shown in FIG. 63, the minimum rectangular areas corresponding to the controls and layouts have been shown, i.e., the areas within the outer frames of the rectangles shown in the figure, which will not be described below. Figure 6a The following describes how to divide the rendering levels according to the intersection conditions, for the control tree scenarios shown in FIG. 64 and FIG. 65.

[0153] Figure 6a Figure 6b

[0154] For example, as shown in FIG. 66, the control tree level information of the control tree is obtained, and the level of absolute layout 0 is control tree level 0, and thus, as shown in FIG. 67, the rendering level of absolute layout 0 is divided as rendering level 0. Figure 6b Figure 7 ​​​​​​​​

[0155] Then, according to the control tree hierarchy order, the next level is determined, that is, the rendering object corresponding to the control tree hierarchy 1 includes absolute layout 1, absolute layout 2 and relative layout 1, as shown in the figure, the minimum rectangular region corresponding to the relative layout 1 does not intersect with the minimum rectangular region corresponding to the absolute layout 1 and the minimum rectangular region corresponding to the absolute layout 2, it is determined that the relative layout 1 does not intersect with the absolute layout 1 and the absolute layout 2, then the rendering level of the relative layout 1 is divided into rendering level 1 corresponding to the control tree hierarchy, as shown in the figure. Figure 6a Figure 7 As shown in the figure, the minimum rectangular region corresponding to the absolute layout 2 intersects with the minimum rectangular region corresponding to the absolute layout 1, it is determined that the absolute layout 2 intersects with the absolute layout 1, and the intersecting region is the absolute layout 2 above the absolute layout 1. Then the rendering level of the absolute layout 1 is divided into rendering level 1 corresponding to the control tree hierarchy, as shown in the figure; the rendering level of the absolute layout 2 is increased by 1 on the corresponding control tree hierarchy, and is divided into rendering level 2. Figure 6a Figure 7

[0156] Then, as shown in the figure, the next level is confirmed again, that is, the rendering object corresponding to the control tree hierarchy 2 includes button 0, button 1, text 0, text 1, button 2, button 3 and absolute layout 3. Among them, as shown in the figure, according to the corresponding minimum rectangular region, it is determined that button 0 and button 1 intersect, and the intersecting region is button 1 above button 0; text 0 and text 1 intersect, and the intersecting region is text 1 above text 0; button 2, button 3 and absolute layout 3 do not intersect with other rendering objects. Then, as shown in the figure, the rendering level of button 0, button 2, button 3 and absolute layout 3 is divided into rendering level 2, and the rendering level of button 1 is divided into rendering level 3. And, since the rendering level of the absolute layout 2 is rendering level 2, text 0 and text 1 are child node controls of the absolute layout 2, then the rendering level of text 0 is increased by 1 based on the rendering level 2, and is divided into rendering level 3; the rendering level of text 1 is increased by 1 based on the rendering level 3, and is divided into rendering level 4. Figure 6b Figure 6a Figure 7

[0157] Finally, as shown in the figure, the next level is confirmed again, that is, the rendering object corresponding to the control tree hierarchy 3 includes text 2 and text 3. Among them, as shown in the figure, according to the corresponding minimum rectangular region, it is determined that text 2 and text 3 intersect, and the intersecting region is text 3 above text 2. Then, as shown in the figure, the rendering level of text 2 is divided into rendering level 3, and the rendering level of text 3 is divided into rendering level 4. Figure 6b Figure 6a Figure 7

[0158] ​​​​​​​​​In this way, by traversing the control tree hierarchy, the rendering hierarchy of the rendering object is divided.

[0159] It is understood that the above rendering levels 0, 1, 2, 3 and 4 are used to represent the rendering order and are only an example. For example, the rendering levels can also be represented as rendering level 1-5, rendering level A- rendering level E, etc.

[0160] It should be noted that the division of rendering levels can be done by the UI thread confirming the control tree level, and then the rendering thread directly obtaining the rendering level for rendering in step S503; or the rendering thread obtaining the control tree level and dividing the rendering level according to the control tree level; or the rendering level being divided by another thread, and the rendering thread receiving the confirmed rendering level division result and data before performing data rendering.

[0161] S503: Call the rendering thread to render controls and layout.

[0162] In some embodiments, the electronic device invokes a rendering thread to render objects (such as controls and layouts) according to the rendering hierarchy order. For example, rendering objects at different rendering levels are rendered sequentially according to the rendering hierarchy order, while rendering objects at the same rendering level are rendered in parallel. Sequential rendering includes invoking a thread to render rendering objects sequentially according to the rendering hierarchy order; parallel rendering includes invoking multiple rendering threads to execute rendering tasks simultaneously.

[0163] For example, such as Figure 7 As shown, the current rendering hierarchy includes 5 levels. The rendering threads are called to perform serial rendering in the order of rendering levels 0 to 4, that is, from the lower rendering levels to the higher rendering levels. For example, the same rendering thread can be called to render absolute layout 0, absolute layout 1, and button 0 in sequence. Furthermore, data within the same rendering level is rendered in parallel. For example, after rendering absolute layout 0, two rendering threads are called to render absolute layout 1 and relative layout 1 in parallel.

[0164] In some scenarios, control data or layout data includes one or more of the following: background layer component data, content layer component data, and foreground layer component data. The background layer component data corresponds to the background of the control or layout after rendering, the content layer component data corresponds to the displayed content of the control or layout, and the foreground layer component data corresponds to the foreground of the control or layout. For example, if the control is a text control, then its content layer component data includes the corresponding text data.

[0165] Therefore, after the electronic device completes the division of the rendering levels, the data of different rendering levels can be rendered in the order of the rendering levels, but the rendering order of the constituent element data in the same rendering level still needs to be further determined. Optionally, the instruction group of the constituent element data in the same rendering level is grouped, and the control and layout rendering is performed according to the grouping result of the instruction group.

[0166] For example, as shown in FIG. 6, button 0 and button 2 are both divided into rendering level 2. It is assumed that button 0 includes background 1 and content 1, and button 2 only includes content 2. Therefore, although button 0 and button 1 are in the same rendering level, the rendering order of background 1, content 1 and content 2 still needs to be divided. Figure 7

[0167] Two instruction group grouping methods of the constituent element data are introduced as follows.

[0168] Method one: the control data and layout data in the same rendering level are grouped according to the data category of the constituent element data. The data of the same data category is divided into the same instruction group, and the data of different data categories is divided into different instruction groups.

[0169] In some embodiments, the background layer constituent element in the same rendering level is divided into one instruction group data, the content layer constituent element is divided into one instruction group data, and the foreground layer constituent element is divided into one instruction group data. For example, the background data in the same rendering level is divided into one instruction group, and the text data is divided into one instruction group. Optionally, the grouping result of the instruction group is marked and recorded in the corresponding control data or layout data. Therefore, in the subsequent rendering process, the background layer constituent element data instruction group, the content layer constituent element data instruction group and the foreground layer constituent element data instruction group can be rendered in the order in the same rendering level.

[0170] For example, as shown in FIG. 6, it is assumed that the method described in step S502 is used to determine that the rendering levels of the control data and layout data included in the control tree include rendering level 1 and rendering level 2. The data corresponding to rendering level 1 includes text 1, text 3, text 5, text 7 and text 9, and the data corresponding to rendering level 2 includes text 2, text 4, text 6, text 8 and text 10. The data in the same rendering level is divided into an instruction group. It is assumed that each text data includes background layer constituent element data and content layer constituent element data (i.e. text content), as shown in FIG. 6. Figure 8 Figure 8 ​​As shown, within each rendering level, the background layer constituent element data can be divided into one instruction group (such as instruction group 1 or instruction group 3), and the content layer constituent element data can be divided into one instruction group (such as instruction group 2 or instruction group 4). After dividing the instruction groups, the data for rendering level 1 includes the data from instruction group 1 and instruction group 2, and the data for rendering level 2 includes the data from instruction group 3 and instruction group 4.

[0171] Method 2: Group control data and layout data within the same rendering level according to the instruction execution order of the constituent element data. Data within the same rendering level with the same instruction execution order are grouped into the same instruction group, while data within the same rendering level but with different instruction execution orders are grouped into different instruction groups.

[0172] In some embodiments, the instruction execution order includes, for example, the original rendering order of the component element data of control data and layout data. Specifically, the original rendering order of the component element data of control data or layout data is generally to render the background layer component element data first, then the content layer component element data, and finally the foreground layer component element data. If the control data or layout data does not include one or more layers of component element data, it is permissible to skip that layer and directly render the next layer of component element data. Therefore, according to the instruction execution order, data with the same instruction execution order at the same rendering level are grouped into the same instruction group, and data with different instruction execution orders are grouped into different instruction groups. Optionally, the instruction grouping results are marked and recorded in the corresponding control data or layout data. Then, during the subsequent rendering process, rendering is performed according to the marked instruction grouping results and the instruction group execution order.

[0173] For example, such as Figure 9 As shown, assuming the method described in step S502 above is used, the rendering levels of the control data and layout data included in the control tree are determined to be rendering level 1 and rendering level 2. The data corresponding to rendering level 1 includes text 1, text 3, text 5, text 7, and text 9, and the data corresponding to rendering level 2 includes text 2, text 4, text 6, text 8, and text 10. Data within the same rendering level is divided into instruction groups, such as... Figure 9As shown, text 1, text 5, text 7 and text 9 in the rendering level 1 all include background layer constituent element data and content layer constituent element data, and text 3 indicated by the reference sign 91 only includes content layer constituent element data and does not include background layer constituent element data. If the instruction execution order is to execute rendering of the background layer constituent element data first and then execute rendering of the content layer constituent element data, then text 3 can directly render the content layer constituent element data in the rendering process because text 3 does not include background layer constituent element data, and other data in the rendering level 1 needs to render the background layer constituent element data first and then render the content layer constituent element data. Therefore, if the instruction groups are divided according to the element categories, it will cause the rendering thread responsible for rendering the background layer constituent element data of text 3 to pause work and wait for the next instruction group to render when rendering the background layer constituent element data of text 1, text 5, text 7 and text 9, thereby increasing unnecessary rendering waiting time. Therefore, the electronic device can divide the instruction groups according to the instruction execution order in the control, for example, as shown in FIG. 5B. Figure 9 As shown, after the instruction groups are divided, the data of the rendering level 1 includes the data of the instruction group 1 and the data of the instruction group 2, the data of the instruction group 1 includes the background layer constituent element data of text 1, text 5, text 7 and text 9, and the content layer constituent element data of text 3; and the data of the instruction group 2 includes the content layer constituent element data of text 1, text 5, text 7 and text 9.

[0174] It should be noted that the instruction group grouping can be confirmed by the UI thread after the rendering level is divided, and then the rendering thread directly obtains the rendering level and the instruction group grouping for rendering in the rendering process; or the rendering thread obtains the control tree level, and confirms the rendering level and the instruction group grouping according to the control tree level; or the electronic device calls other threads to complete the rendering level layering and the instruction group grouping according to the control tree level information determined by the UI thread, and then calls the rendering thread to render according to the rendering level and the instruction group grouping.

[0175] In some embodiments, based on the above step S503, on the basis of serially rendering rendering objects of different rendering levels, the data of different instruction groups in the same rendering level is serially rendered according to the instruction group order, and the data in the same instruction group is parallelly rendered. The instruction group data rendering order corresponds to the rendering order of the background layer constituent element data, the content layer constituent element data and the foreground layer constituent element data. That is, the constituent element data in the same instruction group in the rendering object of the same rendering level is parallelly rendered, and the constituent element data in different instruction groups in the rendering object of the same rendering level is serially rendered.

[0176] For example, as shown in FIG. 5B, the data of the rendering level 1 includes the data of the instruction group 1 and the data of the instruction group 2. Figure 8As shown, the control data and layout data in rendering level 1 should be rendered first, and then the control data and layout data in rendering level 2 should be rendered. Among them, the instruction group 1 data and the instruction group 2 data belonging to the rendering level 1 are rendered in series, the background layer constituent element data corresponding to the instruction group 1 is rendered first, and then the content layer constituent element data corresponding to the instruction group 2 is rendered, and the data in the same instruction group is rendered in parallel. For example, after the electronic device calls threads 1, 2, 3, 4, and 5 to render the background layer constituent element data (i.e., background data) in the instruction group 1 in parallel, it calls threads 1, 2, 3, 4, and 5 to render the content layer constituent element data (i.e., text data) in the instruction group 2 in parallel.

[0177] In some embodiments, the electronic device allocates rendering threads to render data according to the resources available in the CPU and GPU. For example, the UI thread generates rendering instructions to trigger the rendering threads to render data according to the rendering level division result and the instruction group grouping result, and the rendering threads perform data rendering after receiving the rendering instructions.

[0178] For example, after the electronic device completes the instruction group grouping of the data to be rendered, it determines whether the number of parallel callable rendering threads meets the rendering requirements of the instruction group according to the resources available in the CPU and GPU. If the number of parallel callable threads meets the rendering requirements of the instruction group, the electronic device calls the corresponding number of rendering threads to render the data in the instruction group in parallel. If the number of parallel callable threads does not meet the rendering requirements of the instruction group, the electronic device calls the maximum number of rendering threads to render part of the data in the instruction group in parallel, and renders all the data in the instruction group in batches.

[0179] For example, as shown in the scenario of Figure 8 According to the final instruction group grouping result, the electronic device determines that instruction group 1 to instruction group 4 all need to call 5 rendering threads for parallel rendering. The electronic device determines that the resources available in the current CPU and GPU meet the requirements (e.g., the maximum number of parallel callable rendering threads is greater than or equal to 5), that is, the available resources allow calling 5 rendering threads for parallel rendering, and then calls threads 1 to 5 to perform parallel rendering of the data in the instruction group.

[0180] For example, as shown in the scenario of Figure 8 According to the final instruction group grouping result, the electronic device determines that there are 5 constituent element data (e.g., background 1 to background 5 included in instruction group 1) that need to be rendered in parallel. Assuming that the electronic device determines that a maximum of 3 rendering threads can be called for parallel rendering (i.e., the maximum number of parallel callable rendering threads is 3), the electronic device groups the constituent element data that need to be rendered in parallel in the instruction group again and then renders them in series. For example, the electronic device groups the constituent element data that need to be rendered in parallel in the instruction group 1 into two groups, and then calls threads 1 and 2 to render the data in the first group in parallel, and calls threads 3 and 4 to render the data in the second group in parallel. Figure 8The instruction group 1 shown contains five background layer component data items to be rendered, with three of them grouped into one instruction group and the remaining two grouped into another (e.g., background 1-3 in one instruction group, and background 4 and background 5 in another). Then, using three parallel-callable rendering threads, the three background layer component data items are rendered in parallel first. Next, any two of these threads are used to render the remaining two background layer component data items (e.g., background 1-3 are rendered in parallel first, then background 4 and background 5 are rendered in parallel). In other words, when CPU and GPU resources are insufficient, the component data initially allocated for parallel rendering is further divided into partially serially rendered component data, achieving batch rendering of the data to avoid rendering failures.

[0181] In some embodiments, the rendering of the object can be performed by the display device. That is, after the electronic device completes the division of rendering levels and the grouping of instruction groups, it sends the results to the display device (i.e., other electronic devices with display functions) for rendering and display. In this way, the display device can reduce the rendering time while also reducing the power consumption of dividing rendering levels and instruction groups.

[0182] Thus, the method provided in this application embodiment can effectively reduce rendering time and avoid frame drops in the displayed image. In other words, parallel rendering can reduce the possibility of frame drops when electronic devices display images, ensuring the smoothness of the displayed image and thus improving the user's visual experience.

[0183] Furthermore, in some scenarios, the graphic data (such as control data and layout data) included in adjacent image frames does not change, so there is no need to redraw the graphic data. For example, the main interface of an electronic device displays application icons, and the user long-presses and drags one of the application icons to a blank display area. During the dragging of the application icon, the position and display content of other application icons do not change. Therefore, the electronic device does not need to redraw the unchanging controls and layouts in adjacent image frames, thus saving power consumption. That is, the rendering method provided in steps S501-S503 above is applied to the graphic data that needs to be drawn. For graphic data that does not need to be redrawn, the rendering result of the previous frame image can be directly obtained.

[0184] For example, such as Figure 10a As shown, the image display process includes the input event stage, UI data update stage, rendering stage, compositing stage, and display stage. Among them, as... Figure 10aAs shown, different stages are triggered using a vertical synchronization signal (vsync). For example, vertical synchronization signal 1 can be used to trigger the drawing of one or more layers and to trigger the rendering of those layers. In this embodiment, in response to vertical synchronization signal 1, the electronic device can draw one or more layers for each application through each of multiple drawing threads. That is, in response to vertical synchronization signal 1, the electronic device can simultaneously perform drawing tasks for one or more applications to draw one or more layers corresponding to each application. The rendering method for each layer can be the rendering method described in the above embodiment. Vertical synchronization signal 2 can be used to trigger the compositing of rendered image frames. Vertical synchronization signal 3 can be used to trigger the display of the composited image frames. The signal period of each vertical synchronization signal is determined according to the screen refresh rate of the electronic device's display. For example, the signal period of the vertical synchronization signal is the reciprocal of the screen refresh rate of the electronic device's display (such as LCD or OLED). The screen refresh rate of the electronic device can be the same as its frame rate. A high frame rate for the electronic device is equivalent to a high screen refresh rate.

[0185] like Figure 10a As shown, in response to vertical synchronization signal 1, the electronic device processes layer data to obtain the corresponding rendering object in the control tree, as well as the control tree level corresponding to the rendering object, and then determines the rendering level of the rendering object. Finally, it calls the rendering thread to render the rendering object. During the rendering phase, rendering a (i.e., rendering thread a) and rendering b (i.e., rendering thread b) render in parallel, and then rendering c (i.e., rendering thread c) is executed. In this way, the rendering task is completed within the signal period of the vertical synchronization signal, ensuring the execution of subsequent steps. Compared to... Figure 10b As shown, rendering a, b, and c all need to be rendered sequentially. This means that rendering c cannot be completed within the signal period of the vertical synchronization signal. Even after the electronic device detects vertical synchronization signal 2, there are no completed image frames to synthesize, and therefore, in response to vertical synchronization signal 3, the image frame cannot be refreshed. Consequently, the LCD display cannot be updated, resulting in dropped frames. This affects the continuity and smoothness of the displayed image, thus impacting the user's visual experience.

[0186] It should be noted that the name of the vertical synchronization signal may differ in different systems or architectures. For example, in some systems or architectures, the signal used to trigger the processing of one or more layers of data (i.e., vertical synchronization signal 1) may not be called a vertical synchronization signal. However, regardless of the name of the signal, as long as it is a synchronization signal with similar functions and conforms to the technical concept of the method provided in the embodiments of this application, it should be covered within the protection scope of this application.

[0187] Depend on Figure 10aIt can be known that the method provided in the embodiment of the present application can avoid the frame loss phenomenon of the displayed image. That is, the method provided in the embodiment of the present application can reduce the possibility of frame loss when the electronic device displays the image, can ensure the fluency of the image displayed on the display screen, and thus improves the visual experience of the user.

[0188] The above Figures 5-10b The rendering method provided in the embodiment of the present application is described in detail. The following Figure 11 and Figure 12 The electronic device provided in the embodiment of the present application is described in detail.

[0189] In a possible design, Figure 11 A structural schematic diagram of an electronic device provided in the embodiment of the present application is shown in FIG. 11. As shown in FIG. 11, the electronic device 1100 includes an acquisition unit 1101, a division unit 1102, and a rendering unit 1103. The electronic device 1100 can be used to implement the functions of the electronic device involved in the above method embodiments. The electronic device 1100 can be the electronic device itself, or a functional unit or chip in the electronic device, or a device used in conjunction with the electronic device. Figure 11

[0190] Optionally, the acquisition unit 1101 is configured to support the electronic device 1100 to perform step S501 in the method shown in FIG. 5. Figure 5

[0191] Optionally, the division unit 1102 is configured to support the electronic device 1100 to perform step S502 in the method shown in FIG. 5. Figure 5

[0192] Optionally, the rendering unit 1103 is configured to support the electronic device 1100 to perform step S503 in the method shown in FIG. 5. Figure 5

[0193] Optionally, Figure 11 The electronic device 1100 shown in FIG. 11 can further include a sending unit (not shown in FIG. 11), configured to send a signal to another communication device. Figure 11 Optionally,

[0194] The electronic device 1100 shown in FIG. 11 can further include a storage unit (not shown in FIG. 11), which stores a program or instruction. When the acquisition unit 1101, the division unit 1102, and the rendering unit 1103 execute the program or instruction, the electronic device 1100 shown in FIG. 11 can perform the rendering method shown in FIG. 5. Figure 11 Figure 11 Figure 11 Figure 5

[0195] Figure 11 ​​​​​​​​The technical effects of the electronic device 1100 shown can refer to the technical effects described in the above method embodiments, which will not be repeated here.

[0196] The acquisition unit and the sending unit can be collectively referred to as a transceiver unit, can be implemented by a transceiver or a transceiver-related circuit component, and can be a transceiver or a transceiver module. The operations and / or functions of each unit in the electronic device 1100 are respectively used to implement the corresponding processes of the rendering method described in the above method embodiments. For brevity, they will not be repeated here.

[0197] In the case of using integrated units, Figure 12 Another possible composition diagram of the electronic device involved in the above embodiments is shown. As shown in the figure, Figure 12 The electronic device 1200 can include a processing module 1201.

[0198] The processing module 1201 is configured to perform the steps performed by the acquisition unit 1101, the division unit 1102, and the rendering unit 1103 shown in the above Figure 11 The operations and / or functions of the processing module 1201 are used to implement the corresponding processes of the rendering method described in the above method embodiments. For brevity, they will not be repeated here.

[0199] The electronic device 1200 can further include a storage module for storing program codes and data of the electronic device. The storage module can be a memory.

[0200] The processing module 1201 can be a processor or a controller. It can implement or execute various exemplary logical blocks, modules, and circuits described in combination with the disclosure of the embodiments of the present application. The processor can also be a combination of computing functions, such as one or more microprocessor combinations, DSP and microprocessor combinations, etc.

[0201] The embodiments of the present application also provide a chip system, which includes a processor coupled with a memory, the memory being configured to store programs or instructions, when the programs or instructions are executed by the processor, the chip system implements the method in any of the above method embodiments.

[0202] Optionally, the processor in the chip system can be one or more. The processor can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor, which is implemented by reading software codes stored in the memory.

[0203] Optionally, the memory in the chip system can also be one or more. The memory can be integrated with the processor, or can be arranged separately from the processor, and the embodiments of the present application are not limited. Illustratively, the memory can be a non-transient processor, for example, a read-only memory (ROM), which can be integrated on the same chip as the processor, or can be arranged separately on different chips, and the embodiments of the present application do not make specific limitations on the type of memory and the arrangement of the memory and the processor.

[0204] Illustratively, the chip system can be a field programmable gate array (FPGA), can be an application specific integrated circuit (ASIC), can also be a system on chip (SoC), can also be a central processor unit (CPU), can also be a network processor (NP), can also be a digital signal processor (DSP), can also be a micro controller unit (MCU), can also be a programmable logic device (PLD) or other integrated chip.

[0205] It should be understood that each step in the above method embodiments can be completed by the integrated logic circuit of hardware in the processor or the instructions in the form of software. The method steps disclosed in combination with the embodiments of the present application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0206] Embodiments of the present application also provide a storage medium for storing instructions for the above communication device.

[0207] Embodiments of the present application also provide a computer readable storage medium, which stores computer instructions, and when the computer instructions run on a server, the server executes the above related method steps to realize the rendering method in the above embodiments.

[0208] Embodiments of the present application also provide a computer program product, which, when running on a computer, causes the computer to execute the above related steps to realize the rendering method in the above embodiments.

[0209] In addition, the embodiment of the present application further provides a device, which can be a component or a module. The device can include one or more processors and memories connected thereto. The memory is configured to store computer programs. The one or more computer programs include instructions. When the instructions are executed by the one or more processors, the device performs the rendering method in the above-mentioned method embodiments.

[0210] The device, the computer readable storage medium, the computer program product or the chip provided in the embodiments of the present application are used to execute the corresponding methods provided above, and thus the beneficial effects achieved thereby can refer to the beneficial effects of the corresponding methods provided above, which will not be described herein again.

[0211] The steps of the methods or algorithms described in connection with the embodiments disclosed in the present application can be implemented in hardware, or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in a random access memory (RAM), a flash memory, a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a compact disk read only memory (CD-ROM) or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC).

[0212] From the above description of the embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration. In actual applications, the above functions can be completed by different functional modules according to needs, i.e., the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein again.

[0213] In several embodiments provided in the present application, it should be understood that the disclosed methods can be implemented in other manners. For example, the described embodiments of the apparatus are merely schematic, and the division of the modules or units is merely logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, and electrical, mechanical or other forms.

[0214] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place, or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0215] In addition, the functional units in the various embodiments of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0216] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0217] The above is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A rendering method, characterized in that, Applied to electronic devices, the method includes: The electronic device, Obtain the rendering object to be displayed, which includes controls and layout. Obtain the control tree including the rendering object to be displayed. Traverse the control tree starting from the root node: Set different control tree levels to different rendering levels. Intersecting rendering objects at the same control tree level are set to different rendering levels; wherein the smallest rectangular regions corresponding to the intersecting rendering objects intersect. Parallel rendering of rendering objects at the same rendering level. Render objects at different rendering levels sequentially.

2. The method according to claim 1, characterized in that, The parallel rendering of rendering objects at the same rendering level includes: Parallel rendering of constituent element data in the same instruction group within a rendering object at the same rendering level, and serial rendering of constituent element data in different instruction groups within a rendering object at the same rendering level.

3. The method according to claim 2, characterized in that, The constituent element data includes one or more of the following: foreground layer constituent element data, content layer constituent element data, and background layer constituent element data.

4. The method according to claim 2, characterized in that, The data categories of constituent elements within the same instruction group are the same, while the data categories of constituent elements within different instruction groups are different; or, the execution order of the corresponding instructions in the corresponding rendering objects is the same for constituent elements within the same instruction group, while the execution order of the corresponding instructions in the corresponding rendering objects is different for constituent elements within different instruction groups.

5. The method according to any one of claims 1-4, characterized in that, Setting intersecting rendering objects at the same control tree level to different rendering levels includes: Get the smallest rectangular area corresponding to the rendering object at the same control tree level; Set the rendering objects that intersect the smallest rectangular regions in the same control tree hierarchy to different rendering levels.

6. The method according to claim 5, characterized in that, The smallest rectangular local area corresponding to the rendering object is the rectangular region that covers the rendering object and has the smallest area.

7. The method according to any one of claims 1-4, characterized in that, Before rendering rendering objects at the same rendering level in parallel, the method further includes: Resource acquisition status; Based on the resource situation, determine the maximum number of rendering threads that can be invoked in parallel.

8. The method according to claim 7, characterized in that, The parallel rendering of rendering objects at the same rendering level includes: If the first number of rendering threads required to render the same rendering level objects in parallel is less than or equal to the maximum number, then the first number of rendering threads are called to render the same rendering level objects in parallel. If the first number is greater than the maximum number, then the maximum number of rendering threads are invoked to render the rendering objects at the same rendering level in parallel.

9. An electronic device, characterized in that, include: A processor and a memory, the memory being coupled to the processor, the memory being used to store computer program code, the computer program code including computer instructions, which, when the processor reads the computer instructions from the memory, cause the electronic device to perform the rendering method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1-8.

11. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-8.

Images