Graphical display method and related equipment
Patent Information
- Application Number
- CN202311283901.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-28
- Publication Date
- 2025-04-08
Smart Images
Figure CN119781878A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of terminal and communication technology, and in particular to a graphical display method and related equipment. Background Art
[0002] With the development of technology, the applications of mobile phones and other electronic devices are becoming more and more diverse.
[0003] In the current Android system, when multiple applications are displayed in an electronic device, each application will have a corresponding display layer, and all display layers need to be uniformly displayed after a set of display synthesis processing is performed by Surfaceflinger. The display loading speed is slow, the display system process is long, and the delay is large. In complex and high-load scenarios such as games or multi-window displays, the smoothness of the modules that need to be displayed first is easily interfered by other modules of the system, and it is easy to be stuck. Summary of the invention
[0004] The present application provides a graphical display method and related equipment, which can form two display processing channels. When the application has priority display requirements and faces high-load scenarios, it can significantly shorten the data processing process, avoid jamming caused by interference, and reduce graphic display delays.
[0005] In a first aspect, a graphical display method is provided, which is applied to an electronic device including a first graphical display component and a second graphical display component, wherein the first graphical display component is used to perform a first graphical display process on a specific application, and the second graphical display component is used to perform a second graphical display process on other applications other than the specific application. The graphical display method includes:
[0006] When the first application is started, a target graphical display component associated with the first application is determined; the target graphical display component is the first graphical display component or the second graphical display component; graphical display processing is performed on N layers to be displayed of the first application through the target graphical display component, and the first application is displayed on the target screen; N is an integer greater than or equal to 1.
[0007] The first graphic display component forms a display processing channel, and the second graphic display component forms another display processing channel. The two graphic display components provided by the electronic device are used to form two display processing channels for matching and selecting when performing graphic display processing on an application, so as to select a suitable display processing channel for graphic display processing based on the current application.
[0008] For specific applications, newly added graphical display components can be used to perform separate processing through the corresponding display processing channels. For non-specific applications, appropriate graphical display components can be used to perform graphical display processing through the corresponding display processing channels without interfering with the data display processing of specific applications. When the application has priority display requirements and faces high-load scenarios, the data processing process can be significantly shortened, avoiding jamming caused by interference from other applications and processing modules in the system, and reducing graphic display delays.
[0009] In combination with the first aspect, in some implementations of the first aspect, performing graphical display processing on N to-be-displayed layers of the first application by a target graphical display component to display the first application on a target screen includes:
[0010] In the case where the target graphic display component is the first graphic display component, the N layers to be displayed are processed into target graphic data by the first graphic display component, and the first application is graphically displayed on the target screen based on the target graphic data.
[0011] In this case, the first application is a specific application.
[0012] This processing process determines the display processing channel corresponding to the newly added graphical display component from the two display processing channels formed, realizes separate processing of specific applications, avoids the graphical display of specific applications being interfered with by other applications and processing modules in the system and causes jamming, and reduces graphic display delay.
[0013] In combination with the first aspect, in some implementations of the first aspect, processing the N to-be-displayed layers into target graphic data by the first graphic display component includes:
[0014] The N layers to be displayed are directly processed into target graphic data through the first graphic display component; or, the N layers to be displayed are processed into first graphic data through the first graphic display component, and when the first graphic display component receives second graphic data sent by the second graphic display component, the first graphic data and the second graphic data are processed into target graphic data through the first graphic display component.
[0015] In this process, the first graphical display component provides two layer processing methods, one is to process the layer of a specific application or directly display it, and the other is to superimpose the layer data formed by the temporary display requirements of other applications with the layer data of the specific application to obtain graphical data for display.
[0016] It can meet the graphic data processing requirements in different scenarios, such as independent display of specific applications and superimposed display of specific applications with other applications.
[0017] In conjunction with the first aspect, in certain implementations of the first aspect, graphically displaying the first application on the target screen based on the target graphic data includes:
[0018] The graphics library is called to render the target graphics data, and the rendered target graphics data is graphically displayed on the target screen.
[0019] This processing method, with the help of the graphics library, can speed up the processing of graphics data and improve the graphical display effect of the application.
[0020] In combination with the first aspect, in some implementations of the first aspect, the first graphic display component includes a first graphic buffer allocator and a first synthesizer; the first graphic display component processes the N layers to be displayed into target graphic data, and graphically displays the first application on the target screen based on the target graphic data, including:
[0021] A graphics buffer is allocated through a first graphics buffer allocator, and the graphics buffer is used for a first application to draw N layers to be displayed therein; the N layers to be displayed in the graphics buffer are processed into target graphics data through a first synthesizer, and the first application is graphically displayed on a target screen based on the target graphics data.
[0022] In this way, effective implementation of the graphic data processing operation of the newly added graphic display component can be ensured.
[0023] In combination with the first aspect, in some implementations of the first aspect, when the target graphical display component is the first graphical display component, the first application is displayed in full screen on the target screen.
[0024] In this way, the display requirements of specific applications in some scenarios are met and different display implementation methods are provided.
[0025] In combination with the first aspect, in some implementations of the first aspect, performing graphical display processing on N to-be-displayed layers of the first application by a target graphical display component to display the first application on a target screen includes:
[0026] In the case where the target graphic display component is the second graphic display component, the N layers to be displayed are processed into second graphic data by the second graphic display component, and the first application is graphically displayed on the target screen based on the second graphic data.
[0027] In this case, the first application is other application.
[0028] This processing process determines the display processing channel corresponding to other applications from the two display processing channels formed, realizes normal processing of other applications outside the specific application, avoids interference with the data display processing of the specific application, and can significantly shorten the data processing flow and reduce graphics display delay when the application has priority display requirements and faces high-load scenarios.
[0029] In combination with the first aspect, in some implementations of the first aspect, graphically displaying the first application on the target screen based on the second graphic data includes:
[0030] When a graphical display screen of a specific application is displayed on the target screen, the second graphical display component outputs the second graphical data to the first graphical display component, so that the first application is displayed on the target screen through the first graphical display component.
[0031] When a graphical display image of a specific application is displayed on the display screen, if other applications need to be graphically popped up on the same display screen, the first graphical display component used by the other applications needs to send the processing results of layer processing on the application to the second graphical display component used by the specific application, so as to be able to further implement layer synthesis processing between the specific application and the other applications through the second graphical display component.
[0032] In this way, the graphic data processing requirements in the scenario where a specific application is displayed in superposition with other applications are met.
[0033] In combination with the first aspect, in some implementations of the first aspect, graphically displaying the first application on the target screen based on the second graphic data includes:
[0034] In the case that the graphical display screen of the specific application is not displayed on the target screen, the first application is graphically displayed on the target screen directly through the second graphical display component based on the second graphical data.
[0035] This processing method corresponds to the display processing process of other applications other than the specific application. When the graphical display content of the specific application is not displayed on the display screen required by other applications, the graphical display processing results of other applications can be directly displayed to meet the display requirements of other applications.
[0036] In combination with the first aspect, in some implementations of the first aspect, graphically displaying the first application on the target screen based on the second graphic data by the second graphic display component includes:
[0037] The second graphic data is output to the DPU through the second graphic display component to be displayed on the target screen; or, the second graphic data is output to the DPU through the second graphic display component, and when the DPU is unable to perform a display operation in response to the second image data, the graphics library is called to render the second image data, and the rendered second image data is graphically displayed on the target screen.
[0038] This process provides a variety of graphic display processing methods, improves the processing speed of graphic data, and ensures the robustness of the graphic display processing process.
[0039] In combination with the first aspect, in some implementations of the first aspect, the second graphic display component includes a second graphic buffer allocator and a second synthesizer; the second graphic display component processes the N layers to be displayed into second graphic data, and graphically displays the first application on the target screen based on the second graphic data, including:
[0040] A graphics buffer is allocated through a second graphics buffer allocator, and the graphics buffer is used by the first application to draw N layers to be displayed therein; the N layers to be displayed in the graphics buffer are processed into second graphics data through a second synthesizer, and the first application is graphically displayed on a target screen based on the second graphics data.
[0041] In this way, the effective implementation of the graphic data processing operation of the original graphic display component can be ensured.
[0042] In combination with the first aspect, in some implementations of the first aspect, the electronic device is configured with at least two display screens; before graphically displaying the first application on the target screen, the method further includes:
[0043] Determine the display control information associated with the first application in the electronic device; if the display control information indicates that the first application is displayed on the first screen in the electronic device, the first screen is the target screen; if the display control information indicates that the first application is displayed on the second screen in the electronic device, the second screen is the target screen; if the electronic device is in an unfolded state and the display control information indicates that the first application is displayed in full screen in the electronic device, the screen formed by splicing the first screen and the second screen is the target screen.
[0044] This processing method can meet the graphical display requirements of electronic devices with multiple display screens.
[0045] In conjunction with the first aspect, in some implementations of the first aspect, there are multiple first applications, and the multiple first applications include a specific application and other applications; the electronic device is configured with at least a first screen and a second screen; when the first application is started, determining a target graphical display component associated with the first application includes:
[0046] When a specific application is started, the target graphical display component associated with the specific application is determined to be the first graphical display component; when other applications are started, the target graphical display component associated with the other applications is determined to be the second graphical display component.
[0047] Correspondingly, graphical display processing is performed on the N layers to be displayed of the first application through the target graphical display component, and the first application is displayed on the target screen, including: performing graphical display processing on the layers to be displayed of a specific application through the first graphical display component, and displaying the specific application on the first screen with a first resolution and / or a first refresh rate; performing graphical display processing on the layers to be displayed of other applications through the second graphical display component, and displaying the other applications on the second screen with a second resolution and / or a second refresh rate.
[0048] In the split-screen scenario, different screen display control parameters are used to display different images on the corresponding display screens. The two display processing channels can output two images with different frame rates and resolutions to meet differentiated display requirements and reduce the power consumption of the entire system in the split-screen scenario.
[0049] In a second aspect, the present application provides a graphical display device, comprising a first graphical display component and a second graphical display component, wherein the first graphical display component is used to perform a first graphical display process on a specific application, and the second graphical display component is used to perform a second graphical display process on an application other than the specific application; the graphical display device comprises:
[0050] A component determination module, used to determine a target graphical display component associated with the first application when the first application is started; the target graphical display component is the first graphical display component or the second graphical display component;
[0051] The display processing module is used to perform graphical display processing on N to-be-displayed layers of the first application through a target graphical display component, and display the first application on a target screen; N is an integer greater than or equal to 1.
[0052] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the methods of the first aspect when executing the computer program.
[0053] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, which can implement any one of the methods of the first aspect when the computer program is executed by a processor.
[0054] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it can implement any one of the methods of the first aspect.
[0055] It can be understood that the graphical display device provided in the second aspect, the electronic device provided in the third aspect, the computer-readable storage medium provided in the fourth aspect, and the computer program product provided in the fifth aspect are all used to execute any one of the methods provided in the first aspect of the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] Figure 1 An exemplary architecture of a display system in an electronic device is shown;
[0057] Figure 2 Shows a module framework of the SurfaceFlinger component;
[0058] Figure 3 Another exemplary architecture of a display system in an electronic device is shown;
[0059] Figure 4 Shows a module framework of the Compositor component;
[0060] Figure 5 A schematic diagram of triggering an application in a display screen of an electronic device is shown;
[0061] Figure 6 A schematic diagram of a screen in which a graphical display is applied in the first scene;
[0062] Figure 7 is a schematic flow chart of the graphical display method involved in the first scenario;
[0063] Figure 8 A schematic diagram of a screen in which a graphical display is applied in the second scene;
[0064] Fig. 9 is a schematic flow chart of a graphical display method involved in the second scenario;
[0065] Fig.10 A schematic diagram of a screen in which a graphical display is applied in the third scene;
[0066] Fig.11 is a schematic flow chart of a graphical display method involved in the third scenario;
[0067] Fig.12 is a schematic diagram of an electronic device control device according to an embodiment of the present application;
[0068] Fig.13 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0069] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.
[0070] It should be understood that the terms "first", "second", etc. in the specification, claims and drawings of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.
[0071] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.
[0072] The term "user interface (UI)" in the following embodiments of the present application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device, and finally presented as content that can be recognized by the user. The commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.
[0073] An electronic device usually has multiple apps (Applications) installed, and each of the apps has a requirement for performing graphical display on a display screen.
[0074] Electronic devices include, for example, mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), and other devices that can support multiple applications and have graphical display functions. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.
[0075] The multiple apps are specifically App0-AppN. In fact, taking a mobile phone as an example, a mobile phone usually has multiple apps installed, for example, game application software, chat application software, audio and video application software, email software, music player software, etc. In the same period of time, there will be a demand for graphical display on the display screen between multiple apps. For example, there is a demand to display the graphical interface of the game on the display screen, a demand to display the voice and video screen on the display screen, and when the graphical interface of the game is displayed on the display screen, a new email is generated, and a pop-up window is generated for the new email. Or, in the process of displaying the graphical interface of the game on the display screen, in order to realize the dialogue and communication with the game teammates, the chat interface of the chat application software is displayed in the form of a floating small window.
[0076] The graphical display of the above-mentioned multiple apps needs to be realized with the help of a display system provided in the electronic device.
[0077] In one implementation, a graphic display component is provided in a display system of an electronic device for performing graphic display processing on different application programs.
[0078] An exemplary architecture of a display system in an electronic device under the Android system is as follows: Figure 1 shown. Figure 1 The SurfaceFlinger component is formed as a graphical display component.
[0079] When App0 and other apps (App1-AppN) need to be graphically displayed in the same period of time, display overlap will appear between App0-AppN.
[0080] During the specific display process, each App will generate one or more Surface objects, forming N Surfaces. One Surface corresponds to one display layer, and N is an integer greater than or equal to 1.
[0081] Surface provides a graphics container. Applications draw graphic elements onto Surface by creating and manipulating Surface objects. Then, for the Surfaces of multiple apps, Surface submits the drawn content to the SurfaceFlinger component for synthesis and display to display the corresponding graphic elements of the App on the screen.
[0082] Among them, SurfaceFlinger is an important component in the Android system, responsible for managing and rendering all graphics and interface elements on the screen. It obtains graphics data and renders it on the screen.
[0083] The SurfaceFlinger component transmits the graphics data to the DPU (Display Processing Unit). As a hardware device, the DPU is responsible for processing and rendering the graphics data generated by applications and the system, and finally outputs it to the display screen for display.
[0084] In the specific implementation process, a module framework of the SurfaceFlinger component is as follows Figure 2 As shown in the figure, the SurfaceFlinger component includes Gralloc (graphics memory allocator) and HwCompositor (HardwareCompositor, hardware-accelerated synthesis engine).
[0085] Gralloc is a module used by SurfaceFlinger to manage the allocation and release of graphics buffers. It provides a set of APIs for SurfaceFlinger and other system components to apply for and operate graphics buffers.
[0086] When multiple applications (App0-AppN) need to draw graphics on the screen, they can apply for a graphics buffer through SurfaceFlinger and Gralloc.
[0087] This process usually includes the following steps:
[0088] 1. The application sends a request to SurfaceFlinger for a graphics buffer.
[0089] 2.SurfaceFlinger passes this request to the Gralloc module.
[0090] 3. The Gralloc module selects an available memory block from ashmem (anonymous shared memory) according to the system's memory management policy and allocates it to the application.
[0091] 4. The Gralloc module returns a handle to this memory block to SurfaceFlinger.
[0092] 5.SurfaceFlinger passes this handle to the application.
[0093] 6. The application can use the mmap operation to map this handle to its own address space to obtain the corresponding buffer and implement memory allocation.
[0094] Through this process, the application can obtain a graphics buffer that can be directly operated, and then can perform graphics element drawing operations on this buffer. Once the application completes the drawing operation, it can return this buffer to SurfaceFlinger for subsequent rendering and display.
[0095] When SurfaceFlinger receives the graphics content of an application, it sorts the content to determine the order in which they should be displayed on the screen. The sorted overall buffer contains the graphics content of all applications in the correct order.
[0096] Once the sorting is complete, SurfaceFlinger passes the entire buffer to HwCompositor for layer synthesis. HwCompositor can overlay multiple layers into the final image and then send it to the display device for display through the DPU.
[0097] Specifically, in some cases, when HwCompositor cannot process the layers in the entire buffer (there is no HwCompositor hardware, the number of layers exceeds the number of HwCompositor layers), the graphics library OpenGL ES can be used for processing.
[0098] OpenGL ES is an embedded version of the open graphics library OpenGL, used for graphics rendering on mobile devices and embedded systems.
[0099] OpenGL ES provides a cross-platform graphics library with a simplified API for rendering 2D and 3D graphics that can run on a variety of hardware and operating systems. It provides a set of functions and data structures that can handle multiple layers in the overall buffer.
[0100] By using OpenGL ES to process the layers in the overall buffer, the limitations of the HwCompositor can be bypassed and more advanced graphics processing functions can be implemented.
[0101] In the above process, by passing the entire buffer to HwCompositor, SurfaceFlinger can use its hardware acceleration capabilities to improve the performance and efficiency of graphics rendering.
[0102] However, in the above-mentioned graphical display processing method of multiple applications, since the graphical display of the interface in the application requires the SurfaceFlinger component to perform memory allocation, synthesis management, data rendering and other operations on the display layers of all apps, the display smoothness of applications that need to be displayed first is easily interfered by other modules of the system and may become stuck. In addition, the data processing flow of the display system is relatively long as a whole, the graphics display delay is large, and the load is high, which makes the display delay of electronic devices equipped with the Android system relatively large. In complex or high-load scenarios (games, multiple windows, etc.), the system is prone to jams.
[0103] On this basis, the embodiment of the present application proposes another implementation method.
[0104] In order to solve the problems of display delay and system freeze, a first graphical display component and a second graphical display component are set in the electronic device, wherein the first graphical display component is used to perform a first graphical display processing on a specific application, and the second graphical display component is used to perform a second graphical display processing on other applications other than the specific application.
[0105] Some applications need to be set as specific applications. For example, specific applications are applications that need to be used in full-screen mode, such as game applications, or applications with display priority, or some applications set by the user. For example, the user can set the applications set in the whitelist as specific applications by setting a whitelist.
[0106] An exemplary architecture of a display system corresponding to this implementation is as follows: Figure 3 shown.
[0107] exist Figure 3 In the embodiment, the display system is provided with a Compositor component in addition to the SurfaceFlinger component (the second graphical display component). The Compositor is a newly added graphical display component (the first graphical display component).
[0108] In the process of graphical display processing, the Compositor here is a component used to synthesize and render graphics. It can combine different images, videos, special effects, texts and other elements to create the final graphical effect.
[0109] compared to Figure 1 In the system architecture shown in FIG, the newly added Compositor component provides a new display processing channel for the graphical display of the application, while the original SurfaceFlinger component provides an existing display processing channel for the graphical display of the application. The two display processing channels exist together in the electronic device.
[0110] The newly added display processing channel performs the first graphic display processing on the specific application based on the Compositor component. The existing display processing channel performs the second graphic display processing on other applications other than the specific application based on the SurfaceFlinger component.
[0111] The two display processing channels have their own corresponding enabling conditions.
[0112] For the two display processing channels in the electronic device, when it is detected that the application triggered by the user is a specific application, the display processing channel corresponding to the Compositor component is enabled to perform graphical display processing on the specific application.
[0113] When it is detected that the application triggered by the user is another application, the display processing channel corresponding to the SurfaceFlinger component is enabled to perform graphical display processing on the other application.
[0114] When it is detected that other applications need to perform graphical display at the same time during the display processing of a specific application, the display processing channel corresponding to the SurfaceFlinger component is enabled for graphical display processing while the display processing channel corresponding to the Compositor component has been enabled.
[0115] Compared with Compositor, SurfaceFlinger is a more complex component, which is responsible for managing and organizing all Surfaces (including application windows, system UI, etc.) and sending them to the display. SurfaceFlinger needs to handle more tasks, such as window management, animation effects, input event processing, etc. The main function of the Compositor component is to synthesize and render layers. Its functions are relatively simplified and lightweight. And because of its simplified functions, Compositor usually requires fewer system resources, such as memory and processor resources, and takes up less resources. Due to its simplified functions and less resource usage, Compositor is usually able to provide higher performance, can synthesize and render layers faster, and thus provide a smoother user experience.
[0116] Specifically, in Figure 3 In the , the newly added Compositor is used to process the graphical display data of the specific application App0, while the graphical display data of App1-AppN is processed by the SurfaceFlinger component. As a specific application, App0 can use a display processing channel alone for display output, without having to use the display processing channel where the SurfaceFlinger component is located with other applications App1-AppN. When App0 has priority display requirements and faces high-load scenarios, the data processing process can be significantly shortened, and the jamming caused by interference from other modules of the system can be avoided, and the graphics display delay is small.
[0117] In this implementation, a Compositor component is added to the display system, and system resource allocation and data processing priority can be set independently during the graphical display processing process to ensure the smoothness of application display.
[0118] Among them, the implementation process of inputting the layers of App1-AppN to the SurfaceFlinger component for processing can be seen in the above Figure 1 and Figure 2 The related embodiment description process is not described here in detail to avoid repetition.
[0119] The newly added Compositor has two display processing functions.
[0120] On the one hand, Compositor can perform layer integration based on N Surfaces of App0 and send them for display, reducing the load and power consumption of the Android system and speeding up the response speed of specific applications.
[0121] On the other hand, during the process of performing layer integration based on N Surfaces of App0, Compositor can also obtain the layer integration data of App1-AppN transmitted from the SurfaceFlinger component, integrate the layer integration data of App0 and App1-AppN, and then send them for display, so as to meet the needs of other applications for temporary superimposed graphical display (pop-up box display, floating window display, etc.) during the display process of a specific application.
[0122] In the specific implementation process, a module framework of the Compositor component is as follows Figure 4 shown.
[0123] Figure 4 In Compositor, Gralloc1 and Compositor1 are the two submodules. Gralloc1 is used to allocate buffers for layers in a specific application. Compositor1 is used to perform graphic composition rendering for a specific application, or to perform composition rendering for the layer composition data of a specific application and other applications. Figure 4 The dotted line in the figure is a schematic diagram of the newly added data flow corresponding to the Compositor component.
[0124] When a specific application App0 needs to draw graphics on the screen, it can apply for a graphics buffer buffer through Compositor and Gralloc1.
[0125] During this process, the application sends a request to the Compositor for a graphics buffer. The Compositor passes the request to the Gralloc1 module. The Gralloc1 module selects an available memory block from ashmem according to the system's memory management policy and allocates it to App0. The Gralloc1 module returns a handle to the memory block to the Compositor. The Compositor passes the handle to App0. App0 can use the mmap operation to map the handle to its own address space, thereby obtaining the corresponding buffer and implementing memory allocation.
[0126] Through this process, App0 can obtain a graphics buffer that can be directly operated, and then perform graphics element drawing operations on this buffer. After App0 completes the drawing operation, it can return this buffer to the Compositor for composite rendering processing.
[0127] Specifically, Compositor passes the buffer to Compositor 1 to perform layer composition rendering. Compositor 1 can access the GPU with the help of the graphics library OpenGL ES to implement the layer composition rendering process.
[0128] After the GPU has processed the graphics data of App0, it can send the processing results directly to the DPU for direct display.
[0129] During this process, Compositor processes the acquired graphics data of a specific application without being interfered by data from graphic elements in other applications. It requires fewer system resources, can synthesize and render layers faster, and provide smoother graphical display effects.
[0130] Alternatively, the GPU can feed back the processing results to the Compositor, and the Compositor determines whether the graphics data of App0 can be processed and output for display. If it is determined that it can be output, the data will be sent to the DPU for display. If it is determined that it cannot be output, the data will not be sent to the DPU for display for the time being.
[0131] Among them, the Compositor's judgment on whether the graphics data processing results of App0 can be output and displayed can be when the Compositor obtains the graphics integration data of other apps (App1-AppN) from SurfaceFlinger. In this case, the Compositor will determine that the graphics data processing results of App0 cannot be directly output and displayed at present, otherwise, it will determine that the graphics data processing results of App0 can be directly output and displayed at present.
[0132] The Compositor component can control the GPU to directly output the rendered image to the DPU, reducing display delays.
[0133] When the Compositor obtains graphics integration data from other apps in SurfaceFlinger, the Compositor needs to perform composite rendering of the graphics integration data of other apps with the graphics data processing results of App0 fed back by the GPU. The composite rendering process is the same as the aforementioned process of composite rendering of App0's graphics data in the buffer by the Compositor, so it will not be repeated here.
[0134] Finally, the graphical data obtained by further integrating the layer integration data of App0 and App1-AppN is sent for display, which meets the needs of other applications to temporarily overlay graphical displays during the display process of a specific application. The collaborative processing between Compositor and SurfaceFlinger reduces the graphics display delay in high-load scenarios.
[0135] In the present application embodiment, based on the above Figure 3 and Figure 4 The display system and module block diagram shown in the figure also propose a corresponding graphical display method.
[0136] The graphical display method proposed in the embodiment of the present application is described in detail below in combination with three specific implementation application scenarios.
[0137] First scenario: Applications are displayed without overlapping in full-screen display scenarios.
[0138] In this scenario, the application that needs to be displayed in full-screen mode is used as a specific application. When the application displayed in full-screen mode is displayed graphically on the target display screen, only the display screen of a single application is displayed on the target display screen, and the display screens of other applications are not superimposed. Corresponding to this scenario, there is no layer superposition processing between multiple applications in the graphical display processing of the specific application.
[0139] Combination Figure 5 As shown, different application icons are displayed on the display screen of the electronic device. When it is detected that the user clicks the game application icon, the game application (first application) is started. In this process, the electronic device determines whether the started game application is a specific application. When it is detected that the game application is started, and the currently started game application needs to be used in full screen mode, the game application started by the current user is considered to be a specific application.
[0140] Then, when the electronic device determines that the currently launched game application is a specific application, it can determine that the graphical display component associated with the game application is a newly added compositor in the display system. The electronic device will perform graphical display processing on the display layer in the game application through the newly added compositor and output it to the display screen.
[0141] Among them, the image buffer area of the game application is allocated by the Gralloc1 module, and Compositor1 calls the GPU through OpenGL ES to realize the synthetic rendering of the graphic elements of the game application in the image buffer area, and directly sends them to the display if there is no other application that needs to be graphically displayed on the target screen.
[0142] The game application will enter the application display interface in full screen display mode, such as Figure 6 As shown, it is displayed in full screen mode in the electronic device.
[0143] The number of display screens in the electronic device can be one or more. When the game application is graphically displayed on the display screen, if the number of display screens in the electronic device is one, the game application can be directly displayed in full screen on the one display screen.
[0144] When there are multiple display screens in the electronic device, the display control information may be configured based on the current application.
[0145] The display control information may be the display control information manually set by the user when the game application is started, or the display control information pre-configured by the game application. The display control information is used to implement graphical display control of the application. The information includes, for example, which display screen is used, whether the display mode is full screen or non-full screen, the display size, the display position, and other information.
[0146] In this case, the electronic device can directly display the game application in full screen on screen A, in full screen on screen B, or as shown in FIG. Figure 6 The full screen shown in the figure is displayed in the overall display screen formed by screen A and screen B. In particular, the electronic device is, for example, a foldable electronic device. In this case, when the electronic device is in an unfolded state and the display control information indicates that the current application is displayed in full screen in the electronic device, the current application needs to be graphically displayed in the overall display screen formed by splicing screen A and screen B. This meets the display requirements in various scenarios.
[0147] In combination with the description of the first scenario above, the process of an electronic device performing graphical display using the graphical display method in an embodiment of the present application is described in detail.
[0148] Figure 7 It is a schematic flow chart of the graphical display method involved in the first scenario.
[0149] In the first scenario, the process of the electronic device performing graphical display using the graphical display method in the embodiment of the present application can refer to the following description of step 101-step 102.
[0150] Step 101: when a first application is started, determine a target graphical display component associated with the first application.
[0151] See also Figure 3 and Figure 4 As shown, the electronic device is provided with a first graphic display component or a second graphic display component in the display system.
[0152] The target graphical display component is a first graphical display component or a second graphical display component in the electronic device. The first graphical display component is used to perform a first graphical display process on a specific application, and the second graphical display component is used to perform a second graphical display process on other applications other than the specific application.
[0153] Afterwards, it is necessary to perform graphical display processing on the N to-be-displayed layers of the first application through the target graphical display component, so as to display the first application on the target screen; N is an integer greater than or equal to 1.
[0154] In the first scenario, the first application is a specific application, and therefore the target graphical display component corresponding thereto is the first graphical display component.
[0155] Step 102: When the target graphic display component is the first graphic display component, the first graphic display component directly processes the N layers to be displayed into target graphic data, and graphically displays the first application on the target screen based on the target graphic data.
[0156] In the first scenario, the game application is started, and no other application needs to be graphically displayed on the same display screen during the full-screen display of the game. In this case, the first graphical display component only needs to focus on the graphical display processing of the current game application, and the N to-be-displayed layers of the game application can be directly processed into target graphic data through the first graphical display component.
[0157] When N layers to be displayed are processed by the first graphical display component, if there is only one layer to be displayed, that is, N is 1, the display elements in the layer to be displayed are integrated by the first graphical display component to form target graphic data, or the layer to be displayed is directly used as the target graphic data without synthesis.
[0158] If there are multiple layers to be displayed, that is, N is greater than 1, the multiple layers to be displayed are synthesized by the first graphical display component to obtain target layer data.
[0159] The target layer data can be directly rendered by the DPU and then sent to the display. Alternatively, the graphics library can be called to use the GPU to render the target graphics data, and the rendered target graphics data can be directly sent to the display, so that the game application is graphically displayed on the target screen.
[0160] This process uses two graphic display components provided by the electronic device to form two display processing channels, and can select the appropriate display processing channel for graphic display processing based on the current application. For specific applications, the newly added graphic display components can be used to perform separate processing through the corresponding display processing channel. When the application has priority display requirements and faces high-load scenarios, the data processing process can be significantly shortened, avoiding jamming caused by interference from other applications and processing modules in the system, and reducing graphic display delays.
[0161] A structure of the first graphic display component specifically includes a first graphic buffer allocator and a first synthesizer.
[0162] In the process of processing the N to-be-displayed layers into target graphic data through the first graphic display component, and graphically displaying the first application on the target screen based on the target graphic data, the first graphic buffer allocator allocates a graphic buffer for the first application to draw the N to-be-displayed layers therein, and the first synthesizer synthesizes the N to-be-displayed layers in the graphic buffer into target graphic data, and graphically displays the first application on the target screen based on the target graphic data. The effective implementation of the graphic data processing operation of the newly added graphic display component is ensured.
[0163] Second scenario: Application displays are superimposed in a full-screen display scenario.
[0164] In this scenario, the application that needs to be displayed in full screen mode is regarded as a specific application. When the specific application is displaying the screen on the display screen, the graphical display requirements of other applications will appear. When the display screen of the specific application is displayed on the target display screen, the display screen of other applications will be superimposed. In the process of graphical display processing, there will also be layer overlay processing operations across multiple applications.
[0165] The display area of the display screen of other applications in the display screen is smaller than the display area of the display screen of the specific application, for example, graphical display contents such as pop-up boxes and floating small windows in other applications.
[0166] exist Figure 6 In the example, a game is started in the electronic device, and the electronic device determines that the started game application is a specific application, and performs graphical display processing on the display layer in the game application through the newly added Compositor, and outputs it to the display screen. Figure 6 The middle display screen displays a full screen of the game application.
[0167] While the game is running and the screen is continuously displayed, the electronic device detects that another application has a need to display graphics on the target screen.
[0168] In this scenario, while the game is running and displaying, the electronic device detects that the WeChat application running in the background has received a new message. At this time, the new message needs to be prompted to the user in the form of a WeChat pop-up box.
[0169] The electronic device determines whether the started WeChat application is a specific application. When it is determined that the WeChat application is not a specific application, it is necessary to perform corresponding graphical display processing on the WeChat application through the SurfaceFlinger component.
[0170] In one implementation process, the image cache area of the WeChat application is allocated by the Gralloc module in the SurfaceFlinger component, and the HwCompositor combines the GPU or calls the GPU through OpenGL ES to implement the synthetic rendering of the graphic elements corresponding to the WeChat pop-up box in the image cache area. Since the display screen of a specific application is already displayed in the electronic device, the SurfaceFlinger component will not directly display the synthetic rendering processing result in this case, but will output the synthetic rendering processing result to the Compositor component.
[0171] After receiving the composite rendering processing result from HwCompositor, the Compositor component determines that the graphics data processing result of the game application cannot be directly output for display. The Compositor component composites and renders the composite rendering processing result of the WeChat pop-up box sent by the SurfaceFlinger component with the graphics data processing result of the game application generated by itself, realizes the display layer overlay across applications, and finally sends the processing result for display. The display screen is referenced Figure 8 shown.
[0172] Likewise, the number of display screens in the electronic device can be set to one or more.
[0173] If there is only one display screen, the electronic device can directly display the final graphical display processing result in full screen on the one display screen.
[0174] If there are multiple display screens, the final graphical display processing result can be displayed in full screen on screen A, in full screen on screen B, or as shown in FIG. Figure 8 The full screen shown in the figure is displayed in the overall display screen formed by screen A and screen B. If the electronic device is a foldable electronic device, when the electronic device is in an unfolded state and the display control information indicates that the application is displayed in full screen in the electronic device, the application is graphically displayed in the overall display screen formed by splicing screen A and screen B. This meets the display requirements in various scenarios.
[0175] In other scenarios where full-screen display is not required, this display selection method can also be used to adaptively change display parameters such as the graphical display area, display size, and display position applied to the display screen.
[0176] In combination with the description of the second scenario above, the process of an electronic device performing graphical display using the graphical display method in an embodiment of the present application is described in detail.
[0177] Fig. 9 It is a schematic flow chart of the graphical display method involved in the second scenario.
[0178] In the second scenario, the process of the electronic device performing graphical display using the graphical display method in the embodiment of the present application can refer to the following description of steps 201 to 205.
[0179] Step 201, when a first application is started, if the electronic device determines that the target graphical display component associated with the first application is the first graphical display component, the N to-be-displayed layers of the first application are processed into first graphical data by the first graphical display component.
[0180] When an application is started, it is necessary to determine a target graphical display component associated with the application, and the target graphical display component is the first graphical display component or the second graphical display component.
[0181] The first graphic display component is used to perform a first graphic display process on a specific application, and the second graphic display component is used to perform a second graphic display process on other applications other than the specific application.
[0182] If the one application is a specific application, then the target graphical display component corresponding to the application is the first graphical display component. The electronic device processes the N to-be-displayed layers of the specific application into first graphical data through the first graphical display component.
[0183] Here, the implementation process of the electronic device processing the N to-be-displayed layers of a specific application into the first graphic data through the first graphic display component can be referred to the implementation process of processing the N to-be-displayed layers of a specific application into the target graphic data through the first graphic display component in the first scenario. To avoid repetition, it will not be repeated here.
[0184] Step 202, when the second application is started, if the electronic device determines that the target graphical display component associated with the second application is the second graphical display component, the N to-be-displayed layers of the second application are processed into second graphical data by the second graphical display component.
[0185] If the application is not a specific application, the target graphical display component corresponding to the application is the second graphical display component. The electronic device processes the N layers to be displayed of the application into second graphical data through the second graphical display component.
[0186] Here, the second application may be one or more.
[0187] When there are multiple second applications, there are at least two layers to be displayed, that is, N is greater than 1. When the N layers to be displayed of the second application are processed into second graphic data through the second graphical display component, the multiple layers to be displayed are synthesized by the second graphical display component to obtain the second graphic data.
[0188] When there is one second application and one layer to be displayed, that is, N is 1, the display elements in the layer to be displayed are integrated by the second graphic display component to form second graphic data, or the layer to be displayed is directly used as the second graphic data without synthesis.
[0189] When there is one second application, if there are multiple layers to be displayed, that is, N is greater than 1, the multiple layers to be displayed are synthesized by the second graphic display component to obtain the second graphic data.
[0190] Here, for the implementation process of processing the N to-be-displayed layers of the second application into the second graphic data by the second graphic display component, reference can be made to the description process of the functional implementation of the SurfaceFlinger component in the aforementioned embodiment, which will not be repeated here to avoid repetition.
[0191] After the second graphic data is generated, the second application needs to be graphically displayed on the target screen based on the second graphic data.
[0192] In one implementation, the second graphics display component includes a second graphics buffer allocator and a second synthesizer.
[0193] In the process of processing the N layers to be displayed into the second graphic data by the second graphic display component, the second graphic buffer allocator can allocate a graphic buffer for the second application to draw the N layers to be displayed therein, and the second synthesizer can synthesize the N layers to be displayed in the graphic buffer into the second graphic data. The second synthesizer can further display the second application graphically on the target screen based on the second graphic data, thereby ensuring the effective implementation of the graphic data processing operation of the graphic display component.
[0194] Furthermore, in the process of graphically displaying the second application on the target screen based on the second graphic data, there are two possible implementation situations.
[0195] One is to directly display the second application graphically on the target screen based on the second graphic data by the second graphic display component, that is, to execute step 203. Another is to display the first application and the second application graphically on the target screen by the first graphic display component, that is, to execute step 204.
[0196] Specifically, the electronic device may determine whether a specific application is displayed on the target screen where the second application needs to be graphically displayed.
[0197] When there are multiple display screens in an electronic device, if the second application needs to be displayed on screen A, and the specific application is displayed on screen B, then screen A is used as the target screen for the second application. If the specific application is not displayed on the screen, the second application can be displayed on screen A (target screen) directly based on the second graphic data, and step 203 needs to be executed.
[0198] If there is only one display screen in the electronic device, and if the screen does not display the specific application, the second application can also be directly displayed on the display screen based on the second graphic data, and step 203 also needs to be executed.
[0199] When there are multiple display screens in the electronic device, if screen A is the target screen of the second application and a specific application is displayed on the screen, the second application cannot be displayed on screen A directly based on the second graphic data, and step 204 needs to be executed.
[0200] If there is only one display screen in the electronic device, if a specific application is displayed on the screen, the second application cannot be displayed on the display screen directly based on the second graphic data, and step 204 also needs to be executed.
[0201] Step 203 : When no graphical display screen of the specific application is displayed on the target screen, directly display the second application graphically on the target screen through the second graphical display component based on the second graphical data.
[0202] In the process of displaying the second application graphically on the target screen based on the second graphic data by the second graphic display component, the second graphic data can be directly output to the DPU for rendering and then directly displayed. Or when the DPU cannot respond to the display operation of the second image data (GPU hardware is damaged, the number of layers exceeds the set number of layers, etc.), the graphics library is called to use the GPU to render the second image data, and the rendered target graphic data is directly displayed, so that the second application is graphically displayed on the target screen.
[0203] This process uses two graphic display components provided by the electronic device to form two display processing channels, and can select the appropriate display processing channel for graphic display processing based on the current application. For non-specific applications, the existing graphic display components are used to perform graphic display processing through the corresponding display processing channel, which does not interfere with the data display processing of specific applications and reduces the graphic display delay of specific applications with priority display requirements and in high-load scenarios.
[0204] Step 204 : when the target screen displays a graphical display screen of a specific application, output the second graphical data to the first graphical display component through the second graphical display component.
[0205] During the implementation process, the second synthesizer in the second graphical display component is connected to the first synthesizer in the first graphical display component, and the second graphical display component sends the second graphic data to the first synthesizer in the first graphical display component through the second synthesizer, so that the second application and the first application can be displayed together on the target screen with the help of the first graphical display component.
[0206] Step 205, when the first graphic display component receives the second graphic data sent by the second graphic display component, the electronic device synthesizes the first graphic data and the second graphic data into target graphic data through the first graphic display component, and graphically displays the first application and the second application on the target screen based on the target graphic data.
[0207] The implementation process of the layer processing of the first graphical display component can refer to the description in the aforementioned embodiment, which will not be repeated here.
[0208] Specifically, after the first synthesizer in the first graphic display component obtains the second graphic data sent by the second synthesizer in the second graphic display component, the first synthesizer performs layer synthesis processing based on the first graphic data and the second graphic data to obtain target graphic data.
[0209] Here, the second graphical display component, with the help of the first graphical display component, realizes the display layer synthesis processing between the specific application and the non-specific application, and finally further integrates the layer integration data of the specific application and other applications for display, so as to meet the needs of other applications to temporarily superimpose graphical displays during the display process of the specific application, and the two graphical display components provided in the electronic device perform collaborative processing, so that the graphic display delay is small in the face of high-load scenarios.
[0210] The third scenario: Graphical display of applications in a split-screen display scenario.
[0211] This scenario corresponds to the situation where the electronic device includes multiple display screens. Fig.10 As shown, the display screen includes screen A and screen B. In the split-screen display mode, screen A and screen B are used to display different display images respectively.
[0212] In this scenario, the specific application may be an application set in any manner. The other applications may be one or more.
[0213] Specifically, the other application is a photo album application and the specific application is a video call application.
[0214] When the video call application is started, the electronic device uses the newly added Compositor component to perform graphical display processing on it, and outputs the graphical display processing results corresponding to the video call application to screen B for display. For the photo album application, it and the video call application are simultaneously started in some period of time, so the electronic device uses the original SurfaceFlinger component to perform graphical display processing on it, and outputs the graphical display processing results corresponding to the photo album application to screen A for display. See the display effect Fig.10 shown.
[0215] Among them, the graphical display processing process of the Compositor component for the video call application can refer to the description of the relevant functional implementation of the Compositor component in the aforementioned embodiment, and the graphical display processing process of the SurfaceFlinger component for the album application can refer to the description of the relevant functional implementation of the SurfaceFlinger component in the aforementioned embodiment. To avoid repetition, it will not be repeated here.
[0216] In this way, two applications that need to display images at the same time are respectively processed by different graphic display components and finally displayed on different display screens.
[0217] Due to differences in functions or application scenarios, different apps in electronic devices may require different display resolutions and refresh rates when displaying graphics.
[0218] For example, Fig.10 In the photo album preview screen displayed on screen A of the electronic device, in the photo album application, the required resolution for the screen display is high and the refresh rate is low. For example, the required display resolution for the photo album application is Resolution0 and the refresh rate is Refresh0.
[0219] Fig.10In the video call screen displayed on the screen of electronic device B, in the video call application, the resolution required for the screen display is low and the refresh rate is high. For example, the display resolution required by the video call application is Resolution1 and the refresh rate is Refresh1.
[0220] Then, Resolution1 is smaller than Resolution0, and Refresh1 is larger than Refresh0.
[0221] Correspondingly, the resolution and refresh rate of screen A and screen B in the display screen need to be set respectively. Optionally, the display resolution of screen A can be directly set to Resolution0, the refresh rate can be set to Refresh0, and the display resolution of screen A can be set to Resolution1, the refresh rate can be set to Refresh1.
[0222] In a split-screen scenario, the processing requirements of outputting the graphical display processing results formed by different display processing channels to the electronic device are met, and different screen display control parameters are used to display the images separately on the corresponding display screens. The two display processing channels can output two images with different frame rates and resolutions, meeting differentiated display requirements and reducing the power consumption of the entire system in a split-screen scenario.
[0223] In combination with the description of the third scenario above, the process of an electronic device performing graphical display using the graphical display method in an embodiment of the present application is described in detail.
[0224] Fig.11 It is a schematic flow chart of the graphical display method involved in the third scenario.
[0225] In the third scenario, the process of the electronic device performing graphical display using the graphical display method in the embodiment of the present application can refer to the following description of steps 301 to 304.
[0226] Step 301: when a specific application is started, determine that the target graphical display component associated with the specific application is a first graphical display component.
[0227] Step 302: when other applications are started, determine that the target graphical display component associated with the other applications is the second graphical display component.
[0228] When multiple applications are started at the same time, including a specific application and several other applications, and the electronic device includes multiple display screens for graphically displaying these applications, then, in order to ensure the running effect of the specific application, the specific application can be set to be displayed on a single display screen, and other applications can be displayed on another display screen.
[0229] The execution order between step 301 and step 302 is not particularly limited.
[0230] Step 303: Performing graphical display processing on the to-be-displayed layer of the specific application through the first graphical display component, and displaying the specific application on the first screen at the first resolution and / or the first refresh rate.
[0231] Step 304: Perform graphical display processing on the to-be-displayed layers of other applications through the second graphical display component, and display the other applications on the second screen at a second resolution and / or a second refresh rate.
[0232] The execution order between step 303 and step 304 is not particularly limited.
[0233] Among them, at least one of the parameter values of the display parameters in the first screen and the second screen is different, so as to realize differentiated application graphical display.
[0234] In this way, after performing graphical display processing on N layers to be displayed of different applications, the corresponding applications will be displayed on their respective adapted screens with appropriate screen display control parameters, adapting to the differentiated display requirements of different applications and improving the display effect.
[0235] In the embodiments of the present application, there are some similarities and differences in the graphical display methods in different scenarios. The relevant description contents of these parts can be borrowed and integrated with each other, and there are no obstacles to the implementation of technical combination due to differences in scene descriptions.
[0236] Fig.12 is a schematic diagram of a graphical display device according to an embodiment of the present application. Fig.12 As shown, the device 2000 includes a component determination module 2001 and a display processing module 2002 .
[0237] Among them, the component determination module 2001 is used to determine the target graphical display component associated with the first application when the first application is started; the target graphical display component is the first graphical display component or the second graphical display component; the display processing module 2002 is used to perform graphical display processing on the N layers to be displayed of the first application through the target graphical display component, and display the first application on the target screen; N is an integer greater than or equal to 1.
[0238] The device 2000 can be integrated into electronic devices such as mobile phones, tablet computers, and smart wearable devices.
[0239] The device 2000 can be used to execute any one of the above-mentioned graphical display methods.
[0240] In one implementation, the apparatus 2000 may further include a storage unit for storing data such as images, recognized image features, etc. The storage unit may be integrated in any one of the above units, or may be a unit independent of all the above units.
[0241] Fig.13 Schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. Fig.13 As shown, the electronic device 900 may include a processor 910, an external memory interface 920, an internal memory 921, a universal serial bus (USB) interface 930, a charging management module 940, a power management module 941, a battery 942, an antenna 1, an antenna 2, a mobile communication module 950, a wireless communication module 960, an audio module 970, a speaker 970A, a receiver 970B, a microphone 970C, an earphone interface 970D, a sensor module 980, a button 990, a motor 991, an indicator 992, a camera 993, a display screen 994, and a subscriber identification module (SIM) card interface 995, etc. Among them, the sensor module 980 may include a pressure sensor 980A, a gyroscope sensor 980B, an air pressure sensor 980C, a magnetic sensor 980D, an acceleration sensor 980E, a distance sensor 980F, a proximity light sensor 980G, a fingerprint sensor 980H, a temperature sensor 980J, a touch sensor 980K, an ambient light sensor 980L, a bone conduction sensor 980M, etc.
[0242] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 900. In other embodiments of the present application, the electronic device 900 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0243] For example, Fig.13The processor 910 shown may include one or more processing units, for example, the processor 910 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.
[0244] The controller may be the nerve center and command center of the electronic device 900. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.
[0245] The processor 910 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 910 is a cache memory. The memory may store instructions or data that the processor 910 has just used or cyclically used. If the processor 910 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 910, and thus improves the efficiency of the system.
[0246] In some embodiments, the processor 910 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.
[0247] In some embodiments, the I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). The processor 910 may include multiple groups of I2C buses. The processor 910 may be coupled to the touch sensor 980K, the charger, the flash, the camera 993, etc. through different I2C bus interfaces. For example, the processor 910 may be coupled to the touch sensor 980K through the I2C interface, so that the processor 910 communicates with the touch sensor 980K through the I2C bus interface to realize the touch function of the electronic device 900.
[0248] In some embodiments, the I2S interface can be used for audio communication. The processor 910 can include multiple I2S buses. The processor 910 can be coupled to the audio module 970 via the I2S bus to achieve communication between the processor 910 and the audio module 970.
[0249] In some embodiments, the audio module 970 can transmit audio signals to the wireless communication module 960 through the I2S interface to implement the function of answering calls through a Bluetooth headset.
[0250] In some embodiments, the PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. The audio module 970 and the wireless communication module 960 can be coupled via a PCM bus interface.
[0251] In some embodiments, the audio module 970 can also transmit audio signals to the wireless communication module 960 through the PCM interface to implement the function of answering calls through a Bluetooth headset. It should be understood that both the I2S interface and the PCM interface can be used for audio communication.
[0252] In some embodiments, the UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. The UART interface is generally used to connect the processor 910 and the wireless communication module 960. For example, the processor 910 communicates with the Bluetooth module in the wireless communication module 960 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 970 can transmit an audio signal to the wireless communication module 960 through the UART interface to implement the function of playing music through a Bluetooth headset.
[0253] In some embodiments, the MIPI interface can be used to connect the processor 910 with peripheral devices such as the display screen 994 and the camera 993. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. The processor 910 and the camera 993 communicate through the CSI interface to realize the shooting function of the electronic device 900. The processor 910 and the display screen 994 communicate through the DSI interface to realize the display function of the electronic device 900.
[0254] In some embodiments, the GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. The GPIO interface can be used to connect the processor 910 with the camera 993, the display screen 994, the wireless communication module 960, the audio module 970, the sensor module 980, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.
[0255] Exemplarily, the USB interface 930 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 900, and can also be used to transfer data between the electronic device 900 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.
[0256] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 900. In other embodiments of the present application, the electronic device 900 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
[0257] The charging management module 940 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 940 may receive charging input from a wired charger through the USB interface 930. In some wireless charging embodiments, the charging management module 940 may receive wireless charging input through a wireless charging coil of the electronic device 900. While the charging management module 940 is charging the battery 942, it may also power the electronic device through the power management module 941.
[0258] The power management module 941 is used to connect the battery 942, the charging management module 940 and the processor 910. The power management module 941 receives input from the battery 942 and / or the charging management module 940, and supplies power to the processor 910, the internal memory 921, the external memory, the display screen 994, the camera 993, and the wireless communication module 960. The power management module 941 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 941 can also be set in the processor 910. In other embodiments, the power management module 941 and the charging management module 940 can also be set in the same device.
[0259] The wireless communication function of the electronic device 900 can be implemented through antenna 1, antenna 2, mobile communication module 950, wireless communication module 960, modem processor and baseband processor.
[0260] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 900 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.
[0261] The mobile communication module 950 can provide a wireless communication solution applied to the electronic device 900, such as at least one of the following solutions: a second generation (2G) mobile communication solution, a third generation (3G) mobile communication solution, a fourth generation (5G) mobile communication solution, and a fifth generation (5G) mobile communication solution. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 950 can receive electromagnetic waves from the antenna 1, and filter and amplify the received electromagnetic waves, and then transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 950 can also amplify the signal modulated by the modulation and demodulation processor, and the amplified signal is converted into an electromagnetic wave and radiated out through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 910. In some embodiments, at least some of the functional modules of the mobile communication module 950 can be set in the same device as at least some of the modules of the processor 910.
[0262] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 970A, a receiver 970B, etc.), or displays an image or video through a display screen 994. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 910 and be set in the same device as the mobile communication module 950 or other functional modules.
[0263] The wireless communication module 960 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 900. The wireless communication module 960 can be one or more devices integrating at least one communication processing module. The wireless communication module 960 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 910. The wireless communication module 960 can also receive the signal to be sent from the processor 910, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.
[0264] In some embodiments, the antenna 1 of the electronic device 900 is coupled to the mobile communication module 950, and the antenna 2 of the electronic device 900 is coupled to the wireless communication module 960, so that the electronic device 900 can communicate with the network and other electronic devices through wireless communication technology. The wireless communication technology may include at least one of the following communication technologies: global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, IR technology. The GNSS may include at least one of the following positioning technologies: global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and satellite based augmentation systems (SBAS).
[0265] The electronic device 900 implements the display function through a GPU, a display screen 994, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 994 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 910 may include one or more GPUs, which execute program instructions to generate or change display information.
[0266] The display screen 994 is used to display images, videos, etc. The display screen 994 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini-LED, Micro-Led, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 900 may include 1 or N display screens 994, where N is a positive integer greater than 1.
[0267] The electronic device 900 can realize the shooting function through an ISP, a camera 993, a video codec, a GPU, a display screen 994, and an application processor.
[0268] ISP is used to process the data fed back by camera 993. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 993.
[0269] The camera 993 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 900 may include 1 or N cameras 993, where N is a positive integer greater than 1.
[0270] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 900 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.
[0271] Video codecs are used to compress or decompress digital videos. The electronic device 900 may support one or more video codecs. Thus, the electronic device 900 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
[0272] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 900 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.
[0273] The external memory interface 920 can be used to connect an external memory card, such as a secure digital (SD) card, to expand the storage capacity of the electronic device 900. The external memory card communicates with the processor 910 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.
[0274] The internal memory 921 can be used to store computer executable program codes, which include instructions. The processor 910 executes various functional applications and data processing of the electronic device 900 by running the instructions stored in the internal memory 921. The internal memory 921 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 900 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 921 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
[0275] The electronic device 900 can implement audio functions such as music playing and recording through the audio module 970, the speaker 970A, the receiver 970B, the microphone 970C, the headphone jack 970D, and the application processor.
[0276] The audio module 970 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 970 can also be used to encode and decode audio signals. In some embodiments, the audio module 970 can be arranged in the processor 910, or some functional modules of the audio module 970 can be arranged in the processor 910.
[0277] The speaker 970A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 900 can listen to music or listen to a hands-free call through the speaker 970A.
[0278] The receiver 970B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 900 receives a call or voice message, the voice can be received by placing the receiver 970B close to the human ear.
[0279] Microphone 970C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 970C to input the sound signal into the microphone 970C. The electronic device 900 can be provided with at least one microphone 970C. In other embodiments, the electronic device 900 can be provided with two microphones 970C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 900 can also be provided with three, four or more microphones 970C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.
[0280] The earphone interface 970D is used to connect a wired earphone and can be the USB interface 930 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.
[0281] The pressure sensor 980A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 980A can be set on the display screen 994. There are many types of pressure sensors 980A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 980A, the capacitance between the electrodes changes. The electronic device 900 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 994, the electronic device 900 detects the touch operation intensity according to the pressure sensor 980A. The electronic device 900 can also calculate the touch position according to the detection signal of the pressure sensor 980A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.
[0282] The gyroscope sensor 980B can be used to determine the motion posture of the electronic device 900. In some embodiments, the angular velocity of the electronic device 900 around three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 980B. The gyroscope sensor 980B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 980B detects the angle of the electronic device 900 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 900 through reverse movement to achieve anti-shake. The gyroscope sensor 980B can also be used for navigation and somatosensory game scenes.
[0283] The air pressure sensor 980C is used to measure air pressure. In some embodiments, the electronic device 900 calculates the altitude through the air pressure value measured by the air pressure sensor 980C to assist in positioning and navigation.
[0284] The magnetic sensor 980D includes a Hall sensor. The electronic device 900 can use the magnetic sensor 980D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 900 is a flip phone, the electronic device 900 can detect the opening and closing of the flip cover according to the magnetic sensor 980D; according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, set the flip cover to automatically unlock and other features.
[0285] The acceleration sensor 980E can detect the magnitude of the acceleration of the electronic device 900 in all directions (generally three axes). When the electronic device 900 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.
[0286] The distance sensor 980F is used to measure the distance. The electronic device 900 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 900 can use the distance sensor 980F to measure the distance to achieve fast focusing.
[0287] The proximity light sensor 980G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The light-emitting diode may be an infrared light-emitting diode. The electronic device 900 emits infrared light outward through the light-emitting diode. The electronic device 900 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 900. When insufficient reflected light is detected, the electronic device 900 can determine that there is no object near the electronic device 900. The electronic device 900 can use the proximity light sensor 980G to detect that the user holds the electronic device 900 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 980G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.
[0288] The ambient light sensor 980L is used to sense the ambient light brightness. The electronic device 900 can adaptively adjust the brightness of the display screen 994 according to the perceived ambient light brightness. The ambient light sensor 980L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 980L can also cooperate with the proximity light sensor 980G to detect whether the electronic device 900 is in a pocket to prevent accidental touches.
[0289] The fingerprint sensor 980H is used to collect fingerprints. The electronic device 900 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.
[0290] The temperature sensor 980J is used to detect temperature. In some embodiments, the electronic device 900 uses the temperature detected by the temperature sensor 980J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 980J exceeds a threshold, the electronic device 900 reduces the performance of the processor located near the temperature sensor 980J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 900 heats the battery 942 to avoid abnormal shutdown of the electronic device 900 due to low temperature. In some other embodiments, when the temperature is lower than another threshold, the electronic device 900 performs a boost on the output voltage of the battery 942 to avoid abnormal shutdown caused by low temperature.
[0291] The touch sensor 980K is also called a "touch panel". The touch sensor 980K can be set on the display screen 994, and the touch sensor 980K and the display screen 994 form a touch screen, also called a "touch screen". The touch sensor 980K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 994. In other embodiments, the touch sensor 980K can also be set on the surface of the electronic device 900, which is different from the position of the display screen 994.
[0292] The bone conduction sensor 980M can obtain a vibration signal. In some embodiments, the bone conduction sensor 980M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 980M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 980M can also be set in an earphone and combined into a bone conduction earphone. The audio module 970 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 980M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 980M to realize a heart rate detection function.
[0293] The key 990 includes a power key, a volume key, etc. The key 990 may be a mechanical key or a touch key. The electronic device 900 may receive key input and generate key signal input related to user settings and function control of the electronic device 900.
[0294] Motor 991 can generate vibration prompts. Motor 991 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 994, motor 991 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.
[0295] Indicator 992 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.
[0296] The SIM card interface 995 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 900 by inserting it into the SIM card interface 995 or pulling it out from the SIM card interface 995. The electronic device 900 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 995 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 995 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 995 can also be compatible with different types of SIM cards. The SIM card interface 995 can also be compatible with external memory cards. The electronic device 900 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 900 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 900 and cannot be separated from the electronic device 900.
[0297] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.
[0298] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0299] An embodiment of the present application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor can implement the steps in any of the above methods when executing the computer program.
[0300] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.
[0301] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.
[0302] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.
[0303] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0304] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0305] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0306] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0307] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0308] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0309] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.
[0310] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0311] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0312] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A graphical display method, characterized in that: Applied to an electronic device comprising a first graphic display component and a second graphic display component, wherein the first graphic display component is used to perform a first graphic display process on a specific application, and the second graphic display component is used to perform a second graphic display process on other applications other than the specific application; The method comprises: When a first application is started, determining a target graphical display component associated with the first application; the target graphical display component is the first graphical display component or the second graphical display component; Graphical display processing is performed on the N to-be-displayed layers of the first application through the target graphical display component, and the first application is displayed on the target screen; N is an integer greater than or equal to 1.
2. The method according to claim 1, characterized in that The performing graphical display processing on the N to-be-displayed layers of the first application by the target graphical display component to display the first application on the target screen includes: In the case where the target graphical display component is the first graphical display component, the N layers to be displayed are processed into target graphical data by the first graphical display component, and the first application is graphically displayed on the target screen based on the target graphical data.
3. The method according to claim 2, characterized in that The processing of the N to-be-displayed layers into target graphic data by the first graphic display component includes: directly processing the N layers to be displayed into target graphic data through the first graphic display component; or, Processing the N layers to be displayed into first graphic data by the first graphic display component; When the first graphic display component receives the second graphic data sent by the second graphic display component, the first graphic display component processes the first graphic data and the second graphic data into the target graphic data.
4. The method according to claim 2, characterized in that: The step of graphically displaying the first application on the target screen based on the target graphic data includes: The graphics library is called to render the target graphics data, and the rendered target graphics data is graphically displayed on the target screen.
5. The method according to any one of claims 2 to 4, characterized in that: The first graphic display component includes a first graphic buffer allocator and a first synthesizer; The processing of the N to-be-displayed layers into target graphic data by the first graphic display component, and graphically displaying the first application on the target screen based on the target graphic data, includes: Allocate a graphics buffer through the first graphics buffer allocator, wherein the graphics buffer is used by the first application to draw the N layers to be displayed; The N to-be-displayed layers in the graphics buffer are processed into the target graphics data by the first synthesizer, and the first application is graphically displayed on the target screen based on the target graphics data.
6. The method according to any one of claims 1 to 4, characterized in that: When the target graphical display component is the first graphical display component, the first application is displayed on the target screen in full screen mode.
7. The method according to claim 1, characterized in that The performing graphical display processing on the N to-be-displayed layers of the first application by the target graphical display component to display the first application on the target screen includes: In the case where the target graphical display component is the second graphical display component, the N layers to be displayed are processed into second graphical data by the second graphical display component, and the first application is graphically displayed on the target screen based on the second graphical data.
8. The method according to claim 7, characterized in that The step of graphically displaying the first application on the target screen based on the second graphic data includes: When the graphical display screen of the specific application is displayed on the target screen, the second graphical data is output to the first graphical display component through the second graphical display component, so that the first application is displayed on the target screen through the first graphical display component.
9. The method according to claim 7, characterized in that: The step of graphically displaying the first application on the target screen based on the second graphic data includes: In a case where the graphical display screen of the specific application is not displayed on the target screen, the first application is graphically displayed on the target screen directly through the second graphical display component based on the second graphical data.
10. The method according to claim 9, characterized in that The step of graphically displaying the first application on the target screen based on the second graphic data by the second graphic display component includes: outputting the second graphic data to the DPU through the second graphic display component to display on the target screen; or, The second graphic data is output to the DPU through the second graphic display component. When the DPU is unable to perform a display operation in response to the second image data, the graphics library is called to render the second image data, and the rendered second image data is graphically displayed on the target screen.
11. The method according to any one of claims 7 to 10, characterized in that: The second graphic display component includes a second graphic buffer allocator and a second synthesizer; The processing of the N to-be-displayed layers into second graphic data by the second graphic display component, and graphically displaying the first application on the target screen based on the second graphic data, includes: Allocate a graphics buffer through the second graphics buffer allocator, wherein the graphics buffer is used by the first application to draw the N layers to be displayed; The N to-be-displayed layers in the graphics buffer are processed into second graphics data by the second synthesizer, and the first application is graphically displayed on the target screen based on the second graphics data.
12. The method according to claim 1, characterized in that The electronic device is equipped with at least two display screens; before displaying the first application graphically on the target screen, the method further includes: Determining display control information associated with the first application in the electronic device; If the display control information indicates that the first application is displayed on a first screen in the electronic device, the first screen is the target screen; If the display control information indicates that the first application is displayed on a second screen in the electronic device, the second screen is the target screen; If the electronic device is in an unfolded state and the display control information indicates that the first application is displayed in full screen in the electronic device, a screen formed by splicing the first screen and the second screen is the target screen.
13. The method according to claim 1, characterized in that There are multiple first applications, and the multiple first applications include the specific application and the other applications; the electronic device is configured with at least a first screen and a second screen; When the first application is started, determining the target graphical display component associated with the first application includes: When the specific application is started, determining that the target graphical display component associated with the specific application is the first graphical display component; When the other application is started, determining that the target graphical display component associated with the other application is the second graphical display component; The performing graphical display processing on the N to-be-displayed layers of the first application by the target graphical display component to display the first application on the target screen includes: Performing graphical display processing on the to-be-displayed layer of the specific application through the first graphical display component, and displaying the specific application on the first screen at a first resolution and / or a first refresh rate; Graphical display processing is performed on the to-be-displayed layer of the other application through the second graphical display component, and the other application is displayed on the second screen at a second resolution and / or a second refresh rate.
14. A graphical display device, characterized in that: The system comprises a first graphical display component and a second graphical display component, wherein the first graphical display component is used to perform a first graphical display process on a specific application, and the second graphical display component is used to perform a second graphical display process on other applications other than the specific application; The device comprises: a component determination module, configured to determine, when a first application is started, a target graphical display component associated with the first application; the target graphical display component being the first graphical display component or the second graphical display component; The display processing module is used to perform graphical display processing on the N to-be-displayed layers of the first application through the target graphical display component, and display the first application on the target screen; N is an integer greater than or equal to 1.
15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 13 is implemented.
16. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.
Citation Information
Patent Citations
Human-machine interface display processing device and method
CN106354455A
High-energy-efficiency display processing method and device
CN112860428A
Layer synthesis mode configuration method and electronic equipment
CN113012263A
Image display scheduling method applied to Android system, terminal and storage medium
CN113064727A
Method for displaying multiple windows and electronic equipment
CN114040238A