Image processing method and apparatus

Abstract

The present application relates to the technical field of terminals, and provides an image processing method and device. By means of the image processing method and device, when a user uses applications such as an electronic game and an electronic book, the user's hands causes less obstruction to the user's field of view, facilitating improvement of user experience. The method comprises: displaying a first interface of a first application, wherein the first interface comprises a first application picture display area and a first control area, the first application picture display area comprises one or more controllable objects, the first control area comprises one or more controls, the control of the first control area is used for controlling the controllable object in the first application picture display area, and the first application picture display area does not overlap the first control area; and in response to a first operation on a first control in the first control area, controlling a first controllable object in the first application picture display area.

Description

Image processing method and apparatus

[0001] This application claims priority from the Chinese patent application No. 202410682742.0 filed on May 28, 2024, and entitled "Image processing method and apparatus", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of terminals, and in particular to an image processing method and apparatus. BACKGROUND

[0003] With the development of terminal technology, applications such as electronic games and electronic books are widely promoted and applied in terminal devices, such as multiplayer online battle arena games. In order to have a better visual experience, users can also use folding screen devices or tablet computers and other electronic devices with larger display screens to experience electronic games and electronic books and other applications. In this way, during the experience of electronic games and electronic books and other applications, the content such as game characters in the interface displayed by the display screen of the electronic device can be larger.

[0004] However, during the experience of electronic games and electronic books and other applications, the user's field of view can be limited, resulting in poor user experience. SUMMARY

[0005] The present application provides an image processing method and apparatus, which makes the user experience electronic games and electronic books and other applications less obstructed by the user's hands to the user's field of view, helping to improve the user experience.

[0006] In a first aspect, an image processing method is provided, the method comprising: displaying a first interface of a first application, the first interface comprising a first application picture display area and a first control area, the first application picture display area comprising one or more controllable objects, the first control area comprising one or more controls, the controls of the first control area being used to control the controllable objects in the first application picture display area, the first application picture display area and the first control area being non-overlapping; in response to a first operation on a first control in the first control area, controlling a first controllable object in the first application picture display area.

[0007] The image processing method of the present application makes the application picture display area and the control area in the interface of the first application displayed by the electronic device non-overlapping, so that during the process that the user controls the controllable objects in the application picture display area through the controls in the control area, the user's hands are less obstructed to the application picture display area, so that the user's field of view is basically not limited, helping to improve the user experience.

[0008] With reference to the first aspect, in some implementations of the first aspect, the method is applied to a foldable electronic device, the foldable electronic device includes a foldable display screen, a first display part of the foldable display screen is foldable with a second display part of the foldable display screen; and displaying a first interface of a first application includes: displaying the first interface when the foldable display screen is in an unfolded state, the first application picture display area is in the first display part, and the first control area is in the second display part.

[0009] In this way, when the foldable display screen is in the unfolded state, the area of the foldable display screen is relatively large, the controllable objects displayed in the first display part of the foldable display screen and the controls displayed in the second display part do not overlap, so that the user's view is not limited during the process of controlling the controls by the hand.

[0010] With reference to the first aspect, in some implementations of the first aspect, the method further includes: displaying a second interface of the first application when the foldable display screen is in a folded state, the second interface includes a second application picture display area and a second control area, the second application picture display area includes one or more controllable objects, the second control area includes one or more controls, the controls of the second control area are used to control the controllable objects in the second application picture display area, and the second control area is located on the second application picture display area.

[0011] In this way, when the foldable display screen is in the folded state, the area of the available screen in the foldable electronic device is relatively small, and the controllable objects and the controls displayed by the foldable electronic device can overlap, so that the controllable objects and the controls displayed by the foldable electronic device are not too small, thereby not affecting the visual experience of the user.

[0012] In the case of an inner folding foldable device, the available screen in the foldable electronic device can be an outer screen of the foldable electronic device; in the case of an outer folding foldable device, the available screen in the foldable electronic device can be the first display part or the second display part. The present application does not make specific limitations thereto.

[0013] With reference to the first aspect, in some implementations of the first aspect, displaying the first interface of the first application includes: displaying the first interface when the state of the first switch is an open state.

[0014] In this way, the electronic device can determine whether to display the first interface according to the state of the first switch.

[0015] With reference to the first aspect, in some implementations of the first aspect, before displaying the first interface, the method further includes: displaying a third interface, the third interface including a first switch; in response to a second operation on the first switch, a state of the first switch is set to an on state.

[0016] In this way, the user can set the state of the first switch according to the needs, so that the electronic device displays the first interface or the second interface.

[0017] With reference to the first aspect, in some implementations of the first aspect, displaying the first interface of the first application includes: in a case where the first application is in a full-screen display state, displaying the first interface.

[0018] In this way, since in a case where the first application is in a non-full-screen display state, the area of the display screen corresponding to the first application can be small, the controls and the controllable objects displayed in the first interface can be small. Therefore, in a case where the first application is in a full-screen display state, displaying the first interface can provide a better visual experience for the user.

[0019] With reference to the first aspect, in some implementations of the first aspect, the first application is a game application; and displaying the first interface of the first application includes: in a case where the first application is in a match mode, displaying the first interface.

[0020] Since in a case where the first application is not in the match mode, the display screen corresponding to the first application can not include the controls in the first control area, the first screen display area and the first control area do not need to be separated. Therefore, in a case where the first application is in the match mode, the first device displays the first interface.

[0021] With reference to the first aspect, in some implementations of the first aspect, before displaying the first interface of the first application, the method further includes: drawing a first layer, the first layer including a first area and a second area, the first area including one or more controllable objects included in a first application screen display area, and the second area including one or more controls included in a first control area, the one or more controls included in the first control area being drawn in a first window corresponding to the first application, the first window having a same size as the first control area, and the first layer having a same resolution as a resolution of an image displayed by the first interface; and enlarging a size of the first layer to obtain the image displayed by the first interface.

[0022] In this way, the first layer has the same resolution as the image displayed by the first interface, so that the first interface has a higher definition.

[0023] With reference to the first aspect, in some implementations of the first aspect, before drawing the first layer, the method further includes: setting a range parameter of a second window of the first application to obtain the first window, the range parameter being used to describe a position and a size of the second window.

[0024] In this way, by setting the range parameters of the second window of the first application, the size and position of the second window can be adjusted to obtain the first window.

[0025] In conjunction with the first aspect, in some implementations of the first aspect, drawing the first layer includes: enlarging the displayed canvas to obtain a first canvas, the size of which is equal to the sum of the first application screen display area and the first control area; and drawing the first layer on the first canvas.

[0026] In this way, by doubling the size of the canvas corresponding to the first layer, the resolution of the first layer is doubled, resulting in a higher resolution for the first interface.

[0027] In conjunction with the first aspect, in some implementations of the first aspect, before drawing the first layer, the method further includes: determining whether one or more controllable objects included in the first application screen display area can be drawn in the first area, and whether one or more controls included in the first control area can be drawn in the second area; if so, drawing the first layer.

[0028] This helps to reduce the phenomenon of drawing manipulable objects in the second area and / or drawing controls in the first area.

[0029] In conjunction with the first aspect, in some implementations of the first aspect, determining whether one or more controllable objects included in the display area of ​​the first application screen can be drawn in the first area, and whether one or more controls included in the first control area can be drawn in the second area, includes: determining whether the resource library corresponding to the first application includes preset resources; or, determining whether the version of the first application is a preset version.

[0030] In this way, the electronic device can determine whether it is possible to draw a first layer that separates the first screen display area and the first control area based on the resource library corresponding to the first application or the version of the first application.

[0031] In conjunction with the first aspect, in some implementations of the first aspect, drawing the first layer includes: obtaining a first rendering instruction from the rendering thread of the first application, the first rendering instruction including information for instructing the resources used to draw one or more controllable objects included in the display area of ​​the first application screen and the resources used to draw one or more controls included in the first control area; adding a first instruction to the first rendering instruction to obtain a second rendering instruction, the first instruction being used to instruct the resources used to draw one or more controllable objects included in the display area of ​​the first application screen to be drawn in the first area, and instructing the resources used to draw one or more controls included in the first control area to be drawn in the second area; and drawing the first layer in the first window in response to the second rendering instruction.

[0032] In this way, the first device can draw a first layer that separates the controllable objects and controls based on the second rendering instructions.

[0033] In conjunction with the first aspect, in some implementations of the first aspect, the position of the first window is the same as the position of the first control area.

[0034] In this way, one or more controls in the first control area can be drawn in the first window, and the touch logic of one or more controls can also be set in the first window. In subsequent rendering and overlay processing of the first layer, there is no need to modify the touch logic of the controls.

[0035] Secondly, an image processing apparatus is provided for performing the method in any possible implementation of the first aspect described above. Specifically, the apparatus includes a module for performing the method in any possible implementation of the first aspect described above.

[0036] Thirdly, this application provides yet another image processing apparatus, including a processor coupled to a memory, which can be used to execute instructions in the memory to implement the method in any of the possible implementations of the first aspect described above. Optionally, the apparatus further includes a memory. Optionally, the apparatus further includes a communication interface, to which the processor is coupled.

[0037] In another implementation, the device is a chip configured in an electronic device. When the device is a chip configured in an electronic device, the aforementioned communication interface can be an input / output interface.

[0038] Fourthly, a processor is provided, comprising: an input circuit, an output circuit, and a processing circuit. The processing circuit is configured to receive signals through the input circuit and transmit signals through the output circuit, causing the processor to execute the method in any possible implementation of the first aspect described above.

[0039] In the specific implementation process, the processor can be a chip, the input circuit can be an input pin, the output circuit can be an output pin, and the processing circuit can be a transistor, gate circuit, flip-flop, and various logic circuits. The input signal received by the input circuit can be received and input by, for example, but not limited to, a receiver, and the signal output by the output circuit can be output to, for example, but not limited to, a transmitter and transmitted by the transmitter. Furthermore, the input circuit and the output circuit can be the same circuit, which is used as the input circuit and the output circuit at different times. This application does not limit the specific implementation of the processor and various circuits.

[0040] Fifthly, a processing apparatus is provided, including a processor and a memory. The processor is used to read instructions stored in the memory and to receive signals via a receiver and transmit signals via a transmitter to execute the method in any of the possible implementations of the first aspect described above.

[0041] Optionally, the processor may be one or more, and the memory may be one or more.

[0042] Optionally, the memory may be integrated with the processor, or the memory may be separated from the processor.

[0043] In the specific implementation process, the memory can be a non-transitory memory, such as read-only memory (ROM), which can be integrated with the processor on the same chip or set on different chips. This application does not limit the type of memory or the way the memory and processor are set.

[0044] It should be understood that the relevant data interaction process, such as sending instruction information, can be a process of outputting instruction information from the processor, and receiving capability information can be a process of the processor receiving input capability information.

[0045] The processing device in the fifth aspect above can be a chip. The processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. The memory can be integrated into the processor or located outside the processor and exist independently.

[0046] In a sixth aspect, a computer program product is provided, the computer program product comprising: a computer program (also referred to as code or instructions), which, when the computer program is run, causes a computer to perform the method in any of the possible implementations of the first aspect described above.

[0047] In a seventh aspect, a computer-readable storage medium is provided that stores a computer program (also referred to as code or instructions) that, when executed on a computer, causes the computer to perform the methods in any of the possible implementations of the first aspect described above. Attached Figure Description

[0048] Figure 1 is a schematic diagram of a two-fold foldable device;

[0049] Figure 2 is a schematic diagram of a three-fold foldable device;

[0050] Figure 3 is a schematic diagram of the folding process of a two-fold foldable device;

[0051] Figure 4 is a schematic diagram of a two-fold foldable device in landscape mode;

[0052] Figure 5 is a schematic diagram of a two-fold foldable device in its vertical screen configuration;

[0053] Figure 6 is a comparative diagram of a non-foldable mobile phone interface and a two-foldable screen mobile phone interface.

[0054] Figure 7 is a schematic diagram of the game interface of a two-fold foldable screen phone.

[0055] Figure 8 is a schematic diagram of the game interface of another two-fold foldable screen mobile phone provided in the embodiment of this application;

[0056] Figure 9 is a schematic diagram of a synthesis game interface provided in an embodiment of this application;

[0057] Figure 10 is a schematic block diagram of the hardware architecture of the first device provided in an embodiment of this application;

[0058] Figure 11 is a schematic block diagram illustrating the interaction between the software architecture and hardware architecture of the first device provided in an embodiment of this application;

[0059] Figure 12 is a schematic flowchart of an image synthesis method provided in an embodiment of this application;

[0060] Figure 13 is a flowchart illustrating another image synthesis method provided in an embodiment of this application;

[0061] Figure 14 is a schematic diagram of an interface including a super split-screen switch provided in an embodiment of this application;

[0062] Figure 15 is a schematic diagram of the interface of a tablet computer provided in an embodiment of this application;

[0063] Figure 16 is a schematic diagram of a process for synthesizing a first interface according to an embodiment of this application;

[0064] Figure 17 is a schematic block diagram of an image processing apparatus provided in an embodiment of this application. Detailed Implementation

[0065] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0066] 1. Other terms

[0067] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order of execution. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0068] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0069] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0070] 2. Terminal equipment

[0071] The terminal device in this application embodiment can also be any form of electronic device. For example, electronic devices may include handheld devices with image processing functions, vehicle-mounted devices, etc. For example, some electronic devices include: mobile phones, tablets, PDAs, laptops, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, in-vehicle devices, wearable devices, terminal devices in 5G networks, or future evolution of public land mobile communication networks. Terminal devices in a network (PLMN), etc., are not limited to this in the embodiments of this application.

[0072] By way of example and not limitation, in this embodiment, the electronic device can also be a wearable device. Wearable devices, also known as wearable smart devices, are a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothing or accessories. Wearable devices are not merely hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are feature-rich, large in size, and can achieve complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those that focus on a specific type of application function and require the use of other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0073] Furthermore, in this embodiment of the application, the electronic device can also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0074] The electronic devices in the embodiments of this application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0075] In this embodiment, the electronic device or various network devices include a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on top of the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0076] 3. Foldable screen devices

[0077] This can be understood as an electronic device with a foldable screen. In other words, it is an electronic device that achieves a foldable screen display form through flexible components, such as hinges or flexible screens.

[0078] It should be understood that in this application embodiment, the foldable screen device can refer to an outward foldable screen device or an inward foldable screen device. The difference between an outward foldable screen device and an inward foldable screen device is that their folding directions are opposite. Furthermore, the foldable screen device can refer to various types, such as a two-fold foldable screen device or a three-fold foldable screen device. A two-fold foldable screen device can refer to an electronic device capable of folding the screen along one folding axis. For example, as shown in FIG1, a user can fold the screen of the first device 1000 along folding axis 110. A three-fold foldable screen device can refer to an electronic device capable of folding the screen along two folding axes respectively. For example, as shown in FIG2, a user can fold the screen of the electronic device 200 along folding axes 210 and folding axis 220. This application embodiment does not limit the specific type of foldable screen device.

[0079] 4. Folded state, intermediate state, and unfolded state

[0080] These can be used to describe three states of a foldable screen device. The folded state refers to the state where the angle between the two screen parts connected by the flexible component is less than or equal to a first threshold; the unfolded state refers to the state where the angle between the two screen parts connected by the flexible component is greater than or equal to a second threshold; and the intermediate state can be understood as the state between the folded state and the unfolded state.

[0081] In one possible implementation, the first threshold is equal to the second threshold. This implementation can be understood as the foldable screen device being either in a folded state or an unfolded state. That is, the foldable screen device has no intermediate state.

[0082] In another possible implementation, the second threshold is greater than the first threshold. In this implementation, the foldable screen device exists in an intermediate state, and the foldable screen device is in an intermediate state when the included angle between the two screens connected by the flexible component is greater than the first threshold and less than the second threshold.

[0083] It should be understood that the intermediate state can also be called a transitional state, a partially folded state, a partially unfolded state, an incompletely folded state, or an incompletely unfolded state, etc., and this application does not make any specific limitation on it.

[0084] 5. Main screen and cover screen

[0085] Foldable screen devices typically include multiple screens connected by flexible components. Furthermore, for the same foldable screen device, the screen used by the user when the device is in the folded state is usually different from the screen used when the device is in the unfolded state. The screen used by the user when the device is folded can be called the outer screen; the screen used by the user when the device is unfolded can be called the inner screen.

[0086] In other words, the inner screen can refer to the screen that displays the main content when the foldable device is in the unfolded state. It is usually a larger screen formed by multiple screens connected by flexible components, which is used to provide a wider field of view and a more comfortable operating experience.

[0087] The outer screen refers to the screen used to display content when a foldable device is in its folded state. It is usually a smaller screen (relative to the inner screen) and can be used to display basic information, notifications, and quick actions. The outer screen can typically display information such as time, date, and incoming call information, making it easier for users to operate the device when it is folded.

[0088] Below, taking a two-fold inward-folding screen phone as an example of a folding screen device, and referring to Figure 3, we will explain in detail the folding axis, folded state, intermediate state, unfolded state, inner screen and outer screen of the folding screen device.

[0089] Figure 3 is a schematic diagram of the folding process of a foldable screen mobile phone according to an embodiment of this application. In Figure 3, states (a) to (b) to (c) show the folding process of the foldable screen mobile phone 300 from the perspective of the outer screen; states (d) to (e) to (c) show the folding process of the foldable screen mobile phone 300 from the perspective of the inner screen.

[0090] State (a) shows the unfolded state of the foldable phone 300 as viewed from the outer screen. The foldable phone 300 includes an outer screen 301 and a back panel 302. A camera 303 may be disposed in the back panel 302. Folding the foldable phone 300 along the folding axis 304 results in state (b), the intermediate state. Continuing to fold the foldable phone 300, it results in state (c), the folded state. When the foldable phone 300 is in the folded state, the user can operate the foldable phone 300 through the outer screen 301.

[0091] State (d) represents the unfolded state of the foldable phone 300 as seen from the inner screen. The foldable phone 300 includes an inner screen 306. The inner screen 306 can also be understood as a large screen composed of screens 304 and 305. Folding the foldable phone 300 along the folding axis 304, the foldable phone 300 presents state (e), the intermediate state. Continuing to fold the foldable phone 300, the foldable phone 300 presents state (c), the folded state.

[0092] It should be understood that when the foldable screen device is another type of device, such as a three-fold foldable screen device or an outward foldable screen device, the folding process of the foldable screen device is similar to that shown in Figure 3, and will not be described in detail here.

[0093] 6. Landscape and portrait modes of the inner screen

[0094] Landscape and portrait modes describe the usage orientation of the inner screen of a unfolded foldable device. Landscape mode refers to the usage state of the inner screen when the folding axis of the foldable device is horizontal. For example, as shown in Figure 4, the folding axis 410 of the foldable phone is horizontal, and the inner screen is in landscape mode. Portrait mode refers to the usage state of the inner screen when the folding axis of the foldable device is vertical. For example, as shown in Figure 5, the folding axis 510 of the foldable phone is vertical, and the inner screen is in portrait mode.

[0095] It should be understood that the horizontal direction can also be called the transverse direction, and the vertical direction can also be called the vertical direction, etc. A horizontal folding axis can also be understood as the direction when the folding axis is parallel or approximately parallel to the line connecting the user's eyes; a vertical folding axis can also be understood as the direction when the folding axis is perpendicular or approximately perpendicular to the line connecting the user's eyes. This application does not specifically limit this.

[0096] Compared to regular smartphones, electronic devices with larger displays, such as foldable screen devices and tablets, offer users a wider field of view and a more comfortable operating experience. However, the following issues may arise when users utilize electronic devices with larger displays to use various applications.

[0097] Question 1: When using various applications on electronic devices with large displays, such as playing games on the inner screen of a foldable device, the field of view of the game scene displayed on the electronic device's screen is limited because many applications are not adapted to the resolution. In other words, the content displayed on the screen is actually less. In addition, the clarity of the interface displayed on the screen may also be poor.

[0098] Taking a user playing a game on a foldable screen phone as an example, as shown in Figure 6, for the same game scenario, when the user is using a non-foldable screen device 610, the game scenario is shown in interface (a) of Figure 6; when the user is using a bi-foldable screen device 620, the game scenario is shown in interface (b) of Figure 6. Compared to interface (a), interface (b) displays the content between lines 630 and 640 in interface (a), but does not display the content to the left of line 630 and the content to the right of line 640.

[0099] Question 2: Because electronic devices with large displays typically have large screen sizes, controls for manipulating controllable objects in applications may be distributed across various locations on the screen, making operation more difficult for users. For example, manipulating controls in a game can be challenging. For instance, some controls may be located at the top of the screen, while others are at the bottom. Users need to move a greater distance to operate controls located at the top and bottom, potentially leading to longer interaction times and impacting the gaming experience.

[0100] Question 3: Areas in electronic devices housing heat-generating components such as cameras and / or system-on-a-chip (SoC) may experience higher temperatures, resulting in a higher perceived temperature when the user's hand touches these areas. For example, the heat generated by a unfolded foldable screen device may concentrate on the back panel where the camera and / or SoC are located. Therefore, when the user's hand operates the inner screen portion located on the back panel, the temperature felt by the user's hand may be high, leading to a poor user experience.

[0101] Question 4: In most games, the control area overlaps with the field of view, meaning the control area is usually above the field of view. This means that the controls in the control area not only obstruct part of the field of view but also cause the user's hands to partially obstruct the screen when operating it. Furthermore, when the user is using the electronic device to experience other applications, their hands will also partially obstruct the screen when operating it. Therefore, this approach may limit the user's field of view, resulting in a poor user experience.

[0102] The control area refers to the area in the game that the user can operate and control. Controls may include, but are not limited to, one or more of the following types of controls: controls for moving the game character, controls for jumping the game character, controls for attacking the game character, or controls for using items, etc. The field of view refers to the range or angle that the user can see in the game, and it affects the range of scenery the user can observe. The field of view can include one or more game characters that can be controlled by the controls in the control area, as well as the environment in which the game characters are located. The environment in which the game characters are located can refer to scenery, such as trees, houses, or various obstacles. Alternatively, the field of view can also be understood as the game screen outside of the controls within the entire game scene.

[0103] For example, as shown in Figure 7, a user plays a game on a foldable screen phone. The interface 700 shown in Figure 7 includes multiple controls, such as controls 701, 702, 703, 704, 705, 706, 707, and 708. The area occupied by these controls can be understood as the control area. The user can use control 701 to determine or switch the position of the game character; the user can use control 702 to purchase in-game items; and the user can use control 702 to control the movement direction of the game character. The user can use controls 704 through 708 to release various skills. It can be seen that in interface 700, the control area overlaps with the field of view, and the control area is located above the field of view. When the user's hand manipulates the controls, it will obstruct part of the screen, thus obstructing part of the game's view within the field of view, affecting the user's gaming experience.

[0104] It should be noted that the interface shown in the embodiments of this application is only illustrative. The field of view in the interface may also include other game characters and the environment in which they are located. The interface may include more or fewer controls, and the functions of various controls may also be other functions. The positions of various controls may also be other positions in the interface. This application does not make any specific limitations on these aspects.

[0105] In view of this, this application provides an image processing method. The method is applied to a first device, the display screen of the first device including a first part screen and a second part screen. When a user plays a game on the first device, the first device positions the field of view in the first part screen of the display screen and the control area in the second part screen of the inner screen, and the first part screen and the second part screen do not overlap. In this way, when the user's hand operates on the second part screen, it will obstruct the first part screen less, which helps to reduce the obstruction of the user's field of view by the user's hand and improve the user's gaming experience.

[0106] Similarly, when a user is using other applications, the controllable objects in the interface of those other applications can also be located on the first part of the screen, while the controls for manipulating the controllable objects are located on the second part of the screen. This way, when the user's hands are operating on the second part of the screen, they will obstruct the first part of the screen less, which helps to reduce the obstruction of the user's field of vision by their hands.

[0107] For example, taking a foldable screen phone as the first device, as shown in Figure 8, the inner screen of the foldable screen phone consists of screen 810 and screen 820. Interface 800 consists of the game's field of view and control area. Screen 810 can be understood as the first part of the screen, i.e., the game's field of view is located on screen 810; screen 820 can be understood as the second part of the screen, i.e., the game's control area is located on screen 820. When the user's hand operates on screen 820, it will not significantly obstruct the field of view on screen 810.

[0108] It should be understood that the operation area in interface 800 includes multiple controls such as controls 801, 802, 803, 804, 805, 806, 807, and 808, which are similar to the multiple controls such as controls 701, 702, 703, 704, 705, 706, 707, and 708 in the operation area of ​​interface 700, as described above, and will not be repeated here.

[0109] Furthermore, since the area of ​​the second screen is smaller than that of the display screen of the first device, it is easier for the user to operate the controls when the game controls are located on the second screen, thereby helping to further improve the user's gaming experience.

[0110] Optionally, if the game is a mobile game and the aspect ratio of the first part of the screen is the same as that of the non-foldable screen phone, for games that have not undergone resolution adaptation, the interface displayed on the first part of the screen can be the same as the game screen displayed on the non-foldable phone, meaning the game's field of view will not be limited. For example, referring to Figures 6 and 8, the first part of the screen is screen 810, and the game scene displayed on screen 810 is the same as the game scene displayed on the non-foldable screen phone 610 in Figure 6.

[0111] Optionally, if heat-generating components such as the camera and / or system-on-a-chip (SOC) of the first device are located on the first portion of the screen, the temperature of the first portion of the screen may be higher than that of the second portion of the screen. Since the user's hand operates on the second portion of the screen, the user's hand does not need to touch the first portion of the screen, thus the temperature of the first device felt by the user's hand is relatively low, which helps to improve the user's gaming experience. For example, referring to Figure 8, assuming that the back of the screen 810 is a back panel, the camera and SOC of the foldable screen phone are located in the corresponding area of ​​the back panel, which may make the temperature of the back panel higher. When the user's hand operates on the screen 820, it does not need to touch the back panel, thus the temperature of the foldable screen device felt by the user's hand is relatively low.

[0112] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems will be described in detail below with reference to Figures 9 to 16 and specific embodiments. The following specific embodiments can be implemented independently or in combination with each other. The same or similar concepts or processes may not be described again in some embodiments.

[0113] The embodiments shown in this application can be executed by a first device, which may be a foldable screen device or an electronic device with a large screen but without a foldable screen. A foldable screen device can be understood as a terminal device with a foldable screen, such as a mobile phone with a foldable screen; an electronic device with a large screen but without a foldable screen may be a tablet computer. The specific form and number of the devices shown are merely examples and should not constitute any limitation on the implementation of the methods provided in this application.

[0114] The image processing method of this application embodiment will now be described in detail, taking a first device as the execution subject as an example. The first device is a foldable screen device.

[0115] It should be understood that the first device can be the first device itself, or a chip, chip system or processor that supports the first device in implementing the image processing method, or a logic module or software that can implement all or part of the functions of the first device. This application does not make any specific limitations in this regard.

[0116] First, to facilitate understanding of the embodiments of this application, taking the interface 800 that the first device needs to display as an example, the process of the first device synthesizing game images (i.e., the images displayed on interface 800) will be explained in conjunction with FIG9.

[0117] Figure 9 is a schematic diagram of a game image synthesis process 900 provided in an embodiment of this application. Assume the game image is an image displayed on a first device during a user's operation of a game application A, and the inner screen of the first device includes screen 801 and screen 802.

[0118] As shown in Figure 9(a), the first device first sets the size of window 901 corresponding to game application A so that window 901 is located on screen 802 and the size of window 901 is the same as the size of screen 802. Then, the first device obtains window 901 through application A. At this time, the size of the canvas generated by the first device for drawing game layers is the size of window 901.

[0119] Next, the frame buffer (FB) corresponding to the game layer is enlarged by one time. The frame buffer sent to the display can also be understood as FB 0. Alternatively, it can be understood as enlarging the canvas of the game layer by one time. This results in canvas 902 in Figure 9(b).

[0120] It should be understood that a frame buffer can be thought of as a memory area used to store image data, specifically the image data that will be displayed on the screen. FB 0 can be understood as the default frame buffer, which is the frame buffer used by default in the Android operating system, usually identified as FB 0. This frame buffer is the image data storage area that the display controller directly reads and displays on the screen.

[0121] Doubling the default framebuffer (FB 0) can also be understood as doubling the height of FB 0, i.e., FB 0 height × 2. The height of FB 0 refers to the height of the default framebuffer, expressed in pixels. `height` can represent the number of pixels in the vertical direction of FB 0. This vertical direction can also be understood as the y-axis direction. This y-axis can refer to the y-axis in the first device screen coordinate system.

[0122] It should be noted that although the canvas 902 of the game layer is distributed on the inner screen of the first device, the window corresponding to application A is still window 902 located on screen 802.

[0123] Next, the game layer is drawn on canvas 902 of the game layer. The game characters and other content in the field of view are drawn in the area corresponding to screen 801 in canvas 902; the controls in the control area are drawn in the area corresponding to screen 802 in canvas 902.

[0124] However, since the window corresponding to application A is still window 902 on screen 802, the display effect of the game layer drawn on canvas 902 is as shown in Figure 9(c) if it is to be displayed. That is, although the game layer drawn by the first device is a layer of canvas 902 size, the game layer is compressed and displayed in window 902.

[0125] To display the game layer on the entire inner screen of the first device, as shown in Figure 9(c), the first device doubles the vertical dimension of the game layer located on screen 802 as shown in Figure 9(b), so that the game layer covers the entire inner screen, thus obtaining a game image that covers the entire inner screen, i.e., the image displayed on interface 800. The view area 902 is located on screen 801, and the control area 903 is located on screen 802. At this time, although interface 800 is displayed on the entire inner screen, the window corresponding to application A is still window 902 located on screen 802.

[0126] It should be noted that the resolution of the game layer shown in Figure 9(b) is the same as that of the game layer shown in Figure 9(c). That is, compared with the game layer shown in Figure 9(b), the game layer shown in Figure 9(c) is simply enlarged in size.

[0127] It should also be noted that, in this embodiment, enabling the game layer to be displayed on the entire inner screen of the first device does not mean that the size of the game layer is exactly the same as the size of the inner screen of the first device, but rather that the game layer can be displayed on the inner screen of the first device in full-screen mode. The size of the game layer can also be slightly smaller than the size of the inner screen of the first device. This will not be elaborated further below.

[0128] Before rendering the game layer shown in Figure 9(b), the FB 0 of the game layer is doubled. This doubles the resolution (or number of pixels) of the game layer shown in Figure 9(b), resulting in higher clarity for the interface 800. Furthermore, by doubling the FB 0 of the game layer, during the rendering of the game layer by the first device, all controls included in the control area are drawn on the screen 802, ensuring that the positions of the controls in the rendered game layer are the same as the positions of the controls in the interface 800. This allows the first device to avoid changing the touch logic of the controls included in the control area during the compositing of the interface 800. Consequently, the touch stability of the controls is higher, and the game's responsiveness is improved.

[0129] In this way, when a user clicks on control A at location A, the touch logic corresponding to location A becomes the touch logic corresponding to control A. This allows the first device to achieve UI separation of the game layers without changing the touch logic corresponding to each control. This helps maintain the game's responsiveness and touch stability.

[0130] It should be understood that, in the embodiments of this application, the touch logic of a control can be understood as the logic of touching the control, that is, the logic of interaction with the first device generated when the user touches the control. For example, the touch logic of control S is to control the game character S to release skill S, etc. Through the touch logic, the user can interact with the first device by touching the control.

[0131] To better understand the electronic devices in the embodiments of this application, the hardware structure of the first device in the embodiments of this application will be described in detail below with reference to FIG10.

[0132] Figure 10 is a schematic diagram of the structure of the first device 1000 provided in an embodiment of this application. As shown in Figure 10, the first device 1000 may include a processor 1010, an external memory interface 1020, an internal memory 1021, a universal serial bus (USB) interface 1030, a charging management module 1040, a power management module 1041, a battery 1042, an antenna 1, an antenna 2, a mobile communication module 1050, a wireless communication module 1060, an audio module 1070, a sensor module 1080, a button 1090, an indicator 1092, a camera 1093, and a display screen 1094, etc.

[0133] The audio module 1070 may include, but is not limited to, a speaker, a receiver, a microphone, and a headphone jack.

[0134] The sensor module 1080 may include, but is not limited to, one or more of the following sensors: pressure sensor, gyroscope sensor, barometric pressure sensor, magnetic sensor, accelerometer, distance sensor, proximity sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor, and bone conduction sensor, etc.

[0135] It is understood that the structure illustrated in the embodiments of this application does not constitute a specific limitation on the first device 1000. In other embodiments of this application, the first device 1000 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0136] Processor 1010 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Optionally, processor 1010 may also include memory for storing instructions and data. The different processing units may be independent devices or integrated into one or more processors.

[0137] Optionally, if it is determined that the first device 1000 is in a game, the processor 1010 can synthesize a target image and transmit the target image to the display screen 1094 for display. The target image includes the game's field of view and control area, and the field of view and control area do not overlap.

[0138] The wireless communication function of the first device 1000 can be implemented through antenna 1, antenna 2, mobile communication module 1050, wireless communication module 1060, modem processor, and baseband processor.

[0139] The first device 1000 implements display functions through a GPU, a display screen 1094, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 1094 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 910 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0140] The display screen 1094 is used to display images, videos, etc. Exemplarily, the display screen 1094 can display the field of view and control area of ​​a game. In some embodiments, the first device 1000 may include one or N display screens 1094, where N is a positive integer greater than 1. When the first device 1000 is a foldable screen device and includes one display screen 1094, the one display screen 1094 may be an inner screen connected by a flexible component; when the first device 1000 is a foldable screen device and includes multiple display screens 1094, at least one inner screen connected by a flexible component is included among the multiple display screens 1094.

[0141] The external memory interface 1020 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the first device 1000. The internal memory 1021 can be used to store computer executable program code, which includes instructions.

[0142] The software system of the first device 1000 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. The following description, using a layered Android system as an example, illustrates the display process of the first device 1000 in conjunction with Figure 11.

[0143] To facilitate understanding, we will first explain the layered architecture of the Android system and the related terminology.

[0144] As shown in Figure 11, the layered architecture divides the software system of the first device 1000 into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into six layers, from top to bottom: the application layer, the application framework layer, the system libraries and Android Runtime, the hardware abstraction layer, the kernel layer, and the hardware layer.

[0145] 1. The Application Layer may include a series of application packages. For example, an application package may include a camera, gallery, video, SMS, system user interface (UI), layer painter, game manager, and application1, etc.

[0146] The game manager may include a super split-screen switch, which users can use to turn the super split-screen switch on or off. When the super split-screen switch is on, the field of view and control area displayed on the first device 1000 will not overlap during gameplay. Application 1 can be any game application.

[0147] It should be understood that the Super Split Screen Switch is for illustrative purposes only. In some possible implementations, the Super Split Screen Switch may also have other names, and this application does not specifically limit it.

[0148] Application 1 may include a main thread and a rendering thread. During the execution of application 1, the rendering thread of application 1 may issue rendering instruction 1. Rendering instruction 1 may include information indicating the resources required to composite the game layer. For example, if the image to be composited includes a tree and control 1, then rendering instruction 1 may include information indicating the tree and control 1. The game layer can be understood as the layer included in the interface corresponding to application 1 displayed on the first device 1000.

[0149] 2. The Application Framework Layer provides the application programming interface (API) and programming framework for applications in the application layer. The Application Framework Layer includes predefined functions. It may include window managers and game information modules, among other things.

[0150] The window manager is used to manage window applications. The window manager can obtain the screen size, determine if a status bar is present, obtain system status, and set the window size for each application. System status may include, for example, whether the first device 1000 is in split-screen or non-split-screen mode, whether the first device 1000 is in landscape or portrait mode, and whether the first device 1000 is in full-screen or non-full-screen mode. If the first device 1000 is a foldable device, the system status may also include whether the first device 1000 is in a folded or unfolded state.

[0151] Split-screen mode refers to a state where multiple applications or windows are simultaneously displayed on the inner screen of the first device. Through split-screen mode, users can view and operate multiple applications on the inner screen at the same time. For example, referring to Figure 8, application 1 is displayed on screen 810, and shopping application 2 is displayed on screen 820, etc. Full-screen mode can refer to the visual window of application 1 being displayed on the entire inner screen; or, application 1's visual window being displayed on the entire inner screen, but other application visual windows also exist on the inner screen, such as social application 3, which is displayed as a pop-up above application 1's visual window. These visual windows typically include a title bar, menu bar, toolbar, content area, and status bar, and can be understood as providing an interface for users to interact with the application.

[0152] The visualization window is displayed on the entire inner screen, but its size is not limited to be exactly the same as the inner screen size. Please refer to the description above; it will not be repeated here.

[0153] The game information module can obtain game scene information from application 1. Game scene information may include, but is not limited to, one or more of the following: game mode, frame rate, image quality, or settings parameters. The game module can indicate whether the game is in initialization mode, waiting mode, or match mode.

[0154] The initialization mode can be understood as the pre-game phase, used to set the game's initial state, load game resources, and initialize the game interface and various game elements. After initialization mode, application 1 performs a series of preparatory tasks to ensure the game can start and run normally. The waiting mode is the phase between initialization mode and match mode, where users can set game parameters, such as changing character outfits or selecting weapons. Match mode can be understood as the actual gameplay, where players are actively playing and interacting. In match mode, players may battle against other players, challenge levels, complete mission objectives, or perform various in-game actions.

[0155] It should be understood that game mode can also be replaced by game state, etc., and this application does not make specific limitations on this.

[0156] 3. The system library can include multiple functional modules. For example, a game management module, graphics processing acceleration technology (GPU Turbo), image compositing service (Surface Flinger, SF), and 3D graphics processing libraries (e.g., OpenGL ES).

[0157] The game management module can obtain system status from the window manager. For example, the game management module can register a listener with the window manager, and can determine if the system status has changed. The game management module can obtain game scene information from the game information module, and can determine whether the game is in initialization mode, waiting mode or game mode based on the game scene information. The game information module can also obtain the on / off status of the super split screen switch from the game manager. For example, if the user sets the super split screen switch to on (or off), the game manager can instruct the game management module to set the super split screen switch to on (or off).

[0158] Furthermore, when the game management module determines that the switch is on, and that the first device 1000 is in an expanded state, a full-screen state, or a non-split-screen state, and that the game is in match mode, the game management module transmits information 1 to GPU Turbo. Information 1 is used to inquire whether UI separation of the game layer is possible. UI separation means separating the view area and the control area in the interface corresponding to application 1.

[0159] When GPU Turbo instructs the game management module to perform UI separation of the game layer, the game management module can execute the following steps: the game management module instructs the window manager to adjust the size of the window corresponding to application 1; the game management module instructs GPU Turbo to perform UI separation of the game layer; and the game management module instructs the image compositing service to modify the parameters of the game layer. UI separation of the game layer can be understood as separating the resources needed to compose the game layer. For example, resources belonging to the view area are set in the first part of the screen; resources belonging to the control area are set in the second part of the screen, etc.

[0160] When GPU Turbo receives rendering instruction 1 from the rendering thread of application 1, GPU Turbo can set the FB 0 of the game layer corresponding to application 1 and modify rendering instruction 1 to rendering instruction 2. The rendering instruction includes information indicating that resources belonging to the view area of ​​the resources required for compositing the game layer should be set in the first part of the screen, and resources belonging to the control area of ​​the resources required for compositing the game layer should be set in the second part of the screen.

[0161] Based on the instructions from the game management module to modify the parameters of the game layer, the image compositing service modifies the transform parameters of the game layer to enlarge its y-axis. Furthermore, based on rendering instruction 2, the image compositing service can invoke the GPU via the GPU driver to render the game layer.

[0162] It should be understood that the transform parameter typically refers to parameters used for geometric transformations of an image. These transformations can include operations such as rotation, scaling, translation, and shearing. In this embodiment, the transform parameter of the modified game layer can be a parameter used for scaling the image, etc. This parameter for scaling the image can be, for example, a scaling ratio on the y-axis. The y-axis can refer to the y-axis in the first device screen coordinate system.

[0163] The 3D graphics processing library can convert rendering instructions 2 into OpenGL commands, and then call the GPU through the GPU driver to execute the OpenGL commands, so that the GPU can render the game layers and obtain the rendered game layers.

[0164] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0165] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0166] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0167] 4. The Hardware Abstraction Layer (HAL) primarily serves to connect the application framework layer and the kernel layer. The HAL can include a hardware compositor (hwcomposer, HWC), which provides hardware support for image compositing services.

[0168] The Hardware Abstraction Layer (HWC) includes HWC, which can be used to overlay rendered game layers.

[0169] 6. The kernel layer is the layer between hardware and software. In an exemplary embodiment, the kernel layer includes display drivers and GPU drivers, etc.

[0170] 7. The hardware layer can include various hardware components, such as displays and GPUs.

[0171] It should be noted that this application uses the Android system as an example for illustration, but its basic principles are also applicable to electronic devices using other operating systems. This application does not limit the operating system used by the electronic device.

[0172] It should be understood that the display process of the first device 1000 shown in Figure 11 includes process 1 and process 2. Among them, process 1 (the step corresponding to the arrow of the thin line) is the process in which the game management module determines and instructs the window manager, GPU Turbo and image compositing service to perform UI separation on the display image corresponding to application 1.

[0173] Process 1 includes: the game management module obtains the system status from the window manager, the on / off status from the game management module, and the game scene information from the game information module; when the game management module determines that the system status is that the first device 1000 is in full-screen mode and the first device 1000 is in non-split-screen mode, determines that the on / off status is on, and determines that the game mode indicated in the game scene information is match mode, the game management module transmits information 1 to GPU Turbo, information 1 is used to inquire whether UI separation of the game layer can be performed; when GPU Turbo indicates to the game management module that UI separation of the game layer can be performed, the game management module can perform the following steps: the game management module instructs the window manager to adjust the size of the window corresponding to application 1, instructs GPU Turbo to perform UI separation of the game layer, and instructs the image compositing service to modify the parameters of the game layer.

[0174] It is understandable that, in the case that the first device 1000 is a foldable screen device, the game management module also needs to determine whether the first device 1000 is in the unfolded state based on the system status.

[0175] Process 2 (the steps corresponding to the thick arrows) can be performed based on Process 1; that is, Process 2 is the process by which the first device 1000 synthesizes the UI-separated display images. While Application 1 is in a game, the first device 1000 can repeatedly execute Process 2 to synthesize multiple frames of display images during the game.

[0176] Taking the composite display image 1 of the first device 1000 as an example, process 2 will be explained.

[0177] Process 2 includes: Application 1 obtains the window corresponding to Application 1; the rendering thread included in Application 1 sends rendering instructions to GPU Turbo; in response to the instruction from the game management module to separate the UI of the game layer, GPU Turbo modifies rendering instruction 1 to obtain rendering instruction 2; GPU Turbo instructs rendering instruction 2 to the 3D image processing library, the 3D image processing library converts rendering instruction 2 into an OpenGL command, and calls the GPU to execute the OpenGL command through the GPU driver to obtain the rendered game layer; the 3D image processing library transmits the rendered game layer to the image compositing service, the image compositing service transmits the rendered game layer to the hardware compositor, the hardware compositor performs overlay processing on the rendered game layer to obtain the image data of display image 1; then, the display driver calls the display screen to display image 1.

[0178] It should be understood that Figure 11 is merely an example. In some possible implementations, the image compositing service may also call the GPU, or call the GPU and hardware compositor to perform overlay processing on the rendered game layers. This application does not specifically limit this.

[0179] The image processing method of this application embodiment will be described below, taking a foldable screen device with two folds as an example and referring to Figure 12, from the perspective of the interaction of the internal modules included in the first device.

[0180] Figure 12 is a flowchart illustrating the image processing method 1200 provided in an embodiment of this application. Method 1200 can be executed by a first device. The hardware structure of the electronic device can be as shown in Figure 10, and the software structure can be as shown in Figure 11. The functions of each software module shown in Figure 12 can be referred to Figure 11. As shown in Figure 12, method 1200 includes the following steps:

[0181] S1201, The Game Manager indicates that the Super Split Screen switch of the Game Management Module is in the ON state.

[0182] The on / off state of the Super Split Screen switch can be toggled based on user actions. For example, based on user input, the Super Split Screen switch can switch from an on state to a off state, or vice versa. Input actions could include, for example, a click or a swipe.

[0183] Optionally, when the super split-screen switch is toggled, the game manager indicates the new switch state to the game management module, or the game manager indicates to the game management module that the switch state has been toggled. In this way, when the user toggles the switch state, the game management module can determine the new switch state, enabling it to accurately determine the switch state in real time.

[0184] Furthermore, when the game management module receives information from the game manager indicating the switched-off state or that the switch state has been switched, the game management module can record the switched-off state. For example, if the game management module records the initial switch state as "on," and the game manager indicates to the game management module that the switched-off state is "off," then the game management module updates the recorded switch state to "off." Alternatively, if the game manager indicates to the game management module that the switch state has been switched, then the game management module updates the initial switch state to "off."

[0185] Optionally, when the game management module records the first flag, it indicates that the switch is in the "on" state, and / or, when the game management module records the second flag, it indicates that the switch is in the "off" state. For example, the first flag can be 1 and the second flag can be 0; or, the first flag can be true and the second flag can be false, etc.

[0186] It should be understood that the game management module recording the switch state after switching can also be understood as the game management module caching the switch state after switching, or the game management module marking the switch state after switching, etc. This application does not make specific limitations in this regard.

[0187] S1202, Application 1 instructs the game scene information to the game information module.

[0188] S1203, The game information module instructs the game management module on game scene information.

[0189] The game scene information includes information indicating the game mode. This allows the game management module to determine whether the game is in match mode based on this information.

[0190] Optionally, after application 1 starts, application 1 can indicate game scene information to the game information module in real time. Furthermore, after the game information module receives the game scene information from application 1, it can indicate the game scene information to the game management module in real time. This allows the game management module to determine in real time whether the game is in match mode.

[0191] S1204, The window manager indicates the system status to the game management module.

[0192] Optionally, the game management module can register a listener with the window manager. This allows the game management module to determine the new system state when the system state is switched.

[0193] It should be understood that the steps of the game management module to obtain the switch status, system status and game scene information can be executed in parallel or sequentially, and this application does not make specific limitations on this.

[0194] S1205: Based on the switch status, game scene information, and system status, if the first device meets condition 1, the game management module executes S1206 and the steps following S1206.

[0195] Condition 1 includes: the switch is on, the game mode is in match mode, the first device is in unfolded state, and the first device is not in split-screen state. Alternatively, in addition to the above conditions, Condition 1 also includes the first device being in full-screen mode.

[0196] In this context, "the first device is not in split-screen mode" and "the first device is in full-screen mode" can be understood as the visual window of application 1 being displayed on the entire inner screen of the first device.

[0197] In one possible implementation, when application 1 is launched and the first device is not in split-screen mode, the first device can be in full-screen mode. That is, when the foldable screen device is not in split-screen mode, some game applications will be launched and displayed in full-screen mode on the inner screen of the foldable screen device. Therefore, the system state may not include whether the first device is in full-screen mode or not, that is, condition 1 may not include the first device being in full-screen mode.

[0198] It should be noted that if the first device is not a foldable screen device, condition 1 may not include the first device being in an unfolded state.

[0199] It is understandable that if the first device determines that it does not meet one or more of the conditions in condition 1, the first device may repeatedly execute S1201 to S1204 until the first device meets condition 1.

[0200] S1206, The game management module sends information 1 to GPU Turbo. Information 1 is used to ask whether UI separation of game layers is possible.

[0201] In response to message 1, GPU Turbo can determine whether GPU Turbo is capable of separating the UI from the game layers.

[0202] GPU Turbo can determine whether UI separation of game layers is possible using the following methods.

[0203] It is understandable that during the rendering process of application 1, when the rendering thread calls the underlying modules and hardware to draw and render the game layer, it can call the `drawcall()` function to retrieve the resources needed to draw the game layer from resource library 1. Resource library 1 can store the resources needed to draw the game layer corresponding to application 1. For example, assuming the game image to be displayed by the first device includes trees, clouds, game character 1, and control X, the resources needed to draw the game layer can include trees, clouds, game character 1, and control X. Furthermore, each resource needed to draw the game layer can have a corresponding name.

[0204] In this way, GPU Turbo can determine whether each resource belongs to the viewport or the control area based on its name among the resources needed to render the game layer. For example, GPU Turbo can cache or invoke associations, which include the correspondence between resource names and resource attributes. Resource attributes can indicate whether the resource belongs to an object in the viewport or a control in the control area. Thus, through these associations and the resource names obtained by application 1, GPU Turbo can determine whether each resource belongs to the viewport or the control area.

[0205] Therefore, GPU Turbo can determine whether UI separation of game layers is possible using the following methods.

[0206] Method 1: Determine if the resource attributes corresponding to each resource in Resource Library 1 can be determined. That is, if GPU Turbo can determine the resource attributes corresponding to each resource in Resource Library 1 through Resource Library 1 and the association relationship, GPU Turbo can determine that UI separation of the game layer can be performed; otherwise, GPU Turbo can determine that UI separation of the game layer cannot be performed.

[0207] In this way, when resources in resource library 1 are called during the rendering of game layers on the first device, GPU Turbo can determine the resource attributes of any resource.

[0208] Method 2: Determine whether the version of application 1 has been updated, or you can also understand it as determining whether the version of application 1 is the default version 1.

[0209] It should be understood that preset version 1 can also be interpreted as the preset software version number. As the version of application 1 is updated, the resources included in resource library 1 corresponding to application 1 may be updated. Therefore, GPU Turbo may not be able to determine the resource attributes corresponding to some resources in resource library 1, resulting in GPU Turbo being unable to perform UI separation of game layers.

[0210] Method 2 simplifies how GPU Turbo determines whether UI separation of game layers is possible.

[0211] Method 3: Determine whether Resource Library 1 includes one or more preset resources. These preset resources can be resources whose attributes GPU Turbo can determine. For example, one or more preset resources can be resources whose resource attributes GPU Turbo can determine are controls, such as maps in a game.

[0212] In this way, during the rendering of the game layer, GPU Turbo can determine the resource attributes of one or more preset resources. Alternatively, if one or more preset resources are one or more controls needed to render the game layer, GPU Turbo can also determine that the resource attributes of the one or more preset resources are controls, while the resource attributes of the remaining resources are content such as game characters in the field of view.

[0213] S1207 In response to information 1, GPU Turbo instructs the game management module that UI separation of the game layer can be performed. Correspondingly, the game management module continues to execute S1208 and the following steps.

[0214] Optionally, if GPU Turbo determines that UI separation of the game layer cannot be performed, in response to information 1, GPU Turbo indicates to the game management module that UI separation of the game layer cannot be performed. Correspondingly, the game management module does not execute S1207 and the following steps.

[0215] It should be understood that steps S1201 to S1207 can be the process by which the game management module determines whether UI separation of the game layer can be performed. If the game management module determines that UI separation of the game layer can be performed, the first device executes steps S1208 to S1212. Steps S1208 to S1212 can be understood as the process by which the game management module instructs multiple modules to perform UI separation of the game layer.

[0216] In some possible implementations, if the game management module determines that the first device does not meet condition 1, or if GPU Turbo determines that UI separation of the game layer cannot be performed, and GPU Turbo indicates to the game management module that UI separation of the game layer cannot be performed, the first device can synthesize multi-frame images of application 1 according to S1230 to S1235 shown in FIG13.

[0217] S1208, The game management module instructs the window manager to adjust the size of the game window. The game window can be understood as the window corresponding to application 1.

[0218] Optionally, the game management module can indicate the package name of application 1 to the window manager, and indicate that UI separation of game layers is possible. In this way, the window manager can adjust the size of the game window based on the package name of application 1 and the indication that UI separation of game layers is possible.

[0219] It should be understood that the package name of application 1 is merely an example, and the package name of application 1 can also be replaced with other information that can be used to indicate application 1, without any specific limitation in this application.

[0220] S1209. The window manager adjusts the game window to the second part of the inner screen, such as window 901 shown in Figure 9(a). The size of the game window is the same as that of the second part of the screen, and the position of the game window is the same as that of the second part of the screen.

[0221] Optionally, the window manager can adjust the size of the game window by modifying the bounds property of application 1. The bounds property can also be called the bounds parameter. For example, the window manager resets the bounds parameter, which includes the coordinates of the top-left and bottom-right corners of the game window. In this way, the window manager can use the top-left and bottom-right corner coordinates of the game window to position the game window within the second portion of the inner screen of the first device.

[0222] It should be understood that the `bounds` parameter can also include: the coordinates of the top-left corner of the game window, the height of the game window, and the width of the game window, etc. In this way, the window manager can also set the game window in the second part of the inner screen of the first device through the `bounds` parameter. The `bounds` parameter can be understood as a parameter that can be used to describe the size and position of the game window. This application does not specifically limit the parameters included in the `bounds` parameter.

[0223] After S1209, the window manager can instruct application 1 to re-acquire the game window. This game window is the one obtained after the window manager resizes the game window. That is, the game window is located on the second portion of the screen.

[0224] Reacquiring the game window can also be understood as application 1 rereading or determining the size and position of the game window.

[0225] It's understandable that after the window manager modifies the bounds parameter of application 1, it can execute the `makeconfigationchange` operation. The `makeconfigationchange` operation can be understood as the window manager instructing the application to make corresponding adjustments and reconfigurations when some configurations of the electronic device change. It can also be understood as forcing application 1 to obtain the window size, etc. In this way, application 1 can obtain the modified bounds parameter and change the size of the drawn game layer based on this modified bounds parameter.

[0226] Therefore, by having application 1 reacquire the game window, the size of the canvas for drawing the game layer generated by the first device is made the same as the size of the second portion of the screen, as shown, for example, in Figure 9(a).

[0227] Furthermore, by aligning the game window with the control area, the first device can set the touch logic for each space based on its position within the game window. This eliminates the need to modify the touch logic of the controls during the subsequent rendering of the game layer.

[0228] S1210, the game management module instructs GPU Turbo to perform UI separation of game layers.

[0229] In response to instructions from the game management module, GPU Turbo can intercept and modify rendering commands from Application 1 during the rendering of the game layer. This allows other modules to set resources in the view area to the upper half of the second screen and resources in the control area to the lower half of the second screen when drawing and rendering the game layer. For example, as shown in Figure 9(c), the view area is set in the upper half 903 and the control area is set in the lower half 904.

[0230] Optionally, the game management module indicates the layer name of the game layer to GPU Turbo and indicates that UI separation of the game layer is possible.

[0231] In this way, during the subsequent synthesis of multiple frames of images corresponding to application 1 by the first device, if GPU Turbo recognizes that the name of the layer is the name of the game layer, it can intercept and modify the rendering instructions sent by the rendering thread of application 1.

[0232] It should be understood that during the operation of application 1, the layer names of the game layers included in each frame of the multi-frame images corresponding to application 1 drawn by the first device can be the same. For example, during the user's gameplay, at time 1, the first device displays interface 1; at time 2, the first device displays interface 2. The layer names of the game layers corresponding to interface 1 and interface 2 are the same.

[0233] It should also be understood that the layer name of the game layer mentioned above can be replaced with other information that can be used to indicate the game layer, and this application does not specifically limit this.

[0234] S1211, the game management module instructs SF to set the parameters of the game layer. The game layer parameters can be those that adjust the vertical size of the game layer. For example, the game layer's transform parameter. In this way, SF can double the vertical size of the game layer so that the game layer can cover the entire inner screen, as shown in Figure 9(d). Doubling the vertical size of the game layer can also be understood as doubling the y-axis size of the game layer.

[0235] The statement that the game layer can cover the entire inner screen does not mean that the size of the game layer is exactly the same as the size of the inner screen, but rather that the game layer can be displayed on the inner screen in full-screen mode. The size of the game layer can also be slightly smaller than the size of the inner screen; this application does not impose specific limitations on this.

[0236] Optionally, the game management module can indicate the layer name of the game layer to SF and indicate that UI separation of the game layer is possible. In this way, SF can set the transform parameter of the game layer corresponding to application 1. The transform parameter can be, for example, matrix 1. By modifying matrix 1, the vertical size of the game layer can be doubled. Furthermore, before SF modifies the game layer parameters again, during subsequent execution of application 1, the game layer corresponding to each frame in the multi-frame images of application 1 will have its vertical size doubled according to matrix 1.

[0237] S1212 and SF set the parameters of the game layer to parameter 1.

[0238] It can be understood that S1201 to S1212 can correspond to process 1 in Figure 11, that is, the steps corresponding to the thin arrows.

[0239] S1213 to S1220 can be understood as the steps that the first device needs to perform to draw each frame of the image of application 1. Moreover, S1213 to S1220 can be understood as being performed based on S1201 to S1212.

[0240] S1213, The rendering thread in application 1 sends rendering instruction 1 to GPU Turbo.

[0241] Rendering instruction 1 can include the name of the game layer and the names of the resources needed to draw the game layer. In this way, GPU Turbo can respond to the instruction in S1210 and, upon recognizing the name of the game layer, intercept game instruction 1.

[0242] S1214. If GPU Turbo recognizes the name of the game layer, the frame buffer used for display corresponding to the game layer is enlarged by one time, that is, FB 0 is enlarged by one time.

[0243] In this way, the canvas used to draw game layers can be enlarged by one time, as shown in Figure 9(b), for example.

[0244] The resolution of the game layer rendered when FB 0 is magnified by one time is twice the resolution of the game layer rendered when FB 0 is not magnified.

[0245] S1215, GPU Turbo modifies rendering instruction 1 to obtain rendering instruction 2.

[0246] Rendering instruction 1 can include the name of the game layer and the names of the resources needed to draw the game layer. GPU Turbo can then modify rendering instruction 1 by adding instruction 1: setting the resources belonging to the view area of ​​the game layer's canvas in region 1, and setting the resources belonging to the control area of ​​the game layer's canvas in region 2, resulting in rendering instruction 2. Region 1 can be understood as the area corresponding to the first part of the screen; region 2 can be understood as the area corresponding to the second part of the screen.

[0247] It should be understood that GPU Turbo can determine the resource attributes of each resource needed to render a game layer based on the names and relationships of the resources required for rendering the game layer. Specific implementation details can be found in the description above and will not be repeated here.

[0248] It is understandable that, in response to the instruction from the game management module in S1209, GPU Turbo will intercept and modify the rendering instruction when it recognizes that the rendering instruction includes the name of the game layer corresponding to application 1. Therefore, since the game layer corresponding to application 1 is unique, GPU Turbo can intercept and modify rendering instruction 1 when drawing each frame of game image corresponding to application 1.

[0249] S1216, GPU Turbo instructs the 3D graphics processing library to render command 2.

[0250] The GPU Turbo instruction to the 3D graphics processing library for rendering instruction 2 can also be understood as the 3D graphics processing library obtaining the modified rendering instruction 2 from GPU Turbo.

[0251] S1217: The 3D graphics processing library renders the game layer based on rendering instruction 2, resulting in a rendered game layer. In this rendered game layer, the view area is located in the first part of the screen, and the control area is located in the second part of the screen. That is, during the drawing and rendering of the game layer, the controls in the control area are all executed on the second part of the screen, making the positions of the controls in this game layer the same as the positions of the controls in the game image displayed on the first device. Thus, when the user performs input operations on the controls in the game image displayed on the first device, the first device can execute the touch logic corresponding to the controls in the game image. In other words, the positions of the controls in the final game image displayed on the first device are the same as the positions of the controls in this game layer, so that the first device does not need to modify the touch logic of the controls.

[0252] However, since the game window is located in the second half of the screen, if the game layer is displayed, the corresponding display interface for the game layer will be: the field of view is located in the upper half of the second half of the screen, and the control area is located in the lower half of the second half of the screen. For example, as shown in Figure 9(c).

[0253] In this case, the game layer display still has an error. Although the game layer is drawn and rendered on a canvas the size of the inner screen, because the game window is located on the second part of the screen, if the first device wants to display the game layer, the game layer displayed on the first device will be on the second part of the screen. Therefore, the first device can solve this problem through S1218 and S1219.

[0254] Optionally, the 3D graphics processing library can convert rendering instructions 2 into OpenGL commands, and the GPU driver can call the GPU to execute the OpenGL commands to obtain the rendered game layers.

[0255] S1218, the 3D graphics processing library instructs SF on the rendered game layers.

[0256] S1219 and SF synthesize game images based on parameter 1, such as matrix 1 mentioned above.

[0257] It should be understood that SF can process each game layer separately using Matrix 1 before calling the GPU and / or hardware compositor to perform overlay processing on the game layers, thereby doubling the vertical size of each game layer. Then, SF can call the GPU and / or hardware compositor to perform overlay processing on the enlarged game layers. Alternatively, SF can process the overlaid layers using Matrix 1 after calling the GPU and / or hardware compositor to perform overlay processing, thereby doubling the vertical size of the overlaid layers. This application does not specifically limit this.

[0258] Optionally, SF can invoke the GPU and / or hardware synthesizer to overlay game layers.

[0259] S1220, Display Sending. This means the first device can display game images on a screen.

[0260] It should be understood that the game image synthesized in S1219 can undergo various pre-display processing steps, after which the display screen shows the processed game image. This application embodiment does not specifically limit the pre-display processing.

[0261] It should be noted that S1213 to S1220 are the processes by which the first device synthesizes the game image when the game management module determines that the UI separation of the game layer can be performed.

[0262] Since application 1 can indicate the new on / off state to the game management module after the super split-screen switch is switched; the window manager can also indicate the new system state to the game management module after the system state is switched; and the game management module can also determine from the game scene information from the game information module that the game is not in match mode after the game ends. Therefore, after the first device synthesizes one or more frames of image of application 1 according to the process shown in S1213 to S1220, if the game management module determines that the first device no longer meets condition 1, the first device can execute S1221 to S1233 as shown in FIG13.

[0263] It should be noted that the first device may first execute one or more of steps S1221 to S1223.

[0264] S1221, The window manager indicates the updated system status to the game management module. The updated system status may include one or more of the following: the first device is in a collapsed state, the first device is in a split-screen state, or the first device is not in full-screen mode, etc.

[0265] That is, S1221 can be executed when the system state of the first device changes during the user's game.

[0266] S1222, Application 1 sends updated game scene information to the game management module via the game information module. The updated game scene information may include information indicating that the game is not in match mode.

[0267] S1222 can be executed after the user's game ends.

[0268] It is understandable that executing S1222 after a user's game ends does not mean that before the user's game ends, Application 1 does not indicate the updated game scene information to the game management module through the game information module. Rather, it means that before the user's game ends, Application 1 indicates the updated game scene information to the game management module through the game information module, including information indicating that the game is in game mode.

[0269] S1223, The game manager indicates to the game management module that the super split-screen switch is in the off state.

[0270] That is, after the user switches the Super Split Screen switch to the off state, S1223 can be executed.

[0271] It should be understood that the implementation methods of S1221 to S1223 are similar to those of S1201 to S1204, and can be referred to the description above, which will not be repeated here.

[0272] In one or more of the cases shown in S1221 to S1223, the first device may execute S1225 to S1233.

[0273] S1224, The game management module instructs the window manager to adjust the window size.

[0274] S1225, Window Manager adjusts the size and position of the game window.

[0275] Optionally, in the updated system state, including when the first device is in a folded state, the window manager can place the game window on the outer screen of the foldable device, or it can place the game window on either the first or second portion of the foldable device's screen.

[0276] In the updated system state, including when the first device is in split-screen mode, the window manager will not adjust the game window, or will place the game window on the first portion of the foldable screen.

[0277] When S1222 and / or S1223 are executed, the window manager can set the game window on the inner screen of the foldable device.

[0278] In this way, the window manager can set the game window on the screen where the interface of application 1 needs to be displayed, depending on the circumstances.

[0279] After S1225, the window manager can instruct application 1 to obtain the game window.

[0280] It should be understood that the implementation of S1224 and S1225 is similar to that of S1208 and S1209, and can be referred to the description above, which will not be repeated here.

[0281] S1226, the game management module instructs GPU Turbo not to perform UI separation of the game layer. For example, the game management module instructs GPU Turbo to specify the layer name of the game layer and to not perform UI separation. Thus, in the subsequent compositing of multi-frame images from application 1, in response to the instruction in S1226, even if GPU Turbo recognizes the layer name of the game layer, it will not modify the FB 0 corresponding to the game layer, nor will it modify the rendering instructions from application 1.

[0282] It should be understood that the implementation of S1226 is similar to that of S1210, and can be referred to the description above, which will not be repeated here.

[0283] S1227, The game management module instructs SF to set the parameters of the game layer.

[0284] S1228, SF sets the parameters of the game layer to parameter 2.

[0285] It is understandable that in S1212, SF can update parameter 2 to parameter 1; and in S1228, SF can update parameter 1 to parameter 2. That is, if the first device does not meet condition 1, the parameters of the game layer do not need to be scaled up to the size of the game layer.

[0286] It should be understood that the implementation of S1227 and S1228 is similar to that of S1211 and S1212, and can be referred to the description above, which will not be repeated here.

[0287] Through steps S1221 to S1228, the view area and control area in the multi-frame images of application 1 synthesized by the first device are not separated. That is, after S1228, the first device can synthesize the multi-frame images of application 1 in accordance with steps S1229 to S1233.

[0288] S1229, The rendering thread in application 1 instructs the 3D graphics processing library to render command 3.

[0289] S1230, the 3D graphics processing library renders the game layer based on rendering instruction 3, and obtains the rendered game layer.

[0290] S1231, the 3D graphics processing library instructs SF on the rendered game layers.

[0291] S1232 and SF synthesize a game image based on parameter 2. This game image can also be understood as the image data of a game image.

[0292] S1233 and SF will instruct the display screen to show the game image.

[0293] It is understood that the game image synthesized in S1232 can also undergo various pre-display processing before the display screen shows the processed game image. This application embodiment does not specifically limit the various pre-display processing methods.

[0294] It should be understood that, unlike S1213 to S1220, GPU Turbo does not intercept or modify rendering instructions from application 1, nor does it modify the FB 0 of the game layer.

[0295] It should be noted that Method 1200 is merely an example. In some possible implementations, the first device can also be a tri-fold or other foldable screen device, or it can be a non-foldable screen device. In this case, the difference from Method 1200 is that the parameters of the game layer set by SF are different, and the magnification of FB 0 corresponding to the game layer is different. The rest is similar to Method 1200, and can be found in the description above, which will not be listed here.

[0296] For example, the first device is a three-fold foldable screen device, and the inner screen of the foldable screen device includes a first screen, a second screen, and a third screen, which are foldable. Assuming the second part of the screen is the third screen, for example, the part to the right of the folding axis 220 in device 200, parameter 1 can also be a parameter that doubles the vertical size of the game layer; in S1214, GPU Turbo can double the FB 0 corresponding to the game layer.

[0297] Alternatively, the first device is a tablet computer, whose screen comprises a first screen portion and a second screen portion. The first and second screen portions are of equal width, and the height of the first screen portion is X times that of the second screen portion. Then parameter 1 can also be a parameter that enlarges the vertical size of the game layer by a factor of X; in S1214, GPU Turbo can enlarge the FB 0 corresponding to the game layer by a factor of X. Here, enlarging by a factor of X can also be understood as becoming X+1 times the initial state.

[0298] It is understandable that, when X is greater than 1, if the position of the control drawn during the process of the tablet drawing the game image differs from the position of the control in the game image displayed on the tablet, the tablet can also modify the touch logic of the control. For example, suppose that control 1 drawn during the process of the tablet drawing the game image is at position 1, then position 1 is used for the touch logic of control 1. However, if control 1 is displayed on the tablet at position 2, then when the user clicks position 2, the tablet can replace position 2 with position 1 for the click operation and execute the touch logic of control 1 corresponding to position 1.

[0299] The following section, with reference to Figure 14, explains how users can set the on / off state of the super split-screen switch.

[0300] Figure 14 is a schematic diagram of the interface 1400 of a foldable screen device provided in an embodiment of this application. As shown in Figure 14, during the user's gaming process, in response to the user's click on the Game Manager icon 1402 in the side toolbar 1401, the first device can enter the Game Manager settings interface displayed in a pop-up window. In response to the user's input on the Super Split Screen switch control 1403, the on / off state of the Super Split Screen switch control 1403 can be switched. For example, the initial on / off state of the Super Split Screen switch 1403 is the off state, and the Super Split Screen switch displayed in the interface 1400 is in an unselected state; in response to the user's click operation, the on / off state of the Super Split Screen switch control 1403 can be switched to the on state, and the Super Split Screen switch control 1403 displayed in the interface 1400 is in a selected state.

[0301] In some possible implementations, the first device may also respond to the user clicking the Game Manager icon on the desktop to enter the Game Manager's settings interface. In this case, the Game Manager's settings interface may be in full-screen mode, etc. The first device responds to the user's input on the Super Split-Screen switch control in the settings interface, toggling the on / off state of the Super Split-Screen switch control.

[0302] It should be noted that Figure 14 is merely an example. The sidebar 1401 may include more or fewer application or tool icons, and the application icons included in the sidebar 1401 may be replaced with icons of other applications. For example, the gallery icon may be changed to a phone icon, etc. Furthermore, the game manager settings interface displayed in a pop-up window may include more or fewer toggle controls, and the toggle controls in this settings interface may be changed to other toggle controls. For example, the network acceleration toggle control may be changed to a mute toggle control, etc. This application does not impose specific limitations in this regard.

[0303] It should also be noted that the settings interface of the game manager and / or the multiple switch controls included in the settings interface of the game manager displayed in the form of a pop-up window can also be replaced with other forms, such as replacing them with the form of a sliding switch, etc. This application does not make specific limitations on this.

[0304] This application also provides an image processing method 1500. Method 1500 can be executed by an electronic device, the hardware structure of which is shown in FIG10, and the software structure of which is shown in FIG11. Method 1500 includes the following steps:

[0305] S1501. Display the first interface of the first application. The first interface includes a first application screen display area and a first control area. The first application screen display area includes one or more controllable objects. The first control area includes one or more controls. The controls in the first control area are used to control the controllable objects in the first application screen display area. The first application screen display area and the first control area do not overlap.

[0306] The first application can be a game application, such as application 1 in method 1200. The first interface can be understood as a frame of image from application 1. The first application screen display area can be understood as the field of view. One or more controllable objects in the first application screen area can refer to game characters, movable objects in the game, or switchable scenery. The first application screen area may also include the environment in which one or more controllable objects are located, such as grass and mountains in the game screen. The first control area can be understood as the control area in method 1200. The electronic device can be understood as the first device in method 1200. When the electronic device is a foldable screen device that folds in two directions, the first interface can be, for example, as shown in interface 800. When the electronic device is a non-foldable screen device such as a tablet computer, the first interface can be, for example, interface 1500 shown in FIG15. Interface 1500 includes a first application screen display area 1510 and a first control area 1520.

[0307] S1502, In response to a first operation on a first control in a first control area, control a first controllable object in a first application screen display area.

[0308] The first control can refer to a control within one or more controls in the first control area. There can be one or more first controls. The first operation can be, but is not limited to, one or more of the following: single click, swipe, long press, or double click. The first controllable object can be understood as an object within one or more controllable objects in the first application screen area. There can be one or more first controllable objects.

[0309] It should be noted that when there are multiple first controls, the first operation can be multiple operations, and these multiple operations can be the same or different. In response to the first operations of different controls within the first control, the actions performed by the first controllable object can be different. For example, if the first control includes control 1 and control 2, the first operation is a click, and the first controllable object is object 1, in response to a click on control 1, object 1 jumps; in response to a click on control 2, object 1 releases skill 1. Alternatively, in response to different first operations of the same operation within the first control, the actions performed by the first controllable object can be different. For example, if the first control includes control 3, the first operations include a single click and a long press, and the first controllable object is object 3, in response to a single click on control 3, object 3 releases skill 2; in response to a long press on control 3, object 3 releases skill 3. This application does not impose specific limitations on this.

[0310] The image processing method of this application ensures that the application screen display area and the control area of ​​the first application interface displayed on the electronic device do not overlap. This reduces the obstruction of the application screen display area by the user's hand when the user manipulates the controllable object in the application screen display area using the controls in the control area, thus minimizing the restriction of the user's field of vision and improving the user experience.

[0311] When the electronic device is a foldable screen device, the interface displayed on the electronic device can be as follows.

[0312] As an optional embodiment, the electronic device is a foldable electronic device, which includes a foldable display screen. The first display portion and the second display portion of the foldable display screen are foldable. S1501 can be implemented in the following way: when the foldable display screen is in the unfolded state, a first interface is displayed, with a first application screen display area in the first display portion and a first control area in the second display portion.

[0313] In this way, the first application screen display area and the first control area in the first interface can be located on different display parts of the foldable display screen.

[0314] The foldable electronic device can be, for example, a bi-fold or tri-fold electronic device. The foldable display screen can be understood as the inner screen in method 1200. The first display portion can be understood, for example, as the first part of the screen in method 1200, and the second display portion can be understood, for example, as the second part of the screen in method 1200. The first display portion and the second display portion can be connected by a flexible component.

[0315] It should be understood that the unfolded state of the foldable display screen can also be replaced by the unfolded state of the foldable electronic device, and this application does not specifically limit this.

[0316] Optionally, based on the above embodiments, method 1500 further includes: displaying a second interface of the first application when the foldable display screen is in a folded state. The second interface includes a second application screen display area and a second control area. The second application screen display area includes one or more controllable objects, and the second control area includes one or more controls. The controls in the second control area are used to control the controllable objects in the second application screen display area, and the second control area is located on the second application screen display area.

[0317] The second application screen display area can also be understood as the field of view, and the second control area can also be understood as the control area. The second control area being located on top of the second application screen display area can also be understood as the second control area overlapping the second application screen display area, with the second control area situated on top of the second application screen display area. That is, controls in the second control area may obscure part of the content in the second application screen display area. For example, in the case of a foldable electronic device that is bi-foldable, the second interface can be as shown in Figure 7.

[0318] It is understandable that, in one scenario, when the foldable electronic device switches from a folded state to an unfolded state, it switches from displaying the second interface to displaying the first interface; in another scenario, when the foldable electronic device switches from an unfolded state to a folded state, it switches from displaying the first interface to displaying the second interface.

[0319] Thus, the interface displayed on the foldable display screen differs when the foldable electronic device switches from an unfolded state to a folded state, or vice versa.

[0320] Optionally, S1501 can be implemented in the following way: the first interface of the first application is displayed when the electronic device meets one or more of the following conditions: the first condition is that the state of the first switch is on; the second condition is that the first application is in full-screen display state; the third condition is that the first application is in game mode and the first application is a game application.

[0321] Here, the first switch can be understood as the super split-screen switch in method 1200. The first condition, the second condition, and the third condition can be understood as condition 1 in method 1200.

[0322] Optionally, for the first condition, method 1500 further includes: displaying a third interface, the third interface including a first switch; and in response to a second operation on the first switch, setting the state of the first switch to an on state.

[0323] The third interface can be, for example, as shown in interface 1400. Alternatively, the third interface can also be the system settings interface of the foldable electronic device, including the interface for the first switch, etc. The second operation can be, but is not limited to, clicking or swiping.

[0324] In this way, users can choose whether to separate the application screen display area and the control area in the interface of the electronic device according to their needs.

[0325] Based on the above embodiments, the first interface displayed by the electronic device can be synthesized in the following way.

[0326] Optionally, before displaying the first interface of the first application, method 1500 further includes: drawing a first layer, the first layer including a first area and a second area, the first area including one or more controllable objects included in the display area of ​​the first application screen, the second area including one or more controls included in the first control area, the one or more controls included in the first control area being drawn in a first window corresponding to the first application, the size of the first window being the same as that of the first control area, and the resolution of the first layer being the same as the resolution of the image displayed on the first interface; enlarging the size of the first layer to obtain the image displayed on the first interface.

[0327] In this context, the first layer can be understood as the game layer in method 1200, for example, the rendered game layer in S1217; or, it can also refer to the game layer shown in Figure 9(c). The first area can be understood, for example, as the first part of the screen in method 1200, and the second area can be understood, for example, as the second part of the screen in method 1200. The first window can be understood as the game window in method 1200. The second area is the same size as the first window.

[0328] Optionally, the position of the first window is the same as the position of the first control area.

[0329] In this way, one or more controls in the first control area can be drawn in the first window, making the positions of the one or more controls in the first control area the same as the positions of the one or more controls in the first control area of ​​the first interface. During the process of compositing the image displayed on the first interface, there is no need to modify the touch logic of the one or more controls in the first control area. This helps to improve the responsiveness of the first application.

[0330] Optionally, based on the above embodiments, method 1500 further includes: setting a range parameter of the second window of the first application to obtain a first window, wherein the range parameter is used to describe the position and size of the second window.

[0331] The range parameter can be understood as the bounds parameter in method 1200. This allows the size of the first window to be the same as the second area, facilitating the drawing of one or more controls in the first control area within the first window.

[0332] Based on the above embodiments, optionally, the first layer can be drawn in the following way: the canvas to be displayed is enlarged to obtain a first canvas, the size of the first canvas being equal to the sum of the first application screen display area and the first control area; the first layer is drawn on the first canvas.

[0333] Enlarging the displayed canvas can also be understood as doubling the size of FB0 corresponding to the game layer in S1214. The magnification factor of the displayed canvas can be the ratio of the size of the first interface to the size of the first window. For example, referring to Figure 9, the size of interface 800 is twice that of game window 901. In this case, FB0 corresponding to the game layer is enlarged by one time.

[0334] This enlarges the display canvas or framebuffer corresponding to the first layer, making its size the same as the first interface. The first device can draw the first layer on a canvas the same size as the first interface, and the positions of the controls in the drawn first control area are the same as the positions of the controls in the first interface. This preserves the touch logic of the controls in the drawn first interface during subsequent rendering and overlay processing, contributing to higher touch stability.

[0335] Based on the above embodiments, optionally, drawing the first layer includes: obtaining a first rendering instruction from the rendering thread of the first application, the first rendering instruction including information for instructing the resources used to draw one or more controllable objects included in the display area of ​​the first application screen and the resources used to draw one or more controls included in the first control area; adding a first instruction to the first rendering instruction to obtain a second rendering instruction, the first instruction being used to instruct the resources used to draw one or more controllable objects included in the display area of ​​the first application screen to be drawn in the first area, and instructing the resources used to draw one or more controls included in the first control area to be drawn in the second area; and drawing the first layer in the first window in response to the second rendering instruction.

[0336] The first rendering instruction can be, for example, rendering instruction 1 in method 1200; the second rendering instruction can be, for example, rendering instruction 2 in method 1200. The first indication can be, for example, indication 1 in method 1200.

[0337] It should be understood that the method of obtaining the second rendering instruction is similar to the implementation method of S1215, and the method of drawing the first layer based on the second rendering instruction is similar to the implementation method of S1217. Please refer to the description above, and it will not be repeated here.

[0338] It is understandable that, in addition to the first, second, and third conditions, the first device can also determine whether UI separation of the game layer is possible through the following methods.

[0339] Optionally, before drawing the first layer, method 1500 further includes: determining whether one or more controllable objects included in the first application screen display area can be drawn in the first area, and whether one or more controls included in the first control area can be drawn in the second area; if so, drawing the first layer.

[0340] Specifically, determining whether one or more controllable objects included in the first application screen display area can be drawn in the first area, and whether one or more controls included in the first control area can be drawn in the second area, can also be understood as determining whether UI separation of the game layer is possible. The first device can also make this determination using one or more of the methods one, two, or three mentioned above.

[0341] It should be understood that the implementation of this embodiment is similar to the implementation of S1206 to S1207, and can be referred to the above description, which will not be repeated here.

[0342] For example, determining whether one or more controllable objects included in the first application screen display area can be drawn in the first area, and whether one or more controls included in the first control area can be drawn in the second area, includes: determining whether the resource library corresponding to the first application includes preset resources; or determining whether the version of the first application is a preset version.

[0343] Here, the resource library corresponding to the first application is understood as a library used to store the resources required to draw multiple frames of images for the first application. The method for determining whether the resource library corresponding to the first application includes preset resources is similar to the implementation method three described above. Preset resources can be, for example, one or more preset resources. For example, preset resources can be maps, etc., that the first device can recognize and identify as controls.

[0344] The preset version can also be understood as a preset software version number, such as preset version 1 in method two. The method for determining whether the version of the first application is the preset version is similar to the implementation method of method two, and can be referred to the description above, which will not be repeated here.

[0345] In addition to the methods mentioned above, the image displayed on the first interface can also be synthesized using method 1600.

[0346] Optionally, method 1600 includes: setting a third window of the first application, the position of the third window being the same as the position of the display area of ​​the first application screen, and the position of the third window being the same as the size of the display area of ​​the first application screen; creating a fourth window, the position of the fourth window being the same as the first control area, and the size of the fourth window being the same as the first control area; drawing one or more controllable objects included in the display area of ​​the first application screen in the third window; drawing one or more controls included in the first control area in the fourth window; and obtaining the image displayed on the first interface.

[0347] For example, as shown in Figure 16, assuming the electronic device is a foldable screen phone with two folds, the third window can be set on screen 1601, and the fourth window can be set on screen 1602. Furthermore, the controllable objects and other content in the first application screen display area are located in the third window, i.e., screen 1601; the controls and other content in the first control area are located in the fourth window, i.e., screen 1602.

[0348] It's understandable that in method 1600, creating the fourth window means creating a new process. Furthermore, since the one or more controllable objects in the first application screen display area and the one or more controls in the first control area are drawn in different processes, relevant information about the game layer, such as the buffer corresponding to the game layer, needs to be transferred across processes to the process corresponding to the fourth window before drawing the controls in the first control area. Additionally, controlling one or more controllable objects through one or more controls requires cross-process implementation; that is, the touch logic of the controls in the fourth window needs to be transferred across processes to the process corresponding to the third window to control the controllable objects in the third window.

[0349] As can be seen, compared to method 1600, the methods of synthesizing the image displayed on the first interface in method 1500 and synthesizing the game image in method 1200 do not require the creation of new windows and new processes, resulting in lower power consumption for the first device and electronic device. Furthermore, relevant information of the game layer, such as the buffer corresponding to the game layer, does not need to be transmitted across processes; the touch logic of the control also does not need to be transmitted across processes, resulting in higher touch stability and better responsiveness of the control.

[0350] It should be understood that in the embodiments of this application, "size" can also be understood as "dimension". "Magnified by one time" can be understood as "becomes twice the original size". This application does not specifically limit this.

[0351] It should also be understood that the order of the methods listed above does not imply the order of execution. The execution order of each method should be determined by its function and internal logic.

[0352] The image processing method of the present application embodiment has been described in detail above with reference to Figures 9 to 16. The image processing apparatus of the present application embodiment will be described in detail below with reference to Figure 17.

[0353] Figure 17 is a schematic block diagram of an image processing apparatus 1700 provided in an embodiment of this application. The apparatus 1700 includes a processor 1701, a communication interface 1702, and a memory 1703. The processor 1701, communication interface 1702, and memory 1703 communicate with each other via internal connection paths. The memory 1703 is used to store instructions, and the processor 1701 is used to execute the instructions stored in the memory 1703. The communication interface 1702 can be used to send signals to other devices (e.g., the processor 9101 or a touchscreen of an electronic device) and to receive signals from other devices (e.g., the memory 1703). Exemplarily, the communication interface 1702 reads instructions stored in the memory 1703 and sends the instructions to the processor 9101.

[0354] It should be understood that the apparatus 1700 may specifically be the first device or electronic device in the above embodiments, and may be used to perform the various steps and / or processes corresponding to the first device or electronic device in the above method embodiments. Optionally, the memory 1703 may include read-only memory and random access memory, and provide instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information. The processor 9101 may be used to execute instructions stored in the memory, and when the processor 9101 executes instructions stored in the memory, the processor 9101 is used to perform the various steps and / or processes of the above method embodiments.

[0355] It should be understood that, in the embodiments of this application, the processor may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0356] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The steps of the method disclosed in the embodiments of this application can be directly manifested as execution by a hardware processor, or as a combination of hardware and software modules within the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor executes the instructions in the memory, combining them with its hardware to complete the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0357] This application also provides a computer-readable storage medium for storing a computer program for implementing the methods shown in the above-described method embodiments.

[0358] This application also provides a computer program product, which includes a computer program (also referred to as code or instructions) that, when run on a computer, allows the computer to perform the methods shown in the above-described method embodiments.

[0359] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0360] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0361] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0362] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0363] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0364] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0365] The above description is merely a specific embodiment of this application, but the protection scope of the embodiments of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be included within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes: Displaying a first interface of a first application, the first interface includes a first application screen display area and a first control area, the first application screen display area includes one or more controllable objects, the first control area includes one or more controls, the controls in the first control area are used to control the controllable objects in the first application screen display area, the first application screen display area and the first control area do not overlap. In response to a first operation on a first control in the first control area, control a first controllable object in the first application screen display area.

2. The method according to claim 1, characterized in that, The invention relates to a foldable electronic device, the foldable electronic device including a foldable display screen, wherein a first display portion of the foldable display screen and a second display portion of the foldable display screen are foldable. The first interface for displaying the first application includes: When the foldable display screen is in the unfolded state, the first interface is displayed, the first application screen display area is located in the first display portion, and the first control area is located in the second display portion.

3. The method according to claim 2, characterized in that, The method further includes: When the foldable display screen is in a folded state, a second interface of the first application is displayed. The second interface includes a second application screen display area and a second control area. The second application screen display area includes one or more controllable objects, and the second control area includes one or more controls. The controls in the second control area are used to control the controllable objects in the second application screen display area. The second control area is located on the second application screen display area.

4. The method according to any one of claims 1 to 3, characterized in that, The first interface for displaying the first application includes: When the first switch is in the ON state, the first interface is displayed.

5. The method according to claim 4, characterized in that, Before displaying the first interface, the method further includes: A third interface is displayed, the third interface including the first switch; In response to a second operation on the first switch, the state of the first switch is set to the on state.

6. The method according to any one of claims 1 to 5, characterized in that, The first interface for displaying the first application includes: When the first application is in full-screen mode, the first interface is displayed.

7. The method according to any one of claims 1 to 6, characterized in that, The first application is a game application; The first interface for displaying the first application includes: When the first application is in game mode, the first interface is displayed.

8. The method according to any one of claims 1 to 7, characterized in that, Before displaying the first interface of the first application, the method further includes: Draw a first layer, which includes a first area and a second area. The first area includes one or more controllable objects included in the display area of ​​the first application screen, and the second area includes one or more controls included in the first control area. The one or more controls included in the first control area are drawn in a first window corresponding to the first application. The size of the first window is the same as that of the first control area, and the resolution of the first layer is the same as that of the image displayed on the first interface. Enlarge the size of the first layer to obtain the image displayed on the first interface.

9. The method according to claim 8, characterized in that, Before drawing the first layer, the method further includes: The first window is obtained by setting the range parameter of the second window of the first application. The range parameter is used to describe the position and size of the second window.

10. The method according to claim 8 or 9, characterized in that, The drawing of the first layer includes: The canvas to be displayed is enlarged to obtain the first canvas, the size of which is equal to the sum of the first application screen display area and the first control area; Draw the first layer on the first canvas.

11. The method according to any one of claims 8 to 10, characterized in that, Before drawing the first layer, the method further includes: Determine whether one or more controllable objects included in the first application screen display area can be drawn in the first area, and whether one or more controls included in the first control area can be drawn in the second area; If so, draw the first layer.

12. The method according to claim 11, characterized in that, The determination of whether one or more controllable objects included in the first application screen display area can be drawn in the first area, and whether one or more controls included in the first control area can be drawn in the second area, includes: Determine whether the resource library corresponding to the first application includes preset resources; or, Determine whether the version of the first application is the preset version.

13. The method according to any one of claims 8 to 12, characterized in that, The drawing of the first layer includes: Obtain a first rendering instruction from the rendering thread of the first application. The first rendering instruction includes information for instructing the resources used to draw one or more controllable objects included in the display area of ​​the first application screen and the resources used to draw one or more controls included in the first controllable area. A first instruction is added to the first rendering instruction to obtain a second rendering instruction. The first instruction is used to instruct the resources used by one or more controllable objects included in the first application screen display area to be drawn in the first area, and to instruct the resources used by one or more controls included in the first control area to be drawn in the second area. In response to the second rendering instruction, the first layer is drawn in the first window.

14. The method according to any one of claims 8 to 13, characterized in that, The position of the first window is the same as the position of the first control area.

15. An electronic device, characterized in that, The electronic device includes: one or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 14.

16. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 14.

18. A computer program product, characterized in that, The computer program product includes computer program code that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 14.

Images