An image display method and related apparatus

Abstract

The application provides an image display method and related device, in an inner screen use state, a first user interface of a third-party application is displayed through a first display screen, wherein the first user interface is used for displaying a first video call picture, and the first video call picture comprises an image collected through a first camera; in an outer screen use state, a second user interface of the third-party application is displayed through a second display screen, wherein the second user interface is used for displaying a second video call picture, and the second video call picture comprises an image collected through a second camera, the first display screen and the second display screen are located at different surfaces of an electronic device, the first camera is located on the first display screen, and the second camera is located on the second display screen. Therefore, when the electronic device switches from the inner screen use state to the outer screen use state, the video call is not interrupted, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of electronic technology, and in particular to an image display method and related apparatus. Background Technology

[0002] With technological advancements, electronic devices can now be equipped with at least two displays, including a main screen (which can be called the inner screen) and a secondary screen (which can be called the outer screen). Generally, the main screen and the secondary screen are located on different sides of the electronic device. The main screen supports display and is equipped with a camera (which can be called a front-facing camera), while the secondary screen supports display and is equipped with a camera (which can be called a rear-facing camera).

[0003] The advent of foldable screens has diversified the operating states of electronic devices. For example, the physical states of an electronic device can include unfolded, bent, and folded states. When an electronic device is in its unfolded or bent state, a third-party application can be used to activate the front-facing camera for video calls. However, when the electronic device switches from its unfolded or bent state to its folded state, the video call will be interrupted, and the user will need to reconnect to the video call in the folded state, resulting in a poor user experience. Summary of the Invention

[0004] The image display method and related states provided in this application embodiment can improve the user experience when using third-party applications.

[0005] In a first aspect, this application provides an image display method applied to an electronic device, comprising:

[0006] In the internal screen usage state, the first user interface of a third-party application is displayed through the first display screen, wherein the first user interface is used to display the first video call screen, and the first video call screen includes an image captured by the first camera;

[0007] When the external screen is in use, the second user interface of the third-party application is displayed on the second display screen. The second user interface is used to display the second video call screen, which includes the image captured by the second camera. The first display screen and the second display screen are located on different sides of the electronic device. The first camera is located on the first display screen, and the second camera is located on the second display screen.

[0008] As can be seen, when an electronic device switches from using the inner screen to using the outer screen, the video call is not interrupted, and the user can continue the video call, thus improving the user experience.

[0009] In one possible implementation of the first aspect, the method further includes:

[0010] In the state of using the inner screen, the first user interface is also used to display the first image captured by the second camera;

[0011] In the external screen usage state, the second user interface is also used to display a third image, wherein the third image is an image obtained by rotating the second image, and the second image includes the image captured by the second camera in the external screen usage state.

[0012] In the above method, when the electronic device switches from the inner screen usage state to the outer screen usage state, the image captured by the second camera in the outer screen usage state can be rotated to adapt to the outer screen usage state and improve the user experience.

[0013] In one possible implementation of the first aspect, the method further includes:

[0014] The imaging direction of the first image differs from that of the second image by 180 degrees, and the imaging direction of the third image is the same as that of the first image.

[0015] As can be seen, when the electronic device switches from inner screen to outer screen mode, the third image is rotated. Therefore, the image of the third image is not inverted, improving the user experience.

[0016] In one possible implementation of the first aspect, displaying the second user interface of the third-party application via a second display screen while in external screen usage mode includes:

[0017] The physical state of the electronic device is detected to determine whether the electronic device switches from the inner screen usage state to the outer screen usage state;

[0018] Determine the processing strategy corresponding to the third-party application;

[0019] The second user interface is determined according to the processing strategy;

[0020] The second user interface is displayed on the second screen.

[0021] It can be seen that the electronic device can detect the physical state, thereby determining the corresponding processing strategy, and then process the second user interface accordingly, so that the second user interface can be adapted to the second display screen.

[0022] In one possible implementation of the first aspect, determining the processing strategy corresponding to the third-party application includes:

[0023] The processing strategy corresponding to the application scenario is determined based on the application scenario of the third-party application, wherein the application scenario includes one or more of the following: drawing the second user interface through a first view control, drawing the second user interface through a second view control, displaying a Joint Image Experts Group JPEG image in the second user interface, and video calling.

[0024] In one possible implementation of the first aspect, the method further includes:

[0025] In the scenario where the second user interface is drawn using a first view control, the processing strategy includes: setting deformation parameters to rotate the image.

[0026] In one possible implementation of the first aspect, the method further includes:

[0027] In the scenario where the second user interface is drawn using a second view control, the processing strategy includes: rotating the image at the output node.

[0028] In one possible implementation of the first aspect, the method further includes:

[0029] In a scenario where a JPEG image is displayed in the second user interface, the processing strategy includes: rotating the image through video encoding and decoding.

[0030] In one possible implementation of the first aspect, the method further includes:

[0031] In the video call scenario, the processing strategy includes: generating a first instruction, the first instruction including turning off the first camera and turning on the second camera.

[0032] In the above method, the electronic device can select the corresponding strategy according to the actual application scenario of the third-party application, so that the electronic device can switch from the inner screen usage state to the outer screen usage state. The selected strategy can process the images captured by the camera more effectively, ensuring the user experience.

[0033] Secondly, an electronic device is provided in the embodiments of this application, the electronic device comprising: one or more processors; a memory; wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the electronic device to perform the image display method described in the first aspect or any possible implementation of the first aspect.

[0034] Thirdly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to execute the image display method described in the first aspect or any possible implementation thereof. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0035] In one possible implementation, the chip or chip system described above in the embodiments of this application further includes at least one memory, which stores instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (such as a read-only memory or random access memory).

[0036] Fourthly, embodiments of this application provide a computer storage medium storing a computer program that, when executed by a processor, causes the computer to perform an image display method as described in the first aspect or any possible implementation thereof.

[0037] Fifthly, embodiments of this application provide a computer program product that, when run on a communication device, causes the communication device to perform an image display method as described in the first aspect or any possible implementation thereof.

[0038] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0039] The accompanying drawings used in the embodiments of this application are described below.

[0040] Figure 1 This is a schematic diagram of the product form of an electronic device 100 with a longitudinal folding method provided in an embodiment of this application;

[0041] Figure 2 This is a schematic diagram of the product form of an electronic device with a vertical folding method provided in an embodiment of this application;

[0042] Figure 3 This is a schematic diagram illustrating a scenario where a third-party application calls the camera to take pictures, as provided in an embodiment of this application.

[0043] Figure 4 This is a schematic diagram illustrating a scenario where a third-party application uses the camera to make a video call, as provided in an embodiment of this application.

[0044] Figure 5 An exemplary schematic diagram of the hardware structure of an electronic device 100 is shown;

[0045] Figure 6 This is a schematic diagram of the software architecture of the electronic device provided in the embodiments of this application;

[0046] Figures 7A-7D This application provides a set of user interfaces for third-party applications to access the camera;

[0047] Figure 8 This is a schematic diagram of image rotation provided in an embodiment of this application. Detailed Implementation

[0048] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0049] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0050] The term "user interface (UI)" used in the specification, claims, and drawings of this application refers to the medium through which an application or operating system interacts and exchanges information with the user. It facilitates the conversion between the internal form of information and a form acceptable to the user. The user interface of an application is source code written in specific computer languages ​​such as Java or Extensible Markup Language (XML). This source code is parsed and rendered on an electronic device, ultimately presenting user-recognizable content such as images, text, and buttons. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, images, and text. The attributes and content of controls in the interface are defined using tags or nodes, such as XML tags. <textview> 、 <imgview> 、 <videoview>Nodes define the controls contained in the interface. A node corresponds to a control or property in the interface, and after parsing and rendering, the node is presented as the content visible to the user. In addition, many applications, such as hybrid applications, often contain web pages within their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. Web pages are source code written in a specific computer language, such as Hypertext Markup Language (HTML), Cascading Style Sheets (CSS), JavaScript (JS), etc. Web page source code can be loaded and displayed as user-readable content by a browser or a web page display component with browser-like functionality. The specific content contained in a web page is also defined through tags or nodes in the web page source code; for example, HTML uses tags or nodes to define the content. 、 、 <video> 、 <canvas>Used to define the elements and attributes of a webpage.

[0051] The most common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an interface element such as an icon, window, or control displayed on the screen of an electronic device. The control can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0052] The foldable electronic device 100 includes a foldable screen that can be folded to form at least two screens. The folding method of the foldable screen on the electronic device 100 can be divided into two categories: one is a vertical folding method (for example...) Figure 1 The folding method shown is vertical folding (e.g., folding up and down) along the bent part; another type is horizontal folding (e.g., folding horizontally). Figure 2 The folding method shown is to fold horizontally along the bent part (e.g., fold left and right).

[0053] Please see Figure 1 , Figure 1 This is a schematic diagram of an electronic device 100 with a vertically folding configuration provided in an embodiment of this application. For example... Figure 1 As shown, the electronic device 100 may include a first display screen 200, a second display screen 300, a first camera 401, and a second camera 402. The first display screen 200 and the first camera 401 are located on the front of the electronic device 100; the first display screen 200 can be referred to as the inner screen, and the first camera 401 as the front-facing camera. The second display screen 300 and the second camera 402 are located on the back of the electronic device 100; the second display screen 300 can be referred to as the outer screen, and the second camera 402 as the rear-facing camera. The first display screen 200 is larger than the second display screen 300; the first display screen 200 can be referred to as the main screen, and the second display screen 300 as the secondary screen. Figure 1 As shown, the first display screen 200 is a foldable display screen, including region 201, region 202, and region 203. Region 202 is bendable and located on the bent portion of the electronic device 100. The two ends of the bent portion of the electronic device 100 are connected to regions 201 and 203, respectively. The included angle Q between the two ends of the bent portion of the electronic device 100 (which is also the bending angle Q of the first display screen 200) can also be understood as the angle between the plane containing region 201 and the plane containing region 203. The second display screen 300 includes at least one region, namely region 301.

[0054] like Figure 1 As shown in (a), when the first display screen 200 is in the unfolded state, regions 201 and 203 are essentially on the same plane. The light-emitting surface of the second display screen 300 is opposite to the light-emitting surface of the first display screen 200, specifically, the light-emitting surface of region 301 is opposite to the light-emitting surface of region 201. In one embodiment, the second display screen 300 is a non-foldable display screen, and the second display screen 300 is located at one end of the bent portion (i.e., the end where region 201 is connected).

[0055] In one embodiment, the first camera 401 is located above region 201, and the second camera 402 is located above region 301. The second camera 402 may include at least one camera, which can be understood as a camera module. When the electronic device 100 is in the unfolded state, the first camera 401 is located on the front of the electronic device 100 and can function as a front-facing camera, performing the functions of a front-facing camera and displaying the captured images on the first display screen 200; the second camera 402 is located on the back of the electronic device 100 and can function as a rear-facing camera, performing the functions of a rear-facing camera and displaying the captured images on the first display screen 200.

[0056] In one embodiment, the electronic device 100 may be bent along the bent portion in a forward-facing manner to obtain... Figure 1 The electronic device 100 in the bent state shown in (b) of the figure.

[0057] like Figure 1 As shown in (b), the first display screen 200 is bent, the plane containing region 201 intersects the plane containing region 203, the light-emitting surface of region 301 is opposite to the light-emitting surface of region 201, and the plane containing region 301 intersects the plane containing region 203. When the electronic device 100 is in a bent state, the first camera 401 acts as a front-facing camera and performs the functions of a front-facing camera, and the second camera 402 acts as a rear-facing camera and performs the functions of a rear-facing camera.

[0058] In one implementation, Figure 1 The electronic device 100 in the unfolded state shown in (a) can be bent along the bent portion in a manner that bends towards the front, in order to obtain Figure 1 The electronic device 100 in its folded state is shown in (c). In another embodiment, Figure 1 (b) The electronic device 100 in the bent state shown can continue to be folded along the bent portion, forming Figure 1 The electronic device 100 in the folded state shown in (c) is shown in the figure.

[0059] like Figure 1 As shown in (c), the first display screen 200 is in a folded state, and the light-emitting surface of region 201 is opposite to the light-emitting surface of region 203. Therefore, regions 201 and 203 are not visible to the user because... Figure 1 (a) and Figure 1 As shown in (b), the first camera 401 is located on region 201, so it is also invisible to the user at this time. The light-emitting surfaces of the second camera 402 and region 301 are opposite to those of region 201. The plane containing region 301 is parallel to the plane containing region 203. The light-emitting directions of the second camera 402 and region 301 are the same as those of region 203. At this time, region 203 is invisible to the user, while the second camera 402 and region 301 are visible to the user. Therefore, in the folded state, the first camera 401, which performs the function of the front-facing camera, is invisible to the user. Thus, the second camera 402, which is visible to the user, can be used to perform the function of the front-facing camera and display the captured image on the second display screen 300.

[0060] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating the product form of an electronic device with a vertically folding configuration, as provided in an embodiment of this application. For example... Figure 2 As shown, the electronic device 100 may include a first display screen 200, a second display screen 300, a first camera 401, and a second camera 402.

[0061] like Figure 2 As shown in (a), the second display screen 300 of the electronic device 100 may include three regions: region 301, region 302, and region 303. Region 302 is bendable. The two ends of the bendable portion of the electronic device 100 are connected to regions 301 and 303, respectively. When the electronic device is in the unfolded state, Figure 2 The light-emitting surfaces of the first display screen 200 and the second display screen 300 shown in (a) are opposite to each other, that is, the light-emitting surfaces of region 201 and region 301 are opposite to each other.

[0062] In some embodiments, it can be Figure 2 The electronic device 100 in the unfolded state shown in (a) is partially folded to obtain the electronic device 100 in the bent state, for example... Figure 2 The electronic device 100 shown in (b)

[0063] In some embodiments, it can be Figure 2 The electronic device 100 in the unfolded state shown in (a) can be folded to obtain the electronic device 100 in the folded state. Figure 2 The electronic device 100 in the bent state shown in (b) is folded (to obtain the electronic device 100 in the folded state, for example...). Figure 2 The electronic device 100 shown in (c).

[0064] It should be noted that the relevant descriptions of "first display screen 200", "second display screen 300", "first camera 401", "second camera 402", "area 201", "area 203" and "area 301" can be found in the following references. Figure 1 This will not be elaborated upon further.

[0065] In this embodiment, the included angle Q of the electronic device 100 ranges from [0°, 180°]. Specifically, if Q ∈ [0°, P1], the electronic device 100 is in a folded state; if Q ∈ (P1, P2), the electronic device 100 is in a bent state; and if Q ∈ [P2, 180°], the electronic device 100 is in an unfolded state. Wherein, 0° < P1 < P2 < 180°. P1 and P2 can be preset angle thresholds. P1 and P2 can be determined based on the usage habits of a large number of users of foldable screens; alternatively, P1 and P2 can be set by the user in the electronic device 100.

[0066] In some embodiments, according to the usage habits of most users, the preset angle threshold P1 in this application embodiment can be in the range of (0, 30°), and the preset angle threshold P2 can be in the range of (150°, 180°). For example, the preset angle threshold P1 can be 5°, 10°, 15°, 20°, etc. The preset angle threshold P2 can be 155°, 160°, 165°, or 170°, etc.

[0067] Understandably, when the electronic device 100 is in a folded state, a bent state, or an unfolded state, the bending angle of the bent part (for example, the bending angle Q of the first display screen 200) is different, but the specific value of the bending angle Q is not limited.

[0068] The following explanation uses the example of a flexible folding screen (first display screen 200) and a rigid screen (second display screen 300). The first display screen 200 displays a graphical user interface (GUI) through areas 201, 202, and 203 (hereinafter referred to as the user interface). For the electronic device 100 with a vertical folding mechanism, the second display screen 300 displays the user interface through area 301, while areas 302 and 303 are not display areas. For the electronic device 100 with a horizontal folding mechanism, the second display screen 300 displays the user interface through area 303, while areas 301 and 302 are not display areas.

[0069] With the increasing sophistication of electronic devices, electronic devices equipped with cameras can not only start shooting with the camera app, but also start shooting and video calls through third-party applications.

[0070] Please see Figure 3 , Figure 3 This is a schematic diagram of a scenario where a third-party application calls the camera to take pictures, as provided in an embodiment of this application.

[0071] Taking an electronic device 100 with a vertical folding mechanism in its unfolded state as an example, such as Figure 3 As shown, in the unfolded state, i.e., when the first camera 401 (front-facing camera) is facing the user, the electronic device 100 responds to the user's operation by launching a third-party application. Within the third-party application, the electronic device 100, in response to the user's operation, launches the second camera 402 to take a picture and displays the first user interface 31 on the first display screen 200. Then, the electronic device 100 folds along the bending portion, switching from the unfolded state to the folded state. In the folded state, i.e., when the second camera 402 (rear-facing camera) is facing the user, the electronic device 100 displays the user interface of the third-party application in area 301 of the second display screen 300. Because the third-party application is still using the second camera 402 to take pictures, the second user interface 33 captured by the second camera 402 is displayed in area 301.

[0072] from Figure 3 As can be seen, the second user interface 33 displayed on the second display screen 300 is inverted (or flipped) relative to the first user interface 31 displayed on the first display screen 200. This is because the object being photographed displayed in the first user interface 31 is oriented in the same direction as the actual object being photographed. Therefore, the object being photographed displayed in the second user interface 33 is inverted compared to the actual object, thus affecting the user experience. The reason for this inverted display is that the third-party application containing the camera function is developed by a non-manufacturer and cannot perceive the physical state of the electronic device 100 (e.g., unfolded, bent, or folded).

[0073] When the electronic device 100 is in an unfolded or bent state, it uses the first display screen 200 to display the user interface. Since the first display screen 200 can be referred to as the inner screen, the electronic device 100 can be considered to be in inner screen usage mode when it is in an unfolded or bent state. When the electronic device 100 is in a folded state, it uses the second display screen 300 to display the user interface. Since the second display screen 300 can be referred to as the outer screen, the electronic device 100 can be considered to be in outer screen usage mode when it is in a folded state.

[0074] Only when the inner screen is in use, and the second camera 402 is marked as a standard camera, can the underlying layer report the sensor parameters of the second camera 402 (such as the installation angle) to the system layer, and further report them to the third-party application. Therefore, only when the inner screen is in use can the third-party application obtain the sensor parameters of the second camera 402, and then select the corresponding rotation logic to process the image captured by the second camera 402 according to the sensor parameters, thereby ensuring that the processed image can be adapted to the current display state (i.e., displayed on the first display screen 200 in the unfolded state). For example, when the electronic device 100 is not rotated in the inner screen usage state (for example, when the electronic device 100 is used in portrait mode), the installation angle of the second camera 402 reported by the underlying layer is 0. The installation angle of 0 indicates that the image data is consistent with the natural orientation of the device, and no additional rotation is required for the image output by the second camera 402. When the electronic device 100 is rotated 90 degrees clockwise (for example, when the electronic device is used in landscape mode), the installation angle of the second camera 402 reported by the underlying layer is 90 degrees. The installation angle of 0 indicates that the image output by the second camera 402 needs to be rotated 90 degrees clockwise to be consistent with the natural orientation of the electronic device.

[0075] After the electronic device 100 switches from the inner screen usage state to the outer screen usage state, the sensor parameters of the second camera 402 change. Since the underlying layer can only report the camera's sensor parameters to the third-party application when the electronic device 100 is in the inner screen usage state, it cannot report the modified camera's sensor parameters to the third-party application when in the outer screen usage state. Therefore, the sensor parameters obtained by the third-party application are those of the inner screen usage state (i.e., the sensor parameters corresponding to the standard camera), causing the processed image to be unable to adapt to the current display state (i.e., the installation angle of the second camera 402 has been rotated by 180 degrees and it is displayed on the second display screen 300 in the folded state).

[0076] like Figure 3 As shown, after the electronic device 100 switches from inner screen usage mode to outer screen usage mode by bending inward, the third-party application still obtains the sensor parameters of the second camera 402 from the inner screen usage mode (for example, the installation angle is 0 degrees). Therefore, the third-party application does not perform any additional processing on the image output by the second camera 402. Since the installation angle of the second camera 402 has rotated by 180 degrees, the image output by the second camera 402 also rotates by 180 degrees. Because no additional processing is performed on the image output by the second camera 402 based on the sensor parameters from the inner screen usage mode, when the third-party application calls the second camera 402 to take a picture in the outer screen usage mode, the preview and final image show an inverted image compared to the inner screen usage mode.

[0077] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating a scenario where a third-party application uses the camera to make a video call, as provided in an embodiment of this application. Figure 4 (a) is a schematic diagram of a video call scenario using an electronic device 100 with a vertically folding design. Figure 4 (b) is a schematic diagram of a video call scenario using an electronic device 100 with a horizontal folding mechanism.

[0078] from Figure 4 It can be seen that, for video call scenarios, regardless of whether the electronic device 100 has a vertical or horizontal folding design, the video call is interrupted when switching from the inner screen to the outer screen. For example, when the electronic device 100 is in inner screen mode, a third-party application uses the first camera 401 for a video call. When the electronic device 100 switches from inner screen to outer screen mode, the first camera 401 is not visible to the user in outer screen mode. Therefore, the camera used for the video call needs to be switched from the first camera 401 to the second camera 402. Because the first camera 401 and the second camera 402 are installed in different locations and are two different physical cameras, the third-party application cannot switch the camera used for the video call to the second camera 402 in outer screen mode. Generally, the common practice is for the third-party application to directly close the video call, but this will interrupt the user's video call. To resume the video call, the user needs to perform a new operation, which greatly reduces the user experience when using the electronic device 100.

[0079] Based on this, this application provides an image display method applied to an electronic device. The electronic device is equipped with a foldable display screen (which can be called a foldable screen). The electronic device can be called a foldable electronic device. Folding the foldable screen can also be called folding the electronic device. The physical state / attitude of the foldable screen can also be called the object state of the electronic device.

[0080] The electronic device includes a first display screen and a second display screen. A first camera is mounted on the first display screen, and a second camera is mounted on the second display screen. The first and second display screens are located on different sides of the electronic device; for example, the first display screen is the main screen, and the second display screen is the secondary screen. Similarly, the first and second cameras are located on different sides of the electronic device and are used to perform different shooting tasks; for example, the first camera is the front-facing camera, and the second camera is the rear-facing camera. In the inner screen usage state, the first display screen shows the first user interface of a third-party application, which displays the first video call frame, including images captured by the first camera. In the outer screen usage state, the second display screen shows the second user interface of the third-party application, which displays the second video call frame, including images captured by the second camera. Since the first and second display screens are located on different sides of the electronic device, with the first and second cameras mounted on the first and second display screens respectively, the video call remains uninterrupted when the electronic device switches from inner screen to outer screen usage state, improving the user experience.

[0081] In one possible implementation, when the inner screen is in use, the first user interface is also used to display the first image captured by the second camera;

[0082] When the external screen is in use, the second user interface is also used to display a third image, which is obtained by rotating the second image. The second image includes the image captured by the second camera when the external screen is in use. Therefore, when the electronic device switches from the internal screen to the external screen, the displayed image will not be inverted, improving the user experience.

[0083] First, the electronic devices involved in the embodiments of this application will be introduced.

[0084] The electronic devices in this application embodiment may include, but are not limited to: smart screen devices, smart TVs (television, TV), mobile phones, tablet computers, ultra-mobile personal computers (UMPC), netbooks, as well as cellular phones, personal digital assistants (PDAs), wearable devices (such as smartwatches, smart bracelets), and other devices with display functions. This application embodiment does not impose any special restrictions on the specific form of the electronic device.

[0085] Figure 5 An exemplary schematic diagram of the hardware structure of an electronic device 100 is shown.

[0086] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, an angle sensor 180M, etc.

[0087] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0088] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0089] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0090] The processor 110 may also include a memory for storing instructions and data. In one embodiment, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can directly retrieve it from the memory.

[0091] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0092] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In one embodiment, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0093] In one embodiment, the electronic device 100 may include a main screen and a secondary screen. Optionally, the size of the main screen is larger than the size of the secondary screen; the main screen may also be referred to as a large screen, and the secondary screen may also be referred to as a small screen. Optionally, the main screen and the secondary screen are located on different sides of the electronic device 100; for example, the main screen is located on the front of the electronic device 100, and the main screen may also be referred to as an inner screen, while the secondary screen is located on the back of the electronic device 100, and the secondary screen may also be referred to as an outer screen.

[0094] In one embodiment, the main screen of the electronic device 100 is a foldable screen, while the secondary screen is not foldable. The physical states of the electronic device 100 can include an unfolded state, a bent state, and a folded state. When the electronic device 100 is in the unfolded state, the light-emitting surfaces of the main screen and the secondary screen face away from each other. At this time, the main screen can be in the unfolded state. When the electronic device 100 is in the bent or folded state, the main screen can be bent and divided into two display areas. The planes containing these two display areas intersect. At this time, the light-emitting surface of the secondary screen and one of the display areas faces away from each other. Examples of the main screen and the secondary screen can be found above. Figure 1 and Figure 2 The first display screen 200 and the second display screen 300 are shown.

[0095] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0096] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise, brightness, etc. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In one embodiment, the ISP can be set in the camera 193.

[0097] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In one embodiment, electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0098] In one embodiment, the electronic device 100 includes an inner screen and an outer screen, which can be located on different surfaces of the electronic device 100. N cameras 193 of the electronic device 100 can be respectively disposed on different surfaces of the electronic device 100. Optionally, some cameras 193 (i.e., the upper...) Figures 1 to 4 The first camera 401 and the inner screen are located on the same side, and part of the camera 193 (i.e., the upper one) Figures 1 to 4 The second camera 402 (inner screen) and the outer screen are located on the same side. The electronic device 100 can obtain the physical state of the electronic device 100 through the detection signals of the first camera 401 and / or the second camera 402. In some examples, when the electronic device 100 is in an unfolded or bent state, the first camera 401, which is on the same side as the inner screen, can obtain the user's facial information; this state can be referred to as the inner screen usage state. The second camera 402, which is on the same side as the outer screen, cannot obtain the user's facial information; this state can be referred to as the outer screen usage state. When the electronic device is in inner screen mode, for example, if a third-party application in the electronic device 100 initiates a video call, the first camera 401 can be used to conduct the video call, and the video call screen will be displayed on the inner screen. When the electronic device switches from inner screen mode to outer screen mode, the first camera 401, which is on the same side as the inner screen, cannot obtain the user's facial information, while the second camera 402, which is on the same side as the outer screen, can obtain the user's facial information. The electronic device 100 can determine that the current orientation is that the secondary screen and the user's face are facing each other. For example, if the third-party application in the electronic device 100 is still conducting a video call, the device switches from the first camera 401 to the second camera 402, uses the second camera 402 to conduct the video call, and displays the video call screen on the outer screen, ensuring uninterrupted video call.

[0099] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0100] Internal memory 121 can be used to store computer executable program code, which includes instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory located in the processor.

[0101] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In one embodiment, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the intensity of the touch operation based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In one embodiment, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0102] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In another embodiment, touch sensor 180K can also be located on the surface of electronic device 100, in a different position than display screen 194.

[0103] In this application, pressure sensor 180A and / or touch sensor 180K can be disposed in display screen 194. When display screen 194 displays the user interface of the application, pressure sensor 180A and / or touch sensor 180K can detect user operations on the user interface. In response to the user operation, electronic device 100 can perform corresponding tasks based on the application. For example, if a user clicks on the avatar of a friend in a social application, electronic device 100 can display the personal information posted by that friend through the social application on display screen 194.

[0104] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In one embodiment, the gyroscope sensor 180B can determine the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes). In one embodiment, the electronic device 100 can detect whether the bending angle of its folding screen has changed based on the detection signal from the gyroscope sensor 180B. In one embodiment, the electronic device 100 can detect whether it has flipped based on the detection signal from the gyroscope sensor 180B. The gyroscope sensor 180B can also be used for image stabilization, navigation, motion-sensing game scenes, etc. Optionally, the gyroscope sensor 180B can be mounted on the circuit board of the electronic device 100.

[0105] The accelerometer 180E can detect the magnitude of acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of the electronic device. In one embodiment, it can be used to detect whether the bending angle of the folding screen of the electronic device 100 changes; in another embodiment, it can detect whether the electronic device 100 is flipped. It is not limited to these applications and can also be used in applications such as landscape / portrait screen switching and pedometers. Optionally, the accelerometer 180E can be mounted on the circuit board of the electronic device 100.

[0106] In one embodiment, the electronic device 100 may include multiple accelerometers 180E and / or multiple gyroscopes 180B. The electronic device 100 includes a foldable screen (e.g., an inner screen). When the electronic device 100 is in an unfolded state, the foldable screen is in an unfolded state. When the electronic device 100 is in a bent state (also referred to as a half-folded state) and a folded state, the foldable screen is bent and divided into two display areas whose planes intersect. The multiple accelerometers 180E and / or multiple gyroscopes 180B may be respectively disposed on the circuit boards on the sides of the two display areas. The electronic device 100 can obtain changes in its physical state, such as whether the bending angle of the electronic device 100 has changed, based on the detection signals of the multiple accelerometers 180E and / or multiple gyroscopes 180B. In some examples, when the electronic device 100 determines that the physical state is an unfolded state or a bent state based on the detection signals, the electronic device 100 can determine that in the current posture, the inner screen is opposite to the user, and the outer screen is not opposite to the user; this state can be referred to as the inner screen usage state. When electronic device 100 determines that the physical state is folded based on the detection signal, it can determine that in the current posture, the inner screen is not facing the user, while the outer screen is facing the user. This state can be called the outer screen usage state. In the inner screen usage state, for example, if a third-party application in electronic device 100 initiates a video call, it can use the first camera 401 to conduct the video call and display the video call image on the inner screen. When electronic device 100 switches from the inner screen usage state to the outer screen usage state, the first camera 401, which is on the same side as the inner screen, cannot obtain the user's facial information, while the second camera 402, which is on the same side as the outer screen, can obtain the user's facial information. Electronic device 100 can determine that the current posture is that the secondary screen and the user's face are facing each other. For example, if the third-party application in electronic device 100 is still conducting a video call, it can switch from the first camera 401 to the second camera 402, using the second camera 402 to conduct the video call and displaying the video call image on the outer screen, ensuring uninterrupted video call.

[0107] A barometric pressure sensor 180C is used to measure air pressure. A magnetic sensor 180D includes a Hall effect sensor, which the electronic device 100 can use to detect the opening and closing of the flip cover. A distance sensor 180F is used to measure distance. A proximity sensor 180G may include, for example, a light-emitting diode (LED) and a photodetector, such as a photodiode. The electronic device 100 emits infrared light outward through the LED. The electronic device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that an object is near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that no object is near the electronic device 100. A fingerprint sensor 180H is used to collect fingerprints. A temperature sensor 180J is used to detect temperature. An ambient light sensor 180L is used to sense ambient light intensity.

[0108] Angle sensor 180M can acquire angle information and convert it into a usable electrical signal output. In one embodiment, angle sensor 180M can be disposed in display screen 194 for detecting the bending angle of the folding screen. Processor 110 can determine the physical state of electronic device 100 (e.g., unfolded state, bent state, or folded state) and whether the physical state of electronic device 100 has changed based on the signal detected by angle sensor 180M.

[0109] This application does not limit the specific type of sensor used to detect the physical state and motion posture of electronic device 100.

[0110] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100. Motor 191 can generate vibration prompts. Indicator 192 can be an indicator light, used to indicate charging status, battery level changes, and also to indicate messages, missed calls, notifications, etc. SIM card interface 195 is used to connect a SIM card.

[0111] The software system of electronic devices (such as mobile phones) can adopt a layered architecture, transaction-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses the layered architecture of the Android system as an example to illustrate the software architecture of a mobile phone. Please refer to... Figure 6 , Figure 6 This is a schematic diagram of the software architecture of the electronic device provided in the embodiments of this application.

[0112] like Figure 6 As shown, the layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: application layer, application framework layer, hardware abstraction layer (HAL), driver layer, and hardware layer. Wherein:

[0113] The application layer can include a series of packages. For example, applications can include third-party applications, as well as applications such as camera and gallery applications. Third-party applications include those with camera functionality, meaning they can use the electronic device's camera for taking pictures, video calls, etc. Camera applications may include, but are not limited to, a UI module, a photo-taking module, and a gallery module. The UI module can be a cameraUI module, primarily responsible for human-computer interaction in the camera application, such as controlling the preview interface and its display, and receiving and responding to user actions in the preview interface. The photo-taking module provides functions such as taking pictures and focusing. The gallery module can be used to store photos taken by the user in the electronic device's file system or a specific database for retrieval by applications such as the gallery.

[0114] The application framework layer provides the application programming interface (API) and programming framework for applications in the application layer. It primarily involves the camera framework, which may include camera access interfaces such as camera services. It acts as a bridge between the application and application layers, allowing interaction with third-party applications through the application API and with the HAL interface definition language (HIDL). The camera framework provides three data streams for third-party applications to choose from: a preview and callback data stream, a video recording stream, and a photo capture stream.

[0115] The application framework layer may also include a window manager, allowing third-party applications and gallery applications to display captured photos to the user with the support of the window manager. It may also include a physical state management module, which can determine the physical state of the electronic device 100 and, based on that physical state, determine its usage state. For example, if the electronic device 100 is in an unfolded or half-folded state, it is in inner screen usage state; if it is in a folded state, it is in outer screen usage state. The physical state or usage state of the electronic device 100 is then sent to a module that listens for that state, such as the camera service.

[0116] The Camera Service responds to requests from the application layer, providing session management, device management, and policy management. The Camera Service is an independent process that acts as a server, handling cross-process requests from application clients (such as third-party applications), performing internal operations, and then retransmitting the request as a client to the HAL process, which is acting as the server. Therefore, the Camera Service is part of the camera's pathway, playing a crucial bridging role.

[0117] Session management manages the camera device's lifecycle, parameter configuration, input management, and output management. It primarily involves camera preview sessions, camera capture sessions, and so on. A camera preview session is used to preview the camera's image stream, while a camera capture session is used to capture photos or videos. It has a state callback that handles changes in different states, such as ready or starting capture. For example, before using the camera's preview, photo, or video recording functions, a camera session needs to be created, and the relevant configurations completed within the session.

[0118] Device management is used to manage and control camera hardware devices, including connecting to, initializing, configuring, and controlling camera hardware devices, as well as processing data streams from camera devices. The camera service uses device management to coordinate application access to camera hardware devices and is responsible for handling various operations related to the camera hardware devices.

[0119] Policy management is used to manage image processing policies configured for third-party applications. Image processing policies include, but are not limited to, first policy, second policy, third policy, and video call policy. It is understood that different third-party applications may use the same or different image processing policies, primarily depending on the view controls used for display by the third-party application.

[0120] The receiving module is used to manage the whitelist of applications, update the physical state of electronic device 100, register listeners, callback strategies, and so on. For example, the receiving module can listen for third-party applications to start the camera function through a listener based on the whitelist. When the receiving module receives the physical state of electronic device 100 from the physical state management module of the system framework, or the usage status of electronic device 100 related to the physical state, it updates the physical state of electronic device 100 in a timely manner and sends a callback to the policy management.

[0121] The Hardware Abstraction Layer (HAL) is an interface layer located between the application framework layer and the driver layer, providing a virtual hardware platform for the operating system. The HAL can include a camera hardware abstraction layer and a policy execution module.

[0122] The camera hardware abstraction layer can provide virtual hardware for camera device 1 (first camera), camera device 2 (second camera), and other camera devices. For example, the camera hardware abstraction layer runs as an independent process within a native service. It communicates with the camera service across processes via the HIDL interface and issues actual operations for the first and second cameras via the standard HAL interface. This independent process can be a camera provider.

[0123] The strategy execution module stores a variety of image processing algorithms. For example, in this embodiment, the image processing algorithms may include: a video call connection algorithm, a first image processing algorithm, a second image processing algorithm, and a third image processing algorithm.

[0124] The video call continuation algorithm is used to ensure that the video call session is not interrupted when switching from the inner screen to the outer screen, or vice versa.

[0125] The first image processing algorithm is used to achieve rotation and mirroring by setting transformation parameters. Within the camera frame, the transformation parameters are primarily used to handle 3D transformations and rotations, specifically including rotation angles along the X, Y, and Z axes.

[0126] The second image processing algorithm is used to implement image rotation and mirroring functions at the output nodes. These output nodes include image processing nodes in the HAL layer, such as the GPU node. This node executes HAL operations, while the GPU node performs operations like rotation and flipping. In image processing, the rotate operation typically refers to a symmetrical operation of rotating counterclockwise by 2π / n angles around a certain axis, where n is a positive integer and represents the axis of rotation. The flip operation is used to flip a two-dimensional matrix, performing horizontal, vertical, or simultaneous horizontal and vertical flips.

[0127] The third image processing algorithm is used to implement image rotation and mirroring functions through a video codec (mediacodec). Mediacodec is a class provided by the system architecture for encoding and decoding, which accesses the underlying codec to implement encoding and decoding functions. The images processed are Joint Photographic Experts Group (JPEG) images. JPEG images include JPEG meta data, which consists of various tags and information embedded in the image. This information can include image attributes, shooting device information, user settings, etc. Therefore, mediacodec is used to set image attributes in JPEG images to achieve image rotation and mirroring functions.

[0128] For example, a third-party application can obtain a JPEG image by adding the JPEG_ORIENTATION parameter to the created photo request. After receiving the above parameter, the camera HAL layer will send the image generated by the camera driver to the media codec for image processing, thereby obtaining a JPEG image.

[0129] The driver layer is the layer between hardware and software, and it includes drivers for various hardware components. The driver layer can include camera drivers, digital signal processor (DSP) drivers, video codec drivers, and graphics processor (GPU) drivers, among others. Specifically, the camera driver drives the image sensors of one or more cameras to acquire images and drives the image signal processor to preprocess the images. The DSP driver drives the digital signal processor to process images. The video codec driver drives the video codec to process images. The GPU driver drives the graphics processor to process images.

[0130] The hardware layer may include a camera, an image signal processor, a digital signal processor, a video codec, and a graphics processor. The camera may include image sensors (e.g., image sensor 1, image sensor 2, etc.) from one or more cameras (e.g., a first camera and a second camera). Optionally, the camera may also include a camera motor, a lens, a time-of-flight (TOF) sensor, etc.

[0131] The following example, using a third-party application to access the camera, illustrates the workflow of electronic device software and hardware.

[0132] In this embodiment, when the touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the user operation into a raw input event (including touch coordinates, touch operation timestamp, etc.) and identifies the control corresponding to the input event. Taking a touch operation as a user operation acting on a camera control in a third-party application as an example, the third-party application calls the camera access interface of the application framework layer to activate the camera capability and determines the strategy configured for the third-party application and the folding state of the electronic device 100. Then, by calling the camera device in the camera hardware abstraction layer, such as camera device 1, it sends a command to activate the camera capability. The camera hardware abstraction layer sends this command to the camera driver in the driver layer. The camera driver can activate the corresponding camera device (such as the first camera). Then, the sensor in the first camera captures the image light signal and transmits the image light signal to the image signal processor for preprocessing to obtain the raw image stream. Then, the raw image stream is transmitted back to the hardware abstraction layer through the camera driver.

[0133] The Hardware Abstraction Layer (HAL) sends the RAW image stream to the policy execution module. Leveraging the graphics processor, the policy execution module within the HAL processes the raw image stream according to policies configured for third-party applications and the folding state of the electronic device 100. This processing includes actions such as rotation and mirroring to obtain better images. The HAL then returns this image data to the third-party application via the camera interface. Subsequently, the third-party application and gallery applications, with the support of a window manager, can display the photos to the user.

[0134] Figures 7A-7D This is a set of user interfaces for third-party applications to access the camera, provided in the embodiments of this application.

[0135] like Figure 7A As shown, at the first moment, when the physical state of the electronic device 100 is either unfolded or bent, the electronic device 100 is in the inner screen usage state, that is, the first display screen 200 is facing the user's face, and the user interface is displayed through the first display screen 200. For example, when the electronic device 100 is in the inner screen usage state, a third-party application is launched in response to a user operation (e.g., clicking the icon of a third-party application). In the third-party application, in response to a user operation (e.g., clicking the shooting control), the camera capability is activated. A command to activate the camera capability is sent through the camera device 2 in the camera hardware abstraction layer, thereby activating the second camera in the hardware layer, and displaying the first user interface 61 on the first display screen 200. Generally, the invoked camera is the rear camera by default, that is, the second camera 402 located on the display screen (i.e., the second display screen 300) behind the first display screen 200. Therefore, the first image displayed in the first user interface 61 is output by the second camera 402. At the second moment, the user folds the electronic device 100, placing it in a folded state. The electronic device 100 is in external screen usage mode, meaning the second display screen 300 is facing the user's face. The second user interface 62 of the third-party application is displayed on the second display screen 300. At this time, the third-party application is still using the second camera 402, and the third image displayed in the second user interface 62 is the output of the second camera 402. The electronic device 100 draws and displays the user interface of the third-party application (including the first user interface 61 and the second user interface 62) through first view controls, thereby displaying the captured image in the user interface. The first view controls include a surface view and a texture view.

[0136] from Figure 7A As can be seen, the installation angle of the second camera 402 changes when the external screen is in use. In order to ensure the adaptability of the second user interface 62, the application framework layer and the hardware abstraction layer need to process the image output by the second camera 402 accordingly.

[0137] For example, the physical state management module in the application framework layer can determine that the physical state of the electronic device 100 has changed from sensor information reported by the underlying sensors, thereby determining that the physical state of the electronic device 100 is folded, and send the physical state of the electronic device 100 to the receiving module in the camera service. The receiving module then sends the physical state of the electronic device 100 to the policy management, which determines the image processing policy for the third-party application based on the physical state of the electronic device 100 and the view control used by the third-party application. For example, if the physical state of the electronic device 100 is folded, and the third-party application draws the user interface through the first view control, then the policy management determines that the policy configured for the third-party application is the first policy. The policy management sends the first policy to the device management, which sends the first policy to the hardware abstraction layer via a buffer queue using binder communication.

[0138] For example, the camera hardware abstraction layer in the hardware abstraction layer obtains the first policy through a buffer queue and sends the first policy to the policy execution module. The policy execution module stores multiple algorithms. Based on the first policy, the policy execution module determines a first image processing algorithm and, based on the first image processing algorithm, determines a first parameter. Specifically, the first parameter may include one or more rotation angles along the X, Y, and Z axes, and may be a transform parameter. Then, the policy execution module sends the first parameter to the session management in the application framework layer. It is understood that after receiving the first image reported by the hardware layer, the camera device 2 also reports it to the session management in the application framework layer.

[0139] For example, the session management in the application framework layer reports the first image and the first parameter to the first view control via a buffer queue. The first view control draws the user interface of the third-party application based on the first parameter. Further, the first view control rotates the first layer in the user interface used to display the second image based on the first parameter to obtain the third image. The first parameter is used to adjust the rotation angle of the first layer.

[0140] For example, please see Figure 8 , Figure 8 This is a schematic diagram of image rotation provided in an embodiment of this application. For example... Figure 8 As shown, in the inner screen usage state, the second camera 402 outputs the first image. In the outer screen usage state, the second camera 402 is flipped 180 degrees, so the second image output by the second camera 402 is flipped 180 degrees in imaging direction relative to the first image. The first view control is used to draw the layer used to display the second image in the user interface of the third-party application. Therefore, the first view control can rotate the first layer used to display the first image by 180 degrees according to the first parameter to obtain the third image. Consequently, the imaging direction of the third image displayed in the first image also rotates 180 degrees relative to the second image. Finally, from... Figure 7A It can be seen that the third image displayed in the second user interface 62 is compatible with the second display screen 300. That is, the third image is not inverted compared with the first image, and the imaging direction of the third image is consistent with the imaging direction of the first image.

[0141] For example, at the third moment ( Figure 7A (Not shown) After electronic device 100 switches from external screen usage state to internal screen usage state, the physical state management module in the application framework layer senses the change in the physical state of electronic device 100, which is either an unfolded state or a half-folded state. At this time, electronic device 100 is in internal screen usage state. Therefore, the application framework layer can pass the internal screen usage state to the hardware abstraction layer through a buffer queue, and the hardware abstraction layer determines that electronic device 100 is in internal screen usage state. Therefore, the hardware abstraction layer can report the sensor parameters of the second camera 402 to the system layer, and further report them to the third-party application. Thus, the third-party application can obtain the sensor parameters of the second camera 402, and then select the corresponding rotation logic to process the image captured by the second camera 402 according to the sensor parameters, thereby ensuring that the processed image can be adapted to the current display state (i.e., displayed on the first display screen 200 in the unfolded state). That is, when electronic device 100 switches from external screen usage state to internal screen usage state, it no longer processes the image captured by the second camera 402 according to the first parameter, but instead processes the image captured by the second camera 402 according to the sensor parameters reported by the underlying layer.

[0142] like Figure 7B As shown, at the first moment, the electronic device 100 is in inner screen usage mode, displaying the first user interface of a third-party application on the first display screen 200. The first image displayed in the first user interface 61 is output by the second camera 402. At the second moment, the electronic device 100 is in outer screen usage mode, displaying the second user interface 62 of the third-party application on the second display screen 300. The electronic device 100 draws and displays the user interface of the third-party application (including the first user interface 61 and the second user interface 62) through a second view control, thereby displaying the captured image in the user interface. The second view control includes view objects created using the Open Graphics Library (OpenGL).

[0143] from Figure 7B As can be seen, the installation angle of the second camera 402 changes when the external screen is in use. In order to ensure the adaptability of the second user interface 62, the application framework layer and the hardware abstraction layer need to process the image output by the second camera 402 accordingly.

[0144] For example, the physical state management module in the application framework layer can determine that the physical state of the electronic device 100 has changed from sensor information reported by the underlying sensors, thereby determining that the physical state of the electronic device 100 is folded, and send the physical state of the electronic device 100 to the receiving module in the camera service. The receiving module then sends the physical state of the electronic device 100 to the policy management, which determines the image processing policy for the third-party application based on the physical state of the electronic device 100 and the view control used by the third-party application. For example, if the physical state of the electronic device 100 is folded, and the third-party application draws and displays the user interface through a second view control, then the policy management determines the policy configured for the third-party application as the second policy. The policy management sends the second policy to the device management, which sends the second policy to the hardware abstraction layer via a buffer queue using binder communication.

[0145] For example, the camera hardware abstraction layer in the hardware abstraction layer obtains the second strategy through a buffer queue, sends the second strategy to the strategy execution module, and also sends the first image reported by the hardware layer to the strategy execution module. The strategy execution module stores multiple algorithms. Based on the second strategy, the strategy execution module determines a second image processing algorithm and, based on the second image processing algorithm, determines second parameters. Specifically, the second parameters may include parameters for performing rotation operations and / or parameters for performing flip operations. Then, with the support of the graphics processing unit (GPU), the strategy execution module performs rotation and flip operations on the second image to obtain a third image. The second image is the image captured by the second camera 402 after flipping 180 degrees. For example, the image processor rotates the second image 180 degrees to obtain the third image. Then, the hardware abstraction layer returns the third image to the third-party application through the camera interface. Subsequently, the third-party application displays the third image in the second user interface 62 with the support of the second view control. For a description of the first, second, and third images, please refer to... Figure 8 This will not be elaborated upon further.

[0146] For example, at the third moment ( Figure 7B (Not shown) After electronic device 100 switches from external screen usage state to internal screen usage state, the physical state management module in the application framework layer senses the change in the physical state of electronic device 100, which is either an unfolded state or a half-folded state. At this time, electronic device 100 is in internal screen usage state. Therefore, the application framework layer can pass the internal screen usage state to the hardware abstraction layer through a buffer queue, and the hardware abstraction layer determines that electronic device 100 is in internal screen usage state. Therefore, the hardware abstraction layer can report the sensor parameters of the second camera 402 to the system layer, and further report them to the third-party application. Thus, the third-party application can obtain the sensor parameters of the second camera 402, and then select the corresponding rotation logic to process the image captured by the second camera 402 according to the sensor parameters, thereby ensuring that the processed image can be adapted to the current display state (i.e., displayed on the first display screen 200 in the unfolded state). That is, when electronic device 100 switches from external screen usage state to internal screen usage state, it no longer processes the image captured by the second camera 402 according to the second parameter, but instead processes the image captured by the second camera 402 according to the sensor parameters reported by the underlying layer.

[0147] like Figure 7C As shown, at the first moment, the electronic device 100 is in inner screen usage mode, displaying a first user interface 61 of a third-party application on the first display screen 200. The first image displayed in the first user interface 61 is output by the second camera 402. At the second moment, the electronic device 100 is in outer screen usage mode, displaying a second user interface 62 of the third-party application on the second display screen 300. The electronic device 100 draws and displays the user interface of the third-party application (including the first user interface 61 and the second user interface 62) through a second view control, thereby displaying the captured image in the user interface. The image in the third-party application is a JPEG image; that is, the first image displayed in the first user interface 61 is in JPEG format, and the second image displayed in the second user interface 62 is also a JPEG image.

[0148] from Figure 7C As can be seen, the installation angle of the second camera 402 changes when the external screen is in use. In order to ensure the adaptability of the second user interface 62, the application framework layer and the hardware abstraction layer need to process the image output by the second camera 402 accordingly.

[0149] For example, the physical state management module in the application framework layer can determine that the physical state of the electronic device 100 has changed from sensor information reported by the underlying sensors, thereby determining that the physical state of the electronic device 100 is folded, and send the physical state of the electronic device 100 to the receiving module in the camera service. The receiving module then sends the physical state of the electronic device 100 to the policy management, which determines the image processing policy for the third-party application based on the physical state of the electronic device 100 and the JPEG image used by the third-party application. For example, if the physical state of the electronic device 100 is folded, and the third-party application draws and displays the user interface through a second view control, then the policy management determines that the policy configured for the third-party application is the third policy. The policy management sends the third policy to the device management, which sends the third policy to the hardware abstraction layer via a buffer queue using binder communication.

[0150] For example, the camera hardware abstraction layer in the hardware abstraction layer obtains the third strategy through a buffer queue, sends the third strategy to the strategy execution module, and also sends the second image reported by the hardware layer to the strategy execution module. The strategy execution module stores multiple algorithms. Based on the third strategy, the strategy execution module determines the second image processing algorithm and, based on the second image processing algorithm, determines the third parameters. Specifically, the third parameters may include JPEG parameters used for performing rotation operations, such as the JPEG_ORIENTATION parameter. Then, with the support of the video codec, the strategy execution module performs a rotation operation on the second image to obtain the third image. The second image is the image captured by the second camera 402 after being rotated 180 degrees. For example, the media codec rotates the second image 180 degrees to obtain the third image. Then, the hardware abstraction layer returns the third image to the third-party application through the camera interface. Subsequently, the third-party application displays the third image in the second user interface 62 with the support of the window manager. For a description of the first, second, and third images, please refer to [reference needed]. Figure 8 This will not be elaborated upon further.

[0151] For example, at the third moment ( Figure 7C (Not shown) After electronic device 100 switches from external screen usage state to internal screen usage state, the physical state management module in the application framework layer senses the change in the physical state of electronic device 100, which is either an unfolded state or a half-folded state. At this time, electronic device 100 is in internal screen usage state. Therefore, the application framework layer can pass the internal screen usage state to the hardware abstraction layer through a buffer queue, and the hardware abstraction layer determines that electronic device 100 is in internal screen usage state. Therefore, the hardware abstraction layer can report the sensor parameters of the second camera 402 to the system layer, and further report them to the third-party application. Thus, the third-party application can obtain the sensor parameters of the second camera 402, and then select the corresponding rotation logic to process the image captured by the second camera 402 according to the sensor parameters, thereby ensuring that the processed image can be adapted to the current display state (i.e., displayed on the first display screen 200 in the unfolded state). That is, when electronic device 100 switches from external screen usage state to internal screen usage state, it no longer processes the image captured by the second camera 402 according to the third parameter, but instead processes the image captured by the second camera 402 according to the sensor parameters reported by the underlying layer.

[0152] like Figure 7D As shown, in a video call scenario, at the first moment, the electronic device 100 is in inner screen usage mode, and the first display screen 200 displays the first user interface of a third-party application. The first user interface is used to display the first video frame 41, which is the first image stream captured by the first camera 401 (front-facing camera). At the second moment, the electronic device 100 is in outer screen usage mode, and the second display screen 300 displays the second user interface of the third-party application. The second user interface displays the second video frame 42, which is the image captured by the second camera 402 (rear-facing camera).

[0153] from Figure 7D As can be seen, the camera used for video calls changes when switching from the inner screen usage state to the outer screen usage state. In order to ensure the continuity of video calls, the application framework layer and the hardware abstraction layer need to ensure that the current session is not interrupted.

[0154] For example, the physical state management module in the application framework layer can determine that the physical state of electronic device 100 has changed from sensor information reported by the underlying sensors, thereby determining that the physical state of electronic device 100 is folded, and send the physical state of electronic device 100 to the receiving module in the camera service. The receiving module then sends the physical state of electronic device 100 to policy management, and policy management determines the image processing policy for the third-party application based on the physical state of electronic device 100 and the video call scenario of the third-party application. For example, if the physical state of electronic device 100 is folded and the third-party application is conducting a video call, then policy management determines that the policy configured for the third-party application is a video call policy. Policy management sends the video call policy to device management, and device management sends the video call policy to the hardware abstraction layer via a buffer queue using binder communication.

[0155] For example, the camera hardware abstraction layer in the hardware abstraction layer obtains the video call policy through a buffer queue and sends the video call policy to the policy execution module. The policy execution module determines the video call continuation algorithm based on the video call policy and determines a first instruction based on the video call continuation algorithm. Specifically, the first instruction may include an instruction to turn off the first camera 401 and start the second camera 402. Then, the policy execution module sends the first instruction to the camera hardware abstraction layer, which turns off camera device 1 and starts camera device 2 according to the first instruction. Therefore, camera device 1 turns off the first camera 401, and camera device 2 starts the second camera 402, which begins acquiring image data. The second image stream reported by the hardware layer to the hardware abstraction layer is the one acquired by the second camera 402. The hardware abstraction layer returns the second image stream to the third-party application through the camera interface. Subsequently, the third-party application, with the support of a window manager, can display the second video screen 42 to the user. As can be seen, when switching from the inner screen usage state to the outer screen usage state, the session between the application layer, the application framework layer, and the hardware abstraction layer is not interrupted. Instead, the application framework layer senses the state change and restarts the image stream configuration. The third-party application is unaware of this, so the video call is not interrupted, improving the user experience.

[0156] For example, at the third moment ( Figure 7D (Not shown) After electronic device 100 switches from external screen usage state to internal screen usage state, the physical state management module in the application framework layer detects the change in the physical state of electronic device 100, which is either an unfolded state or a half-folded state. At this time, electronic device 100 is in internal screen usage state. Therefore, the application framework layer can pass the internal screen usage state to the hardware abstraction layer through a buffer queue. The hardware abstraction layer determines that electronic device 100 is in internal screen usage state. Therefore, the hardware abstraction layer can issue a command to start the first camera 401 and turn off the second camera 402. So, electronic device 100 restarts streaming, the first camera 401 starts collecting image data, and the hardware layer reports the images collected by the first camera 401 to the hardware abstraction layer.

[0157] It should be noted that, Figure 7D This explanation uses vertically folding electronic devices as an example, but it also applies to horizontally folding electronic devices, which will not be elaborated upon here.

[0158] It should be understood that the steps in the above-described method embodiments provided in this application can be implemented by integrated logic circuits in the processor hardware or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0159] This application also provides an electronic device that may include a memory and a processor. The memory may be used to store a computer program; the processor may be used to invoke the computer program in the memory to cause the electronic device to perform the methods in any of the above embodiments.

[0160] This application also provides a chip system including at least one processor for implementing the functions involved in the methods performed by the electronic device in any of the above embodiments.

[0161] In one possible design, the chip system also includes a memory for storing program instructions and data, which may be located within or outside the processor.

[0162] The chip system can consist of chips or include chips and other discrete components.

[0163] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0164] Optionally, the chip system may contain one or more memories. The memory may be integrated with the processor or disposed separately from it; this application embodiment does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed separately on different chips. This application embodiment does not specifically limit the type of memory or the arrangement of the memory and processor.

[0165] For example, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0166] This application also provides a computer program product comprising: a computer program (also referred to as code or instructions) that, when run, causes a computer to perform the method executed by the electronic device in any of the above embodiments.

[0167] This application also provides a computer-readable storage medium storing a computer program (also referred to as code or instructions). When the computer program is run, it causes the computer to perform the method executed by the electronic device in any of the above embodiments.

[0168] The various embodiments of this application can be combined arbitrarily to achieve different technical effects.

[0169] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive).

[0170] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0171] In summary, the above description is merely an embodiment of the technical solution of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made according to the disclosure of the present invention should be included within the scope of protection of the present invention.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. An image display method, characterized in that, An electronic device with a foldable screen, the electronic device including a first display screen, a second display screen, a first camera, and a second camera, wherein the first display screen and the second display screen are located on different sides of the electronic device, the first camera is located on the first display screen, and the second camera is located on the second display screen, the method comprising: In the internal screen usage state, the first user interface of a third-party application is displayed through the first display screen, wherein the first user interface is used to display the first video call screen, and the first video call screen includes the image captured by the first camera; In response to detecting that the electronic device switches from the inner screen usage state to the outer screen usage state, a processing strategy corresponding to the application scenario is determined at the application framework layer based on the application scenario of the third-party application. When the external screen is in use, the first camera is turned off through the hardware abstraction layer according to the processing strategy, and the second camera is controlled to capture images. The second user interface of the third-party application is displayed on the second display screen to keep the video call of the third-party application uninterrupted. The second user interface is used to display the second video call screen, which includes an image captured by the second camera. The application scenarios include one or more of the following: drawing the second user interface using a first view control, drawing the second user interface using a second view control, displaying a Joint Image Experts Group JPEG image in the second user interface, and video calling; in the scenario where the second user interface is drawn using the first view control, the processing strategy includes: setting deformation parameters to achieve image rotation; in the scenario where the second user interface is drawn using the second view control, the processing strategy includes: achieving image rotation at the output node; in the scenario where a JPEG image is displayed in the second user interface, the processing strategy includes: achieving image rotation through video encoding and decoding; in the video calling scenario, the processing strategy includes: generating a first instruction, the first instruction including turning off the first camera and turning on the second camera.

2. The method according to claim 1, characterized in that, The method further includes: In the state of using the inner screen, the first user interface is also used to display the first image captured by the second camera; In the external screen usage state, the second user interface is also used to display a third image, wherein the third image is an image obtained by rotating the second image, and the second image includes the image captured by the second camera in the external screen usage state.

3. The method according to claim 2, characterized in that, The method further includes: The imaging direction of the first image differs from that of the second image by 180 degrees, and the imaging direction of the third image is the same as that of the first image.

4. An electronic device, characterized in that, The electronic device includes: one or more processors; a memory; wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the electronic device to perform the method as described in any one of claims 1-3.

5. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-3.

6. A computer program product containing instructions, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-3.

7. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-3.

Images