Image display method and related device

Abstract

The invention provides an image display method and a related device, and the method comprises the steps: displaying a first user interface of a third-party application program through a first display screen in an inner screen use state, the first user interface being used for displaying a first video call picture, and the first video call picture comprising an image collected through a first camera; in the outer screen use state, a second user interface of the third-party application program is displayed through the second display screen, the second user interface is used for displaying a second video call picture, a human video call comprises an image collected through the second camera, and the first display screen and the second display screen are located on different faces of the electronic equipment. The first camera is located on the first display screen, and the second camera is located on the second display screen. Therefore, when the electronic equipment is switched from the inner screen use state to the outer screen use state, the video call is not interrupted, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to an image display method and related devices. Background Art

[0002] With technological advancements, electronic devices can be equipped with at least two display screens, including a primary screen (which can be called an inner screen) and a secondary screen (which can be called an outer screen). Generally speaking, the primary screen and the secondary screen are located on different sides of the electronic device. The primary screen supports display and is equipped with a camera (which can be called a front-facing camera), while the secondary screen supports display and is equipped with a camera (which can be called a rear-facing camera).

[0003] The emergence of foldable screens has diversified the working states of electronic devices. For example, the physical states of electronic devices can include unfolded, bent, and folded. When the electronic device is in the unfolded or bent state, a video call with the front camera is activated through a third-party application. When the electronic device switches from the unfolded or bent state to the folded state, the video call will be disconnected, and the user will need to reconnect the video in the folded state, resulting in a poor user experience. Summary of the Invention

[0004] An image display method and related states provided in an embodiment of the present application can improve the user experience when using a third application.

[0005] In a first aspect, the present application provides an image display method, applied to an electronic device, comprising:

[0006] When the inner screen is in use, a first user interface of the third-party application is displayed on the first display screen, wherein the first user interface is used to display a first video call screen, and the first video call screen includes an image captured by the first camera;

[0007] When the external screen is in use, the second user interface of the third-party application is displayed through the second display screen, wherein the second user interface is used to display a second video call screen, and the human video call includes an image captured by the second camera, the first display screen and the second display screen are located on different sides of the electronic device, the first camera is located on the first display screen, and the second camera is located on the second display screen.

[0008] It can be seen that when the electronic device switches from the inner screen usage state to the outer screen usage state, the video call will not be interrupted, and the user can continue the video call, improving the user experience.

[0009] In a possible implementation of the first aspect, the method further includes:

[0010] When the inner screen is in use, the first user interface is further used to display a first image captured by the second camera;

[0011] When the external screen is in use, the second user interface is also used to display a third image, wherein the third image is an image obtained by rotating the second image, and the second image includes the image captured by the second camera when the external screen is in use.

[0012] In the above method, when the electronic device switches from the inner screen usage state to the outer screen usage state, the image captured by the second camera in the outer screen usage state can be rotated to adapt to the outer screen usage state and improve the user experience.

[0013] In a possible implementation of the first aspect, the method further includes:

[0014] An imaging direction of the first image differs from an imaging direction of the second image by 180 degrees, and an imaging direction of the third image is the same as that of the first image.

[0015] It can be seen that when the electronic device switches from the inner screen use state to the outer screen use state, the third image is rotated, so the imaging screen of the third image will not appear inverted, which improves the user experience.

[0016] In a possible implementation of the first aspect, displaying the second user interface of the third-party application on the second display screen when the external screen is in use includes:

[0017] Detecting a physical state of the electronic device to determine whether the electronic device is switched from the inner screen usage state to the outer screen usage state;

[0018] Determining a processing strategy corresponding to the third-party application;

[0019] determining the second user interface according to the processing strategy;

[0020] The second user interface is displayed through the second display screen.

[0021] It can be seen that the electronic device can detect the physical state to determine the corresponding processing strategy, and then perform corresponding processing on the second user interface according to the processing strategy, so that the second user interface can adapt to the second display screen.

[0022] In a possible implementation of the first aspect, determining a processing policy corresponding to the third-party application includes:

[0023] Determine a processing strategy corresponding to the operating scenario based on an application scenario of the third application, wherein the application scenario includes one or more of the following: drawing the second user interface through a first view control, drawing the second user interface through a second view control, displaying a Joint Photographic Experts Group JPEG image in the second user interface, and video calls.

[0024] In a possible implementation of the first aspect, the method further includes:

[0025] In the scenario where the second user interface is drawn through the first view control, the processing strategy includes: setting deformation parameters to realize image rotation.

[0026] In a possible implementation of the first aspect, the method further includes:

[0027] In the scenario where the second user interface is drawn through the second view control, the processing strategy includes: implementing image rotation at the output node.

[0028] In a possible implementation of the first aspect, the method further includes:

[0029] In a scenario where a JPEG image is displayed in the second user interface, the processing strategy includes: implementing image rotation through video encoding and decoding.

[0030] In a possible implementation of the first aspect, the method further includes:

[0031] In the video call scenario, the processing strategy includes: generating a first instruction, where the first instruction includes turning off the first camera and starting the second camera.

[0032] In the above method, the electronic device can select a corresponding strategy based on the actual application scenario of the third-party application, so that the electronic device switches from the internal screen usage state to the external screen usage state. The selected strategy can process the images collected by the camera more efficiently to ensure the user experience.

[0033] In a second aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a memory; wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the image display method described in the first aspect or any possible implementation of the first aspect.

[0034] In a third aspect, the present application provides a chip or chip system, comprising at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via a circuit, and the at least one processor is configured to execute a computer program or instruction to perform the image display method described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip may be an input / output interface, a pin, or a circuit.

[0035] In one possible implementation, the chip or chip system described above in the embodiments of the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit within the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0036] In a fourth aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program. When the computer program is executed by a processor, the computer executes the image display method described in the first aspect or any possible implementation of the first aspect.

[0037] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when running on a communication device, enables the communication device to execute the image display method described in the first aspect or any possible implementation of the first aspect.

[0038] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution or beneficial effect is included in at least one embodiment. Therefore, the description of a technical feature, technical solution or beneficial effect in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in the present embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that the embodiment can be implemented without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in specific embodiments that do not embody all embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The following is an introduction to the drawings used in the embodiments of this application.

[0040] Figure 1 1 is a schematic diagram of a product form of an electronic device 100 with a longitudinal folding method provided in an embodiment of the present application;

[0041] Figure 2 1 is a schematic diagram of a product form of an electronic device with a longitudinal folding method provided in an embodiment of the present application;

[0042] Figure 3 This is a schematic diagram of a scenario in which a third-party application calls a camera for shooting, provided in an embodiment of the present application;

[0043] Figure 4 This is a schematic diagram of a scenario in which a third-party application calls a camera to conduct a video call, provided in an embodiment of the present application;

[0044] Figure 5 A schematic diagram of the hardware structure of an electronic device 100 is shown as an example;

[0045] Figure 6 Schematic diagram of the software architecture of the electronic device provided in the embodiment of the present application;

[0046] Figures 7A-7D A set of user interfaces for third-party applications to call cameras provided in an embodiment of the present application;

[0047] Figure 8 This is a schematic diagram of image rotation provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The terms used in the following examples of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in the specification and appended claims of the present application, the singular expressions "a," "an," "said," "above," "the," and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to and encompasses any or all possible combinations of one or more of the listed items.

[0049] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0050] The term "user interface (UI)" in the specification, claims and drawings of this application refers to the media interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as <head> and <body>. A node corresponds to a control or attribute in the interface, and the node is presented as user-visible content after parsing and rendering. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is a source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code, such as HTML through <body>. 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0051] A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operations that uses graphics. It can be an icon, window, control, or other interface element displayed on the display of an electronic device. Controls can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and other visual interface elements.

[0052] The foldable electronic device 100 includes a foldable screen that can be folded to form at least two screens. The foldable screen on the electronic device 100 can be folded in two ways. One is a longitudinal folding method (for example, Figure 1 The folding method shown in the figure) is to fold vertically according to the bending part (for example, fold up and down); one is the horizontal folding method (for example Figure 2 The folding method shown in the figure) is to fold horizontally according to the bent portion (for example, fold left and right).

[0053] See Figure 1 , Figure 1 FIG. 1 is a schematic diagram of a product form of an electronic device 100 with a longitudinal folding method provided in an embodiment of the present application. Figure 1 As shown, the electronic device 100 may include a first display screen 200, a second display screen 300, a first camera 401, and a second camera 402. The first display screen 200 and the first camera 401 are located on the front of the electronic device 100, the first display screen 200 can be called an inner screen, the first camera 401 can be called a front camera, the second display screen 300 and the second camera 402 are located on the back of the electronic device 100, the second display screen 300 can be called an outer screen, and the second camera 402 can be called a rear camera. The size of the first display screen 200 is larger than that of the second display screen 300, the first display screen 200 can be called a main screen, and the second display screen 300 can be called a secondary screen. As shown Figure 1 As shown, the first display 200 is a foldable display, including regions 201, 202, and 203. Region 202 is bendable and located on the curved portion of the electronic device 100. The ends of the curved portion of the electronic device 100 connect to regions 201 and 203, respectively. The angle Q between the ends of the curved portion of the electronic device 100 (also the bending angle Q of the first display 200) can also be understood as the angle between the plane containing region 201 and the plane containing region 203. The second display 300 includes at least one region, namely, region 301.

[0054] like Figure 1 As shown in (a), when the first display screen 200 is in the unfolded state, regions 201 and 203 are essentially in the same plane. The light-emitting surface of the second display screen 300 faces away from the light-emitting surface of the first display screen 200; specifically, the light-emitting surface of region 301 faces away from the light-emitting surface of region 201. In one embodiment, the second display screen 300 is a non-foldable display screen and is located at one end of the bent portion (i.e., the end connected to region 201).

[0055] In one embodiment, the first camera 401 is located above the area 201, and the second camera 402 is located above the area 301. The second camera 402 may include at least one camera and may be understood as a camera module. When the electronic device 100 is in the unfolded state, the first camera 401 is located on the front of the electronic device 100 and may function as a front-facing camera, performing the work of a front-facing camera and displaying the captured image on the first display 200; the second camera 402 is located on the back of the electronic device 100 and may function as a rear-facing camera, performing the work of a rear-facing camera and displaying the captured image on the first display 200.

[0056] In one embodiment, the electronic device 100 can be bent toward the front along the bent portion to obtain Figure 1 (b) shows the electronic device 100 in a bent state.

[0057] like Figure 1 As shown in (b), the first display screen 200 is bent, the plane containing area 201 intersects the plane containing area 203, the light-emitting surface of area 301 faces away from the light-emitting surface of area 201, and the plane containing area 301 intersects the plane containing area 203. When the electronic device 100 is in the bent state, the first camera 401 functions as a front-facing camera, performing the tasks of a front-facing camera, and the second camera 402 functions as a rear-facing camera, performing the tasks of a rear-facing camera.

[0058] In one embodiment, Figure 1 The electronic device 100 in the unfolded state shown in (a) can be bent along the bending portion in a manner of bending toward the front to obtain Figure 1 The electronic device 100 is in the folded state shown in (c). In another embodiment, Figure 1 The electronic device 100 in the bent state shown in (b) can be further folded along the bent portion to form Figure 1 The electronic device 100 is in the folded state shown in (c).

[0059] like Figure 1 As shown in (c), the first display screen 200 is in a folded state, and the light emitting surface of the area 201 is opposite to the light emitting surface of the area 203. Therefore, the area 201 and the area 203 are not visible to the user because Figure 1 (a) and Figure 1 The first camera 401 shown in (b) is located on area 201, so the first camera 401 is also invisible to the user at this time. The light-emitting surfaces of the second camera 402 and area 301 are opposite to the light-emitting surface of area 201, the plane where area 301 is located is parallel to the plane where area 203 is located, and the light-emitting direction of the second camera 402 and area 301 is consistent with the light-emitting direction of area 203. At this time, area 203 is invisible to the user, while the second camera 402 and area 301 are visible to the user. It can be seen that in the folded state, the first camera 401 used to perform the work tasks of the front camera is invisible to the user. Therefore, the second camera 402, which is visible to the user, can be used to perform the work tasks of the front camera and display the captured image on the second display screen 300.

[0060] See Figure 2 , Figure 2 Schematic diagram of a product form of an electronic device with a longitudinal folding method provided by an embodiment of the present application. Figure 2 As shown, the electronic device 100 may include a first display screen 200 , a second display screen 300 , a first camera 401 and a second camera 402 .

[0061] like Figure 2 As shown in (a), the second display screen 300 of the electronic device 100 may include three areas, namely area 301, area 302 and area 303. Among them, area 302 is bendable. The two ends of the bending part of the electronic device 100 are connected to area 301 and area 303 respectively. When the electronic device is in the unfolded state, Figure 2 As shown in (a), the light emitting surface of the first display screen 200 and the light emitting surface of the second display screen 300 are opposite to each other, that is, the light emitting surface of the area 201 and the light emitting surface of the area 301 are opposite to each other.

[0062] In some embodiments, Figure 2 The electronic device 100 in the unfolded state shown in (a) is partially folded to obtain the electronic device 100 in the bent state, for example Figure 2 The electronic device 100 shown in (b).

[0063] In some embodiments, Figure 2 The electronic device 100 in the unfolded state shown in (a) is folded to obtain the electronic device 100 in the folded state, and the electronic device 100 can also be folded. Figure 2 The electronic device 100 in the bent state shown in (b) is folded (to obtain the electronic device 100 in the folded state, for example Figure 2 The electronic device 100 shown in (c).

[0064] It should be noted that the relevant descriptions of "first display screen 200", "second display screen 300", "first camera 401", "second camera 402", "area 201", "area 203" and "area 301" can be referred to. Figure 1 , I will not go into details this time.

[0065] In the embodiment of the present application, the angle Q of the electronic device 100 has a value range of [0°, 180°]. If Q∈[0°, P1], the electronic device 100 is in a folded state; if Q∈(P1, P2), the electronic device 100 is in a bent state; and if Q∈[P2, 180°], the electronic device 100 is in an unfolded state. 0°<P1<P2<180°. P1 and P2 can be preset angle thresholds. P1 and P2 can be determined based on the usage habits of a large number of users using foldable screens; or, P1 and P2 can be set by the user in the electronic device 100.

[0066] In some embodiments, based on the usage habits of most users, the value range of the preset angle threshold P1 in the embodiment of the present application can be (0, 30°), and the value range of the preset angle threshold P2 can be (150°, 180°). For example, the preset angle threshold P1 can be 5°, 10°, 15°, 20°, etc. The preset angle threshold P2 can be 155°, 160°, 165°, or 170°, etc.

[0067] It is understandable that when the electronic device 100 is in the folded state, the bent state, and the unfolded state, the bending angle of the bent portion (for example, the bending angle Q of the first display screen 200) is different, but the specific value of the bending angle Q is not limited.

[0068] Next, the first display screen 200 is a flexible folding screen, and the second display screen 300 is a rigid screen. The first display screen 200 displays a graphical user interface (GUI) through area 201, area 202, and area 203. The GUI is hereinafter referred to as the user interface. For an electronic device 100 with a longitudinal folding mode, the second display screen 300 displays the user interface through area 301, and areas 302 and 303 are not display areas. For an electronic device 100 with a transverse folding mode, the second display screen 300 displays the user interface through area 303, and areas 301 and 302 are not display areas.

[0069] As the functions of electronic devices become increasingly sophisticated, electronic devices equipped with cameras can not only start the camera to shoot in the camera application, but also start the camera to shoot and make video calls through third-party applications.

[0070] See Figure 3 , Figure 3 This is a schematic diagram of a scenario in which a third-party application calls a camera for shooting, provided in an embodiment of the present application.

[0071] Take the electronic device 100 with a longitudinal folding mode in the unfolded state as an example. Figure 3 As shown, in the unfolded state, that is, when the first camera 401 (front camera) faces the user, the electronic device 100 starts a third-party application in response to the user operation. In the third-party application, the electronic device 100 starts the second camera 402 for shooting in response to the user operation, and displays the first user interface 31 on the first display screen 200. Then, the electronic device 100 is folded along the bent portion and can be switched from the unfolded state to the folded state. In the folded state, that is, when the second camera 402 (rear camera) faces the user, the electronic device 100 displays the user interface of the third-party application in the area 301 of the second display screen 300. Because the third-party application is still calling the second camera 402 for shooting, the second user interface 33 captured by the second camera 402 is displayed in the area 301.

[0072] from Figure 3 It can be seen that the second user interface 33 displayed on the second display 300 is inverted (also called flipped upside down) relative to the first user interface 31 displayed on the first display 200. Because the object displayed in the first user interface 31 is oriented in the same direction as the actual object being photographed, the object displayed in the second user interface 33 is reversed compared to the actual object being photographed, thus affecting the user experience. The reason for the inverted image is that the third-party application that includes the camera function was not developed by the manufacturer and is unable to perceive the physical state of the electronic device 100 (for example, the unfolded, bent, or folded state).

[0073] When the electronic device 100 is in the unfolded state or the bent state, the electronic device 100 uses the first display screen 200 to display the user interface. Since the first display screen 200 can be called the inner screen, when the electronic device 100 is in the unfolded state or the bent state, the electronic device 100 can be considered to be in the inner screen use state. When the electronic device 100 is in the folded state, the electronic device 100 uses the second display screen 300 to display the user interface. Since the second display screen 300 can be called the outer screen, when the electronic device 100 is in the folded state, the electronic device 100 can be considered to be in the outer screen use state.

[0074] Only when the inner screen is in use and the second camera 402 is marked as a standard camera, can the bottom layer report the sensor parameters of the second camera 402 (such as the installation angle) to the system layer and further report them to the third-party application. Therefore, only when the inner screen is in use can the third-party application obtain the sensor parameters of the second camera 402 and then select the corresponding rotation logic based on the sensor parameters to process the image captured by the second camera 402, thereby ensuring that the processed image can adapt to the current display state (i.e., displayed on the first display screen 200 in the expanded state). For example, when the inner screen is in use, the electronic device 100 does not rotate (for example, the electronic device 100 is in portrait mode), the installation angle of the second camera 402 reported by the bottom layer is 0. The installation angle of 0 is used to indicate that the image data is consistent with the natural direction of the device, and no additional rotation is required for the image output by the second camera 402; when the electronic device 100 is rotated 90 degrees clockwise (for example, the electronic device is in landscape mode), the installation angle of the second camera 402 reported by the bottom layer is 90. The installation angle of 0 is used to indicate that the image output by the second camera 402 needs to be rotated 90 degrees clockwise to be consistent with the natural direction of the electronic device.

[0075] After the electronic device 100 switches from the inner screen usage state to the outer screen usage state, the sensor parameters of the second camera 402 change. Since the bottom layer can report the camera's sensor parameters to the third-party application only when the electronic device 100 is in the inner screen usage state, the bottom layer cannot report the modified camera's sensor parameters to the third-party application when the outer screen is in use. Therefore, the sensor parameters obtained by the third-party application are the sensor parameters in the inner screen usage state (i.e., the sensor parameters corresponding to the standard camera), resulting in the processed image being unable to adapt to the current display state (i.e., the installation angle of the second camera 402 is rotated 180 degrees and displayed on the second display screen 300 in the folded state).

[0076] like Figure 3 As shown, after the electronic device 100 switches from the inner screen usage state to the outer screen usage state by bending inward, since in the outer screen usage state, the third-party application still obtains the sensor parameters of the second camera 402 in the inner screen usage state (for example, the installation angle is 0), the third-party application will not perform any additional processing on the image output by the second camera 402. Since the installation angle of the second camera 402 is rotated 180 degrees, the image output by the second camera 402 will also be rotated 180 degrees. Because the image output by the second camera 402 will not be additionally processed according to the sensor parameters in the inner screen usage state, when the third-party application calls the second camera 402 to shoot in the outer screen usage state, the image preview and the finished film are inverted compared to those in the inner screen usage state.

[0077] See Figure 4 , Figure 4 This is a schematic diagram of a scenario in which a third-party application calls a camera to make a video call, as provided in an embodiment of the present application. Figure 4 (a) is a schematic diagram of a scenario in which the electronic device 100 is folded vertically to make a video call. Figure 4 (b) is a schematic diagram of a scenario in which the electronic device 100 is folded horizontally to make a video call.

[0078] from Figure 4 As can be seen, for video calling scenarios, both electronic devices 100 with a vertical folding configuration and those with a horizontal folding configuration experience the phenomenon of video call interruption after switching from the inner-screen to the outer-screen configuration. For example, when the electronic device 100 is in the inner-screen configuration, a third-party application uses the first camera 401 to make a video call. When the electronic device 100 switches from the inner-screen configuration to the outer-screen configuration, the camera used for the video call must be switched from the first camera 401 to the second camera 402, as the first camera 401 is not visible to the user in the outer-screen configuration. Because the first camera 401 and the second camera 402 are mounted in different locations and are physically different cameras, the third-party application cannot switch the camera used for the video call to the second camera 402 in the outer-screen configuration. Typically, the third-party application directly closes the video call, but this operation causes the user's video call to be interrupted. To resume the video call, the user must re-initiate the call, significantly reducing the user experience when using the electronic device 100.

[0079] Based on this, the present application provides an image display method, which is applied to an electronic device, wherein the electronic device is equipped with a foldable display screen (which can be called a folding screen). The electronic device can be called a foldable electronic device, the folding of the folding screen can also be called the folding of the electronic device, and the physical state / posture of the folding screen can also be called the object state of the electronic device.

[0080] The electronic device includes a first display screen and a second display screen, a first camera is installed on the first display screen, and a second camera is installed on the second display screen. The first display screen and the second display screen are located on different sides of the electronic device. For example, the first display screen is the main screen of the electronic device, and the second display screen is the secondary screen of the electronic device. The first camera and the second camera are also located on different sides of the electronic device and are used to perform different shooting tasks. For example, the first camera is the front camera of the electronic device, and the second camera is the rear camera of the electronic device. When the internal screen is used, the first user interface of the third-party application is displayed through the first display screen, wherein the first user interface is used to display the first video call screen, and the first video call screen includes the image captured by the first camera; when the external screen is used, the second user interface of the third-party application is displayed through the second display screen, wherein the second user interface is used to display the second video call screen, and the video call includes the image captured by the second camera. The first display screen and the second display screen are located on different sides of the electronic device, and the first camera is located on the first display screen, and the second camera is located on the second display screen. Therefore, when the electronic device switches from the internal screen use state to the external screen use state, the video call will not be interrupted, thereby improving the user experience.

[0081] In a possible implementation, when the inner screen is in use, the first user interface is further used to display the first image captured by the second camera;

[0082] When the external screen is in use, the second user interface is also used to display a third image. The third image is a rotation of the second image, which includes the image captured by the second camera when the external screen is in use. Therefore, when the electronic device switches from internal screen use to external screen use, the displayed image will not appear inverted, improving the user experience.

[0083] First, the electronic device involved in the embodiments of the present application is introduced.

[0084] The electronic devices in the embodiments of the present application may include, but are not limited to, smart screen devices, smart televisions (TVs), mobile phones, tablet computers, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), wearable devices (such as smart watches and smart bracelets), and other devices with display functions. The embodiments of the present application do not impose any special restrictions on the specific form of the electronic devices.

[0085] Figure 5 A schematic diagram of the hardware structure of an electronic device 100 is shown as an example.

[0086] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, an angle sensor 180M, etc.

[0087] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0088] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0089] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0090] Processor 110 may also include a memory for storing instructions and data. In one embodiment, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the instruction or data again, it can directly call it from the memory.

[0091] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0092] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In one embodiment, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0093] In one embodiment, the electronic device 100 may include a main screen and a secondary screen. Optionally, the size of the main screen is larger than that of the secondary screen, and the main screen may also be referred to as a large screen, and the secondary screen may also be referred to as a small screen. Optionally, the main screen and the secondary screen are located on different sides of the electronic device 100. For example, the main screen is located on the front of the electronic device 100, and the main screen may also be referred to as an inner screen, and the secondary screen is located on the back of the electronic device 100, and the secondary screen may also be referred to as an outer screen.

[0094] In one embodiment, the main screen of the electronic device 100 is a folding screen, and the secondary screen is not foldable. The physical state of the electronic device 100 may include an unfolded state, a bent state, and a folded state. When the electronic device 100 is in the unfolded state, the light-emitting surfaces of the main screen and the secondary screen face each other. At this time, the main screen may be in the unfolded state. When the electronic device 100 is in the bent state or the folded state, the main screen may be bent and divided into two display areas. The planes where the two display areas are located intersect. At this time, the secondary screen and the light-emitting surface of one of the display areas face each other. For examples of the main screen and the secondary screen, please refer to the above. Figure 1 and Figure 2 A first display screen 200 and a second display screen 300 are shown.

[0095] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0096] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and other factors. It can also optimize parameters such as exposure and color temperature of the captured scene. In one embodiment, the ISP can be located within camera 193.

[0097] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In one embodiment, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0098] In one embodiment, the electronic device 100 includes an inner screen and an outer screen. The inner screen and the outer screen can be located on different sides of the electronic device 100. The N cameras 193 of the electronic device 100 can be respectively set on different sides of the electronic device 100. Optionally, some cameras 193 (i.e., the upper Figures 1 to 4 The first camera 401 in the image is located on the same surface as the inner screen, and some cameras 193 (i.e. Figures 1 to 4 The second camera 402 in the inner screen is located on the same surface as the outer screen. The electronic device 100 can obtain the physical state of the electronic device 100 through the detection signals of the first camera 401 and / or the second camera 402. In some examples, when the electronic device 100 is in the unfolded state or the bent state, the first camera 401 located on the same surface as the inner screen obtains the user's facial information, and this state can be referred to as the inner screen use state; the second camera 402 located on the same surface as the outer screen cannot obtain the user's facial information, and this state can be referred to as the outer screen use state. When the inner screen is in use, for example, a third-party application in the electronic device 100 starts a video call, the first camera 401 can be called to make a video call, and the video call screen is displayed on the inner screen; when the electronic device switches from the inner screen use state to the external screen use state, the first camera 401 located on the same side as the inner screen cannot obtain the user's facial information, and the second camera 402 located on the same side as the external screen obtains the user's facial information. The electronic device 100 can determine that the current posture is that the secondary screen and the user's face are facing each other. For example, at this time, the third-party application in the electronic device 100 is still in a video call, and it switches from the first camera 401 to the second camera 402, calls the second camera 402 to make a video call, and displays the video call screen on the external screen to ensure uninterrupted video call.

[0099] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0100] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0101] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In one embodiment, pressure sensor 180A can be located on display screen 194. There are many types of pressure sensors 180A, such as resistive, inductive, and capacitive. A capacitive pressure sensor can include at least two parallel plates made of conductive material. When force acts on pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the intensity of the pressure based on this change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the touch intensity based on pressure sensor 180A. Electronic device 100 can also calculate the touch location based on the detection signal from pressure sensor 180A. In one embodiment, touch operations applied to the same touch location but with different touch intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, a command to view short messages is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to a short message application icon, a command to create a new short message is executed.

[0102] The touch sensor 180K is also referred to as a "touch device." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also referred to as a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In another embodiment, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a location different from that of the display screen 194.

[0103] In the present application, the pressure sensor 180A and / or the touch sensor 180K may be provided in the display screen 194. When the display screen 194 displays the user interface of the application, the pressure sensor 180A and / or the touch sensor 180K may detect user operations performed by the user on the user interface. In response to the user operations, the electronic device 100 may perform corresponding tasks based on the application. For example, if a user clicks on the avatar of a friend in a social application, the electronic device 100 may display the personal information posted by the friend through the social application on the display screen 194.

[0104] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In one embodiment, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. In one embodiment, the electronic device 100 can detect whether the bending angle of the folding screen of the electronic device 100 changes based on the detection signal of the gyroscope sensor 180B. In one embodiment, the electronic device 100 can detect whether the electronic device 100 is flipped based on the detection signal of the gyroscope sensor 180B. The gyroscope sensor 180B can also be used for shooting anti-shake, navigation, somatosensory game scenes, etc. Optionally, the gyroscope sensor 180B can be arranged on a circuit board of the electronic device 100.

[0105] The accelerometer 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device. In one embodiment, it can be used to detect whether the bending angle of the folding screen of the electronic device 100 has changed. In one embodiment, it can detect whether the electronic device 100 is flipped. It is not limited to this and can also be used for applications such as horizontal and vertical screen switching, pedometers, etc. Optionally, the accelerometer 180E can be set on the circuit board of the electronic device 100.

[0106] In one embodiment, the electronic device 100 may include multiple acceleration sensors 180E and / or multiple gyroscope sensors 180B. The electronic device 100 includes a foldable screen (e.g., an inner screen). When the electronic device 100 is in an unfolded state, the foldable screen may be in an unfolded state. When the electronic device 100 is in a bent state (also referred to as a half-folded state) and a folded state, the foldable screen may be bent and divided into two display areas, with the planes of the two display areas intersecting. The multiple acceleration sensors 180E and / or multiple gyroscope sensors 180B may be respectively disposed on circuit boards on the sides of the two display areas. The electronic device 100 may detect changes in the physical state of the electronic device 100, such as whether the bending angle has changed, based on the detection signals of the multiple acceleration sensors 180E and / or multiple gyroscope sensors 180B. In some examples, when the electronic device 100 determines that the physical state is an unfolded state or a bent state based on the detection signals, the electronic device 100 may determine that in the current posture, the inner screen is facing the user and the outer screen is not facing the user. This state may be referred to as the inner screen usage state. When the electronic device 100 determines that the physical state is a folded state based on the detection signal, the electronic device 100 can determine that in the current posture, the inner screen is not facing the user, and the outer screen is facing the user. This state can be referred to as the outer screen use state. In the inner screen use state, for example, if a third-party application in the electronic device 100 initiates a video call, the first camera 401 can be called to conduct the video call, and the video call image is displayed on the inner screen. When the electronic device switches from the inner screen use state to the outer screen use state, the first camera 401 located on the same side as the inner screen cannot obtain the user's facial information, and the second camera 402 located on the same side as the outer screen obtains the user's facial information. The electronic device 100 can determine that the current posture is that the secondary screen and the user's face are facing each other. For example, if the third-party application in the electronic device 100 is still in the video call, the first camera 401 is switched to the second camera 402, the second camera 402 is called to conduct the video call, and the video call image is displayed on the outer screen to ensure uninterrupted video call.

[0107] The air pressure sensor 180C is used to measure air pressure. The magnetic sensor 180D includes a Hall sensor, and the electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. The distance sensor 180F is used to measure distance. The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The fingerprint sensor 180H is used to collect fingerprints. The temperature sensor 180J is used to detect temperature. The ambient light sensor 180L is used to sense the brightness of ambient light.

[0108] The angle sensor 180M can obtain angle information and convert it into a usable electrical signal output. In one embodiment, the angle sensor 180M can be set in the display screen 194 to detect the bending angle of the folding screen. The processor 110 can determine the physical state of the electronic device 100 (e.g., unfolded state, bent state, or folded state) and whether the physical state of the electronic device 100 has changed based on the detection signal of the angle sensor 180M.

[0109] This application does not limit the specific type of sensor used to detect the physical state and motion posture of the electronic device 100.

[0110] Keys 190 include a power button, volume button, and other buttons. Keys 190 can be mechanical or touch-sensitive. Electronic device 100 can receive key inputs and generate key signal inputs related to user settings and function control of electronic device 100. Motor 191 can generate vibration prompts. Indicator 192 can be an indicator light that can be used to indicate charging status, battery level changes, messages, missed calls, notifications, and more. SIM card interface 195 is used to connect a SIM card.

[0111] The software system of an electronic device (such as a mobile phone) can adopt a layered architecture, a transaction-driven architecture, a micro-kernel architecture, a microservice architecture, or a cloud architecture. This application embodiment takes the Android system with a layered architecture as an example to illustrate the software architecture of a mobile phone. Figure 6 , Figure 6 This is a schematic diagram of the software architecture of the electronic device provided in an embodiment of the present application.

[0112] like Figure 6 As shown, the layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: application layer (application), application framework layer (frame), hardware abstraction layer (HAL), driver layer and hardware layer. Among them:

[0113] The application layer (application) may include a series of program packages. For example, the application may include third-party applications, and may also include camera, gallery and other applications. Among them, the third-party application includes an application that can have a camera function, that is, the third application can call the camera of the electronic device 100 to shoot, video call, etc. The camera application may include but is not limited to: UI module, photo module, gallery module and so on. The UI module may be a cameraUI module, which is mainly responsible for the human-computer interaction of the camera application, such as controlling the preview interface and the display of the preview screen therein, receiving and responding to user operations occurring in the preview interface. The photo module provides photo function, focus function, etc. The gallery module can be used to store photos taken by the user in the file system or a specific database of the electronic device for retrieval by applications such as the gallery.

[0114] The application framework layer (frame) provides application programming interfaces (APIs) and a programming framework for applications in the application layer. This primarily involves the camera framework, which includes camera access interfaces such as the camera service. It serves as a link between the upper and lower layers. It interacts with third-party applications through the application API and with the HAL through the HAL interface definition language (HIDL). The camera framework provides three data streams for third-party applications to choose from: a preview and callback data stream, a video stream, and a photo stream.

[0115] The application framework layer may also include a window manager, and third-party applications and gallery applications can display photos taken to users with the support of the window manager. It may also include a physical state management module that can determine the physical state of the electronic device 100, and further determine the usage state of the electronic device 100 based on the physical state of the electronic device 100. For example, if the electronic device 100 is in an unfolded state or a half-folded state, the electronic device 100 is in an inner screen usage state; if the electronic device is in a folded state, the electronic device 100 is in an outer screen usage state, and the physical state or usage state of the electronic device 100 is sent to a module that monitors the state, such as a camera service.

[0116] The Camera Service responds to application layer requests and provides session management, device management, and policy management. The Camera Service is an independent process that acts as a server, handling cross-process requests from application clients (such as third-party applications), performing certain operations internally, and then forwarding the request back to the HAL process, acting as the server. Therefore, the Camera Service is part of the camera pipeline, serving as a link between the upper and lower layers.

[0117] Session management is used to manage the lifecycle, parameter configuration, input and output management of camera devices, and other functions. It primarily involves camera preview sessions and camera capture sessions. A camera preview session is used to preview the camera image stream, while a camera capture session is used to capture photos or videos. It has a state callback that handles state changes, such as ready and start capturing. For example, before using the camera's preview, photo, or video functions, you must first create a camera session and complete the relevant configuration within the session.

[0118] Device management is used to manage and control camera hardware devices, including connecting to, initializing, configuring, and controlling camera hardware devices, as well as processing data streams from camera devices. The camera service coordinates application access to camera hardware devices through device management and is responsible for handling various operations related to camera hardware devices.

[0119] Policy management is used to manage image processing policies configured for third-party applications. Image processing policies include, but are not limited to, the first policy, the second policy, the third policy, and the video call policy. It should be understood that different third-party applications may have the same or different image processing policies, depending on the view controls used by the third-party application.

[0120] The receiving module is used to manage the whitelist of applications, update the physical state of the electronic device 100, register listeners, callback policies, etc. For example, the receiving module can monitor whether a third-party application has started the camera function through the listener according to the whitelist. When the receiving module receives the physical state of the electronic device 100 from the physical state management module of the system framework, or the usage status of the electronic device 100 related to the physical state, it will promptly update the physical state of the electronic device 100 and call back to the policy management.

[0121] The Hardware Abstraction Layer (HAL) is an interface layer between the application framework layer and the driver layer, providing a virtual hardware platform for the operating system. The HAL can include a camera hardware abstraction layer and a policy execution module.

[0122] The Camera Hardware Abstraction Layer (HAL) provides virtual hardware for Camera Device 1 (the first camera), Camera Device 2 (the second camera), and more camera devices. For example, the HAL runs as an independent process in a native service. It communicates with the camera service through a HIDL interface and issues actual operations for the first and second cameras of the camera device through a standard HAL interface. The independent process can be a Camera Provider.

[0123] The policy execution module stores a variety of image processing algorithms. For example, in an embodiment of the present application, the image processing algorithms may include: a video call connection algorithm, a first image processing algorithm, a second image processing algorithm, and a third image processing algorithm.

[0124] The video call continuation algorithm is used to ensure that the conversation path during the video call is not interrupted when switching from the internal screen usage state to the external screen usage state, or when switching from the external screen usage state to the internal screen usage state.

[0125] The first image processing algorithm is used to implement rotation and mirroring functions by setting transformation parameters. In the camera framework, the transformation parameters are mainly used to process 3D transformation and rotation transformation, and the specific parameters include the rotation angles of the X-axis, Y-axis, and Z-axis.

[0126] The second image processing algorithm is used to implement image rotation and mirroring functions in the output node. The output node includes nodes used for image processing in the HAL layer, such as the GPUnode. This node is used to perform HAL GPUNode operations such as rotate and flip. Among them, in image processing, the rotate operation generally refers to a symmetrical operation of rotating 2π / n counterclockwise around an axis, where n is a positive integer and is the axis of rotation. The flip operation flips a two-dimensional matrix, performing a horizontal flip, a vertical flip, or both.

[0127] The third image processing algorithm is used to implement image rotation and image mirroring functions through a video codec (MediaCodec). MediaCodec is a class provided by the system architecture for encoding and decoding. It implements encoding and decoding functions by accessing the underlying codec. The images processed are Joint Photographic Experts Group (JPEG) images. JPEG images include JPEG meta data, which are various tags and information embedded in the image. This information can include image attributes, camera device information, user settings, etc. Therefore, MediaCodec is used to set the image attributes in the JPEG image to implement image rotation and image mirroring functions.

[0128] For example, a third-party application obtains a JPEG image by adding the JPEG_ORIENTATION parameter to the created photo request. After receiving the above parameter, the camera HAL layer sends the image generated by the camera driver to the media codec for image processing, thereby obtaining a JPEG image.

[0129] The driver layer is the layer between hardware and software, and includes drivers for various hardware components. This layer can include camera drivers, digital signal processor drivers, video codec drivers, and graphics processor drivers. The camera driver drives the image sensors of one or more cameras to capture images and the image signal processor to pre-process them. The digital signal processor driver drives the digital signal processor to process images. The video codec driver drives the video codec to process images. The graphics processor driver drives the graphics processor to process images.

[0130] The hardware layer may include a camera, an image signal processor, a digital signal processor, a video codec, and a graphics processor. A camera may include one or more image sensors (e.g., image sensor 1, image sensor 2, etc.) of a camera (e.g., a first camera and a second camera). Optionally, a camera may also include a camera motor, a lens, a time-of-flight (TOF) sensor, and so on.

[0131] The following illustrates the workflow of electronic device software and hardware by combining a scenario where a third-party application calls a camera.

[0132] In an embodiment of the present application, when the touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the user operation into a raw input event (including information such as touch coordinates and the timestamp of the touch operation) and identifies the control corresponding to the input event. Taking the touch operation as a user operation of a camera control in a third-party application as an example, the third-party application calls the camera access interface of the application framework layer to enable the camera capability, and determines the policy configured for the third application and the folding state of the electronic device 100. Then, by calling the camera device in the camera hardware abstraction layer, such as camera device 1, an instruction to enable the camera capability is sent. The camera hardware abstraction layer sends the instruction to the camera driver of the driver layer. The camera driver can then enable the corresponding camera device (for example, the first camera). Then, the sensor in the first camera captures the image light signal and transmits the image light signal to the image signal processor for preprocessing to obtain a raw image stream. The raw image stream is then transmitted back to the hardware abstraction layer through the camera driver.

[0133] The hardware abstraction layer can send the RAW image stream to the policy execution module. Based on the GPU's support, the policy execution module in the hardware abstraction layer can process the raw image stream, such as by rotating or mirroring it, according to the policy configured for the third-party application and the folding state of electronic device 100, to produce a more effective image. The hardware abstraction layer returns this image data to the third-party application via the camera interface. The third-party application and the gallery application can then display the photos to the user with the support of the window manager.

[0134] Figures 7A-7D This is a set of user interfaces for third-party applications to call cameras, provided in an embodiment of the present application.

[0135] like Figure 7A As shown, at the first moment, when the physical state of the electronic device 100 is the unfolded state or the bent state, the electronic device 100 is in the inner screen use state, that is, the first display screen 200 is opposite to the user's face, and the user interface is displayed through the first display screen 200. Exemplarily, when the electronic device 100 is in the inner screen use state, a third-party application is started in response to a user operation (for example, clicking on the icon of a third-party application). In the third-party application, in response to a user operation (for example, clicking on the shooting control), the camera capability is started, and an instruction to start the camera capability is sent through the camera device 2 in the camera hardware abstraction layer, thereby starting the second camera in the hardware layer, and displaying the first user interface 61 on the first display screen 200. Generally speaking, the camera called defaults to the rear camera, that is, the second camera 402 on the display screen (that is, the second display screen 300) located on the back of the first display screen 200. Therefore, the first image displayed in the first user interface 61 is output by the second camera 402. At the second moment, the user folds the electronic device 100 so that the physical state of the electronic device 100 is a folded state, and the electronic device 100 is in an external screen use state, that is, the second display screen 300 is opposite to the user's face, and the second user interface 62 of the third-party application is displayed through the second display screen 300. At this time, the third-party application is still calling the second camera 402, and the third image displayed in the second user interface 62 is output by the second camera 402. Among them, the electronic device 100 draws and displays the user interface of the third-party application (including the first user interface 61 and the second user interface 62) through the first view control, thereby displaying the captured image in the user interface. The first view control includes surfaceview and textureview.

[0136] from Figure 7A It can be seen that when the external screen is in use, the installation angle of the second camera 402 has changed. In order to ensure the adaptability of the second user interface 62, the application framework layer and the hardware abstraction layer need to perform corresponding processing on the image output by the second camera 402.

[0137] Exemplarily, the physical state management module in the application framework layer can determine that the physical state of the electronic device 100 has changed from the sensor information reported by the underlying sensor, and then determine that the physical state of the electronic device 100 is a folded state, and send the physical state of the electronic device 100 to the receiving module in the camera service. The receiving module then sends the physical state of the electronic device 100 to the policy management, and the policy management determines the image processing policy for the third-party application based on the physical state of the electronic device 100 and the view control used by the third-party application. For example, the physical state of the electronic device 100 is a folded state, and the third-party application draws the user interface through the first view control, then the policy management determines that the policy configured for the third-party application is the first policy. The policy management sends the first policy to the device management, and the device management sends the first policy to the hardware abstraction layer through the buffer queue in a binder communication manner.

[0138] Exemplarily, the camera hardware abstraction layer in the hardware abstraction layer obtains the first policy through the buffer queue and sends the first policy to the policy execution module. The policy execution module stores a variety of algorithms. The policy execution module determines the first image processing algorithm according to the first policy, and determines the first parameter based on the first image processing algorithm, wherein the first parameter may specifically include one or more rotation angles of the X-axis, Y-axis, and Z-axis, and the first parameter may be a transform parameter. Then, the policy execution module sends the first parameter to the session management in the application framework layer. It can be understood that after the camera device 2 receives the first image reported by the hardware layer, it also reports it to the session management in the application framework layer.

[0139] Exemplarily, the session management in the application framework layer reports the first image and first parameter to the first view control via a buffer queue. The first view control draws the user interface of the third-party application based on the first parameter. Furthermore, the first view control rotates the first layer used to display the second image in the user interface based on the first parameter to generate a third image. The first parameter is used to adjust the rotation angle of the first layer.

[0140] For example, see Figure 8 , Figure 8 This is a schematic diagram of an image rotation provided by an embodiment of the present application. Figure 8 As shown, when the inner screen is in use, the second camera 402 outputs the first image. When the outer screen is in use, the second camera 402 is flipped 180 degrees, so the second image output by the second camera 402 is flipped 180 degrees relative to the first image. The first view control is used to draw the layer for displaying the second image in the user interface of the third-party application. Therefore, the first view control can rotate the first layer for displaying the first image 180 degrees according to the first parameter to obtain the third image. Then, the imaging direction of the third image displayed in the first image is also rotated 180 degrees relative to the second image. Finally, from Figure 7A It can be seen that the third image displayed in the second user interface 62 can adapt to the second display screen 300, that is, the third image is not inverted compared with the first image, and the imaging direction of the third image is consistent with the imaging direction of the first image.

[0141] For example, at the third moment ( Figure 7A (not shown), after the electronic device 100 switches from the external screen usage state to the internal screen usage state, the physical state management module in the application framework layer senses that the physical state of the electronic device 100 has changed, and the physical state is the unfolded state or the half-folded state. At this time, the electronic device 100 is in the internal screen usage state. Therefore, the application framework layer can pass the internal screen usage state to the hardware abstraction layer through the buffer queue, and the hardware abstraction layer determines that the electronic device 100 is in the internal screen usage state. Therefore, the hardware abstraction layer can report the sensor parameters of the second camera 402 to the system layer, and further report it to the third-party application. Therefore, the third-party application can obtain the sensor parameters of the second camera 402, and then select the corresponding rotation logic according to the sensor parameters to process the image captured by the second camera 402, thereby ensuring that the processed image can adapt to the current display state (that is, displayed on the first display screen 200 in the unfolded state). That is, when the electronic device 100 switches from the external screen usage state to the internal screen usage state, the image captured by the second camera 402 is no longer processed according to the first parameters, but the image captured by the second camera 402 is processed according to the sensor parameters reported by the bottom layer.

[0142] like Figure 7B As shown, at the first moment, the electronic device 100 is in the internal screen use state, and the first user interface of the third-party application is displayed on the first display screen 200. The first image displayed in the first user interface 61 is output by the second camera 402. At the second moment, the electronic device 100 is in the external screen use state, and the second user interface 62 of the third-party application is displayed through the second display screen 300. Among them, the electronic device 100 draws and displays the user interface of the third-party application (including the first user interface 61 and the second user interface 62) through the second view control, thereby displaying the captured image in the user interface. The second view control includes a view object created using the Open Graphics Library (OpenGL).

[0143] from Figure 7B It can be seen that when the external screen is in use, the installation angle of the second camera 402 has changed. In order to ensure the adaptability of the second user interface 62, the application framework layer and the hardware abstraction layer need to perform corresponding processing on the image output by the second camera 402.

[0144] Exemplarily, the physical state management module in the application framework layer can determine that the physical state of the electronic device 100 has changed from the sensor information reported by the underlying sensor, and then determine that the physical state of the electronic device 100 is a folded state, and send the physical state of the electronic device 100 to the receiving module in the camera service. The receiving module then sends the physical state of the electronic device 100 to the policy management, and the policy management determines the image processing policy for the third-party application based on the physical state of the electronic device 100 and the view control used by the third-party application. For example, the physical state of the electronic device 100 is a folded state, and the third-party application draws and displays the user interface through the second view control, then the policy management determines that the policy configured for the third-party application is the second policy. The policy management sends the second policy to the device management, and the device management sends the second policy to the hardware abstraction layer through the buffer queue in a binder communication manner.

[0145] Exemplarily, the camera hardware abstraction layer in the hardware abstraction layer obtains the second policy through the buffer queue, sends the second policy to the policy execution module, and also sends the first image reported by the hardware layer to the policy execution module. The policy execution module stores a plurality of algorithms, and the policy execution module determines the second image processing algorithm according to the second policy, and determines the second parameter based on the second image processing algorithm, wherein the second parameter may specifically include a parameter for performing a rotation operation and / or a parameter for performing a flip operation. Then, the policy execution module performs a rotation and flip operation on the second image with the support of the graphics processing unit (GPU) to obtain a third image. The second image is an image captured after the second camera 402 is flipped 180 degrees. For example, the image processor rotates the second image 180 degrees to obtain a third image. Then, the hardware abstraction layer returns the third image to the third-party application through the camera interface. Subsequently, the third-party application displays the third image in the second user interface 62 with the support of the second view control. Among them, the relevant descriptions of the first image, the second image and the third image can be referred to. Figure 8 , I will not go into details this time.

[0146] For example, at the third moment ( Figure 7B (not shown), after the electronic device 100 switches from the external screen usage state to the internal screen usage state, the physical state management module in the application framework layer senses that the physical state of the electronic device 100 has changed, and the physical state is the unfolded state or the half-folded state. At this time, the electronic device 100 is in the internal screen usage state. Therefore, the application framework layer can pass the internal screen usage state to the hardware abstraction layer through the buffer queue, and the hardware abstraction layer determines that the electronic device 100 is in the internal screen usage state. Therefore, the hardware abstraction layer can report the sensor parameters of the second camera 402 to the system layer, and further report it to the third-party application. Therefore, the third-party application can obtain the sensor parameters of the second camera 402, and then select the corresponding rotation logic according to the sensor parameters to process the image captured by the second camera 402, thereby ensuring that the processed image can adapt to the current display state (that is, displayed on the first display screen 200 in the unfolded state). That is, when the electronic device 100 switches from the external screen usage state to the internal screen usage state, the image captured by the second camera 402 is no longer processed according to the second parameters, but the image captured by the second camera 402 is processed according to the sensor parameters reported by the bottom layer.

[0147] like Figure 7C As shown, at the first moment, the electronic device 100 is in the internal screen use state, and the first user interface 61 of the third-party application is displayed on the first display screen 200, and the first image displayed in the first user interface 61 is output by the second camera 402. At the second moment, the electronic device 100 is in the external screen use state, and the second user interface 62 of the third-party application is displayed through the second display screen 300. Among them, the electronic device 100 draws and displays the user interface of the third-party application (including the first user interface 61 and the second user interface 62) through the second view control, thereby displaying the captured image in the user interface. The image in the third-party application is a JPEG image, that is, the first image displayed in the first user interface 61 is in JPEG format, and the second image displayed in the second user interface 62 is a JPEG image.

[0148] from Figure 7C It can be seen that when the external screen is in use, the installation angle of the second camera 402 has changed. In order to ensure the adaptability of the second user interface 62, the application framework layer and the hardware abstraction layer need to perform corresponding processing on the image output by the second camera 402.

[0149] Exemplarily, the physical state management module in the application framework layer can determine that the physical state of the electronic device 100 has changed from the sensor information reported by the underlying sensor, and then determine that the physical state of the electronic device 100 is a folded state, and send the physical state of the electronic device 100 to the receiving module in the camera service. The receiving module then sends the physical state of the electronic device 100 to the policy management, and the policy management determines the image processing strategy for the third-party application based on the physical state of the electronic device 100 and the JPEG image used by the third-party application. For example, the physical state of the electronic device 100 is a folded state, and the third-party application draws and displays the user interface through the second view control, then the policy management determines that the strategy configured for the third-party application is the third strategy. The policy management sends the third strategy to the device management, and the device management sends the third strategy to the hardware abstraction layer through the buffer queue in a binder communication manner.

[0150] Exemplarily, the camera hardware abstraction layer in the hardware abstraction layer obtains the third policy through the buffer queue, sends the third policy to the policy execution module, and also sends the second image reported by the hardware layer to the policy execution module. The policy execution module stores a variety of algorithms. The policy execution module determines the second image processing algorithm according to the third policy, and determines the third parameter based on the second image processing algorithm, wherein the third parameter may specifically include a JPEG parameter for performing a rotation operation, such as a JPEG_ORIENTATION parameter. Then, the policy execution module performs a rotation operation on the second image with the support of a video codec (media codec) to obtain a third image. The second image is an image captured by the second camera 402 after being flipped 180 degrees. For example, the media codec rotates the second image 180 degrees to obtain a third image. Then, the hardware abstraction layer returns the third image to the third-party application through the camera interface. Subsequently, the third-party application displays the third image in the second user interface 62 with the support of the window manager. For relevant descriptions of the first image, the second image, and the third image, please refer to Figure 8 , I will not go into details this time.

[0151] For example, at the third moment ( Figure 7C (not shown), after the electronic device 100 switches from the external screen usage state to the internal screen usage state, the physical state management module in the application framework layer senses that the physical state of the electronic device 100 has changed, and the physical state is the unfolded state or the half-folded state. At this time, the electronic device 100 is in the internal screen usage state. Therefore, the application framework layer can pass the internal screen usage state to the hardware abstraction layer through the buffer queue, and the hardware abstraction layer determines that the electronic device 100 is in the internal screen usage state. Therefore, the hardware abstraction layer can report the sensor parameters of the second camera 402 to the system layer, and further report it to the third-party application. Therefore, the third-party application can obtain the sensor parameters of the second camera 402, and then select the corresponding rotation logic according to the sensor parameters to process the image captured by the second camera 402, thereby ensuring that the processed image can adapt to the current display state (that is, displayed on the first display screen 200 in the unfolded state). That is, when the electronic device 100 switches from the external screen usage state to the internal screen usage state, the image captured by the second camera 402 is no longer processed according to the third parameter, but the image captured by the second camera 402 is processed according to the sensor parameters reported by the bottom layer.

[0152] like Figure 7D As shown, in the video call scenario, at the first moment, the electronic device 100 is in the internal screen use state, and the first user interface of the third-party application is displayed on the first display screen 200. The first user interface is used to display a first video screen 41, and the first video screen 41 is a first image stream captured by the first camera 401 (front camera). At the second moment, the electronic device 100 is in the external screen use state, and the second user interface of the third-party application is displayed through the second display screen 300. The second user interface user displays a second video screen 42, and the second video screen 42 is a picture captured by the second camera 402 (rear camera).

[0153] from Figure 7D It can be seen that when switching from the internal screen to the external screen, the camera used for the video call has changed. In order to ensure the continuity of the video call, the application framework layer and the hardware abstraction layer need to ensure that the current session call is not interrupted.

[0154] Exemplarily, the physical state management module in the application framework layer can determine that the physical state of the electronic device 100 has changed from the sensor information reported by the underlying sensor, and then determine that the physical state of the electronic device 100 is a folded state, and send the physical state of the electronic device 100 to the receiving module in the camera service. The receiving module then sends the physical state of the electronic device 100 to the policy management, and the policy management determines the image processing policy for the third-party application based on the physical state of the electronic device 100 and the video call scenario of the third-party application. For example, if the physical state of the electronic device 100 is a folded state and the third-party application is making a video call, the policy management determines that the policy configured for the third-party application is a video call policy. The policy management sends the video call policy to the device management, and the device management sends the video call policy to the hardware abstraction layer through the buffer queue in a binder communication manner.

[0155] For example, the camera hardware abstraction layer (HAL) in the hardware abstraction layer (HAL) obtains the video call policy from the buffer queue and sends it to the policy execution module. The policy execution module determines a video call connection algorithm based on the video call policy and, based on the algorithm, a first instruction. The first instruction may specifically include an instruction to shut down first camera 401 and start second camera 402. The policy execution module then sends the first instruction to the camera HAL. Based on the first instruction, the camera HAL shuts down camera device 1 and starts camera device 2. Consequently, camera device 1 shuts down first camera 401, while camera device 2 starts second camera 402, which begins capturing image data. The second image stream reported by the hardware layer to the HAL is captured by second camera 402. The HAL returns this second image stream to the third-party application via the camera interface. The third-party application can then display the second video image 42 to the user with the support of the window manager. It can be seen that when switching from the internal screen usage state to the external screen usage state, the session between the application layer, application framework layer and hardware abstraction layer is not interrupted. Instead, the application framework layer perceives the state change and restarts the configuration of the image stream. The third-party application is not aware of it, so the video call is not interrupted, improving the user experience.

[0156] For example, at the third moment ( Figure 7D (not shown), after the electronic device 100 switches from the external screen usage state to the internal screen usage state, the physical state management module in the application framework layer senses that the physical state of the electronic device 100 has changed, and the physical state is the unfolded state or the half-folded state. At this time, the electronic device 100 is in the internal screen usage state. Therefore, the application framework layer can pass the internal screen usage state to the hardware abstraction layer through the buffer queue, and the hardware abstraction layer determines that the electronic device 100 is in the internal screen usage state. Therefore, the hardware abstraction layer can issue an instruction to start the first camera 401 and turn off the second camera 402. Therefore, the electronic device 100 restarts the flow distribution, the first camera 401 starts to collect image data, and the hardware layer reports the image collected by the first camera 401 to the hardware abstraction layer.

[0157] It should be noted that Figure 7D The description is made using a vertically folding electronic device as an example. Of course, the description is also applicable to a horizontally folding electronic device, which will not be elaborated here.

[0158] It should be understood that each step in the above method embodiments provided herein can be implemented by hardware integrated logic circuits in a processor or by software instructions. The method steps disclosed in the embodiments of this application can be directly implemented as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0159] The present application also provides an electronic device, which may include a memory and a processor, wherein the memory may be used to store a computer program, and the processor may be used to call the computer program in the memory so that the electronic device executes the method in any one of the above embodiments.

[0160] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the method executed by the electronic device in any of the above embodiments.

[0161] In one possible design, the chip system further includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0162] The chip system can be composed of chips, or can include chips and other discrete devices.

[0163] Optionally, there may be one or more processors in the chip system. The processor may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0164] Optionally, the memory in the chip system may be one or more. The memory may be integrated with the processor or may be provided separately from the processor, which is not limited in the embodiments of the present application. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or provided on different chips. The embodiments of the present application do not specifically limit the type of memory or the configuration of the memory and the processor.

[0165] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0166] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which, when executed, enables a computer to execute the method executed by the electronic device in any of the above embodiments.

[0167] The present application also provides a computer-readable storage medium storing a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device in any of the above embodiments.

[0168] The various implementation modes of this application can be combined arbitrarily to achieve different technical effects.

[0169] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described herein are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive).

[0170] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0171] In short, the above description is only an embodiment of the technical solution of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made based on the disclosure of the present invention should be included in the scope of protection of the present invention.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. An image display method, characterized in that: Used in electronic equipment, including: When the inner screen is in use, a first user interface of the third-party application is displayed on the first display screen, wherein the first user interface is used to display a first video call screen, and the first video call screen includes an image captured by the first camera; When the external screen is in use, the second user interface of the third-party application is displayed through the second display screen, wherein the second user interface is used to display a second video call screen, and the human video call includes an image captured by the second camera, the first display screen and the second display screen are located on different sides of the electronic device, the first camera is located on the first display screen, and the second camera is located on the second display screen.

2. The method according to claim 1, characterized in that The method further comprises: When the inner screen is in use, the first user interface is further used to display a first image captured by the second camera; When the external screen is in use, the second user interface is also used to display a third image, wherein the third image is an image obtained by rotating the second image, and the second image includes the image captured by the second camera when the external screen is in use.

3. The method according to claim 2, characterized in that The method further comprises: An imaging direction of the first image differs from an imaging direction of the second image by 180 degrees, and an imaging direction of the third image is the same as that of the first image.

4. The method according to any one of claims 1 to 3, characterized in that The method of displaying the second user interface of the third-party application on the second display screen in the external screen use state includes: Detecting a physical state of the electronic device to determine whether the electronic device is switched from the inner screen usage state to the outer screen usage state; Determining a processing strategy corresponding to the third-party application; determining the second user interface according to the processing strategy; The second user interface is displayed through the second display screen.

5. The method according to claim 4, characterized in that Determining a processing strategy corresponding to the third-party application includes: Determine a processing strategy corresponding to the operating scenario based on an application scenario of the third application, wherein the application scenario includes one or more of the following: drawing the second user interface through a first view control, drawing the second user interface through a second view control, displaying a Joint Photographic Experts Group JPEG image in the second user interface, and video calls.

6. The method according to claim 5, characterized in that The method further comprises: In the scenario where the second user interface is drawn through the first view control, the processing strategy includes: setting deformation parameters to realize image rotation.

7. The method according to claim 5, characterized in that The method further comprises: In the scenario where the second user interface is drawn through the second view control, the processing strategy includes: implementing image rotation at the output node.

8. The method according to claim 5, characterized in that The method further comprises: In a scenario where a JPEG image is displayed in the second user interface, the processing strategy includes: implementing image rotation through video encoding and decoding.

9. The method according to claim 5, characterized in that The method further comprises: In the video call scenario, the processing strategy includes: generating a first instruction, where the first instruction includes turning off the first camera and starting the second camera.

10. An electronic device, characterized in that: The electronic device includes: one or more processors; a memory; wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method as described in any one of claims 1-9.

11. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute the method as described in any one of claims 1-9.

12. A computer program product comprising instructions, characterized in that When the computer program product is run on an electronic device, the electronic device is enabled to perform the method according to any one of claims 1 to 9.

13. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 9.

Images