Display method and electronic equipment

Abstract

The invention provides a display method and electronic equipment, and relates to the technical field of terminals. The electronic equipment obtains the surrounding environment information and displays the surrounding environment information in combination with the surrounding environment, so that the electronic equipment can be integrated into the surrounding environment, and the visual effect is enriched. The electronic equipment is the electronic equipment with a foldable screen, and the method comprises the steps that after the electronic equipment obtains a panoramic image, view angle conversion is carried out on the panoramic image according to the position of a target user in the panoramic image, and a target view angle panoramic image is generated. Wherein the first view angle corresponding to the target view angle panoramic image is matched with the second view angle of the target user. And then, the electronic equipment generates a perspective image according to the target view angle panoramic image. And displaying the perspective image on a target sub-screen corresponding to the target user.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of terminal technology, and in particular to a display method and an electronic device. Background Art

[0002] With the development of terminal technology, electronic devices with foldable screens are widely used in users' daily lives, providing users with a display experience of a larger display screen while ensuring portability. In addition, electronic devices with foldable screens can also realize the function of setting tables. For example, an electronic device with a foldable screen is folded and placed on a desktop, so that the electronic device with a foldable screen can display specific images such as clocks and calendars through a sub-screen. In this way, electronic devices with foldable screens can become decorative tools in the environment, and users can directly view the flat or three-dimensional images displayed on the sub-screen.

[0003] However, since the electronic device with a foldable screen is a real physical device and cannot be integrated into the surrounding environment, the user cannot ignore the electronic device with a foldable screen when viewing the image, resulting in a single visual effect. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a display method and an electronic device. The technical solution provided by the present application enables the electronic device to obtain surrounding environment information and display it in combination with the surrounding environment, so that the electronic device can be integrated into the surrounding environment, enrich the visual effect, and enhance the user experience.

[0005] In order to achieve the above technical objectives, this application provides the following technical solutions:

[0006] In a first aspect, a display method is provided, which is applied to an electronic device, and the electronic device is an electronic device with a foldable screen. The method includes: after the electronic device acquires a panoramic image, according to the position of a target user in the panoramic image, the panoramic image is converted to generate a panoramic image of a target perspective. Among them, the first perspective corresponding to the panoramic image of the target perspective matches the second perspective of the target user. Afterwards, the electronic device generates a perspective image according to the panoramic image of the target perspective. And the perspective image is displayed on the target sub-screen corresponding to the target user.

[0007] Among them, the panoramic image is the image obtained by the electronic device with itself as the center point, so the first perspective corresponding to the panoramic image is the observation perspective of the electronic device. The subsequent folding screen display needs to be presented to the target user for viewing, so the picture displayed on the folding screen needs to be the picture corresponding to the second perspective of the target user. That is, after the electronic device generates the panoramic image, it needs to convert the panoramic image from the perspective of the electronic device into a panoramic image from the target perspective corresponding to the second perspective of the target user.

[0008] The perspective image is a two-dimensional image displayed on the screen of the electronic device, which is obtained by converting the panoramic image of the target perspective seen by the target user through the target sub-screen. By displaying the perspective image, the electronic device is integrated into the surrounding environment to achieve a more realistic panoramic image display and enhance the holographic display effect.

[0009] In this way, the electronic device synthesizes the panoramic image and combines it with the location of the target user to determine the image of the area blocked by the electronic device from the perspective of the target user, and displays this part of the image on the target sub-screen. In this way, visually, the user can see the image of the part blocked by the electronic device through the electronic device, achieving a perspective effect and enriching the user's visual experience.

[0010] According to the first aspect, the screen of the electronic device includes at least three sub-screens, and the at least three sub-screens are folded to enclose a closed area, and the electronic device is in a stationary state.

[0011] According to the first aspect, or any implementation of the first aspect above, obtaining a panoramic image includes: collecting an initial image through cameras configured on at least three sub-screens, and generating a panoramic image based on the initial image.

[0012] According to the first aspect, or any implementation of the first aspect above, at least one camera is configured on each of the at least three sub-screens.

[0013] For example, the panoramic image is taken by taking a group of full-view photos of the scene with the electronic device as the center point, and seamlessly splicing two adjacent flat photos with a certain overlap area through software to combine them into a spherical image that surrounds the whole area. When watching, the panoramic image can look around the scene in six directions: front, back, left, right, top, and bottom, and can also provide users with an immersive interactive experience by zooming in or out of the scene. Then, the panoramic image generated by the electronic device can reflect the environmental information around the electronic device.

[0014] According to the first aspect, or any implementation of the first aspect above, generating a panoramic image based on the initial images includes: generating a panoramic image when at least one of the initial images includes a target user. Alternatively, generating a panoramic image after the position of the electronic device is moved.

[0015] According to the first aspect, or any implementation of the first aspect above, after acquiring the initial image, the method also includes: in the case where the initial image includes a human-shaped object, identifying the object corresponding to the target user based on the initial image, the object corresponding to the target user including any of the following items: a human-shaped object matching the host image, a single human-shaped image included in the initial image, a human-shaped object located in the middle position of multiple human-shaped objects included in the initial image, and a human-shaped object with the highest image quality among the human-shaped objects included in the initial image.

[0016] In some examples, panoramic images mainly serve the user's visual senses. If the electronic device does not detect a human figure, that is, there is no one around the electronic device, even if the electronic device generates a panoramic image, in an unmanned environment, the sensory experience generated is not perceived by anyone, resulting in unnecessary power consumption. Therefore, the electronic device only needs to generate a panoramic image after detecting a human figure.

[0017] In other examples, if the position of the electronic device does not change, the probability of the panoramic image changing is low. Therefore, after the electronic device first captures the initial image and generates the panoramic image, it can stop generating the panoramic image. Later, after the electronic device detects a position change, it can trigger the generation of the panoramic image again.

[0018] According to the first aspect, or any implementation of the first aspect above, the panoramic image is converted into a perspective according to the position of the target user to generate a panoramic image of the target perspective, including: determining a second perspective according to the positional relationship between the target user and the electronic device. A first coordinate system is established with the position of the target user as the origin. The panoramic image is converted into a coordinate system according to the first coordinate system and the second perspective to generate a panoramic image of the target perspective.

[0019] In some examples, after determining the sight angle of the target user, the electronic device may determine the location of the target user as the origin of the panoramic coordinate system. Based on the origin of the panoramic coordinate system, the electronic device performs a coordinate system transformation on the panoramic image, thereby obtaining a panoramic image from the perspective of the target user. Optionally, the origin of the panoramic coordinate system determined based on the sight angle of the target user is, for example, the midpoint of the user's eyes. Optionally, the transformation of the coordinate system includes transformations such as rotation, translation, and scaling.

[0020] In some examples, in a panoramic image, the further away from the center of the line of sight, the greater the image distortion. Therefore, during the image conversion process, the electronic device uses the line of sight angle of the target user as the center line of the line of sight.

[0021] According to the first aspect, or any implementation of the first aspect above, generating a perspective image according to a target perspective panoramic image includes: determining a contour of an electronic device according to the target perspective panoramic image, generating a partial panoramic image according to the contour of the electronic device, and generating a perspective image according to the partial panoramic image.

[0022] According to the first aspect, or any implementation of the first aspect above, determining the outline of the electronic device according to the target perspective panoramic image includes: determining the image acquisition blind area of ​​the electronic device according to the initial image corresponding to the panoramic image. Determining the outline of the electronic device according to the image acquisition blind area and the target perspective panoramic image.

[0023] In some embodiments, during the process of capturing the initial image, due to the angular range of the camera's field of view, there are blind spots in the image capture range of the cameras on adjacent sub-screens. For example, the cameras are located on different sub-screens, and after the display screen is folded, multiple sub-screens are connected to form a polygonal structure, so the positions of multiple cameras do not overlap. Therefore, there is at least a blind spot in the folding angle area of ​​the display screen's rotation axis position, and a blind spot inside the formed polygon. Therefore, the electronic device determines the screen outline of the electronic device in the panoramic image through the blind spot position. That is, the electronic device determines the spatial position of each sub-screen, and then determines the screen coordinates of the electronic device in the panoramic image. In this way, the electronic device can subsequently fill the outline of the electronic device in the blind spot position, so that the electronic device is integrated into the surrounding environment, so as to achieve a more realistic panoramic image display and enhance the holographic display effect.

[0024] According to the first aspect, or any implementation of the first aspect above, the partial panoramic image includes multiple partial panoramic images corresponding to multiple sub-screens of the electronic device, and the perspective image includes multiple perspective images corresponding to the multiple partial panoramic images.

[0025] In some embodiments, in the table mode, the electronic device is in a folded state, and there is an angle between the multiple sub-screens. In the process of converting the three-dimensional partial panoramic image into a two-dimensional perspective image, the three-dimensional image is projected onto a two-dimensional plane. Therefore, in the process of perspective projection of the electronic device, multiple partial panoramic images corresponding to the multiple sub-screens are determined based on each sub-screen of the electronic device. Afterwards, the electronic device performs projection transformation based on the multiple partial panoramic images to obtain multiple perspective images.

[0026] In this way, after generating a perspective image, the electronic device can display the perspective image on the sub-screen, so that the user can watch the perspective image with a perspective effect on the sub-screen, and the electronic device is integrated into the surrounding environment to enrich the visual effect.

[0027] According to the first aspect, or any implementation of the first aspect above, the method further includes: obtaining a preset model. According to the position of the target user and the preset model, generating a visual effect image corresponding to the preset model, the visual effect image being an image corresponding to the second perspective.

[0028] In some scenarios, the sub-screen of the electronic device can display specific images such as clocks and calendars, so that the electronic device can be used as a decoration or tool in the environment. Then, the electronic device can also realize the function of the decoration or tool through image display. In some examples, after the electronic device realizes the perspective effect of the folding screen through the above method, a three-dimensional object can be obtained, and the three-dimensional object can be superimposed and displayed on the basis of the perspective effect. Through the coordination of the three-dimensional object and the perspective effect, when the target user looks at the location of the electronic device, the three-dimensional object is placed at the location of the electronic device.

[0029] In some embodiments, after obtaining the preset model, the electronic device can obtain the coordinate system corresponding to the preset model, which is a default coordinate system (such as a world coordinate system). Therefore, the electronic device can estimate the camera posture of the preset model according to the user's position and the electronic device's position in the target perspective panoramic image. This enables the electronic device to image the preset model and obtain a visual effect image from the user's perspective.

[0030] According to the first aspect, or any implementation of the first aspect above, displaying the perspective image on the target sub-screen corresponding to the target user includes: generating a mixed image according to the visual effect image and the perspective image. Displaying the mixed image on the target sub-screen.

[0031] According to the first aspect, or any implementation method of the first aspect above, a mixed image is generated based on the visual effect image and the perspective image, including: taking the visual effect image as the foreground image, taking the perspective image as the rearview image, and performing a mixed calculation on the visual effect image and the perspective image through a preset algorithm to generate a mixed image.

[0032] According to the first aspect, or any implementation of the first aspect above, the visual effect image includes multiple visual effect images corresponding to multiple sub-screens of the electronic device respectively, and the mixed image displayed on the target sub-screen is a target mixed image generated based on the target visual effect image corresponding to the target sub-screen and the target perspective image.

[0033] In this way, by fusing the perspective image with the preset model, the electronic device can be integrated into the surrounding environment, and the final visual effect image brings a richer visual experience to the user.

[0034] According to the first aspect, or any implementation of the first aspect above, before displaying the perspective image on the target sub-screen corresponding to the target user, the method further includes: determining the target sub-screen corresponding to the target user in the panoramic image according to the spatial position relationship between the electronic device and the target user in the panoramic image. Alternatively, determining that the target initial image of the target user is acquired in the initial image corresponding to the panoramic image. The sub-screen where the camera that captured the target initial image is located is determined as the target sub-screen.

[0035] In some examples, the electronic device simulates the holographic effect of the target user viewing the electronic device based on the position of the target user. Then, the electronic device can determine the target sub-screen according to the sub-screen that the target user can "see". For example, the electronic device determines one or more sub-screens with the smallest distance between the center point of the sub-screen of the electronic device and the position of the target user as the target sub-screen through the target perspective panoramic image.

[0036] In other examples, the sub-screen that the target user can see is the target sub-screen. Then, if the target user can see the sub-screen, the camera on the corresponding sub-screen should also be able to capture the image of the target user. Therefore, after determining the target user, the electronic device determines to recognize the initial image of the target user. Then, the electronic device can determine the camera that captures the initial image, and determine that the sub-screen where the camera is located is the target sub-screen.

[0037] According to the first aspect, or any implementation of the first aspect above, displaying a perspective image on a target sub-screen corresponding to a target user includes: determining a target perspective image corresponding to the target sub-screen from among multiple perspective images acquired according to the panoramic image, and displaying the target perspective image on the target sub-screen.

[0038] Among them, the target sub-screen is part or all of the multiple sub-screens of the electronic device. For example, after the electronic device generates multiple perspective images corresponding to the multiple sub-screens, it displays the corresponding perspective images on the multiple sub-screens respectively. For another example, after the electronic device generates multiple perspective images corresponding to the multiple sub-screens, it only displays the corresponding perspective images on the target sub-screen that the target user can see, thereby reducing display power consumption. For another example, the electronic device only generates perspective images corresponding to the sub-screens that the target user can see, and then displays the corresponding perspective images on these sub-screens.

[0039] According to the first aspect, or any implementation of the first aspect above, after acquiring the panoramic image, the method further includes: determining a position of the target user in the panoramic image based on the initial image.

[0040] In some examples, the user views the surrounding environment through eyes. Then, the position of the target user is, for example, the position of one eye or both eyes of the target user. For example, the electronic device uses the center point position of both eyes of the target user as the position of the target user for subsequent perspective conversion of the panoramic image.

[0041] In this way, by determining the position of the target user in the panoramic image, the electronic device can perform a perspective conversion of the panoramic image in a subsequent process.

[0042] In a second aspect, an electronic device is provided. The electronic device includes: a processor, a memory and a display screen, the memory and the display screen are coupled to the processor, the memory is used to store computer program code, the computer program code includes computer instructions, when the processor reads the computer instructions from the memory, the electronic device executes: after acquiring the panoramic image, according to the position of the target user in the panoramic image, the panoramic image is converted to generate a target perspective panoramic image. Among them, the first perspective corresponding to the target perspective panoramic image matches the second perspective of the target user. Afterwards, a perspective image is generated according to the target perspective panoramic image. And the perspective image is displayed on the target sub-screen corresponding to the target user.

[0043] According to the second aspect, the screen of the electronic device includes at least three sub-screens, and the at least three sub-screens are folded to enclose a closed area, and the electronic device is in a stationary state.

[0044] According to the second aspect, or any implementation of the second aspect, obtaining a panoramic image includes: collecting an initial image through cameras configured on at least three sub-screens, and generating a panoramic image based on the initial image.

[0045] According to the second aspect, or any implementation of the second aspect above, generating a panoramic image based on the initial images includes: generating a panoramic image when at least one of the initial images includes the target user. Alternatively, generating a panoramic image after the position of the electronic device is moved.

[0046] According to the second aspect, or any implementation of the second aspect, at least one camera is configured on each of the at least three sub-screens.

[0047] According to the second aspect, or any implementation of the second aspect above, the panoramic image is transformed in perspective according to the position of the target user to generate a panoramic image of the target perspective, including: determining a second perspective according to the positional relationship between the target user and the electronic device. A first coordinate system is established with the position of the target user as the origin. According to the first coordinate system and the second perspective, the panoramic image is transformed in coordinate system to generate a panoramic image of the target perspective.

[0048] According to the second aspect, or any implementation of the second aspect above, generating a perspective image according to a target perspective panoramic image includes: determining a contour of an electronic device according to the target perspective panoramic image, generating a partial panoramic image according to the contour of the electronic device, and generating a perspective image according to the partial panoramic image.

[0049] According to the second aspect, or any implementation of the second aspect above, determining the outline of the electronic device according to the target perspective panoramic image includes: determining the image acquisition blind area of ​​the electronic device according to the initial image corresponding to the panoramic image. Determining the outline of the electronic device according to the image acquisition blind area and the target perspective panoramic image.

[0050] According to the second aspect, or any implementation of the second aspect above, the partial panoramic image includes multiple partial panoramic images corresponding to multiple sub-screens of the electronic device, and the perspective image includes multiple perspective images corresponding to the multiple partial panoramic images.

[0051] According to the second aspect, or any implementation of the second aspect above, when the processor reads the computer instruction from the memory, the electronic device is also caused to execute: obtaining a preset model. Based on the position of the target user and the preset model, a visual effect image corresponding to the preset model is generated, and the visual effect image is an image corresponding to the second perspective.

[0052] According to the second aspect, or any implementation of the second aspect above, displaying the perspective image on the target sub-screen corresponding to the target user includes: generating a mixed image according to the visual effect image and the perspective image. Displaying the mixed image on the target sub-screen.

[0053] According to the second aspect, or any implementation method of the second aspect above, a mixed image is generated based on the visual effect image and the perspective image, including: taking the visual effect image as the foreground image, taking the perspective image as the rearview image, and performing a mixed calculation on the visual effect image and the perspective image through a preset algorithm to generate a mixed image.

[0054] According to the second aspect, or any implementation of the second aspect above, the visual effect image includes multiple visual effect images corresponding to multiple sub-screens of the electronic device respectively, and the mixed image displayed on the target sub-screen is a target mixed image generated based on the target visual effect image corresponding to the target sub-screen and the target perspective image.

[0055] According to the second aspect, or any implementation of the second aspect above, when the processor reads the computer instruction from the memory, the electronic device is also caused to execute: determining the target sub-screen corresponding to the target user in the panoramic image according to the spatial position relationship between the electronic device and the target user in the panoramic image. Alternatively, determining that the target initial image of the target user is obtained in the initial image corresponding to the panoramic image. The sub-screen where the camera that captured the target initial image is located is determined as the target sub-screen.

[0056] According to the second aspect, or any implementation of the second aspect, displaying a perspective image on a target sub-screen corresponding to a target user includes: determining a target perspective image corresponding to the target sub-screen from among multiple perspective images acquired according to the panoramic image, and displaying the target perspective image on the target sub-screen.

[0057] According to the second aspect, or any implementation of the second aspect above, when the processor reads computer instructions from the memory, it also enables the electronic device to execute: in the case where the initial image includes a human-shaped object, based on the initial image, identify the object corresponding to the target user, and the object corresponding to the target user includes any of the following items: a human-shaped object matching the host image, a single human-shaped image included in the initial image, a human-shaped object located in the middle position among the multiple human-shaped objects included in the initial image, and a human-shaped object with the highest image quality among the human-shaped objects included in the initial image.

[0058] According to the second aspect, or any implementation of the second aspect above, when the processor reads the computer instruction from the memory, it also enables the electronic device to execute: determining the position of the target user in the panoramic image based on the initial image.

[0059] In a third aspect, an electronic device is provided, which has the function of implementing the display method described in the first aspect and any possible implementation thereof. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0060] In a fourth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program (also referred to as an instruction or code), and when the computer program is executed by an electronic device, the electronic device executes the method of the first aspect or any one of the implementations of the first aspect.

[0061] According to a fifth aspect, a computer program product is provided. When the computer program product is run on an electronic device, the electronic device executes the method according to the first aspect or any one of the implementations of the first aspect.

[0062] According to a sixth aspect, a circuit system is provided, the circuit system comprising a processing circuit, wherein the processing circuit is configured to execute the method of the first aspect or any one of the embodiments of the first aspect.

[0063] In the seventh aspect, a chip system is provided, comprising at least one processor and at least one interface circuit, wherein the at least one interface circuit is used to perform transceiver functions and send instructions to the at least one processor, and when the at least one processor executes the instructions, the at least one processor executes the method of the first aspect or any one of the embodiments of the first aspect.

[0064] The technical effects of the aforementioned aspects can be referenced to each other and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A schematic diagram of a folding scenario of an electronic device with a foldable screen provided in an embodiment of the present application;

[0066] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0067] Figure 3 Schematic diagram of the display method provided in the embodiment of the present application Figure 1 ;

[0068] Figure 4 A schematic diagram of a table setting mode scenario provided in an embodiment of the present application;

[0069] Figure 5 A schematic diagram of the camera field of view provided in an embodiment of the present application;

[0070] Figure 6 A schematic diagram of the target user's perspective provided in an embodiment of the present application;

[0071] Figure 7 A schematic diagram of the outline of an electronic device provided in an embodiment of the present application;

[0072] Figure 8 A schematic diagram of a target display area determination scenario provided in an embodiment of the present application;

[0073] Fig. 9 Schematic diagram of the display method provided in the embodiment of the present application Figure 2 ;

[0074] Fig.10 Schematic diagram of the display method provided in the embodiment of the present application Figure 3 ;

[0075] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] The technical solutions in the embodiments of the present application are described below in conjunction with the accompanying drawings in the embodiments of the present application. Wherein, in the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to include expressions such as "one or more", unless there is a clear indication to the contrary in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two).

[0077] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. The term "connection" includes direct connection and indirect connection, unless otherwise specified. "First" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0078] In the embodiments of the present application, the words "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0079] At present, electronic devices (such as mobile phones, computers, etc.) can be roughly divided into foldable screen electronic devices (also called flexible screen electronic devices, foldable screen electronic devices) and non-foldable screen electronic devices based on the display screen. For electronic devices with foldable screens, when their applications are started, the corresponding operating system displays the started applications in full screen or split screen form. The electronic device with a foldable screen includes multiple sub-screens, and after the electronic device with a foldable screen is folded, a certain angle is formed between the multiple sub-screens.

[0080] In some examples, after the multiple sub-screens of the foldable screen electronic device are folded, they can be combined to form a closed area. Figure 1 As shown, the electronic device with a foldable screen includes three sub-screens, namely sub-screen a, sub-screen b and sub-screen c. The user folds sub-screen b and sub-screen a toward sub-screen c to obtain a triangular prism.

[0081] In some embodiments, the electronic device with a foldable screen can realize a table setting function. For example, the electronic device with a foldable screen can be folded and placed on a desktop, so that the electronic device with a foldable screen can display specific images such as a clock and a calendar through a sub-screen. In this way, the electronic device with a foldable screen can become a decorative tool in the environment, and the user can directly view the flat or three-dimensional image displayed on the sub-screen.

[0082] Among them, electronic devices with foldable screens can enhance visual effects through three-dimensional (3D) display functions.

[0083] In some examples, electronic devices with foldable screens use naked-eye 3D technology to achieve 3D display effects on sub-screens.

[0084] Optionally, the current mainstream naked eye 3D technical means include slit liquid crystal grating, cylindrical lens, directional light source, and active backlight. Among them, the technical principle of the slit liquid crystal grating is to add a slit grating in front of the sub-screen, so that when the image seen by the left eye is displayed on the sub-screen during the image display process, the opaque stripes will block the right eye; similarly, when the image seen by the right eye is displayed on the sub-screen, the opaque stripes will block the left eye. In this way, by separating the visible images of the left eye and the right eye, the user can see the 3D picture. The technical principle of the cylindrical lens is to project the pixel points corresponding to the left and right eyes respectively into the left and right eyes through the refraction principle of the lens, so as to achieve image separation and enable the user to see the 3D picture. The technical principle of the directional light source is to accurately control the two sets of screens to project images to the left and right eyes respectively, so that the user can see the 3D picture. The technical principle of the active backlight is to use an optical microstructure to form a backlight module, and the light beam emitted by the backlight can adjust the direction under the control of electronic devices, so that the user can see the 3D picture.

[0085] In other examples, based on the above-mentioned naked-eye 3D technology principles, electronic devices with foldable screens adjust the display content so that two sub-screens that are not on the same plane can be visually seen as a flat image in a better viewing field, thereby achieving a 3D display effect.

[0086] For example, the display content of one of the sub-screens is processed based on a preset field of view, and based on the original display content, it is distorted based on the principle of line of sight projection, so that the distorted display content and the display content on the other sub-screen appear to be on the same plane.

[0087] However, since the electronic device with a foldable screen is a real physical device, it cannot be integrated into the surrounding environment to achieve a perspective effect. When viewing an image, the user cannot ignore the electronic device with a foldable screen, resulting in a monotonous visual effect.

[0088] In addition, when 3D display is achieved by folding the sub-screen, the sub-screen angle is usually fixed and the display content needs to be customized. Once the angle changes, the 3D display effect disappears and cannot be flexibly applied to the folding screen.

[0089] Therefore, the embodiment of the present application provides a display method, whereby an electronic device with a foldable screen obtains information about the surrounding environment and displays it in combination with the surrounding environment, so that the electronic device with a foldable screen can be integrated into the surrounding environment, and while achieving a 3D display effect, it enriches the visual effect and enhances the user experience. Thus, by combining the display of the surrounding environment, a perspective effect is achieved, presenting the user with an illusion of a three-dimensional effect similar to a holographic display.

[0090] In addition, since electronic devices with foldable screens can obtain information about the surrounding environment, the 3D display effect will not be affected when the angle of the sub-screen changes.

[0091] In some embodiments, the display method provided in the embodiments of the present application can be applied to an electronic device 100. The electronic device 100 can be, for example, a folding screen mobile phone, a non-folding screen mobile phone, a tablet computer, a laptop computer, a smart screen, a wearable device, a personal digital assistant (PDA), an artificial intelligence (AI) device, or other terminal devices. The operating system installed in the electronic device 100 includes but is not limited to Or other operating systems. This application does not limit the specific type of the electronic device 100 or the installed operating system.

[0092] In some examples, the electronic device 100 is an electronic device with a foldable screen, and the display screen of the electronic device 100 includes multiple sub-screens.

[0093] Figure 2 A schematic structural diagram of an electronic device 100 is shown.

[0094] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0095] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0096] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0097] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0098] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0099] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0100] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0101] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0102] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0103] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0104] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0105] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0106] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0107] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0108] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device, or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0109] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0110] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0111] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0112] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be manufactured using a liquid crystal display (LCD), such as an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini-led, Micro-led, Micro-oled, quantum dot light emitting diodes (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0113] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0114] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0115] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0116] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110. The electronic device 100 can use the audio module 170, such as music playing, recording, etc. The audio module 170 can include a speaker, a receiver, a microphone, a headphone interface, and an application processor to implement audio functions.

[0117] The sensor module 180 may include a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0118] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0119] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. Motor 191 can also correspond to different vibration feedback effects for touch operations on different areas of display screen 194. Different application scenarios (for example: time reminder, receiving information, alarm clock, game, etc.) can also correspond to different vibration feedback effects.

[0120] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0121] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting or removing the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1.

[0122] Figure 3 This is a flow chart of a display method provided in an embodiment of the present application. It should be noted that this method does not Figure 3 The specific order described below is a limitation, and it should be understood that in other embodiments, the order of some steps in the method can be interchanged according to actual needs, or some steps can be omitted or deleted. The method includes the following steps:

[0123] S301: The electronic device acquires multiple initial images and generates a panoramic image according to the multiple initial images.

[0124] In some embodiments, the electronic device is an electronic device with a foldable screen, and the display screen of the electronic device includes at least two sub-screens.

[0125] In some examples, after the electronic device display screen is folded inward or outward, an angle is formed between adjacent sub-screens in the plurality of sub-screens. In this way, the user can place the folded electronic device vertically on the desktop. For example, Figure 4 In the scenario shown, user 001 folds mobile phone 002 so that the three sub-screens form a certain angle and places it vertically on the desktop.

[0126] In some examples, after the electronic device display screen is folded inward or outward, multiple sub-screens can be combined into a relatively closed area. Figure 4 In the illustrated scenario, after the mobile phone 002 is folded, the three sub-screens are combined to form a relatively closed triangular prism. The following takes the electronic device as a double-folding screen electronic device, and the display screen includes three sub-screens as an example to introduce the specific implementation of the display method provided in the embodiment of the present application.

[0127] In some examples, cameras may be respectively configured on multiple sub-screens of the electronic device, and the multiple cameras may respectively perform image acquisition. In this way, the electronic device may acquire multiple initial images through the multiple cameras.

[0128] For example, Figure 5 As shown, the electronic device display screen includes a sub-screen 51, a sub-screen 52, and a sub-screen 53. Among them, the sub-screen 51 is configured with a camera 511, the sub-screen 52 is configured with a camera 521, and the sub-screen 53 is configured with a camera 531. That is, the electronic device is configured with three cameras. The electronic device can collect three initial images through these three cameras.

[0129] Optionally, the camera can be located at any position on the sub-screen, for example, the camera is located at the middle position of the side of the sub-screen that is perpendicular to the folding side.

[0130] In some embodiments, the electronic device is configured with a table setting mode. For example, in response to a user operation, the electronic device displays a setting interface. In response to the user's operation on the setting interface, the electronic device can turn on the table setting mode. Alternatively, the user turns on the table setting mode of the electronic device through other quick gestures, voice, etc.

[0131] In some examples, after the electronic device turns on the table mode, it detects that the user folds the display screen so that adjacent sub-screens are at a certain angle and then place it on a plane, and determines that the user instructs to run the table mode. Then, the electronic device can call multiple cameras located on multiple sub-screens to collect multiple initial images.

[0132] It should be understood that the table placement mode is only an exemplary name, and the above-mentioned placement state of the electronic device can also be described by other preset modes, and the embodiments of the present application are not limited to this.

[0133] In some embodiments, after acquiring multiple initial images, the electronic device may synthesize the multiple initial images to generate a panoramic image. The field of view angles of multiple cameras of the electronic device overlap at least partially, so that the same object in the physical world will appear simultaneously in the initial images captured by the cameras of adjacent sub-screens. In this way, the electronic device can acquire a panoramic image based on the overlapping initial images.

[0134] Optionally, the panoramic image is taken by taking a group of full-view photos of the scene with the electronic device as the center point, and seamlessly splicing two adjacent flat photos with a certain overlap area through software to combine them into a spherical image that surrounds the whole area. When watching, the panoramic image can look around the scene in six directions: front, back, left, right, top, and bottom, and can also provide users with an immersive interactive experience by zooming in or out of the scene. Then, the panoramic image generated by the electronic device can reflect the environmental information around the electronic device.

[0135] For example, Figure 5 The double-folding folding screen electronic device shown has a field of view angle of 145° for the cameras on the three sub-screens, and is located in the top middle of the sub-screens. In response to user operation, after the electronic device is folded, the three sub-screens are combined to form a relatively closed triangular prism. In this case, the electronic device obtains multiple initial images through multiple cameras, and the initial images obtained by adjacent sub-screens overlap (that is, the same object exists in different initial images). Afterwards, the electronic device processes the multiple initial images according to a preset algorithm to synthesize a panoramic image.

[0136] Optionally, in the process of synthesizing a panoramic image, the electronic device determines a unified spatial coordinate system, which can be defined as a panoramic coordinate system. Afterwards, the electronic device converts the multiple initial images based on the spatial coordinate system through a preset algorithm to generate a panoramic image, which is a 360° panoramic image centered on the electronic device. The process of generating a panoramic image based on a preset algorithm can refer to the prior art, and the embodiments of the present application will not be repeated here.

[0137] Optionally, the center of the electronic device is, for example, the geometric center of a relatively closed three-dimensional space enclosed by the sub-screens, for example, the geometric center of a relatively closed triangular prism enclosed by three sub-screens of the electronic device.

[0138] In some embodiments, during the panoramic image generation process, the electronic device needs to perform a large amount of calculations through a preset algorithm, which consumes a large amount of power. Therefore, after acquiring multiple initial images, the electronic device may not generate a panoramic image temporarily. Instead, after determining that the preset conditions are currently met, the electronic device generates a panoramic image.

[0139] In some examples, the preset condition includes that the electronic device detects a human figure. Among them, the panoramic image mainly serves the user's visual senses. Then if the electronic device does not detect a human figure, that is, there is no one around the electronic device. At this time, even if the electronic device generates a panoramic image, in an unmanned environment, the sensory experience generated is not perceived by anyone, resulting in unnecessary power consumption waste. Therefore, the electronic device only needs to generate a panoramic image after detecting a human figure.

[0140] Optionally, the electronic device detects the surrounding environment through an infrared sensor or other sensors to detect whether there is a human figure. Alternatively, the electronic device determines whether there is a human figure around by performing image recognition on the initial image.

[0141] In other examples, the preset condition includes the electronic device determining a position change. If the electronic device does not change its position, the probability of the panoramic image changing is low. Therefore, after the electronic device first captures the initial image and generates the panoramic image, it may stop generating the panoramic image. Subsequently, after the electronic device detects a position change, it may trigger the generation of the panoramic image again.

[0142] Optionally, the electronic device determines whether the position of the electronic device changes by using an acceleration sensor or other sensors.

[0143] In some embodiments, due to the limited angular range of the camera's field of view, there is a blind spot in the image acquisition range of the camera on the adjacent sub-screen, and the image in the blind spot cannot be captured by the camera. Figure 5As shown, the viewing angles of the cameras on the three sub-screens are 145°, and there are blind spots between adjacent cameras. For example, there is a blind spot 54 between the viewing angles of camera 511 and camera 521.

[0144] Optionally, the blind spot can be reduced by increasing the field of view. Figure 5 In the scenario shown, if the camera's field of view is 175°, the impact of the blind spots between the three cameras can be basically ignored.

[0145] Optionally, the blind area can be reduced by reducing the width of the sub-screen. Figure 5 In the scenario shown, when the field of view of the camera is constant, the wider the width of the sub-screen is, the larger the blind area between adjacent cameras is. Therefore, the blind area can be reduced by reducing the width of the sub-screen.

[0146] It should be understood that in some scenarios, after the electronic device is folded, multiple sub-screens cannot form a relatively closed area, or the electronic device is an electronic device with a non-folding screen. Then, in this case, the panoramic image generated by the electronic device based on the collected multi-channel images may be of poor quality. However, the display method provided in the embodiment of the present application can still be applied to achieve the display of a holographic effect. That is, the embodiment of the present application does not limit whether the electronic device is an electronic device with a foldable screen, and the number of sub-screens of the electronic device with a foldable screen.

[0147] S302: The electronic device determines a target user according to multiple initial images.

[0148] The target user is a user viewing an image presented by the electronic device. Optionally, the target user is, for example, the owner of the device, or one or more users in the surrounding environment.

[0149] In some embodiments, the panoramic image generated by the electronic device is an image generated from a perspective of observing the surrounding environment with the electronic device as the center. However, in the table mode, the electronic device is placed on other planes instead of being held in the hands of the user, that is, the current panoramic image perspective is not the same as the user perspective. Then, if the panoramic image is simply displayed on the electronic device, the electronic device cannot provide the user with an image viewing experience that meets the user's perspective observation needs. Therefore, the electronic device needs to determine the target user, and then determine the position of the target user, so as to determine the perspective of the target user based on the position of the target user, and convert the current panoramic image into a panoramic image viewed from the perspective of the target user. That is, the electronic device needs to convert the perspective of the panoramic image based on the target user.

[0150] In some embodiments, after the electronic device acquires multiple initial images, it can identify whether there is a user in the multiple initial images through image recognition technology. For example, the electronic device identifies whether there is a face in multiple initial images acquired at the same time based on features such as face, nose, eyes, and mouth.

[0151] In some examples, the electronic device determines the presence of faces in multiple initial images through image recognition. Afterwards, the electronic device may further perform facial feature recognition to determine whether the recognized faces include the face of the owner of the machine. If so, the user corresponding to the face of the owner of the machine may be determined as the target user. Optionally, when the electronic device is started for the first time, or when the user account is logged in, or when the table setting mode is turned on, the electronic device may collect the face image of the owner of the machine. In this way, during the target user detection process, the electronic device can determine whether the face of the owner of the machine is detected based on the face image of the owner of the machine. Optionally, the electronic device can also determine whether the owner of the machine is recognized based on multiple indicators such as morphology and gait.

[0152] In some examples, the electronic device does not recognize the owner. In this case, if the electronic device detects only one facial image, the user corresponding to the facial image can be determined as the target user. If the electronic device detects multiple facial images, the electronic device can evaluate and determine the target user based on conditions such as the orientation and quality of the multiple facial images. For example, if there is a facial image located in the middle area of ​​all facial images, the electronic device can determine the user corresponding to the facial image as the target user. For another example, if there is no facial image located in the middle area, the electronic device can determine the user corresponding to the facial image with the highest quality as the target user by judging the quality of multiple facial images. Optionally, the electronic device can evaluate the quality of the facial image based on multiple preset reference dimensions such as facial clarity, facial area, and whether the distance between the eyes is in a reasonable range.

[0153] In some examples, the owner is present in the physical environment and is captured by the camera in the initial image. However, because the owner is far away from the electronic device or the owner's face is blocked by other users or objects, the electronic device fails to recognize the owner in the initial image during the image recognition process. In this case, the electronic device can treat the owner as a general user, that is, the electronic device determines that the owner is not recognized at this time.

[0154] In some examples, when a user is in the overlapping area of ​​the viewing angles of multiple cameras, the user may be captured by multiple cameras at the same time. Then, in the process of identifying the target user based on the initial image, the target user determined by the electronic device based on the initial images identified by different cameras may be different. In this case, the electronic device first determines whether the owner is identified. If the owner is identified, the owner is determined as the target user. If the owner is not identified, or multiple cameras identify the owner, the electronic device can determine the user corresponding to the highest quality facial image as the target user based on the facial image quality of the target user corresponding to the multiple initial images.

[0155] It should be understood that the user may move, so during the process of capturing the initial image, the electronic device may determine whether the target user has changed based on the captured multiple frames of images.

[0156] S303: The electronic device determines the position of the target user according to the initial image, and converts the panoramic image into a panoramic image from a target perspective according to the position of the target user.

[0157] In some embodiments, after the electronic device recognizes the target user based on the initial image, it can locate the position of the target user in the panoramic image, and the target user position is the position of the target user relative to the electronic device. Afterwards, the electronic device can convert the panoramic image from the perspective of the electronic device into a panoramic image from the perspective of the target user through perspective conversion. Thus, in the subsequent process of displaying an image on the electronic device, it can be realized that the displayed image is an image of the environment around the electronic device viewed from the perspective of the target user.

[0158] For example, Figure 5 As shown, the electronic device is a double-folding screen electronic device, and the cameras on the three sub-screens capture one initial image respectively. After the electronic device performs image recognition on the three acquired initial images, the target user can be acquired. Afterwards, the electronic device can calculate the position of the target user relative to the electronic device based on the target user.

[0159] In some examples, the electronic device detects the position of the two eyes of the target user relative to the camera of the electronic device based on an image ranging algorithm. The camera for measuring the position is the camera corresponding to the initial image of the target user. The image ranging algorithm includes a ranging algorithm such as an RGB pixel depth recognition algorithm and a time of flight (TOF) ranging algorithm. For example, Figure 6 As shown, the electronic device detects the target user through camera C, and identifies the positions of the target user's eyes A and B relative to camera C through an image ranging algorithm. Afterwards, the electronic device constructs a triangle based on the camera position and the positions of the target user's two eyes, and calculates the target user's sight angle based on the constructed triangle. For example, Figure 6 As shown, the electronic device constructs a triangle through the position of camera C, the position of eye A, and the position of eye B. Afterwards, the electronic device connects eye A and eye B to obtain straight line L1, connects the center point of straight line L1 with the position of camera C to obtain straight line L2, and makes a line perpendicular to the sub-screen plane where the current camera C is located through the position of camera C to obtain straight line L3. Then, the electronic device can determine that the angle α between straight line L2 and straight line L3 is the user's line of sight.

[0160] Based on the above principle, the electronic device captures multiple frames of images through camera C and performs image distance measurement based on the multiple frames of images to obtain the spatial coordinates of the target user's eyes A and B. Thus, a spatial triangle is constructed based on the spatial coordinates of camera C, eyes A and eyes B. The triangle of the top view angle of the spatial triangle is Figure 6 The triangle shown. Then, the electronic device calculates the spatial coordinates of the midpoint D of the line connecting eye A and eye B, and uses the spatial coordinate position of point D to represent the position of the target user. After calculating the spatial coordinates of point D, the electronic device can calculate the angle between line segment CD and the three coordinate axes of space based on the spatial coordinates of point C and point D. This angle is the sight angle of the target user.

[0161] In some examples, after determining the sight angle of the target user, the electronic device may determine the location of the target user as the origin of the panoramic coordinate system. Based on the origin of the panoramic coordinate system, the electronic device performs coordinate system conversion on the panoramic image to obtain a panoramic image from the perspective of the target user. Optionally, the origin of the panoramic coordinate system determined based on the sight angle of the target user is, for example, the midpoint of the user's eyes, such as Figure 6 Point D is shown. Optionally, the transformation of the coordinate system includes transformations such as rotation, translation and scaling.

[0162] In some examples, in a panoramic image, the further away from the center of the line of sight, the greater the image distortion. Therefore, during the image conversion process, the electronic device uses the line of sight angle of the target user as the center line of the line of sight. For example, Figure 6 As shown, the straight line L2 is the sight line of the target user's eyes. In this way, the sight line angle of the target user is used as the center line of the sight line, so that the electronic device is located on the center line of the sight line to reduce possible distortion of the image near the electronic device.

[0163] Exemplarily, after acquiring a panoramic image through multiple initial images, the electronic device determines the coordinate system of the panoramic image obtained from the perspective of the electronic device. Afterwards, the electronic device determines the target user to obtain the coordinate system from the perspective of the target user. Afterwards, the electronic device can convert each coordinate (x1, y1, z1) in the panoramic image into the coordinate (x2, y2, z2) of the user's perspective based on the conversion relationship between the two coordinate systems. For example, a conversion matrix T is pre-configured in the electronic device, and the electronic device implements the perspective conversion of the panoramic image through the conversion matrix, such as [x2, y2, z2, w2] = T[x1, y1, z1, 1]. Among them, w2 represents the orientation in the coordinate system after the transformation. In this way, the electronic device implements the coordinate system conversion of each coordinate point in the panoramic image through the conversion matrix, and then realizes the conversion of the panoramic image into a panoramic image from the user's perspective. That is, the panoramic image is converted into a panoramic image from the target perspective.

[0164] Optionally, the electronic device is configured with a conversion matrix T, and the conversion matrix T can be adaptively updated during the operation of the electronic device. For example, the electronic device adaptively updates the conversion matrix T based on user feedback on the effect of the electronic device displaying images in the table setting mode, parameter settings (such as depth of field settings, etc.), user habits, etc., so that the subsequent display effect better meets the user's needs. For another example, the electronic device obtains feedback data learned by other electronic devices from the cloud server, thereby realizing the correction of the conversion matrix T based on big data.

[0165] Based on the above principle, after obtaining the panoramic image, the electronic device can transform the coordinate system based on the location of the target user to obtain a new panoramic image observed from the location of the target user. After obtaining the new panoramic image, the electronic device completes the perspective conversion of the panoramic image, realizing the conversion from viewing the panoramic image from the electronic device to viewing the panoramic image by the target user.

[0166] In this way, the electronic device can change the observation position (i.e., the coordinate origin of the panoramic coordinate system when synthesizing the panoramic image) and the observation angle in the panoramic image synthesized based on the multiple initial images to present the panoramic image from different positions and angles. The two-dimensional image finally presented on the screen of the electronic device will be different after the coordinate system transformation. For example, some objects disappear from the presented two-dimensional image, some new objects are added, and the degree of distortion of the same object changes before and after the coordinate system transformation. It should be understood that the closer to the edge of the two-dimensional image, the greater the degree of distortion.

[0167] S304: The electronic device generates a perspective image according to the panoramic image from the target viewing angle.

[0168] In some embodiments, as described in step S301 above, during the process of capturing the initial image, due to the angular range of the camera's field of view, there are blind spots in the image capture range of the cameras on adjacent sub-screens. For example, the cameras are located on different sub-screens, and after the display screen is folded, the first positions of the multiple sub-screens are connected to form a polygonal structure, and the positions of the multiple cameras do not overlap. Therefore, there is at least a blind spot in the folding angle area of ​​the display screen's rotation axis, and a blind spot inside the formed polygon. Figure 5 As shown, there is a blind area 54 at the folding angle position, and a blind area 55 inside the triangular prism formed by the sub-screens. The blind area cannot be captured by the camera, and thus cannot be displayed in the initial image, and further cannot be displayed in the panoramic image. That is, the panoramic images before and after the perspective conversion do not include the images in the blind area.

[0169] In some examples, the electronic device determines the screen outline of the electronic device in the panoramic image through the blind spot position. That is, the electronic device determines the spatial position of each sub-screen, and then determines the screen coordinates of the electronic device in the panoramic image. In this way, the electronic device can fill the outline of the electronic device in the blind spot position, so that the electronic device is integrated into the surrounding environment, so as to achieve a more realistic panoramic image display and improve the holographic display effect.

[0170] It should be understood that although determining the outline of the electronic device based on the position of the blind area will enlarge the screen of the electronic device to a certain extent, since the blind area is small, this effect can be ignored.

[0171] For example, Figure 5 As shown, during the panoramic image synthesis process, the electronic device can determine the initial images taken by adjacent cameras based on the overlapping relationship of the initial images, and then determine the overlapping parts in the two initial images taken by adjacent cameras to perform image synthesis. However, due to the existence of blind spots, during the panoramic image synthesis process, the multiple initial images cannot completely overlap at one point, but can only overlap to three overlapping line segments. Among them, these three overlapping line segments correspond to the intersection lines of the three adjacent sub-screens when the electronic device is in the folded state. These three intersection lines can determine a three-dimensional spatial structure, such as Figure 5 The sub-screens of the double-fold folding screen electronic device shown are combined to form a triangular prism (or if it is a three-fold folding screen electronic device, the three-dimensional space structure is a quadrangular prism). Afterwards, the electronic device can determine the three-dimensional space structure as corresponding to the spatial outline of the electronic device, determine the side of the three-dimensional space structure as corresponding to the sub-screen of the electronic device, and determine the edge of the three-dimensional space structure as corresponding to the rotation axis crease or screen edge at the junction of the sub-screens. Then, when the electronic device performs a panoramic image viewing angle conversion (i.e., coordinate conversion), the electronic device also converts the coordinates of its screen outline.

[0172] In this way, after the screen model of the electronic device is filled in the blind area, the electronic device can determine the partial panoramic image that the target user sees through the sub-screen of the electronic device according to the position of the target user. Figure 4 In the scenario shown, the sub-screen of the mobile phone 002 needs to display a two-dimensional image corresponding to the partial panoramic view that the user 001 can see through the sub-screen of the mobile phone 002, so as to integrate the mobile phone 002 into the surrounding environment.

[0173] In some embodiments, after the electronic device determines the outline of the electronic device based on the target perspective panoramic image and generates a partial panoramic image viewed by the user through the electronic device, the electronic device can generate a two-dimensional perspective image based on the partial panoramic image. The perspective image is a two-dimensional image displayed on the screen of the electronic device after the partial panoramic image viewed by the target user through the target sub-screen is converted.

[0174] For example, Figure 7 As shown, after the double-folding screen electronic device is folded, sub-screen a, sub-screen b and sub-screen c are combined to form a relatively closed triangular prism, in which the display content on the target sub-screen a and the target sub-screen b can be seen from the perspective of user A. Then, based on the perspective of user A, the electronic device can obtain a virtual outline of the electronic device (such as determined based on the blind spot), and a partial panoramic image can be seen through the target sub-screen a and the target sub-screen b. The partial panoramic image is a stereoscopic image with three-dimensional coordinate attributes. Therefore, the electronic device needs to convert the partial panoramic image into a two-dimensional image, and when it is displayed on the target sub-screen a and the target sub-screen b, it has the same visual effect as the partial panoramic image.

[0175] In some examples, after acquiring the partial panoramic image, the electronic device acquires a projection image corresponding to the sub-screen based on the position of the sub-screen, and the projection image is a perspective image corresponding to the sub-screen.

[0176] Exemplarily, the perspective projection matrix is ​​defined as The perspective projection matrix P is used to project points in the three-dimensional space onto a plane, and has the same perspective effect as the three-dimensional view. Parameters A and B are parameters related to the viewing plane. Parameters A and B are preset parameters in the electronic device and can determine the effect of perspective projection.

[0177] Among them, in computer graphics and computer vision, the perspective projection matrix is ​​often used in applications such as three-dimensional rendering and virtual reality. The specific content of the perspective projection matrix can refer to the prior art, and the embodiments of the present application do not limit this.

[0178] In some examples, after acquiring a partial panoramic image, the electronic device performs perspective projection transformation on the three-dimensional coordinate points (x, y, z) in the partial panoramic image based on the perspective projection matrix P, through the matrix multiplication shown in the following formula (1), and outputs the two-dimensional coordinate points (x', y', z', w'). Wherein, w' is a scaling factor used to normalize the projection points.

[0179]

[0180] Optionally, according to the rule of matrix multiplication, the electronic device calculates the coordinates of the projection point (x', y', z', w'). Afterwards, the electronic device performs normalization processing to obtain the final perspective projection point coordinates (xp, yp, zp). Among them, xp = x' / w'; yp = y' / w'; zp = z' / w'.

[0181] In this way, the electronic device uses the above method to transform points in the three-dimensional space onto a two-dimensional viewing plane through perspective projection, thereby producing a perspective effect.

[0182] For example, Figure 7 As shown, after the electronic device determines the partial panoramic image according to the electronic device outline, it uses perspective projection transformation to transform the partial panoramic image onto the screen planes on the target sub-screen a and the target sub-screen b to obtain a perspective image.

[0183] In some embodiments, in the table mode, the electronic device is in a folded state, and there is an angle between the multiple sub-screens. In the process of converting the three-dimensional partial panoramic image into a two-dimensional perspective image, the three-dimensional image is projected onto a two-dimensional plane. Therefore, in the process of perspective projection of the electronic device, multiple partial panoramic images corresponding to the multiple sub-screens are determined based on each sub-screen of the electronic device. Afterwards, the electronic device performs projection transformation based on the multiple partial panoramic images to obtain multiple perspective images.

[0184] S305: The electronic device determines a target display area, and displays a perspective image in the target display area.

[0185] In some embodiments, after generating a perspective image, the electronic device may display the perspective image on a sub-screen, so that the user can view the perspective image with a perspective effect on the sub-screen, and the electronic device is integrated into the surrounding environment to enrich the visual effect.

[0186] In some embodiments, after generating multiple perspective images, the electronic device determines a target display area according to the user's position, and the target display area is a sub-screen that needs to display the perspective image. Afterwards, the electronic device selects a perspective image corresponding to the target display area from the multiple perspective images based on the target display area, and displays the selected perspective image through the target display area.

[0187] In some examples, the display method provided by the embodiments of the present application simulates the holographic effect of the target user viewing the electronic device based on the position of the target user. Then, the electronic device can determine the target display area according to the sub-screen that the target user can "see". For example, the electronic device uses the target perspective panoramic image to determine one or more sub-screens with the smallest distance between the center point of the sub-screen of the electronic device and the position of the target user as the target display area. That is, the electronic device analyzes the panoramic image, determines the target sub-screen in the panoramic image according to the spatial position relationship between the user and the electronic device, and determines the sub-screen in the physical space corresponding to the target sub-screen as the target display area.

[0188] Optionally, the location of the target user is determined based on the center point of the user's eyes, and the location of the electronic device is determined based on the geometric center point of a relatively closed three-dimensional space enclosed by the sub-screens. Therefore, there is a height difference between the location of the target user and the location of the electronic device. Then, the electronic device can connect the center point of the location of the target user and the center point of the location of the electronic device, and project the connection line to the horizontal plane where the center point of the location of the electronic device is located. In addition, the horizontal cross section through the center point of the location of the electronic device intersects with the vertex of the electronic device outline, and multiple straight lines Ln can be obtained. After that, the electronic device determines the angles between the projection line and the multiple straight lines Ln respectively, and determines the minimum angle. If the minimum angle is greater than a preset value, the electronic device determines the closest target sub-screen as the target display area; if the minimum angle is less than or equal to the preset value, the electronic device determines the two closest target sub-screens as the target display area. It should be understood that the number of straight lines Ln is related to the number of contour edges of the non-horizontal plane of the three-dimensional space after the sub-screens are enclosed. If the sub-screens of the double-folding folding screen electronic device are enclosed into a relatively closed triangular prism, the electronic device can obtain three straight lines Ln accordingly.

[0189] Optionally, the preset value can be determined according to experiments, model calculations, etc., and preset in the electronic device.

[0190] For example, Figure 8 As shown in (a), the sub-screens a, b, and c of the double-folding folding screen electronic device are combined to form a relatively closed triangular prism, and the electronic device detects the target user A. Then, if Figure 8As shown in (b), the electronic device connects the center point of the location of user A with the center point 81 of the location of the electronic device (such as the geometric center of the triangular prism), and determines the horizontal plane of the center point 81 of the location of the electronic device, such as the horizontal plane where the center point 81, point 82, point 83, and point 84 are located. Then, the electronic device determines Figure 8 The projection line shown in (b). In addition, the horizontal cross section of the electronic device passing through the center point 81 of the electronic device intersects with the vertex 82 of the electronic device outline to obtain a straight line L1, the horizontal cross section passing through the center point 81 of the electronic device intersects with the vertex 83 of the electronic device outline to obtain a straight line L2, and the horizontal cross section passing through the center point 81 of the electronic device intersects with the vertex 84 of the electronic device outline to obtain a straight line L3. The electronic device determines the angles between the projection line and the straight line L1, the straight line L2, and the straight line L3, respectively, and determines that the angle β between the projection line and the straight line L1 is the minimum angle. If the electronic device determines that the angle β is less than the preset value. Then, the electronic device can determine the two closest target sub-screens as the target display area. That is, the electronic device determines sub-screen a and sub-screen b as the target display area.

[0191] In other examples, the sub-screen that the target user can see is the target display area. Then, if the target user can see the sub-screen, the camera on the corresponding sub-screen should also be able to capture the image of the target user. Therefore, after determining the target user, the electronic device determines to recognize the initial image of the target user. Then, the electronic device can determine the camera that captured the initial image, and determine that the sub-screen where the camera is located is the target display area. That is, the electronic device can also directly determine the target display area based on the multiple initial images captured by the camera.

[0192] In some embodiments, after determining the target display area, the electronic device displays the perspective image corresponding to the target display area (i.e., the target sub-screen) among the acquired multiple perspective images on the target display area. Optionally, other display areas other than the target display area may display the corresponding perspective image, or may not display the perspective image (e.g., a black screen, etc.).

[0193] In other embodiments, the electronic device may also first determine the target display area based on the initial image or the panoramic image. Then, in the above step S303, when the electronic device converts the panoramic image into a target perspective panoramic image, it may directly generate a corresponding target perspective panoramic image according to the target display area. Furthermore, in the above step S304, the electronic device may directly perform perspective projection transformation on the target perspective panoramic image to generate a perspective image that needs to be displayed on the target display area. In step S305, the electronic device directly displays the perspective image on the target display area.

[0194] Alternatively, in the above step S304, after the electronic device obtains the target perspective panoramic image, it can perform perspective projection transformation on the sub-screen as the target display area according to the target display area to generate a perspective image. In step S305, the electronic device directly displays the perspective image on the target display area.

[0195] In this way, the electronic device synthesizes the panoramic image and combines it with the location of the target user to determine the image of the area blocked by the folding screen electronic device from the perspective of the target user, and displays this part of the image in the target display area. In this way, visually, the user can see the image of the part blocked by the folding screen electronic device through the folding screen electronic device, achieving a perspective effect and enriching the user's visual experience in the folding screen electronic device stand mode scene.

[0196] In some scenarios, in the table mode, the sub-screen of the electronic device can display specific images such as clocks and calendars, so that the electronic device can be used as a decoration or tool in the environment. Then, the electronic device can also realize the function of the decoration or tool through image display. In some examples, after the electronic device realizes the perspective effect of the folding screen through the above method, a three-dimensional object can be obtained, and the three-dimensional object can be superimposed and displayed on the basis of the perspective effect. Through the coordination of the three-dimensional object and the perspective effect, when the target user looks at the location of the electronic device, the three-dimensional object is placed at the location of the electronic device.

[0197] Optionally, Fig. 9 This is a flow chart of another display method provided by an embodiment of the present application. After the above step S305, step S901 and step S902 may also be included.

[0198] S901. The electronic device obtains a 3D model, and determines a visual effect image according to the 3D model and a target user position.

[0199] In some embodiments, one or more 3D models are stored in the electronic device. It should be understood that the electronic device may also store a 2D image. That is, the electronic device may also determine a visual effect image based on the 2D image and the target user position, so that the subsequent electronic device displays a mixed effect image corresponding to the 2D image.

[0200] Optionally, the electronic device needs to display the superimposed 3D model by default in the table setting mode. Alternatively, the electronic device determines that the superimposed 3D model needs to be displayed according to user settings.

[0201] Optionally, the electronic device overlays and displays a default 3D model. Alternatively, the electronic device displays the 3D model selected by the user according to the user's selection operation.

[0202] Optionally, the 3D model can be a static 3D model or a dynamically changing 3D model. For example, the 3D model is a dynamically demonstrated fish tank model. For another example, the 3D model is a clock model.

[0203] In some embodiments, after acquiring the 3D model, the electronic device can acquire the coordinate system corresponding to the 3D model, which is a default coordinate system (such as a world coordinate system). Therefore, the electronic device can estimate the camera posture of the 3D model according to the user's position and the electronic device's position in the target perspective panoramic image. This enables the electronic device to image the 3D model and obtain a visual effect image from the user's perspective.

[0204] Optionally, during the camera pose estimation process, the electronic device virtualizes a camera so that the imaging perspective of the camera when imaging the 3D model is the same as the user perspective when the target user views the electronic device. That is, if the 3D model is placed at the location of the electronic device, the location of the camera is the location of the target user, and the direction of the camera is consistent with the line of sight of the target user.

[0205] In some embodiments, in the process of determining the visual effect graph, the electronic device may generate corresponding multiple visual effect graphs for multiple sub-screens of the electronic device respectively. Alternatively, the electronic device may first determine the target display area, and generate corresponding visual effect graphs according to the sub-screens corresponding to the target display area.

[0206] S902: The electronic device generates a mixed image according to the visual effect image and the perspective image, and displays the mixed image in a target display area.

[0207] In some embodiments, after obtaining the perspective image and the visual effect image corresponding to the perspective image, the electronic device can perform a mixed calculation on the visual effect image and the corresponding perspective image through a preset algorithm to generate a mixed image. Afterwards, the electronic device can display the mixed image in the target display area. Among them, when the electronic device displays the mixed image, it can achieve the stereoscopic visual effect of the 3D model while retaining the perspective effect. In addition, the perspective effect is used to enhance the visual fusion of the 3D model and the current physical space to enhance the stereoscopic visual effect.

[0208] In some embodiments, the electronic device may generate a mixed image based on the visual effect image and the perspective image corresponding to the target display area, and display the mixed image in the target display area. Optionally, the non-target display area may generate a corresponding mixed image, or may not generate a corresponding mixed image. That is, the electronic device may display a mixed image, or may not display a mixed image in the non-target display area.

[0209] In some embodiments, the electronic device uses the visual effect image as a foreground image and the perspective image as a background image during the process of generating the mixed image. Afterwards, the electronic device performs image mixing on the foreground image and the background image.

[0210] In some embodiments, during the image blending process, in order to preserve the perspective effect, the electronic device needs to first calculate the transparency of each pixel in the perspective image and the visual effect image, and then use a pixel-by-pixel alpha blending algorithm to blend the perspective image and the visual effect image.

[0211] Optionally, the electronic device achieves mixing of the perspective image and the visual effect image through an alpha channel or a transparency mask.

[0212] Optionally, during the image mixing process, the electronic device needs to consider the relative position and color information of the foreground image and the background image to obtain a natural superposition effect.

[0213] Exemplarily, assume that the transparency of a planar transparent object can be expressed as a transparency function alpha(x,y) in two-dimensional space. Where (x,y) is the coordinate of a pixel on the plane. Then, during the rendering process of the electronic device, final_color = alpha(x,y)*transparent_color+(1-alpha(x,y))*background_color. Where final_color is the color of the pixel of the final rendered mixed image, transparent_color is the color of the pixel on the visual effect image, and background_color is the color of the pixel on the perspective image. The electronic device uses an alpha blending algorithm to mix the transparency and color of the perspective image and the visual effect image to produce a transparent effect. Thereby, when the target user views the location of the electronic device, he or she can have the perception of a 3D model at the location of the electronic device, and can produce a perspective effect of the 3D model.

[0214] The above description takes the foldable screen electronic device as a double-folding folding screen electronic device, and the three sub-screens can be combined to form a relatively closed triangular prism as an example to introduce the display method provided in the embodiment of the present application. It should be understood that the display method provided in the embodiment of the present application can also be applied to single-folding folding screen electronic devices, tri-folding folding screen electronic devices, and other foldable screen electronic devices with more or fewer sub-screens, or non-foldable screen electronic devices, and the specific display method will not be repeated.

[0215] Fig.10 This is a flow chart of another display method provided in the embodiment of the present application. It should be noted that this method does not Fig.10The specific order below is for limitation, and it should be understood that in other embodiments, the order of some steps in the method can be interchanged according to actual needs, or some steps can be omitted or deleted. The method comprises the following steps:

[0216] S1001. The electronic device obtains a panoramic image.

[0217] The electronic device is an electronic device with a foldable screen. The screen of the electronic device includes at least three sub-screens. When the at least three sub-screens are folded to form a closed area and the electronic device is in a stationary state, the electronic device is triggered to run a table mode. Optionally, in the table mode, the electronic device generates and displays a perspective image based on a panoramic image, so that the electronic device is integrated into the surrounding physical environment, so that the user can see the environment blocked by the electronic device through the electronic device, and enrich the visual effect.

[0218] For example, Figure 4 In the scene shown, user 001 folds mobile phone 002 so that the three sub-screens form a certain angle and then places it vertically on the table. The three sub-screens of mobile phone 002 are combined to form a relatively closed triangular prism.

[0219] Optionally, each of the at least three sub-screens of the electronic device is configured with at least one camera.

[0220] For example, Figure 5 As shown, the electronic device display screen includes a sub-screen 51, a sub-screen 52, and a sub-screen 53. Among them, the sub-screen 51 is configured with a camera 511, the sub-screen 52 is configured with a camera 521, and the sub-screen 53 is configured with a camera 531. That is, the electronic device is configured with three cameras. The electronic device can collect three initial images through these three cameras.

[0221] In some embodiments, the electronic device collects an initial image through cameras configured on at least three sub-screens, and then generates a panoramic image based on the initial image.

[0222] In some examples, the electronic device generates the panoramic image when at least one of the initial images includes the target user, or generates the panoramic image after the position of the electronic device moves.

[0223] For example, when the initial image includes a human-shaped object, the electronic device identifies the object corresponding to the target user based on the initial image. The object corresponding to the target user includes any of the following: a human-shaped object matching the host image, a single human-shaped image included in the initial image, a human-shaped object located in the middle of multiple human-shaped objects included in the initial image, and a human-shaped object with the highest image quality among the human-shaped objects included in the initial image.

[0224] For example, the panoramic image is taken by taking a group of full-view photos of the scene with the electronic device as the center point, and seamlessly splicing two adjacent flat photos with a certain overlap area through software to combine them into a spherical image that surrounds the whole area. When watching, the panoramic image can look around the scene in six directions: front, back, left, right, top, and bottom, and can also provide users with an immersive interactive experience by zooming in or out of the scene. Then, the panoramic image generated by the electronic device can reflect the environmental information around the electronic device.

[0225] In some examples, panoramic images mainly serve the user's visual senses. If the electronic device does not detect a human figure, that is, there is no one around the electronic device, even if the electronic device generates a panoramic image, in an unmanned environment, the sensory experience generated is not perceived by anyone, resulting in unnecessary power consumption. Therefore, the electronic device only needs to generate a panoramic image after detecting a human figure.

[0226] In other examples, if the position of the electronic device does not change, the probability of the panoramic image changing is low. Therefore, after the electronic device first captures the initial image and generates the panoramic image, it can stop generating the panoramic image. Later, after the electronic device detects a position change, it can trigger the generation of the panoramic image again.

[0227] S1002: The electronic device converts the perspective of the panoramic image according to the position of the target user in the panoramic image to generate a target perspective panoramic image, wherein a first perspective corresponding to the target perspective panoramic image matches a second perspective of the target user.

[0228] Among them, the electronic device takes its own position as the center to obtain a panoramic image. Therefore, the first perspective corresponding to the panoramic image is the observation perspective of the electronic device. The subsequent folding screen display needs to be presented to the target user for viewing, so the picture displayed on the folding screen needs to be the picture from the perspective of the target user. That is, after the electronic device generates the panoramic image, it needs to convert the panoramic image from the perspective of the electronic device into a panoramic image from the perspective of the target user.

[0229] In some embodiments, after acquiring the panoramic image, the electronic device determines the position of the target user in the panoramic image according to the initial image. The target user is a user viewing the image presented by the electronic device. Optionally, the target user is, for example, the owner of the device, or one or more users in the surrounding environment.

[0230] In some examples, the user views the surrounding environment through eyes. Then, the position of the target user is, for example, the position of one eye or both eyes of the target user. For example, the electronic device uses the center point position of both eyes of the target user as the position of the target user for subsequent perspective conversion of the panoramic image.

[0231] The position of the target user is the position of the target user relative to the electronic device. Afterwards, the electronic device can convert the panoramic image from the perspective of the electronic device into a panoramic image from the perspective of the target user by converting the perspective. Thus, in the subsequent process of displaying an image on the electronic device, the displayed image can be an image of the environment around the electronic device viewed from the perspective of the target user.

[0232] In some embodiments, the electronic device determines the second viewing angle according to the positional relationship between the target user and the electronic device. The electronic device establishes a first coordinate system with the position of the target user as the origin. Afterwards, the electronic device performs a coordinate system conversion on the panoramic image according to the first coordinate system and the second viewing angle to generate a panoramic image of the target viewing angle.

[0233] In some examples, after determining the sight angle of the target user, the electronic device may determine the location of the target user as the origin of the panoramic coordinate system. Based on the origin of the panoramic coordinate system, the electronic device performs a coordinate system transformation on the panoramic image, thereby obtaining a panoramic image from the perspective of the target user. Optionally, the origin of the panoramic coordinate system determined based on the sight angle of the target user is, for example, the midpoint of the user's eyes. Optionally, the transformation of the coordinate system includes transformations such as rotation, translation, and scaling.

[0234] In some examples, in a panoramic image, the further away from the center of the line of sight, the greater the image distortion. Therefore, during the image conversion process, the electronic device uses the line of sight angle of the target user as the center line of the line of sight.

[0235] That is, the electronic device performs coordinate system conversion on the panoramic image according to the first coordinate system and the second viewing angle.

[0236] Exemplarily, after acquiring a panoramic image through multiple initial images, the electronic device determines the coordinate system of the panoramic image obtained from the perspective of the electronic device. Afterwards, the electronic device determines the target user to obtain the coordinate system from the perspective of the target user. Afterwards, the electronic device can convert each coordinate (x1, y1, z1) in the panoramic image into the coordinate (x2, y2, z2) of the user's perspective based on the conversion relationship between the two coordinate systems. For example, a conversion matrix T is pre-configured in the electronic device, and the electronic device implements the perspective conversion of the panoramic image through the conversion matrix, such as [x2, y2, z2, w2] = T[x1, y1, z1, 1]. Among them, w2 represents the orientation in the coordinate system after the transformation. In this way, the electronic device implements the coordinate system conversion of each coordinate point in the panoramic image through the conversion matrix, and then realizes the conversion of the panoramic image into a panoramic image from the user's perspective. That is, the panoramic image is converted into a panoramic image from the target perspective.

[0237] S1003: The electronic device generates a perspective image according to the panoramic image from a target viewing angle.

[0238] In some embodiments, the electronic device determines the outline of the electronic device according to the target viewing angle panoramic image, generates a partial panoramic image according to the outline of the electronic device, and then generates a perspective image according to the partial panoramic image.

[0239] In some examples, the electronic device determines an image acquisition blind area of ​​the electronic device based on an initial image corresponding to the panoramic image, and determines a contour of the electronic device based on the image acquisition blind area and the target viewing angle panoramic image.

[0240] In some embodiments, during the process of capturing the initial image, due to the angular range of the camera's field of view, there are blind spots in the image capture range of the cameras on adjacent sub-screens. For example, the cameras are located on different sub-screens, and after the display screen is folded, multiple sub-screens are connected to form a polygonal structure, so the positions of multiple cameras do not overlap. Therefore, there is at least a blind spot in the folding angle area of ​​the display screen's rotation axis position, and a blind spot inside the formed polygon. Therefore, the electronic device determines the screen outline of the electronic device in the panoramic image through the blind spot position. That is, the electronic device determines the spatial position of each sub-screen, and then determines the screen coordinates of the electronic device in the panoramic image. In this way, the electronic device can subsequently fill the outline of the electronic device in the blind spot position, so that the electronic device is integrated into the surrounding environment, so as to achieve a more realistic panoramic image display and enhance the holographic display effect.

[0241] For example, after the electronic device determines the outline of the electronic device based on the target perspective panoramic image and generates a partial panoramic image viewed by the user through the electronic device, the electronic device can generate a two-dimensional perspective image based on the partial panoramic image. The perspective image is a two-dimensional image displayed on the screen of the electronic device after being converted from the partial panoramic image viewed by the target user through the target sub-screen.

[0242] In some embodiments, in the table mode, the electronic device is in a folded state, and there is an angle between the multiple sub-screens. In the process of converting the three-dimensional partial panoramic image into a two-dimensional perspective image, the three-dimensional image is projected onto a two-dimensional plane. Therefore, in the process of perspective projection of the electronic device, multiple partial panoramic images corresponding to the multiple sub-screens are determined based on each sub-screen of the electronic device. Afterwards, the electronic device performs projection transformation based on the multiple partial panoramic images to obtain multiple perspective images.

[0243] In some examples, the partial panoramic image includes multiple partial panoramic images that respectively correspond to multiple sub-screens of the electronic device, and the perspective image includes multiple perspective images that respectively correspond to the multiple partial panoramic images.

[0244] S1004: The electronic device displays the perspective image on the target sub-screen corresponding to the target user.

[0245] Among them, the target sub-screen is part or all of the multiple sub-screens of the electronic device. For example, after the electronic device generates multiple perspective images corresponding to the multiple sub-screens, it displays the corresponding perspective images on the multiple sub-screens respectively. For another example, after the electronic device generates multiple perspective images corresponding to the multiple sub-screens, it only displays the corresponding perspective images on the target sub-screen that the target user can see, thereby reducing display power consumption. For another example, the electronic device only generates perspective images corresponding to the sub-screens that the target user can see, and then displays the corresponding perspective images on these sub-screens.

[0246] For example, the electronic device determines a target perspective image corresponding to the target sub-screen from among the multiple perspective images acquired according to the panoramic image, and then displays the target perspective image on the target sub-screen.

[0247] In this way, after generating a perspective image, the electronic device can display the perspective image on the sub-screen, so that the user can watch the perspective image with a perspective effect on the sub-screen, and the electronic device is integrated into the surrounding environment to enrich the visual effect.

[0248] In some embodiments, the electronic device determines the target sub-screen corresponding to the target user in the panoramic image based on the spatial position relationship between the electronic device and the target user in the panoramic image. Alternatively, the electronic device determines that the target initial image of the target user is acquired in the initial image corresponding to the panoramic image. The electronic device determines the sub-screen where the camera that captured the target initial image is located as the target sub-screen.

[0249] In some examples, the electronic device simulates the holographic effect of the target user viewing the electronic device based on the position of the target user. Then, the electronic device can determine the target sub-screen according to the sub-screen that the target user can "see". For example, the electronic device determines one or more sub-screens with the smallest distance between the center point of the sub-screen of the electronic device and the position of the target user as the target sub-screen through the target perspective panoramic image.

[0250] In other examples, the sub-screen that the target user can see is the target sub-screen. Then, if the target user can see the sub-screen, the camera on the corresponding sub-screen should also be able to capture the image of the target user. Therefore, after determining the target user, the electronic device determines to recognize the initial image of the target user. Then, the electronic device can determine the camera that captures the initial image, and determine that the sub-screen where the camera is located is the target sub-screen.

[0251] In this way, the electronic device synthesizes the panoramic image and combines it with the location of the target user to determine the image of the area blocked by the electronic device from the perspective of the target user, and displays this part of the image on the target sub-screen. In this way, visually, the user can see the image of the part blocked by the electronic device through the electronic device, achieving a perspective effect and enriching the user's visual experience in the electronic device stand mode scene.

[0252] In some scenarios, the sub-screen of the electronic device can display specific images such as clocks and calendars, so that the electronic device can be used as a decoration or tool in the environment. Then, the electronic device can also realize the function of the decoration or tool through image display. In some examples, after the electronic device realizes the perspective effect of the folding screen through the above method, a three-dimensional object can be obtained, and the three-dimensional object can be superimposed and displayed on the basis of the perspective effect. Through the coordination of the three-dimensional object and the perspective effect, when the target user looks at the location of the electronic device, the three-dimensional object is placed at the location of the electronic device.

[0253] In some embodiments, the electronic device acquires a preset model, and generates a visual effect image corresponding to the preset model according to the position of the target user and the preset model, wherein the visual effect image is an image corresponding to the second perspective.

[0254] Optionally, the preset model is one or more 3D models pre-existing in the electronic device, or one or more 2D images pre-existing in the electronic device.

[0255] In some embodiments, after obtaining the preset model, the electronic device can obtain the coordinate system corresponding to the preset model, which is a default coordinate system (such as a world coordinate system). Therefore, the electronic device can estimate the camera posture of the preset model according to the user's position and the electronic device's position in the target perspective panoramic image. This enables the electronic device to image the preset model and obtain a visual effect image from the user's perspective.

[0256] In some embodiments, the electronic device generates a mixed image based on the visual effect image and the perspective image, and then displays the mixed image on the target sub-screen.

[0257] Optionally, the electronic device uses the visual effect image as a foreground image and the perspective image as a rearview image, and performs a mixed calculation on the visual effect image and the perspective image through a preset algorithm to generate a mixed image.

[0258] Optionally, the visual effect image includes multiple visual effect images corresponding to multiple sub-screens of the electronic device respectively, and the mixed image displayed on the target sub-screen is a target mixed image generated based on the target visual effect image corresponding to the target sub-screen and the target perspective image.

[0259] In this way, by fusing the perspective image with the preset model, the electronic device can be integrated into the surrounding environment, and the final visual effect image brings a richer visual experience to the user.

[0260] Combination of the above Figure 3-Figure 10 The display method provided by the embodiment of the present application is described in detail. Fig.11 The electronic device provided by the embodiments of the present application is described in detail.

[0261] In one possible design, Fig.11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.11 As shown, the electronic device 1100 may include: a transceiver unit 1101, a processing unit 1102, and a display unit 1103. The electronic device 1100 may be used to implement the functions of the electronic device (such as an electronic device with a foldable screen) involved in the above method embodiments.

[0262] Optionally, the transceiver unit 1101 is used to support the electronic device 1100 to execute Figure 3 S301 in; and / or, for supporting the electronic device 1100 to execute Fig.10 S1001 in.

[0263] Optionally, the processing unit 1102 is used to support the electronic device 1100 to execute Figure 3 S301, S302, S303, S304 and S305 in; and / or, for supporting the electronic device 1100 to execute Fig. 9 S901 and S902 in; and / or, for supporting the electronic device 1100 to execute Fig.10 S1001, S1002 and S1003 in.

[0264] Optionally, the display unit 1103 is used to support the electronic device 1100 to execute Figure 3 S305 in; and / or, for supporting the electronic device 1100 to execute Fig. 9 S902 in; and / or, for supporting the electronic device 1100 to execute Fig.10 S1004 in.

[0265] Among them, the transceiver unit may include a receiving unit and a sending unit, and may be implemented by a transceiver or a transceiver-related circuit component, and may be a transceiver or a transceiver module. The operations and / or functions of each unit in the electronic device 1100 are respectively to implement the corresponding process of the display method described in the above method embodiment. All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional unit, and for the sake of brevity, they will not be repeated here.

[0266] Optionally, Fig.11 The electronic device 1100 shown may also include a storage unit ( Fig.11 (not shown), the storage unit stores a program or instruction. When the transceiver unit 1101, the processing unit 1102 and the display unit 1103 execute the program or instruction, Fig.11 The electronic device 1100 shown can execute the display method described in the above method embodiment.

[0267] Fig.11 The technical effects of the electronic device 1100 shown can refer to the technical effects of the display method described in the above method embodiment, and will not be repeated here.

[0268] In addition to being in the form of electronic device 1100, the technical solution provided in the present application may also be a functional unit or chip in the electronic device, or a device used in conjunction with the electronic device.

[0269] An embodiment of the present application also provides a chip system, including: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the chip system implements the method in any of the above method embodiments.

[0270] Optionally, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0271] Optionally, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately arranged with the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transient processor, such as a read-only memory ROM, which may be integrated with the processor on the same chip or may be arranged on different chips respectively. The embodiments of the present application do not specifically limit the type of memory and the arrangement of the memory and the processor.

[0272] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0273] It should be understood that each step in the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0274] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is run on a computer, the computer executes the above-mentioned related steps to implement the display method in the above-mentioned embodiment.

[0275] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the display method in the above-mentioned embodiment.

[0276] In addition, an embodiment of the present application further provides a device. The device may be a component or a module, and the device may include one or more processors and a memory connected to each other. The memory is used to store a computer program. When the computer program is executed by one or more processors, the device performs the display method in each of the above method embodiments.

[0277] Among them, the device, computer-readable storage medium, computer program product or chip provided in the embodiments of the present application are used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0278] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in hardware or by executing software instructions by a processor. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC).

[0279] Through the description of the above implementation methods, those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed; that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0280] In the several embodiments provided in the present application, it should be understood that the disclosed method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of modules or units, which can be electrical, mechanical or other forms.

[0281] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0282] Computer-readable storage media include, but are not limited to, any of the following: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media that can store program codes.

[0283] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A display method, characterized in that: Applied to an electronic device, the electronic device is an electronic device with a foldable screen, the method comprises: Acquire panoramic images; According to the position of the target user in the panoramic image, the panoramic image is converted into a perspective to generate a target perspective panoramic image, wherein a first perspective corresponding to the target perspective panoramic image matches a second perspective of the target user; generating a perspective image according to the target perspective panoramic image; The perspective image is displayed on a target sub-screen corresponding to the target user.

2. The method according to claim 1, characterized in that The screen of the electronic device includes at least three sub-screens, and the at least three sub-screens are folded to enclose a closed area, and the electronic device is in a stationary state.

3. The method according to claim 2, characterized in that The step of acquiring a panoramic image comprises: Capturing an initial image through cameras configured on the at least three sub-screens; The panoramic image is generated according to the initial image.

4. The method according to claim 3, characterized in that Generating the panoramic image according to the initial image includes: generating the panoramic image when at least one of the initial images includes the target user; or, After the position of the electronic device is moved, the panoramic image is generated.

5. The method according to any one of claims 2 to 4, characterized in that: Each of the at least three sub-screens is configured with at least one camera.

6. The method according to any one of claims 1 to 5, characterized in that: The step of converting the perspective of the panoramic image according to the position of the target user to generate a panoramic image from a target perspective includes: determining the second viewing angle according to a positional relationship between the target user and the electronic device; Establishing a first coordinate system with the position of the target user as the origin; The panoramic image is subjected to coordinate system transformation according to the first coordinate system and the second viewing angle to generate the target viewing angle panoramic image.

7. The method according to any one of claims 1 to 6, characterized in that: The step of generating a perspective image according to the target viewing angle panoramic image comprises: Determining the outline of the electronic device according to the target viewing angle panoramic image; generating a partial panoramic image according to the outline of the electronic device; The perspective image is generated according to the partial panoramic image.

8. The method according to claim 7, characterized in that The step of determining the outline of the electronic device according to the target viewing angle panoramic image comprises: Determining an image acquisition blind area of ​​the electronic device according to an initial image corresponding to the panoramic image; The outline of the electronic device is determined according to the image acquisition blind area and the target viewing angle panoramic image.

9. The method according to claim 7 or 8, characterized in that: The partial panoramic image includes a plurality of partial panoramic images respectively corresponding to a plurality of sub-screens of the electronic device, and the perspective image includes a plurality of perspective images respectively corresponding to the plurality of partial panoramic images.

10. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: Get the preset model; According to the position of the target user and the preset model, a visual effect image corresponding to the preset model is generated, and the visual effect image is an image corresponding to the second viewing angle.

11. The method according to claim 10, characterized in that The displaying the perspective image on the target sub-screen corresponding to the target user includes: Generate a mixed image according to the visual effect image and the perspective image; The mixed image is displayed on the target sub-screen.

12. The method according to claim 11, characterized in that The step of generating a mixed image according to the visual effect image and the perspective image comprises: The visual effect image is used as a foreground image, and the perspective image is used as a rear view image. A mixed calculation is performed on the visual effect image and the perspective image through a preset algorithm to generate the mixed image.

13. The method according to any one of claims 10 to 12, characterized in that: The visual effect image includes multiple visual effect images corresponding to multiple sub-screens of the electronic device respectively, and the mixed image displayed on the target sub-screen is a target mixed image generated based on the target visual effect image corresponding to the target sub-screen and the target perspective image.

14. The method according to any one of claims 1 to 13, characterized in that: Before displaying the perspective image on the target sub-screen corresponding to the target user, the method further includes: determining a target sub-screen corresponding to the target user in the panoramic image according to a spatial position relationship between the electronic device and the target user in the panoramic image; or, Determine that a target initial image of the target user is obtained from the initial image corresponding to the panoramic image; The sub-screen where the camera that captured the target initial image is located is determined as the target sub-screen.

15. The method according to any one of claims 1 to 14, characterized in that: The displaying the perspective image on the target sub-screen corresponding to the target user includes: Determining a target perspective image corresponding to the target sub-screen among a plurality of perspective images acquired according to the panoramic image; The target perspective image is displayed on the target sub-screen.

16. The method according to claim 3, characterized in that After acquiring the initial image, the method further includes: In the case where the initial image includes a human-shaped object, the object corresponding to the target user is identified based on the initial image, and the object corresponding to the target user includes any one of the following items: a human-shaped object matching the host image, a single human-shaped image included in the initial image, a human-shaped object located in the middle position among multiple human-shaped objects included in the initial image, and a human-shaped object with the highest image quality among the human-shaped objects included in the initial image.

17. The method according to claim 16, characterized in that After acquiring the panoramic image, the method further includes: The position of the target user is determined in the panoramic image according to the initial image.

18. An electronic device, characterized in that: include: A processor, a memory and a display screen, wherein the memory and the display screen are coupled to the processor, the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor reads the computer instructions from the memory, the electronic device executes the method as described in any one of claims 1 to 17.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises a computer program, and when the computer program is executed on an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 17.

20. A computer program product, characterized in that When the computer program product is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 17.

Images