Display control method and apparatus and electronic device

By obtaining the posture information of electronic devices and dynamically adjusting the image frames on the display screen, the problem of the inability to effectively simulate the changes in user perspectives in the existing technology is solved, the integration effect between the real world and the virtual world is achieved, and the user experience and design vitality is enhanced. .

WO2024234810A1PCT designated stage expired Publication Date: 2024-11-21HUAWEI TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/081473
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-18
Filing Date
2024-03-13
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

When existing electronic devices achieve dynamic effects of blending the real world and the virtual world on the display screen, they cannot effectively simulate changes in user perspectives, resulting in unreal visual effects, poor user experience, and insufficient design vitality.

Method used

By obtaining the posture information of electronic devices, dynamically adjusting the image frames on the display screen, using the posture mapping information and preset animation curves, combining real-time rendering technology and pre-rendering technology to generate video frames that conform to the changes in user's perspectives, real-world and virtualization The effect of the world.

Benefits of technology

It enhances the user's visual experience, provides better integrated reality visual effects, and enhances the interactivity and design vitality of electronic devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024081473_21112024_PF_FP_ABST
    Figure CN2024081473_21112024_PF_FP_ABST
Patent Text Reader

Abstract

Embodiments of the present application provide a display control method and apparatus and an electronic device. The method comprises: acquiring first attitude information of an electronic device; when it is determined that the attitude of the electronic device changes, determining a first image frame according to the first attitude information, wherein the first image frame belongs to a first video stored in the electronic device or the first image frame is obtained by rendering a second image frame, and the second image frame is an image frame displayed on a display screen of the electronic device before the attitude of the electronic device changes; and on the basis of the change of the attitude, displaying the first image frame on the display screen of the electronic device. The method is used for obtaining better visual effects of merged reality in electronic devices.
Need to check novelty before this filing date? Find Prior Art

Description

Display control method, device and electronic equipment

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on May 18, 2023, with application number 202310566992.3 and application name “A display control method, device and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of electronic technology, and in particular to a display control method, device and electronic equipment. Background Art

[0004] With the continuous development of electronic technology, consumer electronics are gradually at a node where the real world and the virtual world merge. Providing users with a user experience that blends the real world and the virtual world (such as Merged Reality (MR)) is an important direction of user experience design at this stage.

[0005] Currently, in the visual design of electronic devices, it is also expected that a dynamic effect (such as dynamic wallpaper) that blends the real world and the virtual world can be presented on the display screen of the electronic device to enhance the user's visual experience.

[0006] Summary of the Invention

[0007] Embodiments of the present application provide a display control method, device, and electronic device for obtaining a better visual effect of fused reality on an electronic device.

[0008] In a first aspect, an embodiment of the present application provides a display control method, the method comprising: when the posture of an electronic device changes, obtaining first posture information of the electronic device; determining a first image frame based on the first posture information, wherein the first image frame belongs to a first video saved in the electronic device, or the first image frame is rendered based on a second image frame, and the second image frame is an image frame displayed on the display screen of the electronic device before the posture of the electronic device changes; based on the change in the posture, displaying the first image frame on the display screen of the electronic device. This method can dynamically display different image frames on the display screen of the electronic device in combination with the posture change of the electronic device, simulating the perception of the user's perspective change, so that the user can obtain a better visual effect of integrated reality on the electronic device.

[0009] In a possible implementation, the first posture information is represented by a device posture angle of the electronic device, where the device posture angle includes at least one of the following: a pitch angle, a roll angle, or a yaw angle.

[0010] In one possible implementation, determining the first image frame based on the first posture information includes: determining the first image frame in the first video based on the first posture information and posture mapping information saved in association with the first video, wherein the posture mapping information is used to describe the mapping relationship between different postures of the electronic device and different image frames of the first video.

[0011] In one possible implementation, displaying the first image frame on the display screen of the electronic device based on the change in the posture includes: displaying the first image frame on the display screen of the electronic device through a preset animation curve based on the change in the posture, wherein the animation curve is associated with the playback frame rate of the first video and the posture change time of the electronic device.

[0012] In one possible implementation, the first posture information is represented by the device posture angle of the electronic device, and the device posture angle includes at least one of the following: pitch angle, roll angle or yaw angle, and the first video includes at least one of the following: a camera angle video saved by the electronic device based on the posture change of the pitch angle in the same three-dimensional environment; or, a camera angle video saved by the electronic device based on the posture change of the roll angle in the same three-dimensional environment; or, a camera angle video saved by the electronic device based on the posture change of the yaw angle in the same three-dimensional environment.

[0013] In a possible implementation, the first posture information is used to describe a first posture of the electronic device relative to a user's viewing angle, where the user is a user of the electronic device.

[0014] In one possible implementation, if the display screen of the electronic device is facing the user's face, and the first posture information is represented by the distance between the display screen of the electronic device and the user's head, the first video includes: a camera angle video saved by the electronic device based on the distance change in the same three-dimensional environment.

[0015] In a possible implementation, the method further includes: generating the first video based on a three-dimensional environment video material.

[0016] In one possible implementation, if the frame rate of the three-dimensional environment video material is lower than the required frame rate of the first video, generating the first video based on the three-dimensional environment video material specifically includes: based on the required frame rate of the first video, supplementing the missing third image frame of the three-dimensional environment video material through pre-rendering technology and interpolation technology; generating the first video based on the three-dimensional environment video material and the third image frame.

[0017] In a possible implementation, if the 3D environment video material is in a compressed format, generating the first video based on the 3D environment video material specifically includes: generating the first video frame based on the 3D environment video material and a frame difference associated with the 3D environment video material.

[0018] In a possible implementation, the method further includes: acquiring scene description information; and determining the first image frame according to the first posture information includes: generating the first image frame according to the first posture information and the scene description information.

[0019] In a possible implementation manner, the first image frame is generated according to the first posture information, the scene description information, and real-time rendering technology.

[0020] In a possible implementation, the scene description information includes weather information.

[0021] In one possible implementation, the display screen of the electronic device includes a foldable display screen.

[0022] In a second aspect, an embodiment of the present application provides an electronic device comprising a display screen, a processor, and a memory; the memory is used to store one or more computer programs; when the one or more computer programs stored in the memory are executed by the processor, the electronic device executes the method described in the first aspect and any possible implementation of the first aspect.

[0023] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program runs on an electronic device, the electronic device executes the method described in the first aspect and any possible implementation of the first aspect.

[0024] In a fourth aspect, an embodiment of the present application provides a chip, which is coupled to a memory and is used to execute a computer program stored in the memory to perform the method described in the first aspect and any possible implementation of the first aspect.

[0025] In a fifth aspect, an embodiment of the present application further provides a device comprising modules / units for executing any one of the possible design methods of any of the above aspects. These modules / units may be implemented by hardware, or by hardware executing corresponding software implementations.

[0026] In a sixth aspect, an embodiment of the present application further provides a method comprising a computer program product, which, when executed on a terminal, enables the electronic device to execute any one of the possible implementation methods described in any of the above aspects.

[0027] These and other aspects of the present application will become more readily apparent from the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] FIG1 shows a schematic diagram of a dynamic wallpaper;

[0029] FIG2 shows a schematic diagram of another dynamic wallpaper;

[0030] FIG3 shows a schematic structural diagram of an electronic device according to an embodiment of the present application;

[0031] FIG4 shows a schematic diagram of the software structure of an electronic device according to an embodiment of the present application;

[0032] FIG5 is a schematic flow chart showing a display control method according to an embodiment of the present application;

[0033] 6A to 6C are schematic diagrams showing a coordinate system conversion according to an embodiment of the present application;

[0034] 7A and 7B are schematic diagrams showing posture changes of an electronic device according to an embodiment of the present application;

[0035] 8A to 8C illustrate dynamic visual display effects of an electronic device based on changes in pitch angle;

[0036] 9A and 9B are schematic diagrams showing changes in the electronic device based on the yaw angle or the roll angle;

[0037] FIG10 is a schematic diagram showing a display interface of an electronic device when the electronic device is away from the user's head;

[0038] FIG11 shows a schematic diagram of an animation curve according to an embodiment of the present application;

[0039] FIG12 is a schematic diagram showing a mapping between angle intervals and videos according to an embodiment of the present application;

[0040] FIG13 shows a schematic diagram of display control of a notebook computer according to an embodiment of the present application;

[0041] FIG14A shows a schematic diagram of display control of a foldable mobile phone according to an embodiment of the present application;

[0042] FIG14B shows a schematic diagram of an interface of a mobile phone in a fully unfolded state at different distances from a user's face according to an embodiment of the present application;

[0043] FIG15 is a schematic diagram showing a weather effect according to an embodiment of the present application;

[0044] FIG16 shows a schematic structural diagram of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] As described in the background art, in the visual design of electronic devices, it is also expected to present a dynamic effect (such as dynamic wallpaper) that blends the real world and the virtual world on the display screen of the electronic device to enhance the user's visual experience.

[0046] However, in some current designs, such as the smartphone display interface shown in Figure 1, dynamic effects are achieved by overlaying multiple images and binding the gyroscope reading to the image's position or displacement. This allows the gyroscope to display image 1 during the first gyroscope reading and image 2 during the second gyroscope reading, thereby creating a staggered image movement effect as the smartphone rotates. However, in this design, the staggered image movement effect fails to convey the distortion caused by changes in perspective when the phone's spatial position changes. Users can clearly notice that the visual effect created by this design does not match the actual movement of the phone.

[0047] In other designs, taking the display screen interface of a smartphone shown in Figure 2 as an example, the implementation of this dynamic effect is bound to the interactive operations on the display screen. For example, when the phone is locked, the display screen presents the picture shown in situation (a). After receiving a specific instruction from the user (such as the user's unlock instruction), the display screen presents the picture shown in situation (b) by moving or scaling the picture position, or changing the camera position. After the phone is locked again, the picture shown in situation (c) is presented. Although this process can achieve a dynamic display effect, this design is limited by the triggering method used, and the effects are more similar to playing and pausing videos. It is difficult for users to form a sensory experience that echoes "interaction-vision". It is easy to cause a situation where the initial experience is good, but the dynamic wallpaper is subsequently abandoned in large quantities, and the design vitality is poor.

[0048] In view of this, the embodiments of the present application provide a display control method, device and electronic device for obtaining a better visual effect of integrated reality in electronic devices. Among them, the method and the device are based on the same technical concept. Since the principles of solving the problems of the method and the device are similar, the implementation of the device and the method can refer to each other, and the repeated parts will not be repeated. Moreover, in the various embodiments of the present application, if there is no special explanation and logical conflict, the terms and / or descriptions between the various embodiments are consistent and can be referenced to each other. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0049] It should be noted that, in the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0050] Furthermore, unless otherwise specified, ordinal numbers such as "first" and "second" in the embodiments of this application are used to distinguish between multiple objects and are not used to define the priority or importance of multiple objects. For example, the first device and the second device are only used to distinguish different electronic devices, and do not indicate a difference in priority or importance between the two devices. For example, in some embodiments, the method steps performed by the first device and the second device can be interchangeable.

[0051] For ease of understanding, a detailed description is given below with reference to the accompanying drawings and embodiments.

[0052] FIG3 shows a schematic structural diagram of an electronic device according to an embodiment of the present application.

[0053] 3 , the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0054] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors. The controller may serve as the nerve center and command center of the electronic device 100. The controller may generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. The processor 110 may also include memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a high-speed cache memory. This memory may store instructions or data that have just been used or are being recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces the processor 110's waiting time, and thus improves system efficiency.

[0055] The USB interface 130 is an interface that complies with USB standards and specifications, and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and peripheral devices. The charging management module 140 is used to receive charging input from the charger. The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160.

[0056] The wireless communication functionality of electronic device 100 can be implemented using antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, a modem processor, and a baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0057] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0058] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0059] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0060] Display screen 194 is used to display application interfaces, such as the desktop of electronic device 100. The desktop may include icons for applications installed on electronic device 100 and shortcut icons created. For example, display screen 194 can display different image frames in different postures as the electronic device changes position, providing the user with a dynamic visual effect that blends reality. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0061] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0062] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. For example, the processor 110 executes the instructions stored in the internal memory 121, so that the electronic device 100 executes the methods described in the various embodiments of the present application. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, and software code of at least one application, etc. The data storage area may store data generated during the use of the electronic device 100 (such as captured images, recorded videos, etc.). In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0063] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as pictures and videos can be stored on the external memory card.

[0064] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0065] Among them, the sensor module 180 may include but is not limited to a pressure sensor 180A, an acceleration sensor 180B, a touch sensor 180C, a gyroscope sensor 180D, a hinge 180E, a rotation vector sensor 180F, a distance sensor 180G, and the like.

[0066] The pressure sensor 180A is used to sense pressure signals and convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194 .

[0067] The touch sensor 180C is also called a "touch panel". The touch sensor 180C can be set on the display screen 194. The touch sensor 180C and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180C is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180C can also be set on the surface of the electronic device 100, at a different position from the display screen 194. The embodiment of the present application does not limit the form of the touch sensor 180C.

[0068] The acceleration sensor 180B, gyroscope sensor 180D, hinge 180E, rotation vector sensor 180F, distance sensor 180G, etc. can be used to detect changes in the posture of the electronic device 100. When a change in the posture of the electronic device 100 is detected, the changed posture information of the electronic device 100 can be provided to the processor in real time, so that the processor can dynamically control the display on the display screen 194 according to the changed posture information of the electronic device to display the changed image frame, presenting a dynamic effect of blending the real world and the virtual world to the user, thereby enhancing the user's visual experience. This will be described in detail below in conjunction with the accompanying drawings and embodiments, and will not be repeated here.

[0069] The buttons 190 include a power button, a volume button, etc. The button 190 can be a mechanical button. It can also be a touch button. The electronic device 100 can receive button input and generate key signal input related to the user settings and function control of the electronic device 100. The motor 191 can generate a vibration prompt. The motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization. The indicator 192 can be an indicator light, which can be used to indicate the charging status, power changes, and can also be used to indicate messages, missed calls, notifications, etc. The SIM card interface 195 is used to connect the SIM card. The SIM card can be inserted into the SIM card interface 195 or pulled out from the SIM card interface 195 to achieve contact and separation with the electronic device 100.

[0070] It is understood that the components shown in FIG3 do not constitute a specific limitation on the electronic device 100. The electronic device may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. In addition, the combination / connection relationship between the components in FIG3 may also be adjusted and modified.

[0071] Figure 4 is a block diagram of the software structure of an electronic device provided in an embodiment of the present application. As shown in Figure 4, the software structure of the electronic device can be a layered architecture. For example, the software can be divided into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the operating system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer (framework, FWK), the runtime and system library, and the kernel layer.

[0072] The application layer can include a series of application packages. As shown in Figure 4, the application layer can include camera, settings, skin modules, user interface (UI), and third-party applications. Among them, third-party applications can include gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0073] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer may include some predefined functions. As shown in Figure 4, the application framework layer may include a window manager, content provider, view system, telephony manager, resource manager, notification manager, etc.

[0074] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, take screenshots, etc. It is understood that in the embodiments of the present application, when multiple windows are displayed on the display screen, the window manager can be used to manage multiple windows simultaneously, such as managing the position and size of multiple windows.

[0075] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0076] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0077] The phone manager is used to provide communication functions for electronic devices, such as call status management (including answering, hanging up, etc.).

[0078] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0079] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0080] The runtime includes the core library and the virtual machine. The runtime is responsible for the scheduling and management of the operating system.

[0081] The core library consists of two parts: one containing the Java language's callable functions and the other containing the operating system's core libraries. The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0082] The system library can include multiple functional modules, such as a surface manager, media libraries, a 3D graphics processing library (such as OpenGL ES), and a 2D graphics engine (such as SGL).

[0083] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0084] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0085] The three-dimensional (3D) graphics processing library is used to implement three-dimensional graphics drawing, image rendering, compositing, and layer processing. The two-dimensional (2D) graphics engine is a drawing engine for 2D drawing.

[0086] In some embodiments, the 3D graphics processing library can be used to draw a 3D motion trajectory image, and the 2D graphics engine can be used to draw a 2D motion trajectory image. In some embodiments, the 3D graphics processing library and the 2D graphics processing engine can be integrated into a graphics processor, which is not limited in this embodiment of the present application.

[0087] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0088] The hardware layer may include various sensors, such as accelerometers, gyroscopes, rotation vector sensors, etc., which may be used to detect the posture of the electronic device (such as motion posture or form, etc.), so as to dynamically display different image frames on the display screen of the electronic device in combination with the posture of the electronic device to obtain a better visual effect of integrated reality.

[0089] Based on the above electronic device structure, the present invention provides a display control method. The method can be implemented by the electronic device or by related components of the electronic device (such as sensors, processors, display screens, etc.) working together to implement the method, which is not limited in the present invention.

[0090] The display control method according to an embodiment of the present application is described below with reference to the accompanying drawings.

[0091] FIG5 is a flow chart showing a display control method according to an embodiment of the present application. Referring to FIG5 , the method may include the following steps:

[0092] S510: When the posture of the electronic device changes, first posture information of the electronic device is acquired.

[0093] In the embodiment of the present application, the posture of the electronic device refers to the device posture or device form of the electronic device in at least one dimension.

[0094] The device posture may specifically be the posture of the electronic device when it is stationary or in motion. If the electronic device is in motion, the device posture may specifically be the posture of the electronic device when it translates or rotates based on a certain spatial component dimension (e.g., the x-axis, y-axis, or z-axis in a geodetic coordinate system), or the posture of the electronic device when it translates or rotates based on at least two spatial component dimensions.

[0095] The device form can be determined based on the form-changing capability of the electronic device. For example, if the electronic device is a straight-type smartphone (i.e., the phone screen is not foldable), the device form is single and fixed. If the electronic device is a foldable phone or a foldable tablet, that is, the display screen of the electronic device has the ability to fold, the device form can include, for example, a folded form, a semi-expanded form, and a fully expanded form. If the electronic device is a clamshell laptop or a clamshell phone, the component with a display screen and the component with an input device (such as a keyboard) are linked as a whole, and the device form can include, for example, a closed form, an open form, etc.

[0096] Posture information is information used to describe the posture of an electronic device. The posture information may be the use of geodetic coordinate parameters (x, y, z) to describe the posture of the electronic device, and the at least one dimension may include at least one of the x-axis, y-axis, or z-axis. Alternatively, the posture information may also be the description of the posture of the electronic device by yaw angle, pitch angle, and roll angle (θ, ψ, φ), and the at least one dimension may include at least one of yaw angle, pitch angle, or roll angle. In some more specific embodiments, the posture information may also be a description of the device posture or device form of the electronic device relative to the user's (e.g., the user of the electronic device) perspective. The embodiment of the present application does not limit the specific content of this posture information.

[0097] In a specific implementation, the electronic device may have an input device, which can obtain first posture information of the electronic device at a first moment in real time, and can provide the first posture information to a processor, and the first posture information can be used to describe a first posture, including a motion posture or device form in at least one dimension. The processor can execute the above S510 to obtain the first posture information of the electronic device. Furthermore, the processor can compare the first posture information with the second posture information at a second moment before the first moment to determine whether the posture of the electronic device has changed, or to determine whether the posture of the electronic device has changed relative to the user's perspective. When it is determined that a posture change has occurred, the processor can perform display control on the electronic device based on the first posture information.

[0098] Exemplarily, the display control process may be as shown in FIG5 , including the following steps:

[0099] S520: Determine a first image frame according to the first posture information.

[0100] S530: Displaying the first image frame on a display screen of the electronic device based on the change in the posture.

[0101] Illustratively, the change in the posture of the electronic device may be a change in the yaw angle, pitch angle, or roll angle of the electronic device.

[0102] Exemplarily, the change in the posture of the electronic device may be a change in the posture of the electronic device relative to the user's viewing angle. For example, the change in relative posture may be caused by the user's head being stationary while the electronic device moves (including translation or rotation). Alternatively, the change in relative posture may be caused by the electronic device being stationary while the user's head moves. Alternatively, the change in relative posture may be caused by both the electronic device and the user's head moving.

[0103] The first image frame may be part of a first video stored on the electronic device, or the first image frame may be rendered based on a second image frame, where the second image frame is the image frame displayed on the display screen of the electronic device at a second moment, i.e., the image frame displayed on the display screen of the electronic device before the posture of the electronic device changes. Thus, the image frame output on the display screen of the electronic device can change as the posture of the electronic device changes, thereby providing the user with a dynamic visual effect that blends the real world with the virtual world.

[0104] Since the image frames displayed on the display screen of the electronic device are related to the posture changes of the electronic device, the posture monitoring method of the electronic device and the mapping relationship between the posture changes of the electronic device and the corresponding image frames are introduced below.

[0105] In an embodiment of the present application, an input device can be used to detect whether the posture of the electronic device has changed, or to detect whether the electronic device has changed relative to the user's viewing angle. Based on the different product types or product forms of the electronic device, the input device can include some or all of the examples of sensor modules shown in Figures 3 or 4, including but not limited to an acceleration sensor, a gyroscope sensor, a hinge sensor, a rotation vector sensor, a distance sensor, etc.

[0106] Based on the different sensors used, different sensor data can be obtained at the first moment to directly or indirectly monitor the posture of the electronic device. For example, based on the acceleration sensor, the gravitational acceleration of the electronic device can be measured, based on the gyroscope sensor, the yaw angular velocity of the electronic device can be measured, based on the hinge sensor, the folding or unfolding state of the electronic device can be measured, based on the rotation vector sensor, the angle of rotation of the electronic device around a fixed axis can be measured, and based on the distance sensor, the distance of the electronic device relative to the user's face (or eyes) can be measured. Different sensor modules of the electronic device can communicate with the processor of the electronic device and can provide the collected sensor data as input information to the processor. The processor can process the acquired sensor data to obtain the first posture information of the electronic device at the first moment.

[0107] Because electronic devices contain multiple sensors that can sense different motions, directions, or external environments, these sensor data are all based on the phone's coordinate system. Therefore, when the position or orientation of the electronic device changes, the collected sensor data will also change accordingly. In reality, due to the individualized user habits, for example, the position of the electronic device's display relative to the user—for example, whether the user is standing, sitting, or lying down, or whether the device is held diagonally upwards facing the user's face or diagonally downwards facing the user's face—all affect the electronic device's posture recognition results and the image frames displayed on the device's display. In other words, in practical applications, given the diversity of user habits and the arbitrary placement of electronic devices, it is necessary to convert the data collected by the electronic device's built-in sensors from the device coordinate system to a common reference coordinate system (e.g., the Earth coordinate system). This conversion provides the sensor data with clearer physical meaning, helping the electronic device dynamically display different image frames based on the user's posture changes, achieving a better integrated reality visual effect.

[0108] As shown in Figure 6A, taking a smartphone as an example (the smartphone or mobile phone mentioned here and in the following description is an example of an electronic device, and will not be distinguished or described one by one in the following), one way to define the geodetic coordinate system is as follows: the positive direction of the x-axis is tangent to the ground at the current location of the mobile phone, pointing directly to the east; the positive direction of the y-axis is also tangent to the ground pointing to the magnetic north pole, and the plane where the x-axis and z-axis are located is the horizontal plane; the positive direction of the z-axis is perpendicular to the horizontal plane and points to the sky.

[0109] As shown in Figure 6B, the determination of the mobile phone coordinate system is related to the mobile phone screen. One way to define the mobile phone coordinate system is as follows: the positive direction of the X-axis is the direction from the center of the mobile phone screen plane to the right, and vice versa is the negative direction of the X-axis; the positive direction of the Y-axis is the direction from the center of the mobile phone screen plane to the upward direction, perpendicular to the X-axis, and vice versa is the negative direction of the Y-axis; and the positive direction of the Z-axis is the direction perpendicular to the mobile phone screen plane and pointing upward from the center of the screen plane, and vice versa is the negative direction of the Z-axis.

[0110] Illustratively, an embodiment of the present application provides a conversion formula for converting a mobile phone coordinate system to a geodetic reference coordinate system, as shown in Formula 1.

[0111] Among them, X / Y / Z are the sensor data in the mobile phone coordinate system, R represents the rotation matrix, and x, y, and z are the sensor data in the earth coordinate system.

[0112] Among them, R is composed of three basic rotation matrices, as shown in Formula 2.

[0113] The variables a, p, and r represent azimuth, pitch, and roll, respectively. Azimuth represents the angle between the magnetic north pole and the Y-axis of the phone's coordinate system. Pitch represents the angle between the X-axis of the phone's coordinate system and the horizontal plane. Roll represents the angle between the Y-axis of the phone's coordinate system and the horizontal plane.

[0114] In other words, based on the coordinate system conversion method described above, the phone can determine its position in the geodetic coordinate system based on the sensor data from the converted geodetic coordinate system, such as whether it is in vertical portrait mode, vertical landscape mode, or portrait or landscape mode with a certain tilt angle. If the user replaces the electronic device with a foldable phone or laptop, the position of the electronic device in the geodetic coordinate system can also be, for example, folded, partially unfolded, or unfolded.

[0115] Exemplarily, the embodiment of the present application can determine the device attitude angle of the mobile phone in the geodetic coordinate system through the converted data of the gyroscope sensor and the gravity sensor (or data of other sensors), and the device attitude angle can be used to characterize the device attitude of the mobile phone. The device attitude angle refers to the angle required to rotate the three coordinate axes of the mobile phone coordinate system to be consistent with the geodetic coordinate system. As shown in Figure 6C, the gravity accelerometer of the mobile phone collects gravity acceleration data (components along the three coordinate axes in the current mobile phone coordinate system). In Figure 6C, assuming that the noise data in the gravity accelerometer data is ignored, the rotation angle of the current device state relative to the horizontal placement state of the mobile phone can be calculated directly through the components of the three axes. In an optional embodiment, the rotation matrix corresponding to the device attitude angle can also be calculated.

[0116] Specifically, using gravity acceleration data as an example, the device attitude calculation process is as follows: First, real-time gravity acceleration data a is acquired from the accelerometer. Then, the data at the end of a queue of length N is pulled out and the acceleration value is saved at the head of the queue. Second, the filtered acceleration value a′ is calculated using the rules of a smoothing filter. Third, the phone's processor calculates the device attitude angles, namely the pitch and roll angles, based on the acceleration a′.

[0117] In practical applications, the output signal of the gravity accelerometer will carry noise due to the influence of the environment. The noise is a high-frequency signal relative to the output of the accelerometer, so the sliding mean filtering method can be used for smoothing. It should be noted that the sliding mean filtering method belongs to the finite impulse response filter in the digital filter. The principle is: regard N measurement data as a queue, the length of the queue is fixed to N, and each time a new sampling is performed, the sampling result is placed at the end of the queue, and the data at the head of the original queue is removed, so that there are always N "latest" data in the queue. Compared with the ordinary mean filtering method (a method in which N data must be read in each time), the processing speed of this method is much faster. In the embodiment of the present application, the improved sliding mean filtering method is adopted, and the data with the largest and smallest modulus values ​​among the N data are removed, which can improve the simulation effect on noise. The improved filtering formula is Formula 3:

[0118] in, is the filter output, X k Indicates the kth data in the current sequence, MAX is the data with the largest modulus value among N data, and MIN is the data with the smallest modulus value among N data.

[0119] It should be understood that the above is only an example of the coordinate transformation of the embodiment of the present application and is not intended to be limiting. In other embodiments, the coordinate transformation method may be adjusted by changing the rotation matrix or the above formula, which is not limited in the embodiment of the present application.

[0120] In an embodiment of the present application, as shown in FIG7A , the face angle of the user when looking forward can be predefined as a reference of 0 degrees, which is called a level viewing angle. When the user's head does not move but the eyeballs are turned downward, or when the user's head is turned downward, it is the user's downward viewing angle. When the user's head does not move but the eyeballs are turned upward, or when the user's head is turned upward, it is the user's upward viewing angle. Taking the user's level viewing angle as an example, FIG7B shows the posture angle of the electronic device or the posture angle relative to the user's viewing angle, including the pitch angle (Pitch), the roll angle (Roll) and the yaw angle (Yaw).

[0121] It should be noted that, in an embodiment of the present application, the first posture information obtained by the electronic device (or the processor of the electronic device) when implementing S510 may be in the form of component information of the earth coordinate system shown in FIG6A, or the first posture information may be in the form of component information of the mobile phone coordinate system shown in FIG6B, or the first posture information may be in the form of the three-component posture angle shown in FIG7B. The embodiment of the present application does not limit the specific implementation method of this posture information.

[0122] After obtaining the first posture information of the electronic device and determining that the posture of the electronic device has changed, when implementing S520, the electronic device (or the processor of the electronic device) can determine the first image frame according to the first posture information through image frame mapping or real-time rendering.

[0123] In one example, when adopting the image frame mapping method, the description method adopted by the posture change of the electronic device can be pre-defined, and then the different posture information of the electronic device is associated and bound with the different image frames of the saved first video, and then the information describing the mapping relationship between the different image frames of the first video and the different posture information of the electronic device is saved, which is called posture mapping information. When implementing S520, the electronic device (or the processor of the electronic device) can determine the first image frame in the first video based on the obtained first posture information and the saved posture mapping information.

[0124] As shown in Figure 8A, taking the posture change of an electronic device (such as a smart phone) based on the pitch angle as an example, it can be predefined that when the electronic device is in a horizontal position with the display screen facing upward, the pitch angle corresponds to 0 degrees; when the electronic device is in a vertical position based on the pitch angle change with the display screen facing the user's face, the pitch angle corresponds to 90 degrees; when the electronic device is in a horizontal position with the display screen facing downward, the pitch angle can correspond to 180 degrees.

[0125] Taking the first video as a 3D snow mountain environment video as an example, the saved posture mapping information can pre-define some key angles of the pitch angle of the electronic device, as well as the mapping between these key angles and different image frames of the first video. For example, as shown in Figure 8B, taking the first video based on the pitch angle rotating from 0 degrees to 90 degrees (equivalent to the camera (the camera angle is equivalent to the user's perspective) panning upward with an arc) as an example, the first video, as a pre-collected video material, can include different image frames (or video materials) of the smartphone based on the pitch angle change, and these image frames are represented by numbers 0-10. Each image frame can be pre-associated with a pitch angle of the mobile phone used by the user in accordance with the change of the user's perspective in the real environment. For example, the image frame numbered 0 is associated with the pitch angle. The elevation angle is 0 degrees, the image frame numbered 1 is associated with a pitch angle of 9 degrees, for example, the image frame numbered 2 is associated with a pitch angle of 18 degrees, the image frame numbered 3 is associated with a pitch angle of 27 degrees, the image frame numbered 4 is associated with a pitch angle of 36 degrees, the image frame numbered 5 is associated with a pitch angle of 45 degrees, for example, the image frame numbered 6 is associated with a pitch angle of 54 degrees, the image frame numbered 7 is associated with a pitch angle of 63 degrees, the image frame numbered 8 is associated with a pitch angle of 72 degrees, the image frame numbered 9 is associated with a pitch angle of 81 degrees, and the image frame numbered 10 is associated with a pitch angle of 90 degrees.

[0126] Among them, the image frame corresponding to number 0 mainly includes a snow-capped mountain road, which is equivalent to the user's perspective of looking down at the snow-capped mountain road, and the visual effect is similar to a bird's-eye view. When the electronic device changes to a pitch angle of 45 degrees based on the pitch angle, the image frame corresponding to number 5 includes more peaks in the content, which is equivalent to the user gradually raising their head (for example, turning it 45 degrees upward) to look down at a slightly farther 3D environment. When the electronic device changes to a pitch angle of 90 degrees based on the pitch angle, the image frame presented on the display mainly includes peaks, which is equivalent to the user looking forward (far away), and the visual effect is similar to a front view. It should be understood that what is not shown in Figure 8B is that when the electronic device changes to a pitch angle of 120 degrees based on the pitch angle, the sky portion in the content included in the image frame presented on the display increases, which is equivalent to the user gradually raising their head to look up at the slightly farther sky. When the electronic device changes to a pitch angle of 150 degrees based on the pitch angle, the sky portion in the content included in the image frame presented on the display further increases, which is equivalent to the user further raising their head to look up at the farther sky. This process combines the posture changes of the electronic device to dynamically switch the image frames displayed on the display screen, as if the user is in a real three-dimensional snow mountain environment, allowing the user to have an immersive sensory experience.

[0127] Based on the above-mentioned posture mapping information, when the posture of the electronic device (or the processor of the electronic device) changes, the first image frame that needs to be displayed on the display screen of the electronic device can be determined based on the first posture information obtained and the saved posture mapping information. For example, as shown in Figure 8C, if the first posture information indicates that the electronic device is at a pitch angle of 0 degrees, it can be determined based on the saved posture mapping information that the first image frame that needs to be presented on the display screen of the electronic device is the image frame corresponding to number 0. If the first posture information indicates that the electronic device is at a pitch angle of 45 degrees, it can be determined based on the saved posture mapping information that the first image frame that needs to be presented on the display screen of the electronic device is the image frame corresponding to number 5. If the first posture information indicates that the electronic device is at a pitch angle of 90 degrees, it can be determined based on the saved posture mapping information that the first image frame that needs to be presented on the display screen of the electronic device is the image frame corresponding to number 10. During the posture change process of the electronic device, as the posture changes, different image frames of the first video can be dynamically switched on the display screen of the electronic device to obtain a better visual effect of fusion reality.

[0128] It should be understood that the above description in conjunction with Figures 8A to 8C merely uses the pitch angle as an example to illustrate the posture changes of the electronic device in a certain dimension and the changes in the image frames presented, and does not constitute a limitation on the posture changes of the electronic device. In other embodiments, the electronic device may also exhibit posture changes based on the flip angle or the yaw angle. The posture changes of the electronic device based on the flip angle or the yaw angle can be referred to as shown in Figure 7B. The mapping relationship between the posture changes in these two dimensions and the different image frames of the first video stored in the corresponding dimensions is similar to the posture changes and posture mapping based on the pitch angle. When the pitch angle shown in Figures 8A to 8C is replaced with the yaw angle or the flip angle, the principle of mapping different image frames based on the changes in the yaw angle or the flip angle is similar to the principle of mapping image frames based on the changes in the pitch angle. For details, please refer to the relevant description of Figures 8A to 8C.

[0129] For example, taking the electronic device as a tablet computer, when the tablet computer's display screen displays a 3D snow mountain environment video, if the tablet computer's display screen faces the user's face and changes its posture based on the yaw angle, the tablet computer's display screen can display different image frames of the surrounding 3D snow mountain environment as the tablet computer's posture changes, as shown in Figure 9A.

[0130] For example, taking the electronic device as a tablet computer, when the display screen of the tablet computer displays the first image frame of the associated first video, if the display screen of the tablet computer faces the user's face and changes its posture based on the flip angle, the display screen of the tablet computer can display the flipped image frame as the posture of the tablet computer changes, as shown in Figure 9B.

[0131] In the example introduced above, if the first video associated with a certain dimension cannot be generated and saved in advance, the electronic device or its processor can also use real-time rendering based on the existing video material to generate image frames associated with different postures under this dimension, and display the corresponding image frames in a timely manner when the electronic device undergoes posture changes based on this dimension.

[0132] In other embodiments, if the posture angle of the electronic device remains unchanged but the distance between the display screen and the user's head (such as human eyes or face, etc.) is changed, as shown in Figure 10, the mobile phone display screen is made to face the face and move away from the user's head (or called moving backward), then the first image frame can also be determined from the first video associated with this posture angle and distance dimension and saved for display. If the electronic device does not pre-save the first video associated with this dimension, the electronic device or its processor can also use rendering technology based on existing materials to imitate the camera moving backward and increasing the focal length to render the first image frame to be displayed. Conversely, if the mobile phone display screen is made to face the face and move closer to the user's head (or called moving forward), the rendering technology can be used to imitate the camera moving forward and reducing the focal length to render the first image frame to be displayed (not shown in the figure).

[0133] It should be understood that the above is only an example of an image frame mapping method or a rendering method, and does not constitute a limitation on the method of determining the first image frame. In a specific implementation, for example, when the electronic device undergoes a posture change based on at least one dimension of the pitch angle, yaw angle or roll angle, the first image frame to be displayed can be determined by rendering. Alternatively, when the electronic device undergoes a posture change based on the distance between the electronic device and the user's head, the first image frame to be displayed can be determined by a saved first video, which will not be repeated here.

[0134] In an optional embodiment, in order to make the image on the display screen present an animation effect that is closer to the real world as the electronic device changes its posture, a spring animation curve (as shown in Figure 11) can be used to dynamically switch the corresponding image frames displayed on the electronic device's display screen during the process of the electronic device changing from the second posture to the first posture. Based on the slight changes in the electronic device's posture angle during this posture change, the electronic device's processor can calculate, decode, and play the first image frame corresponding to the first posture information in real time.

[0135] The Spring animation curve can be used to dynamically bind the video playback speed of the first video to the rate at which the user's handheld electronic device moves (usually the ratio of the difference between the starting posture angle (such as the device posture angle associated with the electronic device at the second moment) and the ending posture angle (such as the device posture angle associated with the electronic device at the first moment) to the movement time difference (such as the difference between the first moment and the second moment)). When the user's handheld electronic device moves slowly, since the posture of the electronic device collected by the sensor changes slowly, the change in the number of video frames corresponding to the posture information is small, so different image frames of the first video can be played at a relatively uniform speed. From the user's perception perspective, the first video is played at a uniform speed as the user holds the electronic device. When the user's handheld electronic device moves quickly, since the posture of the electronic device obtained by the sensor changes quickly, the change in the number of video frames corresponding to the posture information is large, so different image frames of the first video can be played at a relatively fast speed. At the user perception level, the first video is played quickly as the user holds the electronic device.

[0136] Typically, when a user moves an electronic device, the movement speeds up first and then slows down. This Spring animation curve can be more fully applied, allowing the first video to be played faster and then slower. From a user's perspective, the first video plays faster as the user's hand moves quickly, and then plays more slowly (e.g., at a constant speed) as the user's hand moves slowly toward the target location.

[0137] In this way, whether the electronic device plays the first video forward on the display screen based on a positive change in the pitch angle (for example, from a pitch angle of 0 degrees to a pitch angle of 90 degrees), or plays the first video backward on the display screen based on a negative change in the pitch angle (for example, from a pitch angle of 90 degrees to a pitch angle of 0 degrees), the user can watch a dynamic video similar to the change in the user's perspective on the display screen of the electronic device, obtaining a better visual effect of fusion of reality. At the same time, designing the speed of playing the first video forward or backward according to the movement rate of the user's handheld electronic device and the Spring animation curve can also ensure that the video playback is more delicate and smooth, giving the user a better visual experience.

[0138] It should be noted that, in an embodiment of the present application, the saved first video may be generated based on three-dimensional environmental materials (such as video materials or image materials, etc.) associated with the electronic device in a certain dimension (such as the X-axis or pitch angle described above). The three-dimensional environmental materials are collected in the real world by a camera that imitates the changes in the user's perspective in the real environment, that is, the camera's perspective is the same as the user's perspective in the real three-dimensional environment.

[0139] In practical applications, a camera can be used to capture and save video footage of different dimensions in a real three-dimensional environment. Then, based on the captured video footage of the three-dimensional environment, corresponding first videos can be generated and saved based on the posture changes of the electronic device in different dimensions. For example, the first video can be a camera angle video saved by the electronic device based on the posture changes of the pitch angle in the same three-dimensional environment. Or, for example, the first video can be a camera angle video saved by the electronic device based on the posture changes of the roll angle in the same three-dimensional environment. Or, for example, the first video can be a camera angle video saved by the electronic device based on the posture changes of the yaw angle in the same three-dimensional environment. Or, for example, the first video can be a camera angle video saved by the electronic device based on the distance changes in the same three-dimensional environment. It should be understood that this is merely an example of the method for generating the first video and does not constitute any limitation. In other embodiments, the first video can also be generated by a device other than the electronic device (e.g., a device used by a service provider) and provided to the electronic device for storage. In this case, the method for generating the first video is similar to the method for generating the first video by the electronic device itself and will not be further described here.

[0140] At the same time, the electronic device can also store posture mapping information in association with the first video, and the posture mapping information is used to describe the mapping relationship between the posture information of the electronic device in different dimensions and the different image frames of the first video stored in association with the corresponding dimensions so that the electronic device can dynamically display the different image frames of the first video stored in the dimension according to the device posture change of the electronic device in the dimension. In addition, the mapping relationship between the posture angle of the electronic device and the image frame of the first video can be a mapping relationship between the posture angle and the timestamp of the video image frame, or a mapping relationship between the posture angle and the camera angle of the three-dimensional environment material, or a mapping relationship between the posture angle and the node of the displacement curve. The embodiment of the present application does not limit the definition method of this mapping relationship.

[0141] It's worth noting that in actual applications, the format requirements for the captured 3D environment video material and the required first video may differ. For example, the required frame rate of the first video may be lower than the frame rate of the 3D environment video material, or the 3D environment video material may be compressed using a difference in volume. Therefore, when generating the first video based on the 3D environment material, the 3D environment video material can be processed to generate the first video, thereby avoiding issues such as stuttering and frame skipping when the first video is played forward or backward as the electronic device's posture changes.

[0142] The processing of the 3D environment video material can be performed as needed, including but not limited to the following two processing methods:

[0143] (1) If the frame rate of the three-dimensional environment video material is less than the required frame rate of the first video, the third image frame missing from the three-dimensional environment video material can be supplemented based on the required frame rate of the first video through pre-rendering technology and interpolation technology. Further, the first video is generated based on the three-dimensional environment video material and the third image frame. For example, if the required frame rate of the first video is 60 frames per second and the frame rate of the collected three-dimensional environment video material is 30 frames per second, the electronic device (or the processor of the electronic device) can supplement the three-dimensional environment video material to 60 frames per second through pre-rendering technology and interpolation technology to generate a first video that meets the playback requirements.

[0144] It is worth noting that the pre-rendering technology described here and the rendering technology introduced above are both rendering technologies, but the time of application is different. Pre-rendering technology mainly refers to the rendering technology used in pre-rendering, which can usually be used to generate the first video stage. It can be used to pre-render the video materials used to ensure that the first video to be generated meets the playback requirements as much as possible. Rendering technology is mainly the rendering technology used in real-time rendering, which can usually be used to generate image frames to be displayed in real time. It can be used to fill in the gaps in image frames of a certain dimension in a timely manner to reduce freezes, frame skipping, etc.

[0145] (2) If the three-dimensional environment video material is in a compressed format, the first video frame can be generated based on the three-dimensional environment video material and the frame difference value associated with the three-dimensional environment video material. For example, the three-dimensional environment video material in some formats is compressed, that is, the first frame and the frame difference value are in the form of 1, +1, +0.5 to reduce the volume. When the material is used to generate the required first video, the electronic device (or the processor of the electronic device) can be uniformly adjusted to the form of 1, 2, 2.5 so that the generated first video meets the playback requirements.

[0146] Therefore, through the above processing of the three-dimensional environment video material, the generated first video can better meet the playback requirements and bring a smoother dynamic visual experience to the user.

[0147] In addition, due to the personalization of user usage habits, the user's electronic device may mostly undergo posture changes in a certain angle range, and is not often used in other angle ranges. Take the posture change of electronic devices based on pitch angles as an example. For example, it is often used in the angle range of 30 degrees to 100 degrees of pitch angle, and is rarely used in the angle range of 0-30 degrees of pitch angle, or above 100 degrees of pitch angle. Therefore, in other embodiments, in order to improve the vitality of the design, based on the personalization of user usage habits, three-dimensional environment video materials can be obtained according to the angle range to which the posture angle of the electronic device belongs, and a second video corresponding to different angle ranges is generated based on the obtained three-dimensional environment video materials. The first video described above is replaced by the second video, and the angle range to which it belongs is determined according to the posture angle corresponding to the first posture information of the electronic device, and the first image frame to be displayed is determined in the second video associated with the angle range.

[0148] Taking the posture change of an electronic device based on the pitch angle as an example, as shown in FIG12 , for example, a mapping relationship between the posture angle of the electronic device and the image frames of videos with different perspectives can be established in advance by associating a top-down perspective video with the pitch angle interval [0°, 30°), a horizontal perspective video with the pitch angle interval [30°, 100°), and an upward perspective video with the pitch angle interval [100°, 180°]. In a specific application, after identifying the posture angle of the electronic device, the electronic device or its processor can determine the first image frame to display in the corresponding perspective video according to the angle interval to which the posture angle belongs, so that the video played on the display screen of the electronic device is smoother.

[0149] It is understandable that in the mapping method based on angle intervals shown in FIG12 , there may be a back-and-forth jitter problem at the critical angle due to perception differences. For example, at a critical angle pitch angle of 30 degrees, if the posture information obtained fluctuates at a pitch angle of 29 degrees / 30 degrees / 31 degrees, then the first video used will be repeatedly switched. In order to reduce or even avoid this situation, in the mapping method based on angle intervals shown in FIG12 , for example, an angle jitter threshold can be set to expand the critical points of different angle intervals from an angle value to a smaller critical angle interval, so that when the posture of the electronic device changes, different boundaries of the critical angle interval are triggered respectively according to the playback direction of at least one second video to present image frames of different videos on the display screen.

[0150] For example, taking the forward playback of a video based on the change in pitch angle as an example, the upward threshold corresponding to the pitch angle interval [0°, 30°) can be set to 25 degrees, and the downward threshold can be positioned at 35 degrees. If the posture change of the electronic device is from a pitch angle of 20 degrees to a pitch angle of 40 degrees, then at a pitch angle of 25 degrees, the image frame of the video from the bird's-eye view can be switched to the image frame of the video from the head-on view. Conversely, taking the reverse playback of a video based on the change in pitch angle as an example, if the posture change of the electronic device is from a pitch angle of 40 degrees to a pitch angle of 20 degrees, then at a pitch angle of 35 degrees, the image frame of the video from the head-on view can be switched to the image frame of the video from the bird's-eye view. It should be understood that the values ​​based on the jitter angles here are only examples and do not constitute a specific limitation on the implementation methods.

[0151] It is worth noting that in the above introduction, whether it is a smartphone or a tablet computer, the rotation or movement of the display screen is in a single form (for example, the so-called unfolded form). The display control method of the embodiment of the present application can also be applied to other posture changes, such as the shape change of the electronic device.

[0152] For example, as shown in FIG13 , taking a laptop computer as an example, during the process of the laptop computer gradually changing from a folded state to an unfolded state, different image frames of the associated first video are dynamically displayed on the display screen of the laptop computer.

[0153] Alternatively, as shown in FIG14A , taking a foldable phone as an example, when the foldable phone is placed horizontally and gradually changes from a folded state to a fully unfolded state, it unfolds about the Y-axis, and the display screen of the foldable phone dynamically displays different image frames of the associated first video (the content of the image frames is not shown, similar to the switching of image frames shown in FIG13 ). This is equivalent to the visual effect of a camera panning downward in an arc and rotating upward from a top view. Alternatively, as shown in FIG14B , if the foldable phone is in a fully unfolded state and changes from being close to the user's face to being farther away from the face (i.e., moving backward), the change in the image frames presented on the phone's display screen is similar to a camera moving backward and increasing its focal length.

[0154] Moreover, in the above introduction, the user uses the electronic device in the correct posture. The display control method of the embodiment of the present application is also applicable to the case where the user is lying down, lying on his side or in other postures. In these situations, the reference coordinate system or reference angle of the electronic device can be adjusted accordingly and the display control method of the present application can be applied to expand the scope of use of the method, so that no matter what posture the user is in when using the electronic device, the user's perspective change can be simulated based on the posture change of the electronic device, and different image frames can be dynamically displayed, giving the user a better integrated reality use effect.

[0155] In addition, in addition to generating the first video based on the video material collected based on the real three-dimensional environment as described above, in an embodiment of the present application, the electronic device or its processor can also obtain scene description information. The electronic device or its processor can generate and save the first video based on the scene description information and pre-rendering technology. Alternatively, when the electronic device or its processor determines that the posture of the electronic device has changed, the electronic device or its processor can generate the first image frame based on the first posture information and the scene description information. Specifically, for example, the first image frame is generated by real-time rendering technology based on the first posture information and the scene description information. Subsequently, when the posture of the electronic device changes, different image frames can be displayed on the display screen according to the posture change. The display control principle is similar to the above description, and you can refer to the relevant description above, which will not be repeated here.

[0156] For example, the scene description information may include weather information, such as rainy days, snowy days, strong winds, etc. When generating the first video based on the scene description information, similar to the method described above, the effect of a top view of the current weather can be displayed when the mobile phone is placed horizontally on the desktop, corresponding to a pitch angle of 0 degrees. When the mobile phone is rotated based on the pitch angle exceeding a specific preset value, a fade-in and fade-out effect is used to switch the weather view. For example, when the mobile phone is vertical and the display screen faces the user's face (i.e., forward), a horizontal view effect of the current weather is displayed. When the angle of the mobile phone exceeds a specific preset value, a fade-in and fade-out effect is used to switch the weather view. When the mobile phone is vertical and the display screen is facing the user's face (i.e., forward), a horizontal view effect of the current weather is displayed. When the angle of the mobile phone exceeds a specific preset value, a fade-in and fade-out effect is used to switch the weather view. When the mobile phone is vertical and the display screen is downward (for example, the pitch angle is greater than 90 degrees), a weather top view effect is displayed.

[0157] As shown in Figure 15, still taking the posture change of a smartphone based on the pitch angle as an example, in a scene associated with a rainy day, when the phone changes based on the pitch angle, for example, from a pitch angle of 40 degrees to a pitch angle of 70 degrees, and then to a pitch angle of 100 degrees, the content of the image frame displayed on the phone's display changes, from mainly including more raindrops falling on a horizontal plane (such as the ground or the horizontal plane where the phone's display is located), to including fewer raindrops, and finally to including more clouds.

[0158] It is worth noting that the display of the multi-perspective weather effects shown in Figure 15 is only an example. In other embodiments, rendering can also be performed from at least two dimensions to generate the required first image frame. For example, the longitudinal (e.g., corresponding to the pitch angle) posture of the electronic device can be changed to switch different visual state effects of the current weather scene (as shown in Figure 15), and the lateral (e.g., corresponding to the yaw angle) posture of the electronic device can be changed to change the weather scene according to the real-time weather conditions. Moreover, in a specific implementation, when generating the first video or the first image frame based on the scene description information, the real weather effect can be simulated by OpenGL technology and related algorithms, and a particle system and random factors can be added to the weather scene to increase the randomness of the scene, while also trying to avoid the probability of repeated scenes. This scene can get rid of the limitations of fixed scenes such as videos or pictures, and users can naturally interact with weather wallpapers, which helps to increase the fun of the scene.

[0159] In this way, through the above method, whether the first video is generated based on the three-dimensional environment video material, or the first image frame is generated based on the scene description information and / or real-time rendering technology, the dynamic display of the image frame can be switched according to the posture change of the electronic device, so that the user can experience the visual change effect equivalent to the real world as the posture of the electronic device changes, and obtain a better visual effect of integrating reality.

[0160] The embodiment of the present application can dynamically display different image frames in combination with the posture changes of the electronic device through the above method, so as to obtain a better visual effect of fusion of reality on the electronic device.

[0161] In other embodiments of the present application, an electronic device is disclosed, as shown in FIG16 , which may include: a touch screen 1601, wherein the touch screen 1601 includes a touch panel 1607 and a display screen 1608; one or more processors 1602; a memory 1603; one or more application programs (not shown); and one or more computer programs 1604. A sensor 1605 and the above-mentioned components may be connected via one or more communication buses 1606. The one or more computer programs 1604 are stored in the memory 1603 and configured to be executed by the one or more processors 1602. The one or more computer programs 1604 include instructions, which may be used to execute the various method steps in the corresponding embodiment of FIG5 .

[0162] An embodiment of the present application also provides a computer storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the human body posture detection method in the above-mentioned embodiment.

[0163] An embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the human body posture detection method in the above-mentioned embodiment.

[0164] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the human posture detection method for the touch screen in the above-mentioned method embodiments.

[0165] Among them, the electronic device, computer storage medium, computer program product or chip provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0166] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0167] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0168] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0169] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0170] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0171] The above content is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A display control method, characterized in that: include: When the posture of the electronic device changes, obtaining first posture information of the electronic device; Determining a first image frame according to the first posture information, wherein the first image frame belongs to a first video stored in the electronic device, or the first image frame is rendered according to a second image frame, and the second image frame is an image frame displayed on a display screen of the electronic device before the posture of the electronic device changes; Based on the change of the posture, the first image frame is displayed on a display screen of the electronic device.

2. The method according to claim 1, characterized in that: The first posture information is used to describe a first posture of the electronic device relative to a user's viewing angle, and the user is a user of the electronic device.

3. The method according to claim 1, characterized in that The first attitude information is represented by a device attitude angle of the electronic device, and the device attitude angle includes at least one of the following: a pitch angle, a roll angle, or a yaw angle.

4. The method according to any one of claims 1 to 3, characterized in that The determining the first image frame according to the first posture information includes: The first image frame is determined in the first video according to the first posture information and posture mapping information saved in association with the first video, wherein the posture mapping information is used to describe a mapping relationship between different postures of the electronic device and different image frames of the first video.

5. The method according to any one of claims 1 to 4, characterized in that The step of displaying the first image frame on a display screen of the electronic device based on the change in the posture comprises: Based on the change in the posture, the first image frame is displayed on the display screen of the electronic device through a preset animation curve, wherein the animation curve is associated with the playback frame rate of the first video and the posture change time of the electronic device.

6. The method according to any one of claims 1 to 5, characterized in that The first posture information is represented by a device posture angle of the electronic device, and the device posture angle includes at least one of the following: a pitch angle, a roll angle, or a yaw angle, and the first video includes at least one of the following: The camera angle video saved by the electronic device based on the posture change of the pitch angle in the same three-dimensional environment; or, The electronic device stores a video of the camera angle based on the posture change of the flip angle in the same three-dimensional environment; or, The electronic device saves the camera angle video based on the posture change of the yaw angle in the same three-dimensional environment.

7. The method according to any one of claims 1 to 6, characterized in that If the display screen of the electronic device is facing the user's face, and the first posture information is represented by the distance between the display screen of the electronic device and the user's head, the first video includes: a camera angle video saved by the electronic device based on the distance change in the same three-dimensional environment.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: The first video is generated based on the three-dimensional environment video material.

9. The method according to claim 8, characterized in that If the frame rate of the 3D environment video material is less than the required frame rate of the first video, generating the first video based on the 3D environment video material specifically includes: Based on the required frame rate of the first video, the missing third image frame of the three-dimensional environment video material is supplemented by pre-rendering technology and interpolation technology; The first video is generated based on the three-dimensional environment video material and the third image frame.

10. The method according to claim 8 or 9, characterized in that: If the three-dimensional environment video material is in a compressed format, generating the first video based on the three-dimensional environment video material specifically includes: The first video frame is generated based on the three-dimensional environment video material and a frame difference value associated with the three-dimensional environment video material.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: Get scene description information; The determining the first image frame according to the first posture information includes: The first image frame is generated according to the first posture information and the scene description information.

12. The method according to claim 11, characterized in that The scene description information includes weather information.

13. The method according to any one of claims 1 to 12, characterized in that The display screen of the electronic device includes a foldable display screen.

14. An electronic device, characterized in that: Includes display screen, processor and memory; The memory is used to store one or more computer programs; When the one or more computer programs stored in the memory are executed by the processor, the electronic device executes the method according to any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises a computer program, and when the computer program is executed on an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 13.

16. A chip, characterized in that: The chip is coupled to a memory and is configured to execute a computer program stored in the memory to perform the method according to any one of claims 1 to 13.

Citation Information

Patent Citations

  • Virtual reality image display method, virtual reality image display device and terminal equipment using method and device

    CN106249918A

  • Three-dimensional scene display method and device, readable storage medium and electronic equipment

    CN111429519A

  • Picture display method and device based on terminal posture, picture acquisition method and device based on terminal posture, storage medium and terminal

    CN113012290A

  • Three-dimensional image display method and device, electronic equipment and storage medium

    CN113238656A

  • Systems and methods for processing and displaying image data based on attitude information

    US20200221056A1