A folding screen device-based image processing method and folding screen device

Abstract

The application provides a kind of image processing method based on folding screen equipment and folding screen equipment, to improve the display quality of the shooting picture and screen utilization of folding screen equipment in various forms. In the method, when it is detected that the folding screen equipment is in the unfolded state, and in the scene where the main camera is used for shooting, the main camera can be called to shoot to obtain a main camera image, and at the same time, a wide-angle camera can also be called to shoot to obtain a wide-angle image. Then, after aligning the main camera image and the wide-angle image, the target image based on the main camera image and the wide-angle image can be obtained by cropping the wide-angle image based on the height of the main camera image. Compared with only the main camera image, the target image can have a wider viewing angle.

Description

Technical Field

[0001] This application relates to the field of electronic device technology, and in particular to an image processing method based on a foldable screen device and a foldable screen device. Background Technology

[0002] As the functions of electronic devices become increasingly sophisticated and diverse, their hardware forms are also becoming more varied. Taking mobile phones as an example, in addition to conventional candybar phones, there are also foldable screen phones, including but not limited to two-fold or three-fold screen phones, and so on.

[0003] For conventional candybar phones, the physical mounting angle of the camera module ensures that the long side of the camera sensor aligns with the long side of the screen, and the short side aligns with the short side of the screen, allowing the captured image to fully occupy the screen area. However, for foldable phones, the physical mounting angle of the camera sensor is designed for the folded state, similar to conventional candybar phones. But when the foldable phone is unfolded, the long side of the camera sensor may be perpendicular to the long side of the screen, raising the question of how to properly display the image. Summary of the Invention

[0004] This application provides an image processing method and a foldable screen device based on a foldable screen device, which can improve the display quality and screen utilization of images captured by foldable screen mobile phones in various forms.

[0005] A first aspect provides an image processing method based on a foldable screen device, wherein the foldable screen device is the executing entity of the method. In this method, a preset event is detected, the preset event indicating that the foldable screen device is in an unfolded state and a first camera is invoked for image acquisition; in response to the preset event, a first image is acquired through the first camera, and a second image is acquired through a second camera; wherein the focal length of the second camera is shorter than that of the first camera; a first target image is displayed on the screen, the first target image being obtained based on first information, the first information including a cropped second image; wherein the height of the first target image is equal to the height of the first image, and the width of the first target image is equal to the width of the second image.

[0006] In this method, the physical mounting angle of the camera sensor included in the camera module is typically designed with reference to the folded state of the foldable screen device. Therefore, to address various potential image display issues caused by the mismatch between the physical mounting angle of the camera sensor and the unfolded state of the foldable screen device, this method simultaneously utilizes two cameras with different focal lengths to acquire images upon detecting the scenario. The two images are then processed to obtain the final target image for display. Thus, this method can improve the screen utilization rate of the foldable screen device in its unfolded state.

[0007] In one possible design, the first information further includes the first image; the first target image is obtained by stitching together the first image and a cropped second image. For example, the first camera may be a main camera, and the first image corresponds to the main camera image; the second camera may be a wide-angle camera, and the second image corresponds to the wide-angle image.

[0008] In this design, the stitched target image not only possesses the rich detail of the main camera image but also enhances the field of view by combining it with a wide-angle image. Thus, in scenarios where the main camera is used for shooting, the wide-angle image can improve the field of view while maintaining image quality.

[0009] In one possible design, the first target image is obtained by: enlarging the second image based on the first image; replacing pixels in the enlarged second image that belong to the same display position with pixels included in the first image to obtain a third image, wherein the third image is a stitched image of the first image and the second image; and cropping the third image based on the height of the first image to obtain the first target image.

[0010] In this design, the accuracy of image stitching is improved by aligning the main image and the wide-angle image, and then replacing the pixels in the wide-angle image with the pixels included in the main image, thus creating a target image with a wider field of view.

[0011] In one possible design, the step of magnifying the second image based on the first image includes: performing image matching based on the first image and the second image to find M identical pixels, where M is a positive integer greater than 1; determining a first distance between a first pixel and a second pixel in the first image; wherein the first pixel and the second pixel are any two pixels from the M pixels; and magnifying the second image; wherein the distance between the first pixel and the second pixel in the magnified second image is the first distance.

[0012] In this design, by finding the same pixels in two images and adjusting the distance between the pixels, the wide-angle image and the main image after magnification can be guaranteed to have the same display precision, thereby improving the accuracy of image stitching.

[0013] In one possible design, the method further includes: displaying a second target image; wherein the second target image is obtained by image optimization of the first target image.

[0014] In this design, image quality can be improved by optimizing the target image obtained after image stitching. Furthermore, through segmented image processing, the first target image can be displayed first after shooting, thus improving imaging speed; and the second target image can be displayed after image optimization, thereby improving image quality.

[0015] In one possible design, before displaying the first target image on the screen, the method further includes: performing distortion correction processing on the second image. It is understood that the method provided in this application can employ strong distortion correction processing.

[0016] In this design, the method provided in this application is to expand the field of view of the main camera image by combining the wide-angle image. Therefore, the wide-angle image is cropped with reference to the height of the main camera image. Thus, the distortion correction process has little impact on the target image obtained in this application, and it generally belongs to the cropped area. Therefore, the quality of the target image can be further improved by distortion correction process.

[0017] In one possible design, the preset event includes taking a picture by calling the first camera when the foldable screen device is in the unfolded state; or, the preset event includes recording video by calling the first camera when the foldable screen device is in the unfolded state.

[0018] In this design, the method provided in the embodiments of this application can be applied not only to photography scenarios but also to video recording scenarios.

[0019] In one possible design, the preset event includes when the foldable screen device is in the unfolded state and the first camera is invoked to record video, and the first information further includes at least one of the following: the adjacent previous frame of the first image, the adjacent next frame of the first image, the adjacent previous frame of the second image, and the adjacent next frame of the second image.

[0020] In this design, since the video scene includes multiple frames, image optimization can be performed by combining the image information of previous and subsequent frames, thereby improving the quality of image processing.

[0021] A second aspect provides a foldable screen device, the device comprising multiple functional modules; the multiple functional modules interact to implement the methods performed by the foldable screen device in the first aspect and its various embodiments. The multiple functional modules can be implemented based on software, hardware, or a combination of software and hardware, and the multiple functional modules can be arbitrarily combined or divided based on specific implementations.

[0022] A third aspect provides a foldable screen device, including at least one processor and at least one memory, wherein the at least one memory stores computer program instructions, and when the device is running, the at least one processor executes the method executed by the foldable screen device described in the first aspect and its various embodiments.

[0023] The fourth aspect also provides a program product that, when run on a foldable screen device, causes the foldable screen device to perform the methods of any of the above aspects and their embodiments.

[0024] The fifth aspect also provides a readable storage medium storing a program that, when executed by a device, causes the foldable screen device to perform the method of the foldable screen device as described in any of the above aspects and their embodiments.

[0025] The sixth aspect also provides a chip for reading a program stored in a memory and executing the method performed by the foldable screen device in any of the above aspects and their embodiments.

[0026] A seventh aspect also provides a chip system including a processor for supporting a foldable screen device in implementing the methods performed by the foldable screen device in any of the above aspects and embodiments. In one possible design, the chip system further includes a memory for storing the necessary programs and data. The chip system may be composed of chips or may include chips and other discrete devices.

[0027] It should be noted that the beneficial effects of the various designs of the foldable screen devices provided in the second to seventh aspects of the embodiments of this application can be referred to the beneficial effects of any possible design in the first aspect, and will not be repeated here. Attached Figure Description

[0028] Figure 1A This diagram illustrates a scene where a photograph is being taken.

[0029] Figure 1B A schematic diagram is shown illustrating the transformation of a foldable phone from a folded state to an unfolded state.

[0030] Figure 1C This illustrates another scenario for taking the photograph;

[0031] Figure 1D This illustrates yet another scenario for taking the photograph;

[0032] Figure 2 A schematic diagram of the hardware structure of a possible foldable screen device is shown;

[0033] Figure 3 A software architecture block diagram of a foldable screen device provided in an embodiment of this application;

[0034] Figure 4 This is a schematic diagram illustrating a possible scenario for an image processing method based on a foldable screen device, as provided in an embodiment of this application.

[0035] Figure 5 This is a schematic diagram illustrating another possible scenario to which the image processing method based on a foldable screen device provided in this application embodiment may be applied;

[0036] Figure 6 This is a schematic diagram illustrating another possible scenario to which the image processing method based on a foldable screen device provided in this application embodiment may be applied;

[0037] Figure 7 This is a flowchart illustrating an image processing method based on a foldable screen device, as provided in an embodiment of this application. Detailed Implementation

[0038] The embodiments of this application will now be described in detail with reference to the accompanying drawings and examples.

[0039] With the rapid development of society, electronic devices such as mobile phones are becoming increasingly common. These devices not only have communication functions but also powerful processing capabilities, storage capacity, and camera functions. Electronic devices operate through an operating system (e.g., The operating system executes corresponding applications, allowing users to make calls, send text messages, browse the web, take photos, and record videos using electronic devices. Among these, the built-in camera module in the electronic device enables photo taking and video recording through multiple lenses with different focal lengths and a camera sensor. For example, lenses with different focal lengths can include, but are not limited to: wide-angle lenses, ultra-wide-angle lenses, standard lenses, telephoto lenses, prime lenses, zoom lenses, etc.

[0040] For example, Figure 1A This illustration depicts a scenario for taking a photo. A foldable phone is used as an example electronic device. Figure 1AAs shown in interface 10A, this is a schematic diagram of a foldable phone in its folded state. Interface 10A shows that the long side of the physical mounting angle of the camera sensor 101 is parallel to the long side of the foldable phone's screen when it is folded, and the short side is parallel to the short side of the foldable phone's screen when it is folded. It can be understood that the relative relationship between the forward image output size of the camera sensor 101 and the forward screen size of the foldable phone when it is folded corresponds to this.

[0041] Taking the origin of camera sensor 101 at the upper right corner as an example, the hardware abstraction layer (HAL) output can be shown as interface 10B. To achieve normal display of the captured image, the display direction of the HAL output can be turned upright, and the preview screen after being turned upright can be shown as interface 10C.

[0042] After the captured image is sent for display, it can be shown on the foldable phone screen in its folded state, as shown in Interface 10D. Interface 10D shows that the long side of the captured image is parallel to the long side of the foldable phone screen in its folded state, and the short side is parallel to the short side of the foldable phone screen in its folded state, indicating high screen utilization.

[0043] For example, Figure 1B A schematic diagram is shown illustrating the transformation of a foldable phone from a folded state to an unfolded state. Based on the physical relative relationship between the foldable phone and camera sensor 101 in the folded state shown in interface 10A, the physical relative relationship between the foldable phone and camera sensor 101 in the unfolded state can be as shown in interface 10E. It is understood that since the camera sensor 101 is hardware, once the physical mounting angle is determined, it cannot be changed at the hardware level. As shown in interface 10E, the long and short sides of the screen of the foldable phone in the unfolded state have changed compared to the folded state. In the unfolded state, the long side is horizontal, and the short side is vertical. At this time, the long side of the camera sensor 101 at its physical mounting angle is parallel to the short side of the screen of the foldable phone in the unfolded state, and the short side is parallel to the long side of the screen of the foldable phone in the unfolded state. That is, the physical relative relationship between the foldable phone and camera sensor 101 in the folded state is different from that in the unfolded state. In other words, the physical mounting angle of the camera sensor 101 is determined based on the foldable screen phone in the folded state. Therefore, the relative relationship between the forward image output size of the camera sensor 101 and the forward screen size of the foldable screen phone in the unfolded state does not correspond.

[0044] When a foldable phone is in its unfolded state, one possible display interface for showing the captured image is as follows: Figure 1C As shown in interface 10F, another possible display interface is as follows: Figure 1D The interface shown in 10H is shown in the image.

[0045] like Figure 1C As shown in the diagram, the HAL output image obtained by the camera sensor 101 is shown in interface 10B. The HAL output image is then rotated to obtain the preview image shown in interface 10C. Finally, the captured image is displayed on the screen of the foldable phone in its unfolded state, as shown in interface 10F. Interface 10F shows that when the foldable phone is in its unfolded state, the captured image is centered on the screen. However, because the size of the camera sensor's forward output image does not correspond to the size of the foldable phone's forward screen when unfolded, the captured image cannot fully utilize the screen, resulting in large black borders on the left and right sides of the screen, thus wasting screen space.

[0046] like Figure 1D As shown, compared to Figure 1C The preview image shown in interface 10C can be magnified according to the long side of the foldable phone's screen in the unfolded state, as shown in interface 10G. The magnified image shown in interface 10G is then cropped according to the screen size, resulting in interface 10H. As can be seen from interface 10H, the displayed image can fill the entire screen of the foldable phone in the unfolded state, but a significant amount of image information is lost, such as the image information in the cropped area of ​​interface 10G, which cannot be displayed on the screen.

[0047] In view of this, embodiments of this application provide an image processing method based on a foldable screen device. In this method, the foldable screen device captures two images using two lenses with different focal lengths, and then performs image processing such as image fusion on the two images to obtain a target image, which is then displayed. Therefore, the target image displayed on a foldable screen phone in its unfolded state not only retains more detailed image information but also provides a wider field of view, thereby improving screen utilization.

[0048] The technical solutions in this application can be applied to foldable screen devices, which can be any device capable of displaying an interface. For example, a foldable screen device can be a two-fold screen phone, a three-fold screen phone, a foldable display screen, or other foldable screen devices. It is understood that this application does not impose any limitations on the specific type of foldable screen device.

[0049] The embodiments of this application can be applied to foldable screen devices, and exemplary embodiments include, but are not limited to, devices equipped with... Alternatively, it could be a foldable screen device using another operating system. For example, a foldable screen phone could be one of the foldable screen phones described in the preceding embodiments.

[0050] Figure 2 A schematic diagram of a possible hardware structure for a foldable screen device is shown. The foldable screen device 200 includes: a radio frequency (RF) circuit 210, a power supply 220, a processor 230, a memory 240, an input unit 250, a display unit 260, an audio circuit 270, a communication interface 280, a wireless fidelity (Wi-Fi) module 290, a camera 2100, and other components. Those skilled in the art will understand that... Figure 2 The hardware structure of the foldable screen device 200 shown in the figure does not constitute a limitation on the foldable screen device 200. The foldable screen device 200 provided in the embodiments of this application may include more or fewer components than shown, may combine two or more components, or may have different component configurations. Figure 2 The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0051] The following is combined with Figure 2 The various components of the foldable screen device 200 are described in detail below:

[0052] The RF circuit 210 can be used for receiving and transmitting data during communication or a call. Specifically, after receiving downlink data from the base station, the RF circuit 210 sends it to the processor 230 for processing; additionally, it sends uplink data to be transmitted to the base station. Typically, the RF circuit 210 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc.

[0053] Furthermore, the RF circuit 210 can also communicate with other devices via a wireless communication network. The wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Messaging Service (SMS).

[0054] Wi-Fi technology is a short-range wireless transmission technology. The foldable screen device 200 can connect to an access point (AP) via the Wi-Fi module 290, thereby enabling access to the data network. The Wi-Fi module 290 can be used for receiving and sending data during communication.

[0055] The foldable screen device 200 can physically connect to other devices through the communication interface 280. Optionally, the communication interface 280 can be connected to the communication interfaces of other devices via a cable to enable data transmission between the foldable screen device 200 and other devices.

[0056] The foldable screen device 200 can also perform communication services and interact with other foldable screen devices. Therefore, the foldable screen device 200 needs to have data transmission capabilities, meaning that the foldable screen device 200 needs to include a communication module. Although Figure 2 The RF circuit 210, the Wi-Fi module 290, and the communication interface 280 are shown, but it is understood that the foldable screen device 200 contains at least one of the above-mentioned components or other communication modules (such as a Bluetooth module) for data transmission.

[0057] For example, when the foldable screen device 200 is a mobile phone, the foldable screen device 200 may include the RF circuit 210, and may also include the Wi-Fi module 290, or may include a Bluetooth module. Figure 2 (Not shown in the image); When the foldable screen device 200 is a tablet computer, the foldable screen device 200 may include the Wi-Fi module, or may include a Bluetooth module (not shown in the image); Figure 2(Not shown in the image); When the foldable screen device 200 is a smart home device, the foldable screen device 200 may include the Wi-Fi module 290, or may include a Bluetooth module (not shown in the image); Figure 2 (Not shown in the image).

[0058] The memory 240 can be used to store software programs and modules. The processor 230 executes various functional applications and data processing of the foldable screen device 200 by running the software programs and modules stored in the memory 240. Optionally, the memory 240 may mainly include a program storage area and a data storage area. The program storage area may store the operating system (mainly including the software programs or modules corresponding to the kernel layer, system layer, application framework layer, and application layer).

[0059] In addition, the memory 240 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0060] The input unit 250 can be used to receive editing operations on various types of data objects, such as numbers or characters, input by the user, and to generate key signal inputs related to user settings and function control of the foldable screen device 200. Optionally, the input unit 250 may include a touch panel 251 and other input devices 252.

[0061] The touch panel 251, also known as a touchscreen, can collect user touch operations on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 251), and drive corresponding connection devices according to a pre-set program. In this embodiment, the touch panel 251 can collect user operations on it. For example, the user operations could be opening a camera app, clicking on shooting controls, etc.

[0062] Optionally, the other input device 252 may include, but is not limited to, one or more of the following: a physical keyboard, an infrared sensor, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, a joystick, etc. For example, an infrared sensor can be used to acquire the user's air gesture operations.

[0063] The display unit 260 can be used to display information input by the user or information provided to the user, as well as various menus of the foldable screen device 200. The display unit 260 is the display system of the foldable screen device 200, used to present the interface and realize human-computer interaction. The display unit 260 may include a display panel 261. Optionally, the display panel 261 can be configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar form. In this embodiment, the display unit 260 can be used to display a user interface. For example, the user interface can display a target interface obtained based on the method provided in this embodiment; wherein the target interface is obtained by stitching together a main camera image and a wide-angle image.

[0064] The processor 230 is the control center of the foldable screen device 200. It connects to various components via various interfaces and lines, and executes software programs and / or modules stored in the memory 240, as well as calling data stored in the memory 240, to perform various functions and process data of the foldable screen device 200, thereby enabling various services based on the foldable screen device 200. In this embodiment, the processor 230 can be used to implement an image processing method based on a foldable screen device provided in this embodiment.

[0065] The foldable screen device 200 also includes a power supply 220 (such as a battery) for supplying power to various components. Optionally, the power supply 220 can be logically connected to the processor 230 through a power management system, thereby enabling the power management system to manage functions such as charging, discharging, and power consumption.

[0066] like Figure 2 As shown, the foldable screen device 200 also includes an audio circuit 270, a microphone 271, and a speaker 272, providing an audio interface between the user and the foldable screen device 200. The audio circuit 270 converts audio data into signals recognizable by the speaker 272 and transmits the signals to the speaker 272, where the speaker 272 converts them into sound signals for output. The microphone 271 collects external sound signals (such as human speech or other sounds) and converts the collected external sound signals into signals recognizable by the audio circuit 270, sending them to the audio circuit 270. The audio circuit 270 can also convert the signals transmitted by the microphone 271 into audio data, and then output the audio data to the RF circuit 210 for transmission to, for example, another foldable screen device, or output the audio data to the memory 240 for further processing.

[0067] like Figure 2As shown, the foldable screen device 200 also includes a camera module 2100, which can be used to take photos. In this embodiment, the camera module 2100 may include lenses with at least two focal lengths, for example, it may include a main camera and a wide-angle camera.

[0068] although Figure 2 As not shown in the diagram, the foldable screen device 200 may also include at least one sensor, which will not be described in detail here. The at least one sensor may include, but is not limited to, a pressure sensor, a barometric pressure sensor, an accelerometer, a distance sensor, a fingerprint sensor, a touch sensor, a temperature sensor, etc.

[0069] The operating system (OS) involved in this application embodiment is the most basic system software running on the foldable screen device 200. The software system of the foldable screen device 200 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment takes an operating system adopting a layered architecture as an example to illustrate the software architecture of the foldable screen device 200.

[0070] Figure 3 This is a software architecture block diagram of a foldable screen device provided in an embodiment of this application. Figure 3 As shown, the software architecture of a foldable screen device can be a layered architecture. For example, the software can be divided into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the operating system is divided into five layers, from top to bottom: the application layer, the application framework layer (framework, FWK), the runtime and system libraries, the kernel layer, and the hardware layer.

[0071] The application layer can include a series of application packages. For example... Figure 3 As shown, the application layer can include a user interface (UI), camera, photo album, settings, skin modules, third-party applications, etc. Third-party applications can include, for example, wireless local area network (WLAN) applications, music, call, Bluetooth, and video applications.

[0072] In one possible implementation, the application can be developed using Java, by calling the application programming interface (API) provided by the application framework layer. Developers can then interact with the underlying operating system layers (such as the hardware layer and kernel layer) to develop their own applications. This application framework layer primarily consists of a series of services and management systems within the operating system.

[0073] The application framework layer provides application programming interfaces and a programming framework for applications within the application layer. The application framework layer includes some predefined functions. For example... Figure 3 As shown, the application framework layer may include a view system, activity manager, window manager, content provider, phone manager, resource manager, notification manager, etc.

[0074] The Activity Manager manages the lifecycle of each application and provides commonly used navigation and back functions, offering an interactive interface for all program windows.

[0075] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0076] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0077] A view system includes both visual and non-visual controls, such as controls that display text and controls that display images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text message notification icon could include views that display text and views that display images.

[0078] The phone manager is used to provide communication functions for foldable screen devices. For example, it manages call status (including connection, hang-up, etc.).

[0079] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0080] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting alert sounds, vibrating on foldable devices, and flashing indicator lights.

[0081] The runtime includes the core libraries and the virtual machine. The runtime is responsible for the scheduling and management of the operating system.

[0082] The core library consists of two parts: one part contains the functionalities that the Java language needs to call, and the other part contains the core libraries of the operating system. The application layer and application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0083] A system library can include multiple functional modules. For example: a surface manager, a media framework, a 3D graphics processing library (e.g., OpenGL ES), a 2D graphics engine (e.g., SGL), etc.

[0084] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0085] The media framework supports playback and recording of various commonly used audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, and AMR.

[0086] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0087] A two-dimensional graphics engine is a drawing engine for two-dimensional drawing. A two-dimensional graphics engine can perform drawing operations, such as drawing target images obtained through the embodiments of this application on the screen.

[0088] In some embodiments, a 3D graphics processing library can be used to draw 3D motion trajectory images, and a 2D graphics engine can be used to draw 2D motion trajectory images.

[0089] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0090] The hardware layer can include various types of sensors, such as accelerometers, gravity sensors, and touch sensors.

[0091] Typically, foldable screen devices can run multiple applications simultaneously. In simpler cases, one application corresponds to one process; in more complex cases, one application can correspond to multiple processes. Each process has a unique process ID.

[0092] It should be understood that in the embodiments of this application, "at least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, or a, b, and c, where a, b, and c can be single or multiple. "Multiple" refers to two or more. "And / or" is used to describe the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0093] In addition, it should be understood that in the description of this application, the words "first" and "second" are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance or order.

[0094] It should be understood that the hardware structure of foldable screen devices can be as follows: Figure 2 As shown, the software system architecture can be as follows: Figure 3 As shown, the software programs and / or modules corresponding to the software system architecture in the foldable screen device can be stored in the memory 240, and the processor 230 can run the software programs and applications stored in the memory 240 to execute the flow of an image processing method based on a foldable screen device provided in this application embodiment.

[0095] The method provided in this application can be applied to scenarios involving image processing methods based on foldable screen devices. Specifically, this application can be applied to scenarios where a foldable screen device in its unfolded state is used for taking photos or videos. Exemplary scenarios for taking photos or videos include, but are not limited to, the following: scenarios where a camera application (APP) is used for taking photos or videos, scenarios where photos or video calls are made in instant messaging software, and scenarios where photos or videos are taken using a camera control in a search-type APP. It is understood that, in implementing this application, it is not limited to any scenario where a camera module can be used for taking photos or videos.

[0096] It should be noted that the following describes several possible specific scenarios based on the methods provided in the embodiments of this application. It is understood that the methods provided in the embodiments of this application are not limited to the scenarios described below. Furthermore, the following embodiments use a photo-taking scenario as an example; a video recording scenario can be understood as processing multiple frames of images, and the processing of each frame can be found in the following description of the implementation process for the photo-taking scenario.

[0097] For example, Figure 4This is a schematic diagram illustrating a scenario of an image processing method based on a foldable screen device, provided in an embodiment of this application. Figure 1B The example shown in Interface 10E illustrates a foldable phone in its unfolded state. When the unfolded phone detects an image capture operation, it responds by using both the main camera (also known as the "main camera," "main lens," or "standard lens") and the wide-angle camera (also known as the "wide-angle lens") to capture images. The main camera can produce a HAL image as shown in Interface 40A1, and the wide-angle camera can produce a HAL image as shown in Interface 40A2. Understandably, at the same resolution, the image obtained through the main camera has the advantage of rich detail but a smaller field of view (FOV). The image obtained through the wide-angle camera has the advantage of a larger FOV, but due to lens characteristics, there is geometric distortion at the image edges. Furthermore, a wide-angle camera is a camera with a shorter focal length and a wider field of view than the main camera.

[0098] For example, image acquisition operations include, but are not limited to, one of the following operations: opening a camera app, clicking a video call control in an instant messaging software, or clicking a camera control in any app that includes a camera control.

[0099] Based on the HAL output image from the main camera shown in interface 40A1, the display orientation of the HAL output image is rotated to the correct direction. The preview image after rotation is shown in interface 40B1. Similarly, based on the HAL output image from the wide-angle camera shown in interface 40A2, the display orientation of the HAL output image is rotated to the correct direction. The preview image after rotation is shown in interface 40B2.

[0100] Based on the main camera image and the wide-angle image obtained from the wide-angle camera, image fusion can be performed to obtain a stitched image. As shown in interface 40C, by enlarging the wide-angle image based on the main camera image, image matching and stitching between the wide-angle image and the main camera image can be performed to achieve image fusion.

[0101] (A) Image Matching

[0102] One possible implementation is image matching based on feature points; these feature points could be, for example, human eye feature points, mouth feature points, hair feature points, etc. Optionally, the foldable phone can identify identical pixels based on the feature descriptors of the wide-angle and main camera images, determining that the vector values ​​are the same. This allows for the identification of multiple identical pixels from both the wide-angle and main camera images to achieve image matching.

[0103] Additionally, the wide-angle image can be magnified based on multiple sets of identical pixels in both the wide-angle and main image. For example, pixel 1A in the wide-angle image and pixel 1B in the main image are identified as identical pixels, as are pixel 2A in the wide-angle image and pixel 2B in the main image. Then, the distance between pixel 2A and pixel 2B in the main image is determined to be 2AB, and the distance between pixel 1A and pixel 1B in the wide-angle image is magnified to 2AB. It can be understood that by adjusting the distance between pixels, the wide-angle image can be magnified, allowing the magnified wide-angle image to be aligned with the main image. Image alignment can be understood as ensuring that the distance between any two pixels present in both the wide-angle and main images is the same in the magnified wide-angle image as it is in the main image.

[0104] (B) Image stitching

[0105] In one possible approach, the wide-angle image and the main camera image can be stitched together based on pixels extracted from the image edges of the main camera image. Optionally, based on pixels belonging to the image edges of the main camera image, the main camera display area corresponding to the main camera image can be determined in the magnified wide-angle image; then, the pixels in the magnified wide-angle image belonging to the main camera display area are replaced with the pixels included in the main camera image to obtain the stitched image.

[0106] like Figure 4 As shown in interface 40C, this is the stitched image obtained after image fusion. The central region 401 of the stitched image can be the main camera display area in the magnified wide-angle image, and its pixels are based on the main camera image. The pixels in other regions of the stitched image besides the central region 401 can be based on the wide-angle image. Thus, by stitching the wide-angle and main camera images after image matching, the resulting stitched image not only retains the rich detail of the main camera image but also takes advantage of the large field of view (FOV) of the wide-angle camera.

[0107] Based on the stitched image obtained from interface 40C, and using the height of the main camera image as a reference, the stitched image is cropped to obtain the target image shown in region 402 of interface 40D. The foldable phone then displays the cropped target image. Correspondingly, the foldable phone can display interface 40E, in which the target image is displayed.

[0108] As can be seen from interface 40E, compared to Figure 1C The interface is on page 10F, or Figure 1DThe interface 10H in this application provides a target image displayed on a foldable screen phone in the unfolded state using the method provided in this application embodiment. This image not only retains the main camera image with rich details obtained through the main camera, but also stitches together a wide-angle image with a larger FOV to obtain a target image that can fill or almost fill the screen, thereby increasing the screen's screen ratio and improving screen utilization.

[0109] Using the height of the target image equal to the height of the screen as a reference, there are several possible display results:

[0110] (1) When the width of the target image is the same as the width of the screen, the target image can fill the entire screen.

[0111] (2) When the width of the obtained target image is smaller than the width of the screen, there may be images on the left and right sides of the screen, such as... Figure 1C The black border shown is smaller than the one shown. Figure 1C The black border shown.

[0112] (3) When the width of the target image is greater than the width of the screen, the target image can be displayed based on the premise that its width fills the screen. It is understandable that black borders may exist at the top and bottom of the screen, but their area is smaller than [the screen width]. Figure 1C The area of ​​the black border shown.

[0113] It should be noted that, Figure 4 Interfaces 10E, 40A1 to 40D are screens for easy understanding of the methods provided in the embodiments of this application. In actual implementation, they do not need to be displayed on the foldable screen phone when it is in the unfolded state.

[0114] For example, Figure 5 This is another schematic diagram of an image processing method based on a foldable screen device provided in an embodiment of this application. Figure 5 For reference, see interfaces 10E, 40A1, 40A2, 40B1, and 40B2. Figure 4 The details described in the text will not be repeated here.

[0115] and Figure 4 The difference lies in that, based on the wide-angle image shown in interface 40B2, image cropping can be performed after magnification to obtain a cropped wide-angle image, as shown in area 501 of interface 50A. The cropping process can be determined based on the height of the main camera image.

[0116] For example, image matching can first be performed based on feature points; these feature points could be, for example, human eye feature points, mouth feature points, hair feature points, etc. Optionally, the foldable phone can determine the same pixel point by using the feature descriptors of the wide-angle image and the main camera image, and if the vector values ​​are the same. In this way, multiple identical pixels can be found from both the wide-angle image and the main camera image to achieve image matching.

[0117] Then, based on multiple sets of identical pixels in the wide-angle image and the main image, the wide-angle image can be magnified. For example, pixel 1A in the wide-angle image and pixel 1B in the main image are determined to be identical pixels, and pixel 2A in the wide-angle image and pixel 2B in the main image are also determined to be identical pixels. Then, the distance between pixel 2A and pixel 2B in the main image is determined to be 2AB, and the distance between pixel 1A and pixel 1B in the wide-angle image is magnified to 2AB. It can be understood that by adjusting the distance between pixels, the wide-angle image can be magnified, thereby enabling image alignment between the magnified wide-angle image and the main image. Image alignment can be understood as ensuring that the distance between any two pixels that exist in both the wide-angle and main images is the same in the magnified wide-angle image as it is in the main image.

[0118] In this way, based on the magnified wide-angle image, image cropping can be performed according to the height of the main image. Optionally, the top and bottom edges of the main image can be identified in the magnified wide-angle image; then, cropping can be performed based on the top and bottom edges of the main image.

[0119] After cropping the wide-angle image, image fusion is performed based on the cropped wide-angle image and the main camera image to obtain the target image as shown in interface 50B. For example, image fusion may include, but is not limited to, image stitching. The target image shown in interface 50B is obtained by stitching together the main camera image and the wide-angle image. As can be seen from interface 50B, the pixels of the image in the central region 502 are obtained based on the main camera image, while the pixels of the images outside the central region 502 are obtained based on the wide-angle image.

[0120] In addition, among some possible implementations, Figure 4 or Figure 5The scenario described can be the camera in standard shooting mode. It can also be understood as a scenario where the camera uses the main camera to take photos or videos. Using the method provided in this application embodiment, when a preset event is detected that the foldable phone is in an unfolded state and taking photos or videos in standard shooting mode, image processing using the method provided in this application embodiment can be triggered. This allows for simultaneous image acquisition using the wide-angle camera, thereby completing the field of view of the main camera image and improving screen utilization.

[0121] In addition, in some possible recording scenarios, besides based on Figure 4 or Figure 5 The implementation process described here not only performs image processing on each frame of the video, but also optimizes the image by looking at the adjacent frames before and after the current frame.

[0122] For example, the current frame's image can be optimized using the main camera images of adjacent frames. For instance, if the main camera images of adjacent frames include pixels not present in the current frame, then the current frame can be replaced or supplemented using these pixels from the adjacent frames. Optionally, if the pixels in the target image from the wide-angle image include pixels from the main camera images of adjacent frames, then the pixels from the main camera images of adjacent frames are used to replace the pixels from the wide-angle image. Alternatively, if the pixels in the target image from the wide-angle image do not include pixels from the main camera images of adjacent frames, then the pixels from the main camera images of adjacent frames are used to supplement the target image of the current frame.

[0123] Another example is that the image of the current frame can be optimized by using wide-angle images of adjacent frames. For example, if the wide-angle images of adjacent frames include pixels that are not included in the current frame, then these pixels from the adjacent frames can be added to the image of the current frame.

[0124] Due to the characteristics of wide-angle cameras, some distortion may exist at the edges of wide-angle images. Therefore, distortion correction processing is usually required for wide-angle images. For example, Figure 6 This is another schematic diagram illustrating a scenario of an image processing method based on a foldable screen device provided in an embodiment of this application. For example... Figure 6 As shown in interface 60A, due to the screen size of the foldable phone, the area of ​​the wide-angle image other than the central area 601 is cropped and cannot be displayed on the screen. Therefore, the method provided in this application embodiment can use a more powerful distortion correction algorithm, which not only does not affect the display effect on the screen, but also improves the stitching accuracy of the main camera image and the wide-angle image.

[0125] In other possible scenarios, combined Figure 4As shown in interface 40C, during image processing, the wide-angle image needs to be magnified, which can lead to a loss of some image details. Therefore, the method provided in this application embodiment can also optimize the image of regions belonging to the wide-angle image. For example, image optimization can be performed on regions belonging to the wide-angle image using the main camera image. This application embodiment does not limit the image optimization method. For example, image optimization can employ artificial intelligence algorithms, etc.

[0126] In some potential scenarios, foldable phones can employ a segmented shooting approach for image processing during photography. Segmented shooting is a key feature in operating system camera development and can involve two stages.

[0127] Phase 1: Camera photos can be used to output low-quality images, which can be used as thumbnails or previews to improve the user's perception of the shooting speed.

[0128] For example, the first stage may include, but is not limited to: Figure 4 The image stitching process shown in interface 40C and such Figure 4 The image cropping process is shown in the 40D interface.

[0129] For example, the first stage may also include: such as Figure 5 The image cropping process shown in interface 50A is described.

[0130] It is understandable that through the first stage of processing, the target image can be displayed on the screen at a normal size with a fast display speed, but the image quality is low.

[0131] The second stage: The camera takes pictures to obtain high-quality images, so as to ensure that the final image quality reaches the level of the system camera and build the camera's performance competitiveness.

[0132] For example, the second stage may include, but is not limited to: color difference correction processing, such as... Figure 6 The distortion correction process is shown.

[0133] For example, the second stage may also include, but is not limited to: Figure 5 The image fusion processing and color difference correction processing shown are as follows: Figure 6 The distortion correction process is shown.

[0134] It's understandable that the second stage of processing allows not only the target image to be displayed on the screen, but also a high-quality image. This optimizes the system's camera response latency, thereby improving the user experience.

[0135] Based on the content described in the foregoing embodiments, the method provided in this application embodiment, when a foldable screen device in the unfolded state takes a photo or records a video, not only calls the main camera but also the wide-angle camera. This can achieve the completion of the main camera image through the wide-angle image, thereby not only retaining the main camera image with more details but also including part of the content of the wide-angle image with a larger FOV, which can improve screen utilization.

[0136] In another alternative embodiment, Figure 7 Another flowchart illustrates an image processing method based on a foldable screen device, provided as an embodiment of this application. This method can be executed by the foldable screen device. (See reference...) Figure 7 The method includes the following steps:

[0137] Step 701: A preset event is detected. The preset event is used to indicate that when the foldable screen device is in the unfolded state, the first camera is invoked to acquire images.

[0138] Step 702: In response to the preset event, acquire a first image through the first camera and acquire a second image through the second camera; wherein the focal length of the second camera is shorter than the focal length of the first camera.

[0139] For example, the first camera can be the main camera mentioned earlier, and the second camera can be the wide-angle camera mentioned earlier.

[0140] For example, the first image can be the main image mentioned earlier, and the second image can be the wide-angle image mentioned earlier.

[0141] Step 703: Display a first target image on the screen. The first target image is obtained based on first information, which includes a cropped second image. The height of the first target image is equal to the height of the first image, and the width of the first target image is equal to the width of the second image.

[0142] For example, the first target image can be Figure 4 The interface 40E, or Figure 5 In interface 50A, area 501, or... Figure 5 The interface is 40E.

[0143] It should be noted that this application Figure 7 The image processing method based on foldable screen devices shown can be referred to in the above embodiments of this application for specific implementation, and repeated parts will not be described again.

[0144] Based on the above embodiments, this application also provides a foldable screen device, which includes multiple functional modules. These multiple functional modules interact to implement the functions performed by the foldable screen device in the methods described in the embodiments of this application. The multiple functional modules can be implemented based on software, hardware, or a combination of both, and can be arbitrarily combined or divided based on specific implementations. For example, [the following is an example of implementation details]. Figure 7 Steps 701 to 703 are performed by the foldable screen device in the illustrated embodiment.

[0145] Based on the above embodiments, this application also provides a foldable screen device, which includes at least one processor and at least one memory. The at least one memory stores computer program instructions. When the foldable screen device is running, the at least one processor executes the functions performed by the foldable screen device in the various methods described in the embodiments of this application. For example, when executing... Figure 7 Steps 701 to 703 are performed by the foldable screen device in the illustrated embodiment.

[0146] Based on the above embodiments, this application also provides a computer program product, which includes a computer program (also referred to as code or instructions) that, when run, causes a computer to perform the methods described in the embodiments of this application.

[0147] Based on the above embodiments, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, causes the computer to perform the methods described in the embodiments of this application.

[0148] Based on the above embodiments, this application also provides a chip for reading computer programs stored in a memory to implement the methods described in the embodiments of this application.

[0149] Based on the above embodiments, this application provides a chip system including a processor for supporting a computer device in implementing the methods described in the embodiments of this application. In one possible design, the chip system further includes a memory for storing necessary programs and data of the computer device. This chip system may be composed of chips or may include chips and other discrete devices. Those skilled in the art will understand that the embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0151] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0152] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0153] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of protection of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image processing method based on a foldable screen device, characterized in that, The method includes: A preset event is detected, which is used to indicate that the foldable screen device is in the unfolded state and to call the first camera to capture images; In response to the preset event, a first image is acquired through the first camera, and a second image is acquired through the second camera; wherein the focal length of the second camera is shorter than the focal length of the first camera; A first target image is displayed on the screen. The first target image is obtained based on first information, which includes a cropped second image. The height of the first target image is equal to the height of the first image, and the width of the first target image is equal to the width of the second image.

2. The method according to claim 1, characterized in that, The first information also includes the first image; The first target image is obtained by stitching together the first image and the cropped second image.

3. The method according to claim 2, characterized in that, The first target image is obtained in the following manner: The second image is magnified based on the first image; The pixels included in the first image are used to replace the pixels in the magnified second image that belong to the same display position to obtain a third image, which is a spliced ​​image of the first image and the second image. The third image is cropped based on the height of the first image to obtain the first target image.

4. The method according to claim 3, characterized in that, The step of magnifying the second image based on the first image includes: Based on the first image and the second image, image matching is performed to find M identical pixels, where M is a positive integer greater than 1; Determine a first distance between a first pixel and a second pixel in the first image; wherein the first pixel and the second pixel are any two pixels among the M pixels; The second image is magnified; wherein, in the magnified second image, the distance between the first pixel and the second pixel is the first distance.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Display a second target image; wherein the second target image is obtained by optimizing the first target image.

6. The method according to any one of claims 1 to 5, characterized in that, Before displaying the first target image on the screen, the method further includes: The second image is then subjected to distortion correction processing.

7. The method according to any one of claims 1 to 6, characterized in that, The preset event includes taking a picture by calling the first camera when the foldable screen device is in the unfolded state; or, The preset event includes calling the first camera to record video when the foldable screen device is in the unfolded state.

8. The method according to claim 7, characterized in that, The preset events include when the foldable screen device is in the unfolded state and when the first camera is called to record video, and the first information also includes at least one of the following: the adjacent previous frame of the first image, the adjacent next frame of the first image, the adjacent previous frame of the second image, and the adjacent next frame of the second image.

9. A foldable screen device, characterized in that, It includes at least one processor coupled to at least one memory, the at least one processor being configured to read a program stored in the at least one memory to execute the method as described in any one of claims 1-8.

10. A readable storage medium, characterized in that, The readable storage medium stores instructions that, when executed on a foldable screen device, cause the foldable screen device to perform the method as described in any one of claims 1-8.

Images