Image processing method and related apparatus thereof

Abstract

The application relates to the field of image processing, and provides an image processing method and related equipment thereof, which is applied to an electronic device including a first camera and a second camera, and the method includes the following steps: starting a camera application program; displaying a first image, which is obtained by collecting an image by the first camera; receiving a first zooming operation; displaying and saving a second image, which is obtained by collecting an image by the first camera, performing module segmentation on the image by using a parallax between the image and an image collected by the second camera, correcting each module by using a corresponding warp matrix, and splicing the modules. The warp correction and splicing between the modules can realize panoramic depth alignment, and then smooth zooming and smooth switching of the camera can be realized.

Description

Technical Field

[0001] This application relates to the field of image processing, specifically to an image processing method and related equipment. Background Technology

[0002] With the development of shooting functions in electronic devices, camera applications are becoming increasingly widespread. To achieve a better photography experience, current electronic devices typically feature multiple cameras, each with a different focal length.

[0003] During shooting, in response to user input, electronic devices can zoom by switching between cameras with different focal lengths. They can also combine digital zoom to process the captured images to meet various high-magnification shooting scenarios. However, during zooming, due to differences in the depth of the subject, such as the influence of foreground and background, there are abrupt changes in the image content and size between different cameras, making smooth camera switching impossible. Furthermore, digital zoom can only magnify and crop the image content in the center of the field of view, which is very limited.

[0004] Therefore, how to smoothly switch cameras during the shooting process has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides an image processing method and related equipment, which can achieve full depth alignment by performing warp correction between modules and then stitching them together, thereby enabling smooth zoom and smooth switching of cameras.

[0006] In a first aspect, an image processing method is provided, applied to an electronic device, the electronic device including a first camera and a second camera, the method including: opening a camera application; displaying a first image, the first image being obtained by capturing an image from the first camera; receiving a first zoom operation; displaying and saving a second image, the second image being obtained by segmenting the image captured by the first camera into modules using the parallax between the image captured by the first camera and the image captured by the second camera, and then stitching the images together after correcting each module using its corresponding warp matrix.

[0007] In the embodiments of this application, disparity statistics are performed on the images captured by the first camera and the images captured by the second camera; then, the images captured by the first camera and the images captured by the second camera are segmented into modules based on the disparity distribution information; based on each module after segmentation of the images captured by the first camera and the images captured by the second camera, a warp matrix between each pair of corresponding modules is calculated; each module included in the image captured by the first camera is corrected in combination with its corresponding warp matrix, and the stitched second image is then displayed. This application segments the image based on disparity (depth of field) and implements warp correction for each module, thereby enabling warping of both foreground and background, achieving full depth of field alignment, and thus ensuring smooth zooming and smooth switching during camera switching.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: receiving a second zoom operation; displaying and saving a third image, the third image being obtained from an image captured by the second camera.

[0009] In the embodiments of this application, smooth switching between multiple cameras can be achieved.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the image captured by the first camera is used for module segmentation based on the disparity between the image captured by the second camera and the image captured by the second camera, including: acquiring a first initial image captured by the first camera and a second initial image captured by the second camera; performing feature point detection and registration on the first initial image and the second initial image to obtain multiple sets of matching inter-frame feature point pairs; calculating the disparity of each pair of inter-frame feature point pairs to obtain a disparity map; statistically analyzing and classifying the disparity distribution information of the disparity map to determine multiple classes; and performing module segmentation on both the first initial image and the second initial image based on the disparity belonging to the same class.

[0011] Module segmentation, also known as image segmentation, is used to divide images captured by a camera into different image blocks based on parallax. Furthermore, the warp matrix corresponding to each module is unique.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: calculating a first warp matrix between the modules corresponding to the first initial image and the second initial image based on the inter-frame feature point pairs and in combination with the segmented modules; and correcting the modules after the first initial image has been segmented based on the first warp matrix corresponding to each module.

[0013] In this embodiment, after the first initial image and the second initial image are segmented into modules, the first warp matrix corresponding to each module is used for correction. This is equivalent to aligning the first initial image and the second initial image by dividing the blocks based on the disparity between the first initial image and the second initial image.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, after module segmentation, the method further includes: adding a tolerance range around the module, wherein the size and position of the tolerance range are preset.

[0015] In this embodiment, a certain amount of margin is added around the module, so that there is an overlapping area between adjacent modules. This makes it easier to splice the modules together later and can better connect the modules together.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: performing feature point matching on two adjacent modules in the first initial image to obtain multiple sets of matched intra-frame feature point pairs; calculating a second warp matrix between the two adjacent modules based on the intra-frame feature point pairs; and correcting the modules after the first initial image has been segmented based on the second warp matrix corresponding to each module.

[0017] In this embodiment, feature point matching and warp correction are performed on adjacent modules of the first initial image. This is equivalent to aligning the internal content of the first initial image by dividing the blocks based on the similarity between adjacent modules of the first initial image.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, when calculating the second warp matrix between the two adjacent modules, the method includes: filtering feature point pairs located within the tolerance range and feature point pairs located outside the tolerance range according to a preset ratio, wherein the proportion of feature point pairs located within the tolerance range is greater than the proportion of feature point pairs located outside the tolerance range.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes, before calculating the vector formed by each pair of said feature points, removing feature point pairs that are mismatched.

[0020] In the embodiments of this application, after testing, the number of correctly registered feature points is greater than the number of incorrectly registered feature points in each scenario. Therefore, in order to improve processing efficiency and the effect of subsequent processing, the incorrectly registered feature points can be removed, and only the correctly registered feature point pairs can be retained.

[0021] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: overlapping the increased tolerance range during splicing; aligning the overall size after splicing and modifying the edges.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: responding to the first zoom operation; determining that the zoom ratio of the electronic device satisfies the zoom ratio range of the first camera foreground display and the second camera for background operation.

[0023] It should be understood that "foreground display" refers to the image captured by the camera being used for display; "background operation" refers to the image captured by the camera being used for image processing, not for display.

[0024] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: responding to the second zoom operation; determining that the zoom ratio of the electronic device satisfies the zoom ratio range of the first camera running in the background or being turned off, while the second camera is displayed in the foreground.

[0025] In conjunction with the first aspect, in some implementations of the first aspect, the first zoom operation or the second zoom operation includes: a two-finger reverse swipe operation, a swipe operation on the displayed zoom control, a voice operation, or an air gesture operation.

[0026] In conjunction with the first aspect, in some implementations of the first aspect, the first camera includes a wide-angle camera, and / or the second camera includes a telephoto camera or an ultra-wide-angle camera.

[0027] In conjunction with the first aspect, in some implementations of the first aspect, the first zoom operation and the second zoom operation are consecutive zoom operations.

[0028] In a second aspect, an electronic device is provided, comprising: one or more processors, a memory, a first camera, and a second camera; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, and the one or more processors calling the computer instructions to cause the electronic device to execute:

[0029] Open the camera application; display the first image, which is the image captured by the first camera; receive the first zoom operation; display and save the second image, which is the image captured by the first camera. The second image is obtained by segmenting the image into modules using the parallax between the image captured by the first camera and the image captured by the second camera, and then stitching them together after correcting each module using its corresponding warp matrix.

[0030] It should be understood that the extensions, limitations, explanations and descriptions of the relevant content in the first aspect above also apply to the same content in the second aspect.

[0031] Thirdly, a chip system is provided, the chip system being applied to an electronic device, the chip system including one or more processors, the processors being configured to invoke computer instructions to cause the electronic device to perform any of the image processing methods in the first aspect.

[0032] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing computer program code, which, when executed by an electronic device, causes the electronic device to perform any of the image processing methods in the first aspect.

[0033] Fifthly, a computer program product is provided, the computer program product comprising: computer program code, which, when executed by an electronic device, causes the electronic device to perform any of the image processing methods in the first aspect. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of a hardware system for an electronic device applicable to this application;

[0035] Figure 2 This is a schematic diagram of a software system applicable to an electronic device of this application;

[0036] Figure 3 This is a schematic diagram of the arrangement of multiple cameras on an electronic device according to an embodiment of this application;

[0037] Figure 4 This is a schematic diagram illustrating the zoom ratio corresponding to different types of cameras provided in this application embodiment;

[0038] Figure 5 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0039] Figure 6 This is a schematic diagram of images captured by the main camera and the telephoto camera provided in the embodiments of this application;

[0040] Figure 7 This is a set of schematic diagrams of image streams involved in existing technologies;

[0041] Figure 8 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0042] Figure 9 This is a schematic flowchart of another image processing method provided in the embodiments of this application;

[0043] Figure 10 These are a set of schematic diagrams provided in the embodiments of this application;

[0044] Figure 11 This is another set of schematic diagrams provided in the embodiments of this application;

[0045] Figure 12 This is another set of schematic diagrams provided in the embodiments of this application;

[0046] Figure 13 This is another set of schematic diagrams provided in the embodiments of this application;

[0047] Figure 14 This is a schematic flowchart of another image processing method provided in the embodiments of this application;

[0048] Figure 15 This is another set of schematic diagrams provided in the embodiments of this application;

[0049] Figure 16 This is a diagram showing the correspondence between camera operation and zoom ratio provided in an embodiment of this application;

[0050] Figure 17 This refers to a set of image streams involved in the embodiments of this application;

[0051] Figure 18 This is a schematic flowchart of another image processing method provided in the embodiments of this application;

[0052] Figure 19 This is a schematic diagram illustrating an application scenario for zoom switching provided in an embodiment of this application;

[0053] Figure 20 This is a schematic diagram of an application scenario for zoom switching provided in an embodiment of this application. Detailed Implementation

[0054] In the embodiments of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0055] First, some terms used in the embodiments of this application will be explained to facilitate understanding by those skilled in the art.

[0056] 1. Field of View (FOV): In optical instruments, the FOV is the angle between the two edges of the lens, representing the maximum range through which the image of the target object can pass through the lens. The size of the FOV determines the field of view of the optical instrument. The larger the FOV, the wider the field of view, but the lower the optical magnification. In other words, objects beyond this angle will not be captured by the lens. A shorter focal length results in a wider horizontal field of view, and thus a smaller image. The horizontal field of view narrows as the focal length increases, while the size of the object being photographed increases accordingly.

[0057] 2. Registration refers to the matching of geographic coordinates between different images obtained using different imaging methods within the same region. This includes three aspects: geometric correction, projection transformation, and ensuring the images are displayed at the same scale.

[0058] 3. Zoom ratio: Zoom ratio is used to indicate the zoom level of an electronic device when shooting.

[0059] 4. Optical flow algorithm: Optical flow refers to the instantaneous velocity of pixels moving on the imaging plane of a spatially moving object. The optical flow algorithm utilizes the temporal changes of pixels in an image sequence and the correlation between adjacent frames to find the correspondence between the previous and current frames, thereby calculating the motion information of objects between adjacent frames.

[0060] For example, point-by-point matching or tracking of a set of points can be performed on an image. By calculating the offset of pixels on the image, an optical flow field is formed, and pixel-level image registration can be performed using this optical flow field.

[0061] The above is a brief introduction to the terms used in the embodiments of this application, and will not be repeated below.

[0062] The hardware system, software system, and application scenarios of the electronic device provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0063] For example, electronic device 100 may be a mobile phone, smart screen, tablet computer, wearable electronic device, in-vehicle electronic device, augmented reality (AR) device, virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), projector, etc. This application embodiment does not limit the specific type of electronic device 100.

[0064] See Figure 1The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0065] It should be noted that, Figure 1 The structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include... Figure 1 The components shown may include more or fewer components, or the electronic device 100 may include... Figure 1 The components shown may be a combination of certain components, or the electronic device 100 may include... Figure 1 Sub-components of some of the components shown. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.

[0066] Processor 110 may include one or more processing units. For example, processor 110 may include at least one of the following processing units: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and neural network processing unit (NPU). These different processing units may be independent devices or integrated devices.

[0067] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0068] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0069] For example, the processor 110 can be used to execute the image processing method of the embodiments of this application; for example, opening a camera application; displaying a first image, the first image being obtained by capturing an image from a first camera; receiving a first zoom operation; displaying and saving a second image, the second image being obtained by segmenting the image captured by the first camera into modules using the parallax between the image captured by the first camera and the image captured by the second camera, and then stitching them together after correcting each module using its corresponding warp matrix. This application achieves full depth-of-field alignment by performing warp correction and stitching between modules, thereby enabling smooth zooming and smooth camera switching.

[0070] Figure 1 The connection relationships between the modules shown are merely illustrative and do not constitute a limitation on the connection relationships between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of various connection methods described in the above embodiments.

[0071] The wireless communication function of electronic device 100 can be realized through devices such as antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0072] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0073] Electronic device 100 can implement display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0074] Display screen 194 can be used to display images or videos.

[0075] For example, in an embodiment of this application, the display screen 194 may be used to display a second image.

[0076] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0077] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can perform algorithmic optimization of image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0078] The camera 193 (also known as a lens) is used to capture still images or videos. It can be activated via application commands to enable photo-taking, such as capturing images of any scene. The camera may include components such as an imaging lens, filters, and an image sensor. Light emitted or reflected by an object enters the imaging lens, passes through the filter, and is finally focused onto the image sensor. The imaging lens is primarily used to focus and image the light emitted or reflected by all objects within the shooting field of view (also known as the scene to be captured, the target scene, or the scene image the user expects to capture). The filter is primarily used to filter out excess light waves (such as infrared light waves other than visible light). The image sensor can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The image sensor is primarily used to perform photoelectric conversion on the received light signal, converting it into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into standard RGB, YUV, and other image signal formats.

[0079] For example, the gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the electronic device 100 around three axes (i.e., the x-axis, y-axis, and z-axis). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the shake of the electronic device 100, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 100 by moving in the opposite direction, thus achieving image stabilization. The gyroscope sensor 180B can also be used in scenarios such as navigation and motion-sensing games.

[0080] In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0081] The camera 193 can be located in front of the electronic device 100 or on the back of the electronic device 100. The specific number and arrangement of the cameras can be set according to the requirements, and this application does not impose any restrictions.

[0082] For example, the electronic device 100 includes a front-facing camera and a rear-facing camera. For instance, either the front-facing camera or the rear-facing camera may include one or more cameras. Taking an electronic device 100 with four rear-facing cameras as an example, when the electronic device 100 activates the four rear-facing cameras to take pictures, it can use the image processing method provided in the embodiments of this application.

[0083] Alternatively, the camera can be mounted on an external accessory of the electronic device 100, which is rotatably connected to the frame of the phone. The angle formed between the external accessory and the display screen 194 of the electronic device 100 can be any angle between 0 and 360 degrees. For example, when the electronic device 100 takes a selfie, the external accessory rotates the camera to face the user. Of course, when the phone has multiple cameras, only some cameras may be mounted on the external accessory, while the remaining cameras may be mounted on the main body of the electronic device 100. This application embodiment does not impose any restrictions on this.

[0084] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0085] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, and MPEG 4.

[0086] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the electronic device 100 around three axes (i.e., the x-axis, y-axis, and z-axis). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the shake of the electronic device 100, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 100 by moving in the opposite direction, thus achieving image stabilization. The gyroscope sensor 180B can also be used in scenarios such as navigation and motion-sensing games.

[0087] The accelerometer 180E can detect the magnitude of acceleration of the electronic device 100 in various directions (typically the x-axis, y-axis, and z-axis). When the electronic device 100 is stationary, it can detect the magnitude and direction of gravity. The accelerometer 180E can also be used to identify the attitude of the electronic device 100, serving as input parameters for applications such as screen orientation switching and pedometers.

[0088] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. In some embodiments, such as in a shooting scenario, the electronic device 100 can utilize the distance sensor 180F to measure distance for fast focusing.

[0089] The ambient light sensor 180L is used to sense the brightness of ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also work with the proximity sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0090] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to perform functions such as unlocking, accessing application locks, taking photos, and answering calls.

[0091] Touch sensor 180K, also known as a touch device, can be disposed on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a touch screen. Touch sensor 180K is used to detect touch operations applied to or near it. Touch sensor 180K can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be disposed on the surface of electronic device 100, and in a different location from display screen 194.

[0092] The hardware system of electronic device 100 has been described in detail above. The software system of electronic device 100 will be introduced below.

[0093] Figure 2 This is a schematic diagram of the software system of the electronic device provided in the embodiments of this application.

[0094] like Figure 2 As shown, the system architecture may include an application layer 210, an application framework layer 220, a hardware abstraction layer 230, a driver layer 240, and a hardware layer 250.

[0095] Application layer 210 may include applications such as camera application, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0096] The application framework layer 220 provides application programming interfaces (APIs) and programming frameworks for applications in the application layer; the application framework layer may include some predefined functions.

[0097] For example, the application framework layer 220 may include a camera access interface; the camera access interface may include camera management and camera devices. Specifically, camera management can be used to provide an access interface for managing cameras; camera devices can be used to provide an interface for accessing cameras.

[0098] The hardware abstraction layer 230 is used to abstract hardware. For example, the hardware abstraction layer may include a camera abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer can call camera algorithms.

[0099] For example, the hardware abstraction layer 230 includes a camera hardware abstraction layer and a camera algorithm; the camera algorithm may include software algorithms for image processing.

[0100] For example, the camera algorithm library may include algorithms corresponding to the image processing methods provided in the embodiments of this application.

[0101] For example, the algorithm in a camera algorithm can refer to something that does not depend on specific hardware implementation; for example, code that can typically run on a CPU.

[0102] The driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include a camera driver.

[0103] Hardware layer 250 is located at the lowest level of the operating system; such as Figure 2 As shown, hardware layer 250 may include camera 1, camera 2, camera 3, etc. Camera 1, camera 2, and camera 3 may correspond to multiple cameras on an electronic device.

[0104] For ease of understanding, the following description will take a mobile phone with the above-mentioned hardware and software structure as an example, and will first describe in detail the camera on the electronic device 100 to which the method provided in this application embodiment is applicable.

[0105] The electronic device to which the method provided in this application is applicable has at least a plurality of cameras 193, for example, three types of cameras 193; the three types of cameras are a main camera (e.g., a wide-angle camera), an ultra-wide-angle camera, and a telephoto camera; the three cameras can be used to capture the same scene to be captured.

[0106] Optionally, the electronic device 100 may also have other cameras 193. The type of camera 193 and the number of each type of camera 193 can be set as needed, and this application embodiment does not impose any restrictions on this.

[0107] For example, such as Figure 3 The illustration uses an electronic device 100 with three cameras 193 as an example; the arrangement of the three cameras can be as follows: Figure 3 As shown in (a) or as in Figure 3 As shown in (b); for example, the three cameras 193 can be a main camera 1931 (e.g., a wide-angle camera), an ultra-wide-angle camera 1932, and a telephoto camera 1933.

[0108] It should be understood that the above are merely examples of two arrangement methods, and other arrangement methods are also possible; the specific arrangement method can be designed and modified as needed, and the embodiments of this application do not impose any restrictions on this.

[0109] It should be noted that when the three cameras are shooting, the field of view of the main camera 1931 is usually larger than that of the telephoto camera 1933; while the field of view of the ultra-wide-angle camera 1932 is larger than that of the main camera 1931; the field of view of the ultra-wide-angle camera 1932 may overlap with that of the main camera 1931; that is to say, the ultra-wide-angle camera 1932 can capture the scene captured by the main camera 1931 and the surrounding scene.

[0110] It should be understood that the field of view of the telephoto camera 1933 is smaller than that of the main camera 1931, and there may be overlap between the field of view of the main camera 1931 and the telephoto camera 1933; that is, the main camera 1931 can capture the scene captured by the telephoto camera 1933 and its surrounding scene. Similarly, the field of view of the ultra-wide-angle camera 1932 may overlap with that of the telephoto camera 1933; that is, the ultra-wide-angle camera 1932 can capture the scene captured by the telephoto camera 1933 and its surrounding scene.

[0111] Among them, the ultra-wide-angle camera 1932 is suitable for shooting close-ups due to its smaller focusing distance; and, as the name suggests, the ultra-wide-angle camera 1932 is suitable for shooting scenes with a large field of view; the main camera 1931 is more suitable for shooting portraits due to its higher resolution, while the telephoto camera 1933 is more suitable for shooting distant close-ups.

[0112] For example, such as Figure 4 As shown, the zoom ratio of an ultra-wide-angle camera can be less than M times the zoom ratio; the zoom ratio of a wide-angle camera, i.e., the main camera, can be in the range of [M, N); and the zoom ratio of a telephoto camera can be greater than or equal to N times the zoom ratio.

[0113] For example, M can be 1 and N can be 2.5; then the zoom ratio of the ultra-wide-angle camera is less than 1x zoom ratio (1×); the zoom ratio of the wide-angle camera is from 1x zoom ratio to 2.5x zoom ratio (1×~2.5×); the zoom ratio of the telephoto camera is greater than or equal to 2.5x zoom ratio.

[0114] It should be understood that during the shooting process of electronic devices, the greater the zoom ratio, the smaller the corresponding field of view.

[0115] The following is combined with Figure 5 The application scenarios of the image processing method provided in the embodiments of this application are illustrated with examples.

[0116] The method in this application embodiment can be applied to shooting scenarios, previewing scenarios, recording video scenarios, or video call scenarios, etc.; through the method in this application embodiment, smooth zoom and / or smooth switching between different cameras in electronic devices can be achieved, improving the user's shooting experience and image quality.

[0117] For example, the preview scenarios include, but are not limited to, the following scenarios:

[0118] Photo preview, aperture preview, night scene preview, portrait preview, video preview, or professional preview, etc.

[0119] It should be understood that a preview scene can refer to a scene in which an electronic device captures an image before the button indicating that the camera has been clicked, in a certain shooting mode.

[0120] In one example, such as Figure 5 As shown in (a), after the electronic device enters the camera application, it can activate the default shooting mode; in the shooting mode, the electronic device can enter the default shooting mode, which can refer to a shooting mode in which the wide-angle camera is used as the main camera and the zoom ratio is a single zoom ratio (1×), and the electronic device displays the image captured by the main camera; as Figure 5 As shown in (b), in response to user operation, the electronic device can switch to display the image captured by the telephoto camera when the zoom ratio meets the zoom range corresponding to the telephoto camera.

[0121] For example, the image processing method in this application embodiment can also be applied to video call scenarios, wherein the video call scenario may include, but is not limited to, the following scenarios:

[0122] Video calls, video conferencing applications, long and short video applications, live video applications, online video courses, intelligent portrait camera movement applications, video recording by system camera function, video surveillance, or portrait shooting scenarios such as intelligent doorbells, etc.

[0123] It should be understood that the above are illustrative examples of application scenarios and do not limit the application scenarios of this application in any way.

[0124] Currently, during shooting, electronic devices can zoom by switching between cameras with different focal lengths in response to user input. They can also combine digital zoom with image processing to meet various high-magnification shooting scenarios. To achieve smooth zooming, existing technologies typically employ the SAT algorithm (which corrects each pair of simultaneously acquired images from the two cameras used for zoom switching to achieve consistent field of view).

[0125] For example, such as Figure 6 As shown, when switching from the main camera to the telephoto camera, both the main camera and the telephoto camera capture images before the switch, and the image captured by the main camera is displayed. After the switch, both the main camera and the telephoto camera continue to capture images, but the image captured by the telephoto camera is displayed. During this zoom process, to maintain smooth zooming, the field of view (FOV) of each pair of images captured by the main camera and the telephoto camera must remain consistent. To this end, as the zoom magnification increases, the image captured by the telephoto camera is typically used as the reference. Correction parameters are calculated to transform the image captured by the main camera into the image captured by the telephoto camera, and then applied to the image captured by the main camera, thus ensuring that the FOV of the corrected image is consistent with that of the image captured by the telephoto camera. These correction parameters generally include a rotation amount representing the rotation relationship and an offset amount representing the offset relationship. The rotation amount can be obtained using existing related technologies, which will not be discussed here. In existing technologies, the offset amount can be calculated based on the differences between the feature points after registering the images captured by the main camera and the telephoto camera.

[0126] However, this calculation process cannot guarantee the accuracy of registration. For example, in scenes with repetitive textures or weak textures, incorrectly registered feature points may occur. Incorrect feature point registration will lead to errors in the calculation of feature point differences, i.e., offsets, resulting in incorrect correction parameters. Consequently, the FOV of the image corrected using these parameters will not match that of the image captured by the telephoto camera. Repetitive texture scenes refer to images with repetitive patterns or designs, such as floral patterns or tree patterns. Texture scenes refer to texture features with relatively small feature values, such as wall cracks or brick patterns.

[0127] Furthermore, even if all feature points are correctly registered, the offsets of feature points at different depths will differ, and a single, uniform offset cannot represent the offset of all feature points in the image. For example, if the depth of the subject differs—such as the depth of the foreground and background—then the feature points extracted from the foreground and background will correspond to different depths. Since there is a perspective relationship between the foreground and background in the image, for the same physical distance, the offset of feature points in the foreground will be larger than that in the background. Therefore, when performing correction, an offset determined in a fixed way cannot be used as the offset of all feature points in the image.

[0128] However, the center-first and foreground-first processing strategies provided in the existing technology result in the collection of feature points with smaller or closer depths after the depth is calculated to calculate the offset. When there are not enough feature points with closer depths extracted from the foreground during zooming, feature points with farther depths in the background are added to calculate the offset, which leads to abnormal image effects after correction.

[0129] For example, Figure 7 This refers to a set of image streams involved in existing technologies. For example... Figure 7 As shown in (a) to (f), assuming the acquired image only contains foreground and background, during zooming, there is a significant difference between the foreground and background images from the two cameras. If the foreground is to be aligned, the background cannot be aligned; similarly, if the background is to be aligned, the foreground cannot be aligned. Since only one depth can be guaranteed at a time, the algorithm needs to continuously determine whether to align the foreground or the background depth, resulting in repeated jumping between foreground and background in the zooming process, making it impossible to smoothly switch between cameras.

[0130] In view of this, embodiments of this application provide an image processing method and related equipment. In embodiments of this application, disparity statistics are performed on images captured by a first camera and images captured by a second camera. Then, the images captured by the first camera and the second camera are segmented into modules based on the disparity distribution information. Based on each module segmented from the images captured by the first camera and the second camera, a warp matrix is ​​calculated between every two corresponding modules. Each module included in the image captured by the first camera is corrected using its corresponding warp matrix, and the stitched second image is then displayed. This application segments the image based on disparity (depth of field) and implements warp correction for each module, thereby enabling warping of both foreground and background, achieving full depth-of-field alignment, and ensuring smooth zooming and smooth switching during camera switching.

[0131] The following is combined with Figure 8 A schematic flowchart illustrating the image processing method provided in the embodiments of this application is described in detail.

[0132] Figure 8 This is a schematic flowchart of an image processing method provided in an embodiment of this application. The method can be... Figure 1 The electronic device shown executes the method 300, which includes steps S310 to S340. Steps S310 to S340 are described in detail below.

[0133] It should be understood that in the embodiments of this application, the electronic device includes a first camera and a second camera; wherein the first camera and the second camera are cameras of different types.

[0134] S310, launch the camera application.

[0135] For example, a user can instruct an electronic device to launch the camera application by clicking the icon of the "Camera" application.

[0136] For example, when an electronic device is locked, a user can instruct the device to launch the camera application by swiping right on the screen. Alternatively, if the device is locked and the lock screen includes a camera application icon, the user can instruct the device to launch the camera application by tapping the icon. Or, if the device is running another application with permission to access the camera application, the user can instruct the device to launch the camera application by tapping the corresponding control. For instance, if the device is running an instant messaging application, the user can instruct the device to launch the camera application by selecting a control that enables camera functionality.

[0137] It should be understood that the above is an example of how to open a camera application; the camera application can also be opened by voice commands or other methods; this application does not limit this in any way.

[0138] It should also be understood that launching the camera application can mean running the camera application.

[0139] S320. Display the first image, which is obtained by capturing an image from the first camera.

[0140] For example, the first camera can be Figure 3 The wide-angle camera shown can be a main camera, or it can be any other camera with a wider field of view than a telephoto camera.

[0141] For example, the first image can be an RGB image captured by the main camera, or the first image can be an RGB image processed by a series of camera algorithms from a Raw image captured by the main camera.

[0142] For example, Figure 19 The preview image shown in (a) is the first image captured and displayed by the first camera. For example, Figure 20 The video image shown in (a) is the first image captured and displayed by the first camera.

[0143] Optionally, when displaying the first image captured by the first camera, the depth of the subject can also be captured.

[0144] For example, before displaying the first image, the first camera can also perform parallax detection on the content in the first image to determine the depth of the different shooting objects included in the first image.

[0145] For example, such as Figure 20 As shown in (a), when the first camera sends the first image to the display, it can perform parallax detection to determine the depth corresponding to different shooting objects. This depth information can be displayed on the first image or not, and this application does not limit it.

[0146] It should be understood that the first image may include one or more subjects, and when displayed, each subject identified based on AI detection can have a detection box displayed simultaneously.

[0147] It should also be understood that the type of the subject can be preset as needed. For example, if the type is set to a face, the target object can be used to indicate a face identified in the first image. The type of subject can also include plants, animals, etc. This application embodiment does not impose any limitations in this regard.

[0148] S330, received the first zoom operation.

[0149] Optionally, receiving the first zoom operation may include receiving a first zoom operation for the first image.

[0150] It should be understood that zooming on the first image can be a user instruction, or it can be an instruction for zooming automatically triggered by the electronic device based on AI detection. The target object is one of the subjects included in the first image.

[0151] For example, when the zoom operation indicates the operation command triggered by the user, the zoom operation may include a two-finger reverse swipe operation, a click operation, a voice operation, an air gesture operation, etc., for the first image. This application embodiment does not impose any limitations on this.

[0152] For example, such as Figure 20 (a) and Figure 20As shown in (b), when the electronic device displays one or more objects to be photographed and a zoom control, the zoom operation may include a click operation on the target object in the first image and a swipe operation on the zoom control; or, the zoom operation may also include a click operation on the target object in the first image and a two-finger reverse swipe operation. It should be understood that when the zoom operation includes two operations, the user must first perform the click operation, and then perform the swipe operation on the zoom control or the two-finger reverse swipe operation on the target object. The zoom operation may also be other operations, or it may include three or more sub-operations; this application embodiment does not impose any limitations on this.

[0153] S340, Display and save the second image.

[0154] For example, the first image can be an RGB image, and the corresponding second image can be an RGB image.

[0155] The second image is obtained by segmenting the image captured by the first camera into modules based on the disparity between the image captured by the first camera and the image captured by the second camera. Each module is then corrected using its corresponding warp matrix before being stitched together. Module segmentation, also known as image segmentation, is used to divide the image captured by the cameras into different image blocks according to disparity. Furthermore, the warp matrix corresponding to each module is different.

[0156] The warp matrix may include one or more of the rotation matrix, offset matrix, and scaling matrix, or it may be a composite matrix of multiple matrices. In addition, the warp may also include other data, which are not limited in this embodiment.

[0157] It should be understood that, in response to the first zoom operation, disparity statistics are performed on the images captured by the first camera and the images captured by the second camera to obtain a disparity map. For the disparity map, the disparity distribution information is statistically analyzed, and disparities belonging to the same category are segmented into modules. This is equivalent to segmenting the images captured by the first camera and the images captured by the second camera into multiple modules according to different depths. Based on the segmented modules of the images captured by the first camera and the second camera, the warp matrix between each pair of corresponding modules can be calculated. Each module included in the image captured by the first camera is corrected by combining its corresponding warp matrix, and then the images can be stitched together to obtain the second image.

[0158] Optionally, the second image can also be obtained by segmenting the image captured by the first camera into modules using the parallax between the image captured by the first camera and the image captured by the second camera, performing warp correction on each module, and then aligning and stitching adjacent modules together. Here, alignment between adjacent modules refers to warp correction using similar content between adjacent modules.

[0159] It should be understood that, in response to the first zoom operation, after performing disparity statistics on the images captured by the first and second cameras, module segmentation is performed, dividing the images captured by the first and second cameras into multiple modules. Based on each module segmented from the images captured by the first and second cameras, a warp matrix is ​​calculated between corresponding modules in each pair of frames. Then, different modules in the image captured by the first camera are corrected using their respective warp matrices. Afterward, for the corrected image captured by the first camera, a warp matrix is ​​calculated between adjacent modules within the frame. The warp matrices between modules are used to align the modules in the corrected image captured by the first camera. Finally, all modules are stitched together to obtain the second image.

[0160] For example, the first zoom operation is either an operation that increases the zoom magnification, or an operation that decreases the zoom magnification. When the zoom magnification increases, the size of the subject included in the second image is larger than the size of the subject included in the first image. When the zoom magnification decreases, the size of the subject included in the second image is smaller than the size of the subject included in the first image.

[0161] Optionally, the target object in the second image is centered within the second image.

[0162] Optionally, the second image may include all or part of the target object. For example, such as Figure 20 As shown, assuming the target object is the fourth photographed object, after magnification, the second image can include the entire fourth photographed object, or the second image can also include partial content of the fourth photographed object, such as the face of the fourth photographed object; when magnified further, the second image can also include a part of the face of the fourth photographed object.

[0163] Optionally, to ensure smooth magnification of the target object displayed by the electronic device during zooming, the above-mentioned S340 can be executed multiple times, with the target object in each acquired second image being slightly larger than the target object in the previous acquired image. That is, in response to the zoom operation, the electronic device corrects the image acquired by the first camera using the offset between the image acquired by the second camera, acquires and displays multiple second images, all of which include the target object, and the target object in these multiple second images gradually increases in size from its original size. The number of iterations and the magnitude of size change can be set as needed, and this application does not limit this. The original size refers to the size of the target object in the first image.

[0164] It should be understood that S340 can be executed simultaneously with S330, or S340 can be executed after S330. This application embodiment does not impose any restrictions on this.

[0165] For example, a second image can be displayed when in a preview scene. These preview shooting modes include, but are not limited to, shooting modes such as night scene preview mode, video preview mode, photo preview mode, and portrait preview mode.

[0166] For example, when in a recording scene, a second image can be displayed and saved.

[0167] Optionally, the method further includes:

[0168] Responding to the first zoom operation;

[0169] Determine the zoom ratio of the electronic device to meet the zoom ratio range of the first camera for foreground display and the second camera for background operation.

[0170] For example, in response to the first zoom operation, when the zoom ratio has reached the minimum zoom ratio of the second camera but not the maximum zoom ratio of the first camera (e.g., the telephoto camera is at least 1.8× and the main camera is at most 2.5×), the second image acquired by the first camera continues to be displayed. However, this second image is obtained by segmenting the image captured by the first camera into modules using the parallax between the image captured by the first camera and the image captured by the second camera, and then stitching them together after correcting each module with its corresponding warp matrix. The warp matrices corresponding to each module are different.

[0171] For example, such as Figure 16 As shown, the first zoom operation is switching from 1× to 4×. The first camera is always in the foreground display, but when zoomed to 1.8×, the second camera starts running in the background. At this time, the method provided in the embodiment of this application can be triggered to perform module correction on the image captured by the first camera by calculating multiple warp matrices, and then generate a second image for display and saving.

[0172] This application provides an image processing method and related equipment. In this embodiment, disparity statistics are performed on images captured by a first camera and images captured by a second camera. Then, the disparity distribution information is used to segment the images captured by the first and second cameras into modules. Based on each module segmented from the images captured by the first and second cameras, a warp matrix is ​​calculated between every two corresponding modules. Each module included in the image captured by the first camera is corrected using its corresponding warp matrix, and the corrected image is then stitched together to display the second image. This application segments the image based on disparity (depth of field) and implements warp correction for each module, thereby enabling warping of both foreground and background, achieving full depth-of-field alignment, and ensuring smooth zooming and smooth switching during camera switching.

[0173] For example, Figure 17 This refers to a set of image streams involved in this application. For example... Figure 17 (a), (b), (c), (d), (e) Figure 17 As shown in (f), during zooming, assuming that the acquired image is divided into foreground and background modules after module segmentation based on disparity distribution information, then the warp matrix between the foreground module of the image acquired by the first camera and the foreground module of the image acquired by the second camera is calculated, and the foreground module of the image acquired by the first camera is corrected using this warp matrix; and the warp matrix between the background module of the image acquired by the first camera and the background module of the image acquired by the second camera is calculated, and the background module of the image acquired by the first camera is corrected using this warp matrix. Since this application segments the image based on disparity and implements warp correction for each module, both foreground and background can be warped to achieve full depth-of-field alignment, thereby enabling smooth zooming of the FOV of the image stream and smooth switching of cameras during zooming.

[0174] Figure 9 An exemplary flowchart of an image processing method for another electronic device 100 is shown.

[0175] Taking the first camera as the main camera and the second camera as a telephoto camera as an example, the method 400 includes S401 to S410; S401 to S410 are described in detail below.

[0176] S401. Acquire the first initial image captured by the main camera.

[0177] For example, the image captured by the main camera can be a Raw image captured by the wide-angle camera, or a YUV image.

[0178] S402, Acquire the second initial image captured by the telephoto camera.

[0179] For example, the image captured by the telephoto camera can be a Raw image or a YUV image captured by the telephoto camera.

[0180] S403. Perform image feature point detection and registration on the first initial image and the second initial image to obtain multiple pairs of feature points.

[0181] Optionally, image feature point detection and registration can employ any existing image feature point detection and registration algorithm, and this application does not impose any limitations on this. For example, it can be the ORB (Oriented FAST and Rotated BRIEF) algorithm, optical flow algorithm, etc.

[0182] For example, image feature point detection is performed on the first initial image and the second initial image respectively, extracting M feature points from the first initial image and N feature points from the second initial image; then, the M feature points and N feature points are registered. M and N are both integers greater than 0.

[0183] It should be understood that each feature point pair includes one feature point belonging to the first initial image and one feature point belonging to the second initial image; these two feature points are used to indicate the same target. The registered feature points have a one-to-one correspondence.

[0184] Optionally, after S403, the method may further include: filtering correctly registered feature point pairs.

[0185] Specifically, correctly registered feature point pairs can be selected by eliminating incorrectly registered feature point pairs.

[0186] After testing, it was found that the number of correctly registered feature points exceeded the number of incorrectly registered feature points in each scenario. Therefore, in order to improve processing efficiency and the effect of subsequent processing, the incorrectly registered feature points can be removed, and only the correctly registered feature point pairs can be retained.

[0187] S404. Calculate the disparity of each feature point pair in all feature point pairs to obtain a disparity map.

[0188] It should be understood that binocular ranging technology, also known as stereo vision technology, calculates the disparity between two images to obtain three-dimensional information of objects in a scene, including depth (i.e., distance) and size. The core of binocular ranging lies in calculating the disparity between two images. First, the two cameras are calibrated to determine their relative position and orientation. Then, image processing techniques are used to identify corresponding feature points in the images from both cameras. Next, based on the positional relationship of these corresponding feature points, a stereo matching algorithm is used to calculate the disparity. Based on the disparity and the camera calibration parameters, the three-dimensional coordinates of objects in the scene can be calculated, thus obtaining the object's size.

[0189] In the embodiments of this application, the parallax between the first initial image and the second initial image can be determined using binocular ranging technology.

[0190] S405. Statistically analyze and classify the disparity distribution information of the disparity map to determine multiple categories.

[0191] Optionally, for the disparity map, the disparity distribution information is statistically analyzed, and if clustering is used, the map is divided into multiple classes according to the clusters formed by the clustering algorithm.

[0192] Clustering algorithms are a method in unsupervised learning used to divide samples in a dataset into several "clusters" such that samples within the same cluster have high similarity, while samples in different clusters have low similarity.

[0193] It should be understood that if the disparity distribution information is not clustered but dispersed, the method provided in this application embodiment cannot be used for further processing. Other methods can be selected for processing, and this application embodiment does not limit this.

[0194] For example, Figure 10 Image (a) shown in the image is the first initial image. Figure 10 Image (b) shows the second initial image. After image feature point detection and registration are performed on the first and second initial images, multiple pairs of feature points (also called inter-frame feature point pairs) are generated. The disparity of each pair of feature points is calculated to obtain a disparity map, such as... Figure 10 As shown in (c); for this disparity map, the disparity distribution information is statistically analyzed, and the map is divided into multiple classes according to the clusters formed by the clustering algorithm, for example, such as Figure 10 As shown in (d), they can be divided into five categories: a, b, c, d, and e.

[0195] S406. Based on the disparity belonging to the same class, both the first initial image and the second initial image are segmented into modules.

[0196] It should be understood that, based on the classification results, both the first and second initial images are segmented into modules. Disparities belonging to the same class belong to the same module. The number of modules in the segmented first and second initial images may be the same or different.

[0197] It should be noted that modules corresponding to the same type of parallax can include multiple sub-modules, and the positions of the sub-modules do not have to be adjacent; in other words, multiple sub-modules located in different positions may belong to the same type of parallax.

[0198] For example, combined Figure 10 The classification shown in (d) performs module segmentation on disparities belonging to the same class in the first and second initial images. The segmented first initial image is shown below. Figure 11 As shown in (a) of Figure 11, the first initial image comprises five modules enclosed by different dashed lines; the segmented second initial image, as shown in (b) of Figure 11, also comprises five modules enclosed by different dashed lines. The first module in the first initial image may include three sub-modules, and the first sub-module in the second initial image may include two sub-modules.

[0199] Optionally, after S406, S407 to S410 can be executed, or S408 to S410 can be executed.

[0200] S407. Add a certain amount of margin (tolerance range) around the module.

[0201] The size of the margin can be set as needed, and this application embodiment does not limit it. For the perimeter of multiple modules, a margin of the same size can be added, or margins of different sizes can be added; for different edges of each module, a margin of the same size can be added, or margins of different sizes can be added.

[0202] Additionally, margins may not be added for edges belonging to the first or second initial image included in the module.

[0203] For example, such as Figure 11 As shown in (c) in the figure, it is for Figure 11 (a) shows a schematic diagram of adding margins to the first initial image after module segmentation; as shown in Figure (a). Figure 11 As shown in (d) in the figure, it is for... Figure 11 The diagram in (b) shows the second initial image after module segmentation with added margin.

[0204] It should be understood that adding a certain amount of margin around the module creates overlapping areas between adjacent modules, which facilitates subsequent module splicing and allows for better integration of modules together.

[0205] S408. Based on the feature point pairs obtained in S403, and combined with the segmented modules, calculate the warp matrix corresponding to each module.

[0206] It should be understood that, based on feature point pairs and combined with the segmented modules, the warp matrix between corresponding modules of the first initial image and the second initial image can be calculated. For example, if the first initial image and the second initial image include 5 sets of corresponding modules, then 5 warp matrices between these 5 sets of corresponding modules can be calculated.

[0207] For example, such as Figure 12 As shown in (a) and (b), by combining feature point pairs and multiple segmented modules, the warp matrix between the first module in the first initial image and the first module in the second initial image can be calculated using the feature points included in the first module in the first initial image and the matching feature points in the second initial image. Figure 12 Warp1 is shown in (c) in the image; similarly, the warp matrix between the second module in the two images can be calculated in sequence, such as warp2; the warp matrix between the third module in the two images, such as warp3; the warp matrix between the fourth module in the two images, such as warp4; and the warp matrix between the fifth module in the two images, such as warp5.

[0208] It should be understood that for a module that includes multiple submodules, multiple submodules belonging to the same module correspond to the same warp matrix.

[0209] S409. The modules after the first initial image has been segmented are corrected based on the warp matrix corresponding to each module.

[0210] The warp matrix corresponding to this module is the first warp matrix.

[0211] S410, after stitching, the second image is obtained.

[0212] Optionally, when correcting the first initial image, the first initial image can be corrected module by module by combining the warp matrix corresponding to each module. The image obtained by stitching after correction can be saved and displayed as the second image.

[0213] For example, such as Figure 13 As shown in (a), the modules of the first initial image, after being segmented into modules, are corrected based on their respective warp matrices and then stitched together to obtain the following: Figure 13 The image shown in (b) can be displayed and saved as a second image.

[0214] Optionally, for modules that do not execute S407, i.e., do not add margin, after correction, they can be fused into a single image using the Poisson fusion algorithm. This fused image can then be saved and displayed as a second image.

[0215] It should be noted that the Poisson fusion algorithm achieves image fusion by solving the Poisson equation, which can create natural and realistic composite effects.

[0216] In this embodiment, in response to the first zoom operation, disparity statistics are performed on the images captured by the first camera and the second camera to obtain a disparity map. For the disparity map, the disparity distribution information is statistically analyzed, and disparities belonging to the same category are segmented into modules. This is equivalent to segmenting the images captured by the first camera and the second camera into multiple modules based on different depths. Based on each module segmented from the images captured by the first and second cameras, a warp matrix between every two corresponding modules can be calculated. Each module included in the image captured by the first camera is corrected using its corresponding warp matrix, and then the images can be stitched together to obtain the second image. Since both the foreground and background are warped through module segmentation, full depth-of-field alignment can be achieved.

[0217] Figure 14 An exemplary flowchart of an image processing method for another electronic device 100 is shown.

[0218] Taking a first camera as the main camera and a second camera as a telephoto camera as an example, the method 400, based on S401 to S409, may also include S411 to S415; S401 to S409 can refer to the above-mentioned methods for... Figure 9 The description of S411 to S415 will not be repeated here. The following sections will describe S411 to S415 in detail.

[0219] S411. Perform feature point matching on two adjacent modules in the first initial image belonging to S409 to obtain multiple sets of matched feature point pairs (also known as intra-frame feature point pairs).

[0220] It should be understood that, since a certain amount of margin is added to the modules after the first initial image segmentation in S407, there will be a certain overlap between adjacent modules, that is, the area of ​​the margin added by the adjacent modules on the adjacent sides.

[0221] For example, such as Figure 15As shown, for the first and second modules in the first initial image, the first and second modules are adjacent vertically. Therefore, the margin region added to the lower edge of the first module and the margin region added to the upper edge of the second module will have a certain overlap. Thus, multiple sets of feature point pairs may be generated when performing feature point matching.

[0222] S412. Based on the intra-frame feature point pairs, calculate the warp matrix between the two adjacent modules.

[0223] The warp matrix between modules here is the second warp matrix.

[0224] S413. Alignment of modules is achieved by using the warp matrix between two adjacent modules.

[0225] Optionally, after generating feature point pairs, the feature point pairs are filtered according to preset rules. For example, if there is a lot of overlap in the margin regions of adjacent modules, more feature point pairs can be filtered and retained; if there is little overlap in the non-margin regions of adjacent modules, fewer feature point pairs can be filtered and retained.

[0226] For example, such as Figure 15 As shown, for the first and second adjacent modules in the initial image, after feature point matching and generating multiple sets of feature point pairs, feature point pairs in the margin and non-margin regions are filtered according to an 8:2 ratio. Using the filtered feature point pairs, the warp matrix between the two adjacent modules can be calculated, for example, warpA. Then, warpA can be used to correct the first module, achieving alignment between the first and second modules. Similarly, the warp matrix between adjacent modules can be calculated for other modules, and alignment of adjacent modules can be performed.

[0227] It should be understood that this is equivalent to further aligning the margin area after the non-margin area has already been aligned, in order to achieve better splicing between modules.

[0228] S414. After aligning all modules, overlap and splice the margin areas to achieve the splicing of adjacent modules.

[0229] S415. Align the overall dimensions and correct the edges.

[0230] For example, Figure 17 This refers to a set of image streams involved in the embodiments of this application. For example... Figure 17During the zoom process in (a), (b), (c) to (d), by combining the image processing method provided in this application, image stitching can be achieved based on feature points in the Margin region, thereby enabling smooth image changes and seamless switching between multiple cameras.

[0231] In this embodiment, in response to the first zoom operation, after performing disparity statistics on the images captured by the first camera and the images captured by the second camera, module segmentation is performed, dividing the images captured by the first camera and the images captured by the second camera into multiple modules. Based on each module after segmentation of the images captured by the first camera and the images captured by the second camera, a warp matrix between corresponding modules in each pair of frames is calculated. Then, different modules in the image captured by the first camera are corrected by combining their respective warp matrices. Afterwards, for the corrected image captured by the first camera, a warp matrix between adjacent modules within the frame is calculated. For the corrected image captured by the first camera, the modules are aligned using the warp matrices between modules, and then all modules are stitched together to obtain the second image. Since the foreground and background are warped separately through module segmentation, full depth-of-field alignment can be achieved, thereby enabling seamless switching between multiple cameras.

[0232] Figure 18 A schematic flowchart illustrating another image processing method of an electronic device 100 is provided. The method 500 includes steps S501 to S509; steps S501 to S509 are described in detail below.

[0233] S501, Acquire images captured by the main camera.

[0234] For example, the image captured by the main camera can be a raw image captured by the wide-angle camera; or, the image captured by the main camera can be a raw image captured by the ultra-wide-angle camera.

[0235] S502, First front-end processing.

[0236] Optionally, the first front-end processing may include algorithms for converting the Raw image captured by the main camera into a YUV image; this application does not limit the algorithm in any way.

[0237] For example, the first front-end processing may refer to the image processing algorithm performed in the ISP to convert a Raw image to a YUV image. The first front-end processing may also include subsequent operations such as cropping and magnification based on the zoom ratio in response to zoom operations.

[0238] S503: Acquire images captured by the telephoto camera.

[0239] For example, the image captured by the telephoto camera can be a raw image captured by the telephoto camera.

[0240] Optionally, S504 and S501 can be executed simultaneously, or S504 and S501 can be executed sequentially.

[0241] S504, Second Front-End Processing.

[0242] Optionally, the second front-end processing may include algorithms for converting Raw images captured by a telephoto camera into YUV images; this application does not impose any limitations on the algorithms.

[0243] For example, the second front-end processing may refer to the image processing algorithm performed in the ISP to convert the Raw image to a YUV image. The second front-end processing may also include subsequent operations such as cropping and magnification based on the zoom ratio in response to zoom operations.

[0244] After S505 is executed, the processed image undergoes a first back-end processing; the implementation of the first back-end processing can be found in the relevant description of S503.

[0245] For example, such as Figure 16 As shown, when the zoom ratio is between 1× and 1.8×, only S501 is executed; when zoomed to between 1.8× and 2.5×, both S501 and S503 are executed simultaneously; when zoomed to 2.5× and above, only S503 is executed.

[0246] S505, First Backend Processing.

[0247] For example, the first backend processing includes, but is not limited to: brightening processing, noise reduction processing, saturation adjustment processing, cropping processing, or deformation processing.

[0248] S506, Smoothing (or Alignment).

[0249] For example, smoothing is performed on the image stream captured by the main camera and the image stream captured by the telephoto camera; this can be understood as the input data for smoothing being two image streams (e.g., the image stream captured by the main camera and the image stream captured by the telephoto camera), and the smoothed image stream being one image stream. This smoothing process is the one provided in the embodiments of this application. Figure 8 , Figure 9 and / or Figure 14 The image processing method shown.

[0250] For example, when the zoom magnification increases from 1.8× to 2.5×, the image processing method provided in the embodiments of this application is used to process the image streams of the main camera and the telephoto camera.

[0251] In the embodiments of this application, smoothing processing can avoid obvious jumps between foreground and background in the image when switching cameras.

[0252] S507, Second Backend Processing.

[0253] For example, the second back-end processing includes stabilization processing; for instance, image frames in the image stream can be stabilized according to jitter parameters to obtain a processed image.

[0254] S508, Display the processed image.

[0255] For example, such as Figure 16 As shown, when the zoom magnification increases and the zoom ratio is between 1× and 1.8×, the processed image is the image corresponding to the image captured by the main camera; when zoomed to 1.8× to 2.5×, the processed image is the image captured by the main camera, which has been corrected and processed based on the image captured by the telephoto camera; when zoomed to 2.5× and above, the processed image is the image corresponding to the image captured by the telephoto camera.

[0256] S509, Zoom operation detected.

[0257] The zoom operation can be referred to in the description in S230 above.

[0258] Optionally, the electronic device may display an image captured by the main camera, or an image captured by the telephoto camera, and the electronic device may detect user interaction with the displayed image.

[0259] It should be understood that the above examples are provided to help those skilled in the art understand the embodiments of this application, and are not intended to limit the embodiments of this application to the specific values ​​or scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or changes based on the above examples, and such modifications or changes also fall within the scope of the embodiments of this application.

[0260] For example, Figure 19 This is a schematic diagram of an application scenario provided by an embodiment of this application.

[0261] like Figure 19 As shown in (a), in response to a user's operation on the camera application, the electronic device can display a preview interface 1301, which includes a preview window and shooting controls. The preview image 1302 displayed in the preview window can include a first subject, a second subject, and a third subject. Here, when the camera application is opened, the zoom level is 1x by default. Figure 19As shown in (b), suppose a user wants to zoom in on the third subject located in the lower left corner of preview image 1302, the user can perform a two-finger reverse swipe on the screen; in response to this action, for example, as Figure 19 As shown in (c), the zoom ratio can be increased to 10x, the imaging size of the third subject increases from small to large, and the third subject can respond to the movement of the user's finger and be located in the middle of the preview image 1303.

[0262] It can adapt to the process of increasing zoom magnification and switching cameras. Figure 8 , Figure 9 or Figure 14 The image processing methods described herein.

[0263] It should be understood that the above describes a preview scenario in photo shooting mode. This application embodiment can also be applied to preview scenarios in other shooting modes such as video recording. A preview scenario can refer to the scenario where the electronic device captures an image before clicking the shooting control in a certain shooting mode.

[0264] For example, Figure 20 This is a schematic diagram illustrating another application scenario provided by an embodiment of this application.

[0265] Once the camera app is opened, in response to the user's click on the recording controls, the electronic device can begin recording video and display the video recording interface 1401, such as... Figure 20 As shown in (a), the video recording interface 1401 may include a recording window, a pause control, an end control, and a zoom control. The video image 1402 displayed in the recording window may include a first subject, a second subject, a third subject, and a fourth subject. Here, the zoom ratio is set to 1x by default when video recording begins.

[0266] If the electronic device supports and enables AI detection and parallax detection functions, it can detect image content and depth when capturing video image 1402, and display multiple detection boxes when displaying video image 1402. Each detection box is used to indicate the position information of a captured object in video image 1402. Figure 20 As shown in (a), four detection boxes are displayed in video image 1402, which respectively indicate the position information of the first subject to the fourth subject.

[0267] Optionally, if the AI ​​detection and parallax detection functions are only used for depth detection of faces, the electronic device may display only two detection boxes when displaying video image 1402. These two detection boxes indicate the face of the second subject and the face of the fourth subject, respectively.

[0268] like Figure 20As shown in (a), during recording, assuming the user wants to zoom in on the face of the fourth subject located on the left side of video image 1402, the user can click on the detection box corresponding to the fourth subject on the screen. Figure 20 As shown in (b), in response to a user's click operation, the detection box of the fourth subject included in the video image 1403 can be selected, for example, the selected detection box can have a different color and / or style compared to other detection boxes. Then, the electronic device can receive a user's sliding operation on the zoom control, such as... Figure 20 As shown in (c), in response to a sliding operation on the zoom control (assuming a sliding to 8x), the electronic device can display a video image 1404 in which the face of the fourth subject included in the video image 1404 is larger than the face of the fourth subject included in the video image 1402, and the face of the fourth subject can be located in the center of the video image 1404 in response to the user's finger movement.

[0269] It should be noted that if the electronic device detects only one subject or face in the image, the user does not need to click on the detection box corresponding to the subject or face. The electronic device will automatically select it. Then, in response to the user's sliding operation on the zoom control, the electronic device will zoom in and center the subject or face.

[0270] It can adapt during zoom ratio changes and camera switching. Figure 8 , Figure 9 or Figure 14 The image processing methods described herein.

[0271] It should be understood that the above is a video recording scenario, and the embodiments of this application can also be applied to scenarios such as video calls.

[0272] For example, video call scenarios may include, but are not limited to, the following: video calls, video conferencing applications, long and short video applications, live video applications, online video courses, portrait intelligent camera movement applications, video recording by system camera recording function, video surveillance, or shooting scenarios such as smart doorbell cameras.

[0273] The above text combined Figures 1 to 20 The image processing method provided in the embodiments of this application is described in detail.

[0274] This application also provides a computer program product that, when executed by a processor, implements the methods described in any of the method embodiments of this application.

[0275] The computer program product can be stored in memory, for example, it is a program. The program is eventually converted into an executable object file that can be executed by the processor after processes such as preprocessing, compilation, assembly and linking.

[0276] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer, implements the methods described in any of the method embodiments of this application. The computer program may be a high-level language program or an executable object program.

[0277] The computer-readable storage medium is, for example, memory. Memory can be volatile or non-volatile, or it can include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0278] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process and technical effects of the above-described apparatus and equipment can be referred to the corresponding processes and technical effects in the foregoing method embodiments, and will not be repeated here.

[0279] In the several embodiments provided in this application, the systems, apparatuses, and methods disclosed can be implemented in other ways. For example, some features of the method embodiments described above can be omitted or not performed. The apparatus embodiments described above are merely illustrative; the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Multiple units or components can be combined or integrated into another system. Furthermore, the coupling between units or components can be direct coupling or indirect coupling, including electrical, mechanical, or other forms of connection.

[0280] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0281] Furthermore, the terms "system" and "network" are often used interchangeably in this paper. The term "and / or" in this paper merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " in this paper generally indicates that the preceding and following related objects have an "or" relationship.

[0282] In summary, the above description is merely a preferred embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image processing method, characterized by, Applied to an electronic device, the electronic device including a first camera and a second camera, the method includes: Open the camera application; The first image is displayed, which is obtained by the first camera capturing the image. Received the first zoom operation; Displaying and saving a second image, the second image is: after obtaining a disparity map from the first initial image captured by the first camera and the second initial image captured by the second camera, statistically analyzing the disparity distribution information of the disparity map, dividing it into multiple classes according to the clustering algorithm, and performing module segmentation on both the first initial image and the second initial image based on the disparity belonging to the same class, and then stitching together an image after correcting each module after the segmentation of the first initial image using its corresponding first warp matrix and second warp matrix; The first warp matrix is ​​used to indicate the warp matrix between the modules corresponding to the first initial image and the second initial image, and the second warp matrix is ​​used to indicate the warp matrix between two adjacent modules in the first initial image.

2. The image processing method of claim 1, wherein, The method further includes: Received second zoom operation; Display and save a third image, which is obtained from an image captured by the second camera.

3. The image processing method according to claim 1 or 2, characterized in that, The first initial image captured by the first camera is used to obtain a disparity map using the disparity between it and the second initial image captured by the second camera, including: Acquire the first initial image captured by the first camera and the second initial image captured by the second camera; Feature point detection and registration are performed on the first initial image and the second initial image to obtain multiple sets of matching inter-frame feature point pairs; The disparity of each pair of inter-frame feature points is calculated to obtain the disparity map.

4. The image processing method according to claim 3, characterized in that, The method further includes: Based on the inter-frame feature point pairs, and combining the segments of the first initial image and the second initial image, the first warp matrix is ​​calculated.

5. The image processing method according to claim 4, characterized in that, After performing module segmentation, the method further includes: A tolerance range is added around the module, and the size and position of the tolerance range are preset.

6. The image processing method according to claim 5, characterized in that, The method further includes: Feature point matching is performed on two adjacent modules in the first initial image to obtain multiple sets of matched intra-frame feature point pairs. The second warp matrix is ​​calculated based on the intra-frame feature point pairs.

7. The image processing method according to claim 6, characterized in that, The method for calculating the second warp matrix includes: Feature point pairs within the tolerance range and feature point pairs outside the tolerance range are selected according to a preset ratio, wherein the proportion of feature point pairs within the tolerance range is greater than the proportion of feature point pairs outside the tolerance range.

8. The image processing method according to claim 6, characterized in that, The method also includes: removing feature point pairs that are mismatched.

9. The image processing method according to claim 6 or 7, characterized in that, The method further includes: The increased tolerance range is overlapped during splicing; After splicing, align the overall dimensions and modify the edges.

10. The image processing method according to any one of claims 1, 2, 4 to 8, characterized in that, The method further includes: responding to the first zoom operation; The zoom ratio of the electronic device is determined to be within the zoom ratio range that allows the first camera to display data in the foreground and the second camera to operate in the background.

11. The image processing method according to claim 2, characterized in that, The method further includes: responding to the second zoom operation; The zoom ratio of the electronic device is determined to be within the range that the first camera is running in the background or off, while the second camera is displayed in the foreground.

12. The image processing method according to claim 11, characterized in that, The first zoom operation or the second zoom operation includes: a two-finger reverse swipe operation, a swipe operation on the displayed zoom control, a voice operation, or an air gesture operation.

13. The image processing method according to any one of claims 1, 2, 4 to 8, 11, and 12, characterized in that, The first camera includes a wide-angle camera, and / or the second camera includes a telephoto camera or an ultra-wide-angle camera.

14. The image processing method according to claim 2, characterized in that, The first zoom operation and the second zoom operation are consecutive zoom operations.

15. An electronic device, characterized in that, The electronic device includes: one or more processors, a memory, a first camera and a second camera; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to perform the image processing method as described in any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the image processing method according to any one of claims 1 to 14.

Images