Image processing method and related equipment thereof

Abstract

The invention relates to the field of image processing, and provides an image processing method and related equipment thereof, the image processing method is applied to electronic equipment, the electronic equipment comprises a first camera and a second camera, and the method comprises the following steps: starting a camera application program; displaying a first image, wherein the first image is obtained by collecting an image by a first camera; receiving a first zooming operation; and a second image is displayed and stored, the second image is obtained by performing module segmentation on the image collected by the first camera through parallax between the image collected by the first camera and the image collected by the second camera, correcting each module through the corresponding warp matrix and then splicing the modules. According to the method and the device, the modules are spliced after being subjected to warp correction, so that panoramic depth alignment can be realized, and smooth zooming and smooth switching of the camera are further realized.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method and related equipment. Background Art

[0002] With the development of shooting functions in electronic devices, camera applications are increasingly used in electronic devices. To obtain a better photo taking experience, current electronic devices are usually equipped with multiple cameras, each corresponding to a different focal length.

[0003] When shooting, electronic devices can respond to user input by switching between cameras with different focal lengths to perform zoom shots. They can also incorporate digital zoom into the captured image to accommodate a variety of high-magnification shooting scenarios. However, during zooming, due to varying depths of the subject, such as the foreground and background, switching between cameras can cause jumps in image content and size, preventing smooth camera switching. Furthermore, digital zoom can only magnify and crop the image content in the center of the field of view, which is very limited.

[0004] Therefore, during the shooting process, how to smoothly switch cameras becomes a problem that needs to be solved urgently. Summary of the Invention

[0005] The present application provides an image processing method and related equipment, which can achieve full-depth alignment by performing warp correction and stitching between modules, thereby achieving smooth zoom and smooth switching of cameras.

[0006] In a first aspect, an image processing method is provided, which is applied to an electronic device, wherein the electronic device includes a first camera and a second camera, and the method includes: opening a camera application; displaying a first image, where the first image is obtained by capturing an image by the first camera; receiving a first zoom operation; and displaying and saving a second image, where the second image is obtained by segmenting an image captured by the first camera into modules using the parallax between the image captured by the first camera and the image captured by the second camera, and then correcting each module using its corresponding warp matrix and then stitching the images together.

[0007] In an embodiment of the present application, disparity statistics are performed on the image captured by the first camera and the image captured by the second camera; the image captured by the first camera and the image captured by the second camera are then divided into modules based on the disparity distribution information; based on the modules after the segmentation of the image captured by the first camera and the image captured by the second camera, a warp matrix between each two corresponding modules is calculated; the modules included in the image captured by the first camera are corrected in combination with their corresponding warp matrices, and the second image obtained after correction is displayed. The present application divides the image based on disparity (depth of field) and implements warp correction corresponding to each module in a module-by-module manner, so that the foreground and background can be warped separately to achieve full depth alignment, thereby ensuring smooth zooming and smooth switching during camera switching.

[0008] In combination with the first aspect, in some implementations of the first aspect, the method further includes: receiving a second zoom operation; and displaying and saving a third image, where the third image is obtained by capturing an image with the second camera.

[0009] In the embodiments of the present application, smooth switching of multiple cameras can be achieved.

[0010] In combination with the first aspect, in certain implementations of the first aspect, the image captured by the first camera is module-segmented using the disparity between the image captured by the second camera and the image captured by the second camera, including: obtaining a first initial image captured by the first camera and a second initial image captured by the second camera; performing feature point detection and registration on the first initial image and the second initial image to obtain multiple sets of matching inter-frame feature point pairs; calculating the disparity of each pair of the inter-frame feature point pairs to obtain a disparity map; performing statistics and division on the disparity distribution information of the disparity map to determine multiple classes; and module-segmenting the first initial image and the second initial image based on the disparity belonging to the same class.

[0011] Module segmentation, also known as image segmentation, is used to divide the image captured by the camera into different image blocks based on parallax. In addition, the warp matrix corresponding to each module is different.

[0012] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: calculating a first warp matrix between the modules corresponding to the first initial image and the second initial image based on the inter-frame feature point pairs and in combination with the segmented modules; and correcting the modules after the first initial image is segmented based on the first warp matrix corresponding to each module.

[0013] In an embodiment of the present application, after the first initial image and the second initial image are divided into modules, correction is performed using the first warp matrix corresponding to each module, which is equivalent to aligning the first initial image and the second initial image by partitioning blocks based on the parallax between the first initial image and the second initial image.

[0014] In combination with the first aspect, in certain implementations of the first aspect, after module segmentation, the method further includes: adding a tolerance range around the periphery of the module, and the size and position of the tolerance range are preset.

[0015] In the embodiment of the present application, a certain amount of margin is added to the periphery of the module so that there is an overlapping area between adjacent modules, which makes it easier to splice the modules later and enables the modules to be better spliced ​​together.

[0016] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: performing feature point matching on two adjacent modules in the first initial image to obtain multiple sets of matched intra-frame feature point pairs; calculating a second warp matrix between the two adjacent modules based on the intra-frame feature point pairs; and correcting the modules after the first initial image is segmented based on the second warp matrix corresponding to each module.

[0017] In the embodiment of the present application, feature point matching and warp correction are continued for adjacent modules of the first initial image, which is equivalent to aligning the internal content of the first initial image by partitioning blocks based on similar content between adjacent modules of the first initial image.

[0018] In combination with the first aspect, in certain implementations of the first aspect, when calculating the second warp matrix between the two adjacent modules, the method includes: screening feature point pairs within the tolerance range and feature point pairs outside the tolerance range according to a preset ratio, wherein the ratio of feature point pairs within the tolerance range is greater than the ratio of feature point pairs outside the tolerance range.

[0019] In combination with the first aspect, in certain implementations of the first aspect, before calculating the vector formed by each pair of feature point pairs, the method further includes: eliminating feature point pairs with matching errors.

[0020] In the embodiments of the present application, after testing, the number of correctly registered feature points in each scene is greater than the number of incorrectly registered feature points. Therefore, in order to improve processing efficiency and improve the effect of subsequent processing, the incorrectly registered feature points can be eliminated and only the correctly registered feature point pairs are retained.

[0021] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: overlapping the increased tolerance range during splicing; and aligning the overall size and modifying the edges after splicing.

[0022] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: responding to the first zoom operation; determining that the zoom ratio of the electronic device satisfies the zoom ratio range of the first camera foreground display and the second camera for background operation.

[0023] It should be understood that foreground display refers to the use of images captured by the camera for display; background operation refers to the use of images captured by the camera for image processing, not for display.

[0024] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: responding to the second zoom operation; determining that the zoom ratio of the electronic device satisfies the zoom ratio range when the first camera is running in the background or turned off, and the second camera is displayed in the foreground.

[0025] In combination with the first aspect, in some implementations of the first aspect, the first zoom operation or the second zoom operation includes: a two-finger reverse sliding operation, a sliding operation on a displayed zoom control, a voice operation, or an air gesture operation.

[0026] In combination with the first aspect, in some implementations of the first aspect, the first camera includes a wide-angle camera, and / or the second camera includes a telephoto camera or an ultra-wide-angle camera.

[0027] In combination with the first aspect, in some implementations of the first aspect, the first zoom operation and the second zoom operation are continuous zoom operations.

[0028] According to a second aspect, an electronic device is provided, comprising: one or more processors, a memory, a first camera, and a second camera; the memory is coupled to the one or more processors, the memory being configured to store computer program code, the computer program code comprising computer instructions, and the one or more processors invoking the computer instructions to cause the electronic device to execute:

[0029] Open a camera application; display a first image, where the first image is obtained by capturing an image with the first camera; receive a first zoom operation; and display and save a second image, where the second image is obtained by segmenting the image captured by the first camera into modules using the parallax between the image captured by the first camera and the image captured by the second camera, and then correcting each module using its corresponding warp matrix and then stitching the images together.

[0030] It should be understood that the expansion, limitation, explanation and description of the relevant content in the above-mentioned first aspect also apply to the same content in the second aspect.

[0031] In a third aspect, a chip system is provided, which is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute any one of the image processing methods in the first aspect.

[0032] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program code. When the computer program code is executed by an electronic device, the electronic device executes any one of the image processing methods in the first aspect.

[0033] In a fifth aspect, a computer program product is provided, comprising: a computer program code, which, when executed by an electronic device, enables the electronic device to execute any one of the image processing methods in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a schematic diagram of a hardware system of an electronic device applicable to the present application;

[0035] Figure 2 is a schematic diagram of a software system for an electronic device applicable to the present application;

[0036] Figure 3 This is a schematic diagram of an arrangement of multiple cameras on an electronic device provided by an embodiment of the present application;

[0037] Figure 4 This is a schematic diagram of zoom ratios corresponding to different types of cameras provided in an embodiment of the present application;

[0038] Figure 5 This is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0039] Figure 6 This is a schematic diagram of images captured by the main camera and the telephoto camera provided in an embodiment of the present application;

[0040] Figure 7 It is a schematic diagram of a set of image flows involved in the prior art;

[0041] Figure 8 This is a schematic flow chart of an image processing method provided in an embodiment of the present application;

[0042] Figure 9 is a schematic flow chart of another image processing method provided in an embodiment of the present application;

[0043] Figure 10 This is a set of schematic diagrams provided in the embodiments of the present application;

[0044] Figure 11 This is another set of schematic diagrams provided in the embodiments of the present application;

[0045] Figure 12 This is another set of schematic diagrams provided in the embodiments of the present application;

[0046] Figure 13 This is another set of schematic diagrams provided in the embodiments of the present application;

[0047] Figure 14 is a schematic flow chart of another image processing method provided in an embodiment of the present application;

[0048] Figure 15 This is another set of schematic diagrams provided in the embodiments of the present application;

[0049] Figure 16 This is a diagram showing the corresponding relationship between the camera operation status and the zoom ratio provided in the embodiment of the present application;

[0050] Figure 17 is a set of image streams involved in the embodiments of the present application;

[0051] Figure 18 is a schematic flow chart of another image processing method provided in an embodiment of the present application;

[0052] Figure 19 This is a schematic diagram of an application scenario of zoom switching provided by an embodiment of the present application;

[0053] Figure 20 This is a schematic diagram of an application scenario of zoom switching provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In the embodiments of this application, the terms "first," "second," and the like are used for descriptive purposes only and should not be understood to indicate or imply relative importance or to implicitly indicate the number of the technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0055] First, some of the terms used in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0056] 1. Field of view (FOV) in optical instruments is the angle formed by the two edges of the maximum range through which the image of the measured object can pass through the lens, with the lens as the vertex. The size of the field of view determines the field of view of the optical instrument. The larger the field of view, the greater the field of view and the smaller the optical magnification, that is, the target object will not be captured by the lens if it exceeds this angle. The shorter the focal length, the wider the horizontal field of view, and therefore the smaller the image. The horizontal field of view narrows as the focal length increases, and the object being photographed also increases in size.

[0057] 2. Registration refers to the matching of geographic coordinates of different images obtained by different imaging methods within the same area. This includes three aspects: geometric correction, projection transformation, and uniform scale.

[0058] 3. Zoom ratio: The zoom ratio is used to indicate the zoom size of an electronic device when shooting.

[0059] 4. Optical flow algorithm. Optical flow refers to the instantaneous speed of pixel motion of a moving object on the observation imaging plane. Optical flow algorithms use the temporal changes in pixels in an image sequence and the correlation between adjacent frames to find the correspondence between the previous and current frames, thereby calculating the motion information of objects between adjacent frames.

[0060] For example, point-by-point matching or tracking of a group of points can be performed on an image. By calculating the offset of the pixel points on the image, an optical flow field is formed, and pixel-level image registration is performed through this optical flow field.

[0061] The above is a brief introduction to the nouns involved in the embodiments of this application, and no further details will be given below.

[0062] The hardware system, software system and application scenarios of the electronic device provided in the embodiments of the present application are described below with reference to the accompanying drawings.

[0063] Exemplarily, the electronic device 100 may be a mobile phone, a smart screen, a tablet computer, a wearable electronic device, an in-vehicle electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a projector, etc. The embodiments of the present application do not impose any restrictions on the specific type of the electronic device 100.

[0064] See also Figure 1The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0065] It should be noted that Figure 1 The structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include Figure 1 More or fewer components than those shown, or the electronic device 100 may include Figure 1 Combinations of some of the components shown, or alternatively, the electronic device 100 may include Figure 1 Subassemblies of some of the components shown. Figure 1 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0066] The processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). The different processing units may be independent devices or integrated devices.

[0067] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0068] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0069] Exemplarily, the processor 110 can be used to execute the image processing method of an embodiment of the present application; for example, opening a camera application; displaying a first image, where the first image is obtained by capturing an image with a first camera; receiving a first zoom operation; and displaying and saving a second image, where the second image is obtained by segmenting the image captured by the first camera into modules using the parallax between the first camera and the image captured by the second camera, and then correcting each module using its corresponding warp matrix and then stitching them together. By performing warp correction and stitching between modules, the present application can achieve full depth alignment, thereby achieving smooth zooming and smooth camera switching.

[0070] Figure 1 The connection relationship between the modules shown is only for illustrative purposes and does not limit the connection relationship between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of the multiple connection modes in the above embodiments.

[0071] The wireless communication function of the electronic device 100 can be implemented through components such as the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor.

[0072] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0073] Electronic device 100 can implement display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0074] Display screen 194 may be used to display images or videos.

[0075] For example, in an embodiment of the present application, the display screen 194 may be used to display the second image.

[0076] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0077] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can perform algorithmic optimization on image noise, brightness, and color. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0078] The camera 193 (also called the lens) is used to capture still images or videos. It can be triggered to turn on through application instructions to realize the photo function, such as capturing images of any scene. The camera may include components such as an imaging lens, a filter, and an image sensor. The light emitted or reflected by the object enters the imaging lens, passes through the filter, and finally converges on the image sensor. The imaging lens is mainly used to focus the light emitted or reflected by all objects in the photographic field of view (also called the scene to be photographed, the target scene, or the scene image that the user expects to capture) to form an image; the filter is mainly used to filter out excess light waves in the light (for example, light waves other than visible light, such as infrared); the image sensor can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The image sensor is mainly used to perform photoelectric conversion on the received light signal, convert it into an electrical signal, and then transmit the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB, YUV, etc.

[0079] Exemplarily, the gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., the x-axis, the y-axis, and the z-axis) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse motion to achieve anti-shake. The gyroscope sensor 180B can also be used in scenarios such as navigation and somatosensory games.

[0080] In some embodiments, the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1.

[0081] Among them, the camera 193 can be located in the front of the electronic device 100 or in the back of the electronic device 100. The specific number and arrangement of the cameras can be set according to needs, and this application does not impose any restrictions.

[0082] Exemplarily, the electronic device 100 includes a front camera and a rear camera. For example, the front camera or the rear camera may include one or more cameras. Taking the example of the electronic device 100 having four rear cameras, when the electronic device 100 activates the four rear cameras for shooting, the image processing method provided in the embodiments of the present application can be used.

[0083] Alternatively, the camera is provided on an external accessory of the electronic device 100, and the external accessory is rotatably connected to the frame of the mobile phone, and the angle formed between the external accessory and the display screen 194 of the electronic device 100 is any angle between 0 and 360 degrees. For example, when the electronic device 100 takes a selfie, the external accessory drives the camera to rotate to a position facing the user. Of course, when the mobile phone has multiple cameras, only some of the cameras can be provided on the external accessory, and the remaining cameras can be provided on the electronic device 100 body. This embodiment of the application does not impose any restrictions on this.

[0084] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0085] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0086] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., the x-axis, the y-axis, and the z-axis) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for shooting anti-shake. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used in scenarios such as navigation and somatosensory games.

[0087] Accelerometer 180E can detect the magnitude of the acceleration of electronic device 100 in various directions (generally the x-axis, y-axis, and z-axis). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. Accelerometer 180E can also be used to identify the posture of electronic device 100, which can serve as an input parameter for applications such as landscape / portrait switching and pedometers.

[0088] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. In some embodiments, for example, in a photography scenario, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0089] Ambient light sensor 180L is used to sense ambient light brightness. Electronic device 100 can adaptively adjust the brightness of display screen 194 based on the perceived ambient light. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether electronic device 100 is in a pocket to prevent accidental touches.

[0090] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement functions such as unlocking, accessing application locks, taking photos, and answering calls.

[0091] The touch sensor 180K is also referred to as a touch-sensitive device. The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also referred to as a touch screen. The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor 180K can transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100 and at a different location from the display screen 194.

[0092] The hardware system of the electronic device 100 is described in detail above. The software system of the electronic device 100 is introduced below.

[0093] Figure 2 Schematic diagram of the software system of the electronic device provided in an embodiment of the present application.

[0094] like Figure 2 As shown, the system architecture may include an application layer 210 , an application framework layer 220 , a hardware abstraction layer 230 , a driver layer 240 and a hardware layer 250 .

[0095] The application layer 210 may include applications such as camera application, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0096] The application framework layer 220 provides an application programming interface (API) and a programming framework for the application programs of the application layer; the application framework layer may include some predefined functions.

[0097] For example, the application framework layer 220 may include a camera access interface, which may include camera management and camera devices. The camera management may be used to provide an access interface for managing the camera, and the camera device may be used to provide an interface for accessing the camera.

[0098] The hardware abstraction layer 230 is used to abstract the hardware. For example, the hardware abstraction layer may include a camera abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer may call camera algorithms.

[0099] For example, the hardware abstraction layer 230 includes a camera hardware abstraction layer and a camera algorithm; the camera algorithm may include a software algorithm for image processing.

[0100] Exemplarily, the camera algorithm library may include algorithms corresponding to the image processing method provided in the embodiments of the present application.

[0101] For example, the algorithm in the camera algorithm may refer to a code that does not rely on specific hardware implementation; for example, a code that can generally be run in a CPU, etc.

[0102] The driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include a camera driver.

[0103] The hardware layer 250 is located at the bottom layer of the operating system; Figure 2 As shown, the hardware layer 250 may include camera 1, camera 2, camera 3, etc. Among them, camera 1, camera 2, camera 3 may correspond to multiple cameras on the electronic device.

[0104] For ease of understanding, the following takes the electronic device 100 as an example of a mobile phone having the above-mentioned software and hardware structure, and first describes in detail the camera on the electronic device 100 to which the method provided in the embodiment of the present application is applicable.

[0105] The electronic device to which the method provided in the embodiment of the present application is applicable has at least multiple cameras 193, for example, three types of cameras 193; the three types of cameras are a main camera (for example, a wide-angle camera), an ultra-wide-angle camera and a telephoto camera; the three cameras can be used to shoot the same scene to be shot.

[0106] Optionally, the electronic device 100 may also have other cameras 193. The type of cameras 193 and the number of each type of camera 193 may be set as needed, and the embodiment of the present application does not impose any restrictions on this.

[0107] For example, Figure 3 As shown in the figure, the electronic device 100 may have three cameras 193 as an example for explanation; the arrangement of the three cameras can be as follows Figure 3 As shown in (a) in , or as Figure 3 As shown in (b) in FIG; for example, the three cameras 193 can be a main camera 1931 (for example, a wide-angle camera), an ultra-wide-angle camera 1932 and a telephoto camera 1933.

[0108] It should be understood that the above are only examples of two arrangements, and other arrangements are also possible; the specific arrangement can be designed and changed as needed, and the embodiments of the present application do not impose any restrictions on this.

[0109] It should be noted that when the above three cameras are shooting, the field of view angle range corresponding to the main camera 1931 is usually larger than the field of view angle range corresponding to the telephoto camera 1933; and the field of view angle range corresponding to the ultra-wide-angle camera 1932 is larger than the field of view angle range corresponding to the main camera 1931; there may be overlap between the field of view angle of the ultra-wide-angle camera 1932 and the field of view angle of the main camera 1931; that is, the ultra-wide-angle camera 1932 can capture the scene content captured by the main camera 1931 and the scene content around it.

[0110] It should be understood that the field of view of the telephoto camera 1933 is smaller than that of the main camera 1931, and the field of view of the main camera 1931 and the field of view of the telephoto camera 1933 may overlap; that is, the main camera 1931 can capture the scene content captured by the telephoto camera 1933 and the surrounding scene content. The field of view of the ultra-wide-angle camera 1932 and the field of view of the telephoto camera 1933 may overlap; that is, the ultra-wide-angle camera 1932 can capture the scene content captured by the telephoto camera 1933 and the surrounding scene content.

[0111] Among them, the ultra-wide-angle camera 1932 is suitable for shooting close-ups due to its smaller focusing distance; and, as the name suggests, the ultra-wide-angle camera 1932 is suitable for shooting scenes with a larger field of view; the main camera 1931 is more suitable for shooting portraits due to its higher clarity, while the telephoto camera 1933 is more suitable for shooting distant close-ups.

[0112] For example, Figure 4 As shown, the zoom ratio of the ultra-wide-angle camera can be less than M times the zoom ratio; the zoom ratio range of the wide-angle camera, i.e., the main camera, can be [M, N); the zoom ratio of the telephoto camera can be greater than or equal to N times the zoom ratio.

[0113] For example, M can be 1 and N can be 2.5; then the zoom magnification of the ultra-wide-angle camera is less than 1x zoom magnification (1×); the zoom magnification range of the wide-angle camera is 1x zoom magnification to 2.5x zoom magnification [1×~2.5×); the zoom magnification of the telephoto camera is greater than or equal to 2.5x zoom magnification.

[0114] It should be understood that during the shooting process of an electronic device, the greater the zoom ratio, the smaller the corresponding field of view angle.

[0115] The following combination Figure 5 The application scenarios of the image processing method provided in the embodiments of the present application are illustrated with examples.

[0116] The method in the embodiments of the present application can be applied to photo-taking scenarios, preview scenarios, video recording scenarios, or video call scenarios, etc.; through the method in the embodiments of the present application, smooth zoom and / or smooth switching of different cameras can be achieved in electronic devices, thereby improving the user's shooting experience and image quality.

[0117] For example, the preview scenarios include but are not limited to the following scenarios:

[0118] Photo preview, aperture preview, night scene preview, portrait preview, video preview or professional preview, etc.

[0119] It should be understood that the preview scene may refer to a scene in which the electronic device captures an image before a button indicating shooting is clicked in a certain shooting mode.

[0120] In one example, if Figure 5 As shown in (a) in FIG, after the electronic device enters the camera application, the default photo mode can be turned on; in the photo mode, the electronic device can enter the default shooting mode, which can refer to a shooting mode in which the wide-angle camera is used as the main camera and the zoom ratio is a single zoom ratio (1×), and the electronic device displays the image captured by the main camera; Figure 5 As shown in (b), in response to the user's operation, when the zoom ratio satisfies the zoom range corresponding to the telephoto camera, the electronic device can switch to display the image captured by the telephoto camera.

[0121] For example, the image processing method in the embodiment of the present application may also be applied to a video call scenario, wherein the video call scenario may include but is not limited to the following scenarios:

[0122] Video calls, video conferencing applications, long and short video applications, live video applications, online video courses, portrait intelligent mirroring application scenarios, system camera recording function video recording, video surveillance, or smart cat-eye and other portrait shooting scenarios, etc.

[0123] It should be understood that the above is an example of an application scenario and does not limit the application scenario of this application.

[0124] Currently, electronic devices can switch between cameras with different focal lengths to perform zoom shooting in response to user input. They can also combine digital zoom with image processing to accommodate a variety of high-magnification shooting scenarios. To achieve smooth zooming during zooming, the SAT algorithm is commonly used in existing technologies. This algorithm corrects each pair of simultaneously captured images from the two cameras used for zoom switching to achieve consistent FOV.

[0125] For example, Figure 6 As shown, when the zoom is switched from the main camera to the telephoto camera, before the switch, both the main camera and the telephoto camera capture images, and the image captured by the main camera is displayed. After the switch, both the main camera and the telephoto camera continue to capture images, but the image captured by the telephoto camera is displayed. During this zoom process, in order to maintain smooth zooming, the FOV of each pair of images captured by the main camera and the telephoto camera must be kept consistent. To this end, as the zoom ratio increases, the image captured by the telephoto camera is usually used as the basis to calculate the correction parameters for transforming the image captured from the main camera to the image captured by the telephoto camera, and then the correction parameters are applied to the image captured by the main camera, so that the FOV of the corrected image is consistent with the image captured by the telephoto camera. Among them, the correction parameters generally include a rotation amount for representing the rotation relationship and an offset amount for representing the offset relationship. The rotation amount can be obtained by existing related technologies and will not be introduced here. In the prior art, the offset amount can be obtained by aligning the image captured by the main camera and the image captured by the telephoto camera, and then calculating based on the difference in feature points.

[0126] However, this calculation process cannot guarantee the accuracy of the registration. For example, for scenes with repeated textures or weak textures, incorrectly registered feature points may appear. Errors in feature point registration will lead to differences in feature points, that is, offset calculation errors, which will cause errors in the correction parameters. As a result, the image corrected using the correction parameters will not have the same FOV as the image captured by the telephoto camera. Repeated texture scenes refer to image contents with repetitive styles and patterns, such as floral patterns and tree patterns. Texture scenes refer to texture features with relatively small eigenvalues, such as wall cracks and brick patterns.

[0127] In addition, even if all feature points are correctly registered, the offsets of feature points corresponding to different depths are different, and a unified offset cannot be used to represent the offsets of all feature points in the image. For example, if the depths of the photographed objects are different, such as the depths of the foreground and background, the depths corresponding to the feature points extracted in the foreground and the background will be different. Then, since the foreground and background in the image have a perspective relationship of being larger when closer and smaller when farther away, the offset of the feature points in the foreground for the same actual physical distance is larger, and the offset of the feature points in the background is smaller. Therefore, when performing correction, the offset determined in a fixed manner cannot be used as the offset for all feature points in the image.

[0128] However, the center-priority and foreground-priority processing strategies provided in the prior art result in that, after the depth is calculated, feature points with smaller or closer depths are collected to calculate the offset; when the feature points with closer depths extracted from the foreground are insufficient during the zoom process, feature points with farther depths in the background are added to calculate the offset, resulting in abnormal image effects after correction.

[0129] For example, Figure 7 is a set of image streams involved in the prior art. Figure 7 As shown in (a) to (f) of the figure, assuming the captured image only contains the foreground and background, during zooming, the foreground and background images from the two cameras differ significantly. If the foreground is aligned, the background will not align; similarly, if the background is aligned, the foreground will not align. Because only one depth can be guaranteed at a time, the algorithm must constantly determine which depth to align with, the foreground or the background. This causes the foreground and background to jump back and forth during zooming, preventing smooth camera switching.

[0130] In view of this, an embodiment of the present application provides an image processing method and related equipment. In the embodiment of the present application, parallax statistics are performed on the image captured by the first camera and the image captured by the second camera; the image captured by the first camera and the image captured by the second camera are then divided into modules based on the parallax distribution information; based on the modules after the segmentation of the image captured by the first camera and the image captured by the second camera, a warp matrix between each two corresponding modules is calculated; the modules included in the image captured by the first camera are corrected in combination with their corresponding warp matrices, and the stitched second image is displayed after correction. The present application divides the image based on parallax (depth of field) and implements warp correction corresponding to each module in a module-by-module manner, so that the foreground and background can be warped separately to achieve full depth alignment, thereby ensuring smooth zooming and smooth switching during camera switching.

[0131] The following combination Figure 8 The schematic flowchart of the image processing method provided in the embodiment of the present application is described in detail.

[0132] Figure 8 is a schematic flow chart of the image processing method provided in the embodiment of the present application. Figure 1 The electronic device shown is executed; the method 300 includes steps S310 to S340, and steps S310 to S340 are described in detail below.

[0133] It should be understood that in the embodiments of the present application, the electronic device includes a first camera and a second camera; wherein the first camera and the second camera are cameras of different types.

[0134] S310. Start the camera application.

[0135] For example, the user may instruct the electronic device to start the camera application by clicking the icon of the “Camera” application.

[0136] For example, when the electronic device is in the lock screen state, the user can instruct the electronic device to launch the camera application by swiping right on the electronic device's display. Alternatively, when the electronic device is in the lock screen state and the lock screen interface includes a camera application icon, the user can instruct the electronic device to launch the camera application by clicking the camera application icon. Alternatively, when the electronic device is running another application, the application has permission to call the camera application; the user can instruct the electronic device to launch the camera application by clicking the corresponding control. For example, when the electronic device is running an instant messaging application, the user can instruct the electronic device to launch the camera application by selecting the camera function control.

[0137] It should be understood that the above is an example of the operation of opening the camera application; the camera application can also be opened by voice instructions or other operations to instruct the electronic device; this application does not impose any limitation on this.

[0138] It should also be understood that launching the camera application may refer to running the camera application.

[0139] S320: Display a first image, where the first image is obtained by capturing an image with a first camera.

[0140] For example, the first camera may be Figure 3 The wide-angle camera, main camera shown, or other cameras with a larger field of view than the telephoto camera.

[0141] Exemplarily, the first image may be an RGB image captured by the main camera, or in other words, the first image may be an RGB image obtained by processing a series of camera algorithms from a Raw image captured by the main camera.

[0142] For example, Figure 19 The preview image shown in (a) is the first image captured and displayed by the first camera. For another example, Figure 20 The video image shown in (a) is the first image captured and displayed by the first camera.

[0143] Optionally, when displaying the first image captured by the first camera, the depth corresponding to the photographed object may also be captured.

[0144] Illustratively, before displaying the first image, the first camera may further perform disparity detection on the content in the first image to determine the depths corresponding to different photographed objects included in the first image.

[0145] For example, Figure 20 As shown in (a), when the first camera sends the first image for display, it can perform parallax detection to determine the depth corresponding to different photographed objects. The depth information can be displayed on the first image or not, and this application does not limit this.

[0146] It should be understood that the first image may include one or more photographed objects, and when displayed, a detection frame may be displayed simultaneously for each photographed object determined based on AI detection.

[0147] It should also be understood that the type of subject can be preset as needed. For example, when the type is set to face, the target object can be used to indicate a face recognized in the first image. The type of subject can also include plants, animals, etc. This embodiment of the application does not impose any restrictions on this.

[0148] S330: Receive a first zoom operation.

[0149] Optionally, receiving the first zoom operation may include: receiving the first zoom operation for the first image.

[0150] It should be understood that the zoom operation on the first image may indicate an operation instruction of the user, or may indicate a zoom operation instruction automatically triggered by the electronic device based on AI detection. The target object is one of the shooting objects included in the first image.

[0151] Exemplarily, when the zoom operation indicates an operation instruction triggered by the user, the zoom operation may include a two-finger reverse sliding operation, a click operation, a voice operation, an air gesture operation, etc. on the first image, and the embodiments of the present application do not impose any restrictions on this.

[0152] For example, Figure 20 (a) and Figure 20As shown in (b), when the electronic device displays one or more photographic objects and a zoom control, the zoom operation may include a click operation on the target object in the first image and a sliding operation on the zoom control; or, the zoom operation may also include a click operation on the target object in the first image and a two-finger reverse sliding operation. It should be understood that when the zoom operation includes two operations, when executing, the user needs to first perform the click operation, and then perform the sliding operation on the zoom control or the two-finger reverse sliding operation on the target object. The zoom operation can also be other operations, or the zoom operation can also include three or more sub-operations, and the embodiments of the present application do not impose any restrictions on this.

[0153] S340: Display and save the second image.

[0154] Exemplarily, the first image may be an RGB image, and the corresponding second image may be an RGB image.

[0155] The second image is obtained by segmenting the image captured by the first camera into blocks based on the parallax between the image and the second camera. Each block is then corrected for its corresponding warp matrix and stitched together. Block segmentation, also known as image segmentation, divides the camera image into distinct image blocks based on parallax. Furthermore, each block has a different warp matrix.

[0156] Among them, the warp matrix may include one or more of the rotation matrix, offset matrix and scaling matrix, or a composite matrix of multiple items. In addition, the warp may also include other data, which is not limited in the embodiment of the present application.

[0157] It should be understood that in response to the first zoom operation, the disparity statistics of the image captured by the first camera and the image captured by the second camera are performed to obtain a disparity map; for the disparity map, the disparity distribution information is statistically analyzed, and the disparities belonging to the same category are divided into modules; in this way, it is equivalent to dividing the image captured by the first camera and the image captured by the second camera into multiple modules according to the depth; based on the modules after the image captured by the first camera and the image captured by the second camera are divided, the warp matrix between each two corresponding modules can be calculated; the modules included in the image captured by the first camera are corrected in combination with their corresponding warp matrices, and then they can be spliced ​​to obtain the second image.

[0158] Optionally, the second image may be obtained by segmenting the image captured by the first camera into modules using the parallax between the image captured by the second camera, performing warp correction on each module, and then aligning and stitching adjacent modules. Aligning adjacent modules refers to performing warp correction using similar content between adjacent modules.

[0159] It should be understood that in response to the first zoom operation, the image captured by the first camera and the image captured by the second camera are segmented after performing parallax statistics, and the image captured by the first camera and the image captured by the second camera are segmented into multiple modules; based on the modules after the segmentation of the image captured by the first camera and the image captured by the second camera, the warp matrix between the two corresponding modules in the two frames of images is calculated; then, the different modules of the image captured by the first camera are corrected in combination with their corresponding warp matrices. Thereafter, for the corrected image captured by the first camera, the warp matrix between adjacent modules within the frame is calculated, and for the corrected image captured by the first camera, the warp matrix between the modules is used to align the modules, and then all the modules are spliced ​​together to obtain the second image.

[0160] For example, the first zoom operation is an operation of increasing the zoom ratio, or the first zoom operation is an operation of decreasing the zoom ratio. When the zoom ratio is increased, the size of the subject included in the second image is larger than the size of the subject included in the first image. When the zoom ratio is decreased, the size of the subject included in the second image is smaller than the size of the subject included in the first image.

[0161] Optionally, the target object in the second image is centered in the second image.

[0162] Optionally, the second image includes the entire content or a portion of the target object. Figure 20 As shown, assuming that the target object is the fourth subject, after magnification, the second image may include the entire fourth subject, or the second image may also include partial content of the fourth subject, such as the face of the fourth subject; when further magnified, the second image may also include a partial face of the fourth subject.

[0163] Optionally, in order to ensure that the target object displayed by the electronic device can be smoothly enlarged during the zoom process, the above S340 can be executed in a loop multiple times, and the target object in the second image obtained each time is larger than the target object obtained last time. That is, in response to the zoom operation, the electronic device corrects the offset between the image captured by the first camera and the image captured by the second camera, and obtains and displays multiple second images, each of which includes the target object, and the target object in the multiple second images gradually increases from the original size. The number of loop executions and the magnitude of the size change can be set as needed, and this application does not limit this. Among them, the original size indicates the size of the target object in the first image.

[0164] It should be understood that S340 may be executed while S330 is being executed, or S340 may be executed after S330 is executed. This embodiment of the present application does not impose any limitation on this.

[0165] For example, when in the preview scene, the second image can be displayed. Wherein, the preview shooting mode includes but is not limited to: shooting modes such as night scene preview mode, video preview mode, photo preview mode, and portrait preview mode.

[0166] Exemplarily, when in a video recording scene, the second image may be displayed and saved.

[0167] Optionally, the method further includes:

[0168] In response to a first zoom operation;

[0169] Determine whether the zoom ratio of the electronic device meets the zoom ratio range of the first camera foreground display and the second camera background operation.

[0170] Exemplarily, in response to the first zoom operation, when the zoom ratio has reached the minimum zoom ratio of the second camera but has not reached the maximum zoom ratio of the first camera (for example, the telephoto camera is at least 1.8×, and the main camera is at most 2.5×), at this time, the second image acquired by the first camera continues to be displayed, but the second image is obtained by dividing the image captured by the first camera into modules using the parallax between the image captured by the second camera, and then performing corresponding warp matrix correction on each module and then stitching the images together. The warp matrices corresponding to the various modules are different.

[0171] For example, Figure 16 As shown, the first zoom operation is to switch from 1× to 4×, and the first camera is always displayed in the foreground, but when the zoom is changed to 1.8×, the second camera starts to run in the background. At this time, the method provided in the embodiment of the present application can be triggered to calculate multiple warp matrices to perform module correction on the image captured by the first camera, and then generate a second image for display and storage.

[0172] The embodiment of the present application provides an image processing method and related equipment. In the embodiment of the present application, the disparity statistics of the image captured by the first camera and the image captured by the second camera are performed; the image captured by the first camera and the image captured by the second camera are then divided into modules based on the disparity distribution information; based on the modules after the image captured by the first camera and the image captured by the second camera are divided, the warp matrix between each two corresponding modules is calculated; the modules included in the image captured by the first camera are corrected in combination with their corresponding warp matrices, and the second image obtained after correction is displayed. The present application divides the image based on disparity (depth of field) and implements the warp correction corresponding to each module in a module-by-module manner, so that the foreground and background can be warped separately to achieve full depth alignment, thereby ensuring smooth zooming and smooth switching during camera switching.

[0173] For example, Figure 17 is a set of image streams involved in this application. Figure 17 (a), (b), (c), (d), (e) to Figure 17 As shown in (f), during the zoom process, it is assumed that the acquired image is divided into only foreground modules and background modules after module segmentation based on the disparity distribution information; then, the warp matrix between the foreground module of the image acquired by the first camera and the foreground module of the image acquired by the second camera is calculated, and the foreground module of the image acquired by the first camera is corrected using the warp matrix; and, the warp matrix between the background module of the image acquired by the first camera and the background module of the module acquired by the second camera is calculated, and the background module of the image acquired by the first camera is corrected using the warp matrix; since the present application segments the image based on disparity and implements the warp correction corresponding to each module, the foreground and background can be warped separately to achieve full depth alignment, thereby enabling the FOV of the image stream to be smoothly enlarged and the camera to be smoothly switched during the zoom process.

[0174] Figure 9 A schematic flowchart exemplarily shows another image processing method of the electronic device 100 .

[0175] Taking the first camera as a main camera and the second camera as a telephoto camera as an example, the method 400 includes S401 to S410; S401 to S410 are described in detail below.

[0176] S401: Acquire a first initial image captured by a main camera.

[0177] Exemplarily, the image captured by the main camera may be a Raw image captured by a wide-angle camera, or a YUV image.

[0178] S402: Acquire a second initial image captured by the telephoto camera.

[0179] Exemplarily, the image captured by the telephoto camera may be a Raw image or a YUV image captured by the telephoto camera.

[0180] S403 : Perform image feature point detection and registration on the first initial image and the second initial image to obtain multiple pairs of feature point pairs.

[0181] Optionally, the image feature point detection and registration may adopt any existing image feature point detection and registration algorithm, which is not limited in this application, for example, an ORB (Oriented FAST and Rotated BRIEF) algorithm, an optical flow algorithm, etc.

[0182] Exemplarily, image feature point detection is performed on the first initial image and the second initial image, respectively, to extract M feature points from the first initial image and N feature points from the second initial image; then, the M feature points and the N feature points are registered, where both M and N are integers greater than 0.

[0183] It should be understood that each set of feature point pairs includes a feature point belonging to the first initial image and a feature point belonging to the second initial image, and the two feature points are used to indicate the same object. The feature points after registration have a one-to-one correspondence.

[0184] Optionally, after S403 , the method may further include: screening correctly registered feature point pairs.

[0185] Specifically, the correctly registered feature point pairs can be screened out by eliminating the incorrectly registered feature point pairs.

[0186] After testing, in each scene, there are more correctly registered feature points than incorrectly registered feature points. Therefore, in order to improve processing efficiency and the effect of subsequent processing, the incorrectly registered feature points can be eliminated and only the correctly registered feature point pairs are retained.

[0187] S404: Calculate the disparity of each pair of feature point pairs among all feature point pairs to obtain a disparity map.

[0188] It should be understood that binocular ranging technology, also known as stereo vision technology, calculates the disparity between two images to obtain three-dimensional information about objects in the scene, including depth (i.e., distance) and size. The core of binocular ranging lies in calculating the disparity between the two images. First, the two cameras are calibrated to determine their relative position and posture. Then, through image processing technology, corresponding feature points are identified in the images of the two cameras. Next, based on the positional relationship of these corresponding feature points, the disparity is calculated using a stereo matching algorithm. Based on the disparity and the camera calibration parameters, the three-dimensional coordinates of the objects in the scene can be calculated, and the size of the objects can be obtained.

[0189] In the embodiment of the present application, the disparity between the first initial image and the second initial image may be determined using binocular ranging technology.

[0190] S405 , performing statistics and classification on the disparity distribution information of the disparity map to determine multiple classes.

[0191] Optionally, for the disparity map, disparity distribution information is statistically analyzed, and if clustering is performed, the disparity map is divided into multiple classes according to clusters formed by a clustering algorithm.

[0192] Clustering algorithm is a method in unsupervised learning, which is used to divide the samples in the data set into several "clusters" so that the similarity between samples in the same cluster is high, while the similarity between samples in different clusters is low.

[0193] It should be understood that if the disparity distribution information is not aggregated but dispersed, the method provided in the embodiment of the present application cannot be used to continue processing, and other processing methods can be selected, which is not limited in the embodiment of the present application.

[0194] For example, Figure 10 The image shown in (a) is the first initial image. Figure 10 The image shown in (b) is the second initial image. After image feature point detection and registration are performed on the first and second initial images, multiple pairs of feature point pairs (also called inter-frame feature point pairs) are generated. The disparity of each set of feature point pairs is calculated to obtain a disparity map, such as Figure 10 As shown in (c) in the figure; for the disparity map, the disparity distribution information is statistically analyzed and divided into multiple classes according to the clusters divided by the clustering algorithm, for example, Figure 10 As shown in (d), it can be divided into five categories: a, b, c, d and e.

[0195] S406 , performing module segmentation on both the first initial image and the second initial image based on the disparities belonging to the same category.

[0196] It should be understood that here, based on the classification results, both the first initial image and the second initial image are segmented into modules, wherein disparities belonging to the same category belong to the same module, and the number of modules in the segmented first initial image and the second initial image can be the same or different.

[0197] It should be noted that the module corresponding to the same type of disparity may include multiple submodules, and the positions of the submodules may not be adjacent; in other words, multiple submodules located at different positions may belong to the module corresponding to the same type of disparity.

[0198] For example, in combination Figure 10 The classification shown in (d) in FIG, performs module segmentation on the disparity of the first initial image and the second initial image belonging to the same category, and the first initial image after segmentation is as follows Figure 11 As shown in (a) of FIG11 , the first initial image includes five modules surrounded by different dashed lines. The second initial image after segmentation is shown in (b) of FIG11 , which includes five modules surrounded by different dashed lines. The first module in the first initial image may include three submodules, and the first submodule in the second initial image may include two submodules.

[0199] Optionally, after S406 , S407 to S410 may be continued to be executed, or S408 to S410 may be continued to be executed.

[0200] S407: Add a certain amount of margin (tolerance range) around the module.

[0201] The size of the margin can be set as needed and is not limited in this embodiment of the present application. A uniform margin can be added to the periphery of multiple modules, or margins of different sizes can be added; and a uniform margin can be added to the different sides of each module, or margins of different sizes can be added.

[0202] In addition, for edges of the module that belong to the edges of the first initial image or the second initial image, the margin may not be increased.

[0203] For example, Figure 11 As shown in (c) in the figure, Figure 11 Schematic diagram of adding margin to the first initial image after module segmentation shown in (a); Figure 11 As shown in (d) in the figure, Figure 11 Schematic diagram of adding margin to the second initial image after module segmentation shown in (b).

[0204] It should be understood that by adding a certain amount of margin to the periphery of the module, there will be overlapping areas between adjacent modules, which will facilitate subsequent module splicing and enable the modules to be better spliced ​​together.

[0205] S408 . Based on the feature point pairs obtained in S403 and in combination with the segmented modules, calculate the warp matrix corresponding to each module.

[0206] It should be understood that based on the feature point pairs and combined with the segmented modules, the warp matrix between the corresponding modules of the first initial image and the second initial image can be calculated. For example, if the first initial image and the second initial image include 5 groups of corresponding modules, then 5 warp matrices between the 5 groups of corresponding modules can be calculated.

[0207] For example, Figure 12 As shown in (a) and (b) in FIG, combining the feature point pairs and the multiple modules after segmentation, the feature points included in the first module in the first initial image and the feature points matched therewith in the second initial image can be used for calculation to obtain the warp matrix between the first module in the first initial image and the first module in the second initial image, as shown in FIG. Figure 12 Similarly, the warp matrix between the second modules in the two images can be calculated in sequence, such as warp2; the warp matrix between the third modules in the two images can be calculated in sequence, such as warp3; the warp matrix between the fourth modules in the two images can be calculated in sequence, such as warp4; and the warp matrix between the fifth modules in the two images can be calculated in sequence, such as warp5.

[0208] It should be understood that for a module including multiple sub-modules, multiple sub-modules belonging to the same module correspond to the same warp matrix.

[0209] S409 , performing correction on the modules after module segmentation of the first initial image based on the warp matrices corresponding to the modules.

[0210] The warp matrix corresponding to the module here is the first warp matrix.

[0211] S410: obtain a second image after stitching.

[0212] Optionally, when correcting the first initial image, the warp matrix corresponding to each module may be combined to correct the first initial image module by module, and the image obtained by stitching after correction may be saved and displayed as the second image.

[0213] For example, Figure 13 As shown in (a), the modules after module segmentation of the first initial image are corrected and spliced ​​based on their corresponding warp matrices, and the following can be obtained: Figure 13 The image shown in (b) in FIG. 1 can be displayed and saved as the second image.

[0214] Optionally, for the module that does not execute S407, that is, does not add margin, after correction, the modules may be fused into one image using a Poisson Fusion algorithm, and the fused image may be saved and displayed as the second image.

[0215] It should be noted that the Poisson fusion algorithm achieves image fusion by solving the Poisson equation, which can create a natural and realistic synthesis effect.

[0216] In an embodiment of the present application, in response to a first zoom operation, disparity statistics are calculated for the image captured by the first camera and the image captured by the second camera to obtain a disparity map. For the disparity map, disparity distribution information is statistically analyzed, and disparities belonging to the same category are segmented into modules. This is equivalent to segmenting the image captured by the first camera and the second camera into multiple modules according to depth. Based on the modules segmented from the first camera and the second camera, a warp matrix is ​​calculated between each two corresponding modules. The modules included in the image captured by the first camera are corrected using their corresponding warp matrices, and then the modules are stitched together to obtain a second image. Because the foreground and background are each warped separately through module segmentation, full depth alignment can be achieved.

[0217] Figure 14 A schematic flowchart exemplarily shows another image processing method of the electronic device 100 .

[0218] Taking the first camera as the main camera and the second camera as the telephoto camera as an example, the method 400 includes S401 to S409 and can further include S411 to S415; S401 to S409 can refer to the above-mentioned Figure 9 The description of S411 to S415 will be described in detail below.

[0219] S411 , performing feature point matching on two adjacent modules in the first initial image in S409 to obtain multiple sets of matched feature point pairs (also referred to as intra-frame feature point pairs).

[0220] It should be understood that, since a certain amount of margin is added to the modules after segmentation of the first initial image in S407 , adjacent modules will have a certain overlap area, ie, the area of ​​the adjacent modules at adjacent sides is the area of ​​the margin added.

[0221] For example, Figure 15As shown, for the first module and the second module in the first initial image, the first module and the second module are adjacent to each other up and down, and the margin area added by the lower edge of the first module and the margin area added by the upper edge of the second module will have a certain overlap. Therefore, when performing feature point matching, multiple sets of feature point pairs may be generated.

[0222] S412: Calculate the warp matrix between the two adjacent modules based on the feature point pairs within the frame.

[0223] Here, the warp matrix between modules is the second warp matrix.

[0224] S413: Utilize the warp matrix between two adjacent modules to achieve module alignment.

[0225] Optionally, after generating feature point pairs, the feature point pairs are screened according to preset rules. For example, if the margin areas of adjacent modules have more overlapping content, more feature point pairs can be screened and retained; if the non-margin areas of adjacent modules have less overlapping content, fewer feature point pairs can be screened and retained.

[0226] For example, Figure 15 As shown, for the first and second adjacent modules in the first initial image, after feature point matching is performed to generate multiple feature point pairs, the feature point pairs in the margin and non-margin areas are filtered according to an 8:2 ratio. Using these filtered feature point pairs, the warp matrix between the two adjacent modules can be calculated, for example, warpA. At this point, warpA can be used to correct the first module, achieving alignment between the first and second modules. Similarly, the warp matrix between adjacent modules can be calculated for other modules, and alignment of the adjacent modules can be performed.

[0227] It should be understood that this is equivalent to further aligning the margin area after the non-margin area has been aligned, so as to achieve better splicing between modules.

[0228] S414: After all modules are aligned, the margin areas are overlapped and spliced ​​to achieve splicing of adjacent modules.

[0229] S415. Align the overall size and correct the edges.

[0230] For example, Figure 17 A set of image streams involved in the embodiment of this application. Figure 17During the zooming process from (a), (b), (c) to (d), by combining the image processing method provided in this application, image stitching can be achieved based on the feature points in the Margin area, thereby achieving smooth image changes and seamless switching between multiple cameras.

[0231] In an embodiment of the present application, in response to a first zoom operation, the image captured by the first camera and the image captured by the second camera are subjected to parallax statistics and then module segmentation is performed, and the image captured by the first camera and the image captured by the second camera are segmented into multiple modules; based on the modules after segmentation of the image captured by the first camera and the image captured by the second camera, the warp matrix between the two corresponding modules between the two frames of images is calculated; then, the different modules of the image captured by the first camera are corrected in combination with their respective corresponding warp matrices. Afterwards, for the image captured by the corrected first camera, the warp matrix between adjacent modules in the frame is calculated, and for the image captured by the corrected first camera, the warp matrix between the modules is used to align the modules, and then all the modules are spliced ​​together to obtain a second image. Since the foreground and background are each warped by module segmentation, full depth alignment can be achieved, thereby achieving seamless switching between multiple cameras.

[0232] Figure 18 The following is a schematic flow chart showing another image processing method of the electronic device 100. The method 500 includes S501 to S509, and S501 to S509 are described in detail below.

[0233] S501: Acquire an image captured by a main camera.

[0234] Exemplarily, the image captured by the main camera may be a Raw image captured by the wide-angle camera; or, the image captured by the main camera may be a Raw image captured by the ultra-wide-angle camera.

[0235] S502: First front-end processing.

[0236] Optionally, the first front-end processing may include a related algorithm for converting the Raw image captured by the main camera into a YUV image; this application does not impose any limitation on the algorithm.

[0237] For example, the first front-end processing may refer to an image processing algorithm executed in the ISP to convert a raw image into a YUV image. The first front-end processing may also include subsequent operations such as cropping and magnification based on the zoom ratio in response to a zoom operation.

[0238] S503: Acquire an image captured by the telephoto camera.

[0239] Exemplarily, the image captured by the telephoto camera may be a Raw image captured by the telephoto camera.

[0240] Optionally, S504 and S501 may be executed simultaneously, or S504 and S501 may be executed sequentially.

[0241] S504: Second front-end processing.

[0242] Optionally, the second front-end processing may include a related algorithm for converting the Raw image captured by the telephoto camera into a YUV image; this application does not impose any limitation on the algorithm.

[0243] For example, the second front-end processing may refer to an image processing algorithm executed in the ISP to convert a raw image into a YUV image. The second front-end processing may also include subsequent operations such as cropping and magnification based on the zoom ratio in response to a zoom operation.

[0244] After executing S505 , the processed image is subjected to a first back-end processing. For the implementation of the first back-end processing, reference may be made to the relevant description of S503 .

[0245] For example, Figure 16 As shown, when the zoom ratio is 1× to 1.8×, only S501 is executed. When the zoom ratio is 1.8× to 2.5×, S501 and S503 are executed simultaneously. When the zoom ratio is 2.5× or above, only S503 is executed.

[0246] S505: First backend processing.

[0247] Exemplarily, the first back-end processing includes but is not limited to: brightening processing, denoising processing, saturation adjustment processing, cropping processing or deformation processing.

[0248] S506: Smoothing process (or alignment process).

[0249] For example, the image stream captured by the main camera and the image stream captured by the telephoto camera are smoothed; it can be understood that the input smoothing data is two image streams (for example, the image stream captured by the main camera and the image stream captured by the telephoto camera), and the image stream after smoothing is one image stream. This smoothing process is provided in the embodiment of the present application. Figure 8 、 Figure 9 and / or Figure 14 The image processing method shown.

[0250] Exemplarily, when the zoom ratio increases from 1.8× to 2.5×, the image processing method provided in the embodiment of the present application is used to process the image streams of the main camera and the telephoto camera.

[0251] In the embodiment of the present application, smoothing processing can avoid obvious jumps between the foreground and background of the image when switching cameras.

[0252] S507: Second backend processing.

[0253] Exemplarily, the second back-end processing includes anti-shake processing; for example, anti-shake processing may be performed on image frames in the image stream according to a jitter parameter to obtain a processed image.

[0254] S508: Display the processed image.

[0255] For example, Figure 16 As shown, when the zoom ratio increases and the zoom ratio is 1×~1.8×, the processed image is the processed image corresponding to the image captured by the main camera; when the zoom is to 1.8×~2.5×, the processed image is: the image captured by the main camera, the image after correction and other processing based on the image captured by the telephoto camera; when the zoom is to 2.5× and above, the processed image is the processed image corresponding to the image captured by the telephoto camera.

[0256] S509: A zoom operation is detected.

[0257] The zoom operation may refer to the description in the above S230.

[0258] Optionally, the electronic device may display an image captured by the main camera, or an image captured by the telephoto camera, and the electronic device detects a user operation on the displayed image.

[0259] It should be understood that the above examples are intended to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to the specific numerical values ​​or specific scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or variations based on the above examples, and such modifications or variations also fall within the scope of the embodiments of the present application.

[0260] For example, Figure 19 A schematic diagram of an application scenario provided for an embodiment of the present application.

[0261] like Figure 19 As shown in (a) of FIG, in response to a user's operation on a camera application, the electronic device may display a preview interface 1301, which includes a preview window and a shooting control, wherein a preview image 1302 displayed in the preview window may include a first shooting object, a second shooting object, and a third shooting object. Here, when the camera application is opened, the zoom ratio defaults to 1x. Figure 19As shown in (b), assuming that the user wants to zoom in on the third photographed object located at the lower left of the preview image 1302, the user can perform a two-finger reverse sliding operation on the screen; in response to the operation, for example, Figure 19 As shown in (c) in FIG. 13 , the zoom ratio can be increased to 10x, the imaging size corresponding to the third object changes from small to large, and the third object can be located in the middle of the preview image 1303 in response to the movement of the user's finger.

[0262] When the zoom ratio increases and the camera switches, it can adapt to Figure 8 、 Figure 9 or Figure 14 The image processing method introduced.

[0263] It should be understood that the above is a preview scene in the photo mode, and the embodiment of the present application can also be applied to preview scenes in other shooting modes such as video. The preview scene can refer to the scene in which the electronic device captures an image before clicking the shooting control in a certain shooting mode.

[0264] For example, Figure 20 A schematic diagram of another application scenario provided for an embodiment of the present application.

[0265] After the camera application is opened, in response to the user's click operation on the recording control, the electronic device can start recording the video and display the video recording interface 1401, such as Figure 20 As shown in (a) of FIG. 1 , the video recording interface 1401 may include a recording window, a pause control, an end control, and a zoom control. The video image 1402 displayed in the recording window may include a first subject, a second subject, a third subject, and a fourth subject. When video recording begins, the zoom ratio defaults to 1x.

[0266] If the electronic device supports and turns on the AI ​​detection and parallax detection functions, the electronic device can detect the image content and depth when capturing the video image 1402, and display multiple detection frames when displaying the video image 1402, each detection frame is used to indicate the position information of a photographed object in the video image 1402. Figure 20 As shown in (a) in FIG. 1 , four detection frames are displayed in the video image 1402 , and the four detection frames respectively indicate the position information of the first to fourth shooting objects.

[0267] Optionally, if the AI ​​detection and disparity detection functions are only used for depth detection of faces, the electronic device may display only two detection frames when displaying the video image 1402, and the two detection frames respectively indicate the face of the second subject and the face of the fourth subject.

[0268] like Figure 20As shown in (a) in FIG, during the recording process, assuming that the user wants to zoom in on the face of the fourth shooting subject located on the left side of the video image 1402, the user can click on the detection box corresponding to the fourth shooting subject on the screen. Figure 20 As shown in (b) of FIG, in response to the user's click operation, the detection frame of the fourth shooting object included in the video image 1403 can be selected. For example, the selected detection frame can be in a different color and / or style than other detection frames. Then, the electronic device can receive the user's sliding operation on the zoom control, such as Figure 20 As shown in (c) in the figure, in response to a sliding operation on the zoom control (assuming sliding to 8x), the electronic device can display a video image 1404, in which the face of the fourth subject included in the video image 1404 becomes larger relative to the face of the fourth subject included in the video image 1402, and the face of the fourth subject can be located in the middle position of the video image 1404 in response to the movement of the user's finger.

[0269] It should be noted that if the electronic device detects the image content and only detects one subject or one face, the user does not need to click on the detection box corresponding to the subject or face, and the electronic device automatically selects it; then, in direct response to the user's sliding operation on the zoom control, the subject or face is enlarged and displayed in the center.

[0270] When the zoom ratio changes and the camera switches, it can adapt Figure 8 、 Figure 9 or Figure 14 The image processing method introduced.

[0271] It should be understood that the above is a video recording scenario, and the embodiments of the present application can also be applied to scenarios such as video calls.

[0272] Exemplarily, video call scenarios may include but are not limited to the following scenarios: video calls, video conferencing applications, long and short video applications, live video applications, online video courses, portrait smart mirror application scenarios, system camera recording function video recording, video surveillance, or smart cat-eye and other shooting scenarios.

[0273] Combined with the above Figures 1 to 20 The image processing method provided in the embodiments of the present application is described in detail.

[0274] The present application also provides a computer program product, which, when executed by a processor, implements the method described in any method embodiment of the present application.

[0275] The computer program product may be stored in a memory, for example, a program, which is converted into an executable target file that can be executed by a processor after undergoing processes such as preprocessing, compilation, assembly, and linking.

[0276] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer, implements the method described in any method embodiment of the present application. The computer program can be a high-level language program or an executable target program.

[0277] The computer-readable storage medium is, for example, a memory. The memory may be a volatile memory or a non-volatile memory, or the memory may include both volatile memory and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0278] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and equipment and the technical effects produced can refer to the corresponding processes and technical effects in the aforementioned method embodiments, and will not be repeated here.

[0279] In the several embodiments provided in this application, the disclosed systems, devices and methods can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not executed. The device embodiments described above are merely schematic, and the division of units is only a logical function division. There may be other division methods in actual implementation, and multiple units or components may be combined or integrated into another system. In addition, the coupling between the units or the coupling between the components may be direct coupling or indirect coupling, and the above coupling includes electrical, mechanical or other forms of connection.

[0280] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0281] Additionally, the terms "system" and "network" are often used interchangeably. The term "and / or" in this document simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates an "or" relationship between the related objects.

[0282] In short, the above description is only a preferred embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application shall be included in the scope of protection of this application.

Claims

1. An image processing method, characterized in that: Applied to an electronic device, the electronic device includes a first camera and a second camera, and the method includes: Open the Camera app; Display a first image, where the first image is acquired by the first camera; receiving a first zoom operation; Display and save the second image, where the second image is obtained by segmenting the image captured by the first camera into modules using the parallax between the image captured by the second camera and the module, and correcting each module using its corresponding warp matrix and then stitching the resultant image.

2. The image processing method according to claim 1, wherein: The method further comprises: receiving a second zoom operation; Display and save a third image, where the third image is obtained by capturing an image with the second camera.

3. The image processing method according to claim 1 or 2, characterized in that: The image captured by the first camera is segmented into modules using the parallax between the image captured by the second camera and the first camera, including: Acquire a first initial image captured by the first camera and a second initial image captured by the second camera; Performing feature point detection and registration on the first initial image and the second initial image to obtain a plurality of matched sets of inter-frame feature point pairs; Calculating the disparity of each pair of feature points between the frames to obtain a disparity map; The disparity distribution information of the disparity map is counted and divided to determine multiple classes; Based on the disparities belonging to the same category, module segmentation is performed on both the first initial image and the second initial image.

4. The image processing method according to claim 3, wherein: The method further comprises: Calculating a first warp matrix between the modules corresponding to the first initial image and the second initial image based on the inter-frame feature point pairs and in combination with the segmented modules; Correction is performed on the modules obtained by segmenting the first initial image based on the first warp matrix corresponding to each of the modules.

5. The image processing method according to claim 3 or 4, characterized in that: After performing module segmentation, the method further includes: A tolerance range is added to the periphery of the module, and the size and position of the tolerance range are preset.

6. The image processing method according to claim 5, characterized in that The method further comprises: Performing feature point matching on two adjacent modules in the first initial image to obtain a plurality of matched intra-frame feature point pairs; Calculating a second warp matrix between the two adjacent modules based on the feature point pairs within the frame; Correction is performed on the modules obtained by segmenting the first initial image based on the second warp matrix corresponding to each of the modules.

7. The image processing method according to claim 6, characterized in that: When calculating the second warp matrix between the two adjacent modules, the method includes: The feature point pairs within the tolerance range and the feature point pairs outside the tolerance range are screened according to a preset ratio, wherein the ratio of the feature point pairs within the tolerance range is greater than the ratio of the feature point pairs outside the tolerance range.

8. The image processing method according to claim 3 or 6, characterized in that: The method further includes: eliminating feature point pairs with matching errors.

9. The image processing method according to claim 6 or 7, characterized in that: The method further comprises: When splicing, the increased tolerance range will be overlapped; After splicing, align the overall size and modify the edges.

10. The image processing method according to any one of claims 1 to 9, characterized in that: The method further includes: in response to the first zoom operation; Determine that the zoom ratio of the electronic device meets the zoom ratio range of the first camera being displayed in the foreground and the second camera being run in the background.

11. The image processing method according to claim 2, wherein: The method further includes: in response to the second zoom operation; Determine that the zoom ratio of the electronic device meets the zoom ratio range that the first camera runs in the background or is closed, and the second camera is displayed in the foreground.

12. The image processing method according to claim 2 or 11, characterized in that: The first zoom operation or the second zoom operation includes: a two-finger reverse sliding operation, a sliding operation on a displayed zoom control, a voice operation, or an air gesture operation.

13. The image processing method according to any one of claims 1 to 12, characterized in that: The first camera includes a wide-angle camera, and / or the second camera includes a telephoto camera or an ultra-wide-angle camera.

14. The image processing method according to claim 2, wherein: The first zoom operation and the second zoom operation are continuous zoom operations.

15. An electronic device, characterized in that: The electronic device includes: one or more processors, a memory, a first camera and a second camera; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the image processing method as described in any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to execute the image processing method according to any one of claims 1 to 14.

Images