Image processing method and related equipment thereof

Abstract

The invention relates to the field of image processing, and provides an image processing method and related equipment thereof, and the method comprises the steps: starting a camera application program; displaying a first image and a focusing frame, wherein the first image is obtained by an image collected by a first camera; receiving a first zooming operation; a second image is displayed, the second image is obtained by correcting the image collected by the first camera through a spatial alignment transformation matrix between the image collected by the first camera and the image collected by the second camera, and the spatial alignment transformation matrix is feature points of the image collected by the first camera in the second target area; the second target area is obtained by matching feature points of an image collected by the first camera with feature points formed by feature points of the image collected by the second camera, and the second target area is determined by the image collected by the first camera and the focusing frame. According to the invention, the foreground is determined by combining the collected image and the focusing frame, and the spatial alignment transformation matrix is determined through the feature points in the foreground for correction, so that smooth zooming and smooth switching of the camera can be realized.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method and related equipment. Background Art

[0002] With the development of shooting functions in electronic devices, camera applications are increasingly used in electronic devices. To obtain a better photo taking experience, current electronic devices are usually equipped with multiple cameras, each corresponding to a different focal length.

[0003] When shooting, electronic devices can respond to user input by switching between cameras with different focal lengths to perform zoom shots. They can also incorporate digital zoom into the captured image to accommodate a variety of high-magnification shooting scenarios. However, during zooming, due to varying depths of the subject, such as the foreground and background, switching between cameras can cause jumps in image content and size, preventing smooth camera switching. Furthermore, digital zoom can only magnify and crop the image content in the center of the field of view, which is very limited.

[0004] Therefore, during the shooting process, how to smoothly switch cameras becomes a problem that needs to be solved urgently. Summary of the Invention

[0005] The present application provides an image processing method and related equipment, which determines the foreground by combining the captured image and the focus frame, and corrects the foreground by determining the spatial alignment transformation matrix, thereby achieving smooth zoom and smooth switching of the camera.

[0006] In a first aspect, an image processing method is provided, which is applied to an electronic device, wherein the electronic device includes a first camera and a second camera, and the method includes: starting a camera application; displaying a first image and a focus frame, wherein the first image is obtained by obtaining an image captured by the first camera; receiving a first zoom operation; and displaying a second image, wherein the second image is obtained by correcting the image captured by the first camera using a spatial alignment transformation matrix between the image captured by the second camera and the image captured by the second camera, wherein the spatial alignment transformation matrix is ​​obtained by matching feature points in a second target area of ​​the image captured by the first camera with feature points in the image captured by the second camera to form a feature point pair, and the second target area is determined by the image captured by the first camera and the focus frame.

[0007] In an embodiment of the present application, the second target area is determined by combining the image captured by the first camera and the focus frame, which is equivalent to determining a unified and unique foreground for the first camera and the second camera. Then, the feature points of the image captured by the first camera within the second target area are matched with the feature points of the image captured by the second camera to form a feature point pair to determine the spatial alignment transformation matrix, and the image captured by the first camera is corrected according to the spatial alignment transformation matrix to obtain the second image. Because the spatial alignment transformation matrix is ​​calculated based on the feature points in the stable and unchanging second target area during the zoom process, the corrected image can be smoothly magnified based on the second target area, thereby smoothly magnifying the FOV of the image stream and smoothly switching cameras during the zoom process.

[0008] In combination with the first aspect, in some implementations of the first aspect, the method further includes: receiving a second zoom operation; and displaying and saving a third image, where the third image is obtained by capturing an image with the second camera.

[0009] In the embodiment of the present application, when the second zoom operation is performed after the first zoom operation, smooth switching of the camera can be achieved.

[0010] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: obtaining a first initial image captured by the first camera; performing segmentation processing on the first initial image and identifying the segmented first target area; and determining the second target area in combination with the first target area and the focus frame.

[0011] In an embodiment of the present application, the objects in the image captured by the first camera can be determined through segmentation and recognition, and the area where each object is located is a first target area. Based on the first target area and the focus frame, a second target area as the foreground can be determined.

[0012] In combination with the first aspect, in certain implementations of the first aspect, the second target area is determined in combination with the first target area and the focusing frame, including: obtaining the focusing frame; obtaining the mask area corresponding to the first target area; respectively calculating the proportion of each mask area in the focusing frame and sorting them; determining the mask area with the largest proportion as the target mask area; and determining, based on the coordinates of the target mask area, a rectangle including the target mask area as the second target area.

[0013] In the embodiments of the present application, when determining the second target area based on the first target area and the focus frame, the proportion of the mask area corresponding to each first target area within the focus frame is determined, and the mask area with the largest proportion is selected. This is equivalent to combining the photographic object identified from the image and the focus frame to determine the second target area as the foreground. When there are multiple photographic objects in the image, a unified and unique foreground can be clearly identified, thereby ensuring smooth subsequent zooming and avoiding jumps caused by foreground changes.

[0014] In combination with the first aspect, in some implementations of the first aspect, when displaying the focus frame, the method further includes: displaying the mask area corresponding to the first target area, and different mask areas have different colors.

[0015] In the embodiment of the present application, different mask areas have different colors, which can distinguish the same type of shooting objects.

[0016] In combination with the first aspect, in certain implementations of the first aspect, displaying the first image and the focus frame includes: displaying the first image; determining and displaying the focus frame in response to a click operation on the first image; or displaying the first image; determining and displaying the focus frame in response to autofocus processing.

[0017] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: obtaining a second initial image captured by the second camera; extracting a first feature point of the first initial image within the second target area; determining a second feature point that matches the first feature point in the second initial image; calculating a maximum difference parameter corresponding to the target zoom ratio when switching from the first camera to the second camera based on a feature point pair consisting of the matched first feature point and the second feature point; and correcting the first initial image based on the maximum difference parameter to obtain the second image.

[0018] In combination with the first aspect, in certain implementations of the first aspect, before calculating the maximum difference parameter corresponding to the target zoom ratio when switching from the first camera to the second camera, the method further includes: eliminating feature point pairs with incorrect matching.

[0019] In the embodiments of the present application, after testing, the number of correctly registered feature points in each scene is greater than the number of incorrectly registered feature points. Therefore, in order to improve processing efficiency and improve the effect of subsequent processing, the incorrectly registered feature points can be eliminated and only the correctly registered feature point pairs are retained.

[0020] In combination with the first aspect, in certain implementations of the first aspect, the first initial image is corrected based on the maximum difference parameter to obtain the second image, including: assigning different difference parameters to different zoom magnifications in the first zoom magnification range based on the maximum difference parameter; determining the spatial alignment transformation matrix corresponding to the different zoom magnifications in the first zoom magnification range according to the difference parameter; determining the spatial alignment transformation matrix corresponding to the zoom magnification of the electronic device in response to the first zoom operation; correcting the first initial image based on the spatial alignment transformation matrix corresponding to the zoom magnification of the electronic device to obtain the second image; wherein, the first zoom magnification range is the zoom magnification range corresponding to when the first camera is displayed in the foreground and the second camera is running in the background, and the maximum value of the first zoom magnification range is the target zoom magnification; the zoom magnification of the electronic device satisfies the first zoom magnification range.

[0021] Optionally, the difference parameter is assigned in the form of a linear or exponential curve.

[0022] In combination with the first aspect, in some implementations of the first aspect, the difference parameters include at least a rotation angle and an offset, and the maximum difference parameters include at least a maximum rotation angle and a maximum offset.

[0023] It should be noted that, when the difference parameters include a rotation angle and an offset, the spatial alignment transformation matrix includes a rotation transformation matrix and an offset transformation matrix, or may be a combination matrix of the two.

[0024] In addition, the difference parameters may also include a scaling amount; when the difference parameters include a rotation angle, an offset, and a scaling amount, the spatial alignment transformation matrix may include a rotation transformation matrix, an offset transformation matrix, and a scaling transformation matrix, or may be a combination matrix of the three.

[0025] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: responding to the first zoom operation; determining that the zoom ratio of the electronic device satisfies the zoom ratio range of the first camera foreground display and the second camera for background operation.

[0026] It should be understood that foreground display refers to the image captured by the camera being used for display; background operation refers to the image captured by the camera being used for image processing and not for display. The zoom ratio range is the first zoom ratio range mentioned above.

[0027] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: responding to the second zoom operation; determining that the zoom ratio of the electronic device satisfies the zoom ratio range when the first camera is running in the background or turned off, and the second camera is displayed in the foreground.

[0028] In combination with the first aspect, in some implementations of the first aspect, the first zoom operation or the second zoom operation includes: a two-finger reverse sliding operation, a sliding operation on a displayed zoom control, a voice operation, or an air gesture operation.

[0029] In combination with the first aspect, in some implementations of the first aspect, the first camera includes a wide-angle camera, and / or the second camera includes a telephoto camera or an ultra-wide-angle camera.

[0030] In combination with the first aspect, in some implementations of the first aspect, the first zoom operation and the second zoom operation are continuous first zoom operations.

[0031] In combination with the first aspect, in some implementations of the first aspect, the segmentation process is instance segmentation.

[0032] According to a second aspect, an electronic device is provided, comprising: one or more processors, a memory, a first camera, and a second camera; the memory is coupled to the one or more processors, the memory being configured to store computer program code, the computer program code comprising computer instructions, and the one or more processors invoking the computer instructions to cause the electronic device to execute:

[0033] Launch a camera application; display a first image and a focus frame, where the first image is obtained by capturing an image by the first camera; receive a first zoom operation; and display a second image, where the second image is obtained by correcting the image captured by the first camera with an image captured by the second camera using a spatial alignment transformation matrix, where the spatial alignment transformation matrix is ​​obtained by matching feature points within a second target area of ​​the image captured by the first camera with feature points of the image captured by the second camera to form feature point pairs, where the second target area is determined by the image captured by the first camera and the focus frame.

[0034] In combination with the second aspect, in some implementations of the second aspect, the following is further performed: receiving a second zoom operation; and displaying and saving a third image, where the third image is obtained by capturing an image with the second camera.

[0035] In combination with the second aspect, in certain implementations of the second aspect, determining the second target area in combination with the first target area and the focusing frame includes: obtaining a first initial image captured by the first camera; performing segmentation processing on the first initial image and identifying the segmented first target area; and determining the second target area in combination with the first target area and the focusing frame.

[0036] In combination with the second aspect, in certain implementations of the second aspect, the second target area is determined in combination with the first target area and the focusing frame, including: obtaining the focusing frame; obtaining the mask area corresponding to the first target area; respectively calculating the proportion of each mask area in the focusing frame and sorting them; determining the mask area with the largest proportion as the target mask area; and determining, based on the coordinates of the target mask area, a rectangle including the target mask area as the second target area.

[0037] In combination with the second aspect, in some implementations of the second aspect, when the focus frame is displayed, the following is further performed: displaying the mask area corresponding to the first target area, where different mask areas have different colors.

[0038] In combination with the second aspect, in certain implementations of the second aspect, displaying the first image and the focus frame includes: displaying the first image; determining and displaying the focus frame in response to a click operation on the first image; or, displaying the first image; determining and displaying the focus frame in response to autofocus processing.

[0039] In combination with the second aspect, in certain implementations of the second aspect, the following is also performed: obtaining a second initial image captured by the second camera; extracting a first feature point of the first initial image within the second target area; determining a second feature point that matches the first feature point in the second initial image; calculating a maximum difference parameter corresponding to the target zoom ratio when switching from the first camera to the second camera based on a feature point pair consisting of the matched first feature point and the second feature point; and correcting the first initial image based on the maximum difference parameter to obtain the second image.

[0040] In combination with the second aspect, in certain implementations of the second aspect, before calculating the maximum difference parameter corresponding to the target zoom ratio when switching from the first camera to the second camera, the following is further performed: eliminating feature point pairs with incorrect matching.

[0041] In combination with the second aspect, in some implementations of the second aspect, the first initial image is corrected based on the maximum difference parameter to obtain the second image, including: assigning different difference parameters to different zoom magnifications in the first zoom magnification range based on the maximum difference parameter; determining the spatial alignment transformation matrix corresponding to the different zoom magnifications in the first zoom magnification range according to the difference parameter; determining the spatial alignment transformation matrix corresponding to the zoom magnification of the electronic device in response to the first zoom operation; correcting the first initial image based on the spatial alignment transformation matrix corresponding to the zoom magnification of the electronic device to obtain the second image; wherein, the first zoom magnification range is the zoom magnification range corresponding to when the first camera is displayed in the foreground and the second camera is running in the background, and the maximum value of the first zoom magnification range is the target zoom magnification; the zoom magnification of the electronic device satisfies the first zoom magnification range.

[0042] In combination with the second aspect, in certain implementations of the second aspect, the difference parameters include at least a rotation angle and an offset, and the maximum difference parameters include at least a maximum rotation angle and a maximum offset.

[0043] In combination with the second aspect, in certain implementations of the second aspect, the following is further performed: in response to the first zoom operation; determining that the zoom ratio of the electronic device meets the zoom ratio range of the first camera foreground display and the second camera for background operation.

[0044] In combination with the second aspect, in certain implementations of the second aspect, the following is further performed: in response to the second zoom operation; determining that the zoom ratio of the electronic device satisfies the zoom ratio range when the first camera is running in the background or turned off, and the second camera is displayed in the foreground.

[0045] In combination with the second aspect, in some implementations of the second aspect, the first zoom operation or the second zoom operation includes: a two-finger reverse sliding operation, a sliding operation on a displayed zoom control, a voice operation, or an air gesture operation.

[0046] In combination with the second aspect, in some implementations of the second aspect, the first camera includes a wide-angle camera, and / or the second camera includes a telephoto camera or an ultra-wide-angle camera.

[0047] In combination with the second aspect, in some implementations of the second aspect, the first zoom operation and the second zoom operation are continuous first zoom operations.

[0048] In combination with the second aspect, in some implementations of the second aspect, the segmentation process is instance segmentation.

[0049] In a third aspect, a chip system is provided, which is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute any one of the image processing methods in the first aspect.

[0050] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program code. When the computer program code is executed by an electronic device, the electronic device executes any one of the image processing methods in the first aspect.

[0051] In a fifth aspect, a computer program product is provided, comprising: a computer program code, which, when executed by an electronic device, enables the electronic device to execute any one of the image processing methods in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a schematic diagram of a hardware system of an electronic device applicable to the present application;

[0053] Figure 2 is a schematic diagram of a software system for an electronic device applicable to the present application;

[0054] Figure 3 This is a schematic diagram of an arrangement of multiple cameras on an electronic device provided by an embodiment of the present application;

[0055] Figure 4 This is a schematic diagram of zoom ratios corresponding to different types of cameras provided in an embodiment of the present application;

[0056] Figure 5 This is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0057] Figure 6 It is a schematic diagram of a set of image flows involved in the prior art;

[0058] Figure 7 This is a schematic flow chart of an image processing method provided in an embodiment of the present application;

[0059] Figure 8 is a schematic flow chart of another image processing method provided in an embodiment of the present application;

[0060] Figure 9 This is a set of display interface schematic diagrams provided by the embodiments of the present application;

[0061] Figure 10 is another set of display interface schematic diagrams provided in an embodiment of the present application;

[0062] Figure 11is a schematic diagram of the process of obtaining the second target area involved in the embodiment of the present application;

[0063] Figure 12 This is another set of display interface schematic diagrams provided in the embodiments of the present application;

[0064] Figure 13 is a schematic flow chart of another image processing method provided in an embodiment of the present application;

[0065] Figure 14 is a schematic flow chart of another image processing method provided in an embodiment of the present application;

[0066] Figure 15 1 is a diagram showing the correspondence between multiple cameras and zoom ratios provided in an embodiment of the present application;

[0067] Figure 16 is a schematic diagram of a set of image flows involved in the embodiments of the present application;

[0068] Figure 17 is a schematic flow chart of another image processing method provided in an embodiment of the present application;

[0069] Figure 18 This is a schematic diagram of an application scenario of zoom switching provided by an embodiment of the present application;

[0070] Figure 19 This is a schematic diagram of an application scenario of zoom switching provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] In the embodiments of this application, the terms "first," "second," and the like are used for descriptive purposes only and should not be understood to indicate or imply relative importance or to implicitly indicate the number of the technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0072] First, some of the terms used in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0073] 1. Field of view (FOV) in optical instruments is the angle formed by the two edges of the maximum range through which the image of the target object can pass through the lens, with the lens as the vertex. The size of the field of view angle determines the field of view of the optical instrument. The larger the field of view angle, the greater the field of view and the smaller the optical magnification. In other words, the target object will not be captured by the lens if it exceeds this angle. The shorter the focal length, the wider the horizontal field of view, and therefore the smaller the image. The horizontal field of view narrows as the focal length increases, and the object being photographed also increases in size.

[0074] 2. Autofocus (AF) refers to an electronic device adjusting the focus lens position to obtain the highest image frequency component, thereby achieving higher image contrast. Focusing is a continuous, cumulative process. The electronic device compares the contrast of images captured with the lens in different positions to determine the lens position that maximizes image contrast, thereby determining the focal length for focus.

[0075] 3. Registration refers to the matching of geographic coordinates of different images obtained by different imaging methods within the same area. This includes three aspects: geometric correction, projection transformation, and uniform scale.

[0076] 4. Zoom ratio: The zoom ratio is used to indicate the zoom size of an electronic device when shooting.

[0077] The above is a brief introduction to the nouns involved in the embodiments of this application, and no further details will be given below.

[0078] The hardware system, software system and application scenarios of the electronic device provided in the embodiments of the present application are described below with reference to the accompanying drawings.

[0079] Exemplarily, the electronic device 100 may be a mobile phone, a smart screen, a tablet computer, a wearable electronic device, an in-vehicle electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a projector, etc. The embodiments of the present application do not impose any restrictions on the specific type of the electronic device 100.

[0080] See also Figure 1The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0081] It should be noted that Figure 1 The structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include Figure 1 More or fewer components than those shown, or the electronic device 100 may include Figure 1 Combinations of some of the components shown, or alternatively, the electronic device 100 may include Figure 1 Subassemblies of some of the components shown. Figure 1 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0082] The processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). The different processing units may be independent devices or integrated devices.

[0083] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0084] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0085] Exemplarily, the processor 110 can be used to execute the image processing method of the embodiment of the present application; for example, start a camera application; display a first image and a focus frame, where the first image is obtained by capturing an image by the first camera; receive a first zoom operation; display a second image, where the second image is obtained by correcting the image captured by the first camera using a spatial alignment transformation matrix between the image captured by the second camera, where the spatial alignment transformation matrix is ​​obtained by matching feature points in the second target area of ​​the image captured by the first camera with feature points in the image captured by the second camera to form a pair of feature points, where the second target area is determined by the image captured by the first camera and the focus frame.

[0086] Figure 1 The connection relationship between the modules shown is only for illustrative purposes and does not limit the connection relationship between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of the multiple connection modes in the above embodiments.

[0087] The wireless communication function of the electronic device 100 can be implemented through components such as the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor.

[0088] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0089] Electronic device 100 can implement display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0090] Display screen 194 may be used to display images or videos.

[0091] For example, in an embodiment of the present application, the display screen 194 may be used to display the second image.

[0092] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0093] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can perform algorithmic optimization on image noise, brightness, and color. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0094] The camera 193 (also called the lens) is used to capture still images or videos. It can be triggered to turn on through application instructions to realize the photo function, such as capturing images of any scene. The camera may include components such as an imaging lens, a filter, and an image sensor. The light emitted or reflected by the object enters the imaging lens, passes through the filter, and finally converges on the image sensor. The imaging lens is mainly used to focus the light emitted or reflected by all objects in the photographic field of view (also called the scene to be photographed, the target scene, or the scene image that the user expects to capture) to form an image; the filter is mainly used to filter out excess light waves in the light (for example, light waves other than visible light, such as infrared); the image sensor can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The image sensor is mainly used to perform photoelectric conversion on the received light signal, convert it into an electrical signal, and then transmit the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB, YUV, etc.

[0095] Exemplarily, the gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., the x-axis, the y-axis, and the z-axis) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse motion to achieve anti-shake. The gyroscope sensor 180B can also be used in scenarios such as navigation and somatosensory games.

[0096] In some embodiments, the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1.

[0097] Among them, the camera 193 can be located in the front of the electronic device 100 or in the back of the electronic device 100. The specific number and arrangement of the cameras can be set according to needs, and this application does not impose any restrictions.

[0098] Exemplarily, the electronic device 100 includes a front camera and a rear camera. For example, the front camera or the rear camera may include one or more cameras. Taking the example of the electronic device 100 having four rear cameras, when the electronic device 100 activates the four rear cameras for shooting, the image processing method provided in the embodiments of the present application can be used.

[0099] Alternatively, the camera is provided on an external accessory of the electronic device 100, and the external accessory is rotatably connected to the frame of the mobile phone, and the angle formed between the external accessory and the display screen 194 of the electronic device 100 is any angle between 0 and 360 degrees. For example, when the electronic device 100 takes a selfie, the external accessory drives the camera to rotate to a position facing the user. Of course, when the mobile phone has multiple cameras, only some of the cameras can be provided on the external accessory, and the remaining cameras can be provided on the electronic device 100 body. This embodiment of the application does not impose any restrictions on this.

[0100] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0101] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0102] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., the x-axis, the y-axis, and the z-axis) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used in scenarios such as navigation and somatosensory games.

[0103] Accelerometer 180E can detect the magnitude of the acceleration of electronic device 100 in various directions (generally the x-axis, y-axis, and z-axis). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. Accelerometer 180E can also be used to identify the posture of electronic device 100, which can serve as an input parameter for applications such as landscape / portrait switching and pedometers.

[0104] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. In some embodiments, for example, in a photography scenario, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0105] Ambient light sensor 180L is used to sense ambient light brightness. Electronic device 100 can adaptively adjust the brightness of display screen 194 based on the perceived ambient light. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether electronic device 100 is in a pocket to prevent accidental touches.

[0106] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement functions such as unlocking, accessing application locks, taking photos, and answering calls.

[0107] The touch sensor 180K is also referred to as a touch-sensitive device. The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also referred to as a touch screen. The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor 180K can transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100 and at a different location from the display screen 194.

[0108] The hardware system of the electronic device 100 is described in detail above. The software system of the electronic device 100 is introduced below.

[0109] Figure 2 Schematic diagram of the software system of the electronic device provided in an embodiment of the present application.

[0110] like Figure 2 As shown, the system architecture may include an application layer 210 , an application framework layer 220 , a hardware abstraction layer 230 , a driver layer 240 and a hardware layer 250 .

[0111] The application layer 210 may include applications such as camera application, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0112] The application framework layer 220 provides an application programming interface (API) and a programming framework for the application programs of the application layer; the application framework layer may include some predefined functions.

[0113] For example, the application framework layer 220 may include a camera access interface, which may include camera management and camera devices. The camera management may be used to provide an access interface for managing the camera, and the camera device may be used to provide an interface for accessing the camera.

[0114] The hardware abstraction layer 230 is used to abstract the hardware. For example, the hardware abstraction layer may include a camera abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer may call camera algorithms.

[0115] For example, the hardware abstraction layer 230 includes a camera hardware abstraction layer and a camera algorithm; the camera algorithm may include a software algorithm for image processing.

[0116] Exemplarily, the camera algorithm library may include algorithms corresponding to the image processing method provided in the embodiments of the present application.

[0117] For example, the algorithm in the camera algorithm may refer to a code that does not rely on specific hardware implementation; for example, a code that can generally be run in a CPU, etc.

[0118] The driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include a camera driver.

[0119] The hardware layer 250 is located at the bottom layer of the operating system; Figure 2 As shown, the hardware layer 250 may include camera 1, camera 2, camera 3, etc. Among them, camera 1, camera 2, camera 3 may correspond to multiple cameras on the electronic device.

[0120] For ease of understanding, the following takes the electronic device 100 as an example of a mobile phone having the above-mentioned software and hardware structure, and first describes in detail the camera on the electronic device 100 to which the method provided in the embodiment of the present application is applicable.

[0121] The electronic device to which the method provided in the embodiment of the present application is applicable has at least multiple cameras 193, for example, three types of cameras 193; the three types of cameras are a main camera (for example, a wide-angle camera), an ultra-wide-angle camera and a telephoto camera; the three cameras can be used to shoot the same scene to be shot.

[0122] Optionally, the electronic device 100 may also have other cameras 193, such as an ultra-telephoto camera; the type of camera 193 and the number of each camera 193 can be set as needed, and the embodiments of the present application do not impose any restrictions on this.

[0123] For example, Figure 3 As shown in the figure, the electronic device 100 may have three cameras 193 as an example for explanation; the arrangement of the three cameras can be as follows Figure 3 As shown in (a) in , or as Figure 3 As shown in (b) in FIG; for example, the three cameras 193 can be a main camera 1931 (for example, a wide-angle camera), an ultra-wide-angle camera 1932 and a telephoto camera 1933.

[0124] It should be understood that the above are only examples of two arrangements, and other arrangements are also possible; the specific arrangement can be designed and changed as needed, and the embodiments of the present application do not impose any restrictions on this.

[0125] It should be noted that when the above three cameras are shooting, the field of view angle range corresponding to the main camera 1931 is usually larger than the field of view angle range corresponding to the telephoto camera 1933; and the field of view angle range corresponding to the ultra-wide-angle camera 1932 is larger than the field of view angle range corresponding to the main camera 1931; there may be overlap between the field of view angle of the ultra-wide-angle camera 1932 and the field of view angle of the main camera 1931; that is, the ultra-wide-angle camera 1932 can capture the scene content captured by the main camera 1931 and the scene content around it.

[0126] It should be understood that the field of view of the telephoto camera 1933 is smaller than that of the main camera 1931, and the field of view of the main camera 1931 and the field of view of the telephoto camera 1933 may overlap; that is, the main camera 1931 can capture the scene content captured by the telephoto camera 1933 and the surrounding scene content. The field of view of the ultra-wide-angle camera 1932 and the field of view of the telephoto camera 1933 may overlap; that is, the ultra-wide-angle camera 1932 can capture the scene content captured by the telephoto camera 1933 and the surrounding scene content.

[0127] Among them, the ultra-wide-angle camera 1932 is suitable for shooting close-ups due to its smaller focusing distance; and, as the name suggests, the ultra-wide-angle camera 1932 is suitable for shooting scenes with a larger field of view; the main camera 1931 is more suitable for shooting portraits due to its higher clarity, while the telephoto camera 1933 is more suitable for shooting distant close-ups.

[0128] For example, Figure 4 As shown, the zoom ratio of the ultra-wide-angle camera can be less than M times the zoom ratio; the zoom ratio range of the wide-angle camera, i.e., the main camera, can be [M, N); the zoom ratio of the telephoto camera can be greater than or equal to N times the zoom ratio.

[0129] For example, M can be 1 and N can be 2.5; then the zoom magnification of the ultra-wide-angle camera is less than 1x zoom magnification (1×); the zoom magnification range of the wide-angle camera is 1x zoom magnification to 2.5x zoom magnification [1×~2.5×); the zoom magnification of the telephoto camera is greater than or equal to 2.5x zoom magnification.

[0130] It should be understood that the above is only an example, and the zoom ratio ranges applicable to different cameras may also overlap.

[0131] It should also be understood that, during the shooting process of an electronic device, the greater the zoom ratio, the smaller the corresponding field of view angle.

[0132] The following combination Figure 5 The application scenarios of the image processing method provided in the embodiments of the present application are illustrated with examples.

[0133] The method in the embodiments of the present application can be applied to photo-taking scenarios, preview scenarios, video recording scenarios, or video call scenarios, etc.; through the method in the embodiments of the present application, smooth zoom and / or smooth switching of different cameras can be achieved in electronic devices, thereby improving the user's shooting experience and image quality.

[0134] For example, the preview scenarios include but are not limited to the following scenarios:

[0135] Photo preview, aperture preview, night scene preview, portrait preview, video preview or professional preview, etc.

[0136] It should be understood that the preview scene may refer to a scene in which the electronic device captures an image before a button indicating shooting is clicked in a certain shooting mode.

[0137] In one example, if Figure 5 As shown in (a) in FIG, after the electronic device enters the camera application, the default photo mode can be turned on; in the photo mode, the electronic device can enter the default shooting mode, which can refer to a shooting mode in which the wide-angle camera is used as the main camera and the zoom ratio is a single zoom ratio (1×), and the electronic device displays the image captured by the main camera; Figure 5 As shown in (b), in response to the user's operation, when the zoom ratio satisfies the zoom range corresponding to the telephoto camera, the electronic device can switch to display the image captured by the telephoto camera.

[0138] For example, the image processing method in the embodiment of the present application may also be applied to a video call scenario, wherein the video call scenario may include but is not limited to the following scenarios:

[0139] Video calls, video conferencing applications, long and short video applications, live video applications, online video courses, portrait intelligent mirroring application scenarios, system camera recording function video recording, video surveillance, or smart cat-eye and other portrait shooting scenarios, etc.

[0140] It should be understood that the above is an example of an application scenario and does not limit the application scenario of this application.

[0141] Currently, when shooting, in response to user operations, electronic devices can switch cameras with different focal lengths to perform zoom shooting, and can also process the captured images in combination with digital zoom to meet various high-magnification shooting scenarios. However, during the zoom process, when two cameras are switched, due to the different physical positions and field of view ranges of different cameras, the switching process may not be smooth. In particular, when the scene to be shot includes multiple subjects, the depth of each subject is different. During the zoom process, there may be large differences in the subjects in the images of the two cameras, resulting in the inability to align the images, and thus the inability to switch cameras smoothly.

[0142] For example, Figure 6 is a set of image streams involved in the prior art. Figure 6 As shown in (a), (b), (c), (d), (e), and (f), assume that the captured image includes targets 1, 2, and 3, all of which have different depths. During zooming, the image formation of the two cameras differs significantly. If you want to align target 1, targets 2 and 3 may not be aligned. Similarly, if you want to align target 2, targets 1 and 3 may not be aligned. If you want to align target 3, targets 1 and 2 may not be aligned. Because only one depth can be guaranteed at a time, the algorithm must continuously determine which target to align with. This causes the image to jump repeatedly during zooming, preventing smooth camera switching.

[0143] In view of this, an embodiment of the present application provides an image processing method and related equipment. In the embodiment of the present application, the foreground is determined by combining an image captured by a first camera and a focus frame. Then, feature points within the foreground range of the image captured by the first camera are matched with feature points in the image captured by the second camera to form a feature point pair to determine a spatial alignment transformation matrix. The image captured by the first camera is then corrected according to the spatial alignment transformation matrix to obtain a second image. Because the spatial alignment transformation matrix is ​​calculated based on feature points in the stable and unchanging foreground during zooming, the corrected image can be smoothly magnified based on the foreground, thereby smoothly magnifying the FOV of the image stream and smoothly switching cameras during zooming.

[0144] The following combination Figure 7 The schematic flowchart of the image processing method provided in the embodiment of the present application is described in detail.

[0145] Figure 7 is a schematic flow chart of the image processing method provided in the embodiment of the present application. Figure 1The electronic device shown is executed; the method 300 includes steps S310 to S340, and steps S310 to S340 are described in detail below.

[0146] It should be understood that in the embodiments of the present application, the electronic device includes a first camera and a second camera; wherein the first camera and the second camera are cameras of different types.

[0147] S310. Start the camera application.

[0148] For example, the user may instruct the electronic device to start the camera application by clicking the icon of the “Camera” application.

[0149] For example, when the electronic device is in the lock screen state, the user can instruct the electronic device to launch the camera application by swiping right on the electronic device's display. Alternatively, when the electronic device is in the lock screen state and the lock screen interface includes a camera application icon, the user can instruct the electronic device to launch the camera application by clicking the camera application icon. Alternatively, when the electronic device is running another application, the application has permission to call the camera application; the user can instruct the electronic device to launch the camera application by clicking the corresponding control. For example, when the electronic device is running an instant messaging application, the user can instruct the electronic device to launch the camera application by selecting the camera function control.

[0150] It should be understood that the above is an example of the operation of starting the camera application; the camera application can also be started by the electronic device through voice instructions or other operations; this application does not impose any limitation on this.

[0151] It should also be understood that launching the camera application may refer to running the camera application.

[0152] S320: Display a first image and a focus frame, where the first image is obtained by capturing an image with a first camera.

[0153] For example, the first camera may be Figure 3 The wide-angle camera, main camera shown, or other cameras with a larger field of view than the telephoto camera.

[0154] Exemplarily, the first image may be an RGB image captured by the main camera, or in other words, the first image may be an RGB image obtained by processing a series of camera algorithms from a Raw image captured by the main camera.

[0155] For example, Figure 18 The preview image shown in (a) is the first image captured and displayed by the first camera. For example, Figure 19The video image shown in (a) is the first image captured and displayed by the first camera.

[0156] It should be understood that the first image may include one or more photographed objects.

[0157] It should also be understood that the type of the photographed object can be preset as needed. For example, when the type is set to human face, the photographed object can be used to indicate a human face recognized in the first image. The type of photographed object can also include plants, animals, etc. This embodiment of the application does not impose any restrictions on this.

[0158] Optionally, when the first image is displayed, the focus frame is determined and displayed in response to a click operation on the first image; or, when the first image is displayed, the focus frame is determined and displayed in response to an autofocus process.

[0159] S330: Receive a first zoom operation.

[0160] Optionally, receiving the first zoom operation may include: receiving the first zoom operation for the first image.

[0161] It should be understood that the first zoom operation on the first image may indicate an operation instruction of the user, or may also indicate a first zoom operation instruction automatically triggered by the electronic device based on an autofocus process.

[0162] Exemplarily, when the first zoom operation indicates an operation instruction triggered by the user, the first zoom operation may include a two-finger reverse sliding operation, a click operation, a voice operation, an air gesture operation, etc. on the first image. The embodiment of the present application does not impose any restrictions on this.

[0163] For example, Figure 18 As shown in (a) in FIG, when the electronic device displays one or more photographic objects and a zoom control, in response to the autofocus process, the electronic device also displays a focus frame; Figure 18 As shown in (b), thereafter, the first zoom operation may include a sliding operation on the zoom control; or, the first zoom operation may also include a two-finger reverse sliding operation on the first image.

[0164] For example, Figure 19 As shown in (a) of FIG, when the electronic device displays one or more photographic objects and a zoom control, in response to a click operation on the first image, the electronic device determines and displays a focus frame; Figure 19 As shown in (b), thereafter, the first zoom operation may include a sliding operation on the zoom control; or, the first zoom operation may also include a two-finger reverse sliding operation on the first image.

[0165] S340: Display the second image.

[0166] The second image is obtained by correcting the image captured by the first camera using the spatial alignment transformation matrix between the image captured by the second camera. The spatial alignment transformation matrix is ​​obtained by matching the feature points in the second target area of ​​the image captured by the first camera with the feature points of the image captured by the second camera to form feature point pairs. The second target area is determined by the image captured by the first camera and the focus frame.

[0167] Exemplarily, the first image may be an RGB image, and the corresponding second image may be an RGB image.

[0168] It should be understood that in response to the first zoom operation, the second target area is determined by combining the image captured by the first camera and the focus frame, which is equivalent to determining a unique foreground for the first and second cameras. Feature points in the image captured by the first camera within the second target area are then matched with feature points in the image captured by the second camera to form feature point pairs to determine the spatial alignment transformation matrix. The image captured by the first camera is then corrected based on the spatial alignment transformation matrix to obtain the second image. Because the spatial alignment transformation matrix is ​​calculated based on the feature points in the second target area during the zoom process, the corrected image can be smoothly magnified based on the second target area. This allows for smooth enlargement of the FOV of the image stream and smooth camera switching during the zoom process.

[0169] For example, the first zoom operation is an operation of increasing the zoom ratio, or the first zoom operation is an operation of decreasing the zoom ratio. When the zoom ratio is increased, the size of the subject included in the second image is larger than the size of the subject included in the first image. When the zoom ratio is decreased, the size of the subject included in the second image is smaller than the size of the subject included in the first image.

[0170] Optionally, the second target area in the second image is centered in the second image.

[0171] Optionally, the second image includes the entire content or a portion of the content of the second target area. Figure 19 As shown, assuming that the second target area is a rectangle including the fourth subject, after magnification, the second image may include the entire fourth subject, or the second image may also include partial content of the fourth subject, such as the face of the fourth subject; when further magnified, the second image may also include a partial face of the fourth subject.

[0172] Optionally, in order to smoothly enlarge the target object displayed by the electronic device during the zoom process, the above S340 can be executed multiple times in a loop, and the target object in the second image obtained each time is larger than the target object obtained last time. That is, in response to the first zoom operation, the electronic device corrects the image captured by the first camera using the spatial alignment transformation matrix between the images captured by the second camera, and obtains and displays multiple second images, each of which includes a second target area, and the second target area in the multiple second images gradually increases from the original size. The number of loop executions and the magnitude of the size change can be set as needed, and this application does not limit this. Among them, the original size indicates the size of the second target area in the first image.

[0173] It should be understood that S340 may be executed while S330 is being executed, or S340 may be executed after S330 is executed. This embodiment of the present application does not impose any limitation on this.

[0174] For example, when in the preview scene, the second image can be displayed. Wherein, the preview shooting mode includes but is not limited to: shooting modes such as night scene preview mode, video preview mode, photo preview mode, and portrait preview mode.

[0175] Exemplarily, when in a video recording scene, the second image may be displayed.

[0176] Optionally, the method further includes:

[0177] In response to a first zoom operation;

[0178] Determine that the zoom ratio of the electronic device meets the zoom ratio range of the first camera foreground display and the second camera for background operation.

[0179] For example, in response to the first zoom operation, when the zoom magnification has reached the minimum zoom magnification of the second camera but has not reached the maximum zoom magnification of the first camera (for example, Figure 15 As shown, the telephoto camera is at least 1.8x, and the main camera is at most 2.5x). At this time, the second image acquired by the first camera continues to be displayed, but the second image is obtained by correcting the image acquired by the first camera using the spatial alignment transformation matrix (one-to-one corresponding to the zoom ratio) between the image acquired by the second camera. The spatial alignment transformation matrix is ​​obtained by matching the feature points of the image acquired by the first camera in the second target area with the feature points of the image acquired by the second camera to form a feature point pair.

[0180] For example, the first zoom operation is to switch from 1× to 4×, and the first camera is always displayed in the foreground, but when zoomed to 1.8×, the second camera starts running in the background. At this time, the method provided in the embodiment of the present application can be triggered to determine the second target area, calculate the spatial alignment transformation matrix corresponding to 1.8×, correct the image captured by the first camera, and generate a second image for display and storage.

[0181] The present application provides an image processing method and related equipment. In the present application, a second target area is determined by combining an image captured by a first camera and a focus frame, which is equivalent to determining a unique foreground for the first camera and the second camera. Then, feature points in the image captured by the first camera within the second target area are matched with feature points in the image captured by the second camera to form a feature point pair to determine a spatial alignment transformation matrix. The image captured by the first camera is then corrected according to the spatial alignment transformation matrix to obtain a second image. Since the spatial alignment transformation matrix is ​​calculated based on feature points in the stable and unchanging second target area during zooming, the corrected image can be smoothly magnified based on the second target area. This allows the FOV of the image stream to be smoothly magnified, and the camera can be switched smoothly during zooming.

[0182] Figure 8 A schematic flowchart exemplarily shows another image processing method of the electronic device 100 .

[0183] Taking the first camera as the main camera, the second camera as the telephoto camera, the zoom ratio increasing, and the camera switching from the main camera to the telephoto camera as an example, the method 400 includes S401 to S410; S401 to S410 are described in detail below.

[0184] S401: Acquire a first initial image captured by a main camera.

[0185] Exemplarily, the image captured by the main camera may be a Raw image captured by a wide-angle camera, or a YUV image.

[0186] Optionally, after S401, S402 and S403 may be executed; or S404 and S405 may be executed, which is not limited in the embodiment of the present application. In addition, after S401, S406 may also be executed.

[0187] S402: Display a first image obtained by processing the first initial image.

[0188] For example, after the first initial image captured by the main camera is processed by the image sensor and the image processing method in the camera algorithm, the first image can be output to the display screen for display. The first image can be an RGB image.

[0189] S403: In response to a click operation on the first image, determine and display a focus frame.

[0190] For example, Figure 9 As shown in (a) of FIG, the electronic device displays a first image; in response to a click operation on the first image, the electronic device may determine and display a focus frame based on the clicked screen position, such as Figure 9 As shown in (b) in .

[0191] S404: Display a first image obtained by processing the first initial image.

[0192] S405 : In response to the auto-focus process, determine and display a focus frame.

[0193] Among them, the autofocus processing can be any of the following, such as: phase detection autofocus (PhaseDetection Autofocus), contrast detection autofocus (ContrastDetection Autofocus) and hybrid autofocus (Hybrid Autofocus); phase detection autofocus is used to use a dedicated sensor to detect the phase difference value of the image to quickly determine the focus; contrast detection autofocus is used to determine the clearest focus by analyzing the contrast of the image; hybrid autofocus is used to combine the advantages of phase detection and contrast detection to provide faster and more accurate focusing. Of course, the autofocus processing can also be other methods, which are not limited in the embodiments of the present application.

[0194] The focus frame is a component of the autofocus system. It refers to the specific area or areas within an image used to evaluate sharpness. During autofocus, the camera uses the information within the focus frame to adjust the lens for optimal image clarity.

[0195] Optionally, steps S404 and S405 may be interchanged. For example, S404 may be determining a focus frame in response to an auto-focus process; and S405 may be displaying a first image and a focus frame obtained by processing the first initial image.

[0196] For example, Figure 10 As shown in (a) in FIG. 1 , the electronic device displays the first image; in response to the autofocus process, the electronic device may determine and display a focus frame, such as Figure 10 As shown in (b) in .

[0197] S406: Segment the first initial image and identify the segmented first target area.

[0198] Optionally, instance segmentation processing is performed on the first initial image.

[0199] It should be understood that instance segmentation is used to identify each object in the first initial image and assign a pixel-level mask to each object instance. This algorithm can not only identify the object category in the first initial image, but also distinguish different instances of the same object. For example, it can distinguish between two cars or two people.

[0200] The segmentation algorithm used in the instance segmentation process may be an algorithm based on deep learning.

[0201] For example, the segmentation algorithm can be: Mask R-CNN, U-Net, DeepLab, YOLACT and TensorMask, etc. Of course, it can also be an algorithm based on other deep learning, which is not limited in the embodiments of the present application.

[0202] It should be understood that the segmented first target region is the region corresponding to each object instance after instance segmentation. The segmented first initial image can include one or more first target regions, and the parameters such as size and shape of each first target region can be the same or different.

[0203] For example, Figure 11 The image shown in (a) is the first initial image, wherein the first initial image is consistent with the first image and includes target 1, target 2, and target 3; after instance segmentation processing is performed on the first initial image, the first target area a1 corresponding to target 1, the first target area a2 corresponding to target 2, and the first target area a3 corresponding to target 3 can be identified.

[0204] Optionally, after S406 , S408 may be executed; or, S407 may be executed first and then S408 .

[0205] S407. When the aforementioned steps are to execute S402 and S403, in response to a click operation on the first image, while determining and displaying the focus frame, the mask area (also called mask) corresponding to the first target area identified by S406 can also be displayed; when the aforementioned steps are to execute S404 and S405, while displaying the first image and the focus frame, the mask area corresponding to the first target area identified by S406 can also be displayed at the same time.

[0206] It should be understood that the mask region generally refers to an array of the same size as the first target region, wherein the element values ​​are used to indicate whether each pixel in the first initial image or the first image belongs to the specific region or object of interest. In instance segmentation, the instance mask is used to distinguish different instances of the same object. Each instance has a unique mask, and the pixel values ​​in the mask are usually 1, but may contain an additional channel to represent different instances.

[0207] For example, for Figure 11 In the multiple first target areas shown in (b), the electronic device can determine the mask area corresponding to each first target area, such as the first target area a1 corresponds to the mask area b1, the first target area a2 corresponds to the mask area b2, and the first target area a3 corresponds to the mask area b3. Figure 11 As shown in (c) in .

[0208] For example, Figure 12 As shown in (a) in FIG. 1 , the electronic device displays the first image; in response to the autofocus process, the electronic device can determine and display the focus frame, and display the mask areas corresponding to the multiple first target areas, such as Figure 12 As shown in (b) of FIG. To distinguish different mask areas, a semi-transparent colored mask area can be superimposed on the first image; for example, mask area b1 is a semi-transparent red area; mask area b2 is a semi-transparent green area; and mask area b3 is a semi-transparent blue area. The color and transparency can be preset as needed and are not limited in this embodiment of the present application.

[0209] S408 : After S403 or after S405 , determine a second target area by combining the focus frame and the first target area.

[0210] Optionally, when the above method includes S407, the second target area may be determined in combination with the focus frame and the mask area corresponding to the first target area.

[0211] It should be understood that when there are multiple first target areas, the second target area is the first target area with the largest occupancy in the focus frame among the multiple first target areas; or the second target area is the mask area with the largest occupancy in the focus frame among the multiple mask areas. The second target area may also be referred to as the largest target area.

[0212] For example, Figure 11 As shown in (d) in FIG, compared with the mask area b2 and the mask area b3, the mask area b1 occupies the largest proportion in the focus frame. Therefore, the mask area b1 can be determined as the second target area.

[0213] Optionally, on this basis, the diagonal coordinates of the first target area or the mask area that occupies the largest proportion in the focus frame may be determined, and the area determined based on the diagonal coordinates is the second target area.

[0214] When the shape of the first target area or the mask area is irregular, the rectangle including the irregular area may be determined as the second target area. Furthermore, the rectangle including the irregular area may be a rectangle of minimum size.

[0215] S409: Acquire a second initial image captured by the telephoto camera.

[0216] Exemplarily, the image captured by the telephoto camera may be a Raw image or a YUV image captured by the telephoto camera.

[0217] S410: After receiving the first zoom operation, use the second target area as the zoom foreground and perform multi-camera switching in combination with the second initial image.

[0218] In an embodiment of the present application, the first initial image is instance segmented and combined with the focus information to obtain the second target area, and the second target area is determined to be the zoom foreground, which is equivalent to the first camera and the second camera determining a unified and unique foreground; in this way, when the first zoom operation is received, the foreground is stable and unchanged, and smooth zoom between multiple cameras can be achieved.

[0219] Alternatively, as Figure 13 As shown, the above S408 may include S4081 to S4085, and S4081 to S4085 are respectively introduced in detail below.

[0220] S4081. Obtain a focus frame.

[0221] The focus frame determined in S403 may be obtained, or the focus frame determined in S404 may be obtained.

[0222] S4082. Obtain a mask area corresponding to each first target area.

[0223] The mask area determined in S407 may be obtained.

[0224] S4083. Calculate the proportion of each mask area in the focus frame and sort them.

[0225] In other words, the intersection size of each mask area and the focus frame is calculated separately and sorted by size.

[0226] S4084. Select the mask area with the largest proportion as the target mask area.

[0227] For example, Figure 11As shown in (d) in FIG, compared with the mask area b2 and the mask area b3, the mask area b1 occupies the largest proportion in the focus frame. Therefore, the mask area b1 can be determined as the target mask area.

[0228] S4085 . Determine, according to the coordinates of the target mask area, a rectangle including the target mask area as a second target area.

[0229] Optionally, the rectangle including the target mask area may be determined as the second target area according to diagonal coordinates of the target mask area, such as upper left corner coordinates and lower right corner coordinates, or lower left corner coordinates and upper right corner coordinates.

[0230] For example, Figure 11 As shown in (d) in FIG. 5 , the shape of the mask area b1 is irregular. In this case, the rectangle with the smallest size including the irregular area can be determined as the second target area.

[0231] It should be understood that by performing instance segmentation on the first initial image and combining it with the focus information to obtain the second target area, since the two methods are combined to obtain the second target area, the accuracy of the obtained zoom foreground can be improved, so that multiple cameras can be aligned with the same zoom foreground subsequently; in this way, when the first zoom operation is received, smooth zoom between multiple cameras can be achieved.

[0232] Alternatively, as Figure 14 As shown, the above S410 may include S4101 to S4107, and S4101 to S4107 are respectively introduced in detail below.

[0233] S4101: Extract first feature points of the first initial image within the second target area.

[0234] S4102: Determine, in the second initial image, a second feature point that matches the first feature point.

[0235] Optionally, the above S4101 and S4102 can also be: performing image feature point detection and alignment on the first initial image and the second initial image to obtain multiple pairs of feature point pairs; then screening the first feature points in the second target area of ​​the first initial image and the second feature points in the second initial image that match the first feature points.

[0236] Optionally, any existing algorithm may be used for image feature point detection and matching, and this application does not impose any limitation thereto. For example, algorithms such as SIFT, SURF, and FAST may be used.

[0237] Exemplarily, image feature point detection is performed on the first initial image and the second initial image respectively, M feature points in the second target area of ​​the first initial image are extracted, and N second feature points matching the first feature points are determined in the second initial image; M and N are both integers greater than 0.

[0238] It should be understood that each set of feature point pairs includes a feature point belonging to the first initial image and a feature point belonging to the second initial image, and the two feature points are used to indicate the same object. The feature points after registration have a one-to-one correspondence.

[0239] Optionally, after S4102 , the method may further include: screening correctly registered feature point pairs.

[0240] Specifically, the correctly registered feature point pairs can be screened out by eliminating the incorrectly registered feature point pairs.

[0241] After testing, in each scene, there are more correctly registered feature points than incorrectly registered feature points. Therefore, in order to improve processing efficiency and the effect of subsequent processing, the incorrectly registered feature points can be eliminated and only the correctly registered feature point pairs are retained.

[0242] S4103. Based on the feature point pair consisting of the matched first feature point and the second feature point, calculate the maximum difference parameter (i.e., at least including the maximum rotation angle and the maximum offset) corresponding to the target zoom ratio when switching from the camera (i.e., the camera switching point).

[0243] For example, Figure 15 As shown in (a), the applicable zoom magnification range for the main camera is 1× to 2.5×, and the applicable zoom magnification range for the telephoto camera is 1.8× to 10×. The target zoom magnification corresponding to the switching point between the main and telephoto cameras is 2.5×. Therefore, the rotation angle and offset calculated based on the feature point pairs are the maximum rotation angle and maximum offset corresponding to 2.5×.

[0244] In an embodiment of the present application, feature point detection and registration are performed on the two images captured by the first camera and the second camera, and then the feature point pairs are used to determine the rotation angle and offset corresponding to the zoom ratio when switching from the first camera to the second camera. For the two frames of images, the matched feature point pairs are relatively consistent in expression, and the determined rotation angle and offset are representative. For the two image streams captured by the first camera and the second camera, the rotation angle and offset are relatively consistent when the two image streams are corrected, thereby ensuring that the FOV of the image remains consistent during the zoom process and the camera switching process, ensuring a smooth transition.

[0245] S4104: Based on the maximum difference parameter (maximum rotation angle and maximum offset), assign different difference parameters (rotation angle and offset) to different zoom magnifications in the first zoom magnification range.

[0246] The first zoom ratio range is a zoom ratio range corresponding to when the main camera is running in the foreground and the telephoto camera is running in the background, and the maximum value of the first zoom ratio range is the target zoom ratio.

[0247] In the embodiment of the present application, the rotation angle and the offset in the first zoom ratio range are obtained by allocating the total amount (the maximum rotation angle and the maximum offset corresponding to the target zoom ratio).

[0248] Optionally, the rotation angle and offset are distributed in the form of a linear or exponential curve. The unit of the offset is pixels.

[0249] For example, Figure 15 As shown in (a) in FIG. 1 , the zoom ratio range of the telephoto camera running in the background is 1.8× to 2.5×, that is, the first zoom ratio range is 1.8× to 2.5× indicated by H1.

[0250] like Figure 15 As shown in (b) in the figure, taking the offset as an example, assuming that the total offset corresponding to the camera switching point is 100, then based on the allocation logic (for the sake of convenience, it is assumed to be linear), the offset allocated by 1× is 0, the offset allocated by 2.5× is 100, then the offset allocated by 1.1× is 100*(1.1-1) / (2.5-1)≈6.67, the offset allocated by 1.8× is approximately equal to 53.3, and the offset allocated by 2× is approximately equal to 66.7. The calculation process allocated under other zoom magnifications is the same and will not be introduced here. It should be understood that the allocation process of the rotation angle is similar and will not be repeated here.

[0251] S4105: Determine, according to the difference parameter, the spatial alignment transformation matrix corresponding to different zoom magnifications in the first zoom magnification range.

[0252] It should be understood that each zoom factor in the first zoom factor range may be assigned a rotation angle and an offset, and based on the assigned rotation angle and offset, a spatial alignment transformation matrix corresponding to each zoom factor may be determined.

[0253] S4106: Determine a spatial alignment transformation matrix corresponding to the zoom magnification of the electronic device in response to the first zoom operation.

[0254] It should be understood that after determining the spatial alignment transformation matrix corresponding to each zoom magnification in the first zoom magnification range, the spatial alignment transformation matrix corresponding to the zoom magnification of the electronic device may be searched.

[0255] S4107: Correct the first initial image based on the space alignment transformation matrix corresponding to the zoom magnification of the electronic device to obtain a second image.

[0256] It should be understood that in response to the first zoom operation, the electronic device corrects the first initial image based on different spatial alignment transformation matrices corresponding to different zoom ratios during the zoom sliding process, and can obtain and display multiple second images. The multiple second images all include a second target area, and the second target areas in the multiple second images gradually increase from the original size.

[0257] For example, Figure 16 A set of image streams involved in the embodiment of this application. Figure 16 (a), (b), (c), (d), (e) to Figure 16 As shown in (f) in Figure 6 The same as (a) in the figure, assuming that the captured image includes target 1, target 2, and target 3, and the depths of the three targets are different; combined with instance segmentation and focus information, the minimum rectangle including target 1 can be obtained as the second target area. During the zoom process, the rotation angle and offset are mainly calculated based on the feature points of the two images in the second target area. In this way, the corrected image can be smoothly zoomed in based on the second target area, and the FOV of the image stream can be smoothly zoomed in, and the camera can be switched smoothly during the zoom process.

[0258] Figure 17 The following is a schematic flow chart showing another image processing method of the electronic device 100. The method 500 includes S501 to S509, and S501 to S509 are described in detail below.

[0259] S501: Acquire an image captured by a main camera.

[0260] Exemplarily, the image captured by the main camera may be a Raw image captured by the wide-angle camera; or, the image captured by the main camera may be a Raw image captured by the ultra-wide-angle camera.

[0261] S502: First front-end processing.

[0262] Optionally, the first front-end processing may include a related algorithm for converting the Raw image captured by the main camera into a YUV image; this application does not impose any limitation on the algorithm.

[0263] For example, the first front-end processing may refer to an image processing algorithm executed in the ISP to convert a raw image into a YUV image. The first front-end processing may also include subsequent operations such as cropping and magnification based on the zoom ratio in response to the first zoom operation.

[0264] S503: Acquire an image captured by the telephoto camera.

[0265] Exemplarily, the image captured by the telephoto camera may be a Raw image captured by the telephoto camera.

[0266] Optionally, S504 and S501 may be executed simultaneously, or S504 and S501 may be executed sequentially.

[0267] S504: Second front-end processing.

[0268] Optionally, the second front-end processing may include a related algorithm for converting the Raw image captured by the telephoto camera into a YUV image; this application does not impose any limitation on the algorithm.

[0269] For example, the second front-end processing may refer to an image processing algorithm executed in the ISP to convert a raw image into a YUV image. The second front-end processing may also include subsequent operations such as cropping and magnification based on the zoom ratio in response to the first zoom operation.

[0270] After executing S505 , the processed image is subjected to a first back-end processing. For the implementation of the first back-end processing, reference may be made to the relevant description of S503 .

[0271] For example, when the zoom ratio is 1× to 2.5×, only S501 is executed. When the zoom ratio is 2.5× to 5×, S501 and S503 are executed simultaneously. When the zoom ratio is 5× or above, only S503 is executed.

[0272] S505: First backend processing.

[0273] Exemplarily, the first back-end processing includes but is not limited to: brightening processing, denoising processing, saturation adjustment processing, cropping processing or deformation processing.

[0274] S506: Smoothing process (or alignment process).

[0275] For example, the image stream captured by the main camera and the image stream captured by the telephoto camera are smoothed; it can be understood that the input smoothing data is two image streams (for example, the image stream captured by the main camera and the image stream captured by the telephoto camera), and the image stream after smoothing is one image stream. This smoothing process is provided in the embodiment of the present application. Figure 7 、 Figure 8 、 Figure 13 and / or Figure 14 The image processing method shown.

[0276] For example, when the zoom ratio increases from 1.8× to 2.5×, the smoothed image stream is based on the image stream captured by the main camera, and the image processing method provided in the embodiment of the application is used to correct the image stream of the main camera. When the zoom ratio increases from 2.5× to 10×, the smoothed image stream is based on the image stream captured by the telephoto camera.

[0277] In the embodiment of the present application, smoothing processing can avoid obvious jumps between the foreground and background of the image when switching cameras.

[0278] S507: Second backend processing.

[0279] Exemplarily, the second back-end processing includes anti-shake processing; for example, anti-shake processing may be performed on image frames in the image stream according to a jitter parameter to obtain a processed image.

[0280] S508: Display the processed image.

[0281] For example, when the zoom ratio increases and the zoom ratio is 1×~1.8×, the processed image is the processed image corresponding to the image captured by the main camera; when the zoom is to 1.8×~2.5×, the processed image is the image captured by the main camera, which is corrected by the spatial alignment transformation matrix between the image captured by the telephoto camera; when the zoom is to 2.5× and above, the processed image is the processed image corresponding to the image captured by the telephoto camera.

[0282] S509: Detect a first zoom operation.

[0283] The first zoom operation may refer to the description in S330 above.

[0284] Optionally, the electronic device may display an image captured by the main camera, or an image captured by the telephoto camera, and the electronic device detects a user operation on the displayed image.

[0285] It should be understood that the above examples are intended to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to the specific numerical values ​​or specific scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or variations based on the above examples, and such modifications or variations also fall within the scope of the embodiments of the present application.

[0286] For example, Figure 18 A schematic diagram of an application scenario provided for an embodiment of the present application.

[0287] like Figure 18As shown in (a) of FIG, in response to a user's operation on a camera application, the electronic device may display a preview interface 601, which includes a preview window and a shooting control, wherein a preview image 602 displayed in the preview window may include a first shooting object, a second shooting object, and a third shooting object. Here, when the camera application is opened, the zoom ratio defaults to 1×. Figure 18 As shown in (b), in response to the autofocus process, it is assumed that the focus frame is displayed around the third photographic object; at the same time, the mask area corresponding to each photographic object is displayed; when the user performs a two-finger reverse sliding operation on the screen, in response to the operation, for example, Figure 18 As shown in (c), the zoom ratio can be increased to 10×, and the size of the image corresponding to the third object (the image is a rectangle including the third object determined by combining the mask area and the focus frame, that is, the second target area) changes from small to large.

[0288] It should be noted that when zooming in, the mask area corresponding to the third shooting object also becomes larger along with the main body.

[0289] When the zoom ratio increases and the camera switches, you can apply Figure 7 、 Figure 8 、 Figure 13 or Figure 14 The image processing method introduced can be used to identify and track a third object.

[0290] It should be understood that the above is a preview scene in the photo mode, and the embodiment of the present application can also be applied to preview scenes in other shooting modes such as video. The preview scene can refer to the scene in which the electronic device captures an image before clicking the shooting control in a certain shooting mode.

[0291] For example, Figure 19 A schematic diagram of another application scenario provided for an embodiment of the present application.

[0292] After the camera application is opened, in response to the user's click operation on the recording control, the electronic device can start recording the video and display the video recording interface 701, such as Figure 19 As shown in (a) of FIG. 7 , the video recording interface 701 may include a recording window, a pause control, an end control, and a zoom control. The video image 702 displayed in the recording window may include a first subject, a second subject, a third subject, and a fourth subject. When video recording begins, the zoom ratio defaults to 1×.

[0293] like Figure 19As shown in (a) of FIG, during the recording process, assuming that the user clicks on any position on the screen, or the user wants to zoom in on the face of the fourth shooting subject on the left side of the video image 702, the user can click on the corresponding position of the fourth shooting subject on the screen. Figure 19 As shown in (b) of FIG, in response to the user's click operation, a focus frame is superimposed on the fourth shooting object included in the video image 703, and at the same time, a mask area is superimposed on each shooting object, and the color of the mask area corresponding to each shooting object is different. Then, the electronic device can receive the user's sliding operation on the zoom control, such as Figure 19 As shown in (c) in the figure, in response to a sliding operation on the zoom control (assuming sliding to 8×), the electronic device can display a video image 704, in which the face of the fourth subject included in the video image 704 becomes larger relative to the face of the fourth subject included in the video image 702.

[0294] It should be noted that when zooming in, the mask area corresponding to the fourth photographic object also becomes larger along with the main body.

[0295] When the zoom ratio changes and the camera switches, it can adapt Figure 7 、 Figure 8 、 Figure 13 or Figure 14 The image processing method introduced is used to identify and track the subject.

[0296] It should be understood that the above is a video recording scenario, and the embodiments of the present application can also be applied to scenarios such as video calls.

[0297] Exemplarily, video call scenarios may include but are not limited to the following scenarios: video calls, video conferencing applications, long and short video applications, live video applications, online video courses, portrait smart mirror application scenarios, system camera recording function video recording, video surveillance, or smart cat-eye and other shooting scenarios.

[0298] Combined with the above Figures 1 to 19 The image processing method provided in the embodiments of the present application is described in detail.

[0299] The present application also provides a computer program product, which, when executed by a processor, implements the method described in any method embodiment of the present application.

[0300] The computer program product may be stored in a memory, for example, a program, which is converted into an executable target file that can be executed by a processor after undergoing processes such as preprocessing, compilation, assembly, and linking.

[0301] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer, implements the method described in any method embodiment of the present application. The computer program can be a high-level language program or an executable target program.

[0302] The computer-readable storage medium is, for example, a memory. The memory may be a volatile memory or a non-volatile memory, or the memory may include both volatile memory and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0303] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and equipment and the technical effects produced can refer to the corresponding processes and technical effects in the aforementioned method embodiments, and will not be repeated here.

[0304] In the several embodiments provided in this application, the disclosed systems, devices and methods can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not executed. The device embodiments described above are merely schematic, and the division of units is only a logical function division. There may be other division methods in actual implementation, and multiple units or components may be combined or integrated into another system. In addition, the coupling between the units or the coupling between the components may be direct coupling or indirect coupling, and the above coupling includes electrical, mechanical or other forms of connection.

[0305] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0306] Additionally, the terms "system" and "network" are often used interchangeably. The term "and / or" in this document simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates an "or" relationship between the related objects.

[0307] In short, the above description is only a preferred embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application shall be included in the scope of protection of this application.

Claims

1. An image processing method, characterized in that: Applied to an electronic device, the electronic device includes a first camera and a second camera, and the method includes: Launch the Camera app; Displaying a first image and a focus frame, where the first image is obtained from an image captured by the first camera; receiving a first zoom operation; A second image is displayed, where the second image is corrected by using a spatial alignment transformation matrix between the image captured by the first camera and the image captured by the second camera. The spatial alignment transformation matrix is ​​obtained by matching feature points within a second target area of ​​the image captured by the first camera with feature points of the image captured by the second camera to form feature point pairs. The second target area is determined by the image captured by the first camera and the focus frame.

2. The image processing method according to claim 1, wherein: The method further comprises: receiving a second zoom operation; Display and save a third image, where the third image is obtained by capturing an image with the second camera.

3. The image processing method according to claim 1 or 2, characterized in that: The method further comprises: Acquire a first initial image captured by the first camera; performing segmentation processing on the first initial image and identifying a first target area after segmentation; The second target area is determined by combining the first target area and the focus frame.

4. The image processing method according to claim 3, wherein: Determining the second target area by combining the first target area and the focus frame includes: Acquire the focus frame; Acquire a mask area corresponding to the first target area; Calculating the proportion of each mask area in the focus frame respectively and sorting them; Determine the mask area with the largest proportion as the target mask area; According to the coordinates of the target mask area, a rectangle including the target mask area is determined as the second target area.

5. The image processing method according to claim 4, characterized in that When displaying the focus frame, the method further includes: The mask area corresponding to the first target area is displayed, and different mask areas have different colors.

6. The image processing method according to any one of claims 1 to 5, characterized in that: The displaying of the first image and the focus frame comprises: displaying the first image; In response to a click operation on the first image, determining and displaying the focus frame; or, displaying the first image; In response to the auto-focus process, the focus frame is determined and displayed.

7. The image processing method according to any one of claims 3 to 5, characterized in that: The method further includes: acquiring a second initial image captured by the second camera; extracting first feature points of the first initial image within the second target area; determining, in the second initial image, a second feature point that matches the first feature point; calculating, based on the matched feature point pairs consisting of the first feature points and the second feature points, a maximum difference parameter corresponding to a target zoom ratio when switching from the first camera to the second camera; Based on the maximum difference parameter, the first initial image is corrected to obtain the second image.

8. The image processing method according to claim 7, wherein: Correcting the first initial image based on the maximum difference parameter to obtain the second image includes: assigning different difference parameters to different zoom magnifications in the first zoom magnification range based on the maximum difference parameter; determining, according to the difference parameter, spatial alignment transformation matrices corresponding to different zoom magnifications in the first zoom magnification range; determining a spatial alignment transformation matrix corresponding to a zoom magnification of the electronic device in response to the first zoom operation; Correcting the first initial image based on a spatial alignment transformation matrix corresponding to a zoom magnification of the electronic device to obtain the second image; Among them, the first zoom ratio range is the zoom ratio range corresponding to when the first camera is displayed in the foreground and the second camera is running in the background, and the maximum value of the first zoom ratio range is the target zoom ratio; the zoom ratio of the electronic device satisfies the first zoom ratio range.

9. The image processing method according to claim 8, characterized in that: The difference parameters include at least a rotation angle and an offset, and the maximum difference parameters include at least a maximum rotation angle and a maximum offset.

10. The image processing method according to any one of claims 1 to 9, characterized in that: The method further includes: in response to the first zoom operation; Determine that the zoom ratio of the electronic device meets the zoom ratio range of the first camera being displayed in the foreground and the second camera being run in the background.

11. The image processing method according to claim 2, wherein: The method further includes: in response to the second zoom operation; Determine that the zoom ratio of the electronic device meets the zoom ratio range that the first camera runs in the background or is closed, and the second camera is displayed in the foreground.

12. The image processing method according to claim 2 or 11, characterized in that: The first zoom operation or the second zoom operation includes: a two-finger reverse sliding operation, a sliding operation on a displayed zoom control, a voice operation, or an air gesture operation.

13. The image processing method according to any one of claims 1 to 12, characterized in that: The first camera includes a wide-angle camera, and / or the second camera includes a telephoto camera or an ultra-wide-angle camera.

14. The image processing method according to claim 2, wherein: The first zoom operation and the second zoom operation are continuous first zoom operations.

15. The image processing method according to claim 3, wherein: The segmentation process is instance segmentation.

16. An electronic device, characterized in that: The electronic device includes: one or more processors, a memory, a first camera and a second camera; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the image processing method as described in any one of claims 1 to 15.

17. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to execute the image processing method according to any one of claims 1 to 15.

Images