Image Processing Method, Device, and Storage Medium

Abstract

An embodiment of the present application provides an image processing method, device, and storage medium. Among them, the image processing method includes: obtaining a first image corresponding to a first camera, a second image corresponding to a second camera, and a depth image corresponding to a depth camera; the first image, the second image, and the depth image correspond to the same scene; when the user input magnification is in the switching magnification range, determining an affine transformation relationship between the first image and the second image according to the matching relationship between the first image and the second image; calculating the switching magnification according to the calibration data of the first camera and the second camera; processing the first image according to the affine transformation relationship and the depth image so that when the user input magnification reaches the switching magnification, the first image is the same as the second image. The present application can reduce the influence of parallax, reduce the jump of optical zoom, and improve the user experience of optical zoom.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of image processing technology. Specifically, the present application relates to an image processing method, device, and storage medium. Background Art

[0002] Optical zoom is produced by changes in the positions of the lens, object, and focus. When the imaging plane moves horizontally, the viewing angle and focal length change, making more distant scenes clearer and giving the impression of the object approaching.

[0003] Digital video cameras rely on the optical lens structure to achieve zoom. In the field of mobile phone photography, due to the insufficient thickness of a single lens, it is impossible to achieve optical zoom like a digital camera. Therefore, dual-camera and multi-camera modules have become the choice for optical zoom. For example, in dual-camera zoom, if the field of view (FOV) is 80 degrees, the wide angle is 40 degrees, and the telephoto (tele) combination is selected, then changing from 1x (normal shooting size mode) to 2x (shortening the shooting distance and magnifying by 2 times) is digital magnification of the wide lens. When magnified to 2x, it can be switched to tele, and when greater than 2x, it is digital magnification of the tele lens.

[0004] Due to lens and module manufacturing process problems, there is often a large jump in the image content when switching to 2x (i.e., there is a large translation in the same content area, and the image content jump includes jumps such as image scaling, up / down / left / right translation of the image). The main methods to solve the above problems in the prior art are as follows:

[0005] 1. Calibration alignment. By using a calibration method to perform stereo calibration on the image, the alignment above and below the baseline and the alignment at a known distance can be solved.

[0006] 2. Feature point detection alignment.

[0007] The above methods are the most common methods to solve the image content jump currently. When implementing optical zoom using the above methods, parallax cannot be eliminated, that is, within a certain range, the focused area we see does not change, but the background area jumps severely, resulting in a poor user experience. Summary of the Invention

[0008] Embodiments of the present application provide an image processing method, device, and storage medium to solve the problem of background jitter caused by the inability to eliminate parallax when implementing optical zoom in the prior art.

[0009] In a first aspect of the embodiments of the present application, an image processing method is provided. The method includes:

[0010] Obtain a first image corresponding to a first camera, a second image corresponding to a second camera, and a depth image corresponding to a depth camera; the first image, the second image, and the depth image correspond to the same scene;

[0011] When the user input magnification is within a switching magnification range, determine the affine transformation relationship between the first image and the second image according to the matching relationship between the first image and the second image;

[0012] Calculate the switching magnification according to the calibration data of the first camera and the second camera;

[0013] Process the first image according to the affine transformation relationship and the depth image, so that when the user input magnification reaches the switching magnification, the first image is the same as the second image.

[0014] A second aspect of the embodiments of the present application provides an electronic device, including a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of the method described in the first aspect of the present application are implemented.

[0015] A third aspect of the embodiments of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps in the method described in the first aspect of the present application are implemented.

[0016] A fourth aspect of the embodiments of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps in the method described in the first aspect of the present application are implemented.

[0017] In the embodiments of the present application, the first image, the second image, and the depth image are obtained by image acquisition of the same scene through the first camera, the second camera, and the depth camera. When the user input magnification is within the switching magnification range, the affine transformation relationship between the first image and the second image is determined according to the matching relationship between the first image and the second image, and the first image is processed according to the affine transformation relationship and the depth image to ensure that the first image and the second image are the same when the user input magnification reaches the switching magnification, so as to reduce the influence of parallax, reduce the jump of optical zoom, and improve the user experience of optical zoom. Description of the Drawings

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1 is a flowchart of the image processing method according to an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of the switching magnification range according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of the depth image according to an embodiment of the present application;

[0022] Figure 4a is a schematic diagram of the background area covering the focus area according to an embodiment of the present application;

[0023] Figure 4b is a schematic diagram of the background hole according to an embodiment of the present application;

[0024] Figure 5 is a specific implementation flowchart of the image processing method according to an embodiment of the present application;

[0025] Figure 6 is a schematic diagram of the image processing device according to an embodiment of the present application. Detailed implementation manners

[0026] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0027] An embodiment of the present application provides an image processing method, as Figure 1 shown, the method includes:

[0028] Step 101, obtaining a first image corresponding to a first camera, a second image corresponding to a second camera, and a depth image corresponding to a depth camera; the first image, the second image, and the depth image correspond to the same scene.

[0029] The image processing method provided by the embodiment of the present application needs to be implemented in combination with a first camera, a second camera, and a depth camera. The first camera, the second camera, and the depth camera are integrated on the same electronic device. The first camera, the second camera, and the depth camera perform image acquisition on the same scene, and can obtain a first image corresponding to the first camera, a second image corresponding to the second camera, and a depth image corresponding to the depth camera.

[0030] Step 102: When the user input magnification is within the switching magnification range, determine the affine transformation relationship between the first image and the second image according to the matching relationship between the first image and the second image.

[0031] When the user input magnification is within the dual-camera magnification range, both the first camera and the second camera are turned on. The switching magnification range is within the dual-camera magnification range. When the user input magnification is within the switching magnification range, since both the first camera and the second camera are turned on, the affine transformation relationship between the first image and the second image can be determined according to the matching relationship between the first image and the second image. The affine transformation relationship includes at least a rotation relationship and a translation relationship.

[0032] Step 103: Calculate the switching magnification according to the calibration data of the first camera and the second camera.

[0033] According to the calibration data of the first camera and the second camera, the basic information of the first camera and the second camera can be obtained. The basic information may include focal length, pixel size, image resolution, and module assembly position. The switching magnification can be calculated according to the corresponding focal lengths at the same resolution, that is, the switching point between the two cameras.

[0034] That is, the switching magnification is the switching point for switching between the first camera and the second camera. When the user input magnification reaches the switching magnification, the first camera and the second camera are switched. The switching point is a certain magnification in the dual-camera magnification range, dividing the dual-camera magnification range into two intervals. The interval before the switching point is the switching magnification range, as shown in Figure 2 shown. Within the switching magnification range, the image captured by the first camera is displayed in the graphical user interface. At the switching point, the first camera and the second camera are switched. In the interval after the switching point, the image captured by the second camera is displayed in the graphical user interface.

[0035] Step 104: Process the first image according to the affine transformation relationship and the depth image so that when the user input magnification reaches the switching magnification, the first image is the same as the second image.

[0036] When the user input magnification is within the switching magnification range, the affine transformation relationship between the first image and the second image can be determined. According to the determined affine transformation relationship and the depth image, the first image captured by the first camera is processed. The processing of the first image according to the affine transformation relationship includes rotation and translation. The processing of the first image according to the depth image includes parallax adjustment. By rotating, translating, and adjusting the parallax of the first image, when the user input magnification reaches the switching magnification, the processed first image can be made the same as the second image, so as to achieve a smooth transition effect when switching the camera.

[0037] In the above implementation process of the present application, the first camera, the second camera, and the depth camera are used to collect images of the same scene to obtain the first image, the second image, and the depth image. When the magnification input by the user is within the switching magnification range, according to the matching relationship between the first image and the second image, the affine transformation relationship between the first image and the second image is determined, and the first image is processed according to the affine transformation relationship and the depth image to ensure that the first image and the second image are the same when the magnification input by the user reaches the switching magnification, so as to reduce the influence of parallax, reduce the jump of optical zoom, and improve the user experience of optical zoom.

[0038] In an optional embodiment of the present application, the processing of the first image according to the affine transformation relationship and the depth image includes:

[0039] Performing a rotation process on the first image according to the affine transformation relationship;

[0040] Based on the depth image and the focus area of the obtained first image, performing a parallax adjustment process on the first image to eliminate the parallax of the first image after the rotation process;

[0041] Performing a cropping process on the first image according to the affine transformation relationship to obtain the first image after rotation, parallax adjustment, and cropping.

[0042] When processing the first image according to the affine transformation relationship and the depth image, the rotation process of the first image can be first performed according to the affine transformation relationship. After the rotation process is completed, based on the depth image and the focus area of the first image, the parallax adjustment process is performed on the first image to eliminate the parallax of the first image after the rotation process. Finally, the affine transformation relationship is used to perform a cropping process on the first image after rotation and parallax adjustment to obtain the first image after rotation, parallax adjustment, and cropping. Among them, the cropping process includes the scaling and translation of the image. By performing the cropping process, the jump of the image content can be reduced when switching the camera.

[0043] Among them, when processing the first image, within the range where the magnification input by the user reaches the switching magnification, each frame of the first image collected by the first camera is processed.

[0044] The following elaborates on the process of performing a rotation process on the first image according to the affine transformation relationship. When performing a rotation process on the first image according to the affine transformation relationship, it includes:

[0045] Calculating the product of the image rotation parameters corresponding to the first image and the coordinates of each pixel point corresponding to the first image to perform a rotation process on the first image;

[0046] Among them, the image rotation parameter is associated with the image rotation angle of the first camera relative to the second camera.

[0047] When performing rotation processing on the first image based on the affine transformation relationship, it is necessary to obtain the image rotation parameter corresponding to the first image, where the image rotation parameter is associated with the image rotation angle of the first camera relative to the second camera. Specifically, the image rotation parameter H ot is the first matrix, and H r is the second matrix, that is, the image rotation parameter is determined according to the product of the first matrix, the second matrix, and the inverse matrix of the first matrix.

[0048] Among them, the first matrix is:

[0049]

[0050] In the first matrix, u0 represents the abscissa of the center point of the image of the first image, and v0 in the first matrix represents the ordinate of the center point of the image of the first image. Therefore, the first matrix is determined according to the center coordinates of the first image.

[0051] Among them, the second matrix is:

[0052]

[0053] The second matrix is the two-dimensional image rotation formula, θ z is the calibration input angle value of the first camera, switchLevel is the switching magnification, useLevel is the user input magnification, and the warpping length is the length of the switching magnification interval. Therefore, the second matrix is determined according to the calibration input angle value of the first camera, the switching magnification, the user input magnification, and the length of the switching magnification interval. The calibration input angle value of the first camera is the image rotation angle of the image captured by the first camera relative to the image captured by the second camera, and the image rotation angle can be determined based on the calibration data of the first camera and the calibration data of the second camera.

[0054] After determining the image rotation parameter, the first image can be rotated according to the image rotation parameter. When rotating the first image, it is actually calculating the product of the image rotation parameter and the coordinates of each pixel point corresponding to the first image to achieve the rotation processing of the first image. By rotating the first image, it can ensure the smooth transition of the image when the camera is switched.

[0055] The process of rotating the first image is introduced above. Next, the process of performing parallax adjustment on the first image is introduced. Performing parallax adjustment on the first image based on the depth image and the focus area of the obtained first image includes:

[0056] Based on the depth image, obtain the depth value of the first image and convert it into a parallax value;

[0057] According to the parallax value, perform image segmentation on the first image after rotation processing to obtain mask images corresponding to the focus area and the background area of the first image;

[0058] Perform parallax adjustment on the first image according to the mask image.

[0059] See Figure 3 As shown, it is a depth image obtained by using a depth camera to collect an image of a target person. After using a first camera to collect an image of the target person to obtain the first image corresponding to the target person and using a depth camera to collect an image of the target person to obtain the depth image corresponding to the target person, parallax adjustment can be performed on the first image based on the depth image and the focus area of the first image. For the case of collecting an image of a target person, the focus area is the area corresponding to the target person.

[0060] Among them, when performing parallax adjustment on the first image based on the depth image and the focus area of the first image, the depth value of the first image can be first obtained based on the depth image and converted into a parallax value. The conversion formula between the depth value and the parallax value is x = b * f / d, where b represents the baseline, f represents the focal length, d represents the distance (depth value), and x represents the parallax value. When obtaining the depth value of the first image and converting it into a parallax value, specifically: obtain the depth value of the first image, and after obtaining the depth value of the first image, based on the above conversion formula, obtain the parallax value of the first image.

[0061] When performing image segmentation on the first image after rotation processing according to the parallax value, based on the parallax value of the focus area of the first image, intercept the area where the difference from this parallax value is greater than a preset number (such as 3) of pixels (pixels) to achieve the segmentation of the focus area and the background area of the first image, and obtain mask images corresponding to the focus area and the background area of the first image. Then, based on the mask images corresponding to the focus area and the background area, perform parallax adjustment on the first image (the first image after rotation processing). Among them, the process of performing parallax adjustment on the first image can be understood as the process of translating the background area or the focus area. In the embodiments of the present invention, the case of translating the background area is taken as an example for elaboration.

[0062] In the embodiments of the present application, the depth image and the first image are synchronously acquired. Therefore, when performing parallax adjustment processing on the first image based on the depth image, the adjustment can be performed one-to-one. That is, the depth camera and the first camera synchronously acquire images, and a one-to-one correspondence is formed between the depth image and the first image.

[0063] Since the process of performing parallax adjustment processing on the first image can be understood as a process of translating the background area, when performing parallax adjustment processing, it is necessary to determine the corresponding background translation amount. When determining the background translation amount, according to the mask maps corresponding to the focus area and the background area, the parallax (M pixels) of the focus area and the background area is obtained. And the translation of the background area needs to be completed within N frames of images. Then the background translation amount is equal to the ratio of M pixels to N, so as to eliminate the background parallax through N translations. It should be noted that the translation of the background area needs to be completed within N frames of images, which means the moment when the user input magnification reaches the switching magnification, corresponding to N frames of images, that is, the duration of this period is the duration of acquiring N frames of images.

[0064] For the depth image and the first image acquired synchronously next time, when performing parallax adjustment processing on the first image, the background translation amount is determined according to the ratio of the parallax of the focus area and the background area to (N - 1). For each translation of the background area, the parallax of the focus area and the background area usually changes, and the background translation amount is determined according to the parallax of the current focus area and the background area.

[0065] Among them, after the translation of the background area, two problems of covering the focus area and background holes will occur (see Appendix Figure 4a and Figure 4b ). For the situation of covering the focus area ( Figure 4a ), it can be solved by using the method of filling the focus area. For the situation of background holes ( Figure 4b ), it is solved by the method of filling the background edge color and smoothing the filled area.

[0066] In the above implementation process, by obtaining the depth value of the first image and converting it into a parallax value, based on the parallax value, obtaining the mask maps corresponding to the focus area and the background area of the first image, and performing parallax adjustment processing on the first image according to the mask maps, the background parallax can be eliminated, the truncation feeling can be reduced, and the smoothness of the optical zoom can be improved.

[0067] The above introduced the process of performing parallax adjustment on the first image. Next, the process of performing cropping processing on the first image after rotation and parallax adjustment according to the affine transformation relationship will be introduced to obtain the first image after rotation, parallax adjustment and cropping.

[0068] Among them, the process of cropping the first image according to the affine transformation relationship includes:

[0069] Obtain an image magnification parameter according to the ratio of the user input magnification corresponding to the first image to the true zoom magnification;

[0070] Obtain the translation amount between the focus area corresponding to the first camera and the focus area corresponding to the second camera. According to the translation amount, the true zoom magnification corresponding to the first image, the switching magnification, the starting zoom magnification of the switching magnification interval, and the target zoom magnification corresponding to the time when the translation amount is calculated, obtain the image translation parameter corresponding to the first image;

[0071] Determine the cropping parameter corresponding to the first image according to the image magnification parameter, image translation parameter, preset image width, and preset image height corresponding to the first image, and crop the first image.

[0072] When cropping the first image according to the affine transformation relationship, it can be understood as performing scaling and translation processing on the first image. First, the scaling process is introduced. When cropping the first image, it is necessary to determine the image magnification parameter according to the ratio of the user input magnification to the true zoom magnification. In the case of a single camera, the user input magnification is equal to the true zoom magnification of the first camera. When entering the switching magnification interval, the user input magnification is greater than the true zoom magnification of the first camera. Therefore, it is necessary to obtain the image magnification parameter according to the ratio of the user input magnification to the true zoom magnification.

[0073] Then, the translation process is introduced. In this process, it is necessary to obtain the translation amount of the focus area corresponding to the first camera relative to the focus area corresponding to the second camera. After obtaining the translation amount of the focus areas collected by the two cameras, according to the translation amount, the true zoom magnification corresponding to the first image, the switching magnification, the starting zoom magnification of the switching magnification interval, and the target zoom magnification, obtain the image translation parameter corresponding to the first image. Among them, the target zoom magnification is the zoom magnification set when calculating the translation amount between the focus area corresponding to the first camera and the focus area corresponding to the second camera.

[0074] After obtaining the image magnification parameter and the image translation parameter, the cropping parameter corresponding to the first image can be determined according to the image magnification parameter, the image translation parameter, the preset image width, and the preset image height, and the first image is cropped according to the determined cropping parameter.

[0075] Among them, when obtaining the image translation parameter corresponding to the first image according to the translation amount, the true zoom magnification corresponding to the first image, the switching magnification, the starting zoom magnification of the switching magnification interval, and the target zoom magnification, it includes:

[0076] When the user input magnification is less than the switching magnification, determine a first adjustment coefficient according to the true zoom magnification, the target zoom magnification, and the switching magnification corresponding to the first image, and determine the image translation parameter corresponding to the first image according to the product of the first adjustment coefficient and the translation amount;

[0077] When the user input magnification is greater than the switching magnification, determine a second adjustment coefficient according to the true zoom magnification, the target zoom magnification, the switching magnification, and the starting zoom magnification corresponding to the first image, and determine the image translation parameter corresponding to the first image according to the product of the second adjustment coefficient and the translation amount;

[0078] Wherein, the translation amount includes a first translation value in the image width direction and a second translation value in the image height direction, and the image translation parameter includes a first translation parameter corresponding to the first translation value and a second translation parameter corresponding to the second translation value.

[0079] When the user input magnification is less than the switching magnification, a first adjustment coefficient can be determined according to the true zoom magnification, the target zoom magnification, and the switching magnification corresponding to the first image, and the image translation parameter can be determined according to the first adjustment coefficient and the translation amount. This corresponds to the case of camera switching during the process of image magnification, that is, camera switching is performed with the second camera during the process of increasing the zoom magnification of the first camera. It can also be understood as: the user input magnification is continuously increasing, and camera switching is performed when the user input magnification reaches the switching magnification.

[0080] The process of determining the first adjustment coefficient is introduced below. When determining the first adjustment coefficient according to the true zoom magnification, the target zoom magnification, and the switching magnification corresponding to the first image: calculate the difference between the true zoom magnification and the target zoom magnification to obtain a first value, calculate the difference between the switching magnification and the target zoom magnification to obtain a second value, calculate the ratio of the target zoom magnification to the switching magnification to obtain a third value, calculate the ratio of the first value to the second value, and multiply the obtained ratio by the third value to obtain the first adjustment coefficient.

[0081] Among them, by calculating the difference between the true zoom magnification and the target zoom magnification, the change situation of the zoom magnification can be obtained; by calculating the difference between the switching magnification and the target zoom magnification, a first magnification interval can be obtained; by calculating the ratio of the first value to the second value, the change situation of the zoom magnification in the first magnification interval can be obtained, and by multiplying the obtained ratio by the third value, the first adjustment coefficient can be obtained.

[0082] After obtaining the first adjustment coefficient, the image translation parameter corresponding to the first image can be determined according to the product of the first adjustment coefficient and the translation amount. The translation amount includes a first translation value in the image width direction and a second translation value in the image height direction. Accordingly, the first translation parameter can be determined according to the product of the first adjustment coefficient and the first translation value, and the second translation parameter can be determined according to the product of the first adjustment coefficient and the second translation value. That is, the image translation parameter includes a first translation parameter corresponding to the first translation value and a second translation parameter corresponding to the second translation value.

[0083] When the user input magnification is greater than the switching magnification, the second adjustment coefficient can be determined according to the true zoom magnification, the target zoom magnification, the switching magnification, and the starting zoom magnification corresponding to the first image. The image translation parameter is determined according to the second adjustment coefficient and the translation amount. This corresponds to the case of camera switching during the process of image reduction, that is, camera switching is performed with the second camera during the process of reducing the zoom magnification of the first camera. It can also be understood that the user input magnification continuously decreases, and camera switching is performed when the user input magnification reaches the switching magnification.

[0084] The process of determining the second adjustment coefficient is introduced below. When determining the second adjustment coefficient according to the true zoom magnification, the target zoom magnification, the switching magnification, and the starting zoom magnification corresponding to the first image: calculate the difference between the true zoom magnification and the target zoom magnification to obtain a fourth value, calculate the difference between the target zoom magnification and the starting zoom magnification to obtain a fifth value, calculate the ratio of the target zoom magnification to the switching magnification to obtain a sixth value, calculate the ratio of the fourth value to the fifth value, and multiply the obtained ratio by the sixth value to obtain the second adjustment coefficient.

[0085] Among them, by calculating the difference between the true zoom magnification and the target zoom magnification, the change situation of the zoom magnification can be obtained; by calculating the difference between the target zoom magnification and the starting zoom magnification, the second magnification range can be obtained; by calculating the ratio of the fourth value to the fifth value, the change situation of the zoom magnification in the second magnification range can be obtained, and the second adjustment coefficient can be obtained by multiplying the obtained ratio by the sixth value.

[0086] After obtaining the second adjustment coefficient, the image translation parameter corresponding to the first image can be determined according to the product of the second adjustment coefficient and the translation amount. The translation amount includes a first translation value in the image width direction and a second translation value in the image height direction. Accordingly, the first translation parameter can be determined according to the product of the second adjustment coefficient and the first translation value, and the second translation parameter can be determined according to the product of the second adjustment coefficient and the second translation value.

[0087] The process of obtaining the image translation parameters corresponding to the first image is introduced above. After obtaining the image translation parameters and image magnification parameters corresponding to the first image, the cropping parameters corresponding to the first image can be determined according to the image magnification parameters, image translation parameters, preset image width, and preset image height corresponding to the first image. Specifically, the following steps can be included:

[0088] Calculate the ratio of the preset image width to the image magnification parameter corresponding to the first image, obtain half of the difference between the preset image width and the obtained ratio, and obtain the first parameter according to the sum of the obtained value and the first translation parameter;

[0089] Calculate the ratio of the preset image height to the image magnification parameter corresponding to the first image, obtain half of the difference between the preset image height and the obtained ratio, and obtain the second parameter according to the sum of the obtained value and the second translation parameter;

[0090] Obtain the third parameter according to the ratio of the preset image width to the image magnification parameter corresponding to the first image, and obtain the fourth parameter according to the ratio of the preset image height to the image magnification parameter corresponding to the first image;

[0091] Determine the cropping parameters according to the first parameter, the second parameter, the third parameter, and the fourth parameter.

[0092] When determining the cropping parameters based on the image magnification parameter, image translation parameter, preset image width, and preset image height, the first parameter can be obtained based on the preset image width, image magnification parameter, and first translation parameter, the second parameter can be obtained based on the preset image height, image magnification parameter, and second translation parameter, the third parameter can be obtained based on the ratio of the preset image width to the image magnification parameter, and the fourth parameter can be obtained based on the ratio of the preset image height to the image magnification parameter.

[0093] When obtaining the first parameter, it is necessary to calculate the ratio of the preset image width w to the image magnification parameter s, then calculate the difference between the preset image width w and the obtained ratio, and determine the first parameter according to the sum of half of the difference and the first translation parameter tx. It is expressed by the formula: (w - w / s) / 2 + tx.

[0094] When obtaining the second parameter, it is necessary to calculate the ratio of the preset image height h to the image magnification parameter s, then calculate the difference between the preset image height h and the obtained ratio, and determine the second parameter according to the sum of half of the difference and the second translation parameter ty. It is expressed by the formula: (h - h / s) / 2 + ty.

[0095] When obtaining the third parameter, it is necessary to calculate the ratio of the preset width w of the image to the image magnification parameter s, which is expressed by the formula: w / s; when obtaining the fourth parameter, it is necessary to calculate the ratio of the preset height h of the image to the image magnification parameter s, which is expressed by the formula: h / s.

[0096] After obtaining the first parameter, the second parameter, the third parameter, and the fourth parameter, a rectangular area can be delimited according to the first parameter, the second parameter, the third parameter, and the fourth parameter, and then cropping processing can be performed to implement the display of the first image that eliminates background parallax and can be smoothly transitioned between the first camera and the second camera.

[0097] The implementation process of the image processing method of the embodiment of the present application is introduced above. According to the above introduction, the basic idea of the image processing method provided by the embodiment of the present application is as follows: 1. Calculate the switching magnification according to the calibration data of the first camera and the second camera, that is, the switching point of the two cameras. 2. Before reaching the switching point, calculate the translation amount of the focus area of the images collected by the two cameras based on the image. Based on the calibration data, the rotation parameter can be determined, so that at the switching point, the focus area has no rotation and translation. 3. Obtain the depth image according to the depth camera, and perform parallax adjustment processing based on the depth image so that the background area approaches the focus area according to the parallax. 4. Use the fusion method to fuse the focus area and the edge area line to reduce the sense of truncation, so as to achieve a smooth transition of the region of interest.

[0098] The following introduces the image processing method provided by the embodiment of the present application through a specific example. See Figure 5 as shown in the following steps:

[0099] Step 501: Calculate the switching magnification according to the calibration data of the first camera and the second camera.

[0100] Step 502: Calculate the translation amount of the focus area of the first image collected by the first camera and the focus area of the second image collected by the second camera at a specified magnification.

[0101] Step 503: When the magnification input by the user is in the switching magnification range, determine the affine transformation relationship between the first image and the second image according to the matching relationship between the first image and the second image.

[0102] Step 504: Rotate the first image according to the affine transformation relationship between the first image and the second image.

[0103] Step 505: Based on the depth image collected by the depth camera and the focus area of the first image, perform parallax adjustment processing on the first image to eliminate the parallax of the first image after the rotation processing.

[0104] Step 506: Obtain an image magnification parameter according to the ratio of the user input magnification corresponding to the first image to the true zoom magnification.

[0105] Step 507: Obtain an image translation parameter corresponding to the first image according to the translation amount, the true zoom magnification corresponding to the first image, the switching magnification, the starting zoom magnification of the switching magnification range, and the specified magnification corresponding to the time when the translation amount is calculated.

[0106] Step 508: Determine a cropping parameter corresponding to the first image according to the image magnification parameter and the image translation parameter corresponding to the first image, the preset width of the image, and the preset height of the image, and crop the first image.

[0107] Step 509: By respectively performing rotation, parallax adjustment, and cropping processing on multiple frames of the first image collected, when the user input magnification reaches the switching magnification, make the first image the same as the second image.

[0108] The above implementation process can achieve displaying an image with background parallax eliminated, reducing the sense of truncation, and improving the smoothness of optical zoom.

[0109] The above is the overall implementation process of the image processing method provided by the embodiment of the present application. The first image, the second image, and the depth image are obtained by collecting images of the same scene through the first camera, the second camera, and the depth camera. When the user input magnification is in the switching magnification range, according to the matching relationship between the first image and the second image, determine the affine transformation relationship between the first image and the second image, and process the first image according to the affine transformation relationship and the depth image to ensure that the first image and the second image are the same when the user input magnification reaches the switching magnification, so as to reduce the influence of parallax, reduce the jump of optical zoom, and improve the user experience of optical zoom.

[0110] Based on the same inventive concept, an embodiment of the present application provides an image processing apparatus. Refer to Figure 6 , Figure 6 is a schematic diagram of an image processing apparatus proposed by an embodiment of the present application. The image processing apparatus 600 includes:

[0111] An obtaining module 601, configured to obtain a first image corresponding to a first camera, a second image corresponding to a second camera, and a depth image corresponding to a depth camera; the first image, the second image, and the depth image correspond to the same scene;

[0112] A determining module 602, configured to determine an affine transformation relationship between the first image and the second image according to the matching relationship between the first image and the second image when the user input magnification is in the switching magnification range;

[0113] The calculation module 603 is configured to calculate a switching magnification according to the calibration data of the first camera and the second camera;

[0114] The processing module 604 is configured to process the first image according to the affine transformation relationship and the depth image, so that when the user input magnification reaches the switching magnification, the first image is the same as the second image.

[0115] Optionally, the processing module includes:

[0116] The first processing sub-module is configured to perform a rotation process on the first image according to the affine transformation relationship;

[0117] The second processing sub-module is configured to perform a parallax adjustment process on the first image based on the depth image and the focus area of the obtained first image, so as to eliminate the parallax of the first image after the rotation process;

[0118] The third processing sub-module is configured to perform a cropping process on the first image according to the affine transformation relationship, so as to obtain the first image after rotation, parallax adjustment and cropping.

[0119] Optionally, the second processing sub-module includes:

[0120] The first acquisition unit is configured to acquire the depth value of the first image based on the depth image and convert it into a parallax value;

[0121] The second acquisition unit is configured to perform image segmentation on the first image after rotation processing according to the parallax value, and acquire a mask map corresponding to the focus area and the background area of the first image;

[0122] The adjustment unit is configured to perform a parallax adjustment process on the first image according to the mask map.

[0123] Optionally, the first processing sub-module is further configured to:

[0124] Calculate the product of the image rotation parameter corresponding to the first image and the coordinates of each pixel point corresponding to the first image, so as to perform a rotation process on the first image;

[0125] Wherein, the image rotation parameter is associated with the image rotation angle of the first camera relative to the second camera.

[0126] Optionally, the third processing sub-module includes:

[0127] The third acquisition unit is configured to acquire an image magnification parameter according to the ratio of the user input magnification corresponding to the first image to the true zoom magnification;

[0128] A first processing unit, configured to obtain a translation amount between the focus area corresponding to the first camera and the focus area corresponding to the second camera, and obtain an image translation parameter corresponding to the first image according to the translation amount, the true zoom ratio corresponding to the first image, the switching ratio, the starting zoom ratio of the switching ratio range, and the target zoom ratio corresponding to the time when the translation amount is calculated;

[0129] A second processing unit, configured to determine a cropping parameter corresponding to the first image according to the image magnification parameter, the image translation parameter, the preset image width, and the preset image height corresponding to the first image, and crop the first image.

[0130] Optionally, the first processing unit is further configured to:

[0131] When the user input ratio is less than the switching ratio, determine a first adjustment coefficient according to the true zoom ratio, the target zoom ratio, and the switching ratio corresponding to the first image, and determine the image translation parameter corresponding to the first image according to the product of the first adjustment coefficient and the translation amount;

[0132] When the user input ratio is greater than the switching ratio, determine a second adjustment coefficient according to the true zoom ratio, the target zoom ratio, the switching ratio, and the starting zoom ratio corresponding to the first image, and determine the image translation parameter corresponding to the first image according to the product of the second adjustment coefficient and the translation amount;

[0133] Wherein, the translation amount includes a first translation value in the image width direction and a second translation value in the image height direction, and the image translation parameter includes a first translation parameter corresponding to the first translation value and a second translation parameter corresponding to the second translation value.

[0134] Optionally, the second processing unit includes:

[0135] A first processing subunit, configured to calculate a ratio of the preset image width to the image magnification parameter corresponding to the first image, obtain one half of the difference between the preset image width and the obtained ratio, and obtain a first parameter according to the sum of the obtained value and the first translation parameter;

[0136] A second processing subunit, configured to calculate a ratio of the preset image height to the image magnification parameter corresponding to the first image, obtain one half of the difference between the preset image height and the obtained ratio, and obtain a second parameter according to the sum of the obtained value and the second translation parameter;

[0137] A third processing subunit, configured to obtain a third parameter according to a ratio between the preset image width and the image magnification parameter corresponding to the first image, and obtain a fourth parameter according to a ratio between the preset image height and the image magnification parameter corresponding to the first image;

[0138] A fourth processing subunit, configured to determine the cropping parameter according to the first parameter, the second parameter, the third parameter, and the fourth parameter.

[0139] Based on the same inventive concept, another embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps in the image processing method described in any one of the above embodiments of the present application are implemented.

[0140] Based on the same inventive concept, another embodiment of the present application provides an electronic device, including a processor, a memory, and a program or instruction stored on the memory and executable on the processor, and when the program or instruction is executed by the processor, the steps in the image processing method described in any one of the above embodiments of the present application are implemented.

[0141] Based on the same inventive concept, another embodiment of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps in the image processing method described in any one of the above embodiments of the present application are implemented.

[0142] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for the related parts, reference may be made to the partial description of the method embodiment.

[0143] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments may be referred to each other.

[0144] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a device, or a computer program product. Therefore, the embodiments of the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0145] Embodiments of the present application are described with reference to the flowcharts and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate a device for implementing the specified functions in the process Figure 1 frame process or multi-frame process and / or block Figure 1 frame block or multi-frame block.

[0146] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the specified functions in the process Figure 1 frame process or multi-frame process and / or block Figure 1 frame block or multi-frame block.

[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable terminal device provide steps for implementing the specified functions in the process Figure 1 frame process or multi-frame process and / or block Figure 1 frame block or multi-frame block.

[0148] Although the preferred embodiments of the embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the appended claims are intended to be construed as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

[0149] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "including one entity..." does not exclude the existence of additional identical elements in the process, method, article or terminal device including the said element.

[0150] The above has introduced in detail an image processing method, device and storage medium provided by the present application. Specific examples are used in this text to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. An image processing method, characterized in that, The method includes: Obtaining a first image corresponding to a first camera, a second image corresponding to a second camera, and a depth image corresponding to a depth camera; the first image, the second image, and the depth image correspond to the same scene; When the user input magnification is within a switching magnification range, determining an affine transformation relationship between the first image and the second image according to the matching relationship between the first image and the second image; Calculating a switching magnification according to the calibration data of the first camera and the second camera; Processing the first image according to the affine transformation relationship and the depth image, so that when the user input magnification reaches the switching magnification, the first image is the same as the second image; The processing the first image according to the affine transformation relationship and the depth image includes: Performing a rotation process on the first image according to the affine transformation relationship; Based on the depth image and the focus area of the obtained first image, performing a parallax adjustment process on the first image to eliminate the parallax of the first image after the rotation process; Performing a cropping process on the first image according to the affine transformation relationship to obtain the first image after rotation, parallax adjustment, and cropping; The performing a parallax adjustment process on the first image based on the depth image and the focus area of the obtained first image includes: Based on the depth image, obtaining the depth value of the first image and converting it into a parallax value; Performing image segmentation on the first image after the rotation process according to the parallax value to obtain a mask map corresponding to the focus area and the background area of the first image; Performing a parallax adjustment process on the first image according to the mask map.

2. The image processing method according to claim 1, wherein The performing a rotation process on the first image according to the affine transformation relationship includes: Calculating the product of the image rotation parameter corresponding to the first image and the coordinates of each pixel point corresponding to the first image to perform a rotation process on the first image; Wherein, the image rotation parameter is associated with the image rotation angle of the first camera relative to the second camera.

3. The image processing method according to claim 1, characterized in that, The performing a cropping process on the first image according to the affine transformation relationship includes: Obtaining an image magnification parameter according to the ratio of the user input magnification corresponding to the first image to the true zoom magnification; Obtaining the translation amount between the focus area corresponding to the first camera and the focus area corresponding to the second camera, and obtaining the image translation parameter corresponding to the first image according to the translation amount, the true zoom magnification corresponding to the first image, the switching magnification, the starting zoom magnification of the switching magnification range, and the target zoom magnification corresponding to the time when the translation amount is calculated; Determining the cropping parameter corresponding to the first image according to the image magnification parameter, the image translation parameter, the preset image width, and the preset image height corresponding to the first image, and cropping the first image.

4. The image processing method according to claim 3, wherein Obtaining the image translation parameter corresponding to the first image according to the translation amount, the true zoom ratio corresponding to the first image, the switching ratio, the starting zoom ratio of the switching ratio range, and the target zoom ratio corresponding to the time when the translation amount is calculated, includes: In the case where the user input ratio is less than the switching ratio, determining a first adjustment coefficient according to the true zoom ratio corresponding to the first image, the target zoom ratio, and the switching ratio, and determining the image translation parameter corresponding to the first image according to the product of the first adjustment coefficient and the translation amount; In the case where the user input ratio is greater than the switching ratio, determining a second adjustment coefficient according to the true zoom ratio corresponding to the first image, the target zoom ratio, the switching ratio, and the starting zoom ratio, and determining the image translation parameter corresponding to the first image according to the product of the second adjustment coefficient and the translation amount; Wherein, the translation amount includes a first translation value in the image width direction and a second translation value in the image height direction, and the image translation parameter includes a first translation parameter corresponding to the first translation value and a second translation parameter corresponding to the second translation value.

5. The image processing method according to claim 4, characterized in that, Determining the cropping parameter corresponding to the first image according to the image magnification parameter and the image translation parameter corresponding to the first image, the preset image width, and the preset image height, includes: Calculating the ratio of the preset image width to the image magnification parameter corresponding to the first image, obtaining one half of the difference between the preset image width and the obtained ratio, and obtaining a first parameter according to the sum of the obtained value and the first translation parameter; Calculating the ratio of the preset image height to the image magnification parameter corresponding to the first image, obtaining one half of the difference between the preset image height and the obtained ratio, and obtaining a second parameter according to the sum of the obtained value and the second translation parameter; Obtaining a third parameter according to the ratio of the preset image width to the image magnification parameter corresponding to the first image, and obtaining a fourth parameter according to the ratio of the preset image height to the image magnification parameter corresponding to the first image; Determining the cropping parameter according to the first parameter, the second parameter, the third parameter, and the fourth parameter.

6. An electronic device, characterized in that, Including a processor, a memory, and a program or instruction stored on the memory and executable on the processor, the program or instruction, when executed by the processor, implements the steps of the image processing method according to any one of claims 1 to 5.

7. A readable storage medium, characterized in that, The program or instruction is stored on the readable storage medium, and the program or instruction, when executed by the processor, implements the steps of the image processing method according to any one of claims 1 to 5.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method according to any one of claims 1 to 5.

Images