Image processing method and device based on multiple cameras, and computer-readable storage medium

Abstract

An image processing method and device based on multiple cameras, and a computer-readable storage medium, wherein the multiple cameras include a wide-angle camera and a telephoto camera, and the method includes: when focusing is completed, determining the requested zoom ratio and obtaining the current image to be shot; when the requested zoom ratio is in the wide-angle section, obtaining an initial transformation matrix, wherein the initial transformation matrix is ​​used to represent the mapping relationship between the initial wide-angle image and the initial telephoto image, wherein the initial wide-angle image is an image shot by the wide-angle lens at a calibrated distance, and the initial telephoto image is an image shot by the telephoto lens at a calibrated distance; determining the current shooting distance; adjusting the initial transformation matrix according to the current shooting distance and the requested zoom ratio to obtain the current transformation matrix, and processing the image to be shot according to the current transformation matrix. There is no obvious visual jump at the zoom switching point of different cameras, thereby improving the user experience.

Description

Technical Field

[0001] The present invention relates to the field of camera technology, and in particular to an image processing method and device based on multiple cameras, and a computer-readable storage medium. Background Art

[0002] Camera technology is becoming increasingly important in smart terminals (such as mobile phones, computers, etc.). Small camera size and large zoom range are two important features of current terminal cameras. Although optical zoom using optical lenses can obtain high-quality zoom images, it will inevitably increase the size of the camera and greatly increase the cost. Although ordinary single-camera digital zoom can ensure the size of the camera and control the cost at a low level, the image quality after zooming is not satisfactory.

[0003] Therefore, the technology of using multiple cameras with different focal lengths to simulate optical zoom has emerged. At present, there is a simple and low-cost zoom method, that is, in different zoom ranges, simply switch the camera directly. For example, in the wide-angle focal length, the image taken by the wide-angle camera is used for digital zoom, and in the telephoto focal length, the image taken by the telephoto camera is used for digital zoom. However, this zoom method will produce visual jumps at the zoom switching points of different cameras, which reduces the user experience. Summary of the invention

[0004] The technical problem solved by the present invention is how to achieve optical zoom based on multiple cameras without obvious visual jumps at the zoom switching points, thereby improving user experience.

[0005] To solve the above problems, an embodiment of the present invention provides an image processing method based on multiple cameras, wherein the multiple cameras include a wide-angle camera and a telephoto camera, and the method includes: when focusing is completed, determining the requested zoom ratio and obtaining the current image to be captured; when the requested zoom ratio is in the wide-angle segment, obtaining an initial transformation matrix, wherein the initial transformation matrix is ​​used to represent the mapping relationship between an initial wide-angle image and an initial telephoto image, wherein the initial wide-angle image is an image captured by the wide-angle lens at a calibrated distance, and the initial telephoto image is an image captured by the telephoto lens at a calibrated distance; determining the current shooting distance; adjusting the initial transformation matrix according to the current shooting distance and the requested zoom ratio to obtain a current transformation matrix, and processing the image to be captured according to the current transformation matrix.

[0006] Optionally, the adjusting the initial transformation matrix according to the current shooting distance and the requested zoom factor to obtain the current transformation matrix includes: determining the adjustment transformation matrix corresponding to the current shooting distance according to the correspondence between the calibrated distance and the initial transformation matrix; adjusting the adjustment transformation matrix according to each zoom factor of the wide-angle segment to obtain a target transformation matrix corresponding to each zoom factor; and obtaining the target transformation matrix corresponding to the requested zoom factor as the current transformation matrix.

[0007] Optionally, the initial transformation matrix and the current transformation matrix are selected from an affine transformation matrix and a perspective transformation matrix.

[0008] Optionally, the method further includes: when refocusing is detected and the change amount of the voice coil motor exceeds a threshold, re-determining the current shooting distance.

[0009] Optionally, determining the current shooting distance includes: inputting the image to be shot into a trained depth estimation model to obtain the relative depths of different objects in the image to be shot; and determining the current shooting distance according to the relative depths of different objects in the image to be shot.

[0010] Optionally, determining the current shooting distance includes: determining the current shooting distance by an active ranging device.

[0011] Optionally, the method also includes: obtaining an initial brightness parameter, wherein the initial brightness parameter is used to represent the brightness mapping relationship from the wide-angle image to the telephoto image at the calibration distance; after determining the requested zoom factor, it also includes: when the requested zoom factor is in the wide-angle segment, adjusting the initial brightness parameter according to the requested zoom factor to obtain a current brightness parameter, and processing the image to be captured using the current brightness parameter.

[0012] Optionally, the adjusting the initial brightness parameter according to the requested zoom factor to obtain the current brightness parameter includes: adjusting the initial brightness parameter according to each zoom factor of the wide-angle segment to obtain a target brightness parameter corresponding to each zoom factor; and obtaining the target brightness parameter corresponding to the requested zoom factor as the current brightness parameter.

[0013] Optionally, the method further includes: when the requested zoom ratio is in the telephoto range, processing the image to be captured according to a preset zoom method.

[0014] Optionally, the multi-camera also includes an ultra-wide-angle camera, and after obtaining the current image to be captured, it also includes: when the requested zoom factor is in the ultra-wide-angle segment, obtaining an original transformation matrix, the original transformation matrix is ​​used to represent the mapping relationship between an initial ultra-wide-angle image and an initial wide-angle image, the initial ultra-wide-angle image is an image captured by the ultra-wide-angle lens at a calibrated distance, and the initial wide-angle image is an image captured by the wide-angle lens at a calibrated distance; determining a current shooting distance; adjusting the original transformation matrix according to the current shooting distance and the requested zoom factor to obtain a current transformation matrix, and processing the image to be captured according to the current transformation matrix.

[0015] An embodiment of the present invention also provides an image processing device based on multiple cameras, including: an initial transformation matrix acquisition module, used to acquire an initial transformation matrix, wherein the initial transformation matrix is ​​used to represent a mapping relationship between an initial wide-angle image and an initial telephoto image, wherein the initial wide-angle image is an image captured by the wide-angle lens at a calibrated distance, and the initial telephoto image is an image captured by the telephoto lens at a calibrated distance; a requested zoom ratio acquisition module, used to determine a requested zoom ratio and acquire a current image to be captured; a shooting distance determination module, used to determine a current shooting distance; and a processing module, used to adjust the initial transformation matrix according to the current shooting distance and the requested zoom ratio to obtain a current transformation matrix when the requested zoom ratio is in the wide-angle segment, and to process the image to be captured according to the current transformation matrix.

[0016] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, executes any step of the multi-camera-based image processing method.

[0017] An embodiment of the present invention also provides an image processing device based on multiple cameras, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor runs the computer program, the processor executes any step of the image processing method based on multiple cameras.

[0018] Compared with the prior art, the technical solution of the embodiment of the present application has the following beneficial effects:

[0019] In the multi-camera image processing method of the embodiment of the present invention, the image processing algorithm is used to accurately simulate the optical zoom process at different shooting distances in view of the image differences between the wide-angle lens and the telephoto lens. When zooming in a continuous focal length, the captured image will not undergo obvious geometric jumps, thereby providing users with a better zoom experience. Furthermore, when shooting, the terminal can only turn on a single camera to achieve a better zoom effect.

[0020] Furthermore, based on the initial transformation matrix at the calibration distance, the adjustment transformation matrix corresponding to the minimum zoom factor of the wide-angle segment switching to the minimum zoom factor of the telephoto segment at the current shooting distance can be calculated, and the transformation matrix corresponding to each zoom factor of the wide-angle segment can be obtained based on the adjustment transformation matrix, and the corresponding transformation matrix can be selected as the current transformation matrix according to the requested zoom factor. Thus, when the user changes the requested zoom factor, the corresponding transformation matrix can be changed step by step to achieve smooth switching of each zoom factor of the wide-angle segment.

[0021] Furthermore, the depth of the image to be shot can be simulated based on the offline trained depth estimation model (based on lightweight neural network models such as mobilenet, resent18, and resnet34 as the backbone), so as to estimate the current shooting distance based on the image to be shot. Using lightweight neural networks for depth estimation can obtain real-time shooting distance, avoiding the disadvantage of traditional methods that cannot obtain depth. Subsequently, zoom processing can be performed based on the current shooting distance to achieve smooth zoom.

[0022] Furthermore, the current brightness parameter corresponding to the requested zoom factor can be obtained through the initial brightness parameter obtained offline, so as to achieve different brightness adjustments for different zoom factors, so as to achieve a smooth transition of brightness during zooming. Thus, when the user changes the requested zoom factor, the corresponding brightness parameter can be changed step by step, so as to achieve a smooth transition of brightness at each zoom factor in the wide-angle segment. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of a flow chart of an image processing method based on multiple cameras according to an embodiment of the present invention;

[0024] Figure 2 for Figure 1 The flowchart of step S104 in a specific embodiment;

[0025] Figure 3 A flowchart of a photographing step according to an embodiment of the present invention;

[0026] Figure 4 A schematic diagram of a flow chart of another image processing method based on multiple cameras according to an embodiment of the present invention;

[0027] Figure 5 The present invention is a schematic diagram of the structure of an image processing device based on multiple cameras according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] With the development of smart phones, mobile phone cameras are becoming more and more important in users' mobile phones, and have gradually developed from single cameras to dual cameras, triple cameras and even more cameras, achieving a comparable effect to that of SLR cameras. In the early days, if single-camera mobile phones wanted to achieve the zoom function, they could only do so through digital zoom. When the digital zoom is magnified to a certain multiple, the image will become very blurry and the image quality will not be satisfactory. If multiple optical lenses are used to achieve optical zoom, in addition to greatly increasing the cost, it will inevitably lead to an increase in the thickness of the mobile phone.

[0029] In order to avoid excessive use of space by the lens, current mainstream mobile phones usually use wide-angle lenses and telephoto lenses (or add ultra-wide-angle lenses with smaller focal lengths. For the convenience of explaining the present invention, only wide-angle and telephoto lenses are considered below, and the methods for more cameras are similar) to achieve multi-focal-length optical zoom functions. The focal length is divided into wide-angle and telephoto sections. By switching, the zoom range is expanded. When the focal length is in the wide-angle section, the image collected by the wide-angle lens is mainly used. When the focal length is in the telephoto section, the image collected by the telephoto lens is mainly used. Due to the hardware differences between the wide-angle and telephoto lenses themselves (such as focal length, field of view, distortion, etc.) and installation process problems, the images obtained by the wide-angle lens and the telephoto lens when shooting the same scene object inevitably have geometric deviations, such as zoom multiples, rotation and translation, etc. When switching between the two focal lengths, if there is no good image processing algorithm, it is inevitable that obvious visual jumps will occur.

[0030] At present, the more common multi-camera combinations are: a. A wide-angle camera and a telephoto camera combined with dual focal length zoom; b. Adding an ultra-wide-angle camera with a shorter focal length on the basis of a to achieve three-focal length zoom; c. Adding a camera with a longer focal length on the basis of a (considering the size, it can be made into a foldable type) to achieve three-focal length zoom; d. Other multi-camera combinations with different focal lengths. Taking combination a as an example, due to the different hardware parameters of different cameras themselves (such as optical center, focal length, field of view (FOV), distortion and other internal parameters), and different installation arrangements in the module (such as the baseline, relative angle, position, etc. of the arrangement), the wide-angle and telephoto cameras on the same module will inevitably obtain images with different FOVs, different relative positions, different occlusions, and different distortions when shooting the same object.

[0031] Traditionally, a more complex zoom method is provided, which uses image processing technology to achieve the fusion of wide-angle and telephoto images, that is, in the wide-angle focal length, the wide-angle and telephoto images are fused to improve the image quality during zooming, while in the telephoto focal length, the telephoto image is still used for digital zooming. This zoom method has almost no visual jump at the zoom switching point of the two cameras, and it is very natural at any zoom multiple, and it also improves the image clarity during zooming. However, this method requires turning on the wide-angle and telephoto cameras at the same time to facilitate the simultaneous acquisition of wide-angle and telephoto images for image processing, which increases the power consumption of the mobile phone. In addition, image fusion technology often takes a long time and is not convenient for preview and video modes.

[0032] As a compromise between the above two solutions, a third method is given, which is to use camera calibration to obtain certain geometric transformation information, and use this transformation information during the zoom process to achieve a smooth transition of the switching point. In recent years, deep learning methods based on neural networks have continued to develop, adding all aspects of image processing, and have also been well applied in optical zoom. For example, a mobile phone currently uses the main camera for digital zoom at 3x zoom and below, and uses super-resolution technology to improve the quality of zoom images; when zooming at 5x and above, the tele camera is used; when shooting with a zoom between 3 and 5x, the image data of the main camera and the tele camera are combined to generate an image. The best detail image data in the center of the picture comes from the 5x telephoto lens, and the "lost" image area at the edge is filled by the image data of the main camera, and then retouched using artificial intelligence to improve the details of fine patterns and textures. Regarding the direct switching method of wide-angle and telephoto cameras mentioned above, although the calculation is simple, the cost is low, and it is controllable in real time, it will produce obvious visual jumps when switching between zooms, giving users an unnatural experience; and the image fusion method requires the wide-angle and telephoto cameras to be turned on at the same time to obtain wide-angle and telephoto image information for image processing at the same time, which increases the power consumption of the mobile phone. In addition, image fusion technology often takes a long time and is not convenient for preview and video modes; the method of adding artificial intelligence has good effects, but it is often highly complex in calculation and has high hardware requirements. It is difficult to implement in real time on low-cost hardware, which increases the cost; the pre-calibration method is simple and can obtain certain geometric transformation information, but the transformation matrix obtained based on a certain close-range calibration cannot be applied to all shooting distances, especially the translation information. The translation amount is large at close distances, small at long distances, and almost no translation at infinity. In addition, in the smooth switching of preview and video modes, the 3A between multiple cameras is first required to be as synchronized as possible. Therefore, there is rarely a situation where the 3A inconsistency is considered during the smooth switching process. In fact, color and brightness jumps visible to the naked eye often exist to a certain extent.

[0033] In summary, the existing zoom method will produce visual jumps at the zoom switching points of different cameras, reducing the user experience.

[0034] To solve the above problems, an embodiment of the present invention provides an image processing method based on multiple cameras, wherein the multiple cameras include a wide-angle camera and a telephoto camera, and the method includes: when focusing is completed, determining the requested zoom ratio and obtaining the current image to be shot; when the requested zoom ratio is in the wide-angle segment, obtaining an initial transformation matrix, wherein the initial transformation matrix is ​​used to represent the mapping relationship between the initial wide-angle image and the initial telephoto image, wherein the initial wide-angle image is an image taken by the wide-angle lens at a calibrated distance, and the initial telephoto image is an image taken by the telephoto lens at a calibrated distance; determining the current shooting distance; adjusting the initial transformation matrix according to the current shooting distance and the requested zoom ratio to obtain the current transformation matrix, and processing the image to be shot according to the current transformation matrix. As a result, there is no obvious visual jump at the zoom switching point of different cameras, thereby improving the user experience.

[0035] In order to make the above-mentioned objects, features and beneficial effects of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0036] See also Figure 1 , Figure 1 A flowchart of an image processing method based on multiple cameras is provided for an embodiment of the present invention. The multiple cameras include one or more wide-angle cameras and one or more telephoto cameras. The method can be executed by a terminal capable of shooting images or videos, the terminal includes the multiple cameras, and the terminal can be a mobile phone, a computer, a tablet computer, a smart watch, etc.

[0037] First, the wide-angle camera and the telephoto camera are installed on the same electronic device, and objects in the same scene are photographed to obtain wide-angle images and telephoto images respectively. Since the wide-angle image and the telephoto image are processed by different Image Signal Processing (ISP) units, the input images collected by the wide-angle camera and the telephoto camera are affected by different sharpness, contrast, color, brightness, and focus distances. It is necessary to implement 3A synchronization configuration for the two cameras (i.e., the wide-angle camera and the telephoto camera), where 3A includes automatic exposure (AE), automatic focus (AF), and automatic white balance (AWB). When executing Figure 1Before the method is described, it is necessary to process the different distortion parameters between different cameras (here referring to the wide-angle camera and the telephoto camera), that is, to perform distortion correction. After the processing is completed, the method of the embodiment of the present invention can be performed. Among them, a wide-angle lens refers to a lens with a larger field of view and a smaller focal length, which can usually shoot scenes with a wider field of view. A telephoto lens refers to a lens with a smaller field of view and a larger focal length, which can usually shoot scenes with a smaller field of view but richer details. Zoom: Change the focal length and field of view angle to enlarge or reduce the photographed scene to achieve the purpose of shooting scenes at different distances. Optical zoom refers to changing the focal length and field of view angle by changing the distance between optical lenses to achieve the purpose of zooming. Monocular depth estimation refers to the estimation of a depth map based on a single frame image. The estimated depth can reflect the depth of the photographed object, or is called the current shooting distance.

[0038] like Figure 1 The method includes the following steps S101 to S104, which are described in detail as follows:

[0039] Step S101, when focusing is completed, determining the requested zoom factor and acquiring the current image to be captured.

[0040] Here, completing the focus means that the camera on the terminal completes the focus, and the camera includes one or more wide-angle cameras and one or more telephoto cameras.

[0041] The requested zoom factor is a zoom factor set by a user, for example, the requested zoom factor may be 1.0x (x), 1.1x, 2.0x, and so on.

[0042] Optionally, the zoom factor of the wide-angle segment and the zoom factor of the telephoto segment can be preset. For example, the zoom factors of the wide-angle segment include 1.0x, 1, 1x, ..., 1.9x. The zoom factors of the telephoto segment include zoom factors of 2.0x or more. At this time, 2.0x is the zoom switching point from the wide-angle segment to the telephoto segment. Therefore, it can be determined whether the requested zoom factor is in the wide-angle segment or the telephoto segment based on the preset zoom factors of the wide-angle segment and the telephoto segment.

[0043] After determining the requested zoom factor, the corresponding camera is turned on, such as a wide-angle camera below 2.0x, and the current image to be shot at the requested zoom factor is collected through the camera.

[0044] The image to be captured is the current camera preview or the image to be captured by the camera focus. The image can be a single frame image in the video stream. For other frame images in the video stream, refer to the processing method of the single frame image (i.e., the image to be captured).

[0045] Step S102, when the requested zoom ratio is in the wide-angle segment, an initial transformation matrix (denoted as H0) is obtained, where the initial transformation matrix is ​​used to represent a mapping relationship between an initial wide-angle image and an initial telephoto image, where the initial wide-angle image is an image taken by the wide-angle lens at a calibrated distance, and the initial telephoto image is an image taken by the telephoto lens at a calibrated distance.

[0046] Optionally, the initial transformation matrix may be generated offline in advance and stored in the terminal after being generated. When the terminal determines in step S102 that the requested zoom factor is in the wide-angle range, the stored initial transformation matrix H0 is read.

[0047] In a specific embodiment, the initial transformation matrix can be obtained by calibrating a multi-camera camera. As follows, a method for obtaining an initial transformation matrix is ​​provided, and the method includes the following steps A to D, which are described in detail as follows.

[0048] Step A, calibrate the internal and external parameters of the wide-angle lens and the telephoto lens. The internal parameters include the camera's principal point, focal length, scale factor, etc., and the external parameters refer to the rotation factor and translation factor from the wide-angle lens to the telephoto lens, etc. Among them, the camera's principal point can be expressed as a coordinate point (cx, cy), the x-axis and y-axis are the camera's two-dimensional pixel coordinate axes, and fx and fy represent the camera's scale factors on the x-axis and y-axis, respectively.

[0049] Step B, prepare a plane (or two-dimensional) standard chessboard. The size of the standard chessboard should not be too small or too large, and it is best to be able to fill the entire field of view when the telephoto lens is opened. Generally speaking, the distance between the telephoto lens and the standard chessboard is 50 to 80 cm, which is the calibration distance, denoted as D0.

[0050] Step C: capture a wide-angle image at a calibrated distance D0, which is recorded as an initial wide-angle image; capture a telephoto image at a calibrated distance D0, which is recorded as an initial telephoto image.

[0051] Step D: Through corner detection, the transformation matrix H0 of the wide-angle image mapped to the telephoto image at the calibration distance D0 is analyzed and obtained, and the matrix is ​​also the initial transformation matrix. The initial transformation matrix can be a matrix including information such as scale, rotation and translation.

[0052] Optionally, the initial transformation matrix H0 may be an affine transformation matrix, a perspective transformation matrix, or the like.

[0053] Step S103, determining the current shooting distance.

[0054] The current shooting distance (denoted as D1) may also be referred to as the current shooting depth, which may be obtained by analyzing the image to be shot, or by measuring the shooting distance.

[0055] Step S104: adjusting the initial transformation matrix according to the current shooting distance and the requested zoom factor to obtain a current transformation matrix, and processing the image to be shot according to the current transformation matrix.

[0056] Optionally, during camera calibration, the shooting distance may be transformed from the calibration distance D0 to multiple other distances, thereby fitting the transformation matrices corresponding to each other distance, and storing the relationship between the transformation matrices corresponding to the multiple shooting distances and the initial transformation matrix H0 in the terminal. After determining the current shooting distance D1, the terminal may obtain the current transformation matrix, denoted as HA, based on the stored H0 and the relationship between the transformation matrices corresponding to the multiple shooting distances and the initial transformation matrix H0. Furthermore, the transformation matrices corresponding to the multiple shooting distances may be obtained based on the calculation process of obtaining the initial transformation matrix H0 at the calibration distance D0.

[0057] Optionally, the processing of the image to be captured according to the current transformation matrix includes: performing deformation processing (warping) on ​​the image to be captured according to the current transformation matrix HA, which can also be called performing image affine transformation or image distortion processing on the image to be captured, thereby obtaining a processed image.

[0058] Figure 1 In the method, the image difference between wide-angle lens and telephoto lens is considered, and the image processing algorithm is used to accurately simulate the optical zoom process at different shooting distances, and the image within and between consecutive focal lengths will not have obvious geometric jumps, thus bringing a better zoom experience to users. Figure 1 With the method in , when shooting, the terminal can only open a single camera to achieve a better zoom effect.

[0059] In one embodiment, Figure 1 After determining the requested zoom ratio in step S101 and acquiring the current image to be captured, the method may further include: step S105, when the requested zoom ratio is in the telephoto section, processing the image to be captured according to a preset zoom method.

[0060] Optionally, the preset zoom method may use various commonly used digital zoom methods of telephoto cameras. The digital zoom method implements zooming of the image to be captured through an interpolation algorithm, and as the zoom factor increases, the pixel quality of the image to be captured gradually deteriorates.

[0061] Optionally, the image to be captured after being processed in step S104 and step S105 can be saved or displayed. If saved, the capturing of one frame of image is completed.

[0062] In one embodiment, see Figure 2 , Figure 2 for Figure 1 In the flowchart of step S104 in a specific embodiment, the step S104 adjusts the initial transformation matrix according to the current shooting distance and the requested zoom factor to obtain the current transformation matrix, which may include the following steps:

[0063] Step S1041, determining the adjustment transformation matrix (H1) corresponding to the current shooting distance according to the corresponding relationship between the calibration distance (D0) and the initial transformation matrix (H0).

[0064] Optionally, the method in step S104 may be referred to to obtain the adjustment transformation matrix (H1) corresponding to the current shooting distance D1. The adjustment transformation matrix (H1) is a transformation matrix corresponding to switching from the minimum zoom factor of the wide-angle segment to the minimum zoom factor of the telephoto segment, and the transformation matrix may be an affine transformation matrix, a perspective transformation matrix, etc.

[0065] Optionally, the terminal may store the intrinsic parameters, extrinsic parameters, the calibration distance D0 and the initial transformation matrix H0 in the terminal when calibrating the camera, and obtain the adjustment transformation matrix H1 according to the intrinsic parameters, extrinsic parameters and the current shooting distance D1.

[0066] Step S1042: adjusting the adjustment transformation matrix (H1) according to each zoom factor of the wide-angle segment to obtain a target transformation matrix corresponding to each zoom factor.

[0067] The wide-angle segment may include N zoom factors, such as 1.0x, 1, 1x, ..., 1.9x, where N = 10. The adjusting the adjustment transformation matrix may include: dividing the adjustment transformation matrix H1 into N small matrices according to a linear relationship, and each small matrix obtained by dividing is recorded as H1(i), where the value of i is 1, 2, ..., 9, and each zoom factor 1.ix (such as 1.0x~1.9x) corresponds to one H1(i).

[0068] Step S1043: Acquire a target transformation matrix corresponding to the requested zoom factor as the current transformation matrix (HA).

[0069] Get H1(i) corresponding to the requested zoom ratio. For example, if the requested zoom ratio is 1.9x, get H1(9) corresponding to 1.9x.

[0070] In this embodiment, based on the initial transformation matrix at the calibration distance, the adjustment transformation matrix corresponding to the minimum zoom factor of the wide-angle segment switching to the minimum zoom factor of the telephoto segment at the current shooting distance can be calculated, and the transformation matrix corresponding to each zoom factor of the wide-angle segment can be obtained based on the adjustment transformation matrix, and the corresponding transformation matrix can be selected as the current transformation matrix according to the requested zoom factor. Therefore, when the user changes the requested zoom factor, the corresponding transformation matrix can be changed step by step to achieve a geometrically smooth transition of each zoom factor of the wide-angle segment.

[0071] In one embodiment, Figure 1 The determining of the current shooting distance in step S103 may include: the methods of the following examples 1 and 2.

[0072] Example 1: Input the image to be captured into a trained depth estimation model to obtain the relative depths of different objects in the image to be captured; and determine the current shooting distance according to the relative depths of different objects in the image to be captured.

[0073] Optionally, the depth estimation model can be a lightweight neural network model based on Mobilenet, Resent18, Resnet34, etc. as a backbone. The trained depth estimation model can be set in the terminal, or can be communicatively connected to the terminal. The depth estimation model can be obtained based on the sample images used for model training and the relative depth training of each sample image annotated, so that the depth estimation model can obtain the relative depth of different objects in the image to be captured according to the input image to be captured. Among them, the sample image is an image used for model training of the depth estimation model. Optionally, the depth estimation model outputs a depth map or a disparity map of the image to be captured to represent the relative depths of different objects in the image to be captured. The depth estimation model can be for a monocular image, that is, one frame of the image is processed each time.

[0074] Optionally, determining the current shooting distance according to the relative depths of different objects in the image to be captured may include: fitting a parameter R0 according to the relationship between the actual depth and the relative depth of the sample image; calculating the actual depth of the image to be captured according to the parameter R0 according to the relative depths of different objects in the image to be captured, and using the actual depth as the current shooting distance.

[0075] In this embodiment, the depth of the image to be shot can be simulated based on the offline trained depth estimation model, so as to estimate the current shooting distance according to the image to be shot. The use of a lightweight neural network for depth estimation can obtain the real-time shooting distance, avoiding the disadvantage that the traditional method cannot obtain the depth. Subsequently, zoom processing can be performed based on the current shooting distance to achieve smooth zoom.

[0076] Example 2: Determine the current shooting distance using an active distance measuring device.

[0077] Optionally, the active distance measuring device may include a device capable of active distance measuring, such as a laser radar, to measure the current shooting depth as the current shooting distance.

[0078] Comparing Example 1 and Example 2, it can be seen that the solution of Example 1 is based on software implementation, does not require additional hardware equipment on the terminal side, and has lower costs. The solution of Example 2 requires adding active ranging equipment on the terminal side, which may increase the equipment cost of the terminal.

[0079] In one embodiment, the method may further include: when re-focusing is detected and the change amount of the voice coil motor (VCM) exceeds a threshold, re-determining the current shooting distance.

[0080] Among them, the terminal's camera can adjust the position of the lens through VCM to achieve the focusing function to present a clear image.

[0081] exist Figure 1 In the method, for a focus to be completed, it is only necessary to determine the current shooting distance (that is, shooting depth) once. If the focus is no longer adjusted during the zooming process, it means that the shooting depth has not changed, and there is no need to re-determine the current shooting distance. If the focus is re-adjusted at the wide-angle end, the shooting depth is judged to be changed according to the change of VCM. When the change of VCM is within a certain range (that is, it does not exceed the threshold), it is considered that the current shooting depth has not changed or has changed very little, and there is no need to re-determine the current shooting distance, so as to avoid repeated calculations and waste of computing resources. When the change of VCM exceeds the threshold, it is considered that the current shooting depth has changed significantly, and the current shooting distance needs to be re-determined.

[0082] After each new determination of the current shooting distance, you can Figure 1 In step S104, the image to be shot is processed. The method of re-determining the current shooting distance can be referred to the above examples 1 and 2, which will not be described in detail here.

[0083] In one embodiment, Figure 1 In step S101, when focusing is completed, the requested zoom ratio is determined, and after the current image to be captured is obtained, a brightness adjustment method may also be included. The method may include: when the requested zoom ratio is in the wide-angle segment, obtaining an initial brightness parameter, the initial brightness parameter is used to represent the brightness mapping relationship from the wide-angle image to the telephoto image at the calibration distance; adjusting the initial brightness parameter according to the requested zoom ratio to obtain a current brightness parameter, and processing the image to be captured by using the current brightness parameter.

[0084] Optionally, during the calibration process of the multi-camera, the initial wide-angle image and the initial telephoto image may be subjected to brightness analysis respectively, and the brightness mapping of the initial wide-angle image and the initial telephoto image may be obtained and recorded as map0 and saved in the terminal.

[0085] Optionally, the brightness parameters corresponding to different zoom factors can be calculated, and the relationship between the brightness parameters corresponding to different zoom factors and the initial brightness parameters can be determined, so that the initial brightness parameters can be adjusted according to the requested zoom factor to obtain the brightness parameters corresponding to the requested zoom factor, that is, the current brightness parameters, denoted as mapA.

[0086] Optionally, processing the image to be captured by using the current brightness parameter may include: adjusting the brightness of the image to be captured by using the current brightness parameter mapA.

[0087] In a specific embodiment, the initial brightness parameter is adjusted according to the requested zoom factor to obtain the current brightness parameter, including: adjusting the initial brightness parameter according to each zoom factor of the wide-angle segment to obtain the target brightness parameter corresponding to each zoom factor; obtaining the target brightness parameter corresponding to the requested zoom factor as the current brightness parameter.

[0088] The wide-angle segment may include N zoom factors, such as 1.0x, 1, 1x, ..., 1.9x, where N = 10. The adjustment of the initial brightness parameter may include: dividing the initial brightness parameter map0 into N parts according to a linear relationship, each part obtained by the division is recorded as Map0(i), the value of i is 1, 2, ..., 9, and each zoom factor 1.ix (1.0x~1.9x) corresponds to a Map0(i). Obtain the Map0(i) corresponding to the requested zoom factor. For example, if the requested zoom factor is 1.9x, obtain the Map0(9) corresponding to 1.9x.

[0089] In this embodiment, different brightness adjustments can be made for different zoom factors to achieve a smooth transition of brightness during zooming. Thus, when the user changes the requested zoom factor, the corresponding brightness parameters can be changed step by step to achieve a smooth transition of brightness at each zoom factor in the wide-angle segment.

[0090] In one embodiment, when the requested zoom ratio is a zoom switching point from a wide-angle segment to a telephoto segment, an original image captured by a telephoto camera is displayed.

[0091] See also Figure 3 , Figure 3The present invention is a flowchart of a shooting step of an embodiment of the present invention, and the shooting step may include: step S31, 3A synchronization of multiple cameras and distortion correction; step S32, smooth switching processing of the image to be shot, and the processing step is implemented by the various steps of the image processing method based on multiple cameras of the embodiment of the present invention; step S33, displaying or storing the processed image.

[0092] See also Figure 4 , Figure 4 A flowchart of another multi-camera image processing method provided by an embodiment of the present invention, the method implements Figure 3 Step S32 in the embodiment specifically includes the following two steps: offline preprocessing and online processing.

[0093] The offline preprocessing step includes: obtaining the internal and external parameters of the wide-angle lens and the telephoto lens through the camera calibration step, and obtaining H0,map0 under D0 through the initial wide-angle image and the initial telephoto image. The offline preprocessing step also includes: training the depth estimation model based on the sample image and fitting the parameter R0.

[0094] The online processing steps may include: step S321, the camera completes focusing; step S322, determining which focal length (the requested zoom factor) is located; if it is in the wide-angle section, executing steps S323 to S325. If it is in the telephoto section, executing step S326, digital zoom. Step S323, the depth estimation model obtains the current shooting distance; step S324, calculating mapA and H1(A); step S325, performing brightness and deformation processing on the image to be shot.

[0095] Figure 3 and Figure 4 For more information about the working principle and working method, please refer to the above Figure 1 and Figure 2 The related description of the image processing method based on multiple cameras will not be repeated here.

[0096] In the scheme of this embodiment, the transformation matrix between wide angle and telephoto is first calibrated, including scale, rotation and translation parameters, and then the monocular depth estimation based on lightweight neural network is used to provide real-time distance information and obtain the transformation matrix under the real-time shooting distance, which can reduce hardware and computing costs and achieve real-time implementation. At the same time, the brightness inconsistency between multiple cameras is adjusted to avoid obvious "switches" or "jumps" when transitioning from one camera to another, achieving a good visual experience.

[0097] In one embodiment, the multi-camera also includes an ultra-wide-angle camera. After obtaining the current image to be captured, the method may further include: when the requested zoom factor is in the ultra-wide-angle segment, obtaining an original transformation matrix, the original transformation matrix is ​​used to represent a mapping relationship between an initial ultra-wide-angle image and an initial wide-angle image, the initial ultra-wide-angle image is an image captured by the ultra-wide-angle lens at a calibrated distance, and the initial wide-angle image is an image captured by the wide-angle lens at a calibrated distance; determining a current shooting distance; adjusting the original transformation matrix according to the current shooting distance and the requested zoom factor to obtain a current transformation matrix, and processing the image to be captured according to the current transformation matrix.

[0098] Among them, the lens with a particularly wide viewing angle (80-110 degrees) among wide-angle lenses is called an ultra-wide-angle lens.

[0099] The original transformation matrix is ​​obtained based on the initial ultra-wide-angle image and the initial wide-angle image during camera calibration. The method for obtaining the original transformation matrix can refer to the method for obtaining the initial transformation matrix, and the method for obtaining the current transformation matrix can refer to the method for obtaining the current transformation matrix, which will not be repeated here. The original transformation matrix and the current transformation matrix can be selected from an affine transformation matrix, a perspective transformation matrix, and the like.

[0100] For switching between ultra-wide-angle lens and wide-angle lens, please refer to the above Figures 1 to 4 The steps for switching between a medium wide-angle lens and a telephoto lens will not be repeated here.

[0101] See also Figure 5 The embodiment of the present invention further provides an image processing device 40 based on multiple cameras, wherein the multiple cameras include a wide-angle camera and a telephoto camera, and the image processing device 40 based on multiple cameras may include: a zoom ratio determination module 401, used to determine the requested zoom ratio when focusing is completed, and obtain the current image to be shot; an initial transformation matrix acquisition module 402, used to obtain the initial transformation matrix when the requested zoom ratio is in the wide-angle segment, the initial transformation matrix is ​​used to represent the mapping relationship between the initial wide-angle image and the initial telephoto image, the initial wide-angle image is the image captured by the wide-angle lens at a calibrated distance, and the initial telephoto image is the image captured by the telephoto lens at a calibrated distance; a shooting distance determination module 403, used to determine the current shooting distance; a processing module 404, used to adjust the initial transformation matrix according to the current shooting distance and the requested zoom ratio to obtain a current transformation matrix, and process the image to be shot according to the current transformation matrix.

[0102] In one embodiment, the processing module 404 may include: an adjustment transformation matrix determination unit, used to determine the adjustment transformation matrix corresponding to the current shooting distance according to the correspondence between the calibration distance and the initial transformation matrix; a target transformation matrix acquisition unit, used to adjust the adjustment transformation matrix according to each zoom factor of the wide-angle segment to obtain the target transformation matrix corresponding to each zoom factor; and a current transformation matrix acquisition unit, used to obtain the target transformation matrix corresponding to the requested zoom factor as the current transformation matrix.

[0103] In one embodiment, the initial transformation matrix and the current transformation matrix are selected from an affine transformation matrix, a perspective transformation matrix, and the like.

[0104] In one embodiment, the multi-camera based image processing device 40 may further include: a zoom processing module, which is used to re-determine the current shooting distance when re-focusing is detected and the change amount of the voice coil motor exceeds a threshold.

[0105] In one embodiment, the shooting distance determination module 403 may include: a model analysis unit, used to input the image to be shot into a trained depth estimation model to obtain the relative depths of different objects in the image to be shot; and a distance determination unit, used to determine the current shooting distance according to the relative depths of different objects in the image to be shot.

[0106] In one embodiment, the shooting distance determining module 403 may include: determining the current shooting distance through an active distance measuring device.

[0107] In one embodiment, when focusing is completed, the requested zoom ratio is determined, and after the current image to be captured is acquired, the multi-camera based image processing device 40 may further include: an initial brightness parameter acquisition module, used to acquire the initial brightness parameters when the requested zoom ratio is in the wide-angle segment, and the initial brightness parameters are used to represent the brightness mapping relationship from the wide-angle image to the telephoto image at the calibration distance; a brightness adjustment module, used to adjust the initial brightness parameters according to the requested zoom ratio to obtain the current brightness parameters, and process the image to be captured by using the current brightness parameters.

[0108] In one embodiment, the brightness adjustment module may include: a target brightness parameter determination unit, used to adjust the initial brightness parameter according to the zoom factors of the wide-angle segment to obtain the target brightness parameter corresponding to each zoom factor; and a current brightness parameter determination unit, used to obtain the target brightness parameter corresponding to the requested zoom factor as the current brightness parameter.

[0109] In one embodiment, after determining the requested zoom ratio and acquiring the current image to be captured, the multi-camera based image processing device 40 may also include: a telephoto processing module, which is used to process the image to be captured according to a preset zoom method when the requested zoom ratio is in the telephoto segment.

[0110] In one embodiment, the multi-camera further includes an ultra-wide-angle camera. After acquiring the current image to be captured, the multi-camera-based image processing device 40 may further include: an ultra-wide-angle segment calibration module, for acquiring an original transformation matrix when the requested zoom factor is in the ultra-wide-angle segment, the original transformation matrix being used to represent a mapping relationship between an initial ultra-wide-angle image and an initial wide-angle image, the initial ultra-wide-angle image being an image captured by the ultra-wide-angle lens at a calibrated distance, and the initial wide-angle image being an image captured by the wide-angle lens at a calibrated distance; a second distance determination module, for determining a current shooting distance; and an ultra-wide-angle processing module, for adjusting the original transformation matrix according to the current shooting distance and the requested zoom factor to obtain a current transformation matrix, and processing the image to be captured according to the current transformation matrix.

[0111] For more information about the working principle and working method of the multi-camera image processing device 40, please refer to the above Figures 1 to 4 The related description of the image processing method based on multiple cameras will not be repeated here.

[0112] In a specific implementation, the above-mentioned multi-camera based image processing device 40 may correspond to a chip with an image processing function in a terminal, or to a chip with a data processing function, such as a system-on-a-chip (SOC), a radio frequency chip, etc.; or to a chip module in a terminal including a chip with an image processing function; or to a chip module with a chip with a data processing function, or to a terminal.

[0113] Regarding the various modules / units included in the various devices and products described in the above embodiments, they can be software modules / units, or hardware modules / units, or they can be partially software modules / units and partially hardware modules / units. For example, for various devices and products applied to or integrated in a chip, the various modules / units included therein can all be implemented in the form of hardware such as circuits, or at least some of the modules / units can be implemented in the form of software programs, which run on a processor integrated inside the chip, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in a chip module, the various modules / units included therein can all be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (such as a chip, circuit module, etc.) or different components of the chip module, or at least some of the modules / units can be implemented in the form of hardware such as circuits. The element can be implemented in the form of a software program, which runs on a processor integrated inside the chip module, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in the terminal, the various modules / units contained therein can be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (for example, chip, circuit module, etc.) or in different components in the terminal, or, at least some modules / units can be implemented in the form of a software program, which runs on a processor integrated inside the terminal, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits.

[0114] The embodiment of the present invention further provides a storage medium having a computer program stored thereon, wherein the computer program is executed when the processor is running. Figures 1 to 4 Any step of the multi-camera based image processing method. The storage medium may be a computer-readable storage medium, for example, may include a non-volatile memory or a non-transitory memory, and may also include an optical disk, a mechanical hard disk, a solid-state hard disk, etc.

[0115] The embodiment of the present invention further provides an image processing device based on multiple cameras, which may include a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the computer program when running the computer program. Figures 1 to 4 Any step of the multi-camera based image processing method.

[0116] Specifically, in the embodiment of the present invention, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0117] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct rambus RAM (DR RAM).

[0118] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship.

[0119] The "plurality" appearing in the embodiments of the present application refers to two or more.

[0120] The first, second, etc. descriptions appearing in the embodiments of the present application are only used for illustration and distinction of the description objects. There is no order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0121] The "connection" that appears in the embodiments of the present application refers to various connection methods such as direct connection or indirect connection to achieve communication between devices, and the embodiments of the present application do not impose any limitations on this.

[0122] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A multi-camera image processing method, characterized in that: The multi-camera includes a wide-angle camera and a telephoto camera, and the method includes: When focusing is completed, the requested zoom factor is determined, and the current image to be captured is obtained; When the requested zoom ratio is in the wide-angle range, obtaining an initial transformation matrix, wherein the initial transformation matrix is ​​used to represent a mapping relationship between an initial wide-angle image and an initial telephoto image, wherein the initial wide-angle image is an image captured by the wide-angle camera at a calibrated distance, and the initial telephoto image is an image captured by the telephoto camera at a calibrated distance; Determine the current shooting distance; Adjusting the initial transformation matrix according to the current shooting distance and the requested zoom factor to obtain a current transformation matrix, and processing the image to be shot according to the current transformation matrix; The adjusting the initial transformation matrix according to the current shooting distance and the requested zoom factor to obtain the current transformation matrix includes: Determining an adjustment transformation matrix corresponding to the current shooting distance according to a correspondence between the calibration distance and the initial transformation matrix; According to each zoom factor of the wide-angle segment, the adjustment transformation matrix is ​​adjusted to obtain a target transformation matrix corresponding to each zoom factor; A target transformation matrix corresponding to the requested zoom factor is obtained as the current transformation matrix.

2. The method according to claim 1, characterized in that: The initial transformation matrix and the current transformation matrix are selected from an affine transformation matrix and a perspective transformation matrix.

3. The method according to claim 1, characterized in that The method further comprises: When the focus is detected again and the change amount of the voice coil motor exceeds the threshold, the current shooting distance is re-determined.

4. The method according to any one of claims 1 to 3, characterized in that: Determining the current shooting distance includes: Inputting the image to be captured into a trained depth estimation model to obtain relative depths of different objects in the image to be captured; The current shooting distance is determined according to the relative depths of different objects in the image to be shot.

5. The method according to any one of claims 1 to 3, characterized in that: Determining the current shooting distance includes: Determine the current shooting distance through active ranging equipment.

6. The method according to any one of claims 1 to 3, characterized in that: After the focusing is completed, the requested zoom factor is determined, and the current image to be captured is acquired, the method further includes: When the requested zoom ratio is in the wide-angle range, obtaining an initial brightness parameter, where the initial brightness parameter is used to represent a brightness mapping relationship from the wide-angle image to the telephoto image at the calibration distance; The initial brightness parameter is adjusted according to the requested zoom factor to obtain a current brightness parameter, and the image to be captured is processed using the current brightness parameter.

7. The method according to claim 6, characterized in that The adjusting the initial brightness parameter according to the requested zoom factor to obtain a current brightness parameter includes: According to each zoom factor of the wide-angle segment, the initial brightness parameter is adjusted to obtain a target brightness parameter corresponding to each zoom factor; A target brightness parameter corresponding to the requested zoom factor is obtained as the current brightness parameter.

8. The method according to any one of claims 1 to 3, characterized in that: After determining the requested zoom factor and acquiring the current image to be captured, the method further includes: When the requested zoom ratio is in the telephoto range, the image to be captured is processed according to a preset zoom method.

9. The method according to any one of claims 1 to 3, characterized in that: The multi-camera also includes an ultra-wide-angle camera, and after obtaining the current image to be captured, the method further includes: When the requested zoom ratio is in the ultra-wide-angle range, obtaining an original transformation matrix, wherein the original transformation matrix is ​​used to represent a mapping relationship between an initial ultra-wide-angle image and an initial wide-angle image, wherein the initial ultra-wide-angle image is an image captured by the ultra-wide-angle camera at a calibrated distance, and the initial wide-angle image is an image captured by the wide-angle camera at a calibrated distance; Determine the current shooting distance; The original transformation matrix is ​​adjusted according to the current shooting distance and the requested zoom factor to obtain a current transformation matrix, and the image to be shot is processed according to the current transformation matrix.

10. An image processing device based on multiple cameras, characterized in that: The multi-camera includes a wide-angle camera and a telephoto camera, and the device includes: A zoom ratio determination module, used to determine the requested zoom ratio when focusing is completed, and obtain the current image to be captured; an initial transformation matrix acquisition module, used for acquiring an initial transformation matrix when the requested zoom ratio is in the wide-angle segment, wherein the initial transformation matrix is ​​used for representing a mapping relationship between an initial wide-angle image and an initial telephoto image, wherein the initial wide-angle image is an image captured by the wide-angle camera at a calibrated distance, and the initial telephoto image is an image captured by the telephoto camera at a calibrated distance; A shooting distance determination module, used to determine the current shooting distance; A processing module, configured to adjust the initial transformation matrix according to the current shooting distance and the requested zoom factor to obtain a current transformation matrix, and process the image to be shot according to the current transformation matrix; The processing module includes: an adjustment transformation matrix determination unit, which is used to determine the adjustment transformation matrix corresponding to the current shooting distance according to the corresponding relationship between the calibration distance and the initial transformation matrix; a target transformation matrix acquisition unit, which is used to adjust the adjustment transformation matrix according to each zoom factor of the wide-angle segment to obtain the target transformation matrix corresponding to each zoom factor; and a current transformation matrix acquisition unit, which is used to determine the target transformation matrix corresponding to the requested zoom factor as the current transformation matrix.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are performed.

12. An image processing device based on multiple cameras, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor runs the computer program, the steps of the method according to any one of claims 1 to 9 are performed.

Images