Background blur method, device, equipment, storage medium and program product

Abstract

The present application provides a background blur method, device, equipment, storage medium and program product. The method includes: obtaining the current first parameters of the two cameras, then obtaining the current first VCM value, and determining whether to optimize the first parameter according to the first VCM value. If it is determined that the first parameter is to be optimized, the first parameter is optimized, and the current image is subjected to background blur processing based on the optimized first parameter. If it is determined that the first parameter is not to be optimized, the current image is subjected to background blur processing based on the first parameter. The present application does not require the calibration parameters of the camera module, and can determine whether to optimize the known parameters of the camera according to the current shooting scene, so that the parameters used in the current shooting scene are more accurate, thereby achieving a better background blur effect.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a background blur method, device, equipment, storage medium and program product. Background Art

[0002] With the development of mobile communication technology, terminal devices have evolved from single cameras to dual cameras, triple cameras, and even more cameras. However, as mobile terminals, terminal devices are small in size and cannot carry the same hardware as SLR cameras. To achieve a shallow depth of field background blur effect, algorithms are needed.

[0003] In the related art, in a terminal device with dual cameras, background blur can usually be achieved based on the stereoscopic vision of the dual cameras. Specifically, the depth of the scene can be obtained, and recognition can be performed based on the different depths of field between the subject and the background. Different pixels can be gradually blurred to blur the background.

[0004] However, for a pair of images of the same scene captured by dual cameras, since the two cameras have different intrinsic parameters, there is an obvious spatial transformation and scale difference between the perspectives of the images captured, resulting in a poor background blur effect. Summary of the invention

[0005] The present application provides a background blur method, device, equipment, storage medium and program product to solve the problem in the related art that dual cameras have different internal parameters, and there are obvious spatial transformations and scale differences between the perspectives of the captured images, resulting in poor background blur effect.

[0006] In a first aspect, the present application provides a background blurring method, comprising:

[0007] Get the current first parameters of the two cameras;

[0008] Get the current VCM value of the first voice coil motor;

[0009] Determining whether to optimize the first parameter according to the first VCM value;

[0010] If it is determined to optimize the first parameter, the first parameter is optimized, and background blur processing is performed on the current image based on the optimized first parameter;

[0011] If it is determined not to optimize the first parameter, background blur processing is performed on the current image based on the first parameter.

[0012] In a possible implementation manner, determining whether to optimize the first parameter according to the first VCM value includes:

[0013] Obtain the second VCM value recorded when the background blur processing was performed on the image last time;

[0014] When the difference between the first VCM value and the second VCM value is less than or equal to a preset threshold, determining not to optimize the first parameter;

[0015] When the difference between the first VCM value and the second VCM value is greater than the preset threshold, determining to optimize the first parameter;

[0016] The first parameter includes the physical distance between the two camera sensors and the focal length of the lens.

[0017] In a possible implementation manner, the optimizing the first parameter includes:

[0018] Acquire two first images captured by two image sensors;

[0019] Determine a first reprojection error of projection points of feature points in two first images in three-dimensional space;

[0020] constructing a first error function based on the first reprojection error;

[0021] Based on the first error function, optimizing the first parameter to obtain a first error value corresponding to the first error function;

[0022] The optimized first parameter is determined according to the first error value.

[0023] In a possible implementation, determining a first reprojection error of a projection point of a feature point in two first images in a three-dimensional space includes:

[0024] Performing feature point detection on each first image to obtain first feature points, and performing uniform distribution processing on the first feature points to obtain second feature points;

[0025] Performing similarity matching on the second feature points corresponding to the two first images to obtain a first point set and a second point set, wherein the first point set and the second point set respectively store the coordinates of the second feature points that are successfully matched in the two first images;

[0026] Based on the first parameter, a second reprojection error of a projection point of a second feature point in the first point set and the second point set in a three-dimensional space is determined.

[0027] In a possible implementation manner, determining the optimized first parameter according to the first error value includes:

[0028] When the first error value is less than or equal to a preset threshold, determining that a first parameter corresponding to the first error value is an optimized first parameter;

[0029] When the first error value is greater than a preset threshold, two second images newly captured by the two image sensors are obtained, and a second reprojection error of the projection points of the feature points in the two second images in the three-dimensional space is determined, and a second error function is constructed based on the second reprojection error. Based on the second error function, the first parameter is optimized to obtain a second error value corresponding to the second error function, until the second error value is less than or equal to the preset threshold, and the first parameter corresponding to the second error value is determined to be the optimized first parameter.

[0030] In a possible implementation, the method further includes:

[0031] The first VCM value used in the current background blur processing is recorded, and the first VCM value is used to determine whether the parameters used in the next background blur processing on the image are optimized.

[0032] In a possible implementation, the performing background blur processing on the current image based on the optimized first parameter includes:

[0033] Based on the optimized first parameters, performing stereo correction on two first images captured by two image sensors;

[0034] Performing stereo matching on the two corrected first images to obtain a disparity map corresponding to a reference image, wherein the disparity map includes a disparity value corresponding to each pixel in the reference image, and the reference image is one of the two first images;

[0035] Converting the disparity map into a depth map;

[0036] An image after the background of the current image is blurred is determined according to the depth map.

[0037] In a second aspect, the present application provides a background blurring device, comprising:

[0038] A first acquisition module, used to acquire current first parameters of two cameras;

[0039] A second acquisition module is used to obtain the current VCM value of the first voice coil motor;

[0040] A determination module, configured to determine whether to optimize the first parameter according to the first VCM value;

[0041] A first processing module, configured to optimize the first parameter if it is determined that the first parameter is to be optimized, and perform background blur processing on the current image based on the optimized first parameter;

[0042] The second processing module is used to perform background blur processing on the current image based on the first parameter if it is determined that the first parameter is not to be optimized.

[0043] In a possible implementation manner, the determining module is specifically configured to:

[0044] Obtain the second VCM value recorded when the background blur processing was performed on the image last time;

[0045] When the difference between the first VCM value and the second VCM value is less than or equal to a preset threshold, determining not to optimize the first parameter;

[0046] When the difference between the first VCM value and the second VCM value is greater than the preset threshold, determining to optimize the first parameter;

[0047] The first parameter includes the physical distance between the two camera sensors and the focal length of the lens.

[0048] In a possible implementation manner, the first processing module is specifically configured to:

[0049] Acquire two first images captured by two image sensors;

[0050] Determine a first reprojection error of projection points of feature points in two first images in three-dimensional space;

[0051] constructing a first error function based on the first reprojection error;

[0052] Based on the first error function, optimizing the first parameter to obtain a first error value corresponding to the first error function;

[0053] The optimized first parameter is determined according to the first error value.

[0054] In a possible implementation manner, the first processing module is specifically configured to:

[0055] Performing feature point detection on each first image to obtain first feature points, and performing uniform distribution processing on the first feature points to obtain second feature points;

[0056] Performing similarity matching on the second feature points corresponding to the two first images to obtain a first point set and a second point set, wherein the first point set and the second point set respectively store the coordinates of the second feature points that are successfully matched in the two first images;

[0057] Based on the first parameter, a second reprojection error of a projection point of a second feature point in the first point set and the second point set in a three-dimensional space is determined.

[0058] In a possible implementation manner, the first processing module is specifically configured to:

[0059] When the first error value is less than or equal to a preset threshold, determining that a first parameter corresponding to the first error value is an optimized first parameter;

[0060] When the first error value is greater than a preset threshold, two second images newly captured by the two image sensors are obtained, and a second reprojection error of the projection points of the feature points in the two second images in the three-dimensional space is determined, and a second error function is constructed based on the second reprojection error. Based on the second error function, the first parameter is optimized to obtain a second error value corresponding to the second error function, until the second error value is less than or equal to the preset threshold, and the first parameter corresponding to the second error value is determined to be the optimized first parameter.

[0061] In a possible implementation, the device further includes:

[0062] The recording module is used to record the first VCM value used in the current background blur processing, and the first VCM value is used to determine whether the parameters used when the background blur processing is performed on the image next time are optimized.

[0063] In a possible implementation manner, the first processing module is specifically configured to:

[0064] Based on the optimized first parameters, performing stereo correction on two first images captured by two image sensors;

[0065] Performing stereo matching on the two corrected first images to obtain a disparity map corresponding to a reference image, wherein the disparity map includes a disparity value corresponding to each pixel in the reference image, and the reference image is one of the two first images;

[0066] Converting the disparity map into a depth map;

[0067] An image after the background of the current image is blurred is determined according to the depth map.

[0068] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0069] The memory stores computer-executable instructions;

[0070] The processor executes the computer-executable instructions stored in the memory to implement the background blurring method as described in the first aspect.

[0071] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a computer, they are used to implement the background blurring method as described in the first aspect.

[0072] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a computer, is used to implement the background blurring method described in the first aspect.

[0073] In a sixth aspect, an embodiment of the present application provides a chip having a computer program stored thereon, wherein when the computer program is executed by the chip, the background blurring method described in the first aspect is executed.

[0074] In a possible implementation, the chip is a chip in a chip module.

[0075] In a seventh aspect, an embodiment of the present application provides a module device, wherein the module device includes a power module, a storage module, and a chip module;

[0076] Wherein, the power module is used to provide electrical energy to the module device;

[0077] The storage module is used to store data and instructions;

[0078] The chip module is used to execute the background blurring method described in the first aspect.

[0079] The present application provides a background blur method, device, equipment, storage medium and program product, which obtains the current first parameters of the two cameras, then obtains the current first VCM value, and determines whether to optimize the first parameter according to the first VCM value. If it is determined that the first parameter is to be optimized, the first parameter is optimized, and the background blur is performed on the current image based on the optimized first parameter. If it is determined that the first parameter is not to be optimized, the background blur is performed on the current image based on the first parameter. The present application does not require the calibration parameters of the camera module, and can determine whether to optimize the known parameters of the camera according to the current shooting scene, so that the parameters used in the current shooting scene are more accurate, thereby achieving a better background blur effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0081] Figure 1A schematic diagram of an application scenario applicable to the present application;

[0082] Figure 2 A schematic diagram of a process flow of a background blurring method provided in Embodiment 1 of the present application;

[0083] Figure 3 A schematic diagram of a process flow of another background blurring method provided in Embodiment 2 of the present application;

[0084] Figure 4 A schematic diagram of N pairs of pixel points corresponding to a feature point in the example of this application;

[0085] Figure 5 A schematic diagram of the projection points of the example of this application;

[0086] Figure 6 A schematic diagram of the structure of a background blurring device provided in Embodiment 3 of the present application;

[0087] Figure 7 A schematic diagram of the structure of an electronic device provided in Example 4 of the present application.

[0088] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0089] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0090] For ease of understanding, the terms used in this application are first explained:

[0091] (1) Calibration: The process of obtaining the intrinsic and extrinsic parameters of the camera.

[0092] (2) Internal parameters: the hardware parameters of the camera itself, such as principal point coordinates, focal length, lens distortion, etc.

[0093] (3) External parameters: description of the spatial relative position of the two cameras (rotation, translation).

[0094] (4) Baseline: The line connecting the optical centers of the left and right cameras.

[0095] (5) Epipolar plane: The plane determined by the optical centers of the two cameras.

[0096] (6) Pole: the intersection of the baseline and the two camera image planes;

[0097] (7) Epipolar line: the intersection of the epipolar plane and the image plane.

[0098] (8) Parallax: Parallax is the difference in direction when observing the same object from two points at a certain distance.

[0099] (9) Stereo correction: Image stereo correction usually refers to performing a plane projection transformation on two images respectively, so that the corresponding epipolar lines of the two images are in the same horizontal direction, and the poles are mapped to infinity, so that the two images only have parallax in the horizontal direction.

[0100] (10) Stereo matching: Stereo matching is also called disparity estimation or binocular depth estimation. Its input is a pair of left and right images captured at the same time and after epipolar correction. The output of stereo matching is a disparity map composed of the disparity values ​​corresponding to each pixel in the reference image (usually the left image is used as the reference image).

[0101] (11) Depth: The distance of each pixel in the image relative to the source of the image.

[0102] (12) Voice Coil Motor (VCM), VCM can be used to adjust the position of the lens, achieve the focusing function, and present a clear image.

[0103] (13) Active ranging: The light source and sound source are artificially controlled to illuminate the object target, and the three-dimensional information of the photographed target is obtained based on the reflection characteristics, optical and acoustic characteristics of the object surface.

[0104] (14) Passive ranging: No need to artificially set up a radiation source, only the two-dimensional image of the scene under natural light is used to reconstruct the three-dimensional information of the scene.

[0105] (15) Feature points: They can be simply understood as more prominent points in the image, such as contour points, bright spots in darker areas, dark spots in brighter areas, etc.

[0106] (16) Feature point matching: Match the same feature point in two images from different perspectives. If the descriptors are roughly the same, they are considered to be the same point. The descriptor is the common point and connection between the same feature points. The conditions of the descriptor are set by yourself.

[0107] With the development of mobile communication technology, video recording has become an important function in user terminal devices, which has gradually developed from a single camera to dual cameras, triple cameras and even more cameras, achieving an effect comparable to that of a SLR camera. SLR cameras take photos with blurred backgrounds, relying on large photosensitive elements, large apertures and long focal lengths. However, as mobile terminals, terminal devices are small in size and cannot carry the same hardware as SLR cameras. To achieve a blurred background effect with a shallow depth of field, algorithms are needed.

[0108] At present, the commonly used methods for background blurring include: portrait segmentation algorithm, deep learning method, and stereo vision based on dual cameras. Among them, the portrait segmentation algorithm may not be able to accurately identify the outline of the person, especially the details such as the hair ends and cuffs of the person. The incorrectly identified parts will be blurred like the background. The deep learning method requires a large number of accurate data sets. Stereo vision based on dual cameras obtains the depth of the scene, identifies the subject and the background according to the different depths of field, and gradually blurs different pixels to blur the background.

[0109] However, for a pair of images of the same scene captured by dual cameras, due to the different intrinsic parameters of the two cameras, there is an obvious spatial transformation and scale difference between the perspectives of the captured images, resulting in poor background blurring. The premise for accurately obtaining a depth map is to transform the perspectives of the dual camera images to the same plane. In one possible implementation, the two images can be stereo rectified, and the indispensable factor for stereo correction is the camera parameters calibrated during the production of the camera module.

[0110] Calibration is a necessary step before using the dual-camera system. If the camera parameters are not calibrated, stereo correction cannot be performed, making subsequent stereo matching operations such as obtaining disparity maps more difficult and time-consuming. However, when each camera module is produced, due to the existence of process errors, there will be deviations in the lens installation process, so the parameters of each module are different. Each camera module needs to be calibrated separately. The calibration method can be, for example, Zhang Zhengyou calibration method based on a two-dimensional calibration plate under fixed focus conditions.

[0111] However, calibrating each camera module is not only time-consuming and costly, but also due to the limitations of scene conditions in the factory, it is not possible to calibrate at every shooting distance. Only a few sets of shooting distances within a specified range can be calibrated. When the user uses the background blur function, the closest set of parameters is selected for stereo correction and stereo matching. This will cause a large error and lead to unsatisfactory background blur effects, such as edges, hollowing and other detail problems. In addition, the purpose of stereo correction is to correct the actual binocular system to an ideal binocular system, and the ideal binocular system is that the parameters of the two cameras are consistent and the two image planes are parallel. However, in order to avoid excessive use of space by lenses, the current mainstream terminal equipment dual-camera systems usually use wide-angle lenses and telephoto lenses. The dual-camera system composed of two cameras with completely different internal parameters will also affect the calibration accuracy.

[0112] Moreover, due to process errors and other reasons, the parameters of each camera module are different, which means that manufacturers have to calibrate the dual-camera module as many times as they produce terminal devices, which is a step that all manufacturers want to avoid. In recent years, with the development of the diversity of camera types in terminal devices, some manufacturers have used LiDAR in the camera modules of terminal devices. It can obtain the depth information of the scene through active ranging without using camera parameters, but the cost is relatively high.

[0113] Therefore, the present application provides a background blur method. When a user uses the camera function of a terminal device, it is determined whether to optimize the first parameter used for the previous background blur processing according to the current VCM value. If the current VCM value is significantly different from the VCM value recorded when the parameters were optimized for the previous background blur processing of the image, then after optimizing the first parameter, the background blur processing is performed on the current image based on the optimized first parameter, thereby achieving the goal of not requiring calibration parameters of the camera module. It is also possible to determine whether to optimize the known parameters of the camera according to the current shooting scene, so that the parameters used in the current shooting scene are more accurate, thereby achieving a better background blur effect.

[0114] For ease of understanding, the following Figure 1 The examples are used to illustrate the application scenarios to which the embodiments of the present application are applicable.

[0115] Figure 1 For a schematic diagram of an application scenario applicable to this application, please see Figure 1, including a terminal device 101 and a photographed target 102, the terminal device 101 includes at least two cameras, the user photographs the photographed target 102 through the at least two cameras of the terminal device 101, and obtains the first VCM value used for the current shooting, and determines whether to optimize the current first parameters of two cameras of the at least two cameras according to the first VCM value, so as to perform background blur processing on the currently shot image through the first parameter or the optimized first parameter.

[0116] In the embodiment of the present application, the terminal device can be any device with a camera function. The embodiment of the present application can be applicable to preview and video modes, and the device includes but is not limited to: the terminal device can also be called user equipment (User Equipment, UE), mobile phone (Mobile Phone), tablet computer (Pad), tablet, point of sales (Point of sales, POS) machine, computer with wireless transceiver function (such as laptop computer, PDA, etc.), mobile Internet device (Mobile Internet Device, MID), virtual reality (Virtual Reality, VR) device, augmented reality (Augmented Reality, AR) device, extended reality (Extended Reality, XR) device, wireless terminal in industrial control (industrial control), wireless terminal in self-driving (Self Driving), wireless terminal in remote medical (Remote Medical), wireless terminal in smart grid (Smart Grid), wireless terminal in transportation safety (Transportation Safety), wireless terminal in smart city (Smart City), wireless terminal in smart home (Smart Home), cellular phone, cordless phone, session initiation protocol (Session Initiation Protocol (SIP) phones, Wireless Local Loop (WLL) stations, Personal Digital Assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, vehicle-mounted devices, wearable devices, terminal devices in 5G networks, or terminal devices in future evolved public land mobile communication networks (PLMNs), etc.

[0117] Among them, wearable devices can also be called wearable smart devices, which are a general term for the intelligent design and development of wearable devices for daily wear using wearable technology, such as glasses, gloves, watches, clothing and shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also realize powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include full-featured, large-sized, and independent of smartphones to achieve complete or partial functions, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0118] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments may exist independently or in combination with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0119] Figure 2 A flowchart of a background blurring method provided in the first embodiment of the present application is provided. The method can be executed by a terminal device or by a background blurring device provided in the terminal device. The device can be a chip, a chip module, or an integrated development environment (IDE). The following description is made by taking the terminal device as an example. Figure 2 , the method comprises the following steps.

[0120] S201. Obtain current first parameters of two cameras.

[0121] When the user uses the camera function of the terminal device, the terminal device can obtain the current first parameters of two cameras of the at least two cameras in the terminal device, wherein the first parameters include the physical distance between the two camera sensors and the lens focal length.

[0122] It can be understood that the current first parameter may be a parameter used by the terminal device when performing background blur processing on an image last time.

[0123] S202: Obtain the current VCM value of the first voice coil motor.

[0124] The terminal device can obtain the first VCM value corresponding to the position of the lens adjusted by the user to achieve the focusing function and present a clear image. The first VCM value can be used to compare with the VCM value recorded when the terminal device last optimized the parameters for background blur processing of the image to determine whether the depth of the shooting scene has changed, thereby determining whether the first parameter needs to be optimized.

[0125] S203: Determine whether to optimize the first parameter according to the first VCM value.

[0126] After the terminal device obtains the first VCM value, it can determine whether the first parameter needs to be optimized according to the first VCM value.

[0127] Exemplarily, the terminal device may compare the first VCM value with the second VCM value to determine whether the depth of the current shooting scene has changed from the depth of the previous shooting scene. When the difference between the first VCM value and the second VCM value is within a certain range, it can be considered that the depth of the current shooting scene has not changed from the depth of the previous shooting scene, or it can be considered that the change between the depth of the current shooting scene and the depth of the previous shooting scene is small, and then it can be determined not to optimize the first parameter. Conversely, when the difference between the first VCM value and the second VCM value is not within the above range, it can be considered that the depth of the current shooting scene has changed from the depth of the previous shooting scene, and it can be determined to optimize the first parameter.

[0128] The second VCM value is recorded when the terminal device last performed background blur processing on the image.

[0129] S204: If it is determined to optimize the first parameter, the first parameter is optimized, and background blur processing is performed on the current image based on the optimized first parameter.

[0130] When the terminal device determines to optimize the first parameter, the first parameter can be optimized, and then the background of the current image can be blurred based on the optimized first parameter to obtain an image with blurred background.

[0131] Among them, the way in which the terminal device optimizes the first parameter can be, for example: the terminal device can optimize the first parameter based on the spatial constraint relationship between the feature points in the images captured by the two cameras (for example, the reprojection error of the projection points of the feature points in the two first images in the three-dimensional space, etc.).

[0132] S205: If it is determined not to optimize the first parameter, perform background blur processing on the current image based on the first parameter.

[0133] When the terminal device determines not to optimize the first parameter, it indicates that the parameter (ie, the first parameter) used by the terminal device when performing background blur processing on the image last time can be used to perform background blur processing on the current image.

[0134] In this embodiment, the current first parameters of the two cameras are obtained, and then the current first VCM value is obtained, and it is determined whether to optimize the first parameter according to the first VCM value. If it is determined that the first parameter is to be optimized, the first parameter is optimized, and the background of the current image is blurred based on the optimized first parameter. If it is determined that the first parameter is not to be optimized, the background of the current image is blurred based on the first parameter. This application does not require the calibration parameters of the camera module, and it can determine whether to optimize the known parameters of the camera according to the current shooting scene, so that the parameters used in the current shooting scene are more accurate, thereby achieving a better background blur effect.

[0135] Next, another background blurring method provided by the present application is described through Example 2.

[0136] Figure 3 A flowchart of another background blurring method provided in the second embodiment of the present application is provided. The method can be executed by a terminal device or by a background blurring device provided in the terminal device. The device can be a chip, a chip module, or an integrated development environment (IDE). The following description is made by taking the terminal device as an example. Figure 3 , the method comprises the following steps.

[0137] S301, obtaining the current first parameters of two cameras.

[0138] S302: Obtain the current VCM value of the first voice coil motor.

[0139] For the related description of S301 and S302, reference may be made to S201 and S202 in the first embodiment, which will not be repeated here.

[0140] S303: Determine whether to optimize the first parameter according to the first VCM value.

[0141] If it is determined to optimize the first parameter, execute S304 ; if it is determined not to optimize the first parameter, execute S310 .

[0142] In one possible implementation, the terminal device can obtain the second VCM value recorded when the background blur processing was performed on the image last time, and then compare the first VCM value with the second VCM value. When the difference between the first VCM value and the second VCM value is less than or equal to a preset threshold, it is determined not to optimize the first parameter. When the difference between the first VCM value and the second VCM value is greater than the preset threshold, it is determined to optimize the first parameter.

[0143] S304: Acquire two first images captured by two image sensors.

[0144] S305 , determining a first reprojection error of projection points of feature points in the two first images in the three-dimensional space.

[0145] After the terminal device acquires the two first images, the first reprojection error of the projection point of the feature points in the two first images in the three-dimensional space can be determined in the following manner:

[0146] (1) Perform feature point detection on each first image to obtain first feature points, and perform uniform distribution processing on the first feature points to obtain second feature points. Exemplarily, the first feature points can be processed by a quadtree algorithm to obtain second feature points.

[0147] (2) Performing similarity matching on the second feature points corresponding to the two first images to obtain a first point set and a second point set, wherein the first point set and the second point set respectively store the coordinates of the second feature points that are successfully matched in the two first images.

[0148] Optionally, the terminal device may perform similarity matching on the second feature points corresponding to the two first images as follows:

[0149] Taking each feature point as the origin, determine N random pairs of pixels within the range of the circle corresponding to the preset radius, and then compare the grayscale values ​​of each pair of pixels. For example, taking a pair of pixels a and b, if the grayscale value of pixel a is less than the grayscale value of pixel b, the comparison result of this pair of pixels is 0. If the grayscale value of pixel a is greater than or equal to pixel b, the comparison result of this pair of pixels is 1. For ease of understanding, Figure 4 As shown, Figure 4 A schematic diagram of N pairs of pixel points corresponding to a feature point, wherein the coordinates of the N pairs of pixel points are preset in advance.

[0150] Then each feature point can get a descriptor consisting of a string of 0s and 1s, and the descriptor includes N bits. For example, if N is 128, the descriptor of the feature point includes 128 bits.

[0151] Then, the upper left corners of the two first images are taken as the origin to determine the position coordinates of each feature point in the two first images. For example, the two first images are called image A and image B respectively. The terminal device can match the descriptor corresponding to feature point 1 in image A with the descriptor corresponding to the feature point within a preset range based on feature point 2 in image B (the position coordinates of feature point 1 and feature point 2 in image A and image B are the same, respectively).

[0152] Exemplarily, the preset range may be a circle with feature point 2 as the center, or a rectangle with feature point 2 as the upper left corner determined by the pixel value of feature point 2 as the length and width, or other forms, which are not limited in this application.

[0153] There are several ways to determine whether the descriptors match:

[0154] Method 1

[0155] In the method 1, if the feature point and the descriptor corresponding to the feature point are the same, it can be determined as a match; if the feature point and the descriptor corresponding to the feature point are different, it can be determined as a mismatch.

[0156] Method 2

[0157] In mode 2, if 80% of the bits between the descriptors are the same, it can be determined as a match; if the number of the same bits between the descriptors is less than 80%, it can be determined as a mismatch.

[0158] After matching the descriptor corresponding to feature point 1 with the descriptor corresponding to the feature point within a preset range based on feature point 2 in image B, the successfully matched feature points in image A and image B are stored in the first point set and the second point set, respectively. It can be understood that in mode 2, there may be multiple feature points in image B that match a feature point in image A. When there are multiple feature points, the one with the highest matching degree is selected as the feature point that matches the feature point in image A, wherein the one with the highest matching degree may be the one with the largest number of identical bits.

[0159] (3) Based on the first parameter, determine a first reprojection error of a three-dimensional space projection point of a second feature point in the first point set and the second point set.

[0160] The projection point can be determined by formula (1):

[0161]

[0162] Among them, u and v are the coordinates of the projection point, X w , Y w and Z wis the coordinate of the space point corresponding to the second feature point in the three-dimensional space (that is, the actual point in the photographed target corresponding to the second feature point), u0 and v0 are the theoretical principal point coordinates of the camera, R is a 3×3 matrix, the elements in the matrix can be any initial value, such as 0.1, T is a 3×1 matrix, the value of the first element in the matrix is ​​the physical distance between the two camera sensors in the first parameter, the second element and the third element can be any initial value, such as 0.1, f x and f y Refers to the focal length of the secondary camera lens. The secondary camera is the camera corresponding to the image where the projection point is located, for example Figure 5 Schematic diagram of the projection points of the example in , the secondary camera is the camera corresponding to the right picture.

[0163] X w , Y w and Z w It can be determined by the following formula (2):

[0164]

[0165] Among them, Z cleft and Z cright is the scaling factor, for example, Z cleft and Z cright can be set to the effective focal length, i.e. the distance from the optical center to the image plane, u l and v l is the coordinate of the second feature point in the first point set, u r and v r are the coordinates of the feature points in the second point set that can match the second feature points in the first point set, and f lx and f ly It refers to the focal length of the camera lens corresponding to the image where the second feature point in the first point set is located, f rx and f ry It refers to the focal length of the camera lens corresponding to the image where the second feature point in the second point set is located. Rl and Rr are 3×3 matrices respectively. The elements in the matrices can be any initial values. Tl and Tr are 3×1 matrices respectively. The value of the first element in the matrix is ​​the physical distance between the two camera sensors in the first parameter. The second and third elements can be any initial values.

[0166] The solution of the equation system of formula (2) can be solved by singular value decomposition. The eigenvector with the smallest singular value is the solution of the equation system, that is, X w , Y w and Z w .

[0167] The first reprojection error can include two errors in the x direction and the y direction, namely, u and u r The difference between v and v r The difference.

[0168] S306: Construct a first error function based on the first reprojection error.

[0169] Based on the first reprojection error, the constructed first error function can be in the form of formula (3):

[0170]

[0171] S307: Optimize the first parameter based on the first error function to obtain a first error value corresponding to the first error function.

[0172] The terminal device can perform nonlinear least square optimization on the first error function through the value of the first parameter in formula (1) to minimize the difference in formula (3), and then record the adjusted first parameter.

[0173] Then, using the adjusted first parameter, each pair of matching second feature points in the first point set and the second point set is aligned and dedistorted, and the difference of the aligned and dedistorted second feature points is calculated in the y direction to obtain the error value of each pair of matching second feature points, and the error values ​​of multiple pairs of matching second feature points are averaged to obtain the first error value.

[0174] S308. Determine an optimized first parameter according to the first error value.

[0175] In a possible implementation, when the first error value is less than or equal to a preset threshold, it is determined that the first parameter corresponding to the first error value is the optimized first parameter, and the first parameter corresponding to the first error value is the adjusted first error value.

[0176] In another possible implementation, when the first error value is greater than a preset threshold, two second images newly captured by the two image sensors are obtained, and a second reprojection error of the projection points of the feature points in the two second images in the three-dimensional space is determined, and a second error function is constructed based on the second reprojection error. Based on the second error function, the first parameter is optimized to obtain a second error value corresponding to the second error function, until the second error value is less than or equal to the preset threshold, and then the first parameter corresponding to the second error value is determined to be the optimized first parameter.

[0177] S309: Perform background blur processing on the current image based on the optimized first parameter.

[0178] Specifically, the terminal device can perform background blur processing on the current image in the following manner:

[0179] Based on the optimized first parameter, stereo correction is performed on the two first images captured by the two image sensors, that is, a plane projection transformation is performed on the two first images respectively, so that the corresponding epipolar lines of the two first images are in the same horizontal direction, and the poles are mapped to infinity, so that the two first images have only parallax in the horizontal direction.

[0180] Then, stereo matching is performed on the two corrected first images, that is, the two first images after epipolar correction are used as left and right images and input into the stereo matching algorithm, and then the disparity map corresponding to the reference image is obtained. The disparity map includes the disparity value corresponding to each pixel in the reference image. The reference image is one of the two first images, and the left image is generally used as the reference image.

[0181] Then the disparity map is converted into a depth map. Specifically, the terminal device can convert the disparity map into a depth map by the following formula (4):

[0182]

[0183] Among them, d is the depth value that constitutes the depth map, and f is the focal length of the main camera lens, for example Figure 5 For the left camera in the figure, disp is the disparity value in the disparity map.

[0184] After determining the depth map, the terminal device can determine the image after the background of the current image is blurred based on the depth map.

[0185] S310: Perform background blur processing on the current image based on the first parameter.

[0186] In this step, the background blur processing is performed on the current image based on the first parameter. You can refer to S309 in which the background blur processing is performed on the current image based on the optimized first parameter, and replace the optimized first parameter with the first parameter for understanding. No further details will be given here.

[0187] Optionally, the above method further includes:

[0188] The first VCM value used in the current background blur processing is recorded. The first VCM value is used to determine whether the parameters used in the next background blur processing on the image are optimized.

[0189] In this embodiment, the current first parameters of the two cameras are obtained, and then the current first VCM value is obtained, and it is determined whether to optimize the first parameter based on the first VCM value. If it is determined to optimize the first parameter, the two first images captured by the two image sensors are obtained, and the first reprojection error of the projection point of the feature points in the two first images in the three-dimensional space is determined, and then based on the first reprojection error, a first error function is constructed, and then based on the first error function, the first parameter is optimized to obtain the first error value corresponding to the first error function, and then based on the first error value, the optimized first parameter is determined, and the background blurring of the current image is performed based on the optimized first parameter. If it is determined not to optimize the first parameter, the background blurring of the current image is performed based on the first parameter. This application does not require the calibration parameters of the camera module, and can determine whether to optimize the known parameters of the camera based on the current shooting scene, so that the parameters used in the current shooting scene are more accurate, thereby achieving a better background blurring effect.

[0190] In addition, the background blur processing method provided by this application can also have the following technical effects:

[0191] (1) The dual-camera module does not need to be calibrated with a calibration plate in the module factory, providing manufacturers with a more convenient production model.

[0192] (2) When the terminal device module is offset (for example, it is dropped), this method can be used to calibrate the camera parameters at a terminal device maintenance site without a calibration board and calibration environment, providing a more convenient solution for merchants.

[0193] Figure 6 This is a schematic diagram of the structure of a background blur device provided in Example 3 of the present application. Figure 6 The device 60 includes: a first acquisition module 601, a second acquisition module 602, a determination module 603, a first processing module 604 and a second processing module 605.

[0194] The first acquisition module 601 is used to acquire current first parameters of two cameras.

[0195] The second acquisition module 602 is used to acquire the current VCM value of the first voice coil motor.

[0196] The determination module 603 is used to determine whether to optimize the first parameter according to the first VCM value.

[0197] The first processing module 604 is used to optimize the first parameter if it is determined to optimize the first parameter, and perform background blur processing on the current image based on the optimized first parameter.

[0198] The second processing module 605 is configured to perform background blur processing on the current image based on the first parameter if it is determined that the first parameter is not to be optimized.

[0199] In a possible implementation manner, the determination module 603 is specifically configured to:

[0200] Get the second VCM value recorded when the background blur processing is performed on the image last time.

[0201] When the difference between the first VCM value and the second VCM value is less than or equal to the preset threshold, it is determined not to optimize the first parameter.

[0202] When the difference between the first VCM value and the second VCM value is greater than a preset threshold, it is determined to optimize the first parameter.

[0203] The first parameter includes the physical distance between the two camera sensors and the focal length of the lens.

[0204] In a possible implementation, the first processing module 604 is specifically configured to:

[0205] Acquire two first images captured by two image sensors;

[0206] A first reprojection error of projection points of feature points in the two first images in the three-dimensional space is determined.

[0207] Based on the first reprojection error, a first error function is constructed.

[0208] Based on the first error function, the first parameter is optimized to obtain a first error value corresponding to the first error function.

[0209] An optimized first parameter is determined according to the first error value.

[0210] In a possible implementation manner, the first processing module is specifically configured to:

[0211] Feature point detection is performed on each first image to obtain first feature points, and uniform distribution processing is performed on the first feature points to obtain second feature points.

[0212] The second feature points corresponding to the two first images are matched by similarity to obtain a first point set and a second point set, wherein the first point set and the second point set respectively store the coordinates of the second feature points that are successfully matched in the two first images.

[0213] Based on the first parameter, a second reprojection error of a projection point of a second feature point in the first point set and the second point set in the three-dimensional space is determined.

[0214] In a possible implementation, the first processing module 604 is specifically configured to:

[0215] When the first error value is less than or equal to the preset threshold, it is determined that the first parameter corresponding to the first error value is the optimized first parameter.

[0216] When the first error value is greater than a preset threshold, two second images newly captured by the two image sensors are obtained, and the second reprojection errors of the projection points of the feature points in the two second images in the three-dimensional space are determined. Based on the second reprojection error, a second error function is constructed. Based on the second error function, the first parameter is optimized to obtain a second error value corresponding to the second error function, until the second error value is less than or equal to the preset threshold, and the first parameter corresponding to the second error value is determined to be the optimized first parameter.

[0217] In a possible implementation, the device 60 further includes:

[0218] The recording module is used to record the first VCM value used in the current background blur processing, and the first VCM value is used to determine whether the parameters used when the background blur processing is performed on the image next time are optimized.

[0219] In a possible implementation, the first processing module 604 is specifically configured to:

[0220] Based on the optimized first parameters, stereo correction is performed on the two first images captured by the two image sensors.

[0221] Stereo matching is performed on the two corrected first images to obtain a disparity map corresponding to a reference image, wherein the disparity map includes a disparity value corresponding to each pixel in the reference image, and the reference image is one of the two first images.

[0222] Convert the disparity map to a depth map.

[0223] According to the depth map, an image after the background of the current image is blurred is determined.

[0224] The device of this embodiment can be used to execute the technical solution of the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0225] Figure 7 A schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present application is shown in FIG. Figure 7 As shown, the electronic device 70 may include: at least one processor 701 and a memory 702 .

[0226] The memory 702 is used to store programs. Specifically, the programs may include program codes, and the program codes include computer operation instructions.

[0227] The memory 702 may include a random access memory (RAM), and may also include a non-volatile memory (Non-volatile Memory), such as at least one disk memory.

[0228] The processor 701 is used to execute the computer-executable instructions stored in the memory 702 to implement the method described in the above method embodiment. The processor 701 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0229] Optionally, the electronic device 70 may further include: a communication interface 703. In a specific implementation, if the communication interface 703, the memory 702 and the processor 701 are implemented independently, the communication interface 703, the memory 702 and the processor 701 may be interconnected through a bus and communicate with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0230] Optionally, in a specific implementation, if the communication interface 703, the memory 702 and the processor 701 are integrated on a chip, the communication interface 703, the memory 702 and the processor 701 can communicate through an internal interface.

[0231] The electronic device 70 may be a chip, a chip module, an integrated development environment (IDE), a terminal device, etc.

[0232] The electronic device of this embodiment can be used to execute the technical solution of the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0233] Embodiment 5 of the present application provides a computer-readable storage medium, which may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a RAM, a disk, or an optical disk, and other media that can store computer-executable instructions. Specifically, the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a computer, the technical solution shown in the above method embodiment is executed. The specific implementation method and technical effect are similar and will not be repeated here.

[0234] Embodiment 6 of the present application provides a computer program product, including a computer program. When the computer program is executed by a computer, the technical solution shown in the above method embodiment is executed. The specific implementation method and technical effect are similar and will not be repeated here.

[0235] Embodiment 7 of the present application provides a chip having a computer program stored thereon. When the computer program is executed by the chip, the technical solution shown in the above method embodiment is executed.

[0236] In a possible implementation, the chip may also be a chip module.

[0237] Embodiment 8 of the present application provides a module device, which includes a power module, a storage module and a chip module.

[0238] Among them, the power module is used to provide electrical energy to the module equipment.

[0239] The storage module is used to store data and instructions.

[0240] The chip module is used to execute the technical solution shown in the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0241] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0242] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0243] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0244] In this application, "and / or" is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship.

[0245] "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, a and b, a and c, b and c, or a, b and c, where each of a, b, c can be an element itself or a set containing one or more elements.

[0246] In this application, "at least one" means one or more. "Multiple" means two or more. The first, second, etc. descriptions that appear in the embodiments of this application are only used for illustration and distinction of the described objects. There is no order, nor does it represent a special limitation on the number of devices in the embodiments of this application, and cannot constitute any limitation on the embodiments of this application. For example, the first threshold and the second threshold are only used to distinguish different thresholds, and do not represent the difference in size, priority or importance of the two thresholds.

[0247] In this application, "exemplary", "in some embodiments", "in other embodiments", etc. are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present concepts in a concrete way.

[0248] In this application, "of", "corresponding, relevant", "corresponding", and "associated" can sometimes be used interchangeably. It should be noted that when the distinction is not emphasized, the meanings they express are consistent. In the embodiments of this application, communication and transmission can sometimes be used interchangeably. It should be noted that when the distinction is not emphasized, the meanings they express are consistent. For example, transmission can include sending and / or receiving, which can be a noun or a verb.

[0249] In this application, "equal to" can be used in conjunction with "less than" or "greater than", but not with both "less than" and "greater than". When "equal to" is used in conjunction with "less than", it is applicable to the technical solution adopted by "less than". When "equal to" is used in conjunction with "greater than", it is applicable to the technical solution adopted by "greater than".

Claims

1. A background blurring method, characterized in that: The method comprises: Obtaining current first parameters of the two cameras; the current first parameters are the parameters used when the background blur processing was performed on the image last time; wherein the first parameters include the physical distance between the two camera sensors and the lens focal length; Get the current VCM value of the first voice coil motor; Obtain the second VCM value recorded when the background blur processing was performed on the image last time; When the difference between the first voice coil motor VCM value and the second VCM value is less than or equal to a preset threshold, determining not to optimize the first parameter; When the difference between the first voice coil motor VCM value and the second VCM value is greater than the preset threshold, determining to optimize the first parameter; If it is determined to optimize the first parameter, the first parameter is optimized, and background blur processing is performed on the current image based on the optimized first parameter; If it is determined not to optimize the first parameter, background blur processing is performed on the current image based on the first parameter.

2. The method according to claim 1, characterized in that The optimizing the first parameter includes: Acquire two first images captured by two image sensors; Determine a first reprojection error of projection points of feature points in two first images in three-dimensional space; constructing a first error function based on the first reprojection error; Based on the first error function, optimizing the first parameter to obtain a first error value corresponding to the first error function; The optimized first parameter is determined according to the first error value.

3. The method according to claim 2, characterized in that The determining of a first reprojection error of a projection point of a feature point in two first images in a three-dimensional space comprises: Performing feature point detection on each first image to obtain first feature points, and performing uniform distribution processing on the first feature points to obtain second feature points; Performing similarity matching on the second feature points corresponding to the two first images to obtain a first point set and a second point set, wherein the first point set and the second point set respectively store the coordinates of the second feature points that are successfully matched in the two first images; Based on the first parameter, a second reprojection error of a projection point of a second feature point in the first point set and the second point set in a three-dimensional space is determined.

4. The method according to claim 2 or 3, characterized in that: The step of determining the optimized first parameter according to the first error value includes: When the first error value is less than or equal to a preset threshold, determining that a first parameter corresponding to the first error value is an optimized first parameter; When the first error value is greater than a preset threshold, two second images newly captured by the two image sensors are obtained, and a second reprojection error of the projection points of the feature points in the two second images in the three-dimensional space is determined, and a second error function is constructed based on the second reprojection error. Based on the second error function, the first parameter is optimized to obtain a second error value corresponding to the second error function, until the second error value is less than or equal to the preset threshold, and the first parameter corresponding to the second error value is determined to be the optimized first parameter.

5. The method according to claim 4, characterized in that The method further comprises: The VCM value of the first voice coil motor used in the current background blur processing is recorded. The VCM value of the first voice coil motor is used to determine whether the parameters used in the next background blur processing of the image are optimized.

6. The method according to claim 1, characterized in that The performing background blur processing on the current image based on the optimized first parameter includes: Based on the optimized first parameters, performing stereo correction on two first images captured by two image sensors; Performing stereo matching on the two corrected first images to obtain a disparity map corresponding to a reference image, wherein the disparity map includes a disparity value corresponding to each pixel in the reference image, and the reference image is one of the two first images; Converting the disparity map into a depth map; An image after the background of the current image is blurred is determined according to the depth map.

7. A background blur device, characterized in that: include: A first acquisition module, used to acquire current first parameters of the two cameras; the current first parameters are the parameters used when the background blur processing was performed on the image last time; wherein the first parameters include the physical distance between the two camera sensors and the lens focal length; A second acquisition module is used to obtain the current VCM value of the first voice coil motor; A determination module, used to obtain a second VCM value recorded when the background blur processing was performed on the image last time; when the difference between the first voice coil motor VCM value and the second VCM value is less than or equal to a preset threshold, determine not to optimize the first parameter; when the difference between the first voice coil motor VCM value and the second VCM value is greater than the preset threshold, determine to optimize the first parameter; A first processing module, configured to optimize the first parameter if it is determined that the first parameter is to be optimized, and perform background blur processing on the current image based on the optimized first parameter; The second processing module is used to perform background blur processing on the current image based on the first parameter if it is determined that the first parameter is not to be optimized.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the background blurring method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the background blurring method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the background blurring method according to any one of claims 1 to 6 is implemented.

Images