Dual-camera depth estimation method and device based on focal length equivalence and frame stream synchronization

Abstract

The present invention relates to a dual-camera depth estimation method and device based on focal length equivalence and frame stream synchronization, comprising: performing focal length equivalence on the dual cameras according to an iterative FoV cropping technique; performing frame stream synchronization on the heterogeneous cameras based on a preset frame loss threshold; and performing depth estimation on a target image using the dual cameras after focal length equivalence and frame stream synchronization. The present invention adopts an iterative field of view cropping method to synchronize the focal lengths of two cameras on a mobile device to achieve accurate epipolar correction. At the same time, the frame streams captured by the two cameras on the mobile device are synchronized to avoid inter-frame displacement of moving objects on the two cameras at the same time. On this basis, the depth of the target image is estimated using the dual cameras after focal length equivalence and frame stream synchronization, which can reduce the error of depth estimation.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a dual-camera depth estimation method and device based on focal length equivalence and frame stream synchronization. Background Art

[0002] In recent years, augmented reality (AR) and virtual reality (VR) applications on mobile devices have become increasingly popular, and real-time depth estimation technology for augmented reality (AR) and virtual reality (VR) on mobile devices has also received increasing attention. In the context of binocular depth estimation technology having been developed and rear dual cameras being widely installed on mobile devices, researchers hope to use rear dual cameras on mobile devices to achieve depth estimation.

[0003] However, after in-depth research and analysis, we found that using the rear dual cameras on mobile devices to achieve depth estimation faces the following challenges:

[0004] (1): The different focal lengths of the rear dual cameras lead to inaccurate epipolar correction. That is, the original intention of designing the rear dual cameras on mobile devices is to meet application scenarios such as macro photography and wide-angle photography. They have very different focal lengths. If epipolar correction is directly performed on the images taken by the dual cameras, it is very likely that the corrected images will be distorted, affecting the depth estimation effect.

[0005] (2): The frame streams captured by the rear dual cameras are not synchronized, resulting in poor parallax calculation accuracy. That is, due to the slight difference in the frame period of the rear dual cameras, the frame streams of the rear dual cameras are highly asynchronous during operation. When the frame streams of the rear dual cameras are not synchronized, the captured images are displaced in the frames. This displacement will lead to inaccurate parallax calculation, which in turn affects the depth estimation effect.

[0006] At present, no technology has been proposed that can overcome the above-mentioned difficulties. Summary of the invention

[0007] The purpose of the present invention is to provide a dual-camera depth estimation method and device based on focal length equivalence and frame stream synchronization, so as to solve the problem of poor depth estimation effect caused by different focal lengths of dual cameras on mobile devices and asynchronous frame streams, and improve the accuracy of depth estimation.

[0008] In a first aspect, the present invention provides a dual-camera depth estimation method based on focal length equivalence and frame stream synchronization, wherein the dual cameras are heterogeneous cameras disposed on a mobile device, and the method comprises:

[0009] Performing focal length equivalence on the dual cameras according to an iterative FoV cropping technique;

[0010] Based on a preset frame loss threshold, synchronizing the frame streams of the heterogeneous cameras;

[0011] The depth of the target image is estimated by using dual cameras with equivalent focal length and synchronized frame streams.

[0012] According to a dual-camera depth estimation method based on focal length equivalence and frame stream synchronization provided by the present invention, the dual cameras are subjected to focal length equivalence according to an iterative FoV cropping technique, comprising:

[0013] Using the dual cameras, collecting K pairs of chessboard images;

[0014] Filter out the image pairs whose reprojection errors are greater than a preset reprojection error threshold value among the K pairs of checkerboard images, and use the remaining image pairs as original image pairs;

[0015] Determining a cropping factor based on an iterative FoV cropping technique and the original image pair;

[0016] Using the crop factor, digitally zooming the wide field of view camera in the dual cameras to make the focal lengths of the dual cameras equivalent;

[0017] Wherein, K is a first preset value.

[0018] According to a dual-camera depth estimation method based on focal length equivalence and frame stream synchronization provided by the present invention, the determining of the cropping factor according to the iterative FoV cropping technology and the original image pair includes:

[0019] Step A: Initialize the crop factor;

[0020] Step B: calibrating the image with narrow field of view in the original image pair by using Zhang Zhengyou calibration method to obtain the focal length corresponding to the narrow field of view camera in the dual camera;

[0021] Step C: In each iteration, taking the center of the field of view as a base point, the image with the wide field of view in the original image pair is cropped in all directions according to the cropping factor;

[0022] Step D: Calibrate the cropped image using the Zhang Zhengyou calibration method to obtain the focal length corresponding to the wide field of view camera in the dual camera of this iteration;

[0023] Step E: inputting the focal length corresponding to the narrow field of view camera and the focal length corresponding to the wide field of view camera in this iteration into a preset focal length difference loss function to obtain the focal length difference loss function value of this iteration;

[0024] Step F: If the focal length difference loss function values ​​of consecutive G rounds of iterations are all less than the preset loss function value, the current cropping factor is output; otherwise, the cropping factor is updated and the process returns to step C;

[0025] Wherein, G is a second preset value.

[0026] According to a dual-camera depth estimation method based on focal length equivalence and frame stream synchronization provided by the present invention, the expression of the focal length difference loss function is as follows:

[0027]

[0028] In the above formula, J(α i ) is the focal length difference loss function value of the i-th iteration, α i is the reduction factor at the i-th iteration, is the focal length of the wide-field camera corresponding to the i-th iteration, f T is the focal length corresponding to the narrow field of view camera.

[0029] According to a dual-camera depth estimation method based on focal length equivalence and frame stream synchronization provided by the present invention, the updating of the clipping factor includes:

[0030] If the difference between the focal length corresponding to the wide field of view camera and the focal length corresponding to the narrow field of view camera in this iteration is greater than 0, the cropping factor is updated to the difference between the cropping factor and the preset step length; otherwise, the cropping factor is updated to the sum of the cropping factor and the preset step length.

[0031] According to a dual-camera depth estimation method based on focal length equivalence and frame stream synchronization provided by the present invention, the frame stream synchronization of the heterogeneous cameras is performed based on a preset frame loss threshold, including:

[0032] Two frames of images with the same index value in the frame streams of two cameras are regarded as a pair of matching frames; wherein the index value represents the order of the corresponding frame of image in the frame stream;

[0033] Search the target matching frame in a timed manner from the direction of ascending index value;

[0034] When a target matching frame is found, a frame of image with a preceding capture timestamp in the target matching frame is discarded, and the capture timestamps of the frames of image after the discarded frame are updated; wherein the target matching frame is the first matching frame that meets the preset conditions;

[0035] The preset condition is that the difference between the capture timestamps of two frames of images is greater than a preset frame loss threshold.

[0036] According to a dual-camera depth estimation method based on focal length equivalence and frame stream synchronization provided by the present invention, when the mobile device is an Android device, while performing frame stream synchronization on the heterogeneous cameras, the method further includes:

[0037] Using Java reflection mechanism, call the private close() member function in the CameraMetadataNative instance to release the metadata held by the CameraMetadataNative instance;

[0038] The metadata includes: frame rate, image size, shooting time, color depth, camera image distance and camera controllable properties.

[0039] In a second aspect, the present invention provides a dual-camera depth estimation device based on focal length equivalence and frame stream synchronization, wherein the dual cameras are heterogeneous cameras disposed on a mobile device, and the device comprises:

[0040] A focal length equivalence module, used for performing focal length equivalence on the dual cameras according to an iterative FoV cropping technique;

[0041] A frame stream synchronization module, used for performing frame stream synchronization on the heterogeneous cameras based on a preset frame loss threshold;

[0042] The depth estimation module is used to perform depth estimation on the target image using dual cameras after focal length equivalence and frame stream synchronization.

[0043] In a third aspect, the present invention further discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the dual-camera depth estimation method based on focal length equivalence and frame stream synchronization as described in the first aspect is implemented.

[0044] In a fourth aspect, the present invention further discloses a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the dual-camera depth estimation method based on focal length equivalence and frame stream synchronization as described in the first aspect is implemented.

[0045] The dual-camera depth estimation method and device based on focal length equivalence and frame stream synchronization provided by the present invention adopt an iterative field of view cropping method to synchronize the focal lengths of two cameras on a mobile device to achieve accurate epipolar correction. At the same time, the frame streams captured by the two cameras on the mobile device are synchronized to avoid inter-frame displacement of moving objects on the same mobile device. On this basis, the depth of the target image is estimated using the dual cameras after focal length equivalence and frame stream synchronization, which can reduce the error of depth estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 It is a schematic diagram of a dual camera system provided by the prior art;

[0048] Figure 2 It is a flow chart of a dual-camera depth estimation method based on focal length equivalence and frame stream synchronization provided by the present invention;

[0049] Figure 3 is a diagram of the relationship between focal length and field of view provided by the present invention;

[0050] Figure 4 It is a schematic diagram of the principle of the iterative FoV clipping technology provided by the present invention;

[0051] Figure 5 It is a schematic diagram of the frame stream synchronization principle provided by the present invention;

[0052] Figure 6 It is a structural diagram of a dual-camera depth estimation device based on focal length equivalence and frame stream synchronization provided by the present invention;

[0053] Figure 7 It is a schematic diagram of the structure of an electronic device for implementing a dual-camera depth estimation method based on focal length equivalence and frame stream synchronization provided by the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Figure 1 The dual-camera system is demonstrated to simulate the principle of human vision for depth estimation in the 3D world. Figure 1As shown, the two cameras are in the same direction at different positions and the distance is B. According to the binocular depth estimation technology, there are several prerequisites for obtaining the accurate depth Z of point P (xw, yw, zw): 1) The focal length f of the two cameras should be equal, which ensures that the two cameras have the same field of view, so that the imaging points PL (ul, vl) and PR (ur, vr) are located on the same epipolar line. (2) The two cameras need to capture frames at the same time, which avoids the displacement of moving objects between the same pair of frames, which is the basis for accurate disparity estimation. (3) Accurately estimate the disparity dL-dR between the two points PL and PR.

[0056] If the above conditions are met, the depth Z can be obtained by the following formula:

[0057]

[0058] This formula shows that depth can be easily calculated based on the accurate parallax and equivalent focal length of the dual cameras.

[0059] In view of this, the present invention provides a dual-camera depth estimation method based on focal length equivalence and frame stream synchronization, such as Figure 2 As shown, the dual cameras are heterogeneous cameras placed on a mobile device, and the method includes:

[0060] Performing focal length equivalence on the dual cameras according to an iterative FoV cropping technique;

[0061] It is understandable that the original intention of designing dual cameras on existing mobile devices is to meet application scenarios such as macro photography and wide-angle photography, and they have very different focal lengths. The focal length determines the field of view (FoV) of the camera, that is, cameras with different focal lengths have different fields of view, and the sizes of the captured images are different. In this field, when performing depth estimation on moving objects, the images of moving objects taken by dual cameras need to be subjected to epipolar correction in the preparatory stage. If the images of objects taken by dual cameras with different focal lengths are directly corrected, the huge difference in the focal lengths of the dual cameras may cause the corrected images to be distorted, resulting in a catastrophic epipolar correction effect. Therefore, if you want to implement depth estimation on a mobile device, it is extremely necessary to design a method that can zoom the dual cameras to make the focal lengths of the dual cameras equivalent.

[0062] The iterative FoV cropping technology provided by the present invention performs field of view cropping in an iterative manner to crop the fields of view of the dual cameras to be consistent, thereby making the focal lengths of the dual cameras consistent.

[0063] Based on a preset frame loss threshold, synchronizing the frame streams of the heterogeneous cameras;

[0064] It is understandable that for dual cameras on mobile devices, even if the dual cameras are set to have the same frame rate, the frame periods of the dual cameras (i.e., the time interval between consecutive frames) are not completely equal. For example: the frame rate of the dual cameras is set to 30 frames per second, but during operation, it will be found that the average frame length of the main camera is 33.33ms, and the average frame length of the secondary camera is 33.31ms, that is, there is an average time difference of 0.02ms per frame. This difference will continue to accumulate as the dual cameras run, resulting in highly asynchronous frame streams of the dual cameras. When two frames are taken at different times, the same key point (for example, the left hand) may move dozens of pixels in the moving scene, which makes it difficult to accurately calculate the parallax. Therefore, it is particularly necessary to ensure the synchronization of the frame streams of the dual cameras.

[0065] The present invention sets a frame loss threshold to filter out frames with obviously different time generation, thereby ensuring synchronization of frame streams of dual cameras.

[0066] The depth of the target image is estimated by using dual cameras with equivalent focal length and synchronized frame streams.

[0067] The dual-camera depth estimation method based on focal length equivalence and frame stream synchronization provided by the present invention adopts an iterative field of view cropping method to synchronize the focal lengths of two cameras on a mobile device to achieve accurate epipolar correction. At the same time, the frame streams captured by the two cameras on the mobile device are synchronized to avoid inter-frame displacement of moving objects on the same mobile device. On this basis, the depth of the target image is estimated using the dual cameras after focal length equivalence and frame stream synchronization, which can reduce the error of depth estimation.

[0068] Based on the above embodiments, as an optional embodiment, performing focal length equivalence on the dual cameras according to the iterative FoV cropping technology includes:

[0069] Using the dual cameras, collecting K pairs of chessboard images;

[0070] Figure 3 The relationship between focal length and field of view is shown as follows: Figure 3 As shown in Figure 1, the focal lengths of the dual cameras on a mobile device are usually different, resulting in different fields of view (FoV) of the dual cameras. Specifically, according to the lens imaging rule, the field of view of a camera is determined by the focal length of the camera. The camera with a shorter focal length has a larger field of view (FoV). W There is a wide field of view WFoV, a camera with a long focal length f T There is a narrow field of view TFoV. Therefore, by cropping WFoV to make it equal to TFoV, the focal length of both cameras can be made equal to f T Therefore, the problem of equivalent focal lengths of dual cameras is converted into the problem of optimal selection of cropping factors.

[0071] The present invention captures K pairs of calibration pattern images (ie, checkerboard images) at different positions, laying a foundation for accurate acquisition of the cropping factor.

[0072] Filter out the image pairs whose reprojection errors are greater than a preset reprojection error threshold value among the K pairs of checkerboard images, and use the remaining image pairs as original image pairs;

[0073] It is conceivable that the reprojection error threshold is a value selected based on the working conditions. For example, 20 images of a 12×9 checkerboard placed on a 20mm×20mm flat plate are collected, and image pairs with a reprojection error exceeding 0.2 pixels are filtered out.

[0074] The present invention filters out image pairs with large reprojection errors to improve the calibration accuracy of the dual-camera focal lengths.

[0075] Determining a cropping factor based on an iterative FoV cropping technique and the original image pair;

[0076] Existing methods usually use image matching algorithms to crop the field of view, such as SIFT, SURF and ORB. However, they only perform homography transformation, and the cropping accuracy is not satisfactory. In this regard, the present invention analyzes the feasibility of finding the equivalent focal length by cropping the field of view, and then proposes a technology for iteratively cropping the FoV until the focal lengths of the two cameras are equal, where the corresponding cropping factor when the focal lengths of the two cameras are equal is the optimal cropping factor to be obtained.

[0077] Using the crop factor, digitally zooming the wide field of view camera in the dual cameras to make the focal lengths of the dual cameras equivalent;

[0078] It is understandable that after digital zoom, the object images captured by the two cameras have the same frame. At this time, the object images can be passed to the cvStereoRectify function in OpenCV for accurate epipolar correction.

[0079] Wherein, K is a first preset value.

[0080] The present invention designs an iterative FoV cropping technology to synchronize the focal lengths of two cameras on a mobile device to improve the accuracy of epipolar correction.

[0081] Based on the above embodiments, as an optional embodiment, determining the cropping factor according to the iterative FoV cropping technology and the original image pair includes:

[0082] Step A: Initialize the crop factor;

[0083] Step B: calibrating the image with narrow field of view in the original image pair by using Zhang Zhengyou calibration method to obtain the focal length corresponding to the narrow field of view camera in the dual camera;

[0084] Step C: In each iteration, taking the center of the field of view as a base point, the image with the wide field of view in the original image pair is cropped in all directions according to the cropping factor;

[0085] Step D: Calibrate the cropped image using the Zhang Zhengyou calibration method to obtain the focal length corresponding to the wide field of view camera in the dual camera of this iteration;

[0086] Step E: inputting the focal length corresponding to the narrow field of view camera and the focal length corresponding to the wide field of view camera in this iteration into a preset focal length difference loss function to obtain the focal length difference loss function value of this iteration;

[0087] Step F: If the focal length difference loss function values ​​of consecutive G rounds of iterations are all less than the preset loss function value, the current cropping factor is output; otherwise, the cropping factor is updated and the process returns to step C;

[0088] Wherein, G is a second preset value.

[0089] It is understandable that the present invention performs focal length equivalence on dual cameras on a mobile device in an offline manner.

[0090] Figure 4 The following is a schematic diagram of the iterative FoV cropping technology principle, which shows the workflow of iterative FoV cropping, as follows:

[0091] Step I: Set the crop factor α 0 Initialized to 1;

[0092] Step II: Use Zhang Zhengyou calibration method to calibrate the narrow field of view chessboard image to obtain the focal length of the narrow field of view camera;

[0093] Step III: According to α i , the wide field of view chessboard image is cropped in all directions with the center of the field of view as the base point, and the cropped image is calibrated using the Zhang Zhengyou calibration method to obtain the same i The focal length of the corresponding wide-field camera;

[0094] Step IV: Based on the preset focal length difference loss function and α i The focal length of the wide field camera and the focal length of the narrow field camera are used to determine whether the iteration termination condition is met. If so, then α i is the final reduction factor; if it is not satisfied, α i Updated to alpha i+1 , and return to step III.

[0095] Here, the iteration termination condition adopted by the present invention is that the focal length difference loss function values ​​of G consecutive iterations are all less than the preset loss function value. Among them, the preset loss function value Δ is generally set to 0.01; G is generally set to 5.

[0096] This embodiment iterates the FoV cropping technology, that is, it finds the cropping factor α that can make the focal lengths of the two cameras equivalent through iterative cropping, and then uses the cropping factor α to achieve image capture with the same frame, thereby laying the foundation for deep learning.

[0097] Based on the above embodiments, as an optional embodiment, the expression of the focal length difference loss function is as follows:

[0098]

[0099] In the above formula, J(α i ) is the focal length difference loss function value of the i-th iteration, α i is the reduction factor at the i-th iteration, is the focal length of the wide-field camera corresponding to the i-th iteration, f T is the focal length corresponding to the narrow field of view camera.

[0100] The present invention designs a focal length difference loss function to quantify the focal length difference loss of the dual cameras in each iteration, and minimizes the focal length difference loss through iteration to find the optimal cropping factor.

[0101] Based on the above embodiments, as an optional embodiment, the updating of the clipping factor includes:

[0102] If the difference between the focal length corresponding to the wide field of view camera and the focal length corresponding to the narrow field of view camera in this iteration is greater than 0, the cropping factor is updated to the difference between the cropping factor and the preset step length; otherwise, the cropping factor is updated to the sum of the cropping factor and the preset step length.

[0103] In the sight of the present invention, the preset step size is usually set to 0.01.

[0104] Based on the above embodiments, as an optional embodiment, the frame stream synchronization of the heterogeneous cameras based on a preset frame loss threshold includes:

[0105] Two frames of images with the same index value in the frame streams of two cameras are regarded as a pair of matching frames; wherein the index value represents the order of the corresponding frame of image in the frame stream;

[0106] Search the target matching frame in a timed manner from the direction of ascending index value;

[0107] When a target matching frame is found, a frame of image with a preceding capture timestamp in the target matching frame is discarded, and the capture timestamps of the frames of image after the discarded frame are updated; wherein the target matching frame is the first matching frame that meets the preset conditions;

[0108] The preset condition is that the difference between the capture timestamps of two frames of images is greater than a preset frame loss threshold.

[0109] In order to obtain time-aligned frames, additional external hardware signals are usually used to simultaneously trigger two cameras to capture images. However, on most mobile devices, there is no such dedicated hardware to trigger the camera. Therefore, setting the same frame rate for dual cameras to capture time-aligned frames is still the current mainstream trend. As mentioned before, even under the same frame rate setting, the actual frame rates of the dual cameras are slightly different. If not carefully synchronized, the time difference of the frames captured by heterogeneous cameras will accumulate during runtime. To this end, the present invention provides a simple and effective technology to reduce the time difference between heterogeneous camera frames.

[0110] Figure 5 This is a schematic diagram of the frame stream synchronization principle, such as Figure 5 As shown in the figure, TL and TR represent the frame periods of the main camera and the secondary camera respectively. The small circles in the figure represent frames, and the numbers above the circles represent the order of the frames in the frame stream. The synchronization of camera frame streams depends on the timestamps of each frame captured by the two cameras.

[0111] Specifically, the first step is to treat the two frames with the same sequence in the main camera frame stream and the secondary camera frame stream as a frame pair. The second step is to compare the timestamps of the two frames with the same sequence in the main camera frame stream and the secondary camera frame stream. If the timestamp difference of a frame pair is less than the threshold i, the two frames are regarded as a pair of matching frames. Figure 5 Indicated by a solid line in , this pair of matching frames is used as the input for the next step (image correction). If the timestamp difference of a frame pair exceeds the threshold θ, the frame in the faster frame stream of this frame pair is discarded. Step 3: Return to the first step to compare the frame after the discarded frame with the frame in the other frame stream. That is, before being discarded, if the n-th frame pair in sequence is not a matching frame and the main camera frame stream is faster, then the n-th frame in the main camera frame stream is discarded, and the (n+1)-th frame in the main camera frame stream is used to match the n-th frame in the secondary camera frame stream.

[0112] It can be understood that, assuming that the main camera frame stream is faster than the secondary camera frame stream (that is, the secondary camera catches up with the main camera), after discarding a frame in the main camera frame stream, the secondary camera frame stream will become faster than the main camera frame stream (that is, the main camera catches up with the secondary camera), and the cycle will repeat. The timestamp difference of the matching frames will always be constrained within the preset frame loss threshold range, and the frame stream synchronization is almost completed.

[0113] It should be noted that by fine-tuning the frame loss threshold θ, we have made a trade-off between synchronization of the frame stream and frame loss. The larger the value of i, the more frames can be retained, but the time difference of the frames may become larger, which will cause inaccurate depth estimation due to the displacement of the object. Conversely, the smaller the value of i, the smaller the number of frames that can be retained, but the time difference of the frames may become smaller, and the depth estimation is accurate. The value of i should be determined according to the actual application scenario. For example, in a scene where the target object is mostly static, we can use a larger i to tolerate slightly asynchronous frames. If the target object moves faster, the value of θ should be smaller to ensure higher accuracy.

[0114] The present invention synchronizes frame streams captured by two cameras on a mobile device to avoid inter-frame displacement of a moving object on the same mobile device, thereby improving the accuracy of depth estimation.

[0115] Based on the above embodiments, as an optional embodiment, when the mobile device is an Android device, while synchronizing the frame streams of the heterogeneous cameras, the method further includes:

[0116] Using the Java reflection mechanism, the private close() member function in the CameraMetadataNative instance is called to release the metadata held by the CameraMetadataNative instance; wherein the metadata includes: frame rate, image size, shooting time, color bit depth, camera image distance and camera controllable properties.

[0117] While studying frame stream synchronization, researchers found that for the Android system, obtaining frame streams from dual cameras would frequently trigger Android garbage collection (GC) events, causing frame cycle fluctuations of the dual cameras. By tracking memory usage, we found that frequent GCs were due to unrecycled metadata. That is, after running for a period of time, unrecycled metadata will occupy all pre-allocated memory, and the Android system will trigger GC to release memory. In order to solve this problem, the present invention automatically releases metadata before triggering GC to reduce the number of GCs, thereby reducing frame cycle fluctuations of the dual cameras.

[0118] Specifically, when an Android system APP sends a CaptureRequest instance to the Camera2 API, the Camera2API returns a CaptureResult instance: the CaptureResult instance contains a CameraMetadataNative instance, and the class of this instance overrides the finalize() function. For a class that overrides the finalize() function, when the instance of the class is not referenced by any variable, the instance will be added to the FalizerDaemon queue, waiting for memory to be reclaimed. However, if GC does not occur for a period of time and the queue is empty, FalizerDaemon will be blocked, making it impossible for the newly added CameraMetadataNative instance to be reclaimed, resulting in a continuous increase in memory, which in turn triggers frequent GC. Only after GC occurs, the Android system's FalizerDaemon will call the instance's falize() function, thereby indirectly calling the close() function to release the metadata.

[0119] Since the useful metadata in the CameraMetadataNative instance is only related to the current frame, the invalid metadata can be released in time. Considering that finalize() is a private API function that is not exposed to the outside and that the Finalize() function calls the close() function to release the metadata, the present invention uses a Java reflection method to directly call the close() function to release the metadata in the CameraMetadataNative instance.

[0120] The present invention timely publishes metadata created by Android to avoid frequent triggering of garbage collection (GC), thereby reducing frame cycle fluctuations of dual cameras and further reducing the probability of asynchronous frame streams of the dual cameras.

[0121] In the second aspect, the dual-camera depth estimation device based on focal length equivalence and frame stream synchronization provided by the present invention is described. The dual-camera depth estimation device based on focal length equivalence and frame stream synchronization described below and the dual-camera depth estimation method based on focal length equivalence and frame stream synchronization described above can be referenced to each other. Figure 6 A schematic diagram of the structure of a dual-camera depth estimation device based on focal length equivalence and frame stream synchronization is illustrated, wherein the device in the figure includes:

[0122] A focal length equivalent module 21, used for performing focal length equivalent on the dual cameras according to an iterative FoV cropping technique;

[0123] A frame stream synchronization module 22, configured to synchronize the frame streams of the heterogeneous cameras based on a preset frame loss threshold;

[0124] The depth estimation module 23 is used to perform depth estimation on the target image using the dual cameras after focal length equivalence and frame stream synchronization.

[0125] The dual-camera depth estimation device based on focal length equivalence and frame stream synchronization provided by the present invention adopts an iterative field of view cropping method to synchronize the focal lengths of two cameras on a mobile device to achieve accurate epipolar correction. At the same time, the frame streams captured by the two cameras on the mobile device are synchronized to avoid inter-frame displacement of moving objects on the same mobile device. On this basis, the depth of the target image is estimated using the dual cameras after focal length equivalence and frame stream synchronization, which can reduce the error of depth estimation.

[0126] Thirdly, Figure 7 The following is a schematic diagram of the physical structure of an electronic device. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute a dual-camera depth estimation method based on focal length equivalence and frame stream synchronization, wherein the dual cameras are heterogeneous cameras placed on a mobile device, and the method includes: performing focal length equivalence on the dual cameras according to an iterative FoV cropping technique; performing frame stream synchronization on the heterogeneous cameras based on a preset frame loss threshold; and performing depth estimation on a target image using the dual cameras after focal length equivalence and frame stream synchronization.

[0127] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0128] In a fourth aspect, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, it executes a dual-camera depth estimation method based on focal length equivalence and frame stream synchronization. The dual cameras are heterogeneous cameras placed on a mobile device. The method includes: performing focal length equivalence on the dual cameras according to iterative FoV cropping technology; performing frame stream synchronization on the heterogeneous cameras based on a preset frame loss threshold; and performing depth estimation on the target image using the dual cameras after focal length equivalence and frame stream synchronization.

[0129] In a fifth aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon for executing a dual-camera depth estimation method based on focal length equivalence and frame stream synchronization, wherein the dual cameras are heterogeneous cameras placed on a mobile device, and the method comprises: performing focal length equivalence on the dual cameras according to an iterative FoV cropping technique; performing frame stream synchronization on the heterogeneous cameras based on a preset frame loss threshold; and performing depth estimation on a target image using the dual cameras after focal length equivalence and frame stream synchronization.

[0130] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0131] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dual-camera depth estimation method based on focal length equivalence and frame stream synchronization, characterized in that: The dual cameras are heterogeneous cameras placed on a mobile device, and the method includes: According to the iteration Cropping technology to make the focal length of the dual cameras equivalent; Based on a preset frame loss threshold, synchronizing the frame streams of the heterogeneous cameras; Using dual cameras with equivalent focal length and synchronized frame streams, the depth of the target image is estimated. The iteration The cropping technology performs focal length equivalence on the dual cameras, including: Using the dual cameras, collecting K pairs of chessboard images; Filter out the image pairs whose reprojection errors are greater than a preset reprojection error threshold value among the K pairs of checkerboard images, and use the remaining image pairs as original image pairs; According to the iteration Cropping technique and said original image pair, determining a cropping factor; Using the crop factor, digitally zooming the wide field of view camera in the dual cameras to make the focal lengths of the dual cameras equivalent; Wherein, K is a first preset value; The iteration The cropping technique and the original image pair, determining the cropping factor, include: Step A: Initialize the crop factor; Step B: calibrating the image with narrow field of view in the original image pair by using Zhang Zhengyou calibration method to obtain the focal length corresponding to the narrow field of view camera in the dual camera; Step C: In each iteration, taking the center of the field of view as a base point, the image with the wide field of view in the original image pair is cropped in all directions according to the cropping factor; Step D: Calibrate the cropped image using the Zhang Zhengyou calibration method to obtain the focal length corresponding to the wide field of view camera in the dual camera of this iteration; Step E: inputting the focal length corresponding to the narrow field of view camera and the focal length corresponding to the wide field of view camera in this iteration into a preset focal length difference loss function to obtain the focal length difference loss function value of this iteration; Step F: If the focal length difference loss function values ​​of consecutive G rounds of iterations are all less than the preset loss function value, the current cropping factor is output; otherwise, the cropping factor is updated and the process returns to step C; Wherein, G is a second preset value; The step of synchronizing the frame streams of the heterogeneous cameras based on a preset frame loss threshold comprises: Two frames of images with the same index value in the frame streams of two cameras are regarded as a pair of matching frames; wherein the index value represents the order of the corresponding frame of image in the frame stream; Search the target matching frame in a timed manner from the direction of ascending index value; When a target matching frame is found, a frame of image with a preceding capture timestamp in the target matching frame is discarded, and the capture timestamps of the frames of image after the discarded frame are updated; wherein the target matching frame is the first matching frame that meets the preset conditions; The preset condition is that the difference between the capture timestamps of two frames of images is greater than a preset frame loss threshold.

2. The dual-camera depth estimation method based on focal length equivalence and frame stream synchronization according to claim 1, characterized in that: The expression of the focal length difference loss function is as follows: ； In the above formula, For the The focal length difference loss function value of the iteration is: For the The reduction factor at the iteration, For the The focal length corresponding to the wide field of view camera of the iteration, is the focal length corresponding to the narrow field of view camera.

3. The dual-camera depth estimation method based on focal length equivalence and frame stream synchronization according to claim 1, characterized in that: The updating of the clipping factor comprises: If the difference between the focal length corresponding to the wide field of view camera and the focal length corresponding to the narrow field of view camera in this iteration is greater than 0, the cropping factor is updated to the difference between the cropping factor and the preset step length; otherwise, the cropping factor is updated to the sum of the cropping factor and the preset step length.

4. The dual-camera depth estimation method based on focal length equivalence and frame stream synchronization according to claim 1, characterized in that: When the mobile device is an Android device, while synchronizing the frame streams of the heterogeneous cameras, the method further includes: Using the Java reflection mechanism, the private close() member function in the CameraMetadataNative instance is called to release the metadata held by the CameraMetadataNative instance; wherein the metadata includes: frame rate, image size, shooting time, color bit depth, camera image distance and camera controllable properties.

5. A dual-camera depth estimation device based on focal length equivalence and frame stream synchronization, characterized in that: The dual cameras are heterogeneous cameras placed on a mobile device, and the device comprises: Focal length equivalent module, used to iterate The cropping technology performs focal length equivalence on the dual cameras; the performing focal length equivalence on the dual cameras according to the iterative cropping technology includes: Using the dual cameras, collecting K pairs of chessboard images; Filter out the image pairs whose reprojection errors are greater than a preset reprojection error threshold value among the K pairs of checkerboard images, and use the remaining image pairs as original image pairs; Determining a cropping factor based on an iterative cropping technique and the original image pair; Using the crop factor, digitally zooming the wide field of view camera in the dual cameras to make the focal lengths of the dual cameras equivalent; Wherein, K is a first preset value; The determining of the cropping factor according to the iterative cropping technique and the original image pair comprises: Step A: Initialize the crop factor; Step B: calibrating the image with narrow field of view in the original image pair by using Zhang Zhengyou calibration method to obtain the focal length corresponding to the narrow field of view camera in the dual camera; Step C: In each iteration, taking the center of the field of view as a base point, the image with the wide field of view in the original image pair is cropped in all directions according to the cropping factor; Step D: Calibrate the cropped image using the Zhang Zhengyou calibration method to obtain the focal length corresponding to the wide field of view camera in the dual camera of this iteration; Step E: inputting the focal length corresponding to the narrow field of view camera and the focal length corresponding to the wide field of view camera in this iteration into a preset focal length difference loss function to obtain the focal length difference loss function value of this iteration; Step F: If the focal length difference loss function values ​​of consecutive G rounds of iterations are all less than the preset loss function value, the current cropping factor is output; otherwise, the cropping factor is updated and the process returns to step C; Wherein, G is a second preset value; A frame stream synchronization module is used to synchronize the frame streams of the heterogeneous cameras based on a preset frame loss threshold. The frame stream synchronization of the heterogeneous cameras based on the preset frame loss threshold includes: Two frames of images with the same index value in the frame streams of two cameras are regarded as a pair of matching frames; wherein the index value represents the order of the corresponding frame of image in the frame stream; Search the target matching frame in a timed manner from the direction of ascending index value; When a target matching frame is found, a frame of image with a preceding capture timestamp in the target matching frame is discarded, and the capture timestamps of the frames of image after the discarded frame are updated; wherein the target matching frame is the first matching frame that meets the preset conditions; The preset condition is that the difference between the capture timestamps of the two frames of images is greater than a preset frame loss threshold; The depth estimation module is used to perform depth estimation on the target image using dual cameras after focal length equivalence and frame stream synchronization.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the dual-camera depth estimation method based on focal length equivalence and frame stream synchronization is implemented as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the dual-camera depth estimation method based on focal length equivalence and frame stream synchronization is implemented as described in any one of claims 1 to 4.

Images