Image processing method of binocular camera system constructed based on intelligent terminal

Abstract

The invention discloses an image processing method of a binocular camera system constructed based on an intelligent terminal, the image processing method is applied to the intelligent terminal and an extended camera, the intelligent terminal comprises a memory, a processor, a depth information processing module and a camera assembly, the camera assembly comprises at least one camera, the depth information processing module is used for running on the processor, and the extended camera is used for extending the camera. The memory is used for storing the depth information processing module; the method comprises the steps that the camera and the extended camera form a binocular camera system; acquiring calibration parameters of the binocular camera system, and inputting the calibration parameters into a depth information processing module; the binocular camera system shoots a target object to obtain a first target image and a second target image, and inputs the first target image and the second target image into the depth information processing module; a depth information processing module may generate depth information. According to the invention, the image is processed by using the computing power of the intelligent terminal, and the depth information is obtained.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an image processing method of a binocular camera system constructed based on an intelligent terminal. Background Art

[0002] In related technologies, binocular cameras often require an embedded high-computing-power processor to calculate depth values, which increases the cost of binocular camera products. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, one object of the present invention is to propose an image processing method for a binocular camera system constructed based on a smart terminal, construct a binocular camera system based on a camera and an extended camera of the smart terminal, and use the computing power of the smart terminal itself to process the images of the two to generate more accurate and real-time depth information.

[0004] The present invention provides an image processing method for a binocular camera system built based on an intelligent terminal, which is applied to the intelligent terminal and an extended camera. The intelligent terminal includes a memory, a processor, a depth information processing module and a camera component. The camera component includes at least one camera. The depth information processing module is used to run on the processor, and the memory is used to store the depth information processing module. The method includes:

[0005] The camera and the extended camera form a binocular camera system;

[0006] Acquire calibration parameters of the binocular camera system, and input the calibration parameters into the depth information processing module;

[0007] The binocular camera system photographs the target object to obtain a first target image and a second target image, and inputs the first target image and the second target image into the depth information processing module;

[0008] The depth information processing module generates a disparity value according to the first target image and the second target image and generates depth information using the disparity value and the calibration parameter.

[0009] In some embodiments, the smart terminal and the extended camera form a binocular camera system, including:

[0010] The extended camera is communicatively connected to the smart terminal;

[0011] Adjusting the positions of the extended camera and the camera head;

[0012] The pixels of the images taken by the extended camera and the camera are made consistent, and the rows of the images taken by the extended camera and the camera are aligned.

[0013] In some embodiments, aligning the image rows captured by the extended camera and the camera head includes:

[0014] The camera is used to obtain a first alignment image, and the extended camera is used to obtain a second alignment image;

[0015] The first alignment image and the second alignment image are line aligned.

[0016] In some embodiments, the smart terminal includes a display device, and the display device is used to display the first alignment image and the second alignment image;

[0017] The aligning the first alignment image and the second alignment image row by row further comprises:

[0018] The lens of the extended camera and the lens of the camera head are located on the same physical plane;

[0019] The extended camera and the camera head are aligned vertically, and corresponding pixel points of the same object in the first aligned image and the second aligned image are aligned in a vertical direction.

[0020] In some embodiments, the smart terminal further includes an auxiliary alignment module; the auxiliary alignment module is used to obtain the overlap between the first alignment image and the second alignment image, so as to assist in aligning the extended camera and the camera.

[0021] In some embodiments, the depth information processing module has a first preset baseline value and a second preset baseline value; the positions of the extended camera and the camera are adjusted so that the distance between the extended camera and the camera is one of the first preset baseline value and the second preset baseline value, thereby enabling measurement of depth information at different distances.

[0022] In some embodiments, obtaining calibration parameters of the binocular camera system includes:

[0023] Calibrate based on a calibration plate of known size;

[0024] Obtaining a calibration intrinsic parameter matrix and a calibration extrinsic parameter matrix of the binocular camera system;

[0025] Epipolar line correction for stereo matching is performed according to the intrinsic parameter matrix and the extrinsic parameter matrix.

[0026] In some embodiments, the depth information processing module includes:

[0027] An image preprocessing module, the image preprocessing module is used to perform grayscale, denoising and histogram equalization processing on the first target image and the second target image to obtain a first preprocessed image and the second preprocessed image;

[0028] A stereo matching module, wherein the stereo matching module is used to perform stereo matching on the first pre-processed image and the second pre-processed image to obtain a corresponding plurality of feature points;

[0029] A disparity calculation module, wherein the disparity calculation module is used to obtain a disparity value of each of the feature points;

[0030] A depth calculation module is used to obtain a depth value corresponding to each feature point according to the disparity value and the calibration parameter.

[0031] In some embodiments, the depth information processing module further includes a depth map imaging module; the depth map imaging module is used to convert the depth value into a depth map.

[0032] In some embodiments, the binocular camera system also includes a fixed bracket and an extended power supply; the fixed bracket is used to support and fix the positions of the extended camera and the smart terminal; the extended power supply is configured to supply power to the extended camera when the power consumption of the extended camera exceeds a power supply threshold of the smart terminal.

[0033] It can be seen from the technical solution that the embodiments provided by the present invention have the following advantages: (1) By forming a binocular camera system with an extended camera and a camera in a smart terminal, the binocular camera system obtains images of the target object at different viewing angles, namely, the first target image and the second target image, thereby obtaining a disparity value, and using the disparity value information to assist depth estimation. This method can improve the accuracy of depth estimation of a smart terminal with only a single camera. (2) The image processing method provided by the present application can be applied to a binocular camera system constructed based on a smart terminal. The binocular camera system constructed based on the smart terminal and the processor embedded in the smart terminal perform data processing and calculation on the image of the target object, thereby obtaining the depth information of the target object. The computing power of the processor of most smart terminals is sufficient to cope with the calculation and processing of binocular disparity, which can greatly reduce costs and make it more convenient for users to use, with great flexibility in terms of usage scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0035] Figure 1 is a flow chart of an image processing method of a binocular camera system constructed based on a smart terminal according to an embodiment of the present invention;

[0036] Figure 2 This is a simplified model diagram of the binocular camera measuring depth values;

[0037] Figure 3 It is a geometric model diagram that uses triangulation to obtain depth values;

[0038] Figure 4 is a structural schematic diagram of a binocular camera system constructed based on a smart terminal according to an embodiment of the present invention;

[0039] Figure 5 is a flow chart of another image processing method of a binocular camera system constructed based on a smart terminal according to an embodiment of the present invention;

[0040] Figure 6 is a flow chart of another image processing method of a binocular camera system constructed based on a smart terminal according to an embodiment of the present invention;

[0041] Figure 7 is a schematic diagram of a first aligned image and a second aligned image in a row aligned state;

[0042] Figure 8 This is a flow chart of another image processing method of a binocular camera system constructed based on a smart terminal according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0044] The camera industry for smart terminals is constantly introducing new technologies and new features to meet consumers' demand for high-quality shooting experience. Technological innovations include higher pixels, advanced image processing technology, multi-camera systems, variable aperture technology, etc. Technological innovation continues to accelerate. Smart terminals include smartphones, tablets, laptops, smart watches, and smart TVs. The camera pixels on smart terminals are constantly improving, image processing technology is advancing, and multi-camera systems have become an industry development trend.

[0045] Specifically, with the popularization and application of AI technology, the cameras of smart terminals have also made important progress in intelligent recognition and scene optimization. Smart terminals with monocular cameras capture image information through one camera, based on two-dimensional plane imaging, with a simple structure and relatively low cost. Smart terminals with binocular cameras capture images through two cameras, simulate the binocular vision of the human eye, and use parallax to calculate the three-dimensional spatial coordinates of the target object to achieve stereoscopic vision.

[0046] Monocular cameras are widely used in the machine vision industry for tasks such as object recognition, position positioning, and 2D plane measurement. Stereo cameras are more suitable for scenarios that require 3D spatial information, such as robot navigation, object grasping, and autonomous driving.

[0047] Binocular cameras have achieved greater breakthroughs in 3D reconstruction and scene understanding, providing more precise spatial coordinate information and helping the system make more accurate decisions. They also show their unique advantages in some applications that require high depth information, such as face recognition and gesture recognition.

[0048] With the continuous development of machine vision technology, monocular and binocular cameras are also developing. The progress of algorithm optimization and image processing technology has gradually improved the recognition accuracy and stability of monocular cameras. Stereoscopic cameras have made continuous breakthroughs in hardware design and algorithm optimization, achieving higher spatial resolution and more accurate depth measurement.

[0049] In summary, the technical background of expanding monocular cameras into binocular cameras involves the technological innovation of the smart terminal camera industry, the technical differences between monocular and binocular cameras, the advantages and applications of binocular cameras, as well as technical challenges and future development. These factors have jointly promoted the advancement of smartphone camera technology, making it possible to achieve a higher quality shooting experience and more accurate spatial information.

[0050] At present, smart terminals include one or more cameras. When using smart terminals to measure the depth of target objects, the single-camera smart terminal has low depth measurement accuracy when the target object has no obvious texture and the illumination changes greatly; while for smart terminals with multiple cameras, the distances between different cameras are short, resulting in a short baseline and a small parallax value, which in turn reduces the accuracy of depth measurement.

[0051] At the same time, the binocular cameras on the market are usually composed of two specially designed cameras with a fixed structure and poor scalability. The binocular cameras need to be connected to a PC and can only be called through complex codes. In addition, the parallax calculation of the binocular cameras consumes a lot of computing resources and usually requires GPU / FPGA device acceleration. The integrated binocular cameras on the market often have high-computing power processor modules embedded in the products, which leads to a sharp increase in product costs.

[0052] Reference below Figure 1-Figure 8 An image processing method of a binocular camera system constructed based on a smart terminal according to an embodiment of the present invention is described.

[0053] Embodiment 1

[0054] like Figure 1 and Figure 2 As shown, this embodiment provides an image processing method for a binocular camera system built based on a smart terminal, and the method is applied to a smart terminal 1 and an extended camera 2.

[0055] The smart terminal 1 includes a depth information processing module 11 and a camera component 12, and the camera component includes at least one camera. It should be noted that the smart terminal here can be a smart phone, a tablet computer, a laptop computer, a smart watch, a smart TV, etc. These smart terminals are embedded with a processor that can perform data processing and calculation.

[0056] The image processing method of the binocular camera system 100 constructed based on the intelligent terminal includes the following steps:

[0057] S1: The camera and the extended camera 2 form a binocular camera system 100;

[0058] S2: Obtain calibration parameters of the binocular camera system 100, and input the calibration parameters into the depth information processing module 11;

[0059] S3: The binocular camera system 100 photographs the target object to obtain a first target image and a second target image, and inputs the first target image and the second target image into the depth information processing module 11;

[0060] S4: The depth information processing module 11 generates a disparity value according to the first target image and the second target image and generates depth information using the disparity value and calibration parameters.

[0061] In a specific application scenario, a camera of a smart terminal and an extended camera 2 form a binocular camera system 100, and the calibration parameters of the binocular camera system 100 need to be re-determined, and the calibration parameters at least include the actual baseline distance between the camera and the extended camera 2 (i.e., the distance between the camera and the extended camera 2), the focal length of the camera, and the focal length of the extended camera 2. The camera and the extended camera 2 both shoot the target object at the same time, the camera obtains a first target image, and the extended camera 2 obtains a second target image, and the processor of the smart terminal runs the depth information processing module 11, and the depth information processing module 11 generates a disparity value according to the first target image and the second target image, and then generates depth information according to the disparity value and the baseline.

[0062] In the embodiment of the present application, (1) the extended camera 2 and a camera in the smart terminal form a binocular camera system 100, and the binocular camera system 100 obtains images of the target object at different viewing angles, namely, the first target image and the second target image, thereby obtaining the disparity value, and using the disparity value information to assist depth estimation. This method can improve the accuracy of depth estimation of the smart terminal with only a single camera. (2) The image processing method provided in the present application can be applied to the binocular camera system 100 constructed based on the smart terminal, and the binocular camera system 100 constructed based on the smart terminal and the processor embedded in the smart terminal perform data processing and calculation on the image of the target object, thereby obtaining the depth information of the target object. The computing power of the processor of most smart terminals is sufficient to cope with the calculation and processing of binocular disparity, which can greatly reduce the cost, and can make it more convenient for users to use, and has great flexibility in terms of usage scenarios.

[0063] like Figure 4 As shown, further, the depth information processing module 11 includes an image preprocessing module, a stereo matching module, a disparity calculation module and a depth calculation module. The image preprocessing module is used to grayscale, denoise and histogram equalize the first target image and the second target image to obtain a first preprocessed image and a second preprocessed image; the stereo matching module is used to stereo match the first preprocessed image and the second preprocessed image to obtain a feature point pair. The disparity calculation module is used to obtain the disparity value of the feature point pair; the depth calculation module obtains the depth value corresponding to each feature point according to the disparity value and the calibration parameters.

[0064] The stereo matching module uses a stereo matching algorithm, such as semi-global matching or a deep learning-based stereo matching network, to calculate the disparity between the two images captured by the extended camera 2 and the camera. Stereo matching is a computing technique that identifies and matches corresponding feature points in two images by analyzing image pairs captured by two extended cameras 2 and the camera. The positions of these matching points have differences, namely disparity. Disparity information can be obtained by comparing the position differences of corresponding points in the left and right images, and disparity information is the key to achieving depth perception. Once an accurate disparity map is obtained through the stereo matching algorithm, the next step is to convert the disparity information into depth information, which can generate a corresponding depth map. The depth map reflects the distance information from each point in the scene to the extended camera 2 or the camera. The generated depth map can be used in a variety of applications, including three-dimensional reconstruction, augmented reality, obstacle detection in autonomous driving, etc. The depth map provides three-dimensional information of the scene, allowing the device to better understand its surrounding environment.

[0065] like Figure 2 and Figure 3 As shown in the figure, it is necessary to further explain that the disparity information and calibration parameters can be used to convert the disparity value into the depth value through triangulation. The figure shows the imaging model of the binocular camera. , is the center of the left and right apertures, and the box is the imaging plane. is the focal length. and is the coordinate of the imaging plane, according to the coordinate definition in the figure, It should be a negative number, so the distance marked in the figure is - . A binocular camera generally consists of two horizontally placed cameras, a left eye camera and a right eye camera. In the left and right binocular cameras, both cameras are considered as pinhole cameras. They are placed horizontally, which means that the aperture centers of both cameras are located on the x-axis. The distance between the two is called the baseline of the binocular camera (denoted as b), which is an important calibration parameter of the binocular camera system 100. Now, consider a spatial point P, which forms an image in the left eye camera and the right eye camera, denoted as , . Due to the existence of the camera baseline (the distance between the two cameras), the two imaging positions are different. Ideally, since the left and right cameras only have displacement on the x-axis, the image of P also only has differences on the x-axis (corresponding to the u-axis of the image). Its left coordinate is recorded as , the right coordinate is , the geometric relationship is shown on the right side of the figure above. and Similarity relationship:

[0066] =

[0067] After sorting, the disparity value, the focal length in the calibration parameters, and the baseline in the calibration parameters can be used to obtain the depth value through the following formula:

[0068] = ,d = -

[0069] in, is the focal length in the calibration parameters, b is the baseline in the calibration parameters (i.e., the distance between the two cameras), and d is the disparity value (i.e., the horizontal difference of the same object in the two images).

[0070] The present invention converts the image differences captured by the binocular camera system 100 into depth information through calibration, stereo matching, disparity calculation and triangulation, and then generates a depth map, providing an interface for processing various tasks, such as using an API interface on a smartphone. The tasks here include but are not limited to three-dimensional reconstruction, virtual reality imaging and autonomous driving tasks.

[0071] In a specific example, a deep learning module is stored in the memory, and the processor is used to run the deep learning module. The deep learning module includes a convolutional neural network that can automatically learn complex feature representations from raw data, reducing the need to manually design features. In addition, the use of deep learning modules on many visual tasks can improve accuracy. The visual tasks here include image classification, object detection, semantic segmentation, and stereo matching. It has good generalization capabilities and can adapt to different scenarios and conditions. With the development of GPUs and other specialized hardware on smart terminals, the training and reasoning processes of deep learning models can be further accelerated.

[0072] Embodiment 2

[0073] like Figure 5 As shown, the smart terminal and the extended camera 2 form a binocular camera system 100, and this step S1 includes the following specific steps:

[0074] S11: The extended camera 2 is connected to the smart terminal for communication, so that the image acquired by the extended camera 2 can be transmitted to the smart terminal. The extended camera 2 and the smart terminal can be connected by wire or wirelessly, and the specific connection method is not limited here.

[0075] S12: Adjust the positions of the extended camera 2 and the camera. In the related art, for a smartphone with multiple cameras, the baseline of two cameras is about 10 mm (that is, the distance between the two cameras is about 10 mm), and the distance between different cameras is close, resulting in a short baseline and a small parallax value, which in turn reduces the accuracy of depth measurement. In the present application, a camera of the smart terminal forms a binocular camera system 100 with the extended camera 2, and a certain baseline distance is maintained between the extended camera 2 and the camera, and the baseline distance is between 1 cm and 20 cm, and a larger parallax can be obtained to improve the accuracy of depth estimation.

[0076] S13: Make the pixels of the images taken by the extended camera 2 and the camera consistent, and align the rows of the images taken by the extended camera 2 and the camera.

[0077] Specifically, pixels are the basic unit of digital images. To make the pixels of the images taken by the extended camera 2 and the camera consistent means to keep the number of pixels in the horizontal and vertical directions the same. For example, if the image taken by the camera is 1920×1080 pixels (1920 pixels in the horizontal direction and 1080 pixels in the vertical direction), then the image taken by the extended camera 2 also needs to be adjusted to the same 1920×1080 pixels.

[0078] It should also be noted that image row alignment means that the pixels on the corresponding rows in the images taken by the extended camera 2 and the camera should represent the information at the same height position in the actual scene. Each row of the two images must strictly correspond, and for the target object in the actual scene, its projection information can be found on the corresponding rows of the two images. For example, for a can in the actual scene, the pixel points that appear in the 50th row of the image taken by the extended camera 2 must correspond to the pixels that appear in the image taken by the camera, and the corresponding pixel points should appear roughly at the position of the 50th row in the image taken by the camera, thereby satisfying the epipolar geometric relationship of stereo vision, so that the matching point pairs must be located in the same image row, reducing the two-dimensional search to one dimension, greatly improving the efficiency of stereo matching, and thus reducing the amount of computation of the processor.

[0079] Embodiment 3

[0080] like Figure 6 As shown, further, the image rows captured by the extended camera 2 and the camera are aligned, and this step S13 includes the following sub-steps:

[0081] S131: The camera is used to obtain a first alignment image, and the extended camera 2 is used to obtain a second alignment image;

[0082] S132: Align the first alignment image and the second alignment image row.

[0083] That is to say, by adjusting the hardware installation angle or software-assisted correction, the imaging planes of the camera and the extended camera 2 are placed on the same horizontal plane, ensuring that the same scene point in the two images only has a horizontal displacement (X-axis difference), eliminating the vertical (Y-axis) misalignment, thereby satisfying the epipolar geometric relationship of stereo vision, ensuring that the matching point pairs are located in the same image row, reducing the two-dimensional search to one dimension, greatly improving the efficiency of stereo matching, and thus reducing the computational workload of the processor.

[0084] Embodiment 4

[0085] This embodiment provides a specific implementation method for adjusting the hardware installation angle based on the third embodiment.

[0086] like Figure 7 and Figure 8 As shown, further, the intelligent terminal 1 includes a display device 13, and the display device 13 is used to display the first alignment image and the second alignment image; aligning the first alignment image and the second alignment image row, this step further includes the following sub-steps:

[0087] S1321: The display device 13 displays the first alignment image and the second alignment image.

[0088] S1322: Make the lens of the extended camera 2 and the lens of the camera head on the same physical plane. For example, the user first fixes the position of the extended camera 2, and then adjusts the lens of the camera head with the extended camera 2 as a reference, so that the extended camera 2 and the lens of the camera head are on the same physical plane. It is understandable that the user can also fix the position of the camera head (i.e., fix the position of the smart terminal), and then adjust the lens of the extended camera 2 with the lens of the camera head as a reference, so that the extended camera 2 and the lens of the camera head are on the same physical plane. In this process, both the extended camera 2 and the camera head can shoot the target object, and the user can adjust the position by observing the image on the display device 13 to ensure that the same object appears on the imaging plane of the camera head and the extended camera 2 at the same time.

[0089] It should be further explained that this step makes the extended camera 2 and the camera located in the same plane as much as possible, which simplifies the acquisition of the rotation matrix. The rotation matrix is ​​used to describe the rotation relationship between the camera and the extended camera 2. When the two are located in the same plane, the rotation between them is relatively simple, and the parameters and calculation amount required to calculate the rotation matrix are reduced.

[0090] S1323: Align the extended camera 2 and the camera head vertically, so that corresponding pixel points of the same object in the first aligned image and the second aligned image are aligned in the vertical direction.

[0091] In the real three-dimensional space, the direction perpendicular to the ground is usually defined as the vertical direction. The vertical position of the extended camera 2 and the camera is their position in this vertical direction. For example, if the smart terminal is placed flat on the desktop, with the desktop as the horizontal reference plane, then the direction perpendicular to the desktop upward or downward is the vertical direction mentioned here. The extended camera 2 may be directly above the camera, or directly below, or have a certain height difference with the camera in the vertical direction, and these different states represent their different vertical positions. On the display device 13 of the smart terminal, the pixel points corresponding to the same object in the first alignment image (taken by the camera) and the second alignment image (taken by the extended camera 2) will form a connection line. If these lines are not horizontal, it means that the two are not aligned in the vertical direction, that is, there is a deviation in their vertical positions. The user manually moves the extended camera 2 or the smart terminal (including the camera) to change their positions in the vertical direction. For example, when it is found that the line connecting the corresponding pixel points on the display device 13 is tilted, the camera at a higher position is moved downward, or the camera at a lower position is moved upward, until the corresponding pixel points of the same object in the two images are on the same horizontal plane, and the vertical position alignment adjustment is completed at this time.

[0092] In a specific example, two image frames can generate a line corresponding to the same pixel point of the same object as shown in the figure, and the user assists in alignment by adjusting the relative vertical position of the camera and the extended camera 2. The user manually aligns the two cameras so that the same pixel point is located in the same horizontal plane, which simplifies the acquisition of the translation matrix.

[0093] It should be further explained that this step can simplify the acquisition of the translation matrix. The translation matrix is ​​used to describe the translation relationship between the camera and the extended camera 2. When the two are aligned vertically, their translation in the vertical direction becomes simpler, and it is easier to calculate the translation matrix.

[0094] In summary, the calibration parameters include a calibration intrinsic parameter matrix and a calibration extrinsic parameter matrix, and the calibration extrinsic parameter matrix includes translation matrix parameters and rotation matrix parameters between the camera and the extended camera 2. Therefore, after this step, the user manually aligns the calibration extrinsic parameter matrix of the simplified binocular camera system 100, optimizes the user's use, and reduces the amount of processor calculations.

[0095] Embodiment 5

[0096] This embodiment provides a specific implementation method of software-assisted correction based on the third embodiment.

[0097] Furthermore, the intelligent terminal further includes an auxiliary alignment module; the auxiliary alignment module is used to obtain the overlap between the first alignment image and the second alignment image, so as to assist in aligning the extended camera 2 and the camera head.

[0098] The auxiliary alignment module will feed back the calculated coincidence information to the user. For example, the coincidence value is displayed on the display device 13 of the smart terminal, or the degree of coincidence is indicated by intuitive graphics (such as a progress bar, color prompt, etc.), or voice prompts are given on the smart terminal. At the same time, suggestions for adjusting the positions of the extended camera 2 and the camera head are given according to the coincidence situation, prompting the user to move the extended camera 2 up, down, left or right by a certain distance.

[0099] When the user adjusts the positions of the extended camera 2 and the camera according to the feedback information, the auxiliary alignment module will monitor the changes in the overlap between the first alignment image and the second alignment image in real time until the overlap reaches a satisfactory level, thereby achieving precise alignment of the extended camera 2 and the camera.

[0100] Embodiment 6

[0101] Furthermore, the depth information processing module 11 has a first preset baseline value and a second preset baseline value; the positions of the extended camera 2 and the camera are adjusted so that the distance between the extended camera 2 and the camera is one of the first preset baseline value and the second preset baseline value, thereby enabling measurement of depth information at different distances.

[0102] The first preset baseline value and the second preset baseline value are different baseline values. In a specific example, the preset baseline value may also include a third preset baseline value, a fourth preset baseline value, and up to the nth preset baseline value. Different application scenarios may involve depth measurement of objects at different distances. The binocular vision system with a single baseline value has limitations in its measurement range and may only be able to effectively measure objects within a certain distance range. Setting the first preset baseline value and the second preset baseline value can expand the measurement range, and can measure both close objects and distant objects. For example, in robot navigation in advanced mission areas, the robot may need to perceive nearby obstacles and distant target positions at the same time. By switching different preset baselines, the depth measurement requirements of objects at different distances can be met.

[0103] It should be further explained that after the user selects the preset baseline value in the depth information processing module 11, the distance between the extended camera 2 and the camera head is adjusted according to the preset baseline value. In this process, the distance between the extended camera 2 and the camera head is measured by a measuring tool, that is, the distance between the extended camera 2 and the camera head is obtained until the distance between the extended camera 2 and the camera head is the same as the preset baseline value.

[0104] Embodiment 7

[0105] This embodiment is basically the same as the first embodiment, except that:

[0106] The step of obtaining calibration parameters of the binocular camera system 100 further includes the following sub-steps:

[0107] Calibrate based on a calibration plate of known size;

[0108] Obtaining an internal parameter matrix and an external parameter matrix of the binocular camera system 100;

[0109] Epipolar correction for stereo matching based on the intrinsic and extrinsic matrix.

[0110] In a specific scenario, the user needs more accurate depth information, which requires the following series of manual operations. Here, the calibration using the calibration board is to more accurately determine the camera's intrinsic parameter matrix, which includes the main point position, distortion parameters and focus of the binocular camera system 100, and the extrinsic parameter matrix includes the rotation matrix and translation matrix of the binocular camera system 100. The user can place the two cameras according to their own measurement scene, and does not need to simplify the operation of the camera alignment.

[0111] It should be noted that, although the steps of obtaining the calibration parameters of the binocular camera system 100 according to a calibration plate of known size are more complicated, more accurate depth information can be obtained.

[0112] In a specific example, the user needs to prepare a chessboard. The internal point size and the length of the black and white squares measured by a ruler must be known. Multiple pictures need to be taken from different perspectives as a data set, and then calculations are performed in the smart terminal to obtain the calibration parameters of the camera and the extended camera.

[0113] Embodiment 8

[0114] Furthermore, the depth information processing module 11 also includes a depth map imaging module, which is used to convert the depth value into a depth map. The depth map can be a grayscale map, and the size of the grayscale value indicates the distance from the camera. The smaller the grayscale value, the closer the object is to the camera, and the larger the grayscale value, the farther the object is from the camera.

[0115] Embodiment 9

[0116] Furthermore, the binocular camera system 100 also includes a fixed bracket and an extended power supply; the fixed bracket is used to fix the position of the camera and the smart terminal; the extended power supply is configured to supply power to the extended camera 2 when the power consumption of the extended camera 2 exceeds the power supply threshold of the smart terminal.

[0117] The extended power supply can be a battery with a certain capacity or the extended camera 2 can be directly plugged in to provide power for the extended camera 2. In order to ensure the relative position of the camera and the extended camera 2 is stable, a fixed bracket can be set to fix the smart terminal and the extended camera 2. The design of the fixed bracket needs to ensure that the extended camera 2 and the camera of the smart terminal have the correct relative position in the physical space for accurate alignment. The fixed bracket can be designed as a clamp or a magnetic bracket. Ideally, the extended camera 2 should maintain a certain baseline distance from the camera (usually a few centimeters to more than ten centimeters) and be perpendicular to the camera's field of view.

[0118] A specific embodiment is described below by taking the intelligent terminal 1 as an example of a smart phone.

[0119] 1. Hardware level

[0120] The Extended Camera 2 can be used with the existing camera of a smartphone and is compatible with different mobile operating systems (Android, Hongmeng or iOS), and can correctly identify and support image transmission. The Extended Camera 2 also needs to be equipped with a USB-C interface and support OTG connection, and can also support high-speed transmission standard Bluetooth and Wi-Fi transmission.

[0121] The extended camera 2 is also connected to an extended power supply, which is used to charge the extended camera 2. If the extended camera 2 is connected to a smartphone via USB, the battery of the smartphone may be exhausted quickly. Therefore, it is necessary to ensure that the power supply of the extended camera 2 is sufficient. The extended power supply can use a battery of a certain capacity, and can also charge the smartphone and the extended camera 2.

[0122] In order to ensure the relative position of the extended camera 2 and the smartphone is stable, a fixed bracket is used to fix the extended camera 2. The design of the fixed bracket needs to ensure that the extended bracket and the camera of the smartphone have the correct relative position in the physical space for accurate calibration. The fixed bracket can be designed as a clamp or a magnetic bracket, which can be assembled with the extended camera 2 and the smartphone in a simple way. In order to achieve more accurate binocular vision, the relative angle and position of the extended camera 2 and the smartphone camera need to be known and remain fixed during the entire use process. Ideally, the extended camera 2 should maintain a certain baseline distance from the camera (usually a few centimeters to more than ten centimeters) and be perpendicular to the camera's field of view.

[0123] 2. Software level

[0124] The present invention integrates a matching UI interface in a smart phone. After the extended camera 2 is communicatively connected, row alignment is required. At the algorithm level, several known preset baseline values ​​b are specified, corresponding to several different gears, for measuring depth information at different distances. A simple process will be provided in the UI interface to guide the user to complete the camera alignment operation, such as by moving an external camera or adjusting the angle, so that the mobile phone screen can display the overlap or alignment status between the extended camera 2 and the camera shooting picture. During calibration, the alignment effect of the binocular camera system 100 will be displayed in real time on the mobile phone screen to help users ensure that the two cameras are correctly aligned and reduce errors.

[0125] The UI will display the images captured by the two cameras in real time, as well as the processed depth image. The UI will display the depth map to the user in different forms, such as color mapping, grayscale map, 3D point cloud, etc., allowing users to choose different display methods. To help users understand the depth information, specific areas (such as objects closer or farther from the camera) can be highlighted in the depth map. When the user moves the binocular camera system, the depth map can be updated in real time and the depth information changes can be fed back.

[0126] It should be further explained that the user-moved binocular camera system means that the relative positions of the camera and the extended camera are fixed, while the camera and the extended camera as a whole move relative to the target object. That is, the relative positions of the smartphone and the extended camera do not change, and the two move synchronously.

[0127] Since the binocular camera system 100 requires large computing resources to calculate the depth value, the UI can also provide real-time feedback on system performance, such as CPU, GPU load, memory usage, etc. In this way, the user can know whether the system is currently overloaded and take corresponding measures (such as closing other applications or reducing the computing load). At the same time, the frame rate information can be displayed on the screen to help the user understand whether the application is running smoothly.

[0128] When designing a UI for converting a smartphone external expansion camera 2 into a binocular camera system 100, it is critical to help users properly install, calibrate, and operate the system, while providing an easy-to-understand and easy-to-use depth information display. The UI should be concise, intuitive, and able to ensure a smooth user experience under high performance requirements, especially when it comes to real-time depth calculation and image processing.

[0129] Other configurations and operations according to the embodiments of the present invention are known to those skilled in the art and will not be described in detail herein. In the description of the present invention, "first feature" and "second feature" may include one or more of the features. The up-down direction, left-right direction and front-back direction are based on the up-down direction, left-right direction and front-back direction shown in the figure.

[0130] In the description of the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply means that the first feature is higher in level than the second feature.

[0131] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0132] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. An image processing method for a binocular camera system based on an intelligent terminal, characterized in that: Applied to a smart terminal and an extended camera, the smart terminal includes a memory, a processor, a depth information processing module and a camera component, the camera component includes at least one camera, the depth information processing module is used to run on the processor, and the memory is used to store the depth information processing module; The method comprises: The camera and the extended camera form a binocular camera system; Acquire calibration parameters of the binocular camera system, and input the calibration parameters into the depth information processing module; The binocular camera system photographs the target object to obtain a first target image and a second target image, and inputs the first target image and the second target image into the depth information processing module; The depth information processing module generates a disparity value according to the first target image and the second target image and generates depth information using the disparity value and the calibration parameter.

2. The image processing method of the binocular camera system based on the intelligent terminal according to claim 1, characterized in that: The smart terminal and the extended camera form a binocular camera system, including: The extended camera is communicatively connected to the smart terminal; Adjusting the positions of the extended camera and the camera head; The pixels of the images taken by the extended camera and the camera are made consistent, and the rows of the images taken by the extended camera and the camera are aligned.

3. The image processing method of the binocular camera system based on the intelligent terminal according to claim 2, characterized in that: The step of aligning the image rows captured by the extended camera and the camera comprises: The camera is used to obtain a first alignment image, and the extended camera is used to obtain a second alignment image; The first alignment image and the second alignment image are line aligned.

4. The image processing method of the binocular camera system based on the intelligent terminal according to claim 3, characterized in that: The smart terminal includes a display device, and the display device is used to display the first alignment image and the second alignment image; The aligning the first alignment image and the second alignment image row by row further comprises: The lens of the extended camera and the lens of the camera head are located on the same physical plane; The extended camera and the camera head are aligned vertically, and corresponding pixel points of the same object in the first aligned image and the second aligned image are aligned in a vertical direction.

5. The image processing method of the binocular camera system based on the intelligent terminal according to claim 3, characterized in that: The intelligent terminal also includes an auxiliary alignment module; The auxiliary alignment module is used to obtain the overlap between the first alignment image and the second alignment image, so as to assist in aligning the extended camera and the camera head.

6. The image processing method of the binocular camera system based on the intelligent terminal according to claim 2, characterized in that: The depth information processing module has a first preset baseline value and a second preset baseline value; The positions of the extended camera and the camera are adjusted so that the distance between the extended camera and the camera is one of the first preset baseline value and the second preset baseline value, thereby enabling measurement of depth information at different distances.

7. The image processing method of the binocular camera system based on the intelligent terminal according to claim 1, characterized in that: The step of obtaining calibration parameters of the binocular camera system includes: Calibrate based on a calibration plate of known size; Obtaining a calibration intrinsic parameter matrix and a calibration extrinsic parameter matrix of the binocular camera system; Epipolar line correction for stereo matching is performed according to the intrinsic parameter matrix and the extrinsic parameter matrix.

8. The image processing method of the binocular camera system based on the intelligent terminal according to claim 1, characterized in that: The depth information processing module includes: An image preprocessing module, the image preprocessing module is used to perform grayscale, denoising and histogram equalization processing on the first target image and the second target image to obtain a first preprocessed image and the second preprocessed image; A stereo matching module, wherein the stereo matching module is used to perform stereo matching on the first pre-processed image and the second pre-processed image to obtain a plurality of corresponding feature points; A disparity calculation module, wherein the disparity calculation module is used to obtain a disparity value of each of the feature points; A depth calculation module is used to obtain a depth value corresponding to each feature point according to the disparity value and the calibration parameter.

9. The image processing method of the binocular camera system based on the intelligent terminal according to claim 8, characterized in that: The depth information processing module also includes a depth map imaging module; The depth map imaging module is used to convert the depth value into a depth map.

10. The image processing method of the binocular camera system based on the intelligent terminal according to claim 1, characterized in that: The binocular camera system also includes a fixing bracket and an extended power supply; The fixing bracket is used to support and fix the positions of the extended camera and the smart terminal; The extended power supply is configured to supply power to the extended camera when the power consumption of the extended camera exceeds a power supply threshold of the smart terminal.

Images