Self-adaptive amphibious binocular depth camera and electronic equipment

Abstract

The invention discloses a self-adaptive amphibious binocular depth camera and electronic equipment. The self-adaptive amphibious binocular depth camera comprises a waterproof shell; the structured light projector is used for projecting structured light rays; the first receiver is used for receiving a reflection signal of the structured light rays and generating a first image; the second receiver is used for receiving a reflection signal of the structured light rays and generating a second image; the processor is used for performing image mapping on the first image and the second image according to a first calibration file to generate a left image and a right image; judging the distortion degree of the minimum brightness area in the left image and the right image to determine whether to regenerate the left image and the right image according to a second calibration file; generating a disparity map according to the left image and the right image; and generating a depth map according to the internal reference and the external reference of the first calibration file or the second calibration file. According to the invention, the imaging problem of a traditional camera during switching between land and water environments can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of depth cameras, and in particular to an adaptive amphibious binocular depth camera and electronic equipment. Background Art

[0002] In numerous fields, such as ocean exploration, underwater archaeology, water rescue, and some industrial scenarios requiring amphibious operations, there is an urgent need for imaging equipment that can accurately capture environmental information. Traditional camera equipment, whether purely surface or underwater, has significant limitations. When used underwater, conventional surface cameras experience significant image distortion and reduced resolution due to the significant differences between the optical properties of water, such as refraction and scattering, and those of air. This makes it impossible to accurately capture critical information such as the location, shape, and size of underwater objects. Underwater cameras, when used afloat, struggle to adapt to the lighting conditions and imaging requirements of air, similarly failing to provide clear, usable images.

[0003] As amphibious operations continue to intensify, the development of a camera system capable of stable and accurate operation both above and below water becomes increasingly important. Existing camera technology cannot meet the imaging requirements in these complex environments, leading to the urgent need for a novel adaptive amphibious binocular depth camera solution.

[0004] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of this application. Summary of the Invention

[0005] To this end, the present invention ensures that high-quality depth images can be generated in different environments through unique structural design and image processing methods, and can effectively solve the imaging problems of traditional cameras when switching between water and land environments.

[0006] In a first aspect, the present invention provides an adaptive amphibious binocular depth camera, characterized in that it includes:

[0007] Waterproof housing, including optical window;

[0008] A structured light projector, used for projecting structured light and emitting the light through the optical window;

[0009] a first receiver, configured to receive a reflection signal of the structured light passing through the optical window to generate a first image;

[0010] a second receiver, configured to receive a reflection signal of the structured light passing through the optical window, and generate a second image;

[0011] A processor is configured to perform image mapping on the first image and the second image according to a first calibration file to generate a left image and a right image; determine a degree of distortion of a minimum brightness area in the left image and the right image to determine whether to regenerate the left image and the right image according to a second calibration file; generate a disparity map based on the left image and the right image; and generate a depth map based on intrinsic and extrinsic parameters of the first calibration file or the second calibration file.

[0012] Optionally, the adaptive amphibious binocular depth camera is characterized in that the processor includes:

[0013] Step S1: performing image mapping on the first image and the second image according to a first calibration file to generate a left image and a right image;

[0014] Step S2: determining the degree of distortion of the minimum brightness areas in the left image and the right image;

[0015] Step S3: If the distortion degree is greater than a threshold, performing image mapping on the first image and the second image using a second calibration file to regenerate the left image and the right image; otherwise, directly executing step S4;

[0016] Step S4: generating a disparity map according to the left image and the right image;

[0017] Step S5: Generate a depth map according to the intrinsic parameters and extrinsic parameters of the first calibration file or the second calibration file.

[0018] Optionally, the adaptive amphibious binocular depth camera is characterized in that step S2 includes:

[0019] Step S21: determining an image with a greater degree of transformation between the left image and the right image as a reference image;

[0020] Step S22: binarizing the reference image and performing edge detection to obtain the outline of each minimum brightness area;

[0021] Step S23: Compare the outline of each minimum brightness area with the standard outline to obtain the deviation ratio of each minimum brightness area. When the deviation ratio exceeds a first threshold, mark the minimum brightness area as a distortion unit, and calculate the proportion of the distortion unit in the minimum brightness area as the distortion degree.

[0022] Optionally, the adaptive amphibious binocular depth camera is characterized in that the mapping matrix and internal parameters of the first calibration file and the second calibration file are different.

[0023] Optionally, the adaptive amphibious binocular depth camera is characterized in that, compared with the first calibration file, the mapping matrix of the second calibration file also includes an underwater distortion compensation matrix.

[0024] Optionally, the adaptive amphibious binocular depth camera is characterized in that, compared with the first calibration file, the internal parameters of the second calibration file also include a refraction correction coefficient.

[0025] Optionally, the adaptive amphibious binocular depth camera is characterized in that the structured light projector includes a plurality of projection units, and the plurality of projection units are used to project structured light rays with different patterns.

[0026] Optionally, the adaptive amphibious binocular depth camera is characterized in that the waterproof housing includes a sealed cavity, the optical window is made of a light-transmitting material, and the outer surface of the light-transmitting material is provided with a hydrophobic coating.

[0027] Optionally, the adaptive amphibious binocular depth camera is characterized in that the optical window is made of sapphire glass or fused quartz, and is coated with an anti-reflection film layer on the inner surface, and the average transmittance of the anti-reflection film layer in the 400-900nm band is greater than 98%.

[0028] In a second aspect, the present invention provides an electronic device, characterized in that it includes any of the aforementioned adaptive amphibious binocular depth cameras.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention has strong adaptability to amphibious environments and a wide range of application scenarios. The waterproof housing of the present invention is matched with an optical window to ensure that the camera can work stably in complex environments such as land and underwater, and meet the needs of amphibious operations (such as underwater robots, amphibious equipment, ocean exploration, etc.). The sealing design of the optical window effectively isolates water vapor and dust, avoids hardware damage, and improves the durability of the equipment. The processor automatically selects the first calibration file (applicable to air environment) or the second calibration file (applicable to underwater and other refractive environments) by judging the degree of image distortion (such as distortion in the minimum brightness area). This mechanism solves the problem of image distortion caused by optical refraction in different media (air / water) (such as the refractive index of water is about 1.33, which will deflect light), and can adapt to environmental changes without human intervention, significantly expanding the use scenarios of the device.

[0031] The depth measurement of the present invention has high accuracy and strong anti-interference ability. The structured light projector of the present invention actively projects coded patterns (such as stripes and spots), and the reflected signal is obtained by the receiver to generate an image. Compared with passive binocular vision, the active structured light solution can still provide a stable feature matching basis in low-light and texture-missing scenes (such as underwater turbid environments), thereby improving the robustness of depth calculation. The present invention utilizes the binocular imaging principle (left / right image parallax calculation) and combines the internal parameters (camera focal length, distortion coefficient) and external parameters (binocular baseline, relative posture) of the calibration file to generate high-precision disparity maps and depth maps. Dynamic switching of calibration files can compensate for optical distortion in different environments (such as curved surface refraction and optical path offset of underwater windows) in real time to ensure the accuracy of depth data.

[0032] The present invention can perform intelligent adaptive processing, reduce manual intervention, and improve efficiency and adaptability. The processor of the present invention intelligently determines whether the current environment needs to be recalibrated by analyzing the degree of distortion (such as edge blur and pixel offset) in the minimum brightness area of the image. This adaptive mechanism avoids the tedious process of manual recalibration of traditional equipment when the environment switches, and improves the automation level and response speed of the system. The integrated design of the waterproof housing, structured light projector, dual receiver and processor of the present invention realizes hardware compactness and functional modularization. The adaptive calibration and depth calculation process at the algorithm level ensures that the equipment can quickly output reliable depth data in both water and land environments to meet real-time requirements (such as robot navigation and obstacle avoidance).

[0033] The present invention also has the advantages of robustness and reliability. The active light source of the structured light projector of the present invention can resist ambient light fluctuations (such as underwater light attenuation and strong light on land), ensuring that the receiver obtains a stable reflected signal; the waterproof design effectively copes with harsh conditions such as humidity and high pressure, extending the life of the device. By continuously monitoring image distortion and switching calibration files, the system can automatically compensate for slight distortion caused by contamination (such as water stains and attachments) on the optical window surface, rather than directly failing, thereby improving the fault tolerance of the device in complex environments.

[0034] The hardware of the present invention is compatible with both surface and underwater modes, and the algorithms of the two processing modes are also fully compatible. The only difference is the mapping matrix and internal parameters used for the polar constraints. The processes are exactly the same, which greatly improves the integration, modularity and simplicity of the algorithm. It can be used in soc (system on chip) with DPU (data processor) function and dedicated depth chip, and has very good adaptability.

[0035] This invention overcomes the limitations of traditional depth cameras in a single environment through a collaborative approach combining waterproof hardware, active structured light, and an adaptive calibration algorithm, enabling seamless switching and high-precision depth measurement in both amphibious and land scenarios. Its core advantages lie in its environmental adaptability, active high-precision measurement, and intelligent processing. It is suitable for cutting-edge fields requiring high environmental adaptability and precision, such as amphibious robotics, underwater mapping, and intelligent security, and possesses significant engineering application value and technological foresight. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without inventive work. Other features, purposes and advantages of the present invention will become more apparent by reading the detailed description of the non-limiting embodiments with reference to the following drawings:

[0037] Figure 1 Schematic diagram of the structure of an adaptive amphibious binocular depth camera according to an embodiment of the present invention;

[0038] Figure 2 A flowchart of the steps of a processor during processing according to an embodiment of the present invention;

[0039] Figure 3 This is a flowchart of a step for determining the degree of distortion in an embodiment of the present invention;

[0040] Figure 4 Schematic diagram of the structure of another adaptive amphibious binocular depth camera in an embodiment of the present invention;

[0041] Figure 5 Schematic diagram of the structure of an optical window in an embodiment of the present invention;

[0042] Figure 6 Schematic diagram of the structure of another optical window in an embodiment of the present invention.

[0043] 1- Waterproof housing;

[0044] 2-Structured light projector;

[0045] 3- First receiver;

[0046] 4- Second receiver;

[0047] 5-Processor;

[0048] 6-first projection unit;

[0049] 7- Second projection unit;

[0050] 8- Optical window;

[0051] 9-Hydrophobic coating;

[0052] 10-antireflection film layer; DETAILED DESCRIPTION

[0053] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several variations and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0054] The terms "first," "second," "third," "fourth," and the like (if any) in the description and claims of the present invention and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatus.

[0055] An embodiment of the present invention provides an adaptive amphibious binocular depth camera, which aims to solve the problems existing in the prior art.

[0056] The following describes in detail the technical solutions of the present invention and how the technical solutions of this application solve the above-mentioned technical problems using specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The following embodiments of the present invention are described in conjunction with the accompanying drawings.

[0057] Since water and air have different effects on the light path in above-water and underwater scenes, depth cameras in the prior art cannot simultaneously address the depth acquisition issues above and below water.

[0058] The present invention uses structured light as an active light source and a binocular system consisting of a first receiver and a second receiver to obtain depth data. It automatically identifies whether it is above water or underwater based on the left image and the right image, and performs corresponding processing to obtain an accurate above-water or underwater depth map.

[0059] Figure 1FIG. 1 is a schematic diagram of the structure of an adaptive amphibious binocular depth camera according to an embodiment of the present invention. Figure 1 As shown, an adaptive amphibious binocular depth camera in an embodiment of the present invention includes:

[0060] The waterproof housing 1 includes an optical window 8 .

[0061] Specifically, the waterproof housing is the camera's external protective structure, adaptable to diverse underwater and terrestrial environments. It prevents water and dust from entering the camera, protecting the camera's electronic and optical components from damage. The optical window on the housing is the passage for light to enter and exit, and requires excellent optical properties, such as high transmittance and low refractive index, to reduce reflection and refraction at the window, ensuring that structured light can be accurately emitted and the reflected signal can smoothly enter the receiver.

[0062] The structured light projector 2 is used to project structured light and emit it through the optical window.

[0063] Specifically, the main function of a structured light projector is to project structured light. It emits light with a specific structure and pattern. After reflecting off an object's surface, this light carries three-dimensional information about the surface. This structured light is projected onto the external scene through an optical window, providing the basis for subsequent depth information acquisition.

[0064] The first receiver 3 is configured to receive a reflection signal of the structured light passing through the optical window to generate a first image.

[0065] Specifically, the first receiver receives the reflection signal of the structured light passing through the optical window, processes the reflection signal, and converts it into a first image. This image contains information such as the intensity and position of the structured light after it is reflected from the object surface, and is an important data source for subsequent processing.

[0066] The second receiver 4 is configured to receive a reflection signal of the structured light passing through the optical window to generate a second image.

[0067] Specifically, similar to the first receiver, the second receiver also receives the reflection signal of the structured light passing through the optical window. The second receiver generates a second image, which records the reflection of the structured light from a different angle than the first image. This provides two sets of data for binocular vision, which can then be compared and processed to obtain depth information.

[0068] Processor 5 is used to perform image mapping on the first image and the second image according to a first calibration file to generate a left image and a right image; determine the degree of distortion of the minimum brightness area in the left image and the right image to determine whether to regenerate the left image and the right image according to the second calibration file; generate a disparity map based on the left image and the right image; and generate a depth map based on the intrinsic and extrinsic parameters of the first calibration file or the second calibration file.

[0069] Specifically, the processor performs image mapping on the first and second images based on the first calibration file, converting them into left and right images. The first calibration file contains the camera's internal parameters (such as focal length and pixel size) and external parameters (such as camera position and posture). These parameters can be used to calibrate and map the original images to meet the requirements of the binocular vision system, facilitating subsequent processing and analysis.

[0070] Distortion determination and processing: Determine the degree of distortion in the minimum brightness area of the left and right images. If the degree of distortion exceeds a certain range, it indicates that the current environment of the depth camera may not be consistent with the environment specified by the first calibration file. In this case, the left and right images must be regenerated based on the second calibration file. The second calibration file is a parameter file obtained after calibration for different environments to ensure that the camera can adapt to different operating conditions and accurately generate images. One of the first and second calibration files is for underwater environments, and the other is for above-water environments.

[0071] Disparity map generation: Generates a disparity map based on the left and right images. The disparity map represents the disparity information of corresponding points in the left and right images. By calculating the positional differences of the same feature points in the left and right images, the disparity of the object at different positions can be obtained, thereby reflecting the object's depth information.

[0072] Depth map generation: The depth map is generated based on the intrinsic and extrinsic parameters from the first or second calibration file, combined with the disparity map. The intrinsic and extrinsic parameters provide the geometric model and positional relationships of the camera imaging. These parameters, along with the disparity information, are used to calculate the 3D coordinates of each point in the scene, thereby generating a depth map representing the scene's depth information.

[0073] Figure 2 This is a flowchart of the steps of a processor in processing according to an embodiment of the present invention. Figure 2 As shown, the steps of a processor in an embodiment of the present invention during processing include:

[0074] Step S1: performing image mapping on the first image and the second image according to a first calibration file to generate a left image and a right image.

[0075] Specifically, the first calibration file contains the camera's intrinsic and extrinsic parameters. Intrinsic parameters typically include the camera's focal length, principal point coordinates, and pixel size. These parameters determine how the camera projects points in three-dimensional space onto the two-dimensional image plane. Extrinsic parameters represent the camera's position and orientation in the world coordinate system, such as the camera's rotation matrix and translation vector. The first calibration file can be either an above-water or underwater calibration file.

[0076] Since the first image and the second image acquired by the first receiver and the second receiver may have certain geometric distortions and positional deviations, by performing image mapping on the two images using the parameters in the first calibration file, they can be corrected to a unified coordinate system, thereby generating left and right images that meet the requirements of binocular vision processing, providing a basis for subsequent disparity calculations.

[0077] Step S2: Determine the degree of distortion of the minimum brightness areas in the left image and the right image.

[0078] Specifically, the minimum brightness region refers to an area consisting of multiple consecutive bright pixels. For speckle, the minimum brightness region is the area of a single speckle. For stripes, the minimum brightness region is the area of a single stripe. The minimum brightness region is the reflected signal of a single beam of structured light.

[0079] Since the first calibration file is designed to calibrate the camera for specific environmental factors (such as underwater refraction, temperature changes, etc.), if the degree of distortion in the minimum brightness area after mapping the first calibration file exceeds the preset range, it means that the first calibration file does not match the current environment. The degree of image distortion can be determined by analyzing information such as the brightness value changes and pixel position deviations in the left and right images. For example, the degree of distortion can be assessed by calculating the deviation between the actual and theoretical positions of the pixels, or by comparing the statistical characteristics of the pixel brightness values (such as the mean, variance, etc.) with the differences under normal conditions.

[0080] Step S3: If the distortion degree is greater than a threshold, image mapping is performed on the first image and the second image using a second calibration file to regenerate the left image and the right image; otherwise, step S4 is directly executed.

[0081] Specifically, the threshold is a pre-set standard value used to determine whether the degree of image distortion is within an acceptable range. If the degree of distortion is less than or equal to the threshold, the left and right images generated using the first calibration file are of good quality and can be directly used for subsequent disparity calculations. If the degree of distortion is greater than the threshold, the first calibration file may not be able to accurately correct the image distortion, and a second calibration file is required.

[0082] The second calibration file is different from the first. One of the two files is designed for above-water environments, while the other is designed for underwater environments. For example, if the first file is for underwater, the second file is for above-water environments. Conversely, if the first file is for above-water environments, the second file is for underwater environments. When the distortion exceeds a threshold, the second file is used to remap the first and second images to generate new left and right images, improving the accuracy of subsequent depth calculations.

[0083] Step S4: generating a disparity map according to the left image and the right image.

[0084] Specifically, parallax refers to the difference in pixel position between the left and right images of the same object due to the different camera positions of the two binocular cameras. The magnitude of the parallax is inversely proportional to the distance between the object and the camera: the farther the object, the smaller the parallax; the closer the object, the larger the parallax.

[0085] By matching feature points (such as corners and edges) in the left and right images and calculating the difference in pixel positions between them, we can obtain the disparity of each feature point. By integrating the disparity information of all feature points, we can generate a disparity map. Each pixel value in the disparity map represents the disparity of the object at that location.

[0086] Step S5: Generate a depth map according to the intrinsic parameters and extrinsic parameters of the first calibration file or the second calibration file.

[0087] Specifically, based on the principle of binocular vision, if the camera's intrinsic parameters (such as focal length), extrinsic parameters (such as the baseline length between the two cameras) and parallax information are known, the depth of each point in the scene can be calculated through triangulation.

[0088] Based on the intrinsic and extrinsic parameters in the first or second calibration file, combined with the disparity map generated in step S4, the depth of each pixel is calculated to obtain the depth value corresponding to each pixel. The depth values of all pixels are integrated to generate a depth map, in which each pixel value represents the distance from the object at that location to the camera.

[0089] This embodiment does not need to rely on environmental perception judgment of external devices. It can adaptively perform water and underwater identification judgment using only the data of the depth camera, and is real-time, which greatly improves the amphibious adaptability of the depth camera and reduces hardware costs.

[0090] Figure 3 FIG. 1 is a flow chart of steps for determining the degree of distortion in an embodiment of the present invention. Figure 3As shown, in an embodiment of the present invention, a step of determining the degree of distortion includes:

[0091] Step S21: determining the image with a greater degree of transformation between the left image and the right image as a reference image.

[0092] In this step, the left and right images with the highest degree of transformation during the generation process are considered the reference image. In step S1, image mapping is performed on the first and second images to unify their coordinate systems. In practice, one image is often used as the reference for mapping the other. For example, the second image is mapped using the first image as the reference. In this case, the degree of transformation of the second image is greater than that of the first image. Images with higher degrees of transformation are more likely to be distorted, which allows for better distortion assessment and reduces computational complexity.

[0093] Step S22: binarize the reference image and perform edge detection to obtain the outline of each minimum brightness area.

[0094] In this step, binarization converts the image into an image with only two grayscale values (usually 0 and 255). In the reference image, the minimum brightness region typically has a low grayscale value. By selecting an appropriate threshold for binarization, the minimum brightness region can be separated from the background, making subsequent processing simpler and more efficient. For example, the Otsu algorithm can be used to automatically determine an appropriate threshold value, setting pixels in the image with grayscale values below the threshold to 0 (black) and pixels above the threshold to 255 (white).

[0095] Edge detection is a technique used to extract the edges of objects in an image. Performing edge detection on a binarized image can identify the boundaries of areas with minimum brightness. Common edge detection algorithms include the Canny algorithm and the Sobel operator. These algorithms calculate the gradient magnitude and direction of pixels in an image and mark pixels with a gradient magnitude greater than a certain threshold as edge pixels, thereby determining the outline of the area with minimum brightness.

[0096] After edge detection, connected component analysis and other methods can be used to connect adjacent edge pixels to form closed contours, each of which corresponds to a minimum brightness area. This allows the position and shape of each minimum brightness area to be accurately determined.

[0097] Step S23: Compare the outline of each minimum brightness area with the standard outline to obtain the deviation ratio of each minimum brightness area. When the deviation ratio exceeds a first threshold, mark the minimum brightness area as a distortion unit, and calculate the proportion of the distortion unit in the minimum brightness area as the distortion degree.

[0098] In this step, the standard outline is the ideal shape of the minimum brightness region. This can be obtained by analyzing and statistically analyzing a large number of distortion-free images, or by theoretical calculation based on the camera's imaging model and the scene's geometric characteristics. The standard outline serves as a reference template for evaluating the distortion of the minimum brightness region in actual images.

[0099] Deviation Ratio Calculation: The outline of each minimum brightness region is compared with the standard outline. Various methods can be used to calculate the deviation ratio. For example, the Hausdorff distance between the two contours, the difference in contour area, or the difference in contour perimeter can be calculated. For example, the Hausdorff distance is a measure of the distance between two sets of points. By calculating the maximum value of the maximum distance from a point on the minimum brightness region outline to the standard outline and the maximum distance from a point on the standard outline to the minimum brightness region outline, and then normalizing the values, the deviation ratio for that minimum brightness region can be obtained.

[0100] Distortion unit marking: When the deviation ratio of a minimum brightness area exceeds the first threshold, the area is significantly distorted and exceeds the acceptable range, and the area is marked as a distortion unit. The first threshold is a pre-set standard used to determine whether the minimum brightness area has significant distortion.

[0101] Distortion degree calculation: The sum of the number or area of all minimum brightness regions marked as distortion units is divided by the total number or area of minimum brightness regions. This gives the proportion of distortion units in the minimum brightness regions, which is used as the distortion degree of the entire image. This distortion degree can intuitively reflect the distortion of the minimum brightness regions in the image and provide a basis for determining whether to remap the image later.

[0102] This embodiment calculates the deviation ratio of each small maximum brightness area based on the reference image, and then calculates the degree of distortion. While reducing the amount of calculation, it achieves very high precision and greatly improves the accuracy of judgment.

[0103] In some embodiments, the mapping matrix and the internal parameters of the first calibration file are different from those of the second calibration file.

[0104] The mapping matrix plays a central role in the image mapping process. It transforms the original images (the first and second images) from the camera's native coordinate system into a unified coordinate system suitable for subsequent processing, enabling image correction and alignment. This matrix eliminates image offset and rotation caused by factors such as camera position and orientation, ensuring that corresponding points in the left and right images are accurately aligned within the same coordinate system.

[0105] Since one of the first calibration file and the second calibration file is for an underwater environment, the mapping matrices of the two are different to adapt to specific environments. For example, the first calibration file is calibrated under the normal use environment of the camera, when the environment in which the camera is located is relatively stable, such as when used in the air. The second calibration file may be calibrated for a special environment (such as underwater). In an underwater environment, light will refract, which will cause the geometric relationship of the camera imaging to change, making the mapping matrix originally applicable to the air no longer applicable. Therefore, it is necessary to recalculate the mapping matrix to correct image distortion caused by factors such as refraction.

[0106] Different mapping matrices produce different transformation effects on the first and second images. Using the mapping matrix from the first calibration file for image mapping accurately calibrates and aligns the images under standard conditions. However, if the environment or working conditions change and image distortion exceeds acceptable limits, using the mapping matrix from the second calibration file can better adapt to the new situation, calibrating the images more accurately and generating left and right images that better match the actual scene.

[0107] A camera's intrinsic parameters describe its internal imaging characteristics, primarily including focal length, principal point position, and pixel size. These parameters determine how the camera projects objects in three-dimensional space onto a two-dimensional image plane and are the basis for depth calculation and image correction.

[0108] The main reasons why the internal parameters of the first calibration file and the second calibration file are different are:

[0109] Optical property changes: A camera's optical properties may change in different environments. For example, underwater, because water has a different refractive index than air, the refraction of light through the camera lens changes, causing the camera's focal length to shift. Furthermore, water pressure may cause subtle deformations in the camera lens, further affecting the camera's optical properties. Therefore, recalibrating the intrinsic parameters is necessary to accurately reflect the camera's imaging characteristics in the new environment.

[0110] Variations in sensor characteristics: A camera's image sensor may exhibit different characteristics under different environmental conditions. For example, temperature fluctuations may cause the sensor's sensitivity to change, affecting image brightness and contrast. To ensure accurate image acquisition in various environments, internal parameters must be adjusted based on actual conditions.

[0111] Intrinsic parameters play a vital role in depth calculation. According to the principle of binocular vision, depth calculation requires the use of camera intrinsic parameters (such as focal length) and extrinsic parameters (such as the baseline distance between the left and right cameras) as well as parallax information. Different intrinsic parameter values will result in different calculated depth values. When the intrinsic parameters of the first calibration file are used for depth calculation, it is suitable for imaging conditions under normal environments. In special environments or working conditions, using the intrinsic parameters of the second calibration file can more accurately calculate the depth information of objects in the scene and improve the accuracy of the depth map.

[0112] In some embodiments, compared to the first calibration file, the mapping matrix of the second calibration file further includes an underwater distortion compensation matrix.

[0113] In underwater environments, light propagation differs significantly from that in air. Because the refractive index of water is greater than that of air (about 1.33 for water and approximately 1 for air), light is refracted when it enters the camera lens from water. This refraction phenomenon can cause the position and shape of objects in the image to be distorted, making it impossible to accurately correct the image using the mapping matrix of the first calibration file, which was originally designed for use in air. Furthermore, factors such as suspended particles in the water and water flow disturbances can also affect light propagation, further exacerbating image distortion.

[0114] The underwater distortion compensation matrix is specifically designed to correct image distortion caused by underwater environmental factors. It models and calculates distortion factors such as light refraction and transforms the pixels in the image accordingly. For example, for pixels that have shifted due to refraction, the underwater distortion compensation matrix can adjust them to the correct position based on the laws of light refraction. This effectively reduces distortion in underwater imaging, making the image more similar to the geometric features of the real scene, thereby improving image quality and the accuracy of subsequent processing (such as parallax calculation and depth calculation).

[0115] When using the second calibration file for image mapping, the mapping process is different from that of the first calibration file because the mapping matrix includes an underwater distortion compensation matrix. The mapping matrix of the first calibration file mainly deals with conventional distortion and coordinate transformation caused by factors such as camera installation and its own characteristics in the air. On the other hand, the mapping matrix of the second calibration file, on the basis of completing the conventional coordinate transformation, also uses the underwater distortion compensation matrix to compensate for the additional distortion caused by the underwater environment. Specifically, in the process of converting the first image and the second image into the left image and the right image, the pixels will not only be transformed according to the conventional mapping relationship, but will also be additionally adjusted according to the underwater distortion compensation matrix to eliminate the adverse effects of the underwater environment.

[0116] Accurate image mapping is a prerequisite for reliable depth calculation. Because the mapping matrix of the second calibration file improves image accuracy through the underwater distortion compensation matrix, the subsequent disparity map generated from the left and right images is more accurate. The accuracy of the disparity map directly affects the generation of the depth map. A more accurate disparity map can obtain more accurate depth information when calculating depth based on the intrinsic and extrinsic parameters of the first or second calibration file, thereby improving the quality and reliability of depth data acquired by the adaptive amphibious binocular depth camera in underwater environments, enabling it to more accurately perceive the three-dimensional structure of underwater scenes.

[0117] In some embodiments, compared to the first calibration file, the internal parameter of the second calibration file further includes a refraction correction coefficient.

[0118] In air, light propagates relatively evenly, and the camera's intrinsic parameters (such as focal length and the position of the principal point) accurately describe its imaging characteristics. However, when the camera is underwater, light is refracted when it enters the camera lens from water (with a refractive index of approximately 1.33). This refraction changes the original propagation path of the light, causing the actual imaging of the camera to differ from that in air. Specifically, refraction causes an equivalent change in focal length and a possible shift in the position of the principal point, affecting the image geometry and the correspondence between pixels and the actual scene.

[0119] The refraction correction coefficient is a parameter introduced into the internal parameters of the second calibration file specifically to account for the effects of underwater refraction. Its primary function is to quantify and compensate for changes in the camera's internal parameters in underwater environments. By incorporating the refraction correction coefficient into the calculation of the internal parameters, we can correct for changes in parameters such as focal length and principal point position caused by refraction, ensuring that the camera's underwater imaging closely matches its ideal (non-refraction) imaging model.

[0120] The refraction correction coefficient has the following effects on depth calculation:

[0121] Improving disparity calculation accuracy: In a binocular vision system, disparity calculation is based on matching corresponding points in the left and right images. Refraction can cause geometric distortion in the images, causing the positions of corresponding points in the left and right images to shift. The refraction correction coefficient corrects this shift, making the matching of feature points in the left and right images more accurate, thereby improving the accuracy of disparity calculation.

[0122] Optimizing depth calculations: Depth calculations rely on camera intrinsic parameters (such as focal length) and parallax information. Accurate intrinsic parameters are crucial for calculating accurate depth values. The refraction correction factor corrects for variations in these parameters in underwater environments. Combined with accurate parallax information, this allows for more precise depth calculations of objects in the scene, reducing depth calculation errors caused by refraction.

[0123] The refraction correction coefficient has the following significance for the entire processing flow:

[0124] Improving image mapping accuracy: When using the second calibration file for image mapping, the refraction correction coefficients work in conjunction with the underwater distortion compensation matrix. The underwater distortion compensation matrix primarily addresses geometric image distortion, while the refraction correction coefficients correct for camera intrinsics. Together, they improve image mapping accuracy, making the generated left and right images more closely resemble the geometry of the real scene.

[0125] Enhanced camera adaptability: The introduction of a refraction correction coefficient enables the adaptive amphibious binocular depth camera to operate properly in underwater environments, ensuring the accuracy of imaging and depth calculation. This enhances the camera's adaptability in diverse environments, expands its application range, and makes the camera more practical in areas such as underwater detection and underwater robot navigation.

[0126] Figure 4 FIG. 1 is a schematic diagram of the structure of another adaptive amphibious binocular depth camera according to an embodiment of the present invention. Figure 4 As shown, the structured light projector includes a plurality of projection units, and the plurality of projection units are used to project structured light rays with different patterns. Figure 4 The structured light projector 2 includes at least a first projection unit 6 and a second projection unit 7 .

[0127] The structured light projector utilizes multiple projection units primarily to improve the richness, adaptability, and accuracy of scene information. Each projection unit operates independently, projecting a unique pattern of structured light. Each pattern of structured light produces distinct reflections upon encountering an object's surface, incorporating various information about the surface's geometry and texture. By comprehensively analyzing the reflections from these different patterns, a more comprehensive understanding of the scene's characteristics can be achieved, thereby improving the accuracy of tasks such as depth measurement and object recognition.

[0128] The role of different pattern structured light:

[0129] Improved depth resolution: Different structured light patterns can sample an object's surface at different scales. For example, some patterns may feature finer stripes, suitable for measuring minute surface details and improving depth resolution; while other patterns may feature larger spacing, ideal for quickly acquiring an object's general outline and shape. By combining these different structured light patterns, depth measurements can be performed at varying levels of accuracy, meeting the depth information requirements of diverse application scenarios.

[0130] Enhanced anti-interference capabilities: In practical applications, environmental factors such as noise and lighting variations may interfere with the structured light reflection signal. Different structured light patterns have different characteristics and differ in their sensitivity to interference. By simultaneously projecting multiple structured light patterns, the complementarity between the different patterns can be exploited to reduce the impact of interference on measurement results. For example, when one structured light pattern is affected by noise, other structured light patterns may still provide reliable information, thereby ensuring the accuracy of depth measurement.

[0131] Improved object recognition: Different object surfaces reflect different patterns of structured light differently. By analyzing the reflections of multiple structured light patterns, we can extract richer object features, helping to improve object recognition accuracy. For example, some objects may have unique reflection patterns for a specific pattern of structured light. By identifying these patterns, we can more accurately determine the object's type.

[0132] How the projection units work together:

[0133] Multiple projection units need to work together to ensure that the projected structured light can accurately cover the target scene. Usually, these projection units will project structured light with different patterns in a certain order or simultaneously. When the camera receives the reflected signal, it is necessary to accurately distinguish the reflected signals corresponding to different projection units. This can be achieved by encoding the structured light of different projection units (for example, using different colors, frequencies or modulation methods), so that the camera can match it to the corresponding projection unit according to the coded information of the reflected signal, thereby processing the reflection information of different pattern structured light rays separately.

[0134] Improvements to overall camera performance:

[0135] The structured light projector's multiple projection units, which project different patterns of structured light, significantly improve the overall performance of the adaptive amphibious binocular depth camera. This enables the camera to more stably and accurately acquire depth information and object features in complex environments (such as underwater lighting variations and interference from suspended particles). This provides a more reliable data foundation for subsequent image analysis, target detection, and recognition tasks, expanding the camera's application areas and scope.

[0136] Figure 5 FIG. 1 is a schematic diagram of the structure of an optical window in an embodiment of the present invention. Figure 5 As shown, the waterproof housing 1 includes a sealed cavity, the optical window 8 is made of a light-transmitting material, and a hydrophobic coating 9 is provided on the outer surface of the light-transmitting material.

[0137] The sealed cavity is the core component of the waterproof housing. Its primary function is to provide a reliable waterproof protective space for the camera's internal electronic components (such as the structured light projector, first and second receivers, and processor). By sealing these key components within the cavity, water is effectively prevented from entering the camera, preventing damage such as short circuits and corrosion to the electronic components, thereby ensuring the camera's normal operation in underwater environments.

[0138] Sealed cavities are typically made of high-strength, water-resistant materials, such as engineering plastics or metals. Their manufacturing process requires high standards to ensure the cavity's tightness. Common sealing methods include the use of sealing rings and sealants. For example, sealing rings are installed at the cavity's interface. When the various parts of the housing are assembled, the sealing rings are squeezed, forming a good seal and preventing water penetration.

[0139] The optical window is made of a translucent material to ensure that structured light can pass smoothly through the window and be projected onto the external scene, while also allowing reflected light to enter the camera's internal receiver. Common translucent materials include optical glass and optical plastics. These materials have excellent light transmission properties and can maintain high transmittance within a certain spectral range, ensuring that the quality of light transmission is not affected.

[0140] Optical windows must not only have excellent light transmittance but also possess a certain degree of optical stability and wear resistance. In underwater environments, windows may be subject to the impact of water flow and friction from suspended particles, so materials with high hardness and good wear resistance are required. Furthermore, the surface flatness and optical uniformity of the optical window are also very important, as these factors affect the light propagation path and image quality.

[0141] The primary purpose of applying a hydrophobic coating to the outer surface of a light-transmitting material is to reduce the adhesion of water to the optical window surface. The molecular structure of the hydrophobic coating possesses unique properties that reduce the adhesion between water and the window surface. When water contacts the hydrophobic coating, it forms beads rather than spreading evenly across the surface. This allows the water droplets to more easily roll off the window surface under the influence of gravity or currents, thereby reducing any interference with light transmission.

[0142] The hydrophobic coating can effectively prevent water from forming a film on the surface of the optical window, avoiding the refraction and scattering of light by the water film, and ensuring that light can pass through the window with high efficiency. In an underwater environment, the presence of a water film will seriously affect the imaging quality of the camera, resulting in problems such as image blur and distortion. By using a hydrophobic coating, the imaging clarity and stability of the camera in an underwater environment can be improved, thereby enhancing the overall performance of the camera. At the same time, the hydrophobic coating can also reduce the adhesion of impurities in the water to the window surface, reducing maintenance costs and extending the service life of the camera. When the depth camera moves from underwater to above water, the hydrophobic coating can also cause water to flow down quickly, allowing the depth camera to quickly switch from underwater to above water, making calibration on water fast and accurate.

[0143] Figure 6 FIG. 1 is a schematic diagram of the structure of another optical window in an embodiment of the present invention. Figure 6 As shown, the optical window 8 is made of sapphire glass or fused quartz, and is coated with an anti-reflection film layer 10 on the inner surface. The average transmittance of the anti-reflection film layer 10 in the 400-900 nm band is greater than 98%.

[0144] Sapphire glass has an extremely high hardness, reaching 9 on the Mohs hardness scale, second only to diamond. This makes it highly resistant to scratches and abrasions, effectively resisting friction from sand and impurities in water. This ensures that the optical window will not show visible scratches even in complex underwater environments for extended periods of use, thereby maintaining excellent light transmission and optical clarity.

[0145] Sapphire glass has excellent chemical stability and is resistant to corrosion from water and chemicals. Underwater environments may contain water with varying pH levels, but sapphire glass will not react chemically with these substances, affecting its optical properties. This ensures that the camera can function properly in a variety of water conditions.

[0146] Sapphire glass has good light transmittance over a wide spectral range, especially in the visible light and near-infrared bands, and can meet the light propagation requirements required for structured light projection and reception.

[0147] Fused quartz has an extremely low coefficient of thermal expansion, which means it maintains excellent dimensional stability despite temperature fluctuations. In underwater environments, where water temperatures can fluctuate significantly, optical windows made of fused quartz do not experience significant deformation due to temperature fluctuations, thus ensuring optical system stability and imaging accuracy.

[0148] Fused quartz has high purity and excellent optical uniformity, providing high-quality optical transmission. It can reduce the scattering and absorption of light during propagation, allowing structured light to be projected and received more accurately, improving camera measurement accuracy.

[0149] Fused quartz has high transmittance in a wide spectral range from ultraviolet to near-infrared, and can adapt to different types of structured light projection requirements, providing wider applicability for cameras working under different working conditions.

[0150] Anti-reflection coatings utilize the principle of light interference to reduce light reflection on optical window surfaces and increase transmittance. When light strikes the anti-reflection coating, the light reflected from the upper and lower surfaces interfere with each other. By properly designing the film's thickness and refractive index, the reflected light cancels out, allowing more light to pass through the optical window.

[0151] The antireflection coating achieves an average transmittance of over 98% in the 400-900nm wavelength range, a significant performance indicator. The 400-900nm wavelength range encompasses visible light and some near-infrared light, and the light projected by the structured light projector and the reflected signal primarily fall within this wavelength range. This high transmittance minimizes loss of structured light as it passes through the optical window, improving light utilization efficiency and enabling the first and second receivers to receive stronger and more accurate reflected signals, thereby enhancing image quality and depth measurement accuracy.

[0152] In practical applications of adaptive amphibious binocular depth cameras, a high-transmittance anti-reflection coating significantly improves camera performance in a variety of environments. Underwater, light is naturally attenuated by water absorption and scattering. The anti-reflection coating further reduces light loss through the optical window, enhancing the camera's underwater detection capabilities. Even when used in air, it ensures the camera captures clear and accurate image information, providing a reliable data foundation for subsequent image processing and depth calculations.

[0153] This specification also provides an electronic device, including any of the aforementioned adaptive amphibious binocular depth cameras. It should be noted that the electronic device in this embodiment is exemplary and is provided to help those skilled in the art better understand the role of the adaptive amphibious binocular depth camera in an electronic device. It should not constitute any limitation on the electronic device.

[0154] The overall structure of an electronic device varies depending on its specific application scenario and design requirements, but generally speaking, it includes the following main parts:

[0155] Enclosure: This provides physical protection and support for the entire electronic device, typically made of durable, protective materials such as metal alloys or high-strength plastics. Enclosures must be well sealed and waterproof and dustproof to adapt to diverse operating environments, especially when the electronic device is used underwater or in humid environments. Their design also needs to consider heat dissipation, and may incorporate features such as heat dissipation holes or fins.

[0156] Adaptive amphibious binocular depth camera: Located at the front of the electronic device or at a specific detection location, it is isolated from the external environment by a waterproof housing. The camera's optical window faces outward, allowing structured light to be projected and reflected back. The camera contains components such as a structured light projector, a primary receiver, a secondary receiver, and a processor, which work together to complete image acquisition and depth calculation.

[0157] Data processing and storage modules typically include a high-performance processor (such as a CPU or GPU) and a sufficient amount of memory and storage. The processor is responsible for further processing and analyzing the images and depth data captured by the camera, such as image recognition, object detection, and 3D modeling. Storage devices are used to store the captured data and processed results for subsequent query and analysis.

[0158] Power modules provide a stable power supply to various components of electronic devices. These modules can be powered by a built-in battery or an external power adapter. Electronic devices that require extended operation or are used in mobile environments typically feature large-capacity rechargeable batteries and corresponding charging management circuits.

[0159] Communication modules: These modules are used to transmit and communicate data between electronic devices and external devices or systems. Common communication methods include wired communication (such as Ethernet and USB) and wireless communication (such as Wi-Fi, Bluetooth, and 4G / 5G). Through communication modules, electronic devices can upload collected data to cloud servers for further processing and analysis, and can also receive commands and control information from external devices.

[0160] The main functions of electronic equipment are:

[0161] Environmental Perception: Utilizing adaptive amphibious binocular depth cameras, electronic devices can perceive the three-dimensional information of their surroundings in real time, including the position, shape, and distance of objects. This enables electronic devices to navigate and avoid obstacles in complex environments. For example, in underwater exploration, they can avoid obstacles and safely complete detection tasks. In indoor robot navigation, they can accurately identify surrounding objects and passages and plan appropriate paths of action.

[0162] Target detection and recognition: By processing and analyzing collected images and depth data, electronic devices can detect and identify specific targets. For example, in security monitoring, these devices can identify targets such as people and vehicles, analyze their behavior, and provide early warnings. In industrial testing, these devices can detect product defects and flaws to ensure quality.

[0163] 3D Modeling: Based on the depth information and image data captured by a depth camera, electronic devices can create 3D models of the surrounding environment or objects. This has wide-ranging applications in architectural surveying and mapping, cultural heritage preservation, and virtual scene construction. For example, 3D modeling of ancient buildings facilitates digital preservation and research, and in game development, it creates realistic virtual scenes.

[0164] Data Recording and Analysis: Electronic devices can record and store collected images, depth data, and processing results for subsequent analysis and research. By analyzing large amounts of data, patterns of environmental changes, object characteristics, and behavior patterns can be discovered, providing a basis for decision-making.

[0165] The functions of depth cameras in electronic devices are:

[0166] Providing Core Data: Adaptive amphibious binocular depth cameras are key components for electronic devices to acquire three-dimensional information about their surroundings. They project structured light and receive reflected signals to generate left and right images and depth maps, providing the most fundamental and core data for other functional modules in electronic devices. Without this data, electronic devices would be unable to perform functions such as environmental perception, object detection, and 3D modeling.

[0167] Adaptability to diverse environments: The adaptive amphibious nature of depth cameras enables electronic devices to function effectively in diverse environments. Whether in air or underwater, depth cameras can accurately acquire depth information, expanding the application range of electronic devices. For example, in underwater exploration and marine research, electronic devices can use depth cameras for target search and terrain mapping in underwater environments. Their environmental perception and target recognition capabilities can also be utilized in conventional environments, both indoors and outdoors.

[0168] Improved Functional Accuracy: The depth information provided by depth cameras can significantly improve the accuracy of various electronic device functions. In object detection and recognition, depth information can help distinguish objects at different distances and avoid misjudgments. In 3D modeling, depth information makes the constructed models more accurate and realistic. The high-precision data obtained by depth cameras enables electronic devices to better complete various complex tasks, improving work efficiency and quality.

[0169] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be apparent to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0170] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various variations or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. An adaptive amphibious binocular depth camera, characterized in that: include: Waterproof housing, including optical window; A structured light projector, used for projecting structured light and emitting the light through the optical window; a first receiver, configured to receive a reflection signal of the structured light passing through the optical window to generate a first image; a second receiver, configured to receive a reflection signal of the structured light passing through the optical window, and generate a second image; a processor, configured to perform image mapping on the first image and the second image according to a first calibration file to generate a left image and a right image; determining a degree of distortion of minimum brightness areas in the left image and the right image to determine whether to regenerate the left image and the right image according to a second calibration file; A disparity map is generated according to the left image and the right image; and a depth map is generated according to the intrinsic parameters and extrinsic parameters of the first calibration file or the second calibration file.

2. The adaptive amphibious binocular depth camera according to claim 1, characterized in that: The processor includes: Step S1: performing image mapping on the first image and the second image according to a first calibration file to generate a left image and a right image; Step S2: determining the degree of distortion of the minimum brightness areas in the left image and the right image; Step S3: If the distortion degree is greater than a threshold, performing image mapping on the first image and the second image using a second calibration file to regenerate the left image and the right image; otherwise, directly executing step S4; Step S4: generating a disparity map according to the left image and the right image; Step S5: Generate a depth map according to the intrinsic parameters and extrinsic parameters of the first calibration file or the second calibration file.

3. The adaptive amphibious binocular depth camera according to claim 1, characterized in that: Step S2 includes: Step S21: determining an image with a greater degree of transformation between the left image and the right image as a reference image; Step S22: binarizing the reference image and performing edge detection to obtain the outline of each minimum brightness area; Step S23: Compare the outline of each minimum brightness area with the standard outline to obtain the deviation ratio of each minimum brightness area. When the deviation ratio exceeds a first threshold, mark the minimum brightness area as a distortion unit, and calculate the proportion of the distortion unit in the minimum brightness area as the distortion degree.

4. The adaptive amphibious binocular depth camera according to claim 1, characterized in that: The first calibration file and the second calibration file have different mapping matrices and internal parameters.

5. The adaptive amphibious binocular depth camera according to claim 4, characterized in that: Compared with the first calibration file, the mapping matrix of the second calibration file also includes an underwater distortion compensation matrix.

6. The adaptive amphibious binocular depth camera according to claim 4, characterized in that: Compared with the first calibration file, the internal reference of the second calibration file also includes a refraction correction coefficient.

7. The adaptive amphibious binocular depth camera according to claim 1, characterized in that: The structured light projector includes a plurality of projection units, and the plurality of projection units are used to project structured light rays with different patterns.

8. The adaptive amphibious binocular depth camera according to claim 1, characterized in that: The waterproof housing includes a sealed cavity, the optical window is made of a light-transmitting material, and the outer surface of the light-transmitting material is provided with a hydrophobic coating.

9. The adaptive amphibious binocular depth camera according to claim 1, characterized in that: The optical window is made of sapphire glass or fused quartz, and is coated with an anti-reflection film layer on the inner surface. The average transmittance of the anti-reflection film layer in the 400-900nm band is greater than 98%.

10. An electronic device, characterized in that: An adaptive amphibious binocular depth camera comprising any one of claims 1-9.

Images