Transparent object detection segmentation and depth correction method, system and device and storage medium

Abstract

The invention discloses a transparent object detection segmentation and depth correction method, system and device, and a storage medium. The method comprises the following steps: S1, obtaining an RGB image, a speckle infrared image and a depth image; s2, recognizing a transparent object range and a transparent object boundary through a first deep learning network; wherein the first deep learning network obtains the range of the transparent object and the boundary of the transparent object according to the astigmatism characteristic of speckles irradiated on the transparent object; and S3, correcting the depth value in the range of the transparent object according to the depth map and the speckle infrared map in combination with the depth value of the boundary of the transparent object to obtain a final depth map. According to the invention, the depth values of various transparent objects can be accurately identified.

Description

Background Art

[0002] Transparent object detection technology based on depth camera is an advanced computer vision technology that uses depth camera to obtain three-dimensional information of objects, thereby realizing the detection and positioning of transparent objects.

[0003] Depth cameras measure the distance between an object and the camera by emitting and receiving light (such as infrared light), thereby generating depth images of the object. These depth images are combined with color images (RGB images) to provide three-dimensional information of the object.

[0004] Due to their high light transmittance and low reflectivity, transparent objects make it difficult to effectively apply traditional detection methods based on blocking light. Although depth cameras can obtain three-dimensional information of objects, they also face some challenges when facing transparent objects:

[0005] Missing depth information: Transparent objects may not form effective depth information because depth cameras usually rely on the reflection of light on the surface of the object to measure the distance.

[0006] Reflection interference: Reflections from transparent objects may lead to misinterpretation of depth information, especially when ambient light is strongly reflected on the surface of the object.

[0007] To overcome the above challenges, researchers have proposed a variety of transparent object detection techniques based on depth cameras. These techniques usually combine depth images, RGB images, and other sensor data (such as infrared images) to achieve detection and positioning of transparent objects.

[0008] 1. Multimodal Information Fusion:

[0009] The depth camera not only provides depth images, but also RGB images. Combining these two images can achieve preliminary detection of transparent objects.

[0010] Infrared images can also serve as auxiliary information because the reflective properties of transparent objects under infrared light may be different from those under visible light.

[0011] 2. Candidate region extraction:

[0012] Use an active depth sensor (such as Intel RealSense) to retrieve transparent candidate regions from the depth image. These regions are usually areas where depth information is missing or uncertain.

[0013] The corresponding candidate regions are extracted from the RGB image and the infrared image respectively for subsequent processing.

[0014] 3. Feature extraction and classification:

[0015] Use feature extraction algorithms (such as SIFT) to extract feature points from the candidate regions.

[0016] The extracted features are classified using classification algorithms (such as support vector machines, neural networks, etc.) to identify transparent objects.

[0017] 4. 3D reconstruction and positioning:

[0018] By adjusting the direction of the camera so that its optical axis is perpendicular to the normal direction of the plane where the object is located, a new set of RGB images and infrared images are obtained.

[0019] The object contours in these images are extracted separately, and the 3D objects are reconstructed through stereo matching.

[0020] The reconstructed 3D model is used to estimate the object's position and posture information, thereby achieving accurate positioning of transparent objects.

[0021] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0022] To this end, the present invention provides a transparent object detection, segmentation and depth correction solution to solve the requirements for the range and depth of transparent objects in various application scenarios.

[0023] In a first aspect, the present invention provides a transparent object detection, segmentation and depth correction method, characterized by comprising:

[0024] Step S1: Obtain RGB image, speckle infrared image and depth image;

[0025] Step S2: identifying the range of transparent objects and the boundary of transparent objects through a first deep learning network; wherein the first deep learning network obtains the range of transparent objects and the boundary of transparent objects according to the astigmatism characteristics of speckle irradiation on the transparent objects;

[0026] Step S3: According to the depth map and the speckle infrared map, combined with the depth value of the boundary of the transparent object, the depth value within the range of the transparent object is corrected to obtain a final depth map.

[0027] Optionally, the transparent object detection, segmentation and depth correction method is characterized in that step S1 comprises:

[0028] Step S11: obtaining an RGB image and a speckle infrared image; wherein the speckle infrared image has at least two images;

[0029] Step S12: Calculate the disparity according to the two speckle infrared images to obtain a depth map.

[0030] Optionally, the transparent object detection, segmentation and depth correction method is characterized in that the speckle infrared image has two images, and further comprises:

[0031] Step S13: performing pixel-level reprojection alignment on the depth map and the RGB map.

[0032] Optionally, the transparent object detection, segmentation and depth correction method is characterized in that the first deep learning network is a Unet network, comprising an encoder, a first decoder and a second decoder; the first decoder outputs the range of the transparent object; the second decoder outputs the boundary of the transparent object; and there is feature interaction between features of each level of the first decoder and the second decoder.

[0033] Optionally, the transparent object detection, segmentation and depth correction method is characterized in that, in step S3, the depth map, the transparent object range, the transparent object boundary and the speckle infrared image are input into a second deep learning network to obtain a final depth map; wherein, the second deep learning network corrects the depth value within the transparent object range in combination with the transparent object boundary.

[0034] Optionally, the transparent object detection, segmentation and depth correction method is characterized in that step S3 comprises:

[0035] Step S31: performing three-dimensional reconstruction on the portion outside the range of the transparent object according to the depth map and the speckle infrared map;

[0036] Step S32: obtaining three-dimensional information of the boundary of the transparent object, and identifying the shape of the boundary of the transparent object in three-dimensional space;

[0037] Step S33: performing plane fitting according to the boundary of the transparent object to obtain a depth value corresponding to each pixel point within the range of the transparent object.

[0038] Optionally, the transparent object detection, segmentation and depth correction method is characterized in that step S33 comprises:

[0039] Step S331: performing straight line fitting on the boundary of the transparent object to obtain a plurality of straight lines in a three-dimensional space;

[0040] Step S332: fitting the plane of three consecutive straight lines to obtain a transparent object;

[0041] Step S333: Calculate the depth value corresponding to each pixel point within the range of the transparent object according to the transparent object.

[0042] In a second aspect, the present invention provides a transparent object detection, segmentation and depth correction system, which is used to implement any of the transparent object detection, segmentation and depth correction methods described above, and is characterized by comprising:

[0043] An acquisition module is used to acquire RGB images, speckle infrared images and depth images;

[0044] An identification module, used to identify the range of transparent objects and the boundary of transparent objects through a first deep learning network; wherein the first deep learning network obtains the range of transparent objects and the boundary of transparent objects according to the astigmatism characteristics of speckle irradiation on transparent objects;

[0045] A correction module is used to correct the depth value within the range of the transparent object according to the depth map and the speckle infrared map in combination with the depth value of the boundary of the transparent object to obtain a final depth map.

[0046] In a third aspect, the present invention provides a transparent object detection, segmentation and depth correction device, characterized in that it includes:

[0047] processor;

[0048] a memory storing executable instructions of the processor;

[0049] The processor is configured to execute the steps of any of the aforementioned transparent object detection, segmentation and depth correction methods by executing the executable instructions.

[0050] In a fourth aspect, the present invention provides a computer-readable storage medium for storing a program, characterized in that when the program is executed, the steps of any of the aforementioned transparent object detection, segmentation and depth correction methods are implemented.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] The present invention adopts speckle infrared images and utilizes the fact that the reflection of light spots by transparent objects is different from that of non-transparent objects, so that the reflection signals of the light spots present different characteristics on the image, and the calculated depth values ​​also present messy characteristics; the present invention can accurately identify the range of transparent objects and effectively correct the depth values ​​of transparent objects through deep learning of light spots and depth values, and is suitable for various transparent object recognition scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings in the following descriptions are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:

[0054] Figure 1 A flowchart of a method for detecting, segmenting and correcting a transparent object according to an embodiment of the present invention;

[0055] Figure 2 This is a flowchart of the steps of obtaining an RGB image, a speckle infrared image, and a depth image in an embodiment of the present invention;

[0056] Figure 3 This is another flowchart of steps for obtaining an RGB image, a speckle infrared image, and a depth image in an embodiment of the present invention;

[0057] Figure 4 A schematic diagram of a network architecture in an embodiment of the present invention;

[0058] Figure 5 Schematic diagram of various images in an embodiment of the present invention;

[0059] Figure 6 is a flow chart of steps for obtaining a final depth map in an embodiment of the present invention;

[0060] Figure 7 This is a flowchart of the steps of calculating the depth value corresponding to each pixel in an embodiment of the present invention;

[0061] Figure 8 It is a structural schematic diagram of a transparent object detection, segmentation and depth correction system in an embodiment of the present invention;

[0062] Fig. 9 is a schematic structural diagram of a transparent object detection, segmentation and depth correction device in an embodiment of the present invention; and

[0063] Fig.10 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0065] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0066] The embodiment of the present invention provides a transparent object detection, segmentation and depth correction method, which aims to solve the problems existing in the prior art.

[0067] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0068] The present invention is based on an RGBD camera, obtains an RGB image, a speckle infrared image and a depth image, identifies the range of transparent objects through a first deep learning network, and then corrects the depth value to obtain a final depth map. It can accurately identify transparent objects in various application scenarios and solve the problem of inaccurate identification in the prior art.

[0069] Figure 1 FIG. 1 is a flow chart of the steps of a transparent object detection, segmentation and depth correction method according to an embodiment of the present invention. Figure 1 As shown, the steps of a transparent object detection, segmentation and depth correction method in an embodiment of the present invention include:

[0070] Step S1: Obtain RGB image, speckle infrared image and depth image.

[0071] In this step, a common color image sensor, such as a CMOS or CCD photosensitive element in a camera, is used to collect color image information of the scene. This type of sensor uses a filter to separate light into three channels: red, green, and blue, and then combines them into a colorful RGB image. It can intuitively present the color, texture, and other appearance features of objects in the scene, providing an approximate digital presentation for human visual perception.

[0072] The speckle infrared emitter emits a specific pattern of infrared speckle light to the scene. The speckle pattern can be coded structured light or random speckle. The infrared sensor receives the speckle infrared light reflected from the scene, converts it into an electrical signal based on the difference in light intensity, and then generates a speckle infrared image through analog-to-digital conversion and image algorithm processing. This image is critical for the subsequent analysis of transparent objects using speckle characteristics.

[0073] Based on the depth perception technology, the depth map can be calculated based on the structured light principle according to the individual speckle infrared image. This can accurately obtain the depth value for non-transparent objects. However, when the target object is a transparent object, when the light spot is irradiated on the transparent object, the light spot is affected by the transparent object, and its reflected signal will be eliminated to a certain extent and will be relatively weak. At the same time, the transparent object will also change the light path, causing the position and shape of the light spot to shift, and ultimately causing a large deviation in the calculated depth value.

[0074] You can also use binocular technology to calculate the depth map using parallax. In this case, the 3D camera needs to be equipped with two RGB cameras or two structured light receivers. If two RGB cameras are equipped, the depth map is calculated using two RGB images. If two structured light receivers are equipped, the depth map is calculated using two speckle infrared images.

[0075] Step S2: Identify the range and boundary of transparent objects through the first deep learning network.

[0076] In this step, the first deep learning network obtains the transparent object range and the transparent object boundary according to the astigmatism characteristics of the speckle irradiated on the transparent object. Since the refraction and reflection characteristics of transparent objects are different from those of opaque objects, the distribution pattern of the speckle after reflection and penetration on the surface of the transparent object will show a unique regularity. For example, when the light penetrates the transparent object, it will be refracted, so that the shape and density of the speckle will change at the boundary and internal area. The deep learning network learns these astigmatism characteristics to distinguish transparent objects from other types of objects.

[0077] During training, a large number of annotated data sets are collected in advance. The annotated information includes the range and boundaries of transparent objects. The input to the network is the speckle infrared image, RGB image and depth image. The network can use the convolutional neural network (CNN) architecture. The convolution layer is responsible for extracting local features in the speckle image, the pooling layer performs dimensionality reduction to reduce the amount of calculation, and the fully connected layer is used for classification or regression tasks. The output is the predicted range and boundary of transparent objects. Through the back-propagation algorithm, the network weights are continuously adjusted to gradually reduce the error between the predicted results and the true value of the annotation until the network converges and achieves a good recognition accuracy.

[0078] In actual application, the newly collected speckle infrared image is input into the trained first deep learning network, and the network forward propagates calculations to output the range of transparent objects in the image, that is, which pixels belong to the transparent object area, and outline the boundary line between the transparent object and the surrounding environment.

[0079] Step S3: According to the depth map and the speckle infrared map, combined with the depth value of the boundary of the transparent object, the depth value within the range of the transparent object is corrected to obtain a final depth map.

[0080] In this step, due to the optical properties of transparent objects, conventional depth measurement methods are prone to errors. For example, in a depth camera based on structured light, light is refracted when passing through a transparent object, causing the received reflected light coding pattern to deform, resulting in inaccurate calculated depth values; under binocular vision, the light transmittance of transparent objects will also interfere with parallax calculations, making the depth information of transparent objects in the initial depth map unreliable.

[0081] The depth value at the boundary is relatively more accurate because the boundary is the transition area between the transparent object and the surrounding non-transparent objects. The depth measurement accuracy of non-transparent objects is higher and can be used as a reference point for calibration.

[0082] For each pixel point within the range of the transparent object, the depth value is adjusted by using a difference algorithm, a fitting algorithm, etc., combined with the identified boundary depth value. For example, if the transparent object is approximately a plane, the boundary depth value can be used to fit the plane and update the depth values ​​of other points in the plane; if it is a transparent object with a complex surface, a more complex interpolation algorithm can be used to recalculate the depth of the pixels within the range of the transparent object based on the gradient relationship between the boundary points and the depth values ​​of the surrounding non-transparent objects, and finally generate a final depth map that accurately reflects the true depth of the transparent object.

[0083] In some embodiments, in step S3, the depth map, the transparent object range, the transparent object boundary and the speckle infrared map are input into the second deep learning network to obtain the final depth map; wherein, the second deep learning network corrects the depth value within the transparent object range in combination with the transparent object boundary. When entering step S3, the preliminary depth map, the transparent object range identified by the first deep learning network, and the original speckle infrared map have been obtained. These data are fed into the second deep learning network as a whole, providing a comprehensive information basis for subsequent depth correction work. The depth map carries the original distance information of the scene, although there are errors in the transparent object area; the transparent object range clearly defines the target area that needs to be corrected, allowing the network to focus on it; the speckle infrared map contains the unique astigmatism characteristics of transparent objects, providing key clues for correcting the depth value. The second deep learning network can use a convolutional neural network architecture suitable for depth estimation and correction tasks, such as an architecture improved based on the residual network (ResNet). It has multiple convolutional layers, pooling layers and fully connected layers. The convolutional layers are used to extract the features of the input data, the pooling layers are responsible for reducing the data dimension and the amount of calculation, and the fully connected layers integrate the features extracted at different levels to complete the final correction decision. After the network front end receives the input data, each layer processes it step by step and learns the mapping relationship from the combination of input multi-source data to the accurate correction depth value.

[0084] After the input data enters the network, each part is first extracted by different convolutional layers. The features of the depth map reflect the overview of the scene distance, the features of the transparent object range focus on the geometric shape of the target area, and the features of the speckle infrared map capture the scattering characteristics of light on transparent objects. Subsequently, the network merges these features from different sources through a specific fusion layer, so that the information required for correction complements each other and forms a comprehensive judgment basis.

[0085] The fused features are sent to the subsequent network layer, where the depth value within the transparent object range is adjusted based on the pre-learned correction model. The model refers to the astigmatism characteristics in the speckle infrared image, such as speckle deformation and light intensity distribution, and combines the boundary and shape information of the transparent object range to determine the deviation direction and amplitude of the current depth value. For each pixel within the transparent object range, a more accurate depth value is recalculated, and a corrected depth map is gradually constructed.

[0086] After multi-layer network processing, the depth values ​​of all pixels within the transparent object range are corrected, and a complete and accurate final depth map is generated. This depth map not only maintains the original accuracy in the non-transparent object area, but more importantly, solves the problem of inaccurate depth measurement in the transparent object area due to optical characteristics, providing reliable data support for subsequent tasks that rely on accurate depth information, such as 3D scene reconstruction, robot visual navigation, etc.

[0087] Figure 2 FIG. 1 is a flowchart of steps for obtaining an RGB image, a speckle infrared image, and a depth image in an embodiment of the present invention. Figure 2 As shown, in an embodiment of the present invention, a step of obtaining an RGB image, a speckle infrared image, and a depth image includes:

[0088] Step S11: Acquire the RGB image and the speckle infrared image.

[0089] In this step, the speckle infrared image has at least two images. Under the principle of binocular vision, the depth information of the scene can be accurately inferred by comparing the differences between at least two images. Multiple images can capture the scene from different perspectives, providing sufficient data basis for subsequent parallax calculations. With two or more speckle infrared images, it is possible to simulate the way human eyes observe objects and use the perspective difference to mine depth clues.

[0090] When collecting speckle infrared images, a binocular infrared camera system is equipped to emit speckle patterns to the scene at the same time. Two infrared cameras receive the reflected speckle infrared light from different angles, thus obtaining two speckle infrared images. In order to ensure the accuracy of depth calculation, the exposure time, frame rate and other parameters of the infrared camera also need to be strictly synchronized to avoid inconsistent images due to time difference, which affects the subsequent parallax calculation.

[0091] There can also be two RGB images. When collecting RGB images, binocular color cameras can be used. The two cameras are precisely placed in the physical structure to ensure a certain baseline distance. At the moment of shooting, they are triggered synchronously, collecting the scene at the same time, and each generates an RGB image. The two RGB images can be used for depth deduction later because of the slight difference in perspective.

[0092] Step S12: Calculate the disparity according to the two speckle infrared images to obtain a depth map.

[0093] In this step, disparity refers to the position offset of the same object in two images with different viewing angles. For a target object in the scene, its pixel position in the left image is different from that in the right image. The matching points of the corresponding objects in the two images are found through the feature matching algorithm, and the horizontal (or vertical) displacement between these matching points is calculated. This displacement is the disparity. Commonly used feature matching algorithms include SIFT (Scale Invariant Feature Transform), SURF (Speeded Robust Features), ORB (Direction FAST and Rotation BRIEF), etc. These algorithms can extract key feature points in the image and efficiently match them between the two images.

[0094] Once the parallax is calculated, combined with the inherent parameters of the camera, the depth value corresponding to each object in the scene can be quickly calculated, thereby generating a depth map in pixels that marks the depth of each pixel.

[0095] For speckle infrared images, since they have speckle patterns, in addition to the general feature matching method, the characteristics of the speckle itself can also be used when calculating disparity. The distribution law of speckle has specific changes under different viewing angles. Using these laws, corresponding points can be located more accurately, accelerating the disparity calculation process. Subsequently, the disparity is also converted into depth information based on the depth derivation formula and integrated into the final depth map construction.

[0096] This embodiment combines RGB images, speckle infrared images and stereo vision principles to provide high-precision and robust depth information, especially for the recognition and depth reconstruction of transparent objects. This embodiment is not only suitable for the detection and segmentation of transparent objects, but can also be applied to various scenarios that require high-precision depth information, such as augmented reality, industrial automation, and medical imaging.

[0097] Figure 3 FIG. 1 is another flow chart of steps for obtaining an RGB image, a speckle infrared image, and a depth image in an embodiment of the present invention. Figure 3 The speckle infrared image has two images. Compared with the above-mentioned embodiment, another method for obtaining an RGB image, a speckle infrared image and a depth image in the embodiment of the present invention further includes:

[0098] Step S13: performing pixel-level reprojection alignment on the depth map and the RGB map.

[0099] In this step, after the depth map is calculated from the binocular speckle infrared image, the depth map and the RGB image are often not aligned at the pixel level. This is because the camera systems that obtain them may have factors such as differences in installation positions, viewing angle deviations, and lens distortions of different cameras. If the unaligned depth map and RGB image are used directly, subsequent analysis and processing tasks based on the fusion of the two, such as target recognition or scene reconstruction, will have problems such as inaccurate positioning and confusing feature matching, so pixel-level reprojection alignment is required.

[0100] Based on the camera parameters obtained through calibration, for each pixel in the depth map, according to its depth value and the internal and external parameters of the camera, the pixel is reversely projected from the camera coordinate system where the depth map is located to the three-dimensional world coordinate system, and then forward projected to the camera coordinate system where the RGB map is located, so as to obtain the accurate position of the pixel in the RGB map and achieve pixel-level alignment. This process involves matrix operations of coordinate transformation, such as using rotation matrices and translation vectors to rotate and translate coordinates, and combining internal parameters such as focal length to complete the final projection positioning.

[0101] Since the pixel position after reprojection may not be an integer coordinate, it may fall between two pixels in the RGB image, so interpolation is required. Common interpolation methods include bilinear interpolation, which takes the values ​​of four pixels around the target point and averages them according to a certain weight to obtain the pixel value corresponding to the target point, so that the reprojected depth map can be completely and smoothly matched with the RGB image, completing pixel-level reprojection alignment.

[0102] In some embodiments, Figure 4 As shown, the first deep learning network is a Unet network, which includes an encoder, a first decoder and a second decoder; the first decoder outputs the range of the transparent object; the second decoder outputs the boundary of the transparent object; and there is feature interaction between the features of each level of the first decoder and the second decoder.

[0103] Unet is a convolutional neural network with a classic encoder-decoder architecture, which is particularly suitable for image segmentation tasks. Its encoder part is used to gradually extract high-level semantic features of the image, which can capture the complex patterns in the speckle infrared image, and the decoder is responsible for converting these abstract features back to feature maps with a size close to the original image to accurately locate the target area.

[0104] The encoder is usually composed of multiple convolutional modules stacked in sequence, each of which contains a convolutional layer, an activation function (such as ReLU), and a pooling layer. As the number of network layers increases, the convolution kernel continuously extracts more and more abstract features in the speckle infrared image, and the pooling layer gradually reduces the size of the feature map while increasing the receptive field, allowing the network to focus on a wider range of image context information, making it easier to identify the overall astigmatism characteristics associated with transparent objects.

[0105] Corresponding to the encoder, the first decoder gradually upsamples the high-level semantic features transmitted by the encoder to restore the size of the feature map. For example, it uses transposed convolution or up-pooling to gradually move the feature map from abstract to concrete. It finally outputs the range of transparent objects, which means that the features learned by the network can accurately identify which pixels belong to transparent objects after a series of decoding transformations, providing key area information for subsequent processing.

[0106] The second decoder also receives features from the encoder and performs an upsampling process similar to the first decoder. However, it focuses on outputting the boundaries of transparent objects. During the learning process, it focuses on capturing the key features that distinguish transparent objects from the surrounding environment. These features are decoded to outline clear boundary lines, separating transparent objects from the background and other objects.

[0107] The first decoder and the second decoder interact with each other at each level, which is a key design to improve performance. High-level features are rich in semantic information, and low-level features retain more details. The interaction between the two allows the decoder to take into account both semantic understanding and detail restoration. For example, at a certain level, the feature map of the first decoder is spliced ​​with the feature map of the corresponding level of the second decoder, and then subjected to convolution operation. The newly generated feature map combines the advantages of both sides, making the output transparent object range more accurate and the boundaries sharper, reducing misjudgments and unclear segmentation.

[0108] Figure 5 In the figure, a is the speckle infrared image, b is the depth map, c is the transparent object range, and d is the final depth map. As can be seen from a, the speckle infrared image includes multiple scattered spots, which can effectively detect transparent objects. b is the calculated depth map. Due to the existence of transparent objects, the boundaries in the depth map are blurred and the depth values ​​are disordered. The transparent object range c obtained by the first deep learning model can accurately obtain the transparent object range. The final depth map obtained in d can accurately express the depth values ​​including transparent objects in the target scene.

[0109] Figure 6 FIG. 1 is a flowchart of steps for obtaining a final depth map in an embodiment of the present invention. Figure 6 As shown, in an embodiment of the present invention, a step of obtaining a final depth map includes:

[0110] Step S31: performing three-dimensional reconstruction on the portion outside the range of the transparent object according to the depth map and the speckle infrared map.

[0111] In this step, the depth map already carries the distance information of each point in the scene, and the speckle infrared map helps to further understand the surface characteristics of the object. For areas outside the range of transparent objects, since there is no light refraction and reflection interference caused by transparent materials, the data in the depth map is relatively accurate and can be directly used as the basic depth data for 3D reconstruction. The speckle infrared map can supplement the texture and undulation details of some object surfaces and assist in improving the 3D model.

[0112] Common 3D reconstruction methods include reconstruction based on point clouds. First, the pixels in the depth map are converted into points in the 3D space according to their depth values, and many such points form a point cloud. Then, some algorithms, such as Poisson reconstruction, are used to process the point cloud into continuous triangular mesh patches, gradually outlining the 3D shape of the objects in the scene; or the MarchingCubes algorithm is used to voxelize the 3D space, determine the state of the voxels based on the point cloud, and finally generate a high-quality 3D model to restore the scene outside the range of transparent objects.

[0113] Step S32: Obtain three-dimensional information of the boundary of the transparent object, and identify the shape of the boundary of the transparent object in three-dimensional space.

[0114] In this step, since the position of the boundary of the transparent object in the depth map is already known, combined with the depth value in the depth map, the depth of each point on the boundary can be obtained, so that the boundary is converted from the two-dimensional image space to the three-dimensional space and its three-dimensional coordinates are determined. For example, if the depth value of a pixel point on the boundary in the depth map is d, the (x, y, d) coordinates of the point in the three-dimensional space can be calculated based on the conversion relationship between the camera coordinate system and the image coordinate system.

[0115] After obtaining the three-dimensional coordinate set of the boundary, a geometric analysis algorithm is used to identify its shape. For simple shapes, such as planes and cylinders, the mathematical model of the plane and cylinder can be fitted to calculate the error with the coordinate set to determine whether it conforms to the corresponding shape; for complex shapes, the shape classification model in deep learning can be used to input the three-dimensional coordinates of the boundary and output the corresponding shape category to provide a shape basis for subsequent plane fitting.

[0116] Step S33: performing plane fitting according to the boundary of the transparent object to obtain a depth value corresponding to each pixel point within the range of the transparent object.

[0117] In this step, if the shape formed by the boundary of the transparent object is close to a plane as determined in step S32, a plane fitting method can be used. Plane fitting is based on mathematical principles such as the least square method to find a plane equation that best represents the distribution of the boundary point set, and the general form is Ax+By+Cz+D=0. The parameters A, B, C, and D of the plane equation are determined by calculating the minimum sum of square errors of the distances from the three-dimensional coordinate points of the boundary to the plane.

[0118] Depth value derivation: For each pixel point within the range of the transparent object, its two-dimensional coordinates are converted to three-dimensional space, substituted into the fitted plane equation, and the corresponding depth value z is solved. In this way, the depth value correction within the range of the transparent object is completed, so that the final depth map can more accurately reflect the true depth of the transparent object and make up for the initial depth measurement error caused by the optical characteristics of the transparent object. If the shape of the transparent object is non-planar, a more complex surface fitting method is required to complete this correction process.

[0119] This embodiment performs plane fitting on transparent objects according to their boundaries in three-dimensional space, so that the transparent objects can be restored more accurately; it can achieve accurate segmentation, depth value correction and three-dimensional reconstruction of transparent objects, which not only improves the overall accuracy but also ensures the integrity of the reconstruction results; through plane fitting, this embodiment can calculate the depth value corresponding to each pixel point within the range of the transparent object, taking into account the special properties of the transparent object (such as light refraction and reflection), thereby improving the accuracy of the depth value.

[0120] Figure 7 FIG. 1 is a flowchart of a step of calculating the depth value corresponding to each pixel in an embodiment of the present invention. Figure 7 As shown, in an embodiment of the present invention, a step of calculating a depth value corresponding to each pixel point includes:

[0121] Step S331: performing straight line fitting on the boundary of the transparent object to obtain a plurality of straight lines in three-dimensional space.

[0122] In this step, based on the boundary information of the transparent object obtained in the previous step, the boundary information has a corresponding set of coordinate points in the three-dimensional space. Since the boundaries of transparent objects often have linear features, especially some transparent objects with regular geometric shapes, such as the edges of cuboids and prisms, or the contour edge parts of irregular transparent objects with relatively regular shapes, they can be approximated by straight lines.

[0123] Fitting algorithm: The least squares method is used for straight line fitting, which is a classic data fitting method. The least squares method determines the most suitable straight line parameters for the point set by minimizing the sum of the squares of the distances from each boundary point to the straight line, and then fits multiple straight lines on the boundary of the transparent object, using these straight lines to outline the approximate linear framework of the object boundary.

[0124] Step S332: Fit the plane to three consecutively connected straight lines to obtain a transparent object.

[0125] In this step, with the multiple straight lines obtained in step S331, three consecutively connected straight lines are selected to fit the plane. In three-dimensional space, three points that are not on the same straight line can determine a plane, while three connected straight lines can provide enough points at different positions to determine the plane. From a geometric point of view, for the part of the transparent object with plane features, the points around these connected straight lines should be approximately in the same plane in the local area.

[0126] The least squares method is also used to calculate a plane equation for the set of points involved by the three connected straight lines, so that the sum of the squares of the distances from these points to the plane is minimized. By repeating this process for different groups of three connected straight lines, multiple plane pieces are used to splice and approximate the entire transparent object to adapt to the complex geometric shape of the transparent object and complete the construction of the geometric model of the transparent object.

[0127] Step S333: Calculate the depth value corresponding to each pixel point within the range of the transparent object according to the transparent object.

[0128] In this step, we first need to convert the pixels within the transparent object from the 2D image coordinate system to the 3D space coordinate system, which depends on the camera's intrinsic and extrinsic parameters. The camera's intrinsic parameters determine the conversion relationship between the image coordinate system and the camera coordinate system, while the extrinsic parameters relate the camera coordinate system to the world coordinate system. Using these parameters, we "place" the pixels in the 3D space at the position corresponding to the transparent object.

[0129] Substitute the pixel coordinates converted to three-dimensional space into the previously fitted plane equation (if it is a complex surface, it may be a more complex surface equation), and solve the equation to get the depth value corresponding to the pixel. In this way, the depth value that was originally inaccurate due to the optical characteristics of the transparent object is corrected through a series of operations from boundary fitting lines, plane construction to pixel depth calculation, so that the final depth map can accurately reflect the true depth of the transparent object.

[0130] This embodiment can obtain the three-dimensional shape of a transparent object by connecting three straight-line fitting planes. This method takes into account the spatial continuity of the transparent object and can reconstruct the three-dimensional structure of the object more accurately. Even if the transparent object has a complex shape or surface features, a relatively accurate three-dimensional reconstruction result can be obtained by connecting three straight-line fitting planes in sequence. After obtaining the three-dimensional shape of the transparent object, the depth value corresponding to each pixel point within the range of the transparent object can be calculated based on the shape. This depth value calculation method takes into account the three-dimensional structure of the object and is therefore more accurate.

[0131] Figure 8 FIG. 1 is a schematic diagram of the structure of a transparent object detection, segmentation and depth correction system according to an embodiment of the present invention. Figure 8 As shown, a transparent object detection, segmentation and depth correction system in an embodiment of the present invention includes:

[0132] An acquisition module is used to acquire RGB images, speckle infrared images and depth images;

[0133] An identification module, used to identify the range of transparent objects and the boundary of transparent objects through a first deep learning network; wherein the first deep learning network obtains the range of transparent objects and the boundary of transparent objects according to the astigmatism characteristics of speckle irradiation on transparent objects;

[0134] A correction module is used to correct the depth value within the range of the transparent object according to the depth map and the speckle infrared map in combination with the depth value of the boundary of the transparent object to obtain a final depth map.

[0135] This embodiment is based on an RGBD camera to obtain an RGB image, a speckle infrared image and a depth image. The range of transparent objects is identified through a first deep learning network, and then the depth value is corrected to obtain a final depth map. This can accurately identify transparent objects in various application scenarios and solve the problem of inaccurate identification in the prior art.

[0136] The present invention also provides a transparent object detection, segmentation and depth correction device in an embodiment, including a processor and a memory, wherein executable instructions of the processor are stored. The processor is configured to execute the steps of a transparent object detection, segmentation and depth correction method by executing the executable instructions.

[0137] As mentioned above, this embodiment is based on an RGBD camera to obtain an RGB image, a speckle infrared image and a depth map, identifies the range of transparent objects through a first deep learning network, and then corrects the depth value to obtain the final depth map, which can accurately identify transparent objects in various application scenarios and solve the problem of inaccurate recognition in the prior art.

[0138] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0139] Fig. 9 Schematic diagram of a transparent object detection, segmentation and depth correction device in an embodiment of the present invention. Fig. 9The electronic device 600 according to this embodiment of the present invention is described. Fig. 9 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0140] like Fig. 9 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0141] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps of various exemplary embodiments of the present invention described in the above transparent object detection, segmentation and depth correction method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0142] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0143] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a grid environment.

[0144] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0145] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more grids (e.g., a local area network (LAN), a wide area network (WAN), and / or a public grid, such as the Internet) through a grid adapter 660. The grid adapter 660 may communicate with other modules of the electronic device 600 through the bus 630. It should be understood that although Fig. 9 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0146] In an embodiment of the present invention, a computer-readable storage medium is also provided for storing a program, and the steps of a transparent object detection, segmentation and depth correction method are implemented when the program is executed. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above transparent object detection, segmentation and depth correction method section of this specification.

[0147] As shown above, this embodiment is based on an RGBD camera to obtain an RGB image, a speckle infrared image and a depth map, identifies the range of transparent objects through a first deep learning network, and then corrects the depth value to obtain the final depth map, which can accurately identify transparent objects in various application scenarios and solve the problem of inaccurate recognition in the prior art.

[0148] Fig.10 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Fig.10 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0149] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0150] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0151] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of grid, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0152] This embodiment is based on an RGBD camera to obtain an RGB image, a speckle infrared image and a depth image. The range of transparent objects is identified through a first deep learning network, and then the depth value is corrected to obtain a final depth map. This can accurately identify transparent objects in various application scenarios and solve the problem of inaccurate identification in the prior art.

[0153] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this article, but will comply with the widest range consistent with the principles and novel features disclosed herein.

[0154] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A transparent object detection, segmentation and depth correction method, characterized in that: include: Step S1: Obtain RGB image, speckle infrared image and depth image; Step S2: identifying the range of transparent objects and the boundary of transparent objects through a first deep learning network; wherein the first deep learning network obtains the range of transparent objects and the boundary of transparent objects according to the astigmatism characteristics of speckle irradiation on the transparent objects; Step S3: According to the depth map and the speckle infrared map, combined with the depth value of the boundary of the transparent object, the depth value within the range of the transparent object is corrected to obtain a final depth map.

2. A transparent object detection, segmentation and depth correction method according to claim 1, characterized in that: Step S1 includes: Step S11: obtaining an RGB image and a speckle infrared image; wherein the speckle infrared image has at least two images; Step S12: Calculate the disparity according to the two speckle infrared images to obtain a depth map.

3. A transparent object detection, segmentation and depth correction method according to claim 2, characterized in that: The speckle infrared image has two images and also includes: Step S13: performing pixel-level reprojection alignment on the depth map and the RGB map.

4. The method for detecting, segmenting and correcting a transparent object according to claim 1, characterized in that: The first deep learning network is a Unet network, which includes an encoder, a first decoder and a second decoder; the first decoder outputs the range of the transparent object; the second decoder outputs the boundary of the transparent object; and there is feature interaction between features of each level of the first decoder and the second decoder.

5. The method for detecting, segmenting and correcting a transparent object according to claim 1, characterized in that: In step S3, the depth map, the transparent object range, the transparent object boundary and the speckle infrared map are input into a second deep learning network to obtain a final depth map; wherein the second deep learning network corrects the depth value within the transparent object range in combination with the transparent object boundary.

6. The method for detecting, segmenting and correcting a transparent object according to claim 1, characterized in that: Step S3 includes: Step S31: performing three-dimensional reconstruction on the portion outside the range of the transparent object according to the depth map and the speckle infrared map; Step S32: obtaining three-dimensional information of the boundary of the transparent object, and identifying the shape of the boundary of the transparent object in three-dimensional space; Step S33: performing plane fitting according to the boundary of the transparent object to obtain a depth value corresponding to each pixel point within the range of the transparent object.

7. A transparent object detection, segmentation and depth correction method according to claim 6, characterized in that: Step S33 includes: Step S331: performing straight line fitting on the boundary of the transparent object to obtain a plurality of straight lines in a three-dimensional space; Step S332: fitting the plane of three consecutive straight lines to obtain a transparent object; Step S333: Calculate the depth value corresponding to each pixel point within the range of the transparent object according to the transparent object.

8. A transparent object detection, segmentation and depth correction system, used to implement the transparent object detection, segmentation and depth correction method according to any one of claims 1 to 7, characterized in that: include: An acquisition module is used to acquire RGB images, speckle infrared images and depth images; An identification module, used to identify the range of transparent objects and the boundary of transparent objects through a first deep learning network; wherein the first deep learning network obtains the range of transparent objects and the boundary of transparent objects according to the astigmatism characteristics of speckle irradiation on transparent objects; A correction module is used to correct the depth value within the range of the transparent object according to the depth map and the speckle infrared map in combination with the depth value of the boundary of the transparent object to obtain a final depth map.

9. A transparent object detection, segmentation and depth correction device, characterized in that: include: processor; a memory storing executable instructions of the processor; Wherein, the processor is configured to perform the steps of the transparent object detection, segmentation and depth correction method described in any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed, the steps of the transparent object detection, segmentation and depth correction method described in any one of claims 1 to 7 are implemented.

Images