Transparent object detection method, system and equipment based on time sequence and storage medium

Abstract

The invention discloses a transparent object detection method, system and device based on a time sequence, and a storage medium. The method comprises the following steps: S1, obtaining a first image set and a second image set; wherein each of the first image set and the second image set comprises a dense depth map, a sparse depth map and an infrared map; the first image set and the second image set are obtained at different times; s2, inputting the first image set and the second image set into a recognition model, and recognizing a transparent object range according to the difference between the first image set and the second image set; wherein the transparent object range is a point corresponding to the dense depth map. According to the invention, the depth values of various transparent objects can be accurately identified.

Description

Background Art

[0002] In today's era of rapid intelligent development, object detection technology plays a key role in many fields, such as product sorting on industrial automation production lines, target recognition in intelligent security monitoring, and inventory in logistics warehousing. However, most of the existing object detection methods are designed and optimized for opaque objects, and encounter many difficulties when facing transparent objects.

[0003] On the one hand, the unique optical properties of transparent objects pose a detection challenge. Since light can directly penetrate transparent objects, their surfaces lack sufficiently obvious conventional visual features such as texture and color. Traditional detection methods based on visible light images often find it difficult to extract effective identification information, resulting in a significant reduction in detection accuracy. For example, in the food packaging industry, the sorting of products in transparent plastic packaging often results in missed inspections and false inspections, which seriously affects production efficiency and quality control.

[0004] On the other hand, depth maps, as important data for auxiliary detection, also have limitations when dealing with transparent objects. Whether it is a dense depth map or a sparse depth map, the depth measurement of transparent objects is prone to deviations. The refraction phenomenon of transparent objects makes the light propagation path complicated, which in turn interferes with the normal operation of the depth sensor. The acquired depth information is inaccurate and unstable, and cannot accurately reflect the true boundary and position of the object, making it difficult to accurately define the scope of the transparent object by relying solely on the depth map.

[0005] At the same time, most existing recognition models do not fully consider the special needs of transparent object detection in their architecture design, especially the use of temporal information. In dynamic scenes, the subtle movement of transparent objects over time, changes in posture, and dynamic changes in the refraction angle of light all contain valuable recognition clues, but most current models ignore these temporal features and only focus on the static analysis of a single image, resulting in insufficient recognition of transparent objects.

[0006] Although infrared imaging technology helps to highlight the outline of transparent objects to a certain extent, it is still difficult to deal with the confusion between transparent objects and backgrounds in complex scenes by relying solely on infrared images, and misjudgment occurs frequently. Overall, there is currently a lack of a systematic transparent object detection method that can effectively fuse multimodal images and utilize time series differences. Therefore, an innovative detection solution is urgently needed to overcome the industry pain points of low accuracy and poor reliability in transparent object recognition and meet the growing demand for high-precision detection in various fields.

[0007] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0008] To this end, the present invention provides a time-series-based transparent object detection solution to address the requirements for the range and depth of transparent objects in various application scenarios.

[0009] In a first aspect, the present invention provides a time-series-based transparent object detection method, characterized by comprising:

[0010] Step S1: Acquire a first image set and a second image set; wherein the first image set and the second image set both include a dense depth map, a sparse depth map and an infrared map; and the first image set and the second image set are acquired at different times;

[0011] Step S2: input the first image set and the second image set into a recognition model, and identify the range of transparent objects according to the difference between the first image set and the second image set; wherein the range of transparent objects is the points corresponding to the dense depth map.

[0012] Optionally, the time series-based transparent object detection method is characterized in that step S1 comprises:

[0013] Step S11: obtaining an LED infrared image and a speckle infrared image at a first moment, and calculating a dense depth map and a sparse depth map at the first moment;

[0014] Step S12: Obtain the LED infrared image and the speckle infrared image at the second moment, and calculate the dense depth map and the sparse depth map at the second moment.

[0015] Optionally, the time series-based transparent object detection method is characterized in that the infrared bands at the first moment and the second moment are different.

[0016] Optionally, the time series-based transparent object detection method is characterized in that step S2 comprises:

[0017] Step S21: extracting features from the dense depth map, the sparse depth map, and the infrared map at the first moment respectively to obtain first dense features, first sparse features, and first infrared features, and assigning first weights to the dense depth map, the sparse depth map, and the infrared map at the first moment respectively;

[0018] Step S22: extracting features from the dense depth map, the sparse depth map, and the infrared map at the second moment respectively to obtain second dense features, second sparse features, and second infrared features, and assigning second weights according to the dense depth map, the sparse depth map, and the infrared map at the second moment respectively;

[0019] Step S23: According to the first weight, the first dense feature, the first sparse feature and the first infrared feature at the first moment are fused to obtain a first fused image; according to the second weight, the second dense feature, the second sparse feature and the second infrared feature at the second moment are fused to obtain a second fused image;

[0020] Step S24: using a recognition model to recognize the first fusion image and the second fusion image to obtain a transparent object range.

[0021] Optionally, the time series-based transparent object detection method is characterized in that the recognition model is a hierarchical multi-scale convolutional neural network.

[0022] Optionally, the time series-based transparent object detection method is characterized in that the recognition model also includes a self-supervised learning module during training; the common features output by the self-supervised module are injected into the recognition model, and the weights are jointly updated.

[0023] Optionally, the time series-based transparent object detection method is characterized by further comprising:

[0024] Step S3: Correcting the depth value of the transparent object using the depth value of the non-transparent object that the transparent object range contacts.

[0025] In a second aspect, the present invention provides a time-series based transparent object detection system, which is used to implement any of the time-series based transparent object detection methods described above, and is characterized by comprising:

[0026] An acquisition module, configured to acquire a first image set and a second image set; wherein the first image set and the second image set both include a dense depth map, a sparse depth map, and an infrared map; and the first image set and the second image set are acquired at different times;

[0027] A recognition module is used to input the first image set and the second image set into a recognition model, and identify a transparent object range according to the difference between the first image set and the second image set; wherein the transparent object range is the point corresponding to the dense depth map.

[0028] In a third aspect, the present invention provides a time-series-based transparent object detection device, characterized in that it includes:

[0029] processor;

[0030] a memory storing executable instructions of the processor;

[0031] Wherein, the processor is configured to perform the steps of any of the aforementioned time-series-based transparent object detection methods by executing the executable instructions.

[0032] In a fourth aspect, the present invention provides a computer-readable storage medium for storing a program, characterized in that when the program is executed, the steps of any of the aforementioned time-series-based transparent object detection methods are implemented.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention collects dense depth maps, sparse depth maps and infrared maps at the same time to form a rich image set. The dense depth map can capture the fine depth information of the scene and present the precise position of the object in the three-dimensional space; although the sparse depth map has few data points, it can quickly outline the approximate depth contour, and the two complement each other. The infrared image uses the difference in thermal radiation of the object to highlight the outline of the transparent object, and can also highlight the target object even in complex lighting environments. The fusion of the three modal data makes up for the defect of insufficient representation of transparent objects by a single data type, and depicts the characteristics of transparent objects from all directions and angles, so that subsequent identification has a more solid data foundation and reduces the risk of misjudgment.

[0035] The present invention obtains the first image set and the second image set at different times, and cleverly uses the dynamic changes of transparent objects over time. In continuous time segments, transparent objects will produce a series of subtle differences due to their own movement, slight changes in ambient light, and changes in refraction. By inputting image sets at different times into the recognition model, the model can capture these dynamic characteristics and dig out more hidden distinguishing features. Compared with the method of only analyzing a single static image, this significantly improves the recognition accuracy, and can also accurately locate transparent objects that are in a slow motion state or in scenes with unstable ambient light.

[0036] The present invention inputs two sets of multimodal image sets into the recognition model, and uses the model to analyze the differences between the two sets of data to accurately lock the range of transparent objects and specifically indicate the points of the dense depth map corresponding to the range. Due to the high accuracy of the dense depth map, the boundaries of the transparent object range determined based on it are clearer and more accurate, avoiding the problems of blurred boundaries and inaccurate positioning that often occur in traditional methods. It can directly serve industrial scenarios that require high-precision positioning of transparent objects, such as the detection of transparent protective covers on precision electronic component assembly lines, greatly reducing product quality problems caused by detection errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings in the following descriptions are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:

[0038] Figure 1 is a flow chart of the steps of a time-series-based transparent object detection method in an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of a network architecture in an embodiment of the present invention;

[0040] Figure 3 Schematic diagram of various images in an embodiment of the present invention;

[0041] Figure 4 A flowchart of the steps of acquiring a first image set and a second image set in an embodiment of the present invention;

[0042] Figure 5 This is a flowchart of steps for identifying the range of a transparent object in an embodiment of the present invention;

[0043] Figure 6 is a flowchart of another method for detecting transparent objects based on time series in an embodiment of the present invention;

[0044] Figure 7 Schematic diagram of the structure of a time-series-based transparent object detection system in an embodiment of the present invention;

[0045] Figure 8 is a schematic structural diagram of a time-series-based transparent object detection device in an embodiment of the present invention; and

[0046] Fig. 9 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0048] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0049] The embodiment of the present invention provides a time series based transparent object detection method, which aims to solve the problems existing in the prior art.

[0050] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0051] The present invention obtains a first image set and a second image set at different times for comparative identification, and uses two different depth maps, namely a dense depth map and a sparse depth map, in combination with an infrared image, to input a recognition model to obtain the range of transparent objects. It has a better recognition effect on transparent objects and can output an accurate range of transparent objects in real time, greatly improving the recognition effect on transparent objects.

[0052] Figure 1 FIG. 1 is a flow chart of the steps of a time-series-based transparent object detection method according to an embodiment of the present invention. Figure 1 As shown, the steps of a time-series-based transparent object detection method in an embodiment of the present invention include:

[0053] Step S1: Acquire a first image set and a second image set.

[0054] In this step, the first image set and the second image set both include a dense depth map, a sparse depth map, and an infrared map; the first image set and the second image set are obtained at different times. The dense depth map can provide depth information for almost every pixel in the scene, and is acquired through a professional depth sensor, such as a sensor based on structured light and time-of-flight (ToF) principles. It records in detail the distance from the surface of each object in the scene to the sensor, and the pixel-level depth data is very rich, which is critical for the subsequent analysis of the spatial position relationship of the object.

[0055] Compared with dense depth maps, sparse depth maps have fewer data points, and the depth values ​​of specific key points are often obtained through some feature point matching algorithms. For example, feature point extraction algorithms such as ORB and SIFT are used to first find significant feature points and then estimate their depth. Although the data is sparse, it can also reflect the key structural information of the scene and has a low computational cost.

[0056] Infrared images are collected using infrared imaging equipment. Infrared bands are sensitive to temperature. Different objects have different grayscale or color distributions in infrared images due to their own temperature and material thermal radiation characteristics. It can capture the outline of objects that are difficult to see under visible light, especially in complex scenes such as low light and obstruction, and provide key information for subsequent identification.

[0057] The first and second image sets are acquired at different times intentionally. This is to take advantage of the fact that transparent objects will produce different dynamic performances from non-transparent objects at different times due to factors such as their own movement, changes in ambient lighting, and movement of background objects, providing dynamic clues for subsequent detection. For example, as time goes by, the water in a transparent water cup may shake, and the refraction angle of the surrounding light through the water will also change.

[0058] Step S2: input the first image set and the second image set into a recognition model, and identify the range of transparent objects according to the difference between the first image set and the second image set.

[0059] In this step, the transparent object range is the point corresponding to the dense depth map. The two sets of image sets are completely input into the pre-trained recognition model. This recognition model can be built based on a deep learning architecture, such as a convolutional neural network (CNN) and its variants. The model is trained on a large number of image sets containing transparent objects and non-transparent objects in the early stage to learn the characteristic difference patterns of the two types of objects.

[0060] Comparing the dense depth maps in the first and second image sets, the depth values ​​of non-transparent objects are relatively stable at two time points because their physical positions are fixed and their shapes remain unchanged; however, due to their optical properties, such as dynamic changes in refraction and reflection, and their possible displacement, the corresponding depth values ​​of transparent objects will fluctuate significantly. By calculating the depth difference of the pixel at the same spatial position in the dense depth map at different times, the area where the suspected transparent object is located can be preliminarily screened.

[0061] The depth changes of key feature points in the sparse depth map, as well as the changes in the object contour and temperature distribution in the infrared map, are also included in the analysis. The recognition model integrates the differences in these multimodal image information, and uses the built-in feature extraction layer and classification layer to continuously aggregate and classify the feature differences at different times, and finally determines the feature areas that are highly correlated with transparent objects. The corresponding points of these areas on the dense depth map accurately outline the range of transparent objects. For example, transparent glass products have blurred contours and special temperature conduction in infrared images. Combined with abnormal changes in the depth map, the model can accurately lock its location range.

[0062] In some embodiments, the recognition model is a hierarchical multi-scale convolutional neural network. The hierarchical multi-scale convolutional neural network is to process the input fusion graph more finely and comprehensively, and to mine the hidden transparent object features therein. It adopts a hierarchical architecture in the network architecture design and incorporates multi-scale convolution operations to adapt to the feature extraction requirements of objects of different sizes and levels.

[0063] Receive the first fusion image and the second fusion image generated in step S23, perform preliminary formatting on the image data, and convert it into a tensor format that the network can process, in preparation for subsequent convolution calculations. This step ensures that the data dimension and type meet the calculation requirements of the subsequent network layer.

[0064] The recognition model is divided into shallow, medium and deep layers.

[0065] In the shallow part of the network, the convolution kernel size is relatively small, such as 3x3 or 5x5. These small convolution kernels focus on extracting subtle texture features of the local image. Whether it is the light texture anomaly caused by the refraction of transparent objects in the fusion image, or the slight change in the edge of the object contour, the shallow layer can keenly capture it. Moreover, due to the layered design, the shallow layer can quickly process simple basic features, reducing the computational burden of the subsequent deep network and improving the overall computing efficiency.

[0066] The middle-layer network will appropriately increase the size of the convolution kernel, such as 7x7, and increase the number of stacked convolution layers. This allows it to integrate the local features extracted by the shallow layer and begin to capture the structural information of objects at a slightly larger scale. When identifying transparent objects, the middle layer begins to gather scattered texture features, identify the approximate shape of the object, and determine whether it meets some typical shape features of transparent objects, such as a flat glass panel, a round transparent sphere outline, etc.

[0067] In the deepest part of the network, the convolution kernel is further enlarged, and may be 9x9 or larger. The main task of the deep layer is to grasp the scene structure from a macro perspective, integrate the object shape information from the middle layer with the feature relationship of the wider area, and judge the layout and position relationship of the transparent objects in the entire scene, such as determining whether the transparent container is placed in the center of the table or at the edge of a pile of objects.

[0068] There are multiple groups of convolution kernels of different scales in the network, which perform convolution operations on the input fusion map in parallel. For example, a group of 3x3 convolution kernels captures fine textures, while a group of 7x7 convolution kernels captures medium-scale structures, and large-scale convolution kernels such as 9x9 are responsible for overall layout perception. This parallel processing mechanism ensures that features of different sizes within the same level are not missed, and covers all kinds of features that may be presented by transparent objects.

[0069] The feature maps obtained by convolution at different scales are not independent of each other. In the subsequent process, the multi-scale features will be integrated through the feature fusion layer by means of point-by-point addition and splicing, so that the features of each scale can complement each other. This makes the features output by each layer contain both fine local details and macro layout information, and transmits more and more complete transparent object features to deeper layers, helping to accurately locate the range of transparent objects.

[0070] Finally, a feature map corresponding to the size of the input fusion image is output. The pixel range of the corresponding transparent object is marked in the map, clearly defining the area where the transparent object is located after layer-by-layer analysis and feature fusion.

[0071] In some embodiments, the recognition model also includes a self-supervised learning module during training; the common features output by the self-supervised module are injected into the recognition model, and the weights are jointly updated. Self-supervised learning aims to generate supervision signals using the intrinsic structure of the data itself, without relying on large-scale manually annotated data. In this time-series-based transparent object detection and recognition model training scenario, the self-supervised learning module can mine common features in the input image data (such as various images at the first and second moments). These features reflect some common properties of scenes and objects, and are not labeled for specific transparent objects.

[0072] The self-supervised learning module first extracts features from the input image data (including dense depth maps, sparse depth maps, infrared images, etc.) using a method similar to the convolution operation in the recognition model. It uses the hierarchical architecture of the convolutional neural network to gradually extract common features from small-scale local textures to large-scale scene layouts. For example, it can capture the common laws of ambient lighting changes at different times and the fixed geometric shape features of background objects. Although these features seem to have no direct connection with transparent object detection, they contain rich scene context information.

[0073] The extracted universal features will be injected into the recognition model. This injection process occurs at multiple levels of the recognition model, not just the input layer. At the shallow level, universal features can help fine-tune the initial texture features extracted by the small convolution kernel, making the recognition model more robust to the learning of basic textures and avoiding overfitting to specific samples; at the middle level, universal features are combined with the shape features of the objects being aggregated to improve the cognition of the object structure; at the deep level, universal features related to the scene layout further calibrate the macro-level judgment, allowing the recognition model to more accurately grasp the environment in which the transparent object is located from a holistic perspective.

[0074] When the recognition model with common features performs forward operation and outputs the predicted range of transparent objects, it will be compared with the labeled information in the training data (if there are a small number of labeled samples) or the pseudo-labeled information generated based on the data's own rules to calculate the loss value. This loss value measures the degree of deviation between the predicted result and the expected result. Then, through the back-propagation algorithm, the error is transmitted from the output layer to the back layer layer by layer.

[0075] During the back propagation process, not only are the weight parameters of the recognition model itself updated according to the gradient descent rule, but the weights corresponding to the general features output by the self-supervised learning module are also optimized synchronously. A synergistic relationship is formed between the recognition model and the self-supervised learning module, and the two are constantly adapted and adjusted, so that the recognition model can not only learn the detection features unique to transparent objects, but also integrate the general characteristics of the scene with the help of self-supervised learning, thereby improving the generalization ability of transparent object detection in different scenes.

[0076] Figure 2 FIG. 1 is a schematic diagram of a network architecture in an embodiment of the present invention. Figure 2 As shown in the figure, the recognition model includes an encoder and a decoder. The dense depth map, sparse depth map and infrared image are input into the encoder for feature extraction. Then they are decoded step by step by the decoder to finally obtain the range of transparent objects. Figure 2 The image in the middle is a scene arrangement diagram, which is only used to illustrate the scene and does not participate in the processing in this embodiment. Figure 2 It can be seen that the transparent object range is not a strict range of transparent objects, but a depth range corresponding to the depth map, which is different from the range for RGB images in the prior art.

[0077] Figure 3 Schematic diagram of various images in the embodiments of the present invention. Figure 3 As shown, a is a scene layout diagram, which is only used to illustrate the scene situation and does not participate in the calculation and processing of this embodiment. b is an infrared image, c is a dense depth image, and d is a sparse depth image. b, c, and d are input into the recognition model for processing.

[0078] Figure 4 FIG. 1 is a flow chart of steps for obtaining a first image set and a second image set in an embodiment of the present invention. Figure 4 As shown, in an embodiment of the present invention, a step of acquiring a first image set and a second image set includes:

[0079] Step S11: obtaining the LED infrared image and the speckle infrared image at the first moment, and calculating the dense depth map and the sparse depth map at the first moment.

[0080] In this step, the LED infrared image is an image obtained by illuminating the scene with infrared light of a specific wavelength emitted by a light-emitting diode. Different objects reflect and absorb infrared light differently, resulting in different brightness and texture features on the image. The speckle infrared image is generated based on the speckle structured light technology. Speckle is a randomly distributed bright spot pattern. When the infrared light of the speckle pattern is projected onto the surface of the scene object, the reflected speckle will be deformed due to factors such as the distance and roughness of the object surface. This deformation information is critical for depth calculation. At the first moment, these two types of infrared images are collected simultaneously through corresponding infrared cameras and other equipment.

[0081] Using the acquired LED infrared image and speckle infrared image, an algorithm based on the triangulation principle is used to calculate a dense depth map. For example, there is an accurate mathematical relationship between the degree of speckle deformation and the distance to the surface of an object. Combined with the surface texture and light feedback provided by the LED infrared image to assist in positioning, after complex geometric calculations, the depth values ​​of many pixels in the scene are calculated one by one to form a dense depth information map.

[0082] Through feature point extraction algorithms, such as SIFT (Scale-Invariant Feature Transform) and SURF (Speeded Up Robust Features), we first find significant and easy-to-track feature points on the LED infrared image and speckle infrared image, and then estimate the depth of these few feature points based on the binocular vision principle or structured light motion information, thereby obtaining a sparse depth map. Although it has a small amount of data, it can capture the key structural features of the scene.

[0083] Step S12: Obtain the LED infrared image and the speckle infrared image at the second moment, and calculate the dense depth map and the sparse depth map at the second moment.

[0084] In this step, at the second moment, the image acquisition operation in step S11 is repeated, and the same infrared acquisition device is used again to capture the LED infrared image and speckle infrared image at this time. As time has changed, the state of objects, lighting conditions, object positions, etc. in the scene may have changed, and these changes are crucial for the subsequent detection of transparent objects.

[0085] The same depth calculation algorithm as step S11 is used to process the newly acquired LED infrared image and speckle infrared image at the second moment, and respectively calculate the dense depth map and sparse depth map at the second moment. With the depth maps at two different moments and the corresponding infrared images, sufficient data foundation is prepared for the subsequent input recognition model and finding the range of transparent objects.

[0086] In some embodiments, the infrared bands at the first moment and the second moment are different. Different infrared bands correspond to different physical properties and imaging effects. For example, short-wave infrared (SWIR) has weak absorption of moisture, can penetrate a certain degree of fog and smoke, and the imaging picture is relatively clear; long-wave infrared (LWIR) is more sensitive to temperature and is suitable for capturing images formed by the object's own thermal radiation. The LED infrared image and speckle infrared image of a specific infrared band collected at the first moment will present unique scene information based on the band, and the contrast and brightness distribution of objects in the image are closely related to the band.

[0087] To obtain infrared images of different bands, you need to use specially adapted infrared sensors or cameras. Because different bands of infrared have special requirements in detector materials, optical lens coatings, etc., when collecting, you need to switch to the corresponding device that can accurately capture the infrared of the band to capture LED infrared images and speckle infrared images.

[0088] Due to the different infrared bands, the noise characteristics, resolution, contrast, etc. of the image will change, and the algorithm used to calculate the dense depth map may need to be fine-tuned. For example, in the algorithm based on triangulation, some parameters of the light propagation model need to be recalibrated in different bands to cope with changes in image data quality, so as to accurately infer the depth of each pixel. For the calculation of sparse depth maps, the feature point extraction algorithm must also be optimized based on the texture and grayscale distribution characteristics of the image in the new band to ensure that stable and reliable feature points are extracted to estimate the depth.

[0089] At the second moment, switch to another infrared band for image acquisition. Also enable the dedicated infrared acquisition device that matches this band, and capture the LED infrared image and speckle infrared image again. At this moment, the objects in the scene will show a completely new appearance in the new infrared band, which is in sharp contrast to the first moment. This contrast provides more clues for finding transparent objects later.

[0090] When repeating the depth calculation process, the impact of the different bands from the first moment must be fully considered. In addition to recalibrating the parameters of the depth calculation algorithm, attention should also be paid to the normalization of data under different bands, so that the depth map data at two moments can have a reasonable comparison basis in the subsequent recognition model, so as to use the rich information differences due to band switching and time changes to accurately locate the range of transparent objects.

[0091] Figure 5 FIG. 1 is a flow chart of steps for identifying the range of a transparent object in an embodiment of the present invention. Figure 5 As shown, in an embodiment of the present invention, a step of identifying a range of a transparent object includes:

[0092] Step S21: extract features from the dense depth map, sparse depth map and infrared map at the first moment respectively to obtain first dense features, first sparse features and first infrared features, and assign first weights according to the dense depth map, sparse depth map and infrared map at the first moment respectively.

[0093] In this step, for the dense depth map at the first moment, a specific convolution layer in the convolutional neural network (CNN) is used to extract features. The convolution kernel of the CNN slides on the dense depth map to capture depth change patterns of different scales, such as gradient changes in depth values, depth fluctuations in local areas, etc. These captured information are aggregated into the first dense feature. Since the dense depth map is rich in pixel-level depth information, the extracted features can accurately reflect the spatial structure of the scene.

[0094] For the sparse depth map at the first moment, feature extraction focuses on those key feature points that have been determined. Using the local descriptor algorithm, the depth relationship between the sparse feature points and their neighborhood is converted into numerical features, such as a vector representation that describes the depth difference around the feature points, and then the first sparse feature is obtained. Although it has a small amount of data, it captures the key structural information of the scene.

[0095] Using a convolutional neural network pre-trained model for infrared images, or manually designing a convolution kernel suitable for infrared textures, we extract the texture features formed by the object contour and temperature distribution in the infrared image to obtain the first infrared feature. Infrared images can capture the performance of objects hidden under visible light, and the extracted features provide key perspectives for subsequent recognition.

[0096] According to the importance of each image in identifying transparent objects in previous experiments and data set analysis, weights are assigned to the dense depth map, sparse depth map, and infrared image at the first moment. For example, if experience shows that the dense depth map contributes the most to the accuracy of transparent object detection in the early stage, then its corresponding weight is higher, and the weight setting is in preparation for subsequent feature fusion.

[0097] Step S22: extract features from the dense depth map, sparse depth map and infrared map at the second moment respectively to obtain second dense features, second sparse features and second infrared features, and assign second weights according to the dense depth map, sparse depth map and infrared map at the second moment respectively.

[0098] In this step, similar to step S21, the same type of feature extraction algorithm is used for the dense depth map, sparse depth map and infrared map at the second moment to respectively mine the depth change, key structure, infrared texture and other features therein, and obtain the second dense feature, the second sparse feature and the second infrared feature in turn. Even if there are differences between the images at the two moments, the unified extraction method ensures that similar information can be compared.

[0099] Based on past experience and data analysis, we also consider the performance of each image in the transparent object recognition task and assign weights to the three images at the second moment. Due to the different time periods and dynamic changes in the scene, the weights of each image may be slightly different from the first moment to adapt to the characteristics of the scene at this moment for subsequent fusion.

[0100] Step S23: According to the first weight, the first dense feature, the first sparse feature and the first infrared feature at the first moment are fused to obtain a first fusion image; according to the second weight, the second dense feature, the second sparse feature and the second infrared feature at the second moment are fused to obtain a second fusion image.

[0101] In this step, the first dense feature, the first sparse feature and the first infrared feature are weighted fused according to the set first weight. For example, the weighted summation formula is used to add the corresponding elements of different feature matrices according to the weights, and these features are integrated into a unified feature representation, namely the first fusion graph. This fusion graph integrates the key information of different modes at the first moment, avoiding the limitations of a single image information.

[0102] Analogously to the generation process of the first fusion map, the second dense feature, the second sparse feature and the second infrared feature are weighted and fused according to the second weight to create the second fusion map. The two fusion maps condense the key recognition elements of the scenes at two different times, and are ready for comparison in the recognition model.

[0103] Step S24: using a recognition model to recognize the first fusion image and the second fusion image to obtain a transparent object range.

[0104] In this step, the first fusion image and the second fusion image are input into a pre-trained recognition model, which can be based on a long short-term memory network (LSTM) that is good at processing time series information, or a convolutional neural network with a temporal attention mechanism. The model has learned a large number of time series image pairs with transparent objects, and can capture the subtle differences between the two fusion images, such as the evolution of features over time, and then accurately locate those areas that have unique changes due to the optical properties and motion characteristics of transparent objects, and finally obtain the range of transparent objects.

[0105] Figure 6FIG. 1 is a flow chart of another method for detecting transparent objects based on time series in an embodiment of the present invention. Figure 6 As shown, compared with the above-mentioned embodiment, another time-series-based transparent object detection method in the embodiment of the present invention further includes:

[0106] Step S3: Correcting the depth value of the transparent object using the depth value of the non-transparent object that the transparent object range contacts.

[0107] In this step, after completing the identification of the range of transparent objects, due to the special optical properties of transparent objects, such as refraction and reflection, the depth values ​​directly obtained often have large errors, while the depth values ​​of non-transparent objects in contact with them are relatively more accurate, so using the latter to correct the former can improve the accuracy of the overall depth data.

[0108] From the dense depth map obtained in the previous step, extract the depth values ​​of the non-transparent objects at the boundary of the transparent object range and in contact with it. Because non-transparent objects do not produce optical phenomena such as light refraction that interfere with depth measurement, these depth values ​​are more reliable. For example, if a transparent glass cup is placed on a wooden table, the accurate table depth value corresponding to the contact part of the table and the cup is obtained from the dense depth map.

[0109] Similarly, based on the previously generated dense depth map, the depth values ​​within the range of the identified transparent objects are determined. However, these values ​​are affected by refraction and are likely to deviate from the true depth, which is the object of subsequent correction.

[0110] The correction can be based on geometric relationship or multi-contact point weighted correction.

[0111] Correction based on geometric relationships: According to simple geometric principles, it is assumed that the transparent object and the non-transparent object fit closely together, and the depth of the two should be consistent on the contact surface. Taking plane contact as an example, if one side of the transparent object contacts a non-transparent plane object with a depth of d1, the depth value of the transparent object on this side of the contact surface is directly updated to d1. For irregular contact situations, by fitting the contact boundary curve, the depth of the non-transparent object corresponding to each point on the curve is used to correct the depth value of the transparent object point by point.

[0112] Multi-contact point weighted correction: When a transparent object contacts multiple non-transparent objects, the depth values ​​d1, d2, ..., d corresponding to each contact point are obtained. n At the same time, weights w1, w2, …, w are assigned considering factors such as the stability of different contact points and the size of the contact area. n , the sum of weights is 1. Through the weighted summation formula d = e1d1 + w2d2 + ... + w n d nThe corrected comprehensive depth value is calculated and applied to the corresponding area of ​​the transparent object, so that the depth value of the transparent object is more consistent with the real scene depth, providing reliable depth data for subsequent applications such as 3D reconstruction and robot obstacle avoidance.

[0113] Figure 7 FIG. 1 is a schematic diagram of a transparent object detection system based on time series in an embodiment of the present invention. Figure 7 As shown, a time-series-based transparent object detection system in an embodiment of the present invention includes:

[0114] An acquisition module, configured to acquire a first image set and a second image set; wherein the first image set and the second image set both include a dense depth map, a sparse depth map, and an infrared map; and the first image set and the second image set are acquired at different times;

[0115] A recognition module is used to input the first image set and the second image set into a recognition model, and identify a transparent object range according to the difference between the first image set and the second image set; wherein the transparent object range is the point corresponding to the dense depth map.

[0116] This embodiment obtains the first image set and the second image set at different times for comparative identification, and uses two different depth maps, namely a dense depth map and a sparse depth map, combined with an infrared image, to input a recognition model to obtain the range of transparent objects. It has a better recognition effect on transparent objects and can output an accurate range of transparent objects in real time, greatly improving the recognition effect on transparent objects.

[0117] The present invention also provides a time-series based transparent object detection device in an embodiment, comprising a processor and a memory, wherein executable instructions of the processor are stored, wherein the processor is configured to execute the steps of a time-series based transparent object detection method by executing the executable instructions.

[0118] As mentioned above, this embodiment obtains the first image set and the second image set at different times for comparative identification, and uses two different depth maps, namely a dense depth map and a sparse depth map, combined with an infrared image, to input the recognition model to obtain the range of transparent objects. It has a better recognition effect on transparent objects and can output the accurate range of transparent objects in real time, greatly improving the recognition effect on transparent objects.

[0119] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0120] Figure 8 is a schematic diagram of the structure of a time-series based transparent object detection device in an embodiment of the present invention. Figure 8 The electronic device 600 according to this embodiment of the present invention is described. Figure 8 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0121] like Figure 8 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0122] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps of various exemplary embodiments of the present invention described in the above-mentioned time-series-based transparent object detection method section of this specification. For example, the processing unit 610 can execute the following steps: Figure 1 Follow the steps shown in .

[0123] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0124] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a grid environment.

[0125] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0126] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more grids (e.g., a local area network (LAN), a wide area network (WAN), and / or a public grid, such as the Internet) through a grid adapter 660. The grid adapter 660 may communicate with other modules of the electronic device 600 through the bus 630. It should be understood that although Figure 8 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0127] In an embodiment of the present invention, a computer-readable storage medium is also provided for storing a program, and when the program is executed, the steps of a method for detecting a transparent object based on a time series are implemented. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above-mentioned method for detecting a transparent object based on a time series in this specification.

[0128] As shown above, this embodiment obtains the first image set and the second image set at different times for comparative identification, and uses two different depth maps, namely a dense depth map and a sparse depth map, combined with an infrared image, to input the recognition model to obtain the range of transparent objects. It has a better recognition effect on transparent objects and can output the accurate range of transparent objects in real time, greatly improving the recognition effect on transparent objects.

[0129] Fig. 9 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Fig. 9 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0130] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0131] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0132] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of grid, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0133] This embodiment obtains the first image set and the second image set at different times for comparative identification, and uses two different depth maps, namely a dense depth map and a sparse depth map, combined with an infrared image, to input a recognition model to obtain the range of transparent objects. It has a better recognition effect on transparent objects and can output an accurate range of transparent objects in real time, greatly improving the recognition effect on transparent objects.

[0134] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this article, but will comply with the widest range consistent with the principles and novel features disclosed herein.

[0135] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A time series based transparent object detection method, characterized in that: include: Step S1: Acquire a first image set and a second image set; wherein the first image set and the second image set both include a dense depth map, a sparse depth map and an infrared map; and the first image set and the second image set are acquired at different times; Step S2: input the first image set and the second image set into a recognition model, and identify a transparent object range according to the difference between the first image set and the second image set; wherein the transparent object range is the points corresponding to the dense depth map.

2. The method for detecting transparent objects based on time series according to claim 1, characterized in that: Step S1 includes: Step S11: obtaining an LED infrared image and a speckle infrared image at a first moment, and calculating a dense depth map and a sparse depth map at the first moment; Step S12: Obtain the LED infrared image and the speckle infrared image at the second moment, and calculate the dense depth map and the sparse depth map at the second moment.

3. The method for detecting transparent objects based on time series according to claim 2, characterized in that: The infrared bands at the first moment and the second moment are different.

4. The method for detecting transparent objects based on time series according to claim 1, characterized in that: Step S2 includes: Step S21: extracting features from the dense depth map, the sparse depth map, and the infrared map at the first moment respectively to obtain first dense features, first sparse features, and first infrared features, and assigning first weights to the dense depth map, the sparse depth map, and the infrared map at the first moment respectively; Step S22: extracting features from the dense depth map, the sparse depth map, and the infrared map at the second moment respectively to obtain second dense features, second sparse features, and second infrared features, and assigning second weights according to the dense depth map, the sparse depth map, and the infrared map at the second moment respectively; Step S23: According to the first weight, the first dense feature, the first sparse feature and the first infrared feature at the first moment are fused to obtain a first fused image; according to the second weight, the second dense feature, the second sparse feature and the second infrared feature at the second moment are fused to obtain a second fused image; Step S24: using a recognition model to recognize the first fusion image and the second fusion image to obtain a transparent object range.

5. The method for detecting transparent objects based on time series according to claim 1, characterized in that: The recognition model is a hierarchical multi-scale convolutional neural network.

6. The method for detecting transparent objects based on time series according to claim 1, characterized in that: When training, the recognition model also includes a self-supervised learning module; the common features output by the self-supervised module are injected into the recognition model, and the weights are jointly updated.

7. The method for detecting transparent objects based on time series according to claim 1, characterized in that: Also includes: Step S3: Correcting the depth value of the transparent object using the depth value of the non-transparent object that the transparent object range contacts.

8. A time-series based transparent object detection system, used to implement the time-series based transparent object detection method according to any one of claims 1 to 7, characterized in that: include: An acquisition module, configured to acquire a first image set and a second image set; wherein the first image set and the second image set both include a dense depth map, a sparse depth map, and an infrared map; and the first image set and the second image set are acquired at different times; A recognition module is used to input the first image set and the second image set into a recognition model, and identify a transparent object range according to the difference between the first image set and the second image set; wherein the transparent object range is the point corresponding to the dense depth map.

9. A time-series based transparent object detection device, characterized in that: include: processor; a memory storing executable instructions of the processor; Wherein, the processor is configured to perform the steps of the time-series-based transparent object detection method described in any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed, the steps of the time-series-based transparent object detection method described in any one of claims 1 to 7 are implemented.

Images