Garbage detection method and system based on multi-modal information fusion
By combining depth cameras with robust statistics and confidence assessment, the stability and accuracy issues of waste detection under all-weather conditions were resolved, achieving efficient and reliable waste identification and 3D localization under extreme lighting conditions, and outputting structured data to support intelligent decision-making.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- YUNCHUANG ZHIXING TECHNOLOGY (HUZHOU) CO LTD
- Filing Date
- 2025-12-05
- Publication Date
- 2026-04-10
AI Technical Summary
Existing waste detection technologies cannot reliably identify and locate waste in 3D under all-weather conditions. In particular, RGB camera-based solutions cannot meet all-weather requirements, and LiDAR solutions have too sparse point clouds, leading to missed detections. RGB and LiDAR fusion solutions have problems such as significant differences in data source resolution, difficulty in timestamp synchronization, and calibration errors.
The system employs a depth camera integrating an RGB imaging module and an active depth imaging module to acquire synchronized RGB and depth images in real time. By combining robust statistical methods and confidence assessment, the system calculates the three-dimensional spatial coordinates and confidence score of the waste, and outputs structured data records.
It achieves stable and reliable waste detection under extreme lighting conditions, and can identify low and flat waste while providing centimeter-level accurate 3D positioning. It avoids the positioning failure of existing fusion mechanisms, and the output information includes category, location and confidence level to facilitate intelligent decision-making.
Smart Images
Figure CN121837845A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of automatic driving and robots, and in particular, to a garbage detection method and system based on multi-modal information fusion. BACKGROUND
[0002] Under the background of rapid development of automatic driving and robot technology, automatic sanitation operation has become an important part of smart city construction. Autonomous sanitation vehicles need to patrol and locate various types of ground garbage in complex urban environments in real time and accurately, which is the premise and key to realizing autonomous cleaning operation.
[0003] In the prior art, the garbage detection method includes a two-dimensional RGB camera-based scheme, a LiDAR-based scheme, and a fusion scheme based on an RGB camera and a LiDAR. The two-dimensional RGB camera-based scheme uses a standard high-definition color camera to capture road images to identify garbage in the images. This scheme cannot meet the needs of all-weather operation and cannot obtain the target three-dimensional spatial position. The LiDAR-based scheme uses a LiDAR to scan the environment and analyzes the returned point cloud data to cluster and detect protruding objects on the ground. Although it can provide accurate three-dimensional coordinates, for low-beam LiDAR, the point cloud is too sparse, resulting in serious missed detection. The fusion scheme first identifies garbage by an RGB camera, and then finds point clusters corresponding to the identification results from LiDAR point cloud to obtain three-dimensional positions. It faces problems such as resolution disparity between data sources, time stamp synchronization difficulty, and calibration error. When the LiDAR fails to produce effective point cloud for the target garbage, the fusion mechanism completely fails, resulting in positioning failure.
[0004] Therefore, it is of great practical significance and application value to develop an efficient, robust, and all-weather available garbage detection system. SUMMARY
[0005] The present application provides a garbage detection method and system based on multi-modal information fusion to overcome at least one technical problem in the prior art.
[0006] In one aspect, the present application provides a garbage detection method based on multi-modal information fusion, comprising: acquiring scene information in real time by using a depth camera to obtain time and space synchronized and aligned RGB images and depth images; the depth camera integrates an RGB imaging module and an active depth imaging module; detecting the RGB images by using an identification model to output the class label and two-dimensional position information of each detected garbage; based on the two-dimensional position information, determining the corresponding depth sampling region on the depth image; extracting depth values of all pixels in the depth sampling region to form a depth value set; performing robust statistical processing on the depth value set to calculate a depth value representing the distance of the corresponding garbage object; calculating a confidence score based on the depth value set; calculating the three-dimensional spatial coordinates of the corresponding garbage based on the two-dimensional position information, the depth value, and the intrinsic matrix of the depth camera; integrating the category label, the three-dimensional spatial coordinates, and the confidence score of each garbage into a structured data record; outputting all garbage data records to form a target list.
[0007] Optionally, the two-dimensional position information is a two-dimensional bounding box or a pixel-level segmentation mask.
[0008] Optionally, based on the two-dimensional position information, the corresponding depth sampling region is determined on the depth image, specifically: According to the size of the depth image and the size of the RGB image, the proportional mapping relationship between the depth image coordinate system and the RGB image coordinate system is calculated. According to the proportional mapping relationship, the two-dimensional position information is mapped to the depth image coordinate system to obtain the depth sampling region.
[0009] Optionally, the robust statistical processing is specifically: calculating the median of the depth value set.
[0010] Optionally, based on the depth value set, the confidence score is calculated, specifically: calculating the proportion of the number of valid depth values in the depth value set to the total number of pixels in the corresponding depth sampling region as the coverage confidence; calculating the dispersion degree of the depth value set as the consistency confidence; According to the coverage confidence and the consistency confidence, the confidence score is calculated.
[0011] Optionally, the dispersion degree is determined by using the standard deviation.
[0012] Optionally, the three-dimensional spatial coordinates of the corresponding garbage are calculated by using the camera back projection formula.
[0013] On the other hand, the present application also provides a garbage detection system based on multi-modal information fusion, comprising: a depth camera for real-time acquisition of scene information to obtain time and space synchronized and aligned RGB images and depth images; the depth camera integrates an RGB imaging module and an active depth imaging module; A recognition model is used to detect the RGB image, and output a class label and two-dimensional position information of each detected garbage; A determination module is used to determine a corresponding depth sampling region on the depth image based on the two-dimensional position information; An extraction module is used to extract depth values of all pixels in the depth sampling region to form a depth value set; A processing module is used to perform robust statistical processing on the depth value set to calculate a depth value representing a distance of a corresponding garbage body; A first calculation module is used to calculate a confidence score based on the depth value set; A second calculation module is used to calculate three-dimensional space coordinates of the corresponding garbage based on the two-dimensional position information, the depth value, and an intrinsic matrix of the depth camera; An integration module is used to integrate the class label, the three-dimensional space coordinates, and the confidence score of each garbage into a structured data record; An output module is used to output data records of all garbage into a target list.
[0014] Optionally, the determination module is specifically used to: According to the size of the depth image and the size of the RGB image, a proportional mapping relationship between a depth image coordinate system and an RGB image coordinate system is calculated; According to the proportional mapping relationship, the two-dimensional position information is mapped to the depth image coordinate system to obtain the depth sampling region.
[0015] Optionally, the first calculation module is specifically used to: A proportion of a number of valid depth values in the depth value set to a total number of pixels in the corresponding depth sampling region is calculated as a coverage confidence; A dispersion degree of the depth value set is calculated as a consistency confidence; According to the coverage confidence and the consistency confidence, the confidence score is calculated.
[0016] The innovation points of the embodiments of the present application include: In the present embodiment, the depth camera based on the active light source fundamentally solves the dependence on the ambient light, and can stably and reliably work under extreme lighting conditions from complete darkness to outdoor strong light, which is one of the innovation points of the present application.
[0017] In this embodiment, by combining high-resolution RGB recognition with low-resolution depth positioning, both low and flat garbage can be detected like a pure vision scheme, and centimeter-level accurate three-dimensional positioning can be provided like a LiDAR scheme, achieving accurate positioning and avoiding the problem that the existing fusion mechanism completely fails when LiDAR fails to generate effective point cloud for the target garbage, resulting in positioning failure, which is one of the innovations of this embodiment.
[0018] In this embodiment, the output information not only includes the category and position, but also provides the positioning confidence, so that the downstream decision system can adopt a more intelligent strategy, which is one of the innovations of this embodiment.
[0019] In this embodiment, by using robust statistical methods and introducing confidence evaluation, the edge effect and noise interference caused by low depth map resolution are effectively solved, ensuring the stability and reliability of the three-dimensional positioning result, which is one of the innovations of this embodiment. BRIEF DESCRIPTION OF DRAWINGS
[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.
[0021] Figure 1 A flowchart of the method provided by the embodiment of the present application; Figure 2 A flowchart of calculating the confidence score provided by the embodiment of the present application; Figure 3 A structural schematic diagram of the system provided by the embodiment of the present application. DETAILED DESCRIPTION
[0022] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.
[0023] It should be noted that the terms "comprising" and "having" and any variations thereof in the embodiments of the present application and the accompanying drawings are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally further include steps or units not listed, or can optionally further include other steps or units inherent to the process, method, product or device.
[0024] The embodiments of the present application disclose a method and system of a garbage detection method based on multi-modal information fusion. The following are described in detail respectively.
[0025] Figure 1 A flowchart of the method provided by the embodiments of the present application is shown in Figure 1 The garbage detection method based on multi-modal information fusion provided by the embodiments of the present application comprises: Step 1: Real-time collection of scene information by using a depth camera to obtain time and space synchronized and aligned RGB images and depth images; the depth camera integrates an RGB imaging module and an active depth imaging module; Step 2: Detection of the RGB images by using a recognition model to output a class label and two-dimensional position information of each detected garbage; Step 3: Determination of a corresponding depth sampling area on the depth image based on the two-dimensional position information; Step 4: Extraction of depth values of all pixels in the depth sampling area to form a depth value set; Step 5: Robust statistical processing of the depth value set to calculate a depth value representing the distance of the corresponding garbage body; Step 6: Calculation of a confidence score based on the depth value set; Step 7: Calculation of three-dimensional space coordinates of the corresponding garbage based on the two-dimensional position information, the depth value and an intrinsic matrix of the depth camera; Step 8: Integration of the class label, the three-dimensional space coordinates and the confidence score of each garbage into a structured data record; Step 9: Formation of a target list by using all garbage data records for output.
[0026] Specifically, refer to Figure 1The garbage detection method based on multi-modal information fusion provided by the embodiment of the present application uses high-resolution two-dimensional visual information as the main guide for target discovery and classification, and uses low-resolution active depth information as auxiliary for accurate three-dimensional positioning, so that first in step 1, the scene information is collected in real time by using a depth camera integrating an RGB imaging module and an active depth imaging module. The active depth imaging module may be, for example, a DTOF (Direct TOF, direct time of flight) or an active binocular camera.
[0027] The RGB imaging module in the depth camera can collect high-resolution two-dimensional texture and color information, and the active depth imaging module can collect three-dimensional spatial distance information corresponding to each pixel. Real-time synchronous collection of the two can obtain high-resolution RGB images and low-resolution depth images that are synchronized and aligned in time and space, that is, the depth coordinates of each pixel correspond one-to-one to the color values, without the need for RGB and LiDAR fusion, solving the problems of resolution disparity (high-definition image vs. sparse point cloud), timestamp synchronization difficulty, and calibration error in the prior art when RGB and LiDAR are fused.
[0028] The depth camera based on an active light source fundamentally solves the dependence on ambient light and can stably and reliably work under extreme lighting conditions from complete darkness to outdoor strong light, far exceeding existing RGB camera-based solutions. In addition, through high-resolution RGB recognition combined with low-resolution depth positioning, low and flat garbage can be detected like a pure vision solution, and centimeter-level accurate three-dimensional positioning can be provided like a LiDAR solution, realizing visibility and accuracy, and avoiding the problem of complete failure of existing fusion mechanisms and positioning failure when the target garbage fails to generate effective point clouds.
[0029] After obtaining the pixel-level aligned RGB image and depth image, in step 2, the RGB image is input into a pre-trained recognition model (such as a YOLOv8 model), and the model is used to detect garbage in the RGB image and output the class label and two-dimensional position information of each identified garbage. The class label may be, for example, a bottle, a paper ball, etc. The two-dimensional position information may be a two-dimensional bounding box or a pixel-level segmentation mask. The two-dimensional bounding box is in the format of [x min , y min , x max , y max ] rectangular box, and the pixel-level segmentation mask is a binary matrix with the same size as the image, marking the accurate outline of the garbage pixels.
[0030] After obtaining the two-dimensional position information, depth correlation and reliability evaluation can be performed. First, coordinate mapping is performed in step 3 to proportionally map the two-dimensional position information from the coordinate system of the high-resolution RGB image to the coordinate system of the low-resolution depth image. It should be noted that the proportion here can be calculated according to the size of the depth image and the size of the RGB image. In this way, the corresponding depth sampling region can be determined on the depth image.
[0031] When the two-dimensional position information is a two-dimensional bounding box, the proportional mapping relationship between the depth image coordinate system and the RGB image coordinate system is calculated based on the two-dimensional bounding box. When the two-dimensional position information is a pixel-level segmentation mask, the proportional mapping relationship between the depth image coordinate system and the RGB image coordinate system is calculated based on the pixel-level segmentation mask. Each valid pixel in the mask is back-projected to 3D space to form a point cloud of the object, and then calculation is performed based on the point cloud.
[0032] In step 4, depth value extraction and filtering are performed. By extracting the depth values of all pixels in the depth sampling region, a set of depth values is formed.
[0033] To overcome the problem of depth value distortion caused by object edges and sensor noise, the present application does not use simple single-point sampling or mean calculation, but uses a robust statistical method in step 5 to extract the set of depth values, such as calculating the median of the set of depth values, to calculate a single and reliable depth value that represents the distance of the corresponding garbage object. It should be noted that in addition to calculating the median, other robust statistical methods such as trimmed mean (averaging after removing a certain proportion of the maximum and minimum values) can also be used.
[0034] In order to provide more abundant basis for the downstream intelligent decision system, the present application further calculates the confidence score through step 6. The output information not only includes the category and position, but also includes the position confidence, so that the downstream decision system can adopt a more intelligent strategy, such as directly executing cleaning for high-confidence targets and performing secondary confirmation for low-confidence targets.
[0035] Figure 2 A flowchart for calculating the confidence score provided by the embodiments of the present application is shown in Figure 2When calculating the execution score, first, in step 61, the coverage confidence is calculated by calculating the ratio of the number of valid depth values N_valid in the depth value set to the total number of pixels N_total in the corresponding depth sampling region, C1_coverage = N_valid / N_total. The coverage confidence measures how many proportions of the pixels in the sampling region mapped to the depth map are valid. The closer C1 is to 1, the better the depth information coverage, so that it can be prevented that only a few pixels have depth values due to the object being too small or located at the edge of the depth map, thereby causing unreliable positioning.
[0036] Then, in step 62, the consistency confidence is calculated by calculating the dispersion degree of the depth value set. The dispersion degree can be measured by the standard deviation or the median absolute deviation. Specifically, the absolute deviation between each depth value d i and the median m of the depth value set D is calculated to obtain the absolute deviation set A = { |d1 - m|, |d2 - m|,..., |d n - m|}; then the median MAD of the absolute deviation set A is calculated, and the consistency confidence C2_consistency = exp(-k * MAD) is calculated by the median MAD, where exp is the natural exponential function, and k is an adjustable hyperparameter (a positive number) for controlling the speed of attenuation.
[0037] After obtaining the coverage confidence and the consistency confidence, in step 63, the two sub-confidence scores are combined to calculate the confidence score. The combination can be multiplication, such as Confidence_Score = C1_coverage * C2_consistency. In this way, when the confidence of any dimension is low, the final total score will also be low. For example, the depth point coverage is very low (C1 ≈ 0), and even if these points are very concentrated (C2 ≈ 1), the result is unreliable; on the contrary, if the depth point coverage is very high (C1 ≈ 1), but the values of these points are very scattered (C2 ≈ 0), the result is also unreliable. Only when the coverage and consistency are both high, the final confidence score will be high.
[0038] After obtaining the two-dimensional position information and the depth value, in step 7, the three-dimensional space coordinates of the corresponding garbage are accurately calculated by combining the intrinsic matrix of the depth camera and the camera back projection formula. Camera back projection is a standard computer vision concept that can convert a 2D pixel point on the image and a known depth value to a 3D space coordinate system of the camera according to the camera internal parameters.
[0039] Input two-dimensional position information, depth value Z, camera intrinsic matrix K, and use the back projection formula X = (u - cx) *Z / fx, Y = (v - cy) * Z / fy, Z = Z, to calculate 3D coordinates (X, Y, Z). Wherein, the two-dimensional position information (u, v) of the center point of the bounding box is usually taken, the camera intrinsic matrix K is usually represented as a 3x3 matrix, in the form of [fx, 0, cx], [0, fy, cy], [0, 0, 1], wherein (fx, fy) is the focal length of the camera in the x and y directions (in pixels), (cx, cy) is the principal point of the camera, that is, the intersection coordinates of the camera optical axis and the imaging plane (usually close to the image center).
[0040] The application effectively solves the edge effect and noise interference caused by low resolution of the depth map by using a robust statistical method and introducing confidence evaluation, and ensures the stability and reliability of the three-dimensional positioning result.
[0041] After obtaining the three-dimensional space coordinates, in step 8, the category label of each garbage, the three-dimensional space coordinates and the confidence score are integrated into a structured data record, and all garbage data records are formed into a target list through step 9 for output, so that the detection result can be output in a structured format, facilitating the use of the downstream planning control system.
[0042] Based on the same inventive concept, the application also provides a garbage detection system based on multi-modal information fusion, Figure 3 A structural schematic diagram of the system provided by the embodiment of the application is shown in Figure 3 The application provides a garbage detection system 100 based on multi-modal information fusion, which comprises: A depth camera is used to collect scene information in real time, and obtain time and space synchronously aligned RGB images and depth images; the depth camera integrates an RGB imaging module and an active depth imaging module; An identification model is used to detect the RGB images, and output the category label and two-dimensional position information of each detected garbage; A determination module is used to determine the corresponding depth sampling region on the depth image based on the two-dimensional position information; An extraction module is used to extract the depth values of all pixels in the depth sampling region to form a depth value set; A processing module is used to perform robust statistical processing on the depth value set, and calculate a depth value representing the distance of the corresponding garbage body; A first calculation module is used to calculate a confidence score based on the depth value set; The second calculation module is configured to calculate the three-dimensional space coordinates of the corresponding garbage based on the two-dimensional position information, the depth value, and an intrinsic matrix of the depth camera. The integration module is configured to integrate the category label, the three-dimensional space coordinates, and the confidence score of each garbage into a structured data record. The output module is configured to output the data records of all the garbage to form a target list.
[0043] Specifically, referring to Figure 3 The garbage detection system 100 provided by the embodiment of the present application uses high-resolution two-dimensional visual information as the main guide for target discovery and classification, and uses low-resolution active depth information as auxiliary for accurate three-dimensional positioning. Therefore, first, a depth camera integrating an RGB imaging module and an active depth imaging module is used to collect scene information in real time. The active depth imaging module may be, for example, a DTOF (Direct TOF, direct measurement of time of flight) or an active binocular camera.
[0044] The RGB imaging module in the depth camera can collect high-resolution two-dimensional texture and color information, and the active depth imaging module can collect three-dimensional spatial distance information corresponding to each pixel. Real-time synchronous collection of the two can obtain a high-resolution RGB image and a low-resolution depth image that are time and space synchronized and aligned, that is, the depth coordinates of each pixel correspond one-to-one to the color value, without the need for RGB and LiDAR fusion, thus solving the problems of resolution disparity (high-definition image vs. sparse point cloud), timestamp synchronization difficulty, and calibration error in the prior art when RGB and LiDAR are fused.
[0045] The depth camera based on an active light source fundamentally solves the dependence on ambient light and can stably and reliably work under extreme lighting conditions from complete darkness to outdoor strong light, far exceeding existing RGB camera-based solutions. In addition, through high-resolution RGB recognition combined with low-depth positioning, both low and flat garbage can be detected like a pure visual solution, and centimeter-level accurate three-dimensional positioning can be provided like a LiDAR solution, achieving the ability to see and accurately locate, and avoiding the problem of complete failure of the existing fusion mechanism and resulting positioning failure when the target garbage fails to generate effective point cloud.
[0046] After obtaining the pixel-level aligned RGB image and depth image, the RGB image is input into a pre-trained recognition model (such as a YOLOv8 model), which is used to detect garbage in the RGB image and output the category label and two-dimensional position information of each recognized garbage. The category label may be, for example, a bottle or a paper ball, and the two-dimensional position information may be a two-dimensional bounding box or a pixel-level segmentation mask. The two-dimensional bounding box is [xmin , y min , x max , y max A pixel-level segmentation mask is a binary matrix with the same size as the image, marking the precise outline of the garbage pixels.
[0047] After obtaining the two-dimensional position information, depth correlation and reliability evaluation can be performed. First, the coordinate mapping is performed by using the determination module, so that the two-dimensional position information is proportionally mapped from the coordinate system of the high-resolution RGB image to the coordinate system of the low-resolution depth image. It should be noted that the proportion here can be calculated according to the size of the depth image and the size of the RGB image. In this way, the corresponding depth sampling region can be determined on the depth image.
[0048] When the two-dimensional position information is a two-dimensional bounding box, the proportional mapping relationship between the depth image coordinate system and the RGB image coordinate system is calculated based on the two-dimensional bounding box. When the two-dimensional position information is a pixel-level segmentation mask, the proportional mapping relationship between the depth image coordinate system and the RGB image coordinate system is calculated based on the pixel-level segmentation mask. Each valid pixel in the mask is back-projected to the 3D space to form a point cloud of the object, and then the point cloud is calculated.
[0049] The extraction module extracts and filters the depth values. By extracting the depth values of all pixels in the depth sampling region, a set of depth values is formed.
[0050] In order to overcome the distortion of depth values caused by the edge of the object and the noise of the sensor, the present application does not use simple single-point sampling or mean calculation, but uses a robust statistical method to process the extracted depth value set by the processing module, such as calculating the median of the depth value set, to calculate a single and reliable depth value that can represent the distance of the corresponding garbage object. It should be noted that in addition to calculating the median, other robust statistical methods such as trimmed mean (calculating the average after removing a certain proportion of the maximum and minimum values) can also be used.
[0051] In order to provide more abundant basis for the downstream intelligent decision system, the present application also calculates the confidence score by the first calculation module. The output information not only includes the category and position, but also includes the position confidence, so that the downstream decision system can adopt a more intelligent strategy, such as directly executing cleaning for high-confidence targets and performing secondary confirmation for low-confidence targets.
[0052] When calculating the execution score, first, the coverage confidence is calculated by calculating the ratio of the number of valid depth values N_valid in the depth value set to the total number of pixels N_total in the corresponding depth sampling region C1_coverage = N_valid / N_total. The coverage confidence measures how many proportions of pixels in the sampling region mapped to the depth map are valid. The closer C1 is to 1, the better the depth information coverage, so that it can be prevented that only a few pixels have depth values due to the small size of the object or the edge of the depth map, thereby generating unreliable positioning.
[0053] Then, the consistency confidence is calculated by calculating the dispersion degree of the depth value set. The dispersion degree can be measured by the standard deviation or the median absolute deviation. Specifically, the absolute deviation between each depth value d i and the median m of the depth value set D is calculated to obtain the absolute deviation set A = { |d1 - m|, |d2 - m|,..., |d n - m|}; then the median MAD of the absolute deviation set A is calculated MAD = median(|D - m|); and the consistency confidence C2_consistency = exp(-k * MAD) is calculated by the median MAD, where exp is the natural exponential function, and k is an adjustable hyperparameter (a positive number) for controlling the speed of attenuation.
[0054] After obtaining the coverage confidence and the consistency confidence, the two sub-confidence scores are combined to calculate the confidence score. The combination method can be multiplication, such as Confidence_Score = C1_coverage * C2_consistency, so that when the confidence of any dimension is low, the final total score will be low, for example, the depth point coverage is low (C1 ≈ 0), even if these points are very concentrated (C2 ≈ 1), the result is unreliable; on the contrary, if the depth point coverage is high (C1 ≈ 1), but the values of these points are very scattered (C2 ≈ 0), the result is also unreliable, only when the coverage and consistency are high at the same time, the final confidence score will be high.
[0055] After obtaining the two-dimensional position information and the depth value, the second calculation module combines the intrinsic matrix of the depth camera to accurately calculate the three-dimensional space coordinates of the corresponding garbage through the camera back projection (Back-Projection) formula. Camera back projection is a standard computer vision concept, which can convert a 2D pixel point on the image and a known depth value to a camera 3D space coordinate system according to the camera internal parameters.
[0056] Input two-dimensional position information, depth value Z, camera intrinsic matrix K, and use the back projection formula X = (u - cx) *Z / fx, Y = (v - cy) * Z / fy, Z = Z, to calculate 3D coordinates (X, Y, Z). Wherein, usually take the center point coordinate two-dimensional position information (u, v) of the bounding box, the camera intrinsic matrix K is usually represented as a 3x3 matrix, in the form of [fx, 0, cx], [0, fy, cy], [0, 0, 1], wherein (fx, fy) is the focal length (in pixels) of the camera in the x and y directions, (cx, cy) is the principal point (Principal Point) of the camera, that is, the intersection coordinates of the camera optical axis and the imaging plane (usually close to the image center).
[0057] The application effectively solves the edge effect and noise interference caused by low resolution of the depth map by using a robust statistical method and introducing confidence evaluation, and ensures the stability and reliability of the three-dimensional positioning result.
[0058] After obtaining the three-dimensional space coordinates, the integration module integrates the class label of each garbage, the three-dimensional space coordinates and the confidence score into a structured data record, and outputs all garbage data records in the form of a target list through the output module, so that the detection result can be output in a structured format, facilitating the use of the downstream planning control system.
[0059] Those skilled in the art can understand that the drawings are only schematic diagrams of an embodiment, and the modules or processes in the drawings are not necessarily necessary for implementing the application.
[0060] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed and located in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or can be further split into multiple sub-modules.
[0061] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application.
Claims
1.A method for garbage detection based on multi-modal information fusion, characterized in that, The method comprises the following steps: Real-time acquisition of scene information by using a depth camera to obtain time and space synchronously aligned RGB images and depth images; The depth camera integrates an RGB imaging module and an active depth imaging module; Using a recognition model to detect the RGB images and output the class label and two-dimensional position information of each detected garbage; Based on the two-dimensional position information, determining the corresponding depth sampling area on the depth image; Extracting the depth values of all pixels in the depth sampling area to form a depth value set; Robust statistical processing of the depth value set to calculate a depth value representing the distance of the corresponding garbage; Based on the depth value set, calculating a confidence score; Based on the two-dimensional position information, the depth value, and the intrinsic matrix of the depth camera, calculating the three-dimensional spatial coordinates of the corresponding garbage; Integrating the class label, three-dimensional spatial coordinates, and confidence score of each garbage into a structured data record; Outputting the data records of all garbage to form a target list. 2.The multi-modal information fusion based garbage detection method of claim 1, wherein, The two-dimensional position information is a two-dimensional bounding box or a pixel-level segmentation mask. 3.The multi-modal information fusion based garbage detection method of claim 1, wherein, Based on the two-dimensional position information, determining the corresponding depth sampling area on the depth image, specifically: According to the size of the depth image and the size of the RGB image, calculating the proportional mapping relationship between the depth image coordinate system and the RGB image coordinate system; According to the proportional mapping relationship, mapping the two-dimensional position information to the depth image coordinate system to obtain the depth sampling area. 4.The multi-modal information fusion based garbage detection method of claim 1, wherein, The robust statistical processing specifically includes: Calculating the median of the depth value set. 5.The multi-modal information fusion based garbage detection method according to claim 1, characterized in that, Based on the depth value set, calculating a confidence score, specifically: Calculating the proportion of the number of valid depth values in the depth value set to the total number of pixels in the corresponding depth sampling area as the coverage confidence; Calculating the dispersion degree of the depth value set as the consistency confidence; According to the coverage confidence and the consistency confidence, calculating the confidence score. 6.The multi-modal information fusion based garbage detection method according to claim 1, characterized in that, The dispersion degree is determined by using standard deviation. 7.The multi-modal information fusion based garbage detection method according to claim 1, characterized in that, The three-dimensional spatial coordinates of the corresponding garbage are calculated using the camera back projection formula. 8.A garbage detection system based on multi-modal information fusion, characterized in that, The method comprises the following steps: A depth camera is used to real-time acquisition of scene information to obtain time and space synchronously aligned RGB images and depth images; The depth camera integrates an RGB imaging module and an active depth imaging module; A recognition model is used to detect the RGB images and output the class label and two-dimensional position information of each detected garbage; A determination module is used to determine the corresponding depth sampling area on the depth image based on the two-dimensional position information; An extraction module is used to extract the depth values of all pixels in the depth sampling area to form a depth value set; A processing module is used to robust statistical processing of the depth value set to calculate a depth value representing the distance of the corresponding garbage; A first calculation module is used to calculate a confidence score based on the depth value set; A second calculation module is used to calculate the three-dimensional spatial coordinates of the corresponding garbage based on the two-dimensional position information, the depth value, and the intrinsic matrix of the depth camera; An integration module is configured to integrate the category label, the three-dimensional spatial coordinate and the confidence score of each piece of garbage into a structured data record; An output module is configured to output all the data records of the garbage into a target list. 9.The multi-modal information fusion based garbage detection system of claim 8, wherein, The determination module is specifically configured to: According to the size of the depth image and the size of the RGB image, calculate a proportional mapping relationship between a depth image coordinate system and an RGB image coordinate system; According to the proportional mapping relationship, map the two-dimensional position information to the depth image coordinate system to obtain a depth sampling region. 10.The multi-modal information fusion based garbage detection system according to claim 1, wherein, The first calculation module is specifically configured to: Calculate a proportion of the number of valid depth values in the depth value set to the total number of pixels in the corresponding depth sampling region as a coverage confidence; Calculate a dispersion degree of the depth value set as a consistency confidence; According to the coverage confidence and the consistency confidence, calculate the confidence score.