A monitoring method and monitoring system based on a medical waste intelligent collection vehicle

Abstract

The application discloses a kind of based on medical waste intelligent collection car's monitoring method and monitoring system, it is related to medical waste collection technical field, including the following steps: for by multiple sensors being set on collection car acquisition medical waste stacking area Multimodal perception data;For the multimodal perception data of acquisition is handled in space-time registration, establishes the coordinate conversion relationship between different sensor data;For based on the multimodal perception data after registration, generate comprehensive confidence chart;For using neural network to repair the missing area in image, the training process of neural network is optimized using multi-objective loss function;For based on the image data and three-dimensional point cloud data after repair, construct and update environment three-dimensional representation, based on multimodal feature carries out waste container identification and state monitoring;The application ensures that clear medical waste container image information can be acquired under different depth of field conditions.

Description

Technical Field

[0001] This invention relates to the field of medical waste collection technology, specifically to a monitoring method and system based on an intelligent medical waste collection vehicle. Background Technology

[0002] Medical waste generally refers to waste generated by medical and health institutions during medical treatment, prevention, health care and other related activities that has direct or indirect infectiousness, toxicity and other hazards. Therefore, it is usually necessary to collect and dispose of this medical waste in isolation. At present, the main method is to use color-coded collection bins to collect this waste and then transport it to landfills.

[0003] However, in complex medical waste storage areas, waste is densely packed and often obstructed. In this environment, the onboard cameras of intelligent collection vehicles have difficulty effectively capturing the status of all waste containers, especially when medical waste packaging and stacking are dense. Image information is prone to overlap, obstruction or loss, thus affecting the accuracy of waste sorting. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a monitoring method and system based on an intelligent medical waste collection vehicle.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] This invention provides a monitoring method based on an intelligent medical waste collection vehicle, comprising the following steps:

[0007] The data acquisition step is used to collect multimodal sensing data of the medical waste stacking area through multiple sensors installed on the collection vehicle. The multimodal sensing data includes image data, temperature distribution data and three-dimensional point cloud data. The image data acquisition process is based on the depth distribution characteristics determined by the three-dimensional point cloud data to adjust the focal length.

[0008] The data registration step is used to perform spatiotemporal registration processing on the collected multimodal sensing data, establish coordinate transformation relationships between data from different sensors, and perform correction processing on the data from each sensor.

[0009] The confidence fusion step is used to calculate confidence indices reflecting the reliability of each data source based on the registered multimodal sensing data, and to fuse multiple confidence indices to generate a comprehensive confidence map.

[0010] The image restoration step is used to repair the missing areas in the image using a neural network. The training process of the neural network is optimized using a multi-objective loss function, which includes a restoration error term weighted based on the comprehensive confidence map.

[0011] The state analysis step is used to construct and update a 3D representation of the environment based on the repaired image data and 3D point cloud data, and to identify and monitor the state of waste containers based on multimodal features.

[0012] The present invention also provides a monitoring system based on a medical waste intelligent collection vehicle, the monitoring system comprising:

[0013] The multispectral imaging module is used to acquire visible light, near-infrared, and short-wave infrared images of the medical waste storage area.

[0014] The thermal imaging module is used to acquire infrared temperature distribution images of the area.

[0015] The depth perception module is used to collect three-dimensional point cloud data of the area.

[0016] The computational control unit is used to control the multispectral imaging module to adjust the focal length based on the depth distribution characteristics determined by the three-dimensional point cloud data, and to perform spatiotemporal registration processing on the acquired multimodal sensing data to establish coordinate transformation relationships between different sensor data.

[0017] The computational control unit is also used for: calculating confidence indices reflecting the reliability of each data source based on the registered multimodal sensing data; fusing multiple confidence indices to generate a comprehensive confidence map; using a neural network to repair the missing areas in the image, wherein the training process of the neural network is optimized using a multi-objective loss function that includes a repair error term weighted based on the comprehensive confidence map; constructing and updating a three-dimensional representation of the environment based on the repaired image data and three-dimensional point cloud data; and identifying and monitoring the status of waste containers based on multimodal features.

[0018] The beneficial effects of this invention are:

[0019] This invention effectively solves the monitoring challenges caused by dense occlusion, varying depths, and complex material reflection characteristics in medical waste storage areas through the collaborative acquisition and processing of multimodal sensing data. The data acquisition process utilizes a mechanism for dynamically adjusting the focal length based on 3D point cloud data, combined with the synchronous acquisition of multispectral image sequences and thermal imaging image data. This ensures that clear images of medical waste containers can be obtained under different depth-of-field conditions, reducing information loss caused by occlusion and unclear focus from the data source. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0021] Figure 1 This is a schematic diagram illustrating the workflow of the monitoring method based on an intelligent medical waste collection vehicle according to the present invention.

[0022] Figure 2 This is a schematic diagram of the overall structure of the monitoring system based on the intelligent medical waste collection vehicle of the present invention. Detailed Implementation

[0023] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0024] like Figure 1 As shown, a monitoring method based on a smart medical waste collection vehicle includes the following steps:

[0025] The data acquisition step involves collecting multimodal sensing data of the medical waste storage area using multiple sensors mounted on the collection vehicle. The multimodal sensing data includes image data, temperature distribution data, and three-dimensional point cloud data. The image data acquisition process involves adjusting the focal length based on the depth distribution characteristics determined by the three-dimensional point cloud data. The focal length adjustment refers to automatically adjusting the focal length parameters of the multispectral imaging module according to the depth distribution characteristics of the medical waste storage area to ensure that clear images can be acquired under different depth conditions.

[0026] After the intelligent medical waste collection vehicle reaches the waste dumping area, the multispectral imaging module, thermal imaging module, and depth sensing module installed on the collection vehicle are activated to work together. The multispectral imaging module includes a high dynamic range camera in three bands: visible light (400-700nm), near-infrared (700-1100nm), and short-wave infrared (1100-2500nm), used to collect image data of the medical waste dumping area. The thermal imaging module is a long-wave infrared thermal imaging camera with a wavelength range of 8-14μm, used to collect temperature distribution data of the medical waste dumping area. The depth sensing module is a 32-line rotating lidar scanner used to collect three-dimensional point cloud data of the medical waste dumping area. The operating wavelength is 905nm, the ranging range is 0.1-200m, the horizontal field of view is 360°, the vertical field of view is -15° to +15°, the angular resolution is 0.1°-0.4° horizontally and 0.4° vertically, the point cloud output frequency is 10Hz, and the ranging accuracy is ±2cm. The depth perception module is connected to the computing control unit via an Ethernet interface and transmits point cloud data using the TCP / IP protocol.

[0027] The data registration step is used to perform spatiotemporal registration processing on the collected multimodal sensing data, establish coordinate transformation relationships between data from different sensors, and perform correction processing on the data from each sensor.

[0028] After completing the multimodal sensing data acquisition, the system performs a data registration step, which involves spatiotemporal registration processing of the acquired multimodal sensing data. Specifically, the spatiotemporal registration process includes: establishing a unified coordinate system, including a world coordinate system based on the global positioning of the collection vehicle, a camera coordinate system with the vehicle-mounted sensor installation location as the origin, and an image coordinate system based on pixel grids. Through a coordinate transformation matrix, the system determines the transformation relationship between different coordinate systems, enabling all sensor data to be analyzed within the same spatial framework. Specifically, a feature matching method based on ORB feature descriptors is used, combined with the unique color and geometric features of medical waste containers, to accurately register point cloud data with multispectral images, achieving a registration error of less than 0.05 meters.

[0029] The confidence fusion step is used to calculate confidence indices reflecting the reliability of each data source based on the registered multimodal sensing data, and to fuse multiple confidence indices to generate a comprehensive confidence map.

[0030] The image restoration step is used to repair the missing areas in the image using a neural network, especially for the image loss problem caused by mutual occlusion or sensor field of view limitation in medical waste stacking areas. The training process of the neural network is optimized using a multi-objective loss function, which includes a restoration error term weighted based on the comprehensive confidence map.

[0031] The neural network is a convolutional neural network based on the U-Net structure. Its training process adopts multi-objective loss function optimization. The loss function includes reconstruction loss term, depth consistency loss term, confidence weighted loss term and edge preservation loss term. The training data comes from labeled images of actual medical waste dumping scenarios. The training iteration count is 10,000 times. The optimizer used is Adam.

[0032] The state analysis step is used to construct and update a 3D representation of the environment based on the repaired image data and 3D point cloud data, and to identify and monitor the state of waste containers based on multimodal features.

[0033] Through the coordinated operation of the above five steps, this invention can effectively solve the problem of image acquisition and analysis in complex environments for intelligent medical waste collection vehicles, achieve accurate identification and status monitoring of medical waste containers, and provide reliable technical support for the safe collection and treatment of medical waste.

[0034] Furthermore, the dynamic focal length adjustment is achieved through the following steps:

[0035] The initial depth map data of the medical waste storage area is obtained through the depth sensing module. The depth statistical feature parameters of the medical waste storage area in the initial depth map are calculated. Based on the depth statistical feature parameters, the focal length sequence is calculated through the depth-focal length mapping model. The gimbal system is controlled to move according to the preset motion trajectory. Multispectral image sequence and thermal imaging image data are collected at each focal length point in the focal length sequence.

[0036] The workflow of the depth-focal length mapping model in practical applications is as follows: First, the system calculates the median depth of the initial depth map and determines that the center value of the focal length sequence is located near the median depth. Then, the number of focal points is determined based on the magnitude of the depth variance: 3 focal points are selected when the depth variance is less than 0.08 square meters, 4 focal points are selected when the depth variance is between 0.08 and 0.15 square meters, and 5 focal points are selected when the depth variance is greater than 0.15 square meters. Next, the distribution of focal points is determined based on the shape of the depth distribution histogram. If the depth distribution exhibits a single-peak shape, the focal points are evenly distributed on both sides of the center value; if the depth distribution exhibits a bimodal or multimodal shape, additional focal points are added near each peak. Finally, the system calculates the specific focal length sequence values ​​based on the determined number and distribution shape of the focal points, ensuring coverage of all possible medical waste container depth ranges and higher focal point density in key depth areas.

[0037] Specifically, when the intelligent medical waste collection vehicle travels to a position 1.5 meters in front of the waste dumping area, the depth perception module activates the lidar to scan the area at a frequency of 10 Hz, with a horizontal field of view of 360 degrees and a vertical field of view of ±15 degrees. The lidar emits a laser beam and receives the reflected signal, calculates the distance to each point using the time-of-flight method, and generates initial depth map data containing approximately 32,000 points. This initial depth map data covers the medical waste dumping area within a range of 0.5 meters to 3 meters in front of the collection vehicle, with a resolution of 640×480 pixels and a depth measurement accuracy of ±2 centimeters.

[0038] Next, the system calculates the depth statistical feature parameters of the medical waste dumping area in the initial depth map. First, the system preprocesses the initial depth map, including removing outliers and filling in missing values. Then, for the medical waste dumping area (the region of interest determined through coarse segmentation), the system calculates the following depth statistical feature parameters:

[0039] Median Depth: This indicator reflects the central tendency of the main body of the waste pile. It is calculated by taking the median value after arranging all depth values ​​in ascending order. In typical medical waste disposal scenarios, the median depth is usually between 1.2 meters and 1.8 meters, corresponding to the average distance from the main waste container in front of the collection vehicle.

[0040] Depth variance: a dispersion index characterizing the degree of irregularity in stacking. It is calculated as the average of the squares of the differences between each depth value and the mean depth. The magnitude of the depth variance directly reflects the drastic change in the height of the waste pile. When the stacking height of medical waste containers varies greatly, the depth variance value is high, usually in the range of 0.05 to 0.2 square meters.

[0041] Depth distribution histogram: quantifies the distribution characteristics of different height regions. The depth range is divided into 10 equal-width intervals, and the proportion of points in each interval is counted. The depth distribution histogram can intuitively show the hierarchical structure of the waste pile, such as the distribution of the bottom layer container (0.5-1.0 meters), the middle layer container (1.0-2.0 meters), and the top layer container (2.0-3.0 meters).

[0042] These depth statistical feature parameters collectively describe the three-dimensional spatial distribution characteristics of the medical waste dumping area, providing a key basis for subsequent focal length sequence calculations.

[0043] Then, based on the depth statistical feature parameters, the focal length sequence is calculated through the depth-focal length mapping model. The depth-focal length mapping model is a nonlinear function that is pre-calibrated through experiments and maps the depth statistical feature parameters to the optimal focal length value.

[0044] In practical applications, the system determines the center value of the focal length sequence based on the median depth, the interval of the focal length sequence based on the depth variance, and the distribution pattern of the focal length sequence based on the depth distribution histogram. For example, when the median depth is 1.5 meters and the depth variance is 0.1 square meters, the focal length sequence calculated by the system may be {0.9m, 1.3m, 1.7m, 2.1m}, with a total of 4 focal length points, covering the focal length range from 0.8 meters to 2.5 meters, ensuring that it can adapt to medical waste containers of different heights. The number of focal length points n is dynamically determined based on the depth variance, usually between 3 and 5. The larger the depth variance, the more focal length points there are.

[0045] Finally, the gimbal system is controlled to move along a preset motion trajectory, acquiring multispectral image sequences and thermal imaging data at each focal point in the focal length sequence. The gimbal system is a six-degree-of-freedom precision motion platform with a positioning accuracy of ±0.1 degrees. The system adjusts the lens focal length of the multispectral imaging module according to each focal length value in the focal length sequence and controls the gimbal system to move along the preset motion trajectory. The preset motion trajectory includes a combination of horizontal rotation ±30 degrees and vertical pitch ±15 degrees to ensure a comprehensive scan of the medical waste storage area from multiple angles.

[0046] Through the aforementioned dynamic focal length adjustment mechanism, the intelligent medical waste collection vehicle can acquire multiple sets of image data at different focal lengths during a single scan, effectively solving the problem of drastic changes in depth of field caused by differences in the height of medical waste stacking.

[0047] Furthermore, the multiple confidence levels include spectral confidence, temperature confidence, and depth confidence;

[0048] The calculation process of the spectral confidence score includes obtaining the spectral intensity vector of the pixel in the visible light, near infrared and short-wave infrared bands, calculating the Euclidean distance between the vector and the mean spectral vector of typical medical waste materials, and calculating the spectral confidence score value by combining the spectral variance vector and the specular reflectance factor parameter.

[0049] Specifically, the calculation process is based on the following formula:

[0050] ;

[0051] Wherein, S(p) is the spectral intensity vector of pixel p in the visible light band, near-infrared band and short-wave infrared band, representing the reflectivity or radiation intensity of the pixel in the three bands. In the medical waste monitoring scenario, medical waste containers of different materials (such as plastic sharps boxes, metal needle boxes, glass medicine bottles, etc.) have unique spectral characteristics, and these characteristics exhibit different response modes in different bands.

[0052] μ s This is the spectral mean vector of typical medical waste materials, which is obtained by performing spectral measurements on various typical medical waste containers (including sharps containers, infectious waste bins, drug packaging, etc.) under laboratory conditions.

[0053] σ s 2 The spectral variance vector represents the range of spectral fluctuations of the same type of medical waste material under different lighting conditions and observation angles.

[0054] ||S(p)-μ s ||² represents the squared Euclidean distance between the pixel spectral intensity vector and the mean spectral vector of typical medical waste material, used to quantify the degree of difference between the current pixel spectral features and typical medical waste material.

[0055] The process of calculating temperature confidence includes obtaining the temperature measurement value of the pixel, calculating the sigmoid function transformation value of the temperature value and the medical waste temperature anomaly threshold, and combining the temperature field gradient magnitude parameter to calculate the temperature confidence value.

[0056] Specifically, the calculation process is based on the following formula:

[0057] ;

[0058] Where T(p) is the temperature value of pixel p in degrees Celsius, which is obtained by measuring through the thermal imaging module. In the medical waste monitoring scenario, infectious medical waste may produce temperature anomalies due to microbial activity, which are usually higher than the ambient temperature. The system obtains the temperature distribution of the medical waste stacking area through the thermal imaging module and assigns a temperature value to each pixel.

[0059] T0=37.5℃ is the abnormal temperature threshold for medical waste. This threshold is determined based on medical research and actual measurement data.

[0060] τ is a parameter representing the width of the temperature transition range. This parameter controls the smoothness of the transition from low to high temperature confidence. The choice of τ value is based on the uncertainty of temperature measurement and the requirements of actual application scenarios. The temperature measurement uncertainty of the thermal imaging module is approximately 0.5℃. Considering ambient temperature fluctuations and measurement errors, setting τ=2.5 (approximately 5 times the measurement uncertainty) can ensure sensitivity while avoiding noise interference. When the temperature value is far from the threshold, the confidence level rapidly approaches 0 or 1; when the temperature value is close to the threshold, the confidence level transitions smoothly, avoiding drastic changes in confidence level due to small temperature fluctuations.

[0061] (T(p)-T0) / τ represents the standardized temperature difference, which converts the difference between the actual temperature and the threshold into a dimensionless relative difference. The standardized temperature difference is negative, then exponentially calculated and incremented by 1, and finally the reciprocal is taken. The result is a sigmoid function transformation value. The sigmoid function is chosen based on its S-curve characteristics, which can map the temperature value to the [0,1] interval and has a smooth transition characteristic near the threshold. When T(p) < T0, the temperature confidence is close to 0, indicating that the temperature in this area is normal; when T(p) > T0, the temperature confidence is close to 1, indicating that there may be a temperature anomaly in this area.

[0062] The process of calculating depth confidence includes obtaining the depth value measured by lidar and the depth value predicted based on multi-view geometry, calculating the absolute difference between the two depth values, and combining the depth measurement uncertainty parameter and the point cloud curvature estimate to calculate the depth confidence value.

[0063] Specifically, the calculation process is based on the following formula:

[0064] ;

[0065] Where D(p) is the depth value measured by lidar, in meters, representing the distance from the collection vehicle to the surface of the medical waste.

[0066] D pred(p) is the depth value based on multi-view geometric prediction, in meters, which is calculated by feature matching and triangulation of adjacent frame images. Specifically, the system uses the camera pose changes and feature point correspondence between consecutive frames to calculate the predicted depth of the current pixel in the current frame through epipolar constraints and triangulation principles.

[0067] |D(p)-D pred (p)| represents the absolute difference between the depth value measured by lidar and the depth value predicted based on multi-view geometry, reflecting the degree of consistency between the two depth acquisition methods. Ideally, the depth values ​​obtained by the two methods should be consistent; however, when there are measurement errors or dynamic changes in the scene, the two will differ.

[0068] σ d The depth measurement uncertainty is determined based on the technical specifications of the depth sensing module and actual test results. The nominal ranging accuracy of the lidar is ±2 cm. However, in actual medical waste monitoring environments, the uncertainty of depth measurement increases due to the diverse surface materials (such as plastic, metal, and fabric) and surface characteristics (such as transparency, reflectivity, and light absorption) of medical waste. Through depth measurement experiments on different surface materials, the average measurement uncertainty was statistically obtained to be 4.7 cm. Considering a safety margin, the depth measurement uncertainty σ is... d Set to 5 cm (0.05 m), this parameter is used to standardize depth differences, making the confidence calculation adaptable to depth differences at different scales.

[0069] The formula for calculating depth confidence represents the probability distribution of depth consistency, using an exponential decay form. This confidence level is achieved when the measured depth and predicted depth are perfectly consistent (i.e., |D(p) - D)). pred (p)|=0), the depth confidence reaches its maximum value of 1; as the difference increases, the confidence decreases exponentially, which can effectively reflect the reliability of depth measurement.

[0070] Finally, the fusion confidence level C f (p) is represented as:

[0071] ;

[0072] Where, ω s The spectral confidence weighting coefficient is determined primarily based on the core requirements of medical waste classification and management. Accurately identifying the type of waste material (such as infectious waste, sharps waste, and pharmaceutical waste) is the most basic and crucial requirement during medical waste treatment. Spectral information provides the most direct basis for material identification, especially the short-wave infrared band, which has unique advantages in identifying materials such as plastics and organic matter. Through the analysis of several medical waste classification cases, the preferred spectral confidence weighting coefficient ω is... sSet to 0.5;

[0073] ω t The temperature confidence weighting coefficient is determined based on the importance of biosafety risk assessment. Infectious medical waste may carry pathogenic microorganisms, posing a threat to the environment and human health. Therefore, identifying areas with abnormal temperatures has a high priority. However, temperature anomalies can also be caused by environmental factors, and their specificity is not as good as spectral information. Preferably, the temperature confidence weighting coefficient ω... t Set to 0.3;

[0074] ω d The depth confidence weighting coefficient is determined based on the auxiliary role of 3D structural information. Depth information is mainly used to determine the spatial location and shape of medical waste, and its direct contribution to material identification and risk assessment is relatively small. However, it plays an important role in processing occluded and repairing images. Preferably, the depth confidence weighting coefficient ω... d Set it to 0.2.

[0075] This confidence fusion mechanism can effectively solve common problems such as occlusion and noise interference in medical waste storage areas, and provide reliable confidence guidance for subsequent image restoration and status analysis.

[0076] Furthermore, the process of constructing the loss function of the neural network includes the following steps:

[0077] A reconstruction loss term is constructed to calculate the mean square error between the image restored in the occluded region and the ideal image, and the mean square error between the image restored in the non-occluded region and the input image.

[0078] The calculation formula is as follows:

[0079]

[0080] ;

[0081] Where N represents the total number of pixels, p represents the pixel position index, and M(p)∈{0,1} is the binary occlusion mask. The determination of M(p) is based on the depth map obtained by the depth perception module and multi-view geometric analysis. Specifically, the system first obtains the depth map of the medical waste stacking area through the depth perception module, then analyzes the depth discontinuity region and the visibility occlusion relationship to generate the binary occlusion mask M(p). When pixel p is located in an area occluded by other medical waste containers (such as the gap between stacked containers, or the part of the bottom container covered by the upper container), M(p)=1; when pixel p is located in the visible area, M(p)=0.

[0082] I repair (p) represents the pixel value of the restored image, I ideal(p) represents the ideal image pixel value, I input (p) represents the pixel value of the input damaged image;

[0083] The reconstruction loss term uses mean squared error instead of absolute error as the difference measure because mean squared error is more sensitive to large errors and can effectively avoid obvious flaws or discontinuities in the restoration results. In medical waste image restoration, even small obvious flaws may lead to incorrect identification of container edges, which in turn affects the waste classification results. Therefore, a higher penalty needs to be imposed on large errors.

[0084] A depth consistency loss term is constructed to calculate the mean square error between the estimated depth map and the measured depth map and the difference between the depth gradient field;

[0085] The calculation formula is as follows:

[0086]

[0087] ;

[0088] Among them, D est (p) represents the depth map estimated from the restored image. The system uses a pre-trained monocular depth estimation network (such as MiDaS or LeRes) to estimate the depth map from the restored RGB image. D(p) represents the measured depth map, which is directly acquired by the depth sensing module (32-line rotating LiDAR). ∇D est (p) represents the gradient field of the estimated depth map, ∇D(p) represents the gradient field of the measured depth map, and α represents the gradient weight coefficient. When α=0.5, the depth consistency loss term achieves the best balance between maintaining overall depth consistency and preserving edge features.

[0089] The depth consistency loss term is designed based on the importance of the three-dimensional structure of medical waste containers for their identification and status monitoring. In the scenario of medical waste stacking, the geometry of the container (such as the rectangle of a sharps box and the cylinder of an infectious waste bin) is a key feature for identifying the container type, while the depth distribution of the container reflects the filling status and stacking stability. If the repair result is inconsistent with the measured data in terms of depth, it will lead to serious errors in subsequent container identification and status analysis. For example, if the depth distribution of the repaired sharps box is abnormal, it may be misidentified as other types of containers or its filling status may be incorrectly judged.

[0090] A confidence-weighted loss term is constructed to calculate the repair error of different regions based on the fused confidence map.

[0091] The calculation formula is as follows:

[0092] ;

[0093] In the medical waste monitoring scenario, there are significant differences in confidence levels across different regions:

[0094] Planar regions of containers typically have high spectral confidence (due to distinct material characteristics) and high depth confidence (due to flat surfaces), thus resulting in high fusion confidence.

[0095] Container edge regions typically have lower depth confidence (due to high point cloud curvature) but may have higher spectral confidence (if the edges are clearly visible).

[0096] High reflectivity areas (such as plastic surfaces contaminated with liquid) typically have lower spectral confidence (because specular reflection causes spectral distortion).

[0097] Areas with abnormal temperatures (which may indicate biological hazards) typically have a high temperature confidence level;

[0098] The confidence-weighted loss term is designed based on the varying importance of different areas in medical waste monitoring scenarios. In medical waste classification and status monitoring, certain areas (such as container label areas and temperature anomaly areas) are more important than others and require higher accuracy in restoration. For example, if the biohazard marking area on a sharps container is incorrectly restored, it may lead to misclassification; if the temperature anomaly area on an infectious waste bin is not accurately restored, it may result in missed biohazard risks. Through confidence weighting, the system can ensure that these critical areas receive priority and accurate restoration.

[0099] An edge-preserving loss term is constructed to calculate the difference between the gradient field of the restored image and the gradient field of the ideal image;

[0100] The calculation formula is as follows:

[0101]

[0102] ;

[0103] Among them, ∇I ideal (p) represents the gradient field of the ideal image, ∇I repair (p) represents the gradient field of the restored image. The gradient field reflects the direction and intensity of the image brightness change and is crucial for maintaining the edge features of medical waste containers. In medical waste monitoring scenarios, container edges (such as the corners of sharps boxes and the opening edges of infectious waste bins) usually exhibit abrupt changes in image gradients. Traditional image restoration methods often lead to blurred edges, especially in the restoration of occluded areas, where edge continuity is difficult to maintain. This invention ensures that the restoration result maintains appropriate sharpness in edge areas and consistency in smooth areas by using an edge preservation loss term, thus avoiding artifacts caused by over-sharpening.

[0104] The four loss terms are combined according to their weighting coefficients to form the final loss function:

[0105]

[0106] ;

[0107] Where, λ recon , λ depth , λ conf and λ edge These are the weight coefficients for the reconstruction loss term, the depth consistency loss term, the confidence-weighted loss term, and the edge-preserving loss term, respectively.

[0108] Preferably, λ recon =0.6, the reconstruction loss term has the highest weight, ensuring pixel-level accuracy of the restored image. In medical waste classification, pixel-level accuracy is crucial for material identification, especially for distinguishing different types of plastic containers (such as sharps containers and medicine packaging boxes), therefore it is given the highest weight;

[0109] λ depth =0.2, with the depth consistency loss term having the next highest weight, ensuring that the repaired result matches the actual 3D structure. Depth information is crucial for resolving occlusion issues and determining the spatial location of containers, but its accuracy is slightly lower than pixel-level accuracy, hence the weight is set to 0.2;

[0110] λ conf =0.15, the confidence-weighted loss term has a moderate weight, emphasizing the repair quality of high-confidence areas. This loss term plays an important role in improving the repair quality of key areas (such as temperature anomaly areas and container label areas), but its effect depends on the accuracy of confidence calculation, so the weight is set to 0.15;

[0111] λ edge =0.05, the edge preservation loss term has a low weight, but it is crucial for key edge features. Edge features play a decisive role in container recognition, but since the first three loss terms already partially include edge information, the weight is set to 0.05.

[0112] In specific medical waste monitoring scenarios, this loss function exhibits the following advantages:

[0113] For the bottom sharps container that is obscured by the upper container, the system can accurately repair its surface labels and biohazard symbols, ensuring the accuracy of waste sorting. This is because the reconstruction loss item and the confidence-weighted loss item work together to prioritize the repair quality of critical areas.

[0114] For partially occluded infectious waste bins, the system maintains the integrity of their cylindrical outline, avoiding shape distortion caused by repair. This is because the depth consistency loss term and the edge preservation loss term work together to ensure that the repair result conforms to the actual 3D structure and edge features;

[0115] For highly reflective plastic container surfaces contaminated with chemicals, the system avoids over-repairing that could lead to material distortion, maintaining the realism of the plastic material. This is because the confidence-weighted loss term reduces the repair weight for highly reflective areas, causing the network to adopt a more conservative repair strategy for these areas.

[0116] For areas with abnormal temperatures (which may indicate biological hazards), the system prioritizes the quality of repair in these areas, ensuring that the abnormal temperature characteristics are not erased during the repair process. This is because areas with higher temperature confidence receive a higher weight in the confidence-weighted loss term.

[0117] Furthermore, the process of updating the 3D representation of the environment includes the following steps:

[0118] The three-dimensional space is divided into a voxel grid, and each voxel stores the truncation symbol distance function value and weight value;

[0119] First, the three-dimensional space is divided into a voxel grid. The system divides the space of the medical waste storage area into a regular voxel grid of 0.01m × 0.01m × 0.01m. This resolution is chosen based on the typical size of medical waste containers (sharpware containers are approximately 20 × 20 × 30 cm, and infectious waste bins are approximately 30 × 30 × 50 cm) and the ranging accuracy of the depth sensing module (±2 cm). The truncated signed distance function (TSDF) value represents the distance from the voxel to the surface of the medical waste, with a value range of [-1, 1], where a positive value indicates that the voxel is located at... Outside the surface, a negative value indicates that the voxel is inside the surface, and a value of 0 indicates that the voxel is on the surface. The weight value reflects the reliability of the voxel data. The initial value is 0, and it increases with the number of observations. In the medical waste monitoring scenario, the cutoff distance of the truncated sign distance function is set to 3 times the depth measurement uncertainty (0.15 meters). This value is determined based on the technical specifications of the depth sensing module and the typical curvature of the medical waste container surface. If the cutoff distance is too small, the surface reconstruction will be incomplete. If the cutoff distance is too large, it will introduce too much noise. Through experiments, 0.15 meters was determined to be the optimal cutoff distance.

[0120] For each new input depth frame data, calculate the truncated symbolic distance function observation for each voxel; update the truncated symbolic distance function value and weight value of the voxel based on Bayesian inference rules; and smooth the voxel update process using the exponential moving average method.

[0121] The process first transforms the depth frame data into the world coordinate system. Then, it uses ray casting to determine the intersection point of each voxel with the depth surface along the line of sight. For each voxel, the system calculates its signed distance to the nearest surface and weights it according to the depth measurement uncertainty. Specifically, based on the pose information of the collecting vehicle, the system projects the depth frame data into three-dimensional space and then calculates the Euclidean distance from the center point of each voxel to the depth surface. This distance value is weighted according to the depth measurement uncertainty σ. d =0.05m is normalized and restricted to the range [-1,1] to form the truncated sign distance function observations.

[0122] The process of waste container segmentation and identification includes: initial region segmentation based on multispectral features to generate a set of candidate container regions;

[0123] The system utilizes the spectral response characteristics of multispectral images in the visible, near-infrared, and short-wave infrared bands to extract the material boundaries of medical waste containers. Specifically, the system calculates the spectral intensity vector of each pixel in the three bands and matches it with the spectral features of typical medical waste materials to generate a material similarity map. Then, the system applies an adaptive threshold segmentation algorithm to identify regions with material similarity higher than a threshold as candidate container regions. The threshold selection is based on the spectral mean and variance vectors of typical medical waste materials, and the optimal threshold is determined by maximizing the inter-class variance. In the medical waste monitoring scenario, the short-wave infrared band has unique absorption characteristics for plastic materials, and the system has specifically strengthened the weight of this band, improving the recognition accuracy of plastic containers (such as sharps containers and infectious waste bins). The initial region segmentation process also incorporates depth information to exclude regions that do not match the size of the medical waste container (such as regions smaller than 10×10×10 cm or larger than 50×50×80 cm), generating a preliminary set of candidate container regions.

[0124] Based on the temperature anomaly detection results, candidate container areas are screened and classified.

[0125] The system analyzes the temperature distribution in thermal imaging images and identifies areas with temperatures exceeding 37.5℃ as potential biohazard areas. For each candidate container area, the system calculates its average temperature and temperature variance and compares them with a medical waste temperature anomaly threshold. If the average temperature of the candidate container area exceeds 37.5℃ and the temperature variance is less than the set threshold (indicating a relatively uniform temperature distribution), it is classified as an infectious waste container. If the average temperature is normal but there are local high-temperature points (large temperature variance), it is classified as a pharmaceutical waste container that may contain chemical reactions. If the temperature distribution is uniform and normal, it is classified as a general medical waste container.

[0126] Based on geometric shape features, the DBSCAN clustering algorithm is used to segment container instances;

[0127] The system analyzes the geometric features of candidate container regions, including contour curvature, boundary continuity and region compactness. For each candidate container region, the system extracts its 3D point cloud data and calculates principal curvature, surface normal vector and convex hull features. Then, the system applies the improved DBSCAN clustering algorithm to cluster point clouds with similar geometric shapes into independent container instances.

[0128] The DBSCAN clustering algorithm preferably uses the following parameters: neighborhood radius = 0.05m, minimum number of points = 10, and geometric features including principal curvature, surface normal vector consistency, point cloud compactness, and boundary continuity. The specific implementation steps are as follows: calculate the principal curvature of the point cloud in the candidate container region, and filter points with curvature values ​​between 0.05 and 0.25; calculate the surface normal vector of the point cloud, and filter points with normal vector consistency greater than 0.85; apply the DBSCAN algorithm for clustering, clustering point clouds with similar geometric features into independent container instances; calculate the convex hull volume and bounding box for each clustering result, and filter out clusters with volumes less than 0.001m³ or greater than 0.5m³, thereby achieving container instance segmentation.

[0129] By combining time series data analysis, the state changes of the container can be tracked.

[0130] Finally, the system maintains historical state data for each identified container instance, including location, size, temperature distribution, and filling status. For each new frame, the system calculates the similarity of container features (including geometric shape similarity, spectral feature similarity, and temperature distribution similarity) and matches the container instances in the current frame with historical data to establish temporal continuity.

[0131] The specific implementation of the time series data analysis includes the following steps: The system establishes and maintains a status archive for each identified container instance. This archive records the complete historical information of the container instance, including the location coordinate sequence, size parameter sequence, temperature distribution feature sequence, and filling status sequence. When new monitoring data arrives, the system first calculates the feature similarity between each container instance in the current frame and the container instances in the historical archive. This similarity consists of three parts: geometric shape similarity, calculated by comparing the three-dimensional size ratio (height / diameter ratio), principal curvature distribution characteristics, and surface normal vector consistency of the container; spectral feature similarity, calculated by comparing the spectral reflectance characteristics of the container surface in the visible, near-infrared, and short-wave infrared bands; and temperature distribution similarity, calculated by comparing the average temperature, temperature variance, and hotspot distribution pattern of the container surface. The system sets different weights for each type of similarity. Optionally, the weight for geometric shape similarity is 0.5, the weight for spectral feature similarity is 0.3, and the weight for temperature distribution similarity is 0.2. A comprehensive similarity value is obtained by weighted summation.

[0132] Based on the comprehensive similarity value, the system uses a threshold matching method to associate container instances in the current frame with historical archives: when the comprehensive similarity value is greater than 0.75, the system determines that they are continuous observations of the same container instance; when the comprehensive similarity value is between 0.6 and 0.75, the system further checks the continuity of the container position. If the position change conforms to the movement pattern of the collection vehicle (i.e., the position change is within the expected range), it is still determined to be the same container instance; when the comprehensive similarity value is less than 0.6, the system determines that it is a newly appeared container instance.

[0133] Based on the matching results, the system updates the state information of each container instance and detects state changes:

[0134] Fill status change: By comparing the volume change of the point cloud inside the container, it is determined whether the container is filled or empty. When the volume increases by more than 10% and there is no significant change in temperature distribution, it is determined to be filled; when the volume decreases by more than 10% and there is no significant change in temperature distribution, it is determined to be empty.

[0135] Abnormal temperature changes: By monitoring the temperature change trend on the container surface, potential changes in biological activity can be identified. When the temperature continues to rise by more than 0.5°C / minute and exceeds 37.5°C, a biohazard alarm is triggered.

[0136] Stacking stability changes: By analyzing changes in container position and orientation, stacking stability is assessed. When the container tilt angle exceeds 15 degrees or the position deviation exceeds 5 centimeters, a tipping risk alarm is triggered.

[0137] Furthermore, the monitoring method also includes a fault handling step, which is used to set up a multi-sensor redundancy backup mechanism. When a sensor fails, the system switches to the backup sensor to continue data acquisition, thereby realizing remote monitoring and diagnostic functions and reporting system operating status and abnormal information records.

[0138] First, the system is configured with redundant backups for key sensors. In the monitoring system of the intelligent medical waste collection vehicle, key sensors include a multispectral imaging module, a thermal imaging module, and a depth sensing module. These sensors are crucial for the identification and status monitoring of waste containers. The system is configured with a backup sensor of equivalent performance for each key sensor. For example, the system is configured with a main multispectral imaging module and a backup multispectral imaging module, a main thermal imaging module and a backup thermal imaging module, and a main depth sensing module and a backup depth sensing module.

[0139] In practical applications of intelligent medical waste collection vehicles, the multi-sensor redundancy backup mechanism effectively addresses various sensor failure scenarios. For example, when the main multispectral imaging module experiences image quality degradation due to lens contamination, the system switches to the backup multispectral imaging module within 3.2 seconds, ensuring uninterrupted image data acquisition. When the main thermal imaging module experiences temperature drift due to high-temperature environments, the system switches to the backup thermal imaging module within 4.7 seconds, maintaining continuous temperature monitoring. When the main depth sensing module experiences laser emitter failure due to vibration, the system switches to the backup depth sensing module within 2.8 seconds, ensuring the reliability of depth data. Experimental data shows that this redundancy mechanism reduces the monitoring interruption time caused by system sensor failures from an average of 15.3 minutes to 0.5 minutes, significantly improving system reliability.

[0140] When the system detects a continuous decrease in the signal-to-noise ratio of the multispectral imaging module, it automatically reports the abnormality. After analyzing the data, the remote monitoring center determines that the lens is contaminated and sends a cleaning command. The system then controls the automatic cleaning device to perform the cleaning, avoiding on-site maintenance. When the system frequently triggers re-acquisition in the waste storage room of a specific hospital, the remote monitoring center analyzes the data and finds that the lighting conditions are special, so it remotely adjusts the parameters of the multispectral imaging module. When the temperature of the calculation control unit rises abnormally, the system reports the temperature data. The remote monitoring center diagnoses the problem as a failure of the heat dissipation system and arranges maintenance personnel to bring replacement parts to handle the issue, avoiding system downtime.

[0141] Furthermore, the correction process includes the following steps: registration of point cloud data with multispectral images by combining the unique color and geometric features of medical waste containers; radiometric correction using a relative radiometric calibration method based on a standard reference plate; temperature calibration based on Planck's blackbody radiation law and calibration by measuring the radiation source of a blackbody at a known temperature; and geometric distortion correction using the Brown-Conrady lens distortion model and obtaining distortion parameters through a camera calibration plate for correction and compensation.

[0142] During the monitoring process of the intelligent medical waste collection vehicle, the multispectral imaging module, thermal imaging module, and depth sensing module come from different manufacturers and have different working principles and coordinate systems, resulting in temporal and spatial inconsistencies in the collected data. This inconsistency can lead to errors in subsequent confidence fusion and image restoration steps, affecting the identification and status monitoring of waste containers.

[0143] To address this technical problem, this invention implements four key correction processes in the data registration step: point cloud and image registration based on ORB feature descriptors, radiometric correction based on a standard reference plate, temperature calibration based on Planck's blackbody radiation law, and geometric distortion correction based on the Brown-Conrady model.

[0144] Point cloud and image registration processing based on ORB feature descriptors: This invention employs an improved iterative nearest-point algorithm, combined with the unique color and geometric features of medical waste containers for accurate registration. The system first extracts ORB feature points from the multispectral image, paying particular attention to the container edges and label areas; then, it extracts geometric feature points from the point cloud data, selecting points with curvature values ​​greater than 0.15 as registration features; next, it performs bidirectional matching by combining color and geometric features, applying a random sampling consensus algorithm to eliminate erroneous matches; finally, it uses the improved iterative nearest-point algorithm for accurate registration, dynamically adjusting distance thresholds based on container size (0.03 meters for close distance, 0.05 meters for medium distance, and 0.08 meters for far distance).

[0145] Radiometric correction processing based on standard reference plates: This processing adopts the relative radiometric calibration method. By measuring the standard reference plates with known reflectance (20%, 50%, 80%), the mapping relationship between image grayscale values ​​and actual reflectance is established. The system establishes correction models for visible light, near-infrared and short-wave infrared bands respectively, taking into account the influence of illumination direction and ambient light intensity.

[0146] Temperature calibration processing based on Planck's blackbody radiation law: This processing establishes an accurate mapping between thermal imaging data and actual temperature by measuring blackbody radiation sources at known temperatures (25℃ and 50℃). The system dynamically adjusts emissivity parameters according to the material of the medical waste container, applies different corrections to different material areas, and corrects the fixed-mode noise of the thermal imaging module.

[0147] Geometric distortion correction based on the Brown-Conrady model: This process obtains distortion parameters through a camera calibration board and performs pixel-by-pixel correction using the Brown-Conrady model. The system calculates distortion parameters for each of the three bands, establishes a focal length-distortion model, and considers the influence of temperature on lens distortion.

[0148] The four correction processes are executed sequentially: first geometric distortion correction, then radiometric correction, then temperature calibration, and finally point cloud and image registration. The correction processes share environmental parameters and establish a feedback mechanism. When one process detects an environmental change, it notifies other adjustment parameters. The system dynamically adjusts the correction intensity according to the scene complexity and evaluates the correction quality in real time, triggering recalibration when necessary.

[0149] The specific registration process is as follows: The system first identifies the unique color marking features of medical waste containers (infectious waste containers are marked in red, sharps waste containers are marked in yellow, and pharmaceutical waste containers are marked in white). In the multispectral image, the system performs enhancement processing on these specific color areas, paying special attention to the container edges and label areas, extracting high-contrast color feature points, and adjusting the corresponding band weights according to different label types.

[0150] Secondly, the system analyzes the geometric features of medical waste containers, including typical shapes (sharp waste containers are rectangular prisms, and infectious waste containers are cylindrical), size ranges (e.g., height 0.5-1.0 meters), and structural features. Based on this, the system extracts matching geometric feature points from the point cloud data: high curvature (e.g., >0.25) straight edge points are extracted from sharp waste containers, medium curvature (0.1-0.2) cylindrical surface feature points are extracted from infectious waste containers, and low curvature (<0.1) planar area points are extracted from pharmaceutical waste containers. The system also uses size information to filter out interfering point clouds.

[0151] The system employs a phased verification strategy for feature fusion and matching: first, candidate regions with standard color identifiers are identified; then, matching geometric features are searched in the point cloud; finally, a correspondence is established. The system dynamically adjusts the registration parameters based on the container distance: a strict threshold (0.03 meters) is used for close distances (0.5-1.0 meters); a threshold of 0.05 meters is used for medium distances (1.0-2.0 meters); and a threshold of 0.08 meters is used for long distances (2.0-3.0 meters). The system also considers the influence of the container surface material, enhancing the weight of geometric features for high-reflectivity surfaces and enhancing the weight of color features for low-reflectivity surfaces.

[0152] refer to Figure 2 The present invention also provides a monitoring system based on a medical waste intelligent collection vehicle, the monitoring system comprising:

[0153] The multispectral imaging module is used to acquire visible light, near-infrared, and short-wave infrared images of the medical waste storage area.

[0154] The thermal imaging module is used to acquire infrared temperature distribution images of the area.

[0155] The depth perception module is used to collect three-dimensional point cloud data of the area.

[0156] The computational control unit is used to control the multispectral imaging module to adjust the focal length based on the depth distribution characteristics determined by the three-dimensional point cloud data, and to perform spatiotemporal registration processing on the acquired multimodal sensing data to establish coordinate transformation relationships between different sensor data.

[0157] The computational control unit is also used for: calculating confidence indices reflecting the reliability of each data source based on the registered multimodal sensing data; fusing multiple confidence indices to generate a comprehensive confidence map; using a neural network to repair the missing areas in the image, wherein the training process of the neural network is optimized using a multi-objective loss function that includes a repair error term weighted based on the comprehensive confidence map; constructing and updating a three-dimensional representation of the environment based on the repaired image data and three-dimensional point cloud data; and identifying and monitoring the status of waste containers based on multimodal features.

[0158] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A monitoring method based on an intelligent medical waste collection vehicle, characterized in that, Includes the following steps: The data acquisition step is used to collect multimodal sensing data of the medical waste stacking area through multiple sensors installed on the collection vehicle. The multimodal sensing data includes image data, temperature distribution data and three-dimensional point cloud data. The image data acquisition process is based on the depth distribution characteristics determined by the three-dimensional point cloud data to adjust the focal length. The data registration step is used to perform spatiotemporal registration processing on the collected multimodal sensing data, establish coordinate transformation relationships between data from different sensors, and perform correction processing on the data from each sensor. The confidence fusion step is used to calculate confidence indices reflecting the reliability of each data source based on the registered multimodal sensing data, and to fuse multiple confidence indices to generate a comprehensive confidence map. The image inpainting step is used to repair the missing areas in the image using a neural network. The training process of the neural network is optimized using a multi-objective loss function, which includes a repair error term weighted based on the comprehensive confidence map. The process of constructing the loss function of the neural network includes the following steps: A reconstruction loss term is constructed to calculate the mean square error between the image restored in the occluded region and the ideal image, and the mean square error between the image restored in the non-occluded region and the input image. A depth consistency loss term is constructed to calculate the mean square error between the estimated depth map and the measured depth map and the difference between the depth gradient field; A confidence-weighted loss term is constructed to calculate the repair error of different regions based on the fused confidence map. An edge-preserving loss term is constructed to calculate the difference between the gradient field of the restored image and the gradient field of the ideal image; The four loss terms are combined according to their weighting coefficients to form the final loss function; The status analysis step is used to construct and update a 3D representation of the environment based on the repaired image data and 3D point cloud data, and to identify and monitor the status of waste containers based on multimodal features. The process of updating the three-dimensional representation of the environment includes the following steps: The three-dimensional space is divided into a voxel grid, and each voxel stores the truncation symbol distance function value and weight value; For each new input depth frame data, calculate the truncated symbolic distance function observation for each voxel; update the truncated symbolic distance function value and weight value of the voxel based on Bayesian inference rules; and smooth the voxel update process using the exponential moving average method. The process of waste container segmentation and identification includes: initial region segmentation based on multispectral features to generate a set of candidate container regions; Based on the temperature anomaly detection results, candidate container areas are screened and classified. Based on geometric shape features, the DBSCAN clustering algorithm is used to segment container instances; By combining time series data analysis, the state changes of the container can be tracked.

2. The monitoring method based on a medical waste intelligent collection vehicle according to claim 1, characterized in that, The focal length adjustment is achieved through the following steps: The initial depth map data of the medical waste storage area is obtained through the depth sensing module. The depth statistical feature parameters of the medical waste storage area in the initial depth map are calculated. Based on the depth statistical feature parameters, the focal length sequence is calculated through the depth-focal length mapping model. The gimbal system is controlled to move according to the preset motion trajectory. Multispectral image sequence and thermal imaging image data are collected at each focal length point in the focal length sequence.

3. The monitoring method based on a medical waste intelligent collection vehicle according to claim 1, characterized in that, The multiple confidence levels include spectral confidence, temperature confidence, and depth confidence; The calculation process of the spectral confidence score includes obtaining the spectral intensity vector of the pixel in the visible light, near infrared and short-wave infrared bands, calculating the Euclidean distance between the vector and the mean spectral vector of typical medical waste materials, and calculating the spectral confidence score value by combining the spectral variance vector and the specular reflectance factor parameter. The process of calculating temperature confidence includes obtaining the temperature measurement value of the pixel, calculating the sigmoid function transformation value of the temperature measurement value and the temperature anomaly threshold of medical waste, and calculating the temperature confidence value by combining the temperature field gradient magnitude parameter. The process of calculating depth confidence includes obtaining the depth value measured by lidar and the depth value predicted based on multi-view geometry, calculating the absolute difference between the two depth values, and combining the depth measurement uncertainty parameter and the point cloud curvature estimate to calculate the depth confidence value.

4. The monitoring method based on a medical waste intelligent collection vehicle according to claim 1, characterized in that, The monitoring method also includes a fault handling step, which is used to set up a multi-sensor redundancy backup mechanism. When a sensor fails, the system switches to the backup sensor to continue data acquisition, thereby realizing remote monitoring and diagnosis functions and reporting system operating status and abnormal information records.

5. The monitoring method based on a medical waste intelligent collection vehicle according to claim 1, characterized in that, The correction process includes the following steps: registration of point cloud data with multispectral images based on the unique color and geometric features of medical waste containers; radiometric correction using a relative radiometric calibration method based on a standard reference plate; temperature calibration based on Planck's blackbody radiation law and calibration using a blackbody radiation source with known temperature; and geometric distortion correction using the Brown-Conrady lens distortion model and distortion parameters obtained through a camera calibration plate for correction and compensation.

6. A monitoring system based on a medical waste intelligent collection vehicle, used to implement the monitoring method based on a medical waste intelligent collection vehicle as described in any one of claims 1-5, characterized in that, The monitoring system includes: The multispectral imaging module is used to acquire visible light, near-infrared, and short-wave infrared images of the medical waste storage area. The thermal imaging module is used to acquire infrared temperature distribution images of the area. The depth perception module is used to collect three-dimensional point cloud data of the area. The computational control unit is used to control the multispectral imaging module to adjust the focal length based on the depth distribution characteristics determined by the three-dimensional point cloud data, and to perform spatiotemporal registration processing on the acquired multimodal sensing data to establish coordinate transformation relationships between different sensor data. The computational control unit is also used for: calculating confidence indices reflecting the reliability of each data source based on the registered multimodal sensing data; fusing multiple confidence indices to generate a comprehensive confidence map; using a neural network to repair the missing areas in the image, wherein the training process of the neural network is optimized using a multi-objective loss function that includes a repair error term weighted based on the comprehensive confidence map; constructing and updating a three-dimensional representation of the environment based on the repaired image data and three-dimensional point cloud data; and identifying and monitoring the status of waste containers based on multimodal features.

Images