3D object recognition and counting method based on yolov5 and point cloud clustering

Abstract

The application relates to a 3D object recognition and counting method based on YOLOv5 and point cloud clustering, and belongs to the technical field of computer vision target detection and point cloud processing, and specifically relates to a 3D object recognition and counting method based on YOLOv5 and point cloud clustering. The method comprises the following steps: constructing a target detection data set, and training a YOLOv5 model according to the data set; performing near-distance scene adaptive parameter initialization configuration, and setting a plane filtering double threshold value; acquiring a candidate target region consistent with a pixel coordinate system of a to-be-detected RGB image; generating and pre-processing a near-distance substitute depth map; extracting and quantizing effective depth values of the candidate target region, and performing first screening of plane interference based on the plane filtering double threshold value, to obtain an effective three-dimensional object candidate frame; generating a three-dimensional point cloud based on the effective three-dimensional object candidate frame, and performing second screening of plane interference, to obtain a real three-dimensional object detection frame; and performing point cloud density clustering and accurate counting driven by depth abnormity.

Description

Technical Field

[0001] This invention relates to the field of point cloud processing technology for computer vision target detection, and specifically to a 3D object recognition and counting method based on YOLOv5 and point cloud clustering. Background Technology

[0002] In close-range identification and counting scenarios such as industrial small parts inspection, retail merchandise inventory, laboratory consumables statistics, and pharmaceutical sorting and verification, it is often necessary to quickly identify and count multiple objects on desktops, shelves, conveyor belts, or sorting tables. The target objects in these scenarios are often small in size, similar in appearance, and densely packed, and are prone to mutual obstruction, stacking, edge overlap, and interference from packaging patterns. Inaccurate identification and counting results will lead to extensive manual verification during subsequent inventory, sorting, verification, or warehouse management, reducing operational efficiency and affecting the reliability of the automated system in practical applications.

[0003] In existing technologies, target detection and counting methods based on two-dimensional images typically acquire images using ordinary RGB cameras and utilize target detection models to locate, classify, and count target regions within the images. These methods offer advantages such as low deployment cost and fast processing speed, but they primarily rely on the target's color, texture, and contour appearance features, lacking the ability to determine the target's true spatial thickness, three-dimensional protrusion status, and depth hierarchy. In close-range desktop scenarios, planar images such as posters, screen displays, printed patterns on packaging boxes, and labels may have similar visual textures to real objects, leading to misidentification as real three-dimensional objects. Furthermore, for multiple objects stacked vertically, occluding each other, or densely packed together, the two-dimensional detection bounding box struggles to accurately distinguish the spatial hierarchy and true boundaries of different objects, easily resulting in missed detections, duplicate counting, or misclassifying multiple objects as a single object.

[0004] Another type of existing technology uses RGB-D devices, structured light cameras, time-of-flight cameras, binocular vision devices, or 3D point cloud processing to acquire spatial information, and combines 2D image features and depth information for target detection and counting. While this type of solution can, to some extent, utilize depth differences to determine the spatial structure of a target, it typically relies on dedicated depth acquisition equipment, resulting in high hardware costs and installation and calibration requirements. Furthermore, in close-range detection scenarios of 0.2 meters to 1.0 meters, depth data is easily affected by factors such as camera minimum ranging, depth quantization errors, edge flying spots, local holes, reflective materials, and insufficient thickness of small, thin, sheet-like objects, making it difficult to stably distinguish the depth differences between real objects and planar interference.

[0005] Furthermore, existing deep filtering or point cloud processing methods typically employ fixed thresholds or global point cloud computing for judgment, making it difficult to adapt to scene variations such as different detection distances, target heights, and stacking states. When the threshold is set too strictly, small real objects or objects with low heights are easily mistakenly rejected; when the threshold is set too leniently, printed patterns, screen images, or packaging textures are easily retained as false detections. Performing complex point cloud analysis on all detection areas would also increase unnecessary computation, making it difficult for the system to balance detection accuracy and real-time performance in batch processing scenarios.

[0006] Therefore, existing technologies still suffer from problems such as high false detection rate of planar interference, inaccurate counting in stacked occlusion scenarios, high cost and adaptation requirements of dedicated depth equipment, and heavy computational burden of point cloud processing in close-range 3D object recognition and counting scenarios. There is an urgent need for a 3D object recognition and counting method that is suitable for close-range scenarios and can balance the removal of planar interference, the accuracy of stacked object counting, and processing efficiency. Summary of the Invention

[0007] In the context of close-range object recognition and counting, existing 2D image detection methods struggle to distinguish between planar patterns and real 3D objects, and are unable to accurately handle stacked occluded targets. Existing RGB-D or point cloud detection methods suffer from high equipment costs, insufficient close-range depth stability, and heavy computational burden. The purpose of this invention is to propose a 3D object recognition and counting method based on YOLOv5 and point cloud clustering.

[0008] The method includes the following steps: S1. Construct an object detection dataset and train a YOLOv5 model based on the dataset; S2. Initialize and configure adaptive parameters for close-range scenes, and set dual thresholds for planar filtering; S3. Based on the RGB image to be detected and the trained YOLOv5 model, obtain candidate target regions that are consistent with the pixel coordinate system of the RGB image to be detected. S4. Generate and preprocess a near-field surrogate depth map for the RGB image to be detected; S5. Extract and quantify the effective depth values ​​of the candidate target regions, and perform the first screening of planar interference based on the dual threshold of planar filtering to obtain effective 3D object candidate boxes. S6. Based on the effective 3D object candidate boxes, generate 3D point clouds and perform a second screening of planar interference to obtain the real 3D object detection boxes. S7, Deep Anomaly-Driven Point Cloud Density Clustering and Precise Counting.

[0009] Furthermore, in step S2, the adaptive initialization configuration includes the following parameters: effective detection depth range and camera intrinsic parameter matrix; Planar filtering with dual thresholds includes: depth variance threshold. and depth / thickness threshold ; The formula for calculating the dual threshold of planar filtering is: , ,in, This represents the basic threshold of depth variance. Indicates the basic threshold for depth and thickness. Indicates the detection distance. Indicates the baseline detection distance.

[0010] Furthermore, in step S3, obtaining candidate target regions that are consistent with the pixel coordinate system of the RGB image to be detected includes the following steps: S3.1 Input the RGB image to be detected into the trained YOLOv5 model to obtain several detection boxes; S3.2 Setting the confidence threshold When the confidence level of the detection box is greater than or equal to If the current detection box is not found, it is recorded as a candidate target region; otherwise, the current detection box is discarded. S3.3 Convert the coordinates of the candidate target region into pixel-level detection box coordinates in the RGB image to be detected. ; S3.4 The rectangular area enclosed by the pixel-level detection box coordinates is used as a candidate target area that is consistent with the pixel coordinate system of the RGB image to be detected.

[0011] Furthermore, for the RGB image to be detected, generating a near-field surrogate depth map includes the following steps: Step 1: Perform grayscale conversion and denoising on the RGB image to be detected sequentially to obtain a denoised grayscale image. ; Step 2: Targeting Perform Sobel gradient calculations to obtain gradient maps along the x-axis. and y-axis gradient plot and according to and Calculate gradient magnitude map ; Step 3: Based on and Constructing an initial near-field alternative depth map ,

[0012] in, This represents the luminance inverse response weighting coefficient. Represents the gradient weight coefficients. ; Step 4: Mapped to near-field surrogate depth map ,

[0013] in, and These represent the initial near-field surrogate depth maps. The minimum and maximum pixel values, and These represent the minimum and maximum values ​​of the effective detection depth range, respectively.

[0014] Furthermore, the preprocessing of the near-field replacement depth map includes: clearing or overwriting the previous frame's depth map cache, data conversion, and filtering invalid values.

[0015] Furthermore, the extraction and quantification of the effective depth values ​​of the candidate target regions specifically involves mapping the candidate target regions, which are consistent with the pixel coordinate system of the RGB image to be detected, to the preprocessed near-field surrogate depth map. The corresponding pixel regions in the depth map are used to obtain the pixel coordinates of the left, right, upper, and lower boundaries of the candidate target region. , , and ; according to , , and In the preprocessed near-field alternative depth map The process involves cropping to obtain rectangular pixel-level regions of interest (ROIs) corresponding to the candidate target regions. Traverse the pixel-level Region of Interest (ROI), read the depth value of each pixel, and store the depth values ​​of pixels that meet the preset effective range into the effective depth value set. ; against Numerical statistical calculations were performed to obtain the average depth. Depth variance Minimum depth Maximum depth and depth and thickness .

[0016] Furthermore, the planar interference first-stage screening based on planar filtering with dual thresholds specifically involves: like and If the current candidate target area is determined to be a planar texture interference, it will be directly eliminated. Conversely, it is determined to be a valid candidate bounding box for a 3D object.

[0017] Furthermore, the generation of a 3D point cloud based on valid 3D object candidate boxes specifically involves: Based on the camera intrinsic parameter matrix, each pixel within the valid 3D object candidate box is converted into the camera's 3D coordinate system, generating a 3D point cloud set. ,

[0018]

[0019]

[0020] in, This represents the index value of a pixel within a valid 3D object candidate bounding box. , , They represent the first The coordinates of each pixel in the camera's 3D coordinate system, along the x, y, and z axes. and They represent The x-axis and y-axis coordinates of each pixel in the RGB image pixel coordinate system. Indicates the first The effective depth value of each pixel. Indicates the first The depth value of each pixel in the camera's 3D coordinate system , These are the focal lengths along the camera's x and y axes, respectively. and This represents the x-axis and y-axis coordinates of the center point of the image captured by the camera.

[0021] Furthermore, the second screening of planar interference specifically involves: targeting the three-dimensional point cloud set. Perform noise reduction preprocessing; Set the threshold for determining point cloud thickness. Calculate the set of 3D point clouds after denoising preprocessing Thickness in the Z-axis direction ; like If the current 3D point cloud is identified as a valid 3D object candidate box, and the candidate box is identified as an interference from the surface map, then the candidate box is proposed. Conversely, if the bounding box is not found, it is determined to be a real 3D object detection box.

[0022] Furthermore, the depth anomaly-driven point cloud density clustering and accurate counting includes the following steps: S7.1, if If the current real 3D object detection box is determined to be a depth anomaly region, proceed to step S7.2 to perform cluster counting; like If the current real 3D object detection box is determined to be a single object without stacking, it is directly counted as 1 target and no clustering counting is performed; S7.2. The DBSCAN density clustering algorithm is used to cluster and segment the deep anomaly region into Q independent point cloud clusters. S7.3. Use the value of Q as the number of targets in the real 3D object detection box corresponding to the depth anomaly region; The total number of objects in the entire image is obtained by summing the counts of all real 3D object detection boxes.

[0023] The beneficial effects of the method described in this invention are as follows: (1) When there is no real depth map input, the present invention uses Gaussian denoising, Sobel gradient extraction and brightness weighted fusion to generate a near-distance alternative depth map and makes it maintain pixel-level correspondence with the RGB image, thereby providing auxiliary depth input for subsequent depth feature extraction, planar interference filtering and point cloud approximation processing, reducing the dependence on dedicated RGB-D cameras or industrial depth cameras.

[0024] (2) The present invention extracts depth variance and depth thickness from the candidate detection boxes output by YOLOv5, and adaptively adjusts the variance threshold and thickness threshold according to the average depth of the candidate region. This enables the identification of planar textures, screen patterns or poster interference lacking three-dimensional structure from the perspective of depth fluctuation and spatial thickness, reducing false detections caused by relying solely on two-dimensional texture and contour features.

[0025] (3) The present invention generates a three-dimensional point cloud for the candidate region after the first screening, and further makes a second judgment based on the Z-axis thickness. This can verify whether the candidate target has an actual protruding structure from the three-dimensional spatial geometric features, reducing the probability that flat patterns, printed images or screen display content are misidentified as real objects.

[0026] (4) The present invention first determines whether there is a risk of stacking or occlusion based on the depth anomaly of the candidate region, performs DBSCAN point cloud density clustering only on the region with risk, and corrects the counting result based on the number of clusters, thereby improving the problem of undercounting or missing counts when multiple targets are occluded, densely placed or stuck together.

[0027] (5) The present invention first removes low-confidence detection boxes, and then filters planar interference through depth double threshold. Point cloud processing is only performed on the remaining candidate regions or regions with stacking risk, thereby avoiding indiscriminate three-dimensional calculations on all detection boxes, reducing the amount of point cloud computing, and improving the overall recognition and counting efficiency. Attached Figure Description

[0028] Figure 1 This is a flowchart of the method described in this invention. Detailed Implementation

[0029] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Example 1 This embodiment provides a 3D object recognition and counting method based on YOLOv5 and point cloud clustering. The flowchart of the method is as follows: Figure 1 As shown, the method includes the following steps: S1. Construct an object detection dataset and train a YOLOv5 model based on the dataset; The relevant operations in step S1 will be introduced with specific examples: Object detection dataset construction: RGB images of the targets to be detected were collected in close-range desktop scenes (shooting distance 0.2-1.0 meters), covering various scenes such as single objects, stacked occlusions, and scenes with planar texture interference; the images were labeled using the Labelimg tool to generate YOLO format label files, with the label content including the target category and the normalized center coordinates of the detection box. The detection box width w and height h are set; the dataset is divided into training set, validation set and test set in a ratio of 8:1:1.

[0031] YOLOv5 model training: Based on the YOLOv5s network architecture (balancing accuracy and inference speed), pre-trained weights are loaded, and training hyperparameters are set as follows: input image size 640×640, batch size=16, initial learning rate 0.001, training epochs=100, intersection-over-union threshold IOU=0.5, confidence threshold conf_thres=0.25; the model is trained using the training set, and the model weights are adjusted with the help of the validation set, finally obtaining a YOLOv5 pre-trained weight file adapted to this scenario.

[0032] S2. Initialize and configure adaptive parameters for close-range scenes, and set dual thresholds for planar filtering; The relevant operations in step S2 will be introduced with specific examples: For close-range desktop detection scenarios, the effective depth range, dual thresholds for planar filtering, and camera intrinsic parameters are pre-configured to provide a unified parameter benchmark for subsequent depth map filtering, mapping of 2D detection boxes to 3D point cloud regions, and planar interference removal.

[0033] Depth range configuration: Set the effective detection depth range to 0.2 m to 1.0 m, and the optimal adaptation range to 0.3 m to 0.8 m. Depth data outside this range is considered invalid and will be removed in subsequent depth analysis and point cloud generation processes to reduce the impact of near-distance occlusion, far-distance background and irrelevant areas on target recognition and counting results.

[0034] Camera intrinsic parameter initialization: Configure the camera intrinsic parameter matrix K, as shown in the following formula: in, , These are the normalized focal lengths (in pixels) along the camera's x and y axes, respectively. , These are the x and y coordinates (in pixels) of the center point of the image captured by the camera; this intrinsic parameter matrix is ​​used for coordinate mapping in the subsequent conversion of the depth map to a 3D point cloud.

[0035] Set a dual threshold for planar filtering, the dual threshold for planar filtering including a depth variance threshold. and depth / thickness threshold Among them, the depth variance threshold Used to determine the overall fluctuation of depth values ​​within a candidate region. Depth / thickness threshold. This is used to determine the spatial thickness difference between the maximum and minimum depth values ​​within a candidate region, and simultaneously, the threshold is adaptively updated based on the detection distance. , Based on the detection distance (i.e.) The formula for adaptive adjustment is as follows:

[0036]

[0037] In the formula, This represents the basic threshold of depth variance. Indicates the basic threshold for depth and thickness. and The value range is 3mm to 10mm. Meters serve as the reference detection distance; this allows for tighter thresholds at close range and wider thresholds at long range, addressing the poor adaptability issue of fixed thresholds. This represents the detection distance, specifically calculated from the average depth obtained in step S5. Substitute the detection distance This can prevent threshold fluctuations caused by noise, reflections, and edge holes affecting the depth value of a single pixel.

[0038] S3. Based on the RGB image to be detected and the trained YOLOv5 model, obtain candidate target regions that are consistent with the pixel coordinate system of the RGB image to be detected. The relevant operations in step S3 will be introduced with specific examples: This system batch reads RGB images to be detected, supporting formats such as JPG and PNG, automatically filters invalid image files, and is compatible with Chinese paths and case-sensitive file extensions. The RGB images are then input into a trained YOLOv5 object detection model to obtain candidate object detection boxes, object categories, and corresponding confidence scores. Preliminary object detection using YOLOv5 is performed to quickly identify candidate regions where potential objects may exist in 2D images, limiting subsequent depth analysis and point cloud processing to these candidate regions to reduce unnecessary computation caused by performing depth calculations and point cloud clustering on the entire image.

[0039] In the YOLOv5 inference detection process, a pre-threshold for confidence is set. The The value range is 0.3 to 0.7, preferably 0.5. Read the detection results output by YOLOv5 to obtain the bounding box coordinates, confidence score, and category information. When the bounding box confidence score is lower than... If the detection box is deemed a low-confidence candidate box, it is discarded, and subsequent depth extraction, planar filtering, and point cloud clustering processes are no longer performed on it; if the confidence level of the detection box is not lower than a certain threshold, it is discarded. When this happens, the detection box is retained as a candidate target region.

[0040] The normalized bounding box coordinates output by YOLO are converted into pixel-level bounding box coordinates in the RGB image to be detected. The specific process is as follows: Let the normalized coordinates of the center point of the candidate target region be... The normalized width and height are w and h, respectively. The width and height of the input RGB image to be detected are set to be... , (Unit: pixels), then the pixel boundary coordinates of the detection box are represented as:

[0041]

[0042]

[0043]

[0044] in, , , , These are the pixel coordinates of the left, right, top, and bottom boundaries of the detection box, respectively. The rectangular region enclosed by these pixel-level detection box coordinates is considered as a candidate target region consistent with the pixel coordinate system of the RGB image to be detected.

[0045] ROI normalization is performed on candidate target regions, cropping those exceeding the boundaries of the RGB image to the effective image area. Abnormal candidate target regions are then removed based on preset minimum width, minimum height, minimum area, and aspect ratio ranges. The ROI-normalized candidate target regions are used for subsequent depth map region cropping, planar interference detection, and point cloud clustering to ensure a consistent spatial correspondence between the subsequent 3D analysis objects and the 2D detection results.

[0046] S4. Generate and preprocess a near-field surrogate depth map for the RGB image to be detected; The relevant operations in step S4 will be introduced with specific examples: This step is used to obtain depth map data that has a pixel-level correspondence with the input RGB image, providing a depth foundation for subsequent candidate region depth feature calculation, planar interference filtering, and 3D point cloud generation. The depth map can be the original depth map acquired by a depth camera, or a near-range substitute depth map generated based on the RGB image when no original depth map is available.

[0047] S4.1 Depth Map Acquisition Mode Determination: Determine whether the current input data contains the original depth map corresponding to the RGB image. If the input data contains the original depth map, perform size alignment, coordinate registration, and invalid value processing on the original depth map before proceeding to step S4.4; if the input data does not contain the original depth map, perform a near-field replacement depth map generation process based on the input RGB image and proceed to step 4.2.

[0048] S4.2 Image Preprocessing: The input RGB image is converted to grayscale to obtain a grayscale image. Use a 5×5 Gaussian kernel to check the grayscale image. Gaussian blurring is performed to obtain the denoised grayscale image. Gaussian blurring suppresses high-frequency noise and fine texture interference in the image, reducing the probability that noise points will be misidentified as target edges or regions of abrupt depth changes in subsequent gradient calculations.

[0049] 4.3 Generation of Near-Range Substitute Depth Maps: Denoising grayscale image Perform Sobel gradient calculations to obtain gradient maps along the x-axis. and y-axis gradient plot And according to the standard formula for calculating the magnitude of the Sobel gradient:

[0050] Calculate gradient magnitude map The gradient magnitude map Used to characterize edge strength, contour abrupt changes, and object spatial boundaries in an image, its pixel value ranges from 0 to 255.

[0051] Perform brightness-weighted fusion to simulate the imaging patterns of a real depth camera: using the desktop background as the far field (large depth value) and protruding objects as the near field (small depth value), weighted fusion of the gradient magnitude map and brightness map is performed to construct an initial near-field substitute depth map. The calculation formula is as follows:

[0052] in, This is the brightness weighting coefficient, with a value ranging from 0.6 to 0.8; The gradient response weight coefficients range from 0.2 to 0.4 and satisfy the following conditions: ; This is the brightness inverse response term, used to characterize the difference in grayscale response between the target area and the background area in a close-up desktop scene.

[0053] Replace the initial near-field depth map Linear mapping to 16-bit normalized near-field surrogate depth map The effective detection depth range corresponds to the 0.2 m to 1.0 m effective depth range set in step S2, and the mapping formula is as follows:

[0054] in, and These represent the initial near-field surrogate depth maps. The minimum and maximum pixel values, rice, The meters represent the minimum and maximum effective detection depth range, respectively; the 16-bit standardized near-field surrogate depth map generated by the above formula... It achieves pixel-level registration by ensuring that the pixel size is completely consistent with the RGB image to be detected.

[0055] 4.4 Depth Map Caching and Preprocessing: For each frame of input image, the depth map cache of the previous frame is cleared or overwritten, and the current RGB image is bound to the current depth map by frame number or timestamp to avoid the reuse of historical depth maps, which would lead to depth analysis errors.

[0056] Subsequently, the 16-bit depth map was converted into floating-point depth data and then uniformly converted into millimeter units; Invalid depth values ​​less than 0.2 meters or greater than 1.0 meters are filtered out, and holes, abnormal noise points, and pixels that do not meet the effective depth range are also removed to obtain a preprocessed near-field alternative depth map. .

[0057] The preprocessed near-field substitution depth map Used for subsequent calculation of candidate region depth dispersion, depth and thickness, and point cloud coordinate generation.

[0058] S5. Extract and quantify the effective depth values ​​of the candidate target regions, and perform the first screening of planar interference based on the dual threshold of planar filtering to obtain effective 3D object candidate boxes. The relevant operations in step S5 will be introduced with specific examples: This step is used to perform a preliminary depth geometry screening of the candidate target regions with consistent pixel coordinate systems of the RGB image to be detected obtained in step S3. Before entering point cloud clustering, false detection regions that lack three-dimensional convex features, such as planar textures, screen images, and packaging printing patterns, are first removed based on the depth distribution characteristics of the candidate regions, and the depth features required for subsequent point cloud processing are extracted.

[0059] S5.1 Extracting the effective depth values ​​of candidate target regions: For the candidate target regions obtained in step S3 that are consistent with the pixel coordinate system of the RGB image to be detected, map the target regions to the corresponding pixel regions in the depth map output in step S4. Let the width and height of the depth map output in step S4 be W and H, respectively; The normalized center coordinates of the candidate target region are: Normalize the width and height w and h, and map them to pixel-level regions of interest (ROIs) in the depth map output in step S4. The mapping formula is as follows: ; ; ; .

[0060] and These represent the maximum value function and the minimum value function, respectively. , , and These are the left, right, top, and bottom pixel coordinates of the candidate target region in the depth map. Based on these pixel boundary coordinates of the candidate target region in the depth map, a rectangular pixel-level region of interest (ROI) corresponding to the candidate target region is cropped from the pixel boundary coordinates in the depth map.

[0061] If the YOLOv5 output has already been converted to RGB image pixel coordinates in step 3, and the depth map is replaced by a close-up view... If the image has the same size and pixel-level correspondence as the RGB image, then the corresponding ROI is directly extracted from the depth map using that pixel coordinate; if a closer image is used to replace the depth map... If the size is different from that of the RGB image, the coordinates of the RGB detection box are mapped to the depth map coordinate system according to the scale ratio of the two or the camera registration parameters.

[0062] Traverse the ROI pixel by pixel and read the depth value of each pixel. Considering that data acquired by depth cameras (such as structured light or ToF cameras) often contains invalid measurement points due to occlusion, reflection, or depth boundaries, this step only retains measurement points that meet the preset effective range. The pixel depth value (based on the shooting distance setting in step 2, taken as...) , Store the depth values ​​of all valid pixels that meet the conditions into a valid depth value set. :

[0063] in, Indicates the 1st generation within the ROI region The depth value of each valid pixel. Indicates the 1st generation within the ROI region The effective pixels in the preprocessed near-field surrogate depth map The horizontal pixel coordinates in Indicates the 1st generation within the ROI region The effective pixels in the preprocessed near-field surrogate depth map The vertical pixel coordinates in the image. This represents the total number of valid depth pixels within the ROI region. Below the preset minimum effective pixel count threshold If the detection box region lacks sufficient depth support information, it may be a false detection at a long distance or a false detection due to low texture. In this case, the detection box is directly marked as an invalid candidate box and all subsequent calculation steps are skipped to save computing power.

[0064] S5.2, Deep Feature Quantization Calculation: For Numerical statistics (mean, variance, minimum, and maximum) are calculated to obtain the following deep quantization features: Average depth ,for The average of all valid depth values ​​within the range; Depth Variance ,for The variance of the effective depth values ​​within the range characterizes the degree of dispersion of the depth values. Minimum depth Maximum depth , respectively Minimum and maximum values ​​of the effective depth within the interior; Depth : , characterization The spatial thickness of an internal object.

[0065] S5.3, First screening of dual-threshold plane interference: Using depth variance threshold Depth and thickness thresholds A dual judgment is performed, and the judgment rules are as follows: If the candidate target region simultaneously meets the following conditions and If the condition is met, the candidate target area is determined to be planar texture interference and is directly removed; otherwise, it is determined to be a valid 3D object candidate box and enters the subsequent point cloud processing stage.

[0066] The effective depth value of any pixel within the valid 3D object candidate bounding box is denoted as . , The index value of a pixel within a valid 3D object candidate bounding box. , This represents the total number of pixels within the candidate bounding box of a valid 3D object.

[0067] S6. Based on the effective 3D object candidate boxes, generate a 3D point cloud and perform a second filtering of planar interference to obtain the true 3D object detection boxes: The relevant operations in step S6 will be introduced with specific examples: This step converts the valid 3D object candidate boxes into 3D spatial point clouds, achieving a second precise removal of planar interference from a geometrical perspective, thus ensuring the authenticity of the detected targets.

[0068] S6.1 Based on the camera intrinsic parameter matrix K initialized in step 2, each pixel within the effective 3D object candidate box is transformed from the RGB image pixel coordinate system to the camera 3D coordinate system, based on the camera intrinsic parameter matrix initialized in step 2. Each pixel within the valid 3D object candidate bounding box is transformed from the image pixel coordinate system to the camera 3D coordinate system (this coordinate system has the camera optical center as its origin). (Axis along the optical axis), generating a 3D point cloud set. .in , These are the normalized focal lengths (in pixels) along the camera's x and y axes, respectively. , These are the x and y coordinates (in pixels) of the center point of the image captured by the camera. A 3D point cloud set is generated. The coordinate transformation formula is as follows:

[0069]

[0070]

[0071] in, For the first The coordinates of each pixel in the image coordinate system For the first The depth value (in millimeters) corresponding to each pixel. For the first The depth value of each pixel in the camera's 3D coordinate system (unit: meters). , , , Indicates the first The coordinates of each pixel on the x, y, and z axes in the camera's 3D coordinate system are used to obtain the 3D point cloud of the ROI region. .

[0072] S6.2 Point Cloud Denoising Preprocessing: A statistical filtering algorithm is used to remove outlier noise points in the point cloud and retain the main point cloud data; the number of neighboring points k=20 and the standard deviation multiple threshold is set to 2.0 to filter outlier points that exceed twice the standard deviation of the average neighborhood distance.

[0073] S6.3 Secondary Precise Filtering Based on Z-Axis Thickness of Point Cloud: Calculating the 3D Point Cloud Set After Denoising Preprocessing Thickness in the Z-axis direction The formula is as follows:

[0074] in, , These are the 3D point cloud sets after denoising preprocessing. Maximum and minimum coordinate values ​​in the Z-axis direction; set the point cloud thickness threshold. (Adjustable range 3mm~8mm), if If the bounding box is not found to be a planar texture interference, it is determined to be a true 3D object bounding box and is removed; otherwise, it is determined to be a true 3D object bounding box, and is retained and proceeded to the subsequent counting stage.

[0075] S7. Deep anomaly-driven point cloud density clustering and precise counting: The relevant operations in step S7 will be introduced with specific examples: This step only performs point cloud clustering operations on areas with a risk of stacking occlusion, balancing the processing speed of single-object scenes with the counting accuracy of stacked scenes.

[0076] S7.1 Depth Anomaly Detection: Based on the true 3D object detection boxes retained in step S6, the depth variance calculated in step S5 is used to determine the depth anomaly. To determine whether there is a risk of overlapping occlusion in the area, the following rules are used: like If the region is identified as having an abnormal depth and poses a risk of stacking occlusion, proceed to step S7.2 to perform cluster counting; if If the object is a single object with no stacked area, it is directly counted as one target, and no clustering operation is performed, thus improving processing speed.

[0077] S7.2 Point Cloud Density Clustering: For 3D point clouds in depth anomaly regions, the DBSCAN density clustering algorithm is used for clustering and segmentation, automatically distinguishing multiple stacked targets; Clustering hyperparameters are set: neighborhood radius. Meter (10mm), minimum number of neighborhood points After clustering, Q independent point cloud clusters are obtained.

[0078] S7.3 Counting result correction: The number of point cloud clusters Q obtained by clustering is used as the number of targets within the detection box to complete the accurate counting of the stacked scene; the counting results of all real 3D object detection boxes are summed to obtain the total number of targets in the entire image.

[0079] S7.4 Counting Results Visualization and Structured Storage Output Visualization rendering: Draw the preserved real 3D object detection boxes on the original RGB image, and label the object category, count, average depth, and point cloud thickness information above the detection boxes. Set the label font to a Chinese-supporting sans-serif font with the font size adapted to the size of the detection boxes to generate a visualization effect.

[0080] Results storage: Save the visualization to the specified output folder with the original filename; at the same time, generate a structured result file in CSV format, which includes information such as image filename, detection box pixel coordinates, object category, depth features, point cloud thickness, and number of counts.

[0081] Memory reclamation optimization: After batch processing is completed, the memory occupied by the model, image, and point cloud data is automatically released to avoid memory overflow and ensure the stability of large-scale batch data processing.

Claims

1. A 3D object recognition and counting method based on YOLOv5 and point cloud clustering, characterized in that, The method includes the following steps: S1. Construct an object detection dataset and train a YOLOv5 model based on the dataset; S2. Initialize and configure adaptive parameters for close-range scenes, and set dual thresholds for planar filtering; S3. Based on the RGB image to be detected and the trained YOLOv5 model, obtain candidate target regions that are consistent with the pixel coordinate system of the RGB image to be detected. S4. Generate and preprocess a near-field surrogate depth map for the RGB image to be detected; S5. Extract and quantify the effective depth values ​​of the candidate target regions, and perform the first screening of planar interference based on the dual threshold of planar filtering to obtain effective 3D object candidate boxes. S6. Based on the effective 3D object candidate boxes, generate 3D point clouds and perform a second screening of planar interference to obtain the real 3D object detection boxes. S7, Deep Anomaly-Driven Point Cloud Density Clustering and Precise Counting.

2. The 3D object recognition and counting method based on YOLOv5 and point cloud clustering according to claim 1, characterized in that, In step S2, the adaptive initialization configuration includes the following parameters: effective detection depth range and camera intrinsic parameter matrix; Planar filtering with dual thresholds includes: depth variance threshold. and depth / thickness threshold ; The formula for calculating the dual threshold of planar filtering is: , ,in, This represents the basic threshold of depth variance. Indicates the basic threshold for depth and thickness. Indicates the detection distance. Indicates the baseline detection distance.

3. The 3D object recognition and counting method based on YOLOv5 and point cloud clustering according to claim 2, characterized in that, Step S3, obtaining candidate target regions that are consistent with the pixel coordinate system of the RGB image to be detected, includes the following steps: S3.1 Input the RGB image to be detected into the trained YOLOv5 model to obtain several detection boxes; S3.2 Setting the confidence threshold When the confidence level of the detection box is greater than or equal to If the current detection box is not found, it is recorded as a candidate target region; otherwise, the current detection box is discarded. S3.3 Convert the coordinates of the candidate target region into pixel-level detection box coordinates in the RGB image to be detected. ; S3.4 The rectangular area enclosed by the pixel-level detection box coordinates is used as a candidate target area that is consistent with the pixel coordinate system of the RGB image to be detected.

4. The 3D object recognition and counting method based on YOLOv5 and point cloud clustering according to claim 3, characterized in that, Generating a near-field surrogate depth map for the RGB image to be detected includes the following steps: Step 1: Perform grayscale conversion and denoising on the RGB image to be detected sequentially to obtain a denoised grayscale image. ; Step 2: Targeting Perform Sobel gradient calculations to obtain gradient maps along the x-axis. and y-axis gradient plot and according to and Calculate gradient magnitude map ; Step 3: Based on and Constructing an initial near-field alternative depth map , in, This represents the luminance inverse response weighting coefficient. Represents the gradient weight coefficients. ; Step 4: Mapped to near-field surrogate depth map , in, and These represent the initial near-field surrogate depth maps. The minimum and maximum pixel values, and These represent the minimum and maximum values ​​of the effective detection depth range, respectively.

5. The 3D object recognition and counting method based on YOLOv5 and point cloud clustering according to claim 4, characterized in that, The preprocessing of the near-field replacement depth map includes: clearing or overwriting the previous frame's depth map cache, data conversion, and filtering invalid values.

6. The 3D object recognition and counting method based on YOLOv5 and point cloud clustering according to claim 5, characterized in that, The extraction and quantification of the effective depth values ​​of the candidate target regions specifically involves mapping the candidate target regions, which are consistent with the pixel coordinate system of the RGB image to be detected, onto the preprocessed near-field surrogate depth map. The corresponding pixel regions in the depth map are used to obtain the pixel coordinates of the left, right, upper, and lower boundaries of the candidate target region. , , and ; according to , , and In the preprocessed near-field alternative depth map The process involves cropping to obtain rectangular pixel-level regions of interest (ROIs) corresponding to the candidate target regions. Traverse the pixel-level Region of Interest (ROI), read the depth value of each pixel, and store the depth values ​​of pixels that meet the preset effective range into the effective depth value set. ; against Numerical statistical calculations were performed to obtain the average depth. Depth variance Minimum depth Maximum depth and depth and thickness .

7. The 3D object recognition and counting method based on YOLOv5 and point cloud clustering according to claim 6, characterized in that, The first screening of planar interference based on planar filtering with dual thresholds is as follows: like and If the current candidate target area is determined to be a planar texture interference, it will be directly eliminated. Conversely, it is determined to be a valid candidate bounding box for a 3D object.

8. The 3D object recognition and counting method based on YOLOv5 and point cloud clustering according to claim 7, characterized in that, The process of generating a 3D point cloud based on valid 3D object candidate boxes specifically involves: Based on the camera intrinsic parameter matrix, each pixel within the valid 3D object candidate box is converted into the camera's 3D coordinate system, generating a 3D point cloud set. , in, This represents the index value of a pixel within a valid 3D object candidate bounding box. , , They represent the first The coordinates of each pixel in the camera's 3D coordinate system, along the x, y, and z axes. and They represent The x-axis and y-axis coordinates of each pixel in the RGB image pixel coordinate system. Indicates the first The effective depth value of each pixel. Indicates the first The depth value of each pixel in the camera's 3D coordinate system , These are the focal lengths along the camera's x and y axes, respectively. and This represents the x-axis and y-axis coordinates of the center point of the image captured by the camera.

9. The 3D object recognition and counting method based on YOLOv5 and point cloud clustering according to claim 8, characterized in that, The second screening of planar interference specifically involves: targeting the 3D point cloud set. Perform noise reduction preprocessing; Set the threshold for determining point cloud thickness. Calculate the set of 3D point clouds after denoising preprocessing Thickness in the Z-axis direction ; like If the current 3D point cloud is identified as a valid 3D object candidate box, and the candidate box is identified as an interference from the surface map, then the candidate box is proposed. Conversely, if the bounding box is not found, it is determined to be a real 3D object detection box.

10. The 3D object recognition and counting method based on YOLOv5 and point cloud clustering according to claim 9, characterized in that, The depth anomaly-driven point cloud density clustering and accurate counting includes the following steps: S7.1, if If the current real 3D object detection box is determined to be a depth anomaly region, proceed to step S7.2 to perform cluster counting; like If the current real 3D object detection box is determined to be a single object without stacking, it is directly counted as 1 target and no clustering counting is performed; S7.

2. The DBSCAN density clustering algorithm is used to cluster and segment the deep anomaly region into Q independent point cloud clusters. S7.

3. Use the value of Q as the number of targets in the real 3D object detection box corresponding to the depth anomaly region; The total number of objects in the entire image is obtained by summing the counts of all real 3D object detection boxes.

Images