A method and apparatus for detecting the tilt of finished grain stacks based on computer vision

Abstract

This invention discloses a method and apparatus for detecting the tilt of finished grain stacks based on computer vision, belonging to the field of grain depot management technology. The method includes: using a binocular vision camera as the measuring device and performing stereo calibration; acquiring left and right images of the stack at fixed points and times; using deep learning-based stereo matching technology to obtain disparity maps of the left and right images, and acquiring image depth information and point cloud information; comparing the point cloud information of the current stack image with the point cloud information of the reference image, and determining whether the stack is tilted based on the comparison result. This invention employs a domain adaptive technique based on color distribution transfer, enabling the model to maintain efficient feature extraction capabilities even under complex lighting conditions in grain depots. It can be deployed without manual annotation, reducing application costs. The invention introduces a residual network to extract multi-scale features and combines it with an adaptive aggregation module, resulting in clearer object contours in the output disparity map and improved detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of grain depot management technology, specifically a method and device for detecting the tilt of finished grain stacks based on computer vision. Background Technology

[0002] Monitoring the tilting of packaged grain stacks within grain depots has become a crucial task in daily grain depot management due to its implications for worker safety. However, relying on manual inspections is often limited by time and manpower, making continuous and effective monitoring difficult. In recent years, with the development of information technology in grain depots, high-definition cameras have been deployed inside and around grain depots, providing convenience for managers to remotely inspect warehouses via video. However, this method is also limited by working hours and lacks intelligent video analysis and automatic early warning mechanisms. The identification of abnormal events and risk prediction still depend on manual judgment, making it difficult to promptly detect safety hazards such as tilting stacks.

[0003] Although existing research has attempted to introduce advanced technologies into grain depot management, the application and implementation of technologies such as artificial intelligence in grain warehouse monitoring remain relatively lagging due to the unique nature of the grain storage environment, and the overall level of intelligence needs to be improved. In the daily management of grain depots storing finished grain, preventing stack tilt is of paramount importance. Currently, the primarily manual inspection method is not only limited by manpower and working hours but is also prone to inaccuracies due to subjective experience and visual fatigue, making it difficult to detect early, minor tilts. This could lead to stack collapse, causing personal injury and property damage. Therefore, achieving automated, high-precision tilt monitoring is of significant practical importance.

[0004] Computer vision technology is increasingly used in agriculture and warehousing, mainly focusing on pest detection and grain condition monitoring. However, real-time monitoring technology for dynamic risks such as stack tilting is relatively scarce. Traditional two-dimensional image processing is difficult to obtain depth information and cannot accurately quantify the tilt displacement of the stack in the direction perpendicular to the ground. Moreover, existing solutions mostly rely on traditional image processing algorithms, which are poorly adaptable to complex environments such as low lighting in warehouses and weak textures in packaging bags, and the detection accuracy is difficult to meet practical needs.

[0005] Stereo matching is the core of binocular vision, aiming to find corresponding pixels in the left and right views to calculate disparity. However, traditional stereo matching algorithms (such as SGM and BM) have significant shortcomings in grain depot scenarios: although computationally efficient, they are sensitive to uniform texture and local occlusion on the stacked surface in a grain warehouse environment, resulting in large disparity map errors, insufficient accuracy in 3D coordinate calculation, and inability to accurately capture subtle tilts of the stacks. With recent developments, stereo matching methods based on convolutional neural networks (CNNs) have gradually demonstrated powerful feature extraction capabilities. These methods can handle weakly textured regions and utilize architectures such as ResNet to extract multi-scale features, thereby improving matching accuracy.

[0006] However, due to the lack of public datasets for stacking images, existing stereo matching models are difficult to directly transfer and apply. Directly using models trained on public datasets to process grain warehouse data will result in a significant performance drop due to domain shift. Although domain adaptation technology can solve the problem of data distribution differences across scenarios, there is little targeted research in grain warehouse stacking monitoring, and a mature technical solution has not yet been formed, resulting in insufficient model generalization ability and inability to meet actual usage needs.

[0007] Based on this, a method and apparatus for detecting the tilt of finished grain stacks based on computer vision are now provided, which can eliminate the drawbacks of existing technical solutions. Summary of the Invention

[0008] The purpose of this invention is to provide a method and apparatus for detecting the tilt of finished grain stacks based on computer vision, so as to solve the problems of the shortcomings of existing detection technologies in the background art.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] A computer vision-based method for detecting the tilt of finished grain stacks includes the following steps:

[0011] Step S1: Use a binocular vision camera with the same parameters and configuration as the measurement device, and perform stereo calibration on the binocular vision camera to obtain the internal and external parameters of the binocular camera.

[0012] Step S2: Acquire left and right images of the stack at fixed points and time intervals using the binocular vision camera, obtain the disparity map of the left and right images using deep learning-based stereo matching technology, obtain image depth information, and obtain the point cloud information of the current stack image by combining calibration parameters.

[0013] Step S3: Compare the point cloud information of the current stacking image with the point cloud information of the reference image, and determine whether the stacking is tilted based on the comparison result. The reference image is the image when the stacking is not tilted.

[0014] The comparison operation of the point cloud information specifically includes:

[0015] Select the point cloud information of the central region of the reference image, calculate the average physical distance and segmented average distance of the central region, and save the results;

[0016] Select the point cloud information of the corresponding central region in the current stacking image and calculate the average physical distance;

[0017] The average physical distance of the current stacking image is compared with the average physical distance of the reference image. If the difference exceeds a preset threshold, an alarm is triggered, and different alarm modes are selected according to the size of the difference.

[0018] Furthermore, the stereo calibration in step S1 specifically includes:

[0019] Establish the geometric relationship between the pixel coordinates of the two-dimensional image and the coordinates of the three-dimensional world, and fix the binocular vision camera at a predetermined monitoring position at a distance from the front of the stack;

[0020] In a well-lit and uniform environment, several sets of left and right images of the stack were collected by adjusting the position and angle of the black and white checkerboard calibration board.

[0021] The acquired left and right images are input into the stereo calibration program to calculate the intrinsic parameter matrix, distortion coefficients and extrinsic parameter matrix of the binocular vision camera, which are then used as the calibration results.

[0022] The reprojection matrix is ​​derived from the calibration results. The disparity map is then mapped to three-dimensional coordinate information using the reprojection matrix. The intrinsic matrix, distortion coefficients, extrinsic matrix, and reprojection matrix are used as calibration parameters obtained from stereo calibration.

[0023] Furthermore, in step S2, after acquiring left and right images of the stacked structure at fixed points and times using a binocular vision camera, a stereoscopic correction operation is performed on the left and right images:

[0024] Using the calibration parameters obtained from stereo calibration, distortion correction and stereo correction are performed on the original left and right images of the stack, so that the corrected left and right images satisfy the epipolar geometric constraint, that is, any pixel in the left image and its corresponding matching pixel in the right image are located on the same horizontal scan line, satisfying the condition that the epipolar lines are parallel and the rows are aligned.

[0025] Furthermore, in step S2, the deep learning-based stereo matching technique outputs a disparity map through a deep learning model, specifically including:

[0026] A domain adaptation technique based on color distribution transfer is used to transfer the color statistical features of unlabeled stacked images from a grain warehouse to a public dataset with real parallax labels. Specifically, the public dataset images and unlabeled stacked images are converted from RGB space to CIE Lab color space. The global mean and variance of the stacked image set are calculated using formulas. The color distribution of the public dataset images is adjusted to match the lighting and texture features of the grain warehouse environment. The transformed images are then converted back to RGB format.

[0027] The stereo-corrected left and right images are input into a pre-trained deep neural network. The deep neural network uses a residual network as the backbone network to extract multi-scale features of the image. After multi-scale feature extraction, the normalized correlation coefficient is used to calculate the matching cost of the left and right feature maps and construct a multi-scale matching cost volume.

[0028] A stacked adaptive aggregation module is used to aggregate matching cost volumes at different scales. The SoftArgmax function is used to regress the aggregated cost and output a disparity map. The adaptive aggregation module adopts a cost aggregation strategy that combines intra-scale aggregation and cross-scale aggregation.

[0029] The cost aggregation strategy is expressed by the following formula: ,in, Indicated in pixels Point parallax is The cost of aggregation The number of sampling points. For the first Aggregate weights of points, For pixels The fixed offset of the point For additional offset, For specific location weights, The matching cost at a certain scale and satisfying D, H, and W represent the current maximum parallax value, height, and width, respectively.

[0030] Furthermore, the formulas involved in calculating the global mean and variance of the stacked image set and adjusting the color distribution of the images in the public dataset specifically include:

[0031] ;

[0032] ;

[0033] ;

[0034] in, and Let represent the global mean and variance of the stacked image set, respectively. and The mean and variance of the stacked images in the current batch. and This represents the mean and variance of the public dataset in the current batch.

[0035] Furthermore, the formula for calculating the matching cost based on the normalized correlation coefficient is as follows:

[0036] ;

[0037] in, The disparity value represents the positional difference of matching pixels along the X-axis. and Represents the pixel coordinates of the image. This represents the inner product of two eigenvectors. and Features of the left and right images.

[0038] Furthermore, it also includes: before the comparison operation, performing a preprocessing operation on the central region of the current stacking image and the reference image, wherein the central region is a strip region with a width of W and a height of H extracted with the vertical axis of the center of the left image as the reference, the central region covers the vertical features of the stacking in the image from top to bottom, the preprocessing operation includes removing invalid points with a depth value of 0 and infinity, and removing noise points using median filtering.

[0039] Furthermore, it also includes a three-dimensional coordinate calculation step: based on the disparity map output by the deep learning model and the reprojection matrix obtained from stereo calibration, the reprojection principle is used to calculate the coordinates of each pixel in the left image. and the corresponding three-dimensional spatial coordinates The calculation formula for the reprojection principle includes:

[0040] ;

[0041] ;

[0042] ;

[0043] ;

[0044] in, For parallax, The depth value is calculated based on disparity. The focal point of the binocular vision camera. The distance between the two binocular vision cameras. and The optical center coordinates of the image. and The focal length of a binocular vision camera. and For coordinate information on the image, This represents the depth value, which is the vertical distance from the object's surface to the optical center of the camera.

[0045] Furthermore, it also includes a stack tilt detection step based on the central region:

[0046] The sum of the depth values ​​of all valid points in the central region of the current stacking image and the reference image are calculated respectively, and then the difference between the two is calculated. The alarm threshold is compared with the difference to determine whether an alarm occurs. The alarm threshold is 20% of the sum of the depth values ​​of valid points in the central region of the reference image. If an alarm occurs, an audible and visual alarm is triggered and a warning message is pushed to the management terminal.

[0047] A computer vision-based finished grain stack tilt detection device, applied to a computer vision-based finished grain stack tilt detection method, includes measuring equipment and a management terminal interconnected via a network:

[0048] The measuring device is used to acquire stacking image data and consists of a binocular vision camera and a black and white checkerboard calibration plate. The binocular vision camera includes a binocular vision depth camera and a binocular structured light depth camera for acquiring image data. The black and white checkerboard calibration plate adopts a calibration plate with a specification of 9×12 and a square side length of 30mm.

[0049] The management terminal is used to receive image data transmitted by the measuring device, store the depth information of the reference image, and determine whether it exceeds the threshold by comparison. If it does, an alarm is triggered. The management terminal has a built-in deep learning model.

[0050] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0051] 1. This invention adopts a domain adaptation technique based on color distribution transfer. By transferring the color statistical features of unlabeled images of grain warehouses to a public dataset, the model is trained using the transferred data. This overcomes the domain differences across scenes, enabling the model to maintain efficient feature extraction capabilities even under complex lighting conditions in grain warehouses. It can be deployed without manual labeling, reducing application costs.

[0052] 2. This invention introduces a residual network to extract multi-scale features of images and combines it with an adaptive aggregation module to smooth the cost surface using contextual information, making the object outlines in the output disparity map clearer, overcoming the matching defects of traditional algorithms in weak texture regions, and improving detection accuracy.

[0053] 3. This invention achieves non-contact, real-time quantitative early warning with high sensitivity. The invention proposes a depth difference method based on the central region. By calculating the cumulative depth and difference of a specific region between the reference image and the current stacking image, minute three-dimensional deformations can be reflected by quantified values. This eliminates the need for contact sensors on the grain stack, avoiding interference with the stack structure. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the steps in the finished grain stack tilt detection method of the present invention; Figure 2 This is a schematic diagram of the stack tilt detection process of the present invention; Figure 3 A schematic diagram illustrating the principle of stereo calibration for the binocular vision camera of the present invention; Figure 4 This is a schematic diagram of the image correction method of the present invention; Figure 5 This is a comparative diagram of the image correction before and after the present invention; Figure 6 This is a schematic diagram of the architecture of the stereo matching technology based on deep learning of the present invention; Figure 7 This is a schematic diagram comparing the results of the method of this invention with the SGBM stereo matching algorithm built into OpenCV; Figure 8 This is a schematic diagram of the three-dimensional point cloud data of the present invention; Figure 9 This is a schematic diagram illustrating the extraction of the central region of the reference image and the current stacking image for this invention; Figure 10 This is a schematic diagram of the structure of the finished grain stack tilt detection device of the present invention; Figure 11 This is a schematic diagram illustrating the image disparity and depth calculation of the present invention; Figure 12 This is a schematic diagram of the tilt detection and alarm of the present invention.

[0055] Figure label annotation: Measuring equipment 10, management terminal 20. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0057] Example 1

[0058] In this embodiment, as Figures 1-9 As shown, this invention proposes a method for detecting the tilt of finished grain stacks based on computer vision, specifically including the following steps:

[0059] Step S1: Use a binocular vision camera with the same parameters and configuration as the measuring device 10, and perform stereo calibration on the binocular vision camera to obtain the internal and external parameters of the binocular camera.

[0060] Specifically, through the analysis of the causes of stack collapse, it is known that judging the collapse of the stack requires detecting whether the finished grain packaging bags have shifted or deformed. Therefore, this invention proposes to use a binocular vision camera combined with deep learning technology to obtain the three-dimensional coordinate information of the stack detection surface, and to determine whether the stack has shifted or deformed. In order to ensure that the problem of finished grain stack collapse detection and early warning is effectively solved, this invention proposes to use two kinds of depth cameras to obtain three-dimensional coordinate information: a binocular vision depth camera and a binocular structured light depth camera. The binocular vision depth camera requires the object to have texture information, and its measurement accuracy is strongly related to the camera calibration and the texture of the object. The binocular structured light depth camera can obtain higher accuracy measurement results, but it is easily affected by lighting and the measurement distance is limited, making it unsuitable for outdoor operations and large-distance measurements. Therefore, this invention combines the above two depth cameras as the measurement device 10.

[0061] Step S2: Acquire left and right images of the stack at fixed points and times using a binocular vision camera, obtain disparity maps of the left and right images using deep learning-based stereo matching technology, obtain image depth information, and obtain point cloud information of the current stack image (including the physical distance between the stack and the left camera) by combining calibration parameters.

[0062] Step S3: Compare the point cloud information of the current stacking image with the point cloud information of the reference image, and determine whether the stacking is tilted based on the comparison result. The reference image is the image when the stacking is not tilted.

[0063] As shown above, this method uses a binocular vision camera combined with deep learning technology to obtain the three-dimensional coordinate information of the stacking detection surface, and determines whether the stack has shifted or deformed. In the point cloud information comparison process, to ensure the effective implementation of this method, this embodiment employs a simple and efficient comparison method, specifically including:

[0064] Select the point cloud information of the central region of the reference image (e.g., 5 pixels to the left and right of the center line of the image), calculate the average physical distance and segmented average distance of the central region (considering removing the non-stacked areas on the ground and top of the image), and save the results;

[0065] Select the point cloud information of the corresponding central region in the current stacking image and calculate the average physical distance;

[0066] The average physical distance of the current stacking image is compared with the average physical distance of the reference image. If the difference exceeds a preset threshold, an alarm is triggered, and different alarm modes are selected according to the size of the difference.

[0067] Among them, such as Figure 3 As shown, the stereo calibration in step S1 specifically includes:

[0068] Establish the geometric relationship between the pixel coordinates of the two-dimensional image and the coordinates of the three-dimensional world, and fix the binocular vision camera at a predetermined monitoring position at a distance from the front of the stack to obtain accurate three-dimensional spatial information. In this embodiment, the ZED2 binocular camera is used.

[0069] In a well-lit and uniform environment, several sets of left and right images of the stack were collected by adjusting the position and angle of the black and white checkerboard calibration board.

[0070] The acquired left and right images are input into the stereo calibration program (input management terminal 20) to calculate the intrinsic parameter matrix, distortion coefficients and extrinsic parameter matrix of the binocular vision camera, and these are used as the calibration results.

[0071] Specifically, Figure 3 This diagram illustrates the principle of stereo calibration for a binocular vision camera, using a black and white checkerboard calibration plate as the world coordinate system. The reference carrier, the points on the black and white checkerboard calibration board. These are three-dimensional points in the world coordinate system. Camera 1 (left camera) and Camera 2 (right camera) correspond to the stereo vision cameras mentioned above. Each stereo vision camera has its own camera coordinate system. , The camera's optical center is the origin of the coordinate system, the camera's imaging plane is the projection surface of the image pixels, and points in the world coordinate system... They will be projected onto the imaging planes of the left and right cameras respectively, forming pixels. , The stereo calibration operation involves acquiring multiple sets of images of the black and white checkerboard calibration board from different angles and positions, and utilizing world coordinate system points. and the imaging points of the left and right cameras , Based on the correspondence, the intrinsic parameter matrix, distortion coefficient, and extrinsic parameter matrix of the binocular vision camera are calculated.

[0072] Among them, such as Figure 4 and Figure 5 As shown, in step S2, after acquiring left and right images of the stack at fixed points and times using a binocular vision camera, a stereo correction operation is also performed on the left and right images:

[0073] Using the calibration parameters obtained from stereo calibration, distortion correction and stereo correction are performed on the original left and right images of the stack, so that the corrected left and right images satisfy the epipolar geometric constraint, that is, any pixel in the left image and its corresponding matching pixel in the right image are located on the same horizontal scan line, satisfying the condition that the epipolar lines are parallel and the rows are aligned.

[0074] Specifically, considering that the original acquired binocular vision images suffer from lens distortion and the optical axes of the left and right cameras are difficult to make perfectly parallel, direct matching is inefficient and prone to large errors. Therefore, this invention utilizes calibration parameters obtained from stereo calibration to perform distortion correction and stereo correction operations on the original acquired images. This step eliminates vertical parallax, simplifies the complex two-dimensional stereo matching search into a one-dimensional horizontal search, and lays the foundation for subsequent deep learning-based stereo matching techniques. An example of image correction is shown below. Figure 4 As shown, point The target pixel on the imaging plane of the left camera (corresponding to a spatial feature point on the stacked surface). On the imaging plane of the right camera, the point is the same as the point in the left image. The corresponding uncorrected pixel (with vertical offset). On the imaging plane of the right camera, the point is the same as the point in the left image. The corresponding corrected pixels (polar alignment). , Let these be the optical centers of the left and right cameras, each corresponding to the left camera coordinate system. Right camera coordinate system , The extrinsic parameters of the right camera relative to the left camera ( It is a rotation matrix. (It is a translation vector), before correction: left image points Compared with the right figure because (Rotational deviation) and lens distortion, with vertical offset, make direct matching impossible. After correction: using intrinsic parameters (focal length, optical center coordinates) and extrinsic parameters obtained from stereo calibration, distortion correction and epipolar correction are performed on the image, making... Adjusted to This makes the point in the left figure Compared with the right figure Projections along the Z-axis (depth direction) are on the same horizontal line, improving the efficiency and accuracy of stereo matching.

[0075] In step S2, the deep learning-based stereo matching technique outputs a disparity map through a deep learning model. The stereo matching architecture is as follows: Figure 6 As shown, it specifically includes:

[0076] (1) A domain adaptation technique based on color distribution transfer is adopted. This technique addresses the problem that the lack of ground truth labels in grain warehouse stacking images makes it impossible to directly train deep neural networks. The color statistical features of unlabeled stacking images in the grain warehouse are transferred to a public dataset with ground truth labels. Specifically, the public dataset images (source domain images) and unlabeled stacking images (target domain images) are converted from RGB space to CIE Lab color space to reduce the correlation between channels. The global mean and variance of the stacking image set are calculated using formulas, and the color distribution of the public dataset images is adjusted to match the lighting and texture features of the grain warehouse environment. Then, the transformed images are converted back to RGB format. The model is trained using the public dataset images (source domain images) after the color distribution transfer, so that the model can adapt to the complex lighting and texture environment of the grain warehouse, effectively improving the generalization ability of the model, effectively overcoming cross-scene domain differences, and can be deployed without manual annotation, greatly reducing the application cost. The formulas involved in calculating the global mean and variance of the stacking image set and adjusting the color distribution of the public dataset images include:

[0077] ;

[0078] ;

[0079] ;

[0080] in, and Let represent the global mean and variance of the stacked image set, respectively. and The mean and variance of the stacked images in the current batch. and This represents the mean and variance of the public dataset in the current batch;

[0081] (2) Input the stereo-corrected left and right images into a pre-trained deep neural network. The deep neural network uses a ResNet as the backbone network (Encoder) to extract multi-scale features of the image. By introducing a Shortcut Connection, the ResNet effectively solves the gradient vanishing problem of deep networks and enhances the feature representation ability of weak and repetitive texture areas on the surface of the grain bag. After multi-scale feature extraction, the matching cost of the left and right feature maps is calculated using the Normalized Correlation Coefficient (NCC) to construct a multi-scale matching cost volume. The Normalized Correlation Coefficient (NCC) can effectively resist radiation differences and improve the robustness of matching. The formula for calculating the matching cost based on the Normalized Correlation Coefficient is as follows:

[0082] ;

[0083] in, The disparity value represents the positional difference of matching pixels along the X-axis. and Represents the pixel coordinates of the image. This represents the inner product of two eigenvectors. and Features of the left and right images;

[0084] (3) In order to smooth the cost surface and eliminate local noise, the present invention uses stacked adaptive aggregation modules (AAModules) to aggregate matching cost volumes at different scales (1 / 3, 1 / 6, and 1 / 12 resolution of the original image). The Soft Argmax function is used to regress the aggregated cost and output the disparity map, which avoids the discretization error caused by traditional Argmax and improves the fineness of depth estimation. The adaptive aggregation module adopts a cost aggregation strategy that combines intra-scale aggregation and cross-scale aggregation. It uses deformable convolution to adapt to the geometry of the object edge and effectively solves the matching problem of grain bag edge and occluded area through the interaction and fusion of multi-scale information.

[0085] The cost aggregation strategy can be expressed by the following formula: ,in, Indicated in pixels Point parallax is The cost of aggregation The number of sampling points. For the first Aggregate weights of points, For pixels The fixed offset of the point For additional offset, For specific location weights, The matching cost at a certain scale and satisfying D, H, and W represent the current maximum disparity value, height, and width, respectively. The model achieves adaptive cost aggregation by learning additional offsets and specific location weights.

[0086] Among them, such as Figure 9As shown, it also includes: before the comparison operation, performing a preprocessing operation on the central region of the current stacking image and the reference image. The central region is a strip region with a width of W and a height of H extracted with the vertical axis of the left image center as the reference. W can be 5% to 10% of the image width, and H can be the image height. The central region covers the vertical features of the stacking in the image from top to bottom. The preprocessing operation includes removing invalid points with a depth value of 0 and infinity, and removing noise points using median filtering.

[0087] In this embodiment, Figure 5 The image shows examples of the original acquired image before and after image correction. This image shows the effect of distortion correction and stereo correction of the original acquired image using the calibration parameters of the present invention. As can be seen from the image, in the corrected left and right views, the corresponding feature points are strictly distributed on the same horizontal scan line, which satisfies the epipolar geometry constraint, proving the effectiveness of the present invention.

[0088] Figure 7 This diagram illustrates the comparison between the method of this invention and the SGBM stereo matching algorithm built into OpenCV. The left side of the diagram shows the disparity map generated by this invention based on a deep learning model, while the right side shows the disparity map generated by the traditional SGBM algorithm. As can be seen from the comparison, in areas with repetitive and weak textures on the surface of the grain bag, the SGBM algorithm exhibits large areas of matching holes and noise. In contrast, the disparity map generated by the method of this invention is smooth, complete, and has clear object outlines, demonstrating that this invention has significantly superior matching accuracy and robustness in the specific environment of a grain warehouse.

[0089] Figure 8 This is a schematic diagram of three-dimensional point cloud data display. The diagram shows the point cloud effect after the management terminal 20 reprojects the disparity map into three-dimensional space by combining it with the camera Q matrix. The present invention can accurately recover the spatial three-dimensional shape of the stack, and the Z-axis depth information is continuous and accurate, providing reliable three-dimensional data support for subsequent deformation analysis.

[0090] Figure 9 This diagram illustrates the extraction of the central region (red area in the middle) in the reference image and the current stacking image. It visually demonstrates the detection logic of the central region depth difference method proposed in this invention. The vertical strip region at the center of the stack is locked as the key feature area. By comparing the cumulative depth change of this region in the reference image and the current stacking image, background interference can be effectively eliminated and the displacement trend of the stacking body can be sensitively captured.

[0091] Example 2

[0092] In this embodiment, as Figures 1-9 As shown, the present invention also includes a three-dimensional coordinate calculation step: based on the disparity map output by the deep learning model and the reprojection matrix obtained by stereo calibration, the reprojection principle is used to calculate the coordinates of each pixel in the left image. and the corresponding three-dimensional spatial coordinates The calculation formulas for the reprojection principle include:

[0093] ;

[0094] ;

[0095] ;

[0096] ;

[0097] in, For parallax, The depth value is calculated based on disparity. The focal point of the binocular vision camera. The distance between the two binocular vision cameras. and The optical center coordinates of the image. and The focal length of a binocular vision camera. and For coordinate information on the image, This represents the depth value, which is the vertical distance from the object's surface to the optical center of the camera.

[0098] Among them, such as Figure 9 As shown, considering the whole Figure 3 3D point cloud processing involves large computational loads and is easily affected by background noise. Therefore, this invention proposes a stacking tilt detection method based on the depth difference method of the central region. The stacking tilt detection steps based on the central region specifically include:

[0099] In this embodiment, assuming the stack is located in the center of the image during capture, a long strip region with width W and height H is extracted based on the vertical axis of the left image center. This region covers the key vertical features of the stack from top to bottom. Then, preprocessing operations are performed on the central region of the current stack image and the reference image to remove invalid points with depth values ​​of 0 and infinity, and median filtering is used to remove noise. Finally, the sum of the depth values ​​of all valid points in the central region of the current stack image and the reference image is calculated respectively. , Then calculate the difference between the two. , This is the sum of depth values ​​within the central region of the reference image. The depth values ​​in the central region of the current stacking image are accumulated. An alarm is determined by comparing the difference with the alarm threshold. The alarm threshold is 20% of the sum of the effective point depth values ​​in the central region of the reference image. The alarm threshold can be adjusted according to the stacking type and the on-site environment. According to the experiment in the table below, when the stacking is tilted or displaced in a visible manner, the accumulated depth value in the central region will change significantly. Therefore, if an alarm occurs, an audible and visual alarm is triggered and a warning message is pushed to the management terminal 20.

[0100] Specifically, to verify the effectiveness of this method, field tests were conducted at a grain depot in Zhongshan City, covering various forms such as five-star stacking and pallet stacking. Taking the five-star stacked ton bag experiment as an example, images of the stack before tilting were selected as references. Multiple frames of images were collected over time during the tilting process. The depth accumulation difference values ​​calculated using this method are shown in Table 1 below:

[0101] Table 1 - Comparison of Experimental Data for Five-Star Stack Tilting Detection

[0102] Image status Description of tilt Cumulative depth difference (mm) Judgment Result Reference image The stack is in an initial stable state 0 normal Tilt image 1 In the early stages, tilting is visible to the naked eye. 1216.73 Alarm Tilt image 2 The degree of tilting has increased 1236.90 Alarm Tilt image 3 Severely tilted (close to collapse) 1495.14 Alarm Tilt image 4 Severely tilted (at different angles) 1448.34 Alarm

[0103] As shown in the table above, when the stack is in a stable state, the difference value is 0. When the stack tilts visibly, the depth difference value calculated by this invention rapidly rises to over 1200mm. Experiments show that this numerical index can sensitively and quantitatively reflect the tilting process of the stack. By setting a reasonable alarm threshold, this method can accurately identify abnormal tilting and issue an alarm, effectively reducing the risk of collapse caused by missed detection during manual inspection.

[0104] Example 3

[0105] In this embodiment, as Figures 10-12 As shown, the present invention also provides a computer vision-based finished grain stack tilt detection device, applied to the above-mentioned finished grain stack tilt detection method, including a measuring device 10 and a management terminal 20 interconnected via a network:

[0106] The measuring device 10 is used to acquire stacking image data. It consists of a binocular vision camera and a black and white checkerboard calibration plate. The binocular vision camera includes a binocular vision depth camera and a binocular structured light depth camera for acquiring image data. The black and white checkerboard calibration plate adopts a calibration plate with a specification of 9×12 and a square side length of 30mm.

[0107] The management terminal 20 is used to receive image data transmitted by the measuring device 10, store the depth information of the reference image, and determine whether the threshold is exceeded by comparison. If the threshold is exceeded, an alarm is triggered. The management terminal 20 has a built-in deep learning model. The management terminal 20 also includes a stereo correction module, a stereo matching module, a three-dimensional coordinate calculation module, a tilt judgment module, and an alarm management module. The modules cooperate with each other to achieve the above judgment function.

[0108] The management terminal 20 also includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it implements the above-mentioned finished grain stack tilt detection method and steps. A computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by the processor, it implements the above-mentioned finished grain stack tilt detection method.

[0109] In this embodiment, Figure 11 This diagram illustrates the current image disparity and depth calculation, showcasing the real-time processing capability of the detection device. The management terminal 20 can calculate and display the disparity map of the current monitoring screen online, and the depth distance of any pixel can be read in real time through mouse interaction. This demonstrates that the method of the present invention has a fast calculation speed and meets the real-time requirements of online monitoring of grain depots.

[0110] Figure 12 This diagram illustrates tilt detection and alarm. It shows that when the stack shifts and the depth difference exceeds a preset threshold, an alarm is automatically triggered. The interface prominently displays the detected tilt and alarm prompts, verifying the practicality and implementation capability of this invention as a safety early warning device.

[0111] In summary, this invention can overcome cross-scene domain differences, enabling the model to maintain efficient feature extraction capabilities even under complex lighting conditions in grain silos. It also overcomes the matching defects of traditional algorithms in weak texture areas, and can reflect minute three-dimensional deformations with quantified values. This eliminates the need to install contact sensors on the grain stacks, avoiding interference with the grain stack structure, and has good application prospects.

[0112] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for detecting the tilt of finished grain stacks based on computer vision, characterized in that, Specifically, the following steps are included: Step S1: Use a binocular vision camera with the same parameters and configuration as the measurement device, and perform stereo calibration on the binocular vision camera to obtain the internal and external parameters of the binocular camera. Step S2: Acquire left and right images of the stack at fixed points and time intervals using the binocular vision camera, obtain the disparity map of the left and right images using deep learning-based stereo matching technology, obtain image depth information, and obtain the point cloud information of the current stack image by combining calibration parameters. Step S3: Compare the point cloud information of the current stacking image with the point cloud information of the reference image, and determine whether the stacking is tilted based on the comparison result. The reference image is the image when the stacking is not tilted. The comparison operation of the point cloud information specifically includes: Select the point cloud information of the central region of the reference image, calculate the average physical distance and segmented average distance of the central region, and save the results; Select the point cloud information of the corresponding central region in the current stacking image and calculate the average physical distance; The average physical distance of the current stacking image is compared with the average physical distance of the reference image. If the difference exceeds a preset threshold, an alarm is triggered, and different alarm modes are selected according to the size of the difference.

2. The method for detecting the tilt of finished grain stacks based on computer vision according to claim 1, characterized in that, The stereo calibration in step S1 specifically includes: Establish the geometric relationship between the pixel coordinates of the two-dimensional image and the coordinates of the three-dimensional world, and fix the binocular vision camera at a predetermined monitoring position at a distance from the front of the stack; In a well-lit and uniform environment, several sets of left and right images of the stack were collected by adjusting the position and angle of the black and white checkerboard calibration board. The acquired left and right images are input into the stereo calibration program to calculate the intrinsic parameter matrix, distortion coefficients and extrinsic parameter matrix of the binocular vision camera, which are then used as the calibration results. The reprojection matrix is ​​derived from the calibration results. The disparity map is then mapped to three-dimensional coordinate information using the reprojection matrix. The intrinsic matrix, distortion coefficients, extrinsic matrix, and reprojection matrix are used as calibration parameters obtained from stereo calibration.

3. The method for detecting the tilt of finished grain stacks based on computer vision according to claim 2, characterized in that, In step S2, after acquiring left and right images of the stacked structure at fixed points and times using a binocular vision camera, a stereoscopic correction operation is performed on the left and right images: Using the calibration parameters obtained from stereo calibration, distortion correction and stereo correction are performed on the original left and right images of the stack, so that the corrected left and right images satisfy the epipolar geometric constraint, that is, any pixel in the left image and its corresponding matching pixel in the right image are located on the same horizontal scan line, satisfying the condition that the epipolar lines are parallel and the rows are aligned.

4. The method for detecting the tilt of finished grain stacks based on computer vision according to claim 3, characterized in that, In step S2, the deep learning-based stereo matching technique outputs a disparity map through a deep learning model, specifically including: A domain adaptation technique based on color distribution transfer is used to transfer the color statistical features of unlabeled stacked images from a grain warehouse to a public dataset with real parallax labels. Specifically, the public dataset images and unlabeled stacked images are converted from RGB space to CIE Lab color space. The global mean and variance of the stacked image set are calculated using formulas. The color distribution of the public dataset images is adjusted to match the lighting and texture features of the grain warehouse environment. The transformed images are then converted back to RGB format. The stereo-corrected left and right images are input into a pre-trained deep neural network. The deep neural network uses a residual network as the backbone network to extract multi-scale features of the image. After multi-scale feature extraction, the normalized correlation coefficient is used to calculate the matching cost of the left and right feature maps and construct a multi-scale matching cost volume. A stacked adaptive aggregation module is used to aggregate matching cost volumes at different scales. The Soft Argmax function is used to regress the aggregated cost and output a disparity map. The adaptive aggregation module adopts a cost aggregation strategy that combines intra-scale aggregation and cross-scale aggregation. The cost aggregation strategy is expressed by the following formula: ,in, Indicated in pixels Point parallax is The cost of aggregation The number of sampling points. For the first Aggregate weights of points, For pixels The fixed offset of the point For additional offset, For specific location weights, The matching cost at a certain scale and satisfying D, H, and W represent the current maximum parallax value, height, and width, respectively.

5. The method for detecting the tilt of finished grain stacks based on computer vision according to claim 4, characterized in that, The formulas involved in calculating the global mean and variance of the stacked image set and adjusting the color distribution of the publicly available dataset images specifically include: ； ； ； in, and Let represent the global mean and variance of the stacked image set, respectively. and The mean and variance of the stacked images in the current batch. and This represents the mean and variance of the public dataset in the current batch.

6. The method for detecting the tilt of finished grain stacks based on computer vision according to claim 4, characterized in that, The formula for calculating the matching cost based on the normalized correlation coefficient is as follows: ; in, The disparity value represents the positional difference of matching pixels along the X-axis. and Represents the pixel coordinates of the image. This represents the inner product of two eigenvectors. and Features of the left and right images.

7. The method for detecting the tilt of finished grain stacks based on computer vision according to claim 1, characterized in that, Also includes: Before the comparison operation, the central region of the current stacking image and the reference image is preprocessed. The central region is a strip region with a width of W and a height of H extracted with the vertical axis of the left image center as the reference. The central region covers the vertical features of the stacking in the image from top to bottom. The preprocessing operation includes removing invalid points with a depth value of 0 and infinity, and removing noise points using median filtering.

8. The method for detecting the tilt of finished grain stacks based on computer vision according to claim 4, characterized in that, It also includes a 3D coordinate calculation step: based on the disparity map output by the deep learning model and the reprojection matrix obtained from stereo calibration, the reprojection principle is used to calculate the coordinates of each pixel in the left image. and the corresponding three-dimensional spatial coordinates The calculation formula for the reprojection principle include: ； ； ； ； in, For parallax, The depth value is calculated based on disparity. The focal point of the binocular vision camera. The distance between the two binocular vision cameras. and The optical center coordinates of the image. and The focal length of a binocular vision camera. and For coordinate information on the image, This represents the depth value, which is the vertical distance from the object's surface to the optical center of the camera.

9. A method for detecting the tilt of finished grain stacks based on computer vision according to claim 8, characterized in that, It also includes a stack tilt detection step based on the central region: The sum of the depth values ​​of all valid points in the central region of the current stacking image and the reference image are calculated respectively, and then the difference between the two is calculated. The alarm threshold is compared with the difference to determine whether an alarm occurs. The alarm threshold is 20% of the sum of the depth values ​​of valid points in the central region of the reference image. If an alarm occurs, an audible and visual alarm is triggered and a warning message is pushed to the management terminal.

10. A computer vision-based finished grain stack tilt detection device, applied to the computer vision-based finished grain stack tilt detection method according to any one of claims 1-9, characterized in that, This includes measurement devices and management terminals that are interconnected via a network: The measuring device is used to acquire stacking image data and consists of a binocular vision camera and a black and white checkerboard calibration plate. The binocular vision camera includes a binocular vision depth camera and a binocular structured light depth camera for acquiring image data. The black and white checkerboard calibration plate adopts a calibration plate with a specification of 9×12 and a square side length of 30mm. The management terminal is used to receive image data transmitted by the measuring device, store the depth information of the reference image, and determine whether it exceeds the threshold by comparison. If it does, an alarm is triggered. The management terminal has a built-in deep learning model.

Images