A Computer Vision-Based Method and System for Safety Monitoring of Cargo Stacking in Freight Yards

Abstract

This invention discloses a computer vision-based method and system for safety monitoring of cargo stacking in freight yards, comprising: acquiring cargo image data; acquiring semantic features of the cargo based on the cargo image data; acquiring three-dimensional pose parameters of the cargo based on the semantic features; adjusting the weights of different cargo regions based on the three-dimensional pose parameters to acquire pose information of occluded cargo; and analyzing the spatial relationships and forces between cargo based on the pose information of the occluded cargo to predict dangerous situations. This invention improves the accuracy, robustness, and real-time performance of cargo pose recognition in complex freight yard environments, realizing intelligent monitoring and early warning of cargo stacking.

Description

Technical Field

[0001] This invention belongs to the field of information monitoring technology, and in particular relates to a method and system for safety monitoring of cargo stacking in freight yards based on computer vision. Background Technology

[0002] When goods are stacked in a freight yard, their posture directly affects the safety and stability of the stacking. Traditional cargo stacking safety monitoring mainly relies on manual inspections, which suffers from low monitoring efficiency, limited monitoring frequency, and susceptibility to subjective factors. Cargo posture recognition and monitoring technology based on computer vision offers a new approach to solving these problems, but it still faces many technical challenges in practical applications.

[0003] First, the freight yard environment is complex and variable, with unstable lighting conditions and a wide variety of goods, differing greatly in appearance, size, material, and packaging, posing challenges to image acquisition and processing. Second, goods undergo diverse posture changes during stacking, including translation, rotation, tilting, and deformation, requiring visual algorithms to accurately capture and quantify these posture changes. Third, goods are typically stacked at high density, with occlusion and interference between different goods, increasing the difficulty of posture recognition. Furthermore, freight stacking safety monitoring requires real-time performance, demanding that visual algorithms quickly process image data and provide judgment results, placing high demands on the algorithm's timeliness. Finally, in practical applications, it is necessary to fuse visual algorithms with data from other sensors and integrate them with the freight yard management system to build a complete freight stacking safety monitoring system, which presents challenges to the system's robustness, scalability, and ease of use. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a computer vision-based method and system for safe monitoring of cargo stacking in freight yards. This method can improve the accuracy, robustness, and real-time performance of cargo posture recognition in complex freight yard environments, and realize intelligent monitoring and early warning of cargo stacking.

[0005] This invention provides a computer vision-based method for safety monitoring of cargo stacking in freight yards, including:

[0006] Acquire cargo image data;

[0007] Based on the cargo image data, the semantic features of the cargo are obtained;

[0008] Based on the semantic features, the three-dimensional pose parameters of the cargo are obtained;

[0009] Based on the three-dimensional pose parameters, the weights of different cargo areas are adjusted to obtain the pose information of the occluded cargo.

[0010] Based on the posture information of the obscured goods, the spatial relationship and force conditions between the goods are analyzed to predict dangerous situations.

[0011] Optionally, based on the cargo image data, obtaining the semantic features of the cargo includes:

[0012] The cargo image data is preprocessed to obtain high-quality cargo images;

[0013] Based on the high-quality cargo image, obtain the semantic features of the cargo.

[0014] Optionally, preprocessing the cargo image data to obtain high-quality cargo images includes:

[0015] An adaptive algorithm is used to dynamically adjust the contrast, brightness, and saturation of the cargo image data to obtain an adjusted cargo image;

[0016] The adjusted cargo image is evaluated using a convolutional neural network model to determine whether the image quality meets the standards. Based on the determination result, the high-quality cargo image is obtained.

[0017] Optionally, obtaining the semantic features of the goods based on the high-quality goods image includes:

[0018] Extract the color features, texture features, shape features, and size features of the high-quality cargo image to obtain a multi-dimensional feature vector;

[0019] A convolutional neural network model is used to perform deep learning and extraction on the multidimensional feature vectors to obtain the semantic features of the goods.

[0020] Optionally, based on the semantic features, obtaining the three-dimensional pose parameters of the cargo includes:

[0021] Based on the semantic features, a three-dimensional model of the cargo is established, and key points on the cargo are predefined, including corner points and edge points.

[0022] Based on predefined key points, key points are detected in cargo images, and the pixel coordinates of the key points are obtained.

[0023] The pixel coordinates are converted into three-dimensional coordinates, and the spatial distance and angular relationship between key points are calculated based on the three-dimensional coordinates to obtain the spatial relationship matrix between key points.

[0024] Based on the spatial relationship matrix, the three-dimensional attitude parameters of the cargo are estimated.

[0025] Optionally, based on the three-dimensional pose parameters, adjusting the weights of different cargo regions to obtain the pose information of the occluded cargo includes:

[0026] Based on the three-dimensional pose parameters, the attention module adaptively adjusts different acquisition areas in the image to acquire the target cargo area;

[0027] Based on the target cargo area, construct a spatial topology diagram of the cargo relationships;

[0028] A convolutional neural network model is used to extract features from the weighted cargo image to obtain a visual feature representation of the cargo;

[0029] The spatial topology graph and visual features are fused together, and the pose information of the occluded goods is inferred through graph convolutional networks.

[0030] Optionally, based on the posture information of the obscured goods, combined with the three-dimensional posture parameters of the goods, the spatial relationship and force situation between the goods are analyzed, and dangerous situations are predicted, including:

[0031] Based on the posture information of the obscured goods, combined with the three-dimensional posture parameters of the goods and the spatial relationship between the goods, a physical simulation engine is used to simulate the stress on the goods and evaluate the stability of the stacked goods.

[0032] This invention also provides a computer vision-based cargo stacking safety monitoring system in a freight yard, comprising: an image acquisition module, a feature extraction module, a pose estimation module, and a 3D analysis module.

[0033] The image acquisition module is used to acquire images of the cargo;

[0034] The feature extraction module is used to extract semantic features from the cargo image;

[0035] The attitude estimation module is used to estimate the three-dimensional attitude of the cargo;

[0036] The three-dimensional analysis module is used to analyze the spatial relationships and stress conditions between goods and predict dangerous situations.

[0037] Compared with the prior art, the present invention has the following advantages and technical effects:

[0038] This invention employs an adaptive image enhancement algorithm to preprocess cargo images, improving image quality; it constructs a multi-level cargo feature extraction model, comprehensively utilizing features such as color, texture, and shape, and extracts deep semantic features of the cargo through a convolutional neural network; it introduces an attention mechanism and contextual information to highlight target cargo, suppress background interference, and infer the posture of occluded cargo; it uses a lightweight posture recognition model to improve model inference speed; and it combines depth information of the cargo yard environment to construct a 3D scene model of cargo stacking, analyze the spatial relationships and stress conditions of the cargo, assess stacking stability and safety, and predict potential hazards such as tipping over and collapse. This invention improves the accuracy, robustness, and real-time performance of cargo posture recognition in complex cargo yard environments, realizing intelligent monitoring and early warning of cargo stacking. Attached Figure Description

[0039] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0040] Figure 1 This is a flowchart of a computer vision-based safety monitoring method for cargo stacking in a freight yard, according to an embodiment of the present invention.

[0041] Figure 2 This is a flowchart of the data preprocessing process according to an embodiment of the present invention;

[0042] Figure 3 This is a flowchart illustrating the process of obtaining the semantic features of goods according to an embodiment of the present invention;

[0043] Figure 4 This is a flowchart illustrating the acquisition of the three-dimensional attitude parameters of cargo according to an embodiment of the present invention;

[0044] Figure 5 This is a flowchart of an embodiment of the present invention for obtaining the posture information of an obscured cargo;

[0045] Figure 6 This is a structural diagram of the attention module according to an embodiment of the present invention;

[0046] Figure 7 This is a structural diagram of a computer vision-based cargo stacking safety monitoring system in a freight yard, according to an embodiment of the present invention. Detailed Implementation

[0047] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0048] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0049] The following explains the technical terms used in the embodiments:

[0050] The novel convolutional neural network model is based on a convolutional neural network, uses an improved threshold function as the activation function tReLU, introduces residual neurons in the middle layers of the convolutional neural network, connects them alternately using convolution and pooling, and performs softmax classification throughout the entire structure to generate the novel convolutional neural network model RLCNN; the activation function tReLU is shown in formula (1):

[0051]

[0052] The residual neuron is represented by formula (2):

[0053] F(x)=W2f(W1x+b)+b (2)

[0054] In formula (2), x represents the input of the current layer, F(x) represents the input of the next layer, W1 and W2 represent the weights of the current layer and the next layer, respectively, f(.) represents the tReLU activation function, and b represents the bias.

[0055] The novel convolutional neural network model is a six-layer convolutional neural network. The kernel sizes of the first, second, fifth, and sixth layers are 5×5, 3×3, 3×3, and 3×3, respectively, with 32, 64, 128, and 256 kernels. Each layer uses a max-pooling layer with a size of 2×2. The third and fourth layers are residual neuron layers built using residual neurons. The kernel sizes of the residual neurons in the third and fourth layers are 1×1, 3×3, and 1×1, respectively, with 16, 16, and 64 kernels. The structural parameters of the six-layer convolutional neural network are shown in Table 1.

[0056] The input to the novel convolutional neural network model is the pixel intensity matrix of a grayscale image. The pixel intensity matrix of the grayscale image is obtained through two-dimensional image processing. Specifically, the two-dimensional image is preprocessed into a vibration signal dataset, and then the data from the vibration signal dataset is converted into the pixel intensity matrix of the grayscale image using formula (3).

[0057]

[0058] Where M represents a vibration data sequence of length M, j = 1…M, k = 1…M, represents the pixel intensity of the image, and round(·) represents normalizing the pixel values ​​to 0-255.

[0059] Pose estimation algorithms are algorithms that estimate the pose of an object or human body using sensor data or image data as input. They have wide applications in computer vision, robotics, virtual reality, and other fields. Common pose estimation algorithms include: 1. Sensor-based pose estimation algorithms: These acquire pose information of an object or human body using sensors such as accelerometers, gyroscopes, and magnetometers, and then estimate the pose using algorithms such as filtering and integration. 2. Image-based pose estimation algorithms: These acquire images of an object or human body using cameras, and then estimate the pose using algorithms such as feature point matching and model fitting. 3. Deep learning-based pose estimation algorithms: These train images of an object or human body using deep learning models, and then predict the pose information using the model. 4. Sensor and image fusion pose estimation algorithms: These fuse sensor and image data and estimate the pose using algorithms such as Kalman filtering and extended Kalman filtering. Pose estimation algorithms have a wide range of applications. For example, in the field of robotics, they can be used for autonomous navigation and operation of robots; in the field of virtual reality, they can be used for tracking the hand and head postures of users; and in the medical field, they can be used to monitor and evaluate the postures of patients.

[0060] The attention module based on the convolutional neural network includes an attention vector generation unit, which is configured to feed the feature vector input from the residual module to a first branch and a second branch; wherein, the first branch is configured to perform deformable convolution, channel decay and global pooling operations on the feature vector in the horizontal direction, and the second branch is configured to perform deformable convolution, channel decay and global pooling operations on the feature vector in the vertical direction.

[0061] The attention vector generation unit is also configured to concatenate the output of the first branch with the output of the second branch to obtain a concatenated vector, and then transform the concatenated vector using a convolution transformation function. The transformed concatenated vector is then fed into a fully connected layer, and the input of the fully connected layer is convolved in both the horizontal and vertical directions to obtain the attention vector of the input feature vector in the horizontal direction and the attention vector of the input feature vector in the vertical direction.

[0062] The attention module involved in this embodiment uses deformable convolution for feature extraction in both the horizontal and vertical directions, making it easier to capture the positional information of objects during subsequent encoding. Specifically, the first branch performs deformable convolution and channel attenuation operations on the feature vector in the horizontal direction to obtain a feature map in the horizontal direction. The second branch performs deformable convolution and channel attenuation operations on the feature vector in the vertical direction to obtain a feature map in the vertical direction. That is, deformable convolution is performed in only one direction on each branch. Compared with the traditional deformable convolution method that performs it in two directions simultaneously, this embodiment can enhance the feature extraction effect and improve the detection and recognition accuracy of the convolutional neural network. Furthermore, this embodiment employs a dual-branch design, pooling the horizontal and vertical feature maps separately. The attention module can capture long-range dependencies along one spatial direction while retaining precise positional information along another, thus preserving information in both the vertical and horizontal directions. After a series of transformations, the attention vector is obtained and used as a weight factor to multiply back the original feature vector. In this way, spatial attention and channel attention can be fused, solving the problem of unifying spatial and channel operations in existing attention mechanisms and improving the accuracy of convolutional neural networks.

[0063] The attention module also includes a weight allocation unit, which is configured to allocate weights to the input feature vector based on the attention vector in the horizontal direction and the attention vector in the vertical direction, so as to obtain a weighted feature vector.

[0064] This embodiment proposes a computer vision-based method for safety monitoring of cargo stacking in freight yards, such as... Figure 1 As shown, the specific steps include:

[0065] Acquire cargo image data;

[0066] Based on cargo image data, semantic features of the cargo are obtained;

[0067] Based on semantic features, obtain the three-dimensional pose parameters of the cargo;

[0068] Based on the three-dimensional pose parameters, the weights of different cargo regions are adjusted to obtain the pose information of the occluded cargo.

[0069] Based on the posture information of the obscured goods, the spatial relationship and force conditions between the goods are analyzed to predict dangerous situations.

[0070] Specifically, this embodiment employs an adaptive image enhancement algorithm to preprocess cargo images, improving image quality; it constructs a multi-level cargo feature extraction model, comprehensively utilizing features such as color, texture, and shape, and extracts deep semantic features of cargo through a convolutional neural network; it introduces an attention mechanism and contextual information to highlight target cargo, suppress background interference, and infer the posture of occluded cargo; it adopts a lightweight posture recognition model to improve model inference speed; and it combines the depth information of the cargo yard environment to construct a three-dimensional scene model of cargo stacking, analyze the spatial relationships and stress conditions of cargo, assess stacking stability and safety, and predict potential dangerous situations such as tipping over and collapse.

[0071] Furthermore, based on cargo image data, the semantic features of the cargo are obtained, including:

[0072] Preprocess the cargo image data to obtain high-quality cargo images;

[0073] Semantic features of goods are obtained from high-quality cargo images.

[0074] Specifically, the algorithm acquires cargo images collected in the cargo yard environment; determines whether the image quality meets a preset threshold; if not, it executes an adaptive image enhancement algorithm; dynamically adjusts the image contrast based on the brightness distribution of the cargo image to improve the image's sense of depth; adaptively adjusts the overall brightness of the image based on the average brightness value of the cargo image to make the image brightness moderate; dynamically optimizes the image saturation based on the color distribution of the cargo image to make the colors more vivid; evaluates the enhanced cargo image through a convolutional neural network to determine whether the image quality meets the standards; if the image quality meets the standards, it outputs a high-quality cargo image for subsequent cargo recognition and classification tasks.

[0075] Furthermore, such as Figure 2 As shown, preprocessing the cargo image data to obtain high-quality cargo images includes:

[0076] An adaptive algorithm is used to dynamically adjust the contrast, brightness, and saturation of cargo image data to obtain the adjusted cargo image;

[0077] The adjusted cargo image is evaluated using a convolutional neural network model to determine whether the image quality meets the standards. Based on the judgment result, a high-quality cargo image is obtained.

[0078] Specifically, adaptive image enhancement algorithms refer to enhancement algorithms that adjust parameters based on the characteristics of the image itself. For example, for images with insufficient lighting, histogram equalization-based algorithms can be used to redistribute pixel values, thereby improving image contrast and brightness. If the image is blurry, algorithms based on desharpening masks can be used to enhance image edges and details. Adaptive parameter adjustment can better adapt to image quality issues in different scenarios. Taking histogram equalization as an example, the mapping relationship of pixel values ​​can be adaptively adjusted according to the histogram distribution of the image, thus avoiding the tedious process of manually adjusting parameters and improving the enhancement effect. Based on the brightness distribution of a cargo image, the image contrast can be dynamically adjusted to improve the image's sense of depth. For example, if the brightness distribution of a cargo image is relatively concentrated, it indicates low image contrast and a lack of clear depth. Contrast can be improved by stretching the image's brightness histogram. Specifically, the darker parts of the brightness histogram are darkened, and the brighter parts are brightened, thereby expanding the dynamic range of brightness values. This makes the details in the dark areas of the image clearer and the details in the bright areas more prominent, thus improving the image's sense of depth. For example, stretching pixel values ​​originally between 50-150 brightness to the range of 0-255 improves image contrast and makes image layers more distinct. Based on the average brightness value of the image, the overall image brightness is adaptively adjusted to achieve a moderate level of brightness. For instance, if the average brightness value is low, indicating an overall dark image, increasing the overall brightness can improve the visual effect. Assuming the average brightness value is 50, increasing the brightness value of each pixel by 30 raises the average brightness value to 80, achieving a moderate brightness. Conversely, if the average brightness value is too high, the overall image brightness can be reduced. Adaptive brightness adjustment avoids images that are too bright or too dark, ensuring optimal visual quality. Based on the color distribution of the image, image saturation is dynamically optimized to make colors more vivid. For example, if the image has a relatively uniform color distribution and low saturation, increasing saturation can make the colors more vivid. For instance, for an image of pale fruit, increasing saturation can make the fruit's color more vibrant and closer to its true color. Conversely, if the image colors are too vibrant, decreasing saturation can make the colors softer. Dynamically optimizing saturation can make image colors more natural and realistic. A convolutional neural network (CNN) is used to evaluate the enhanced cargo image to determine if its quality meets the standards. A CNN model is trained to score the quality of the cargo image. The enhanced image is input into the trained CNN to obtain an image quality score. For example, images with a score higher than 9 are considered high-quality images, and images with a score lower than 6 are considered low-quality images. This method can effectively evaluate image quality, and the scoring threshold can be adjusted according to actual needs. If the image quality meets the standards, a high-quality cargo image is output for subsequent cargo recognition and classification tasks.For example, enhanced cargo images with a quality score of 95 will be output. These high-quality images will be used to train cargo recognition and classification models, or for cargo recognition and classification in real-world scenarios. High-quality images can improve the accuracy of recognition and classification, thereby improving the efficiency of cargo yard management.

[0079] Furthermore, such as Figure 3 As shown, based on high-quality cargo images, the semantic features of the cargo are obtained, including:

[0080] Extract color, texture, shape, and size features from high-quality cargo images to obtain multidimensional feature vectors;

[0081] A convolutional neural network model is used to perform deep learning and extraction of multi-dimensional feature vectors to obtain the semantic features of goods.

[0082] Specifically, the color, texture, shape, and size features of the goods are extracted to obtain a multi-dimensional feature vector. Based on this multi-dimensional feature vector, a convolutional neural network model is used to perform deep learning and extraction of the goods' features, resulting in a deep semantic feature representation of the goods.

[0083] These multidimensional features collectively constitute the feature vector of the goods, used for subsequent classification and recognition. The extracted multidimensional feature vector is input into a Convolutional Neural Network (CNN) for deep learning and feature extraction. CNNs can automatically learn local and global features of an image through multiple convolution and pooling operations. For example, the first convolutional layer of a CNN can learn the edge information of the goods, the second convolutional layer can learn the texture information of the goods, and deeper convolutional layers can learn the shape and structure information of the goods. Assuming that a 1024-dimensional feature vector is obtained after extraction by the CNN, this vector represents the deep semantic feature representation of the goods, which is more expressive and discriminative than manually designed features. Using the deep semantic features extracted by the CNN, a Support Vector Machine (SVM) classification model is constructed.

[0084] Furthermore, such as Figure 4 As shown, based on semantic features, the three-dimensional pose parameters of the cargo are obtained as follows:

[0085] Based on semantic features, a 3D model of the cargo is established, and key points on the cargo are predefined, including corner points and edge points.

[0086] Based on predefined key points, key points are detected in cargo images, and the pixel coordinates of the key points are obtained.

[0087] Convert pixel coordinates to 3D coordinates, calculate the spatial distance and angular relationship between key points based on the 3D coordinates, and obtain the spatial relationship matrix between key points;

[0088] Based on the spatial relationship matrix, the three-dimensional attitude parameters of the cargo are estimated.

[0089] Specifically, based on the deep semantic features of the goods, a 3D model of the goods is established, and key points on the goods, such as corner points and edge points, are predefined. A key point detection algorithm is used to detect key points in the goods image and obtain their pixel coordinates. Based on camera calibration parameters, the pixel coordinates of the key points are converted into 3D coordinates in the camera coordinate system. A spatial relationship matrix between key points is constructed by calculating the spatial distances and angular relationships between them. A pose estimation algorithm, such as the PnP algorithm, is used to estimate the 3D pose parameters of the goods relative to the camera, including translation vectors and rotation matrices, based on the 3D coordinates of the key points and the spatial relationship matrix. Based on the estimated pose parameters, it is determined whether the goods have undergone pose changes such as translation, rotation, tilting, or deformation, and the degree of these changes is quantified. If the pose change of the goods exceeds a preset threshold, the goods are considered to have an abnormal pose, triggering an alarm or taking appropriate action.

[0090] In this step, as an additional embodiment, the PnP algorithm is used to estimate the three-dimensional pose parameters of the cargo relative to the camera, including the translation vector and rotation matrix, based on the three-dimensional coordinates of the key points and the spatial relationship matrix.

[0091] The system acquires images of the cargo using a camera, obtaining the 2D coordinates of key points within these images. Based on a pre-calibrated camera intrinsic parameter matrix and distortion coefficients, the 2D coordinates of the key points are distorted, resulting in corrected key point coordinates. Combining this with the cargo's 3D model information, the 2D coordinates of the key points are converted to 3D coordinates through a spatial mapping relationship. A spatial relationship matrix between the camera and the cargo is constructed, including the camera's intrinsic and extrinsic parameter matrices. Using the 3D coordinates of the key points and the spatial relationship matrix as input, the PnP algorithm is employed for attitude estimation, solving for the cargo's rotation matrix and translation vector relative to the camera. Based on the obtained rotation matrix and translation vector, the 3D attitude information of the cargo in the camera coordinate system is calculated, determining the cargo's spatial position and orientation relative to the camera. This 3D attitude information of the cargo is then converted into attitude parameters for the actual scene, obtaining the cargo's position and orientation in real space, providing a reference for subsequent cargo operations.

[0092] Furthermore, such as Figure 5 As shown, based on 3D pose parameters, the weights of different cargo regions are adjusted to obtain the pose information of the occluded cargo, including:

[0093] Based on the 3D pose parameters, the attention module adaptively adjusts different acquisition regions in the image to obtain the target cargo region. The attention module, for example... Figure 6 As shown;

[0094] Based on the target cargo area, construct a spatial topology diagram of the cargo relationships;

[0095] A convolutional neural network model is used to extract features from the weighted cargo image to obtain a visual feature representation of the cargo;

[0096] By fusing spatial topology graphs and visual features, the pose information of occluded goods is inferred through graph convolutional networks.

[0097] Specifically, based on the density of the stacked goods, the system determines whether there is severe occlusion and interference. If so, an attention mechanism and contextual information are introduced for processing. Images of the stacked goods scene are acquired, and the weights of different goods regions in the image are adaptively adjusted using an attention module to highlight the target goods region and suppress background interference regions. Based on the contextual information of the stacked goods, a spatial topological relationship graph between the goods is constructed to infer the pose of occluded goods. A convolutional neural network is used to extract features from the weighted goods images to obtain visual feature representations of the goods. The visual features of the goods are fused with the contextual spatial topological relationship, and the pose of the occluded goods is inferred using a graph convolutional network. Based on the inferred pose of the goods, combined with the visual features, a multilayer perceptron is used to classify the goods' poses, resulting in a goods pose recognition result. The pose recognition result is compared with the reference pose of the target goods to calculate the accuracy of the pose recognition. The model's performance is evaluated based on the accuracy, and parameter tuning is performed as necessary to improve recognition accuracy.

[0098] A convolutional neural network is used to extract features from the weighted cargo image, resulting in a visual feature representation of the cargo including:

[0099] The process involves acquiring an image of the goods to be identified and preprocessing it, including size normalization and pixel value normalization. Using a pre-trained convolutional neural network model, features are extracted from the preprocessed image to obtain feature vectors representing the visual characteristics of the goods. A support vector machine (SVM) classifier is then used to determine the category of the goods based on these feature vectors. If the confidence level of the determined category is lower than a preset threshold, the category is re-evaluated using a nearest neighbor algorithm based on a pre-established goods image feature library. Based on the category determination, the corresponding goods attribute information, including name and specifications, is obtained. This attribute information is then associated with the goods image and stored in a goods recognition result database. The accuracy of the recognition results is calculated for different goods categories. For categories with accuracy rates below a preset threshold, the corresponding convolutional neural network model is retrained.

[0100] More specifically, suppose we have a warehouse environment where cameras capture images of goods that may include various types of goods, such as boxed beverages and bagged rice. Image preprocessing is an essential step. Size normalization unifies images of different resolutions to the same size, for example, adjusting all images to 256x256 pixels for subsequent processing. Pixel value normalization normalizes the pixel value range of the image from 0-255 to 0-1, which accelerates the model training process and improves the model's generalization ability. Next, a pre-trained convolutional neural network (CNN) model is used to extract features from the preprocessed images. Assuming a ResNet-50 model is used, this model can extract deep features from the image, forming a set of 1024-dimensional feature vectors. These feature vectors can effectively represent the visual features of the goods, such as color, texture, and shape. Based on the extracted feature vectors, a support vector machine (SVM) classifier is used to determine the goods category. Assume there are five goods categories: beverages, rice, flour, cooking oil, and detergent. SVM classifiers learn the boundaries between different categories through training data. When a new feature vector is input, the SVM can determine the category to which the vector belongs. For example, a feature vector is classified as a beverage with a confidence score of 85. If the confidence score of the determined category is lower than a preset threshold, such as 8, further judgment is needed. At this point, a pre-established feature library of goods images can be used, employing the nearest neighbor algorithm (KNN) for re-judgment. Assuming the feature library already contains a large number of labeled feature vectors of goods images, the KNN algorithm will find the K samples closest to the current feature vector, vote on the categories of these samples, and finally determine the category of the goods. For example, if 4 out of the 5 closest samples in the feature library belong to the rice category, then the goods are judged to be rice. Based on the goods category judgment result, the corresponding category's attribute information is obtained. Assuming there is a database storing detailed information about various goods, such as the attributes of beverages including brand, volume, production date, etc., the specific attribute information of a particular brand of 500ml beverage can be obtained by querying the database. The system associates cargo attribute information with cargo images and stores the results in a cargo recognition database. This facilitates subsequent queries and management; for example, in a warehouse management system, detailed information about a specific cargo can be quickly found through its image. The accuracy of the recognition results is then statistically analyzed for different cargo categories. Assuming 100 recognitions were performed for each category, the accuracy rate for beverages was 95%, while the accuracy rate for rice was only 75%. For cargo categories with accuracy rates below a preset threshold (e.g., 80%), retraining of the corresponding convolutional neural network model is required. This retraining can be achieved by increasing the number of training samples for that category and adjusting model parameters.For example, for rice, more images of rice under different lighting and angles can be added, and the ResNet-50 model can be retrained to improve its accuracy in identifying rice. Through these steps, not only can efficient identification of goods be achieved, but the identification model can also be continuously optimized, improving the overall system's accuracy and reliability. Size normalization and pixel value normalization ensure the consistency of image input; the feature vectors extracted by the CNN model provide rich visual information; the combined use of SVM and KNN classifiers improves the accuracy of category judgment; the association and storage of goods attribute information facilitates subsequent management; and accuracy statistics and model retraining form a closed-loop optimization mechanism, ensuring continuous system improvement. This multi-layered, multi-technology combination not only improves the accuracy of goods identification but also enhances the system's robustness and adaptability, enabling it to cope with complex and ever-changing warehouse environments and ensuring the efficiency and accuracy of goods management.

[0101] Image data is preprocessed, including image size normalization and pixel normalization. For example, all images are resized to 224x224 pixels, and pixel values ​​are normalized to between 0 and 1 to eliminate differences in size and brightness between different images, improving model training efficiency. The preprocessed image data is then input into a pre-trained lightweight pose recognition model, such as MobileNetV3. Lightweight models are chosen because they have fewer parameters, faster computation speed, and are suitable for deployment on resource-constrained edge devices, meeting real-time requirements. Based on the output of the cargo pose recognition model, the current pose state of the cargo is determined. The model output can be pose angle values, such as Euler angles (yaw, pitch, roll), or pose categories, such as "upright," "tilted," or "inverted." If the cargo pose deviates significantly from the preset standard pose, such as a tilt angle exceeding 30 degrees, an alert is triggered, outputting cargo pose anomaly information, such as "Cargo A is tilted at an angle of 45 degrees." This anomaly information is also sent to the monitoring platform so that staff can take timely action. The model's recognition accuracy in real-world applications is evaluated regularly, such as calculating precision and recall on a test dataset weekly or monthly. The model structure is continuously optimized by pruning redundant parameters, for example, by removing neural connections that contribute little to recognition accuracy, thus reducing model size and computational cost, and improving inference speed while maintaining accuracy. Assuming an initial inference speed of 15 frames per second, the pruned model can achieve 25 frames per second, meeting real-time requirements. Incremental learning is employed, periodically fine-tuning and updating the model using newly acquired cargo pose data. For example, every 1000 newly added cargo images with pose annotations are used to fine-tune the model, allowing it to adapt to new cargo pose changes, such as adding a new cargo stacking method. This incremental learning approach avoids retraining the entire model, saving time and resources. During model inference, by setting batch size and parallel computing (e.g., setting the batch size to 32), the parallel computing capabilities of hardware accelerators such as GPUs are fully utilized to further improve inference speed and real-time performance. For example, by using batch processing and GPU acceleration, the model's inference speed can be increased from 25 frames per second to 40 frames per second. Multiple specialized pose recognition sub-models are built for different types of goods, such as boxed goods, bagged goods, and barrelled goods. Because different types of goods have different shape and pose change characteristics, using specialized models can improve recognition accuracy. The corresponding sub-model is dynamically selected for pose recognition based on the type of goods; for example, the type of goods can be identified based on the goods' barcode, and then the corresponding sub-model is selected. Distributed edge computing nodes are deployed to perform local computation for pose recognition at the goods monitoring site; for example, an edge computing device can be deployed near each camera. This reduces data transmission latency and improves the system's real-time response capability.For example, uploading image data to the cloud for processing might take one second, while processing it on an edge device only takes one second. The central node aggregates and analyzes the recognition results from each edge node, such as counting the frequency and type of various pose anomalies, to provide decision support for warehouse management.

[0102] Furthermore, based on the attitude information of the obstructed goods, combined with the three-dimensional attitude parameters of the goods, the spatial relationships and force conditions between the goods are analyzed, and dangerous situations are predicted, including:

[0103] Based on the attitude information of the obscured goods, combined with the three-dimensional attitude parameters of the goods and the spatial relationship between the goods, a physical simulation engine is used to simulate the force on the goods and evaluate the stability of the stacked goods.

[0104] Specifically, the process involves acquiring depth information of the cargo yard environment by collecting 3D spatial data using devices such as depth cameras or LiDAR. Based on this depth information, 3D reconstruction techniques, such as point cloud stitching and surface reconstruction algorithms, are used to construct a 3D scene model of the cargo stacking. Within this model, the 3D pose information of each cargo is identified, including attributes such as position, orientation, and size. The spatial relationships between cargo in the 3D scene model are analyzed, calculating parameters such as distance and contact area to determine the rationality of the stacking. Based on the material and weight attributes of the cargo, combined with the spatial relationships between them, a physics simulation engine is used to simulate the stress on the cargo and assess its stability. A safety threshold for cargo stacking stability is set; when the stability falls below this threshold, potential safety hazards such as tipping or collapse are identified. Based on the stability analysis results, warnings are issued for cargo with safety hazards in the 3D scene, and suggestions for optimizing cargo stacking methods are provided to guide staff in re-stacking the cargo and eliminating safety hazards.

[0105] This embodiment also provides a computer vision-based cargo stacking safety monitoring system in freight yards, such as... Figure 7 The module includes: image acquisition module, feature extraction module, pose estimation module, and 3D analysis module.

[0106] The image acquisition module is used to acquire images of the goods.

[0107] The feature extraction module is used to extract semantic features from cargo images;

[0108] The attitude estimation module is used to estimate the three-dimensional attitude of the cargo.

[0109] The 3D analysis module is used to analyze the spatial relationships and stress conditions between goods and predict dangerous situations.

[0110] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A computer vision-based method for safety monitoring of cargo stacking in a freight yard, characterized in that, include: Acquire cargo image data; Based on the cargo image data, the semantic features of the cargo are obtained; Based on the semantic features, the three-dimensional pose parameters of the cargo are obtained; Based on the semantic features, the three-dimensional pose parameters of the cargo are obtained as follows: Based on the semantic features, a three-dimensional model of the cargo is established, and key points on the cargo are predefined, including corner points and edge points. Based on predefined key points, key points are detected in cargo images, and the pixel coordinates of the key points are obtained. The pixel coordinates are converted into three-dimensional coordinates, and the spatial distance and angular relationship between key points are calculated based on the three-dimensional coordinates to obtain the spatial relationship matrix between key points. Based on the spatial relationship matrix, the three-dimensional attitude parameters of the cargo are estimated; Based on the three-dimensional pose parameters, the weights of different cargo areas are adjusted to obtain the pose information of the occluded cargo. Based on the aforementioned three-dimensional pose parameters, adjusting the weights of different cargo regions to obtain the pose information of the occluded cargo includes: Based on the three-dimensional pose parameters, the attention module adaptively adjusts different acquisition areas in the image to acquire the target cargo area; Based on the target cargo area, construct a spatial topology diagram of the cargo relationships; A convolutional neural network model is used to extract features from the weighted cargo image to obtain a visual feature representation of the cargo; The spatial topology graph and visual features are fused together, and the posture information of the occluded goods is inferred through a graph convolutional network. Based on the posture information of the occluded goods, the spatial relationships and forces between the goods are analyzed, and dangerous situations are predicted.

2. The computer vision-based method for safety monitoring of cargo stacking in a freight yard according to claim 1, characterized in that, Based on the cargo image data, the semantic features of the cargo are obtained, including: The cargo image data is preprocessed to obtain high-quality cargo images; Based on the high-quality cargo image, obtain the semantic features of the cargo.

3. The computer vision-based safety monitoring method for cargo stacking in a freight yard according to claim 2, characterized in that, Preprocessing the cargo image data to obtain high-quality cargo images includes: An adaptive algorithm is used to dynamically adjust the contrast, brightness, and saturation of the cargo image data to obtain an adjusted cargo image; The adjusted cargo image is evaluated using a convolutional neural network model to determine whether the image quality meets the standards. Based on the determination result, the high-quality cargo image is obtained.

4. The computer vision-based safety monitoring method for cargo stacking in a freight yard according to claim 2, characterized in that, Based on the high-quality cargo image, the semantic features of the cargo are obtained, including: Extract the color features, texture features, shape features, and size features of the high-quality cargo image to obtain a multi-dimensional feature vector; A convolutional neural network model is used to perform deep learning and extraction on the multidimensional feature vectors to obtain the semantic features of the goods.

5. The computer vision-based method for safety monitoring of cargo stacking in a freight yard according to claim 1, characterized in that, Based on the posture information of the obstructed goods, combined with the three-dimensional posture parameters of the goods, the spatial relationship and force situation between the goods are analyzed, and dangerous situations are predicted, including: Based on the posture information of the obscured goods, combined with the three-dimensional posture parameters of the goods and the spatial relationship between the goods, a physical simulation engine is used to simulate the stress on the goods and evaluate the stability of the stacked goods.

6. A computer vision-based safety monitoring system for cargo stacking in a freight yard, characterized in that, include: Image acquisition module, feature extraction module, pose estimation module, and 3D analysis module: The image acquisition module is used to acquire images of the cargo; The feature extraction module is used to extract semantic features from the cargo image; The attitude estimation module is used to estimate the three-dimensional attitude of the cargo; Based on the semantic features, the three-dimensional pose parameters of the cargo are obtained as follows: Based on the semantic features, a three-dimensional model of the cargo is established, and key points on the cargo are predefined, including corner points and edge points. Based on predefined key points, key points are detected in cargo images, and the pixel coordinates of the key points are obtained. The pixel coordinates are converted into three-dimensional coordinates, and the spatial distance and angular relationship between key points are calculated based on the three-dimensional coordinates to obtain the spatial relationship matrix between key points. Based on the spatial relationship matrix, the three-dimensional attitude parameters of the cargo are estimated; Based on the aforementioned three-dimensional pose parameters, adjusting the weights of different cargo regions to obtain the pose information of the occluded cargo includes: Based on the three-dimensional pose parameters, the attention module adaptively adjusts different acquisition areas in the image to acquire the target cargo area; Based on the target cargo area, construct a spatial topology diagram of the cargo relationships; A convolutional neural network model is used to extract features from the weighted cargo image to obtain a visual feature representation of the cargo; The spatial topology graph and visual features are fused together, and the pose information of the occluded goods is inferred through a graph convolutional network. The three-dimensional analysis module is used to analyze the spatial relationships and stress conditions between goods and predict dangerous situations.

Images