A Package Detection System and Method Based on Conditional Random Fields

Abstract

This invention relates to the field of smart logistics technology in industrial parks, and discloses a parcel detection method based on conditional random fields (CRF). The method includes: a site vehicle management system pre-scheduling the volume of transport vehicles for each day, each time period, and each pallet; collecting parcel images at the pallets and processing the images using the 3DCRFYOLO7 algorithm model to identify each parcel, and calculating the volume of each identified parcel; continuously accumulating the volume of each parcel to obtain the total volume of the parcels; comparing the total volume of the parcels with the volume of the transport vehicle to obtain the current loading rate; and determining whether to push dispatch information to relevant staff based on the current loading rate and a set loading rate threshold. The parcel detection system and method based on CRF provided by this invention solve the problems of low accuracy in parcel detection, identification, and volume calculation for parcels of different shapes, colors, under different lighting conditions, with low image resolution, and with unclear images.

Description

Technical Field

[0001] This invention relates to the field of smart logistics technology in industrial parks, and in particular to a package detection system and method based on conditional random fields. Background Technology

[0002] In the actual environment of express delivery and logistics, most packages are irregularly shaped, making calculations using traditional methods relatively inaccurate. Calculations performed on packages packed in regular boxes result in measurements that are larger than the actual volume, leading to significant errors. An unloaded truck wastes vehicle resources. Problems that could be solved with a fully loaded truck become unmanageable due to insufficient truck space utilization, necessitating additional vehicles and increasing transportation costs.

[0003] The cargo loading rate of transport vehicles is usually estimated based on the subjective feeling of loading staff or by using laser rangefinders. The calculated results often deviate significantly from the actual situation, especially when calculating the volume of irregularly shaped packages, such as cones, trapezoids, cylinders, and shapeless plastic bags. Based on this, several methods for calculating loading rates have emerged. One method is based on image comparison. This method uses cameras to capture images of the cargo compartment when it is empty and after loading, showing the bottom, front, and sides of the compartment. The empty and loaded images are then compared to identify points of color difference. The pixels in the corresponding images are quantified to represent the corresponding area of ​​the cargo and the area of ​​each surface of the cargo compartment, thus calculating the loading rate. However, this method is computationally complex, difficult to implement, and has low accuracy and efficiency. Second, a monocular image depth estimation loading rate recognition method and device. This method inputs images of loaded vehicles and empty vehicles into a deep convolutional neural network model to obtain the depth of the corresponding cargo compartment images. The sum of the two depths is then compared to obtain the corresponding loading rate. However, this method has poor real-time performance in loading rate calculation, and the placement of packages within the cargo compartment, lighting, and other factors significantly affect the depth calculation. Third, a depth camera-based package volume measurement method and implementation. This method obtains the depth map of the image using a depth camera, obtains the depth value of each pixel, and accumulates the volumes of the pixels using the relationship between depth values ​​and volume to obtain the final package volume. While this method can calculate the volume well, it is greatly affected by the external environment and cannot determine whether an item is a package. For example, if a package is transported on a conveyor belt, this method will include the volume of the conveyor belt itself. Summary of the Invention

[0004] This invention provides a package detection system and method based on conditional random fields. It uses the 3DCRFYOLO-V7 algorithm model to calculate the volume of packages at the top of the stack and compares it with the volume of the transport vehicle. This solves the current problems of low accuracy in package detection, recognition and volume calculation for packages with different shapes, colors, lighting conditions, low image resolution, and unclear images.

[0005] This invention provides a package detection method based on conditional random fields, comprising:

[0006] The on-site vehicle management system pre-schedules the capacity of transport vehicles for each day, each time period, and each stack.

[0007] Images of packages at the top of the stack are captured by a 3D camera, and the images are processed using the 3DCRFYOLO-V7 algorithm model to identify each package at the top of the stack. The volume of each identified package is calculated to obtain the volume of each package.

[0008] The total volume of the package is obtained by continuously summing up the calculated volume of each package.

[0009] The current loading rate is obtained by comparing the total volume of the packages with the volume of the transport vehicle.

[0010] The decision to send the departure information to relevant staff is based on the current loading rate and the set loading rate threshold.

[0011] Furthermore, the training process of the 3DCRFYOLO-V7 algorithm model is as follows:

[0012] The image of the package is resized to the set size and input into the BACKBONE network in YOLO-V7 for preliminary feature extraction and noise removal. After passing through the HEAD network, it is convolved into the REP network for high-dimensional effective information feature extraction to obtain the first feature map.

[0013] The depth map corresponding to the image is adjusted to a set size and input into the CRFCONV network. After convolution, regularization, and activation, a second feature map with the same size as the output of the REP network is obtained.

[0014] The second feature map obtained from the CRFCONV network is concatenated with the first feature map obtained from the REP network, and then convolved again for output.

[0015] The accuracy of object detection and volume calculation is adjusted using a loss function, with the total loss being the predicted center coordinate loss. , prediction boundary width and height loss Predicted category loss Confidence loss of prediction Conditional random field sharpness loss The sum of these five factors is used in repeated iterative training, employing stochastic gradient descent, until the loss function of the 3DCRFYOLO-V7 algorithm model converges and becomes stable, thus obtaining the model parameters of the 3DCRFYOLO-V7 algorithm model.

[0016] Further, before the step of resizing the package image to 640x640 and inputting it into the BACKBONE network in YOLO-V7 for preliminary feature extraction and noise removal, and then convolving it through the HEAD network into the REP network for high-dimensional effective information feature extraction to obtain the first feature map, the method further includes:

[0017] Image sample data of packages on site are obtained by acquiring image sample data of packages on site by 3D camera, as well as the depth corresponding to each image, to obtain sample images; wherein, the sample images include different types of package colors, package shapes, and package sizes, and the sample images are package images under different lighting, different pallet scenes, and different interference factors;

[0018] A first predetermined number of sample images are selected as the package test set images, and a second predetermined number of sample images are selected as the package training set images.

[0019] Furthermore, the set size is 640x640, the first set quantity is 8000 images, and the second set quantity is 100000 images.

[0020] Furthermore, the predicted center coordinate loss for:

[0021]

[0022] The width and height loss of the prediction boundary for:

[0023]

[0024] The predicted category loss for:

[0025]

[0026] The predicted confidence loss for:

[0027]

[0028] The sharpness loss of the conditional random field for:

[0029]

[0030]

[0031] in, For normalization operations, The parameters represent contribution parameters, s represents the grid, and B represents the prediction box. This indicates whether the j-th preselected box in the i-th grid is responsible for being detected as a package. If it is responsible, If they are not responsible, ; This indicates that the j-th preselected box in the i-th grid is not responsible for predicting this target; and The coordinates representing the center of the prediction, and Represents the actual center point coordinates; and These represent the probability that the predicted bounding box belongs to category C and the probability that the labeled bounding box belongs to category C, respectively. and These represent the probability value and the true value of the target object, respectively, where N is the number of pixels. Let i represent the image pixel value at position i, and k represent the image pixel value at position k. The predicted coordinate position on the top Describes the neighborhood of point k in the image. This represents the depth value at point i. and These represent the feature values ​​at pixels i and j, respectively.

[0032] Furthermore, the steps of acquiring images of packages at the stack opening using a 3D camera, processing the images using the 3DCRFYOLO-V7 algorithm model to identify each package at the stack opening, and calculating the volume of each identified package to obtain the volume of each package include:

[0033] The package image is resized to 640x640 and input into the BACKBONE network in YOLO-V7 for preliminary feature extraction and noise removal. After passing through the HEAD network, it is convolved into the REP network for high-dimensional effective information feature extraction, resulting in the feature map output by the REP network.

[0034] The depth map corresponding to the package image is resized to 640x640 and input into the CRFCONV network. After convolution, regularization, and activation, a feature map of the same size as the output of the REP network is obtained.

[0035] The feature map obtained from the CRFCONV network is concatenated with the feature map obtained from the REP network, and then convolved again for output.

[0036] The YOLO-V7 pre-selection box matching strategy is used to accurately detect the package corresponding to the package image, and the volume of the package corresponding to the package image is obtained by calculus using the detected area and the depth value of the image points.

[0037] Furthermore, before the step of determining whether to push the departure information to relevant staff based on the current loading rate and the set loading rate threshold, the method further includes:

[0038] The video data collected by the 3D camera on the package is aggregated into the video network platform, and the package AI analysis platform pulls the corresponding video data through the video network platform;

[0039] The results of processing the video data using the 3DCRFYOLO-V7 algorithm model in the cloud GPU computing resource pool are then displayed on the interface of the package AI analysis platform.

[0040] The present invention also provides a package detection system based on conditional random fields, comprising:

[0041] The scheduling module is used to pre-schedule the volume of transport vehicles for each day, each time period, and each stack in the on-site vehicle management system.

[0042] The calculation module is used to acquire images of packages at the top of the stack using a 3D camera, and to process the images using the 3DCRFYOLO-V7 algorithm model to identify each package at the top of the stack, and to calculate the volume of each identified package.

[0043] The accumulation module is used to continuously accumulate the volume of each package calculated to obtain the total volume of the packages;

[0044] The comparison module is used to compare the total volume of the package with the volume of the transport vehicle to obtain the current loading rate;

[0045] The push module is used to determine whether to push departure information to relevant staff based on the current loading rate and the set loading rate threshold.

[0046] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0047] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0048] The beneficial effects of this invention are as follows:

[0049] This invention obtains the volume of the corresponding pallet transport vehicle based on the on-site vehicle management system, uses a 3D depth camera to obtain package data on the conveyor belt, and then processes the package data using the 3DCRFYOLO-V7 algorithm model to accurately identify and detect each package and obtain the corresponding volume of each package. These data are then accumulated sequentially, and finally the current loading rate (the ratio of the accumulated package volume to the total vehicle volume) is obtained in real time. Based on the set departure loading rate threshold, departure information is pushed to relevant staff, thereby reducing vehicle transportation costs and improving vehicle transportation efficiency. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of a method flow according to an embodiment of the present invention.

[0051] Figure 2 This is a schematic diagram of the YOLO-V7 package loading rate algorithm model architecture based on conditional random fields in this invention.

[0052] Figure 3 This is a schematic diagram of the loss training process of the 3DCRFYOLO-V7 model and the YOLO-V7 model in this invention.

[0053] Figure 4 This is a graph showing the change in AP of YOLO-V7 in this invention.

[0054] Figure 5 This is a curve showing the AP variation of 3DCRFYOLO-V7 in this invention.

[0055] Figure 6 This is a schematic diagram showing the test results of the 3DCRFYOLO-V7 algorithm model on package data in this invention.

[0056] Figure 7 This is a schematic diagram of the construction architecture of the package loading rate AI analysis platform in this invention.

[0057] Figure 8 This is a schematic diagram of the device structure according to an embodiment of the present invention.

[0058] Figure 9 This is a schematic diagram of the internal structure of a computer device according to an embodiment of the present invention.

[0059] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0060] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0061] This invention leverages the advantages of YOLO-V7, including its rich network architecture, effective feature integration methods, accurate detection methods, precise loss functions, and effective label allocation methods, to achieve object detection and recognition at a speed of 160 FPS. In practical applications, while YOLO-V7's detection speed has been greatly improved, it cannot accurately detect and recognize objects when the image resolution is low or the image edges are blurred. Considering the respective advantages of deep convolutional neural networks and graph models, and utilizing the powerful feature representation capabilities of deep convolutional neural networks and the ability of conditional random fields (CRFs) to establish local and global relationships, this invention combines the image depth map acquired by a 3D camera to construct the correlation between the image and the depth map, as well as between pixels at different locations in the image. A YOLO-V7 package loading rate algorithm model based on CRFs is proposed, abbreviated as 3DCRFYOLO-V7 model, to improve the accuracy of object recognition, detection, and volume calculation.

[0062] like Figure 1 As shown, this invention provides a package detection method based on conditional random fields, comprising:

[0063] S1, the capacity of pre-scheduled transport vehicles for each day, each time period, and each stacking point in the on-site vehicle management system;

[0064] S2. Capture images of packages at the stack opening using a 3D camera, or capture video data of the packages. Process the images using the 3DCRFYOLO-V7 algorithm model to accurately detect and identify each package at the stack opening. After the 3DCRFYOLO-V7 algorithm model detects the packages in the image, it calculates the volume of each identified package using a depth map to obtain the volume of each package.

[0065] like Figure 2 As shown, the YOLO-V7 package loading rate algorithm model architecture based on Conditional Random Fields (CRF) adds a CRFCONV (Conditional Random Field) network to the YOLO-V7 network architecture to optimize the HEAD output network. First, the image is resized to 640x640 and input into the BACKBONE network. Then, the optimized HEAD network outputs three layers of feature maps of different sizes, which are then processed by CONV to output the prediction results. This invention uses package data collected by a 3D camera under different scenes and lighting conditions for training, testing, detection, recognition, and calculation. Finally, it outputs the (X, Y, W, H, C, V) coordinates, attribute category, and volume of each package.

[0066] Using the 3DCRFYOLO-V7 algorithm model requires training. The training process for the 3DCRFYOLO-V7 algorithm model is as follows:

[0067] T1. Obtain sample image data of packages on site through a 3D camera, as well as the depth corresponding to each image, to obtain sample images; wherein, the sample images include different types of package colors, package shapes, and package sizes, and the sample images are package images under different lighting conditions, different pallet scenes, and different interference factors;

[0068] T2. Select a first predetermined number of sample images as the package test set images, and select a second predetermined number of sample images as the package training set images. The first predetermined number is 8000 images, and the second predetermined number is 100000 images. The main purpose is to improve the model's generalization ability and make it applicable to more scenarios. Maintain the original YOLO-V7 model architecture, and when outputting the final feature map, use the deep feature map of the associated images through a conditional random field to train the 3DCRF YOLO-V7 algorithm model.

[0069] T3. Adjust the size of the package image to the set size and input it into the BACKBONE network in YOLO-V7 for preliminary feature extraction and noise removal. After passing through the HEAD network, it is convolved into the REP network for high-dimensional effective information feature extraction to obtain the first feature map. The set size is 640x640.

[0070] T4. Adjust the size of the depth map corresponding to the image to the set size, input it into the CRFCONV network, and after convolution, regularization and activation, obtain the second feature map with the same size as the output of the REP network;

[0071] T5. Concatenate the second feature map obtained from the CRFCONV network with the first feature map obtained from the REP network, and then convolve again to output the result.

[0072] T6. Adjust the accuracy of object detection and volume calculation using a loss function. The total loss is the predicted center coordinate loss. , prediction boundary width and height loss Predicted category loss Confidence loss of prediction Conditional random field sharpness loss The sum of these five factors is used in repeated iterative training, employing stochastic gradient descent, until the loss function of the 3DCRFYOLO-V7 algorithm model converges and becomes stable, thus obtaining the model parameters of the 3DCRFYOLO-V7 algorithm model.

[0073] The predicted center coordinate loss for:

[0074]

[0075] The width and height loss of the prediction boundary for:

[0076]

[0077] The predicted category loss for:

[0078]

[0079] The predicted confidence loss for:

[0080]

[0081] The sharpness loss of the conditional random field for:

[0082]

[0083]

[0084] in, This is a normalization operation, serving a normalization purpose. The parameters represent contribution parameters, s represents the grid, and B represents the prediction box. This indicates whether the j-th preselected box in the i-th grid is responsible for being detected as a package. If it is responsible, If they are not responsible, ; This indicates that the j-th preselected box in the i-th grid is not responsible for predicting this target; and The coordinates representing the center of the prediction, and This represents the actual center point coordinates; similarly, width and height. and These represent the probability that the predicted bounding box belongs to category C and the probability that the labeled bounding box belongs to category C, respectively. and These represent the probability value and the true value of the target object, respectively, where N is the number of pixels. Let i represent the image pixel value at position i, and k represent the image pixel value at position k. The predicted coordinate position on the top Describes the neighborhood of point k in the image. This represents the depth value at point i. and These represent the feature values ​​of pixels at points i and j, respectively. When the difference between the values ​​of these two pixels is large, the closer their depth and pixel values ​​are, the better the smoothness, the greater the correlation between pixels, and the higher the overall image clarity.

[0085] To demonstrate the superiority of the proposed 3DCRFYOLO-V7 model, the YOLO-V7 model is trained simultaneously during the training of the 3DCRFYOLO-V7 model. For example... Figure 3 As shown in the training results, the 3DCRFYOLO-V7 model reduces its loss from over 1000 to 70 after approximately 3 iterations, and converges around a loss of 10. The YOLO-V7 model, trained on top of a pre-trained model using wrapper data, has an initial loss of around 12 and converges after approximately 50 iterations, exhibiting slower convergence compared to the proposed 3DCRFYOLO-V7 model. The training process demonstrates that the 3DCRFYOLO-V7 model outperforms the YOLO-V7 model.

[0086] like Figure 4 , Figure 5 As shown, the AP (mean and average precision) curve of the 3DCRFYOLO-V7 model shows that when the recall is around 1.0, the AP value is 99.97%; while the AP curve of the YOLO-V7 model shows that when the recall is around 0.8, the AP value is 98.56%. Therefore, the accuracy of the 3DCRFYOLO-V7 model is better than that of the YOLO-V7 model.

[0087] like Figure 6 As shown, the 3DCRFYOLO-V7 algorithm model was used to test the on-site package data. The test data shows that: 1. For regular packages, the error rate between the measured value and the actual value is around 2%-4%. 2. For irregular packages, the error rate between the measured value and the actual value is around 5%-14.5%. Combining 1 and 2, the overall package measurement error is between 3%-9%, where negative values ​​indicate a higher error than the actual value.

[0088] After training the 3DCRFYOLO-V7 algorithm model, the image is processed using the 3DCRFYOLO-V7 algorithm model to identify each package at the top of the stack, and the volume of each identified package is calculated. The steps to obtain the volume of each package include:

[0089] S21. Adjust the size of the package image to 640x640 and input it into the BACKBONE network in YOLO-V7 for preliminary feature extraction and noise point removal. After passing through the HEAD network, it is convolved into the REP network for high-dimensional effective information feature extraction to obtain the feature map output by the REP network.

[0090] S22. Adjust the size of the depth map corresponding to the package image to 640x640, input it into the CRFCONV network, and after convolution, regularization and activation, obtain a feature map with the same size as the output of the REP network.

[0091] S23. Concatenate the feature map obtained from the CRFCONV network with the feature map obtained from the REP network, and then convolve again to output the result.

[0092] S24. Accurately detect the package corresponding to the package image using the YOLO-V7 pre-selection box matching strategy, and use the detected area and image point depth value to perform calculus to obtain the volume of the package corresponding to the package image.

[0093] S3. Continuously accumulate the calculated volume of each package to obtain the total volume of the package;

[0094] S4. Ratio the total volume of the package to the volume of the transport vehicle to obtain the current loading rate;

[0095] This invention combines cloud computing, cloud storage, and the Internet of Things to build an AI analysis platform for package loading rates, such as... Figure 7 The diagram illustrates the architecture of the package loading rate AI analysis platform. First, the terminal device's 3D camera collects package data. The video data collected by the 3D camera is aggregated to a video network platform. The package AI analysis platform retrieves the corresponding video data through this platform. It then uses the 3DCRFYOLO-V7 algorithm model from the cloud GPU computing resource pool to process the video data and displays the results on the platform's interface. The platform displays the loading rate status of each pallet in real-time via a split-screen display. Each pallet interface shows the total number of packages, the total volume of packages, and the total loading rate. The platform implements real-time calculation and accumulation of package volume and vehicle loading rate, real-time cumulative statistics of package count, real-time monitoring of package transportation status, and pushes departure information to relevant staff when the vehicle loading rate reaches a preset threshold.

[0096] S5. Based on the current loading rate and the set loading rate threshold, it determines whether to push the departure information to relevant staff, which reduces vehicle transportation costs and improves vehicle transportation efficiency.

[0097] This invention improves the accuracy of object recognition and detection for images with low resolution or blurriness through its provided algorithm model. For irregularly shaped packages or in poorly lit environments, the algorithm model enhances the accuracy of package volume calculation. By integrating advanced technologies such as cloud computing, cloud storage, and the Internet of Things, and based on the provided algorithm model, an AI analysis platform for package loading rate is built, realizing the application of the algorithm model.

[0098] The proposed algorithm model adds a CRFCONV (Conditional Random Field) network to the YOLO-V7 network architecture and provides a loss function to adjust the accuracy of object detection and volume calculation. Through iterative training until the loss function converges, the corresponding model parameters are obtained, and the HEAD output network is optimized. Leveraging the advantages of YOLO-V7's rich network architecture, effective feature ensemble methods, accurate detection methods, and effective label allocation methods, and combining conditional random fields, the model constructs the correlation between the image and depth map, as well as between pixels at different locations in the image. Based on this algorithm model, and combined with advanced technologies such as cloud computing, cloud storage, and the Internet of Things, a package loading rate AI analysis platform is built, realizing the application of the algorithm model and improving the accuracy of package detection, recognition, and volume calculation for packages of different shapes, colors, lighting conditions, low image resolution, and unclear images.

[0099] The algorithm model proposed in this invention transforms the logistics and transportation industry from subjective judgment by workers to intelligent judgment by AI, providing effective data for decisions such as whether to dispatch a vehicle, and increasing the reliability and accuracy of decision-making. Furthermore, by combining advanced technologies such as cloud computing, cloud storage, and the Internet of Things, an easy-to-use parcel loading rate platform is built, which greatly increases the number of parcels that can be loaded onto a vehicle, reduces transportation costs, and improves transportation efficiency.

[0100] like Figure 8 As shown, the present invention also provides a package detection system based on conditional random fields, comprising:

[0101] The scheduling module 1 is used to pre-schedule the volume of transport vehicles for each day, each time period, and each stack in the on-site vehicle management system.

[0102] The calculation module 2 is used to acquire images of packages at the top of the stack using a 3D camera, and to process the images using the 3DCRFYOLO-V7 algorithm model to identify each package at the top of the stack, and to calculate the volume of each identified package.

[0103] Accumulation module 3 is used to continuously accumulate the volume of each package calculated to obtain the total volume of the packages;

[0104] Comparison module 4 is used to compare the total volume of the package with the volume of the transport vehicle to obtain the current loading rate;

[0105] The push module 5 is used to determine whether to push the departure information to relevant staff based on the current loading rate and the set loading rate threshold.

[0106] In one embodiment, the training process of the 3DCRFYOLO-V7 algorithm model in computing module 1 is as follows:

[0107] The first feature unit is used to adjust the size of the package image to a set size and input it into the BACKBONE network in YOLO-V7 for preliminary feature extraction and noise point removal. After passing through the HEAD network, it is convolved into the REP network for high-dimensional effective information feature extraction to obtain the first feature map.

[0108] The second feature unit is used to adjust the size of the depth map corresponding to the image to a set size, input it into the CRFCONV network, and after convolution, regularization and activation, obtain a second feature map with the same size as the output of the REP network.

[0109] The concatenation unit is used to concatenate the second feature map obtained by the CRFCONV network with the first feature map obtained by the REP network, and then convolve the output again.

[0110] The iterative unit is used to adjust the accuracy of object detection and volume calculation using a loss function. The total loss is the predicted center coordinate loss. , prediction boundary width and height loss Predicted category loss Confidence loss of prediction Conditional random field sharpness loss The sum of these five factors is used in repeated iterative training, employing stochastic gradient descent, until the loss function of the 3DCRFYOLO-V7 algorithm model converges and becomes stable, thus obtaining the model parameters of the 3DCRFYOLO-V7 algorithm model.

[0111] In one embodiment, the computing module 1 further includes:

[0112] The acquisition unit is used to acquire image sample data of packages on site through a 3D camera, as well as the depth corresponding to each image, to obtain sample images; wherein, the sample images include different types of package colors, package shapes, and package sizes, and the sample images are package images under different lighting, different pallet scenes, and different interference factors;

[0113] The selection unit is used to select a first predetermined number of sample images as package test set images and to select a second predetermined number of sample images as package training set images.

[0114] In one embodiment, in the calculation module 1, the set size is 640x640, the first set quantity is 8000 images, and the second set quantity is 100000 images.

[0115] In one embodiment, in calculation module 1, the predicted center coordinate loss for:

[0116]

[0117] The width and height loss of the prediction boundary for:

[0118]

[0119] The predicted category loss for:

[0120]

[0121] The predicted confidence loss for:

[0122]

[0123] The sharpness loss of the conditional random field for:

[0124]

[0125]

[0126] in, For normalization operations, The parameters represent contribution parameters, s represents the grid, and B represents the prediction box. This indicates whether the j-th preselected box in the i-th grid is responsible for being detected as a package. If it is responsible, If they are not responsible, ; This indicates that the j-th preselected box in the i-th grid is not responsible for predicting this target; and The coordinates representing the center of the prediction, and Represents the actual center point coordinates; and These represent the probability that the predicted bounding box belongs to category C and the probability that the labeled bounding box belongs to category C, respectively. and These represent the probability value and the true value of the target object, respectively, where N is the number of pixels. Let i represent the image pixel value at position i, and k represent the image pixel value at position k. The predicted coordinate position on the top Describes the neighborhood of point k in the image. This represents the depth value at point i. and These represent the feature values ​​at pixels i and j, respectively.

[0127] In one embodiment, the computing module 1 includes:

[0128] The first adjustment unit is used to adjust the size of the package image to 640x640 and input it into the BACKBONE network in YOLO-V7 for preliminary feature extraction and noise point removal. After passing through the HEAD network, it is convolved into the REP network for high-dimensional effective information feature extraction to obtain the feature map output by the REP network.

[0129] The second adjustment unit is used to adjust the size of the depth map corresponding to the package image to 640x640, input it into the CRFCONV network, and after convolution, regularization and activation, obtain a feature map with the same size as the output of the REP network.

[0130] The output unit is used to concatenate the feature map obtained from the CRFCONV network with the feature map obtained from the REP network, and then perform another convolution to output the result.

[0131] The calculus unit is used to accurately detect the package corresponding to the package image using the pre-selection box matching strategy of YOLO-V7, and to perform calculus using the detected area and the depth value of the image points to obtain the volume of the package corresponding to the package image.

[0132] In one embodiment, it also includes:

[0133] The aggregation unit is used to aggregate the video data collected by the 3D camera on the package to the video network platform. The package AI analysis platform pulls the corresponding video data through the video network platform.

[0134] The feedback unit is used to call the 3DCRFYOLO-V7 algorithm model in the cloud GPU computing resource pool to process the video data and then feed the results back to the interface of the package AI analysis platform for display.

[0135] Each of the above modules and units is used to perform the respective steps in the above-mentioned package detection method based on conditional random fields. The specific implementation method is as described in the above-mentioned method embodiments, and will not be repeated here.

[0136] like Figure 9As shown, the present invention also provides a computer device, which may be a server, and its internal structure may be as follows: Figure 9 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores all data required for the process of the conditional random field-based package detection method. The network interface is used for communication with external terminals via a network connection. The computer program is executed by the processor to implement the conditional random field-based package detection method.

[0137] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment on which the present application is applied.

[0138] An embodiment of this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the above-described package detection methods based on conditional random fields.

[0139] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in this application and in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0140] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0141] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A package detection method based on conditional random fields, characterized in that, include: The on-site vehicle management system pre-schedules the capacity of transport vehicles for each day, each time period, and each stack. Images of packages at the top of a stack are captured using a 3D camera, and the 3DCRFYOLO-V7 algorithm model is used to process these images to identify each package at the top of the stack and calculate the volume of each package. The training process of the 3DCRFYOLO-V7 algorithm model is as follows: the package image is resized to a set size and input into the BACKBONE network of YOLO-V7 for initial feature extraction and noise removal. After passing through the HEAD network, it is convolved into the REP network for high-dimensional effective feature extraction, resulting in a first feature map. The corresponding depth map is resized to a set size and input into the CRFCONV network. After convolution, regularization, and activation, a second feature map of the same size as the REP network output is obtained. The second feature map obtained from the CRFCONV network is concatenated with the first feature map obtained from the REP network, and then convolved again for output. A loss function is used to adjust the accuracy of object detection and volume calculation; the total loss is the predicted center coordinate loss. , prediction boundary width and height loss Predicted category loss Confidence loss of prediction Conditional random field sharpness loss The sum of the five factors is used for iterative training, employing stochastic gradient descent, until the loss function of the 3DCRFYOLO-V7 algorithm model converges and becomes stable, thus obtaining the model parameters of the 3DCRFYOLO-V7 algorithm model. The total volume of the package is obtained by continuously summing up the calculated volume of each package. The current loading rate is obtained by comparing the total volume of the packages with the volume of the transport vehicle. The decision to send the departure information to relevant staff is based on the current loading rate and the set loading rate threshold.

2. The package detection method based on conditional random fields according to claim 1, characterized in that, Before the step of resizing the package image to a set size and inputting it into the BACKBONE network in YOLO-V7 for initial feature extraction and noise removal, and then convolving it through the HEAD network into the REP network for high-dimensional effective feature extraction to obtain the first feature map, the following steps are also included: Image sample data of packages on site are obtained by acquiring image sample data of packages on site by 3D camera, as well as the depth corresponding to each image, to obtain sample images; wherein, the sample images include different types of package colors, package shapes, and package sizes, and the sample images are package images under different lighting, different pallet scenes, and different interference factors; A first predetermined number of sample images are selected as the package test set images, and a second predetermined number of sample images are selected as the package training set images.

3. The package detection method based on conditional random fields according to claim 2, characterized in that, The set size is 640x640, the first set quantity is 8000 images, and the second set quantity is 100000 images.

4. The package detection method based on conditional random fields according to claim 2, characterized in that, The predicted center coordinate loss for: The width and height loss of the prediction boundary for: The predicted category loss for: The predicted confidence loss for: The sharpness loss of the conditional random field for: in, For normalization operations, The parameters represent contribution parameters, where 's' represents the number of grid cells on a single side, and 'B' represents the number of preselected boxes. This indicates whether the j-th preselected box in the i-th grid is responsible for being detected as a package. If it is responsible, If they are not responsible, ; This indicates that the j-th preselected box in the i-th grid is not responsible for predicting this target; and The coordinates representing the center of the prediction, and Represents the actual center point coordinates; and These represent the probability that the predicted bounding box belongs to category C and the probability that the labeled bounding box belongs to category C, respectively. and These represent the probability value and the true value of the target object, respectively, where N is the number of pixels. Let i represent the image pixel value at position i, and k represent the image pixel value at position k. The predicted coordinate position on the top Describes the neighborhood of point k in the image. This represents the depth value at point i. and These represent the feature values ​​at pixels i and j, respectively.

5. The package detection method based on conditional random fields according to claim 4, characterized in that, The steps of acquiring images of packages at the stack opening using a 3D camera, processing the images using the 3DCRFYOLO-V7 algorithm model to identify each package at the stack opening, and calculating the volume of each identified package to obtain the volume of each package include: The package image is resized to 640x640 and input into the BACKBONE network in YOLO-V7 for preliminary feature extraction and noise removal. After passing through the HEAD network, it is convolved into the REP network for high-dimensional effective information feature extraction, resulting in the feature map output by the REP network. The depth map corresponding to the package image is resized to 640x640 and input into the CRFCONV network. After convolution, regularization, and activation, a feature map of the same size as the output of the REP network is obtained. The feature map obtained from the CRFCONV network is concatenated with the feature map obtained from the REP network, and then convolved again for output. The YOLO-V7 pre-selection box matching strategy is used to accurately detect the package corresponding to the package image, and the volume of the package corresponding to the package image is obtained by calculus using the detected area and the depth value of the image points.

6. The package detection method based on conditional random fields according to claim 1, characterized in that, Before the step of determining whether to push the departure information to relevant staff based on the current loading rate and the set loading rate threshold, the method also includes: The video data collected by the 3D camera on the package is aggregated into the video network platform, and the package AI analysis platform pulls the corresponding video data through the video network platform; The results of processing the video data using the 3DCRFYOLO-V7 algorithm model in the cloud GPU computing resource pool are then displayed on the interface of the package AI analysis platform.

7. A package detection system based on conditional random fields, characterized in that, include: The scheduling module is used to pre-schedule the volume of transport vehicles for each day, each time period, and each stack in the on-site vehicle management system. The calculation module is used to acquire images of packages at the top of the stack using a 3D camera, and process the images using the 3DCRFYOLO-V7 algorithm model to identify each package at the top of the stack and calculate the volume of each identified package. The training process of the 3DCRFYOLO-V7 algorithm model is as follows: the package image size is adjusted to a set size and input into the BACKBONE network in YOLO-V7 for preliminary feature extraction and noise removal. After passing through the HEAD network, it is convolved into the REP network for high-dimensional effective information feature extraction, resulting in a first feature map. The corresponding depth map size is adjusted to a set size and input into the CRFCONV network. After convolution, regularization, and activation, a second feature map with the same size as the REP network output is obtained. The second feature map obtained from the CRFCONV network and the first feature map obtained from the REP network are concatenated and convolved again for output. A loss function is used to adjust the accuracy of object detection and volume calculation; the total loss is the predicted center coordinate loss. , prediction boundary width and height loss Predicted category loss Confidence loss of prediction Conditional random field sharpness loss The sum of the five factors is used for iterative training, employing stochastic gradient descent, until the loss function of the 3DCRFYOLO-V7 algorithm model converges and becomes stable, thus obtaining the model parameters of the 3DCRFYOLO-V7 algorithm model. The accumulation module is used to continuously accumulate the volume of each package calculated to obtain the total volume of the packages; The comparison module is used to compare the total volume of the package with the volume of the transport vehicle to obtain the current loading rate; The push module is used to determine whether to push departure information to relevant staff based on the current loading rate and the set loading rate threshold.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images