Artificial intelligence-based warehouse access visual detection system and method

Abstract

The application relates to the warehouse technology field, in particular to an in-out warehouse visual detection system and method based on artificial intelligence, the method comprising the following steps: acquiring in-warehouse stage image data, wherein the in-warehouse stage image data comprises a to-be-warehoused article image, a to-be-warehoused article point cloud image, a mechanical arm in-warehouse operation image sequence and a goods shelf article stacking image; acquiring out-warehouse stage image data, wherein the out-warehouse stage image data comprises a to-be-out-warehoused article image, a to-be-out-warehoused article point cloud image and a mechanical arm out-warehouse operation image sequence; in the in-warehouse stage, the in-warehouse stage image data is analyzed to realize tilt detection, deformation identification and tilt cause analysis of goods; in the out-warehouse stage, the out-warehouse stage image data is analyzed to realize deformation degree detection, damage prediction and out-warehouse condition judgment of the goods package, and the out-warehouse strategy is updated according to the damage prediction and the out-warehouse condition judgment result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of warehousing, in particular to an in-out warehouse visual detection system and method based on artificial intelligence. BACKGROUND

[0002] Machine vision is widely used in warehouse in-out. It mainly uses image processing, pattern recognition and artificial intelligence technology to automate key links, improve efficiency, reduce errors and optimize management. For example, it can be used for cargo identification, volume and weight anomaly detection and packaging box deformation of packaged goods.

[0003] In reality, common use cases include intelligent warehousing systems that detect goods through machine vision, identify goods information, and transport goods to the corresponding position of the shelf according to the preset in-out warehouse route for storage or retrieval. In this process, the visual recognition of traditional systems is mostly focused on cargo information identification (identifying packaging box surface labels and packaging box sizes, etc.), and does not take into account the deformation of the external packaging box (especially the carton) caused by the mechanical arm pressure, other goods extrusion and environmental temperature and humidity during transportation and storage. When the deformation exceeds a certain threshold, the internal goods may be damaged, and such damage is hidden (relative to the surface damage of the box) and difficult to detect. SUMMARY

[0004] The present application identifies packaging box deformation by collecting packaging box surface image data and point cloud data, and the operation pressure of the mechanical arm during transportation, and predicts whether the goods inside the packaging box are damaged based on the deformation information.

[0005] The technical solution of the present application is as follows: an in-out warehouse visual detection method based on artificial intelligence, the method comprising:

[0006] Obtaining in-out warehouse image data, the in-out warehouse image data comprising in-warehouse item image, in-warehouse item point cloud image, mechanical arm in-warehouse operation image sequence and shelf item stacking image;

[0007] Obtaining out-of-warehouse image data, the out-of-warehouse image data comprising out-of-warehouse object image, out-of-warehouse item point cloud image and mechanical arm out-of-warehouse operation image sequence;

[0008] In the in-warehouse stage, the in-warehouse image data is analyzed to realize the tilt detection, deformation identification and tilt reason analysis of the goods, including the following sub-steps:

[0009] Tilt detection, deformation identification and tilt reason analysis of goods in the stacking process;

[0010] After the stacking is completed, the tilt detection of the stacked multi-layer packaging box is performed to identify the risk of tilting;

[0011] In the warehouse-out phase, the deformation degree detection, damage prediction and warehouse-out condition judgment of the cargo package are realized by analyzing the warehouse-out phase image data, and the warehouse-out strategy is updated according to the damage prediction and warehouse-out condition judgment results.

[0012] Preferably, before the tilt detection, deformation identification and tilt cause analysis of the cargo in the stacking process, it comprises:

[0013] Obtaining point cloud data of the packaging box through a 3D camera;

[0014] Filtering the point cloud data to remove outliers to form a time sequence point cloud sequence;

[0015] Obtaining multiple angle RGB image data of the packaging box in the stacking area through a multi-angle camera; obtaining pressure data when the mechanical arm operates and adding a mechanical arm operation timestamp through a mechanical arm end sensor; and collecting mechanical arm front end image data through a mechanical arm end camera;

[0016] Aligning the mechanical arm coordinate system with the multi-angle camera coordinate system based on a calibration board;

[0017] The tilt detection, deformation identification and tilt cause analysis of the cargo in the stacking process comprises:

[0018] Detecting the tilt degree of the packaging box in the stacking and positioning the tilt point;

[0019] Detecting the surface deformation of the packaging box;

[0020] Correlating the mechanical arm operation with the packaging box deformation.

[0021] Preferably, the detection of the tilt degree of the packaging box in the stacking and the positioning of the tilt point comprises the following sub-steps:

[0022] Performing Canny edge detection on the RGB image data to extract a packaging box contour image;

[0023] Performing Hough transform on the packaging box contour image to detect the straight line of the packaging box edge;

[0024] Calculating the angle deviation of the detected straight line; that is, Wherein, Δy and Δx represent the deviation of the straight line relative to the vertical reference axis and the deviation relative to the horizontal reference axis, respectively;

[0025] Based on the point cloud data, the offset amount of the packaging box straight line position and the geometric center is calculated, specifically:

[0026] Wherein, N represents the number of point clouds, x i , y i , z irespectively represent point cloud coordinates;

[0027] The surface deformation of the packaging box is detected, including the following sub-steps:

[0028] A reference point cloud model of the packaging box without deformation is obtained from a database;

[0029] The current packaging box point cloud is aligned with the reference point cloud model using an iterative closest point (ICP) algorithm; the ICP objective function is: wherein R represents a rotation matrix, t i represents a translation vector, p i , q i respectively represent vectors of corresponding points in the current point cloud and the reference point cloud;

[0030] The displacement of the current packaging box and the reference point cloud model is compared point by point to obtain a plurality of displacement amplitude values;

[0031] If the displacement amplitude value exceeds a preset displacement threshold, the point is determined to be a deformation point, and a plurality of deformation points constitute a deformation region;

[0032] The Gaussian curvature value of the deformation region is obtained through local surface healing calculation;

[0033] After the obtained displacement amplitude value and the Gaussian curvature value are standardized, they are input into a pre-trained support vector machine (SVM) model to output the deformation type of the deformation region; the deformation type includes concave, protruding, and twisting;

[0034] The association of the mechanical arm operation and the packaging box deformation includes:

[0035] The point cloud data acquisition timestamp, the RGB image data acquisition time, and the mechanical arm operation timestamp are obtained;

[0036] The point cloud data acquisition timestamp, the RGB image data acquisition time, and the mechanical arm operation timestamp are aligned on a time axis;

[0037] The pressure value F J corresponding to the mechanical arm operation timestamp is obtained;

[0038] The association probability of the mechanical arm operation and the packaging box deformation is wherein F0 represents a pressure threshold; ROI B , ROI J respectively represent the packaging box deformation region and the contact region between the mechanical arm and the packaging box.

[0039] Preferably, the detection of the surface deformation of the packaging box further includes:

[0040] The point cloud of the packaging box was segmented into side, top, and bottom regions, and corresponding region identification tags were added to each region. Specifically:

[0041] The principal orientation of the packaging box is calculated using the principal component analysis (PCA) algorithm.

[0042] The major axis direction is represented by vector v1, the width axis direction by vector v2, and the height axis direction by vector v3.

[0043] Take the top 10% of the point cloud region along the v3 direction as the top region, and take the bottom 10% of the point cloud region along the opposite direction of v3 as the bottom region.

[0044] The point cloud of the packaging box is divided into four side regions using v1 and v2;

[0045] Perform local surface fitting on each region and calculate the Gaussian curvature K′ and mean curvature H′ for each region;

[0046] Where λ1, λ2, and A represent the principal curvature of local surface one to be fitted, the principal curvature of local surface two, and the neighborhood area, respectively.

[0047] The volume of the point cloud in each region is calculated using the convex hull algorithm. The calculated volume of each region is then compared with the volume of the corresponding region in the reference point cloud to obtain the volume difference ΔV. i =|V i -V i0 |;Among them, V i V represents the point cloud volume of region i. i0 This represents the reference volume of region i corresponding to the reference point cloud;

[0048] The translation vector t of each point in the point cloud of each region is obtained using the ICP algorithm. j =p ij -q ij Calculate the average displacement value of the region Where, m is the number of points within the region; p ij q ij Let represent the vector of point j in the point cloud of region i and the vector of point j in the reference point cloud corresponding to region i, respectively.

[0049] Obtain the maximum Gaussian curvature K′ for each region max Maximum mean curvature H′ max Regional average displacement value and volume difference ΔV i After standardization, feature vectors for each region are constructed.

[0050] The feature vector of each region is input into a pre-trained random forest model, and the deformation type label of each region is output; the type label includes category 1, category 2 and category 3;

[0051] Type 1 is lateral deformation, that is, the average displacement value of the lateral region is greater than a preset lateral deformation threshold and the maximum Gaussian curvature of the lateral region is greater than a preset lateral deformation curvature threshold.

[0052] Type 2 is top deformation, that is, the volume difference of the top region is greater than the preset region volume difference threshold;

[0053] Type 3 is bottom deformation, that is, the average displacement value of the bottom region is greater than the preset bottom deformation threshold and the maximum Gaussian curvature of the bottom region is greater than the preset bottom deformation curvature threshold.

[0054] Preferably, the step of detecting the degree of deformation of the goods packaging, predicting damage, and determining the outbound conditions by analyzing image data during the outbound stage includes:

[0055] The images of stacked goods are processed using a semantic segmentation network to distinguish between the visible and occluded areas of the packaging boxes; specifically:

[0056] RGB image data of packaging boxes from multiple angles within the stacking area are acquired, and the acquisition timestamps of the RGB images are obtained. The RGB images are then arranged according to the acquisition timestamps to form an RGB image sequence.

[0057] Obtain the image sequence of the robotic arm's warehouse entry operation, and align the RGB image sequence with the images within the robotic arm's warehouse entry operation image sequence on the time axis;

[0058] The occlusions in the RGB image are segmented using a segmentation model to obtain a segmentation mask; the occlusions include packaging boxes and shelves.

[0059] The segmentation mask is projected onto the point cloud of the packaging box, and the occluded areas of the point cloud are marked as invalid areas, while the unoccluded areas of the point cloud are marked as visible areas.

[0060] Predict the deformation of the occluded region based on the deformation characteristics of the visible region;

[0061] Based on the deformation of the obstructed area, cargo damage is predicted, and the prediction results are used to determine whether the outbound conditions are met.

[0062] If so, a scheduling strategy is generated to dynamically plan a new pickup route; otherwise, the original pickup route is maintained.

[0063] The dynamic planning of the new pickup route includes:

[0064] The warehouse shelving is abstracted as a graph G = (V, E); where node V represents a storage location and edge E represents a feasible path.

[0065] A new pickup route is calculated using the A* algorithm to avoid goods that do not meet the outbound conditions.

[0066] Preferably, the prediction of the deformation of the occluded region based on the deformation features of the visible region includes:

[0067] Deformation and displacement features are captured by pixel motion between consecutive image frames;

[0068] Calculate the displacement vector of each point in the point cloud using a time-series point cloud sequence;

[0069] Based on deformation displacement features and the displacement vector of each point in the point cloud, the deformation of the occluded region is predicted using an LSTM-UNet hybrid network.

[0070] Preferably, the step of capturing deformation displacement features through pixel motion between consecutive image frames includes:

[0071] Dynamic deformation features of visible regions are extracted from RGB image sequences. These dynamic deformation features include one or more of the following: translation of each point within the point cloud of each visible region, Gaussian curvature of each region, average curvature, and texture features; specifically:

[0072] Establish the optical flow equation I x u+I y v+I t =0, where I x I y I t represents the spatial gradient of the image along the x and y axes, respectively; t represents the temporal gradient of the image along the t-axis.

[0073] For each pixel's neighborhood, an overdetermined equation is constructed, and the least squares solution is:

[0074] Where B and b represent the gradient matrix and temporal gradient vector of all pixels in the neighborhood, respectively; u and v represent the instantaneous displacement velocities of pixels in the RGB image along the x and y axes in space.

[0075] The Gaussian pyramid algorithm is used to generate an α-layer multi-scale image, and optical flow is calculated layer by layer to solve the large displacement problem; the neighborhood size is 15×15 pixels, α≥3, and the scaling factor of the first layer is 0.5.

[0076] Output a dense optical flow field V(x,y,t)=(u,v) to capture pixel-level displacement;

[0077] The step of calculating the displacement vector of each point in the time-series point cloud sequence includes:

[0078] Obtain the temporal point cloud sequence and the baseline point cloud model;

[0079] Registering the time-series cloud sequence and the reference point cloud model involves: following a preset time step t. step Iteratively optimize the rotation matrix R and translation vector to minimize the distance between the current point cloud and the reference point cloud model;

[0080] After registration, the displacement vector of each point in the point cloud is d. i (x,y,z,t i ) = p i -(R·p i +t i );

[0081] The method of predicting the deformation of the occluded region using an LSTM-UNet hybrid network based on deformation displacement features and the displacement vector of each point in the point cloud includes:

[0082] Obtain the average displacement of the point cloud in the visible region Where N′ represents the number of points in the visible area point cloud;

[0083] Obtaining the standard deviation of displacement

[0084] Obtain the maximum curvature K′ max ;

[0085] Obtaining optical flow energy W f =∑(u 2 +v 2 );

[0086] Constructing eigenvectors

[0087] Constructing time-series feature sequences The temporal feature sequence is input into a pre-trained LSTM-UNet hybrid network, which outputs the predicted displacement vector of the occluded region. UNet is the decoder;

[0088] After the stacking is completed, the multi-layered packaging boxes are tilted to identify the risk of tipping over, including:

[0089] Constructing damage prediction features in, This represents the displacement vector with the largest magnitude among the predicted displacement vectors;

[0090] A damage prediction model was constructed using a support vector machine (SVM).

[0091] The damage prediction features are input into the pre-trained damage prediction model, which outputs the exit condition judgment value p.out ∈[0,1]; if p out =1, then the goods meet the outbound conditions; if p out If the value is 0, then the goods inside the packaging box are considered damaged and do not meet the conditions for shipment.

[0092] Preferably, after the stacking is completed, a tilt detection is performed on the stacked multi-layer packaging boxes to identify the risk of tipping over, including:

[0093] By combining the deformation data upon receipt, it is determined whether the damage to goods upon departure is related to the deformation data upon receipt. Specifically:

[0094]

[0095] Calculate the centroid C′0 of the packaging box in the stack and the offset ΔC0 from the reference centroid.

[0096] Where, ΔC max θ s These represent the centroid offset threshold and the tilt angle threshold, respectively.

[0097] Obtain the type of surface deformation and identification information of the packaging box upon entry into the warehouse;

[0098] If a package with the same identification information is identified as not meeting the outbound requirements when it leaves the warehouse, it is determined that there is a correlation between the surface deformation of the package when it enters the warehouse and the failure to meet the outbound requirements.

[0099] Obtain the feature vector of each region of the packaging box upon receipt. And mark;

[0100] Using the labeled feature vector To determine whether newly received packaging boxes meet the warehousing conditions, the following steps are taken:

[0101] Let the deformation characteristic vector of the newly received packaging boxes be...

[0102]

[0103] The present invention also provides an artificial intelligence-based visual inspection system for inbound and outbound operations, the system being used to execute the aforementioned artificial intelligence-based visual inspection method for inbound and outbound operations.

[0104] The present invention also provides a computer-readable storage medium storing a computer program, which is executed by a processor to implement the artificial intelligence-based visual inspection method for inbound and outbound operations.

[0105] The beneficial effects of this invention are:

[0106] 1. This invention collects RGB images and point cloud data of the outer packaging box of goods upon entry into the warehouse to determine whether the goods meet the entry conditions (overweight, oversized, etc.). When stacking goods, it detects the tilt angle and center of gravity offset of the goods to determine whether there is a possibility of the goods tipping over.

[0107] 2. In this invention, during outbound shipment, the deformation of the sides, top, and bottom of the packaging box of the goods to be shipped is obtained by using images from various directions. This allows for the prediction of whether the goods inside the packaging box are damaged. If damage is predicted, information indicating that the goods do not meet the outbound conditions is fed back to the warehouse management center, and the picking route is adjusted to avoid the corresponding goods and find alternative goods. Attached Figure Description

[0108] Figure 1 This is a flowchart of the AI-based visual inspection method for warehouse entry and exit according to the present invention. Detailed Implementation

[0109] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will be apparent to those skilled in the art. The basic principles of the invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the invention.

[0110] It is understood that the term "a" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "a" should not be understood as a limitation on the number.

[0111] Example 1:

[0112] refer to Figure 1 The technical solution provided by this invention is: an artificial intelligence-based visual inspection system and method for inbound and outbound operations, comprising the following steps:

[0113] Step 1: Obtain image data for the warehousing stage. The image data for the warehousing stage includes images of items to be warehoused, point cloud images of items to be warehoused, image sequences of robotic arm warehousing operations, and images of items stacked on shelves.

[0114] Step 2: Obtain image data for the outbound stage. The image data for the outbound stage includes images of the object to be outbound, point cloud images of the items to be outbound, and a sequence of images of the robotic arm's outbound operation.

[0115] Before steps 1 and 2, there are also data acquisition and data preprocessing steps, specifically:

[0116] Point cloud data of the packaging box was acquired using a 3D camera;

[0117] The point cloud data is filtered and outliers are removed to form a time-series point cloud sequence.

[0118] RGB image data of packaging boxes from multiple angles within the stacking area are obtained using multi-angle cameras, which cover the top, bottom, and sides of the stacked packaging boxes.

[0119] Pressure data during robotic arm operation is acquired by sensors at the end of the robotic arm and a timestamp of the operation is added. Image data of the front end of the robotic arm is acquired by a camera at the end of the robotic arm. The coordinate system of the robotic arm is aligned with the coordinate system of the multi-angle camera based on a calibration board.

[0120] Step 3: During the warehousing stage, the tilt detection, deformation recognition, and tilt cause analysis of the goods are achieved by analyzing the image data during the warehousing stage. This includes the following sub-steps:

[0121] Step 3.1: Detection of cargo tilting, identification of deformation, and analysis of tilting causes during stacking. This includes the following steps:

[0122] Step 3.1.1: Detect the degree of tilt of the packaging boxes in the stack and locate the tilt point. Specifically:

[0123] Perform Canny edge detection on RGB image data to extract the outline image of the packaging box;

[0124] Perform Hough transform on the outline image of the packaging box to detect straight lines on the edge of the packaging box;

[0125] Calculate the angular deviation of the detected straight line; that is... Wherein, Δy and Δx represent the deviation of the straight line relative to the vertical reference axis and the deviation relative to the horizontal reference axis, respectively;

[0126] Based on point cloud data, the offset between the straight-line position of the packaging box and its geometric center is calculated, specifically:

[0127] Where N represents the number of point clouds, x i y i z i These represent the point cloud coordinates.

[0128] Step 3.1.2: Detect the surface deformation of the packaging box. Specifically, obtain the baseline point cloud model of the undeformed packaging box from the database;

[0129] The Iterative Nearest Point (ICP) algorithm is used to align the current packing box point cloud with the baseline point cloud model; the ICP objective function is: Where R represents the rotation matrix, t i p represents the translation vector. i qi These represent the vectors of corresponding points in the current point cloud and the reference point cloud, respectively.

[0130] Compare the displacement of the current packaging box with the reference point cloud model point by point to obtain multiple displacement amplitude values;

[0131] If the displacement amplitude exceeds the preset displacement threshold, the point is determined to be a deformation point, and multiple deformation points constitute a deformation region.

[0132] The Gaussian curvature value of the deformed region is obtained by local surface bridging calculation;

[0133] After standardizing the obtained displacement amplitude and Gaussian curvature values, they are input into a pre-trained vector machine (SVM) model to output the deformation type of the deformed region; the deformation type includes depression, protrusion, and twist.

[0134] Step 3.1.3, linking the robotic arm operation with the deformation of the packaging box, specifically:

[0135] Obtain the point cloud data acquisition timestamp, RGB image data acquisition time, and robotic arm operation timestamp;

[0136] Align the point cloud data acquisition timestamp, RGB image data acquisition time, and robotic arm operation timestamp on the timeline;

[0137] Obtain the pressure value F of the robotic arm operation corresponding to the timestamp of the robotic arm operation. J ;

[0138] The probability of the association between the robotic arm operation and the deformation of the packaging box is: Where F0 represents the pressure threshold; ROI B ROI J These represent the deformation area of ​​the packaging box and the contact area between the robotic arm and the packaging box, respectively.

[0139] Step 3.2: After stacking, perform a tilt test on the multi-layered packaging boxes to identify the risk of tipping over. This includes the following steps:

[0140] By combining the deformation data upon receipt, it is determined whether the damage to goods upon departure is related to the deformation data upon receipt. Specifically:

[0141]

[0142] Calculate the centroid C′0 of the packaging box in the stack and the offset ΔC0 from the reference centroid.

[0143] Where, ΔC max θ s These represent the centroid offset threshold and the tilt angle threshold, respectively.

[0144] Obtain the type of surface deformation and identification information of the packaging box upon entry into the warehouse;

[0145] If a package with the same identification information is identified as not meeting the outbound requirements when it leaves the warehouse, it is determined that there is a correlation between the surface deformation of the package when it enters the warehouse and the failure to meet the outbound requirements.

[0146] Obtain the feature vector of each region of the packaging box upon receipt. And mark;

[0147] Using the labeled feature vector To determine whether newly received packaging boxes meet the warehousing conditions, the following steps are taken:

[0148] Let the deformation characteristic vector of the newly received packaging boxes be...

[0149]

[0150] For example, during warehousing, displacement of the side of the goods' packaging box is detected. Bottom curvature K Tmax =0.12mm -1 This allows us to determine if the bottom and sides of the packaging box have deformed.

[0151] The labeled feature vector f i =[4.5, 0.01, 0.1, 0.2] indicates that the packaging box of the incoming goods is deformed and does not meet the warehousing requirements. The information indicating that the goods cannot be stored is sent to the warehouse management center. The goods are then placed in a temporary storage area. A positive bottom curvature indicates that the packaging box is bulging downwards, and if this curvature is much greater than 0.01, it suggests that the goods may be overweight.

[0152] Step 4: During the outbound stage, image data is analyzed to detect the degree of deformation of the goods packaging, predict damage, and determine outbound conditions. The outbound strategy is updated based on the damage prediction and outbound condition assessment results. This includes the following steps:

[0153] Step 4.1: Process the stacked goods image using a semantic segmentation network to distinguish between the visible and occluded areas of the packaging boxes; specifically:

[0154] RGB image data of packaging boxes from multiple angles within the stacking area are acquired, and the acquisition timestamps of the RGB images are obtained. The RGB images are then arranged according to the acquisition timestamps to form an RGB image sequence.

[0155] Obtain the image sequence of the robotic arm's warehouse entry operation, and align the RGB image sequence with the images within the robotic arm's warehouse entry operation image sequence on the time axis;

[0156] The occlusions in the RGB image are segmented using a segmentation model to obtain a segmentation mask; the occlusions include packaging boxes and shelves.

[0157] The segmentation mask is projected onto the point cloud of the packaging box, and the occluded areas of the point cloud are marked as invalid areas, while the unoccluded areas of the point cloud are marked as visible areas.

[0158] Predict the deformation of the occluded region based on the deformation characteristics of the visible region.

[0159] Step 4.2: Based on the deformation of the obstructed area, predict cargo damage and determine whether the outbound conditions are met based on the prediction results. If so, generate a scheduling strategy and dynamically plan a new pickup route; otherwise, maintain the original pickup route. This includes:

[0160] The warehouse shelving is abstracted as a graph G = (V, E); where node V represents a storage location and edge E represents a feasible path.

[0161] A new pickup route is calculated using the A* algorithm to avoid goods that do not meet the outbound conditions.

[0162] For example, the goods to be shipped are the goods in box 002 on the shelf.

[0163] The damage prediction feature input to the damage prediction model is s = [10, -0.2, 86], i.e. K′ max = -0.2mm -1 F J =86N;

[0164] The damage prediction model outputs 0, indicating that the goods do not meet the outbound requirements. This information is then fed back to the warehouse management center.

[0165] The A* algorithm is used to redesign a new pickup route, bypassing box 002, and selecting alternative goods.

[0166] Example 2:

[0167] In some application scenarios, when detecting surface deformation of packaging boxes, it is necessary to determine the specific location of the deformation, namely the side, top, and bottom.

[0168] Therefore, based on Embodiment 1, we propose the following technical solution:

[0169] Detecting surface deformation of packaging boxes also includes:

[0170] The point cloud of the packaging box was segmented into side, top, and bottom regions, and corresponding region identification tags were added to each region. Specifically:

[0171] The principal orientation of the packaging box is calculated using the principal component analysis (PCA) algorithm.

[0172] The major axis direction is represented by vector v1, the width axis direction by vector v2, and the height axis direction by vector v3.

[0173] Take the top 10% of the point cloud region along the v3 direction as the top region, and take the bottom 10% of the point cloud region along the opposite direction of v3 as the bottom region.

[0174] The point cloud of the packaging box is divided into four side regions using v1 and v2;

[0175] Perform local surface fitting on each region and calculate the Gaussian curvature K′ and mean curvature H′ for each region;

[0176] Where λ1, λ2, and A represent the principal curvature of local surface one to be fitted, the principal curvature of local surface two, and the neighborhood area, respectively.

[0177] The volume of the point cloud in each region is calculated using the convex hull algorithm. The calculated volume of each region is then compared with the volume of the corresponding region in the reference point cloud to obtain the volume difference ΔV. i =|V i -V i0 |;Among them, V i V represents the point cloud volume of region i. i0 This represents the reference volume of region i corresponding to the reference point cloud;

[0178] The translation vector t of each point in the point cloud of each region is obtained using the ICP algorithm. j =p ij -q ij Calculate the average displacement value of the region Where, m is the number of points within the region; p ij q ij Let represent the vector of point j in the point cloud of region i and the vector of point j in the reference point cloud corresponding to region i, respectively.

[0179] Obtain the maximum Gaussian curvature K′ for each region max Maximum mean curvature H′ max Regional average displacement d and volume difference ΔV i After standardization, feature vectors for each region are constructed.

[0180] The feature vector of each region is input into a pre-trained random forest model, and the deformation type label of each region is output; the type label includes category 1, category 2 and category 3;

[0181] Type 1 is lateral deformation, that is, the average displacement value of the lateral region is greater than a preset lateral deformation threshold and the maximum Gaussian curvature of the lateral region is greater than a preset lateral deformation curvature threshold.

[0182] Type 2 is top deformation, that is, the volume difference of the top region is greater than the preset region volume difference threshold;

[0183] Type 3 is bottom deformation, that is, the average displacement value of the bottom region is greater than the preset bottom deformation threshold and the maximum Gaussian curvature of the bottom region is greater than the preset bottom deformation curvature threshold.

[0184] Example 3:

[0185] When there are multiple layers of goods on a shelf, each layer of goods will be squeezed by the upper and / or lower layers, causing deformation of the outer packaging. In this case, because bottom images are not available, a complete picture of the product's condition cannot be obtained, making it impossible to determine whether the internal goods are damaged due to compression. Therefore, based on Embodiment 2, a technical solution is proposed that can deduce the deformation of the obscured area using known image data. Specifically:

[0186] Deformation displacement features are captured by pixel motion between consecutive image frames. This includes the following steps:

[0187] Dynamic deformation features of visible regions are extracted from RGB image sequences. These dynamic deformation features include one or more of the following: translation of each point within the point cloud of each visible region, Gaussian curvature of each region, average curvature, and texture features; specifically:

[0188] Establish the optical flow equation I x u+I y v+I t =0, where I x I y I t represents the spatial gradient of the image along the x and y axes, respectively; t represents the temporal gradient of the image along the t-axis.

[0189] For each pixel's neighborhood, an overdetermined equation is constructed, and the least squares solution is:

[0190] Where B and b represent the gradient matrix and temporal gradient vector of all pixels in the neighborhood, respectively; u and v represent the instantaneous displacement velocities of pixels in the RGB image along the x and y axes in space.

[0191] The Gaussian pyramid algorithm is used to generate an α-layer multi-scale image, and optical flow is calculated layer by layer to solve the large displacement problem; the neighborhood size is 15×15 pixels, α≥3, and the scaling factor of the first layer is 0.5.

[0192] Output a dense optical flow field V(x,y,t)=(u,v) to capture pixel-level displacement.

[0193] The displacement vector of each point in the point cloud is calculated using a time-series point cloud sequence, including the following steps: obtaining the time-series point cloud sequence and the reference point cloud model;

[0194] Registering the time-series cloud sequence and the reference point cloud model involves: following a preset time step t. step Iteratively optimize the rotation matrix R and translation vector to minimize the distance between the current point cloud and the reference point cloud model;

[0195] After registration, the displacement vector of each point in the point cloud is d. i (x,y,z,t i ) = p i -(R·p i +t i );

[0196] The method of predicting the deformation of the occluded region using an LSTM-UNet hybrid network based on deformation displacement features and the displacement vector of each point in the point cloud includes:

[0197] Obtain the average displacement of the point cloud in the visible region Where N′ represents the number of points in the visible area point cloud;

[0198] Obtaining the standard deviation of displacement

[0199] Obtain the maximum curvature K′ max ;

[0200] Obtaining optical flow energy W f =∑(u 2 +v 2 );

[0201] Constructing eigenvectors

[0202] Constructing time-series feature sequences The temporal feature sequence is input into a pre-trained LSTM-UNet hybrid network, which outputs the predicted displacement vector of the occluded region. UNet is the decoder.

[0203] Based on deformation displacement features and the displacement vector of each point in the point cloud, an LSTM-UNet hybrid network is used to predict the deformation of the occluded region. This includes the following steps:

[0204] Constructing damage prediction features in, This represents the displacement vector with the largest magnitude among the predicted displacement vectors;

[0205] A damage prediction model was constructed using a support vector machine (SVM).

[0206] The damage prediction features are input into the pre-trained damage prediction model, which outputs the exit condition judgment value p. out ∈[0,1]; if p out =1, then the goods meet the outbound conditions; if p out If the value is 0, then the goods inside the packaging box are considered damaged and do not meet the conditions for shipment.

[0207] For example, if goods on the left side of the second layer are obstructed, it is necessary to predict the deformation of the obstructed surface. First, the operating pressure of the robotic arm when the goods were put into storage is obtained from the database, and image data is acquired at a sampling frequency of 30Hz.

[0208] Optical flow data from the first 4 seconds was extracted, and the average displacement of the visible region increased from 0.5 mm to 3.2 mm. The maximum curvature of the visible region was -0.12 mm. -1 By inputting the features from the first 4 seconds into LSTM-UNet, the displacement of the invisible region in the 5th second is predicted to be 4.1 mm.

[0209] The predicted displacement, robotic arm operating pressure, and maximum curvature are input into the damage prediction model. If the output result is 1, it is determined that the goods are not damaged and meet the conditions for leaving the warehouse.

[0210] The damage prediction model is trained using historical database data, including the deformation of each side of the packaging box when the goods are damaged and the pressure value of the robotic arm when the goods are put into storage. This allows the model to learn the relationship between the damage to the goods and the deformation and the pressure of the robotic arm, and to predict whether the goods are damaged.

[0211] The present invention also provides an artificial intelligence-based visual inspection system for inbound and outbound operations, the system being used to execute the aforementioned artificial intelligence-based visual inspection method for inbound and outbound operations.

[0212] The present invention also provides a computer-readable storage medium storing a computer program, which is executed by a processor to implement the artificial intelligence-based visual inspection method for inbound and outbound operations.

[0213] The processes described above with reference to the flowcharts in the embodiments disclosed in this invention can be implemented as computer software programs. The embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), it performs the functions defined in the methods of this application. It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wire segments, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless segments, wire segments, optical fibers, RF, etc., or any suitable combination thereof.

[0214] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0215] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functions and structural principles of the present invention have been shown and explained in the embodiments. Without departing from the principles described, the implementation of the present invention may have any changes or modifications.

Claims

1. An AI-based warehouse access visual detection method, characterized in that, The method comprises: acquiring warehouse-in stage image data, the warehouse-in stage image data comprising a to-be-warehoused article image, a to-be-warehoused article point cloud image, a mechanical arm warehouse-in operation image sequence, and a goods shelf article stacking image; acquiring warehouse-out stage image data, the warehouse-out stage image data comprising a to-be-warehoused article image, a to-be-warehoused article point cloud image, and a mechanical arm warehouse-out operation image sequence; in the warehouse-in stage, tilt detection, deformation recognition, and tilt cause analysis of the goods are achieved by analyzing the warehouse-in stage image data, comprising the following sub-steps: tilt detection, deformation recognition, and tilt cause analysis of the goods in the stacking process; after the stacking is completed, tilt detection is performed on the stacked multi-layer packaging boxes to identify the risk of dumping; in the warehouse-out stage, deformation degree detection, damage prediction, and warehouse-out condition judgment of the goods packaging are achieved by analyzing the warehouse-out stage image data, and the warehouse-out strategy is updated according to the damage prediction and the warehouse-out condition judgment result, comprising: processing the stacked goods image through a semantic segmentation network to distinguish the visible area and the occluded area of the packaging box; specifically: acquiring RGB image data of the packaging box at multiple angles in the stacking area, and acquiring the acquisition time stamp of the RGB image, arranging the RGB images according to the acquisition time stamp to form an RGB image sequence; acquiring the mechanical arm warehouse-in operation image sequence, and aligning the RGB image sequence with the images in the mechanical arm warehouse-in operation image sequence on the time axis; segmenting the occlusion in the RGB image using a segmentation model to obtain a segmentation mask; the occlusion includes the packaging box and the goods shelf; projecting the segmentation mask to the point cloud of the packaging box, marking the occluded area of the point cloud as an invalid area, and marking the non-occluded area of the point cloud as a visible area; predicting the deformation of the occluded area based on the deformation features of the visible area, comprising: capturing deformation displacement features through the pixel motion between consecutive image frames; calculating the displacement vector of each point of the point cloud through the time sequence point cloud sequence; predicting the deformation of the occluded area based on the deformation displacement features and the displacement vector of each point of the point cloud using an LSTM-UNet hybrid network; based on the deformation of the occluded area, predicting the damage of the goods, and judging whether the warehouse-out condition meets the requirements according to the prediction result; if yes, generating a scheduling strategy and dynamically planning a new goods picking path; otherwise, keeping the original goods picking path; the dynamic planning of the new goods picking path comprises: The warehouse racks are abstracted as a graph ; wherein the nodes represent the storage locations, and the edges represent feasible paths; calculating a new goods picking path by A* algorithm to avoid goods that do not meet the warehouse-out condition. 2.The AI-based warehouse access visual detection method according to claim 1, characterized in that, Before the tilt detection, deformation recognition, and tilt cause analysis of the goods in the stacking process, comprising: acquiring point cloud data of the packaging box through a 3D camera; performing filtering processing on the point cloud data to remove outliers to form a time sequence point cloud sequence; acquiring RGB image data of the packaging box at multiple angles in the stacking area through multi-angle cameras; acquiring pressure data when the mechanical arm operates through a mechanical arm end sensor and adding a mechanical arm operation time stamp, and acquiring mechanical arm front end image data through a mechanical arm end camera; aligning the mechanical arm coordinate system with the multi-angle camera coordinate system based on a calibration board; the tilt detection, deformation recognition, and tilt cause analysis of the goods in the stacking process, comprising: Detecting the degree of tilt of the packaging box in the stack and locating the tilt point; Detecting the surface deformation of the packaging box; Correlating the operation of the mechanical arm with the deformation of the packaging box. 3.The AI-based warehouse entry and exit visual detection method according to claim 2, characterized in that, The detecting the degree of tilt of the packaging box in the stack and locating the tilt point comprises the following sub-steps: Performing Canny edge detection on the RGB image data to extract a packaging box contour image; Performing Hough transform on the packaging box contour image to detect the edge straight line of the packaging box; calculating an angular deviation for the detected straight line; i.e. wherein, respectively represent the deviation of the straight line with respect to a vertical reference axis and with respect to a horizontal reference axis. Based on the point cloud data, calculating the offset of the packaging box straight line position and the geometric center, specifically: wherein, denotes the number of point clouds, denotes the point cloud coordinates, respectively; The detecting the surface deformation of the packaging box comprises the following sub-steps: Obtaining a reference point cloud model of an undeformed packaging box from a database; The current packaging box point cloud is aligned with the reference point cloud model using an iterative closest point (ICP) algorithm; an ICP objective function is: ; wherein, represents a rotation matrix, represents a translation vector, respectively represent vectors of corresponding points in the current point cloud and the reference point cloud. Comparing the displacement of the current packaging box with the reference point cloud model point by point to obtain a plurality of displacement amplitude values; If the displacement amplitude value exceeds a preset displacement threshold value, the point is judged to be a deformation point, and a plurality of deformation points constitute a deformation region; Obtaining the Gaussian curvature value of the deformation region through local surface closure calculation; After standardizing the obtained displacement amplitude value and Gaussian curvature value, inputting them into a pre-trained support vector machine SVM model to output the deformation type of the deformation region; the deformation type includes concave, protruding and twisting; The correlating the operation of the mechanical arm with the deformation of the packaging box comprises: Obtaining the point cloud data acquisition timestamp, the RGB image data acquisition time and the mechanical arm operation timestamp; Aligning the point cloud data acquisition timestamp, the RGB image data acquisition time and the mechanical arm operation timestamp on the time axis; acquiring a pressure value of the robot operation corresponding to the robot operation timestamp ; The probability of the association between the mechanical arm operation and the deformation of the packaging box is ; wherein, represents a pressure threshold value; respectively represent a deformation region of the packaging box and a contact region between the mechanical arm and the packaging box. 4.The AI-based warehouse entry and exit visual detection method according to claim 3, characterized in that, The detecting the surface deformation of the packaging box further comprises: Segmenting the packaging box point cloud into a side region, a top region and a bottom region, and adding corresponding region identification labels to the side region, the top region and the bottom region, specifically: By the principal component analysis, PCA, algorithm, the main directions of the packaging box are calculated, i.e. by the vector denoting the long axis direction, by the vector denoting the wide axis direction and by the vector denoting the high axis direction; will along The top 10% of the point cloud region is taken as the top region, and along... Take 10% of the point cloud area in the opposite direction as the bottom area; By and The package box point cloud is divided into four side regions; Local surface fitting is performed for each region, and the Gaussian curvature of each region is calculated and the mean curvature ; , ; wherein, respectively denote the principal curvatures of the local surface one and the local surface two and the neighborhood area to be fitted. The volume of each region point cloud is calculated by a convex hull algorithm, and the volume difference is obtained by comparing the volume of each region point cloud with the volume of the corresponding region of the reference point cloud ; wherein represents the volume of the region point cloud, represents the reference volume of the corresponding region of the reference point cloud; The translation vectors of each point in the point cloud of each region are obtained by the ICP algorithm , the regional average displacement value is calculated ; wherein, the number of points in the region; respectively represent the vector of the point in the point cloud of the region and the vector of the point corresponding to the region in the reference point cloud . maximum gaussian curvature of each region , maximum average curvature , region average displacement value , and volume difference , after standardization, the feature vector of each region is constructed ; Inputting the feature vector of each region into a pre-trained random forest model to output the deformation type label of each region; the type label includes category 1, category 2 and category 3; The type 1 is side deformation, that is, the region average displacement value of the side region is greater than a preset side deformation threshold value and the maximum Gaussian curvature of the side region is greater than a preset side deformation curvature threshold value; The type 2 is top deformation, that is, the volume difference of the top region is greater than a preset region volume difference threshold value; The type 3 is bottom deformation, that is, the average displacement value of the bottom region is greater than a preset bottom deformation threshold value and the maximum Gaussian curvature of the bottom region is greater than a preset bottom deformation curvature threshold value. 5.The AI-based warehouse entry and exit visual detection method according to claim 4, characterized in that, The capturing deformation displacement features through the pixel motion between consecutive image frames comprises: Extracting dynamic deformation features of the visible region from the RGB image sequence, the dynamic deformation features including one or more of the translation of each point in each visible region point cloud, the Gaussian curvature of each region, the average curvature and the texture features; specifically: establishing an optical flow equation wherein, respectively represent spatial gradients of the image in the spatial axis direction; represent a temporal gradient of the image in the temporal axis direction; Constructing an over-determined equation for each pixel neighborhood, and the least squares solution is: wherein, respectively represent the gradient matrix and the temporal gradient vector of all pixels within the neighborhood; represents the instantaneous displacement velocity of the RGB image pixel in the spatial axis direction; By the Gaussian pyramid algorithm, generate Layer multiscale image, and calculate the optical flow layer by layer to solve the large displacement problem; the neighborhood size is 15x15 pixels, , and the first layer scaling factor is 0.5; Output dense optical flow field to capture pixel-level displacements; The calculating the displacement vector of each point of the point cloud through the time series point cloud sequence comprises: Obtaining the time series point cloud sequence and the reference point cloud model; registering the time-sequential cloud sequence and the fiducial point cloud model, specifically, iteratively optimizing a rotation matrix and a translation vector to minimize a distance between a current point cloud and the fiducial point cloud model according to a preset time step ​​ After registration, the displacement vector of each point of the point cloud is ; The predicting the occlusion region deformation based on the deformation displacement features and the displacement vector of each point of the point cloud using an LSTM-UNet hybrid network comprises: Obtaining average displacement of point cloud of visible region ; wherein, denotes the number of points of the visible region point cloud; Acquiring displacement standard deviation ; acquiring maximum curvature ; acquiring optical flow energy ; constituent feature vector ; Constructing a time-series feature sequence The time-series feature sequence is input as an input variable into a pre-trained LSTM-UNet hybrid network, and a predicted displacement vector of the occluded region is output ; wherein the UNet is a decoder After the stacking is completed, the multi-layer packaging boxes of the stack are subjected to inclination detection to identify a tipping risk, including: Constructing damage prediction features ; wherein, denotes the displacement vector with the largest magnitude in the prediction displacement vector; An injury prediction model is constructed by using a support vector machine (SVM); The damage prediction feature is input into a pre-trained damage prediction model, and an outbound condition judgment value is output If , it is judged that the goods meet the outbound condition; if , it is judged that the goods in the packaging box are damaged and do not meet the outbound condition. 6.The AI-based warehouse entry and exit visual detection method according to claim 5, characterized in that, After the stacking is completed, the multi-layer packaging boxes of the stack are subjected to inclination detection to identify a tipping risk, including: The correlation between the damage of the goods when the goods are delivered out of the warehouse and the deformation data when the goods are delivered into the warehouse is determined by combining the deformation data when the goods are delivered into the warehouse, and specifically: acquisition ; Computing the center of mass of a package of boxes and offset from a reference center of mass ; pouring probability wherein, respectively represent the centroid shift threshold and the tilt angle threshold. The type and identification information of the surface deformation of the packaging box when the goods are delivered into the warehouse are obtained; If the packaging box with the same identification information is identified as not meeting the delivery requirements when the goods are delivered out of the warehouse, it is determined that there is a correlation between the surface deformation of the packaging box when the goods are delivered into the warehouse and the delivery requirements not being met. Acquiring feature vectors of each area of a packing box at the time of warehousing and marking; Utilizing the labeled feature vectors whether the newly-arrived packaging box meets the storage condition, specifically: Set the deformation feature vector of the newly-warehoused packaging box ; 。 7. The warehouse access visual detection system based on artificial intelligence, characterized in that, The system is used to execute the artificial intelligence-based visual detection method for delivery into and out of the warehouse according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the artificial intelligence-based visual detection method for delivery into and out of the warehouse according to any one of claims 1-6.

Images