A goods quantity identification and inventory method and system based on instance segmentation

Abstract

The application provides a kind of instance segmentation-based cargo quantity identification inventory method and system, including collecting cargo picture dataset and carrying out labelization processing;Multiple instance segmentation models are trained;Cargo image is identified and segmented using instance segmentation model, to obtain the segmentation mask and mask category information of cargo image;According to the mask quantity and mask category information of cargo image, it is determined whether it is a legal case;And determine whether the top layer of goods is full load, if full load, the total cargo quantity is directly calculated, otherwise, based on the area ratio, the box quantity of any layer of goods in the non-top layer is estimated, and then the total cargo box quantity is calculated.The application uses instance segmentation model to realize the identification of goods in complex stacking scene, and generates goods segmentation mask and mask type, and the total cargo quantity is calculated by identifying and distinguishing different layers of goods boxes;Can directly carry out accurate quantity statistics in the case of full top layer of goods.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of goods inventory, and particularly relates to a goods quantity recognition and inventory method and system based on instance segmentation. BACKGROUND

[0002] With the rapid development of the logistics industry, warehouse management plays an increasingly important role in the supply chain. In order to improve the utilization efficiency of the warehouse and reduce the labor cost, automated goods management and inventory technology has gradually become a research hotspot. In the warehouse environment, the number of goods boxes directly affects the utilization rate of the warehouse space and the efficiency of goods allocation. Therefore, how to efficiently and accurately count the number of goods boxes has become an important problem in current warehouse management.

[0003] At present, the intelligent inventory method for warehouse goods quantity mainly includes two types. The first type of method is based on RFID (Radio Frequency Identification) recognition and counting technology. This method installs RFID tags on goods, uses radio wave readers to identify each tag and obtains real-time goods quantity information. The inventory efficiency of RFID technology is high, and it does not depend on manual intervention. However, due to the high implementation and replacement cost of RFID tags, its popularity in large warehouse environments is limited.

[0004] The second type of method is to use UAV (Unmanned Aerial Vehicle) or AGV (Automated Guided Vehicle) combined with identification codes for inventory. This method has a lower cost than RFID, and can reduce labor costs to a certain extent by using automated equipment to read the barcodes or QR codes of goods. However, this method also depends on electronic tags, has high requirements for the integrity of the identification code, and may be affected by factors such as obstruction in complex warehouse environments, affecting the accuracy of identification.

[0005] In addition to the tag-based inventory method, some factories also use sensor-based measurement methods to achieve inventory by comparing the state before and after the goods are stacked, such as a scheme combining visual perception and weight sensors. Although these methods have certain effects in specific environments, their applicability is limited and cannot meet the needs of warehouse with various complex stacking forms.

[0006] In recent years, with the development of computer vision and deep learning technologies, inventory counting tasks have gradually become possible by using visual algorithms to analyze images of stacked goods in warehouse environments, achieving the goal of automatically counting the number of goods. For example, some researchers have proposed an inventory counting method based on target detection and post-processing modules, which can accurately count the number of goods in a three-dimensional warehouse environment with good lighting conditions. However, in actual warehouse environments, the stacking structure of goods is complex and varied, especially in dark warehouses. Existing visual algorithms have difficulty accurately identifying and counting goods. Dark warehouses usually only turn on the lighting near the camera during inventory counting, which leads to a significant increase in factors such as shadows and uneven lighting in the image, posing a huge challenge to inventory counting algorithms.

[0007] While existing vision-based object counting methods have addressed the label dependency issue to some extent, they still face significant limitations in complex warehouse environments. First, the diverse stacking patterns of goods, especially when top-level items are scattered or have gaps, make it difficult for traditional object detection algorithms to accurately count. Furthermore, in real-world warehouses, environmental factors such as camera angle, lighting, and occlusion can affect the stability of segmentation and detection results, further impacting counting accuracy. Summary of the Invention

[0008] In view of the shortcomings of the existing technology, the present invention provides a method and system for identifying and counting the quantity of goods based on instance segmentation. The present invention identifies the quantity of goods on the top layer and non-top layer according to different stacking situations.

[0009] The technical solution of the present invention is: a method for identifying and counting the quantity of goods based on instance segmentation, comprising the following steps:

[0010] S1) Collecting a cargo image dataset and labeling the cargo image dataset;

[0011] S2), using the cargo image dataset in step S1) to train multiple instance segmentation models;

[0012] S3) using the trained instance segmentation model to identify and segment the cargo image, and obtain the segmentation mask and mask category information of the cargo image;

[0013] S4), based on the number of masks and mask category information of the cargo image, preliminarily determine whether it is legal. If illegal, report the data; otherwise, go to step S5);

[0014] S5) Calculate the convex hull of the mask, and determine whether the top layer of the cargo is fully loaded based on the geometric relationship of the points in the convex hull. If it is fully loaded, directly calculate the total quantity of cargo; otherwise, proceed to step S6);

[0015] S6), according to the geometric relationship of the mask convex hull, the number of the area ratio is used to estimate the number of the box of any layer of the non-top layer of the goods, and then the total number of the box of the goods is calculated.

[0016] Preferably, in step S1), the labeling processing refers to marking each box of the goods on the top layer of the goods image as upbox and as category 1, and marking each layer of the non-top layer as downbox and as category 0.

[0017] Preferably, in step S2), the instance segmentation model adopts a Mask2former model, or a SOLOv2 model, a QueryInst model, or a SparseInst model.

[0018] Preferably, in step S3), the instance segmentation model identifies the target in the goods image and generates a mask for each object, and classifies each mask as upbox or downbox, and the value of the mask matrix is 1 or 0, 0 indicating that the pixel point has no mask, and 1 indicating that the pixel point has a mask.

[0019] Preferably, in step S4), when the number of the mask of the downbox category is 0, it is considered as an illegal case.

[0020] Preferably, in step S5), it is determined whether the top layer of the goods is full, which specifically includes the following steps:

[0021] S51), according to the mask of the goods and the mask type, a convex hull HM corresponding to all masks is calculated, wherein the convex hull is a geometry of a plurality of two-dimensional points;

[0022] S52), selecting a mask of the upbox category closest to the bottom of the convex hull point of the goods image, and calculating a center point center_point of the convex hull corresponding to the mask;

[0023] S53), selecting a convex hull corresponding to the mask of the downbox category closest to the top of the image, and calculating a highest point ymin_point of the convex hull;

[0024] S54), comparing the y value of the center point center_point of the convex hull in step S53) with the y value of the highest point ymin_point of the convex hull in step S53), if the y value of the center point center_point of the convex hull in step S53) is less than the y value of the highest point ymin_point of the convex hull in step S53), it is determined that the current case is that the top layer of the goods is full, otherwise it is not full.

[0025] Preferably, in step S5), if only the mask of the upbox category exists, it is considered that the top layer of the goods is full.

[0026] As preferred, in step S6), the number of the whole layer of boxes in the non-top layer is estimated by area ratio-based quantity estimation, which specifically includes the following steps:

[0027] S61), according to the type of the mask, the convex hull set HM of all masks calculated in step S5) is divided into UH and DH, wherein UH is the convex hull set of the mask upbox, and DH is the convex hull set of the mask downbox;

[0028] S62), for a single top layer box, its upper surface area S1 is estimated;

[0029] S63), the upper surface area S2 of the whole layer of box in the non-top layer is estimated;

[0030] S64), according to the ratio of the upper surface area S2 of the whole layer of box to the upper surface area S1 of the single top layer box, and then rounding down, the estimated value N of the whole layer of box is obtained layer ;

[0031] S65), the total number of goods N total is calculated, i.e.:

[0032] N total = N downbox *N layer +N upbox

[0033] In the formula, N downbox , N upbox are the number of masks of categories 0 and 1 respectively.

[0034] As preferred, in step S62), the following steps are specifically included:

[0035] S621), in the convex hull set UH, find the convex hull y m_HU with the largest y value, and sort all points in the convex hull y m_HU in ascending order according to the x value, if the x value is the same, sort the points in ascending order according to the y value and prefer the point with smaller y value; mark the first sorted point as point A;

[0036] S622), calculate the point closest to the lower left corner of the goods image in the convex hull y m_HU and mark it as point B using Euclidean distance; calculate the points closest to the right lower corner and the right upper corner of the goods image and mark them as points C and E respectively; meanwhile, find the point with the smallest y value and the smallest x value in the convex hull y m_HU and mark it as point D;

[0037] S623), calculate the center point of the convex hull y m_HU ; and filter out the convex hull ym_HU points with y value greater than the center point y value, and put all filtered points into set below_center;

[0038] S624), sort the points in set below_center in descending order of x value, and if x value is the same, sort in descending order of y value, then mark the first sorted point as point F;

[0039] S625), calculate the intersection points according to the slope relationship between the points, and take the average position of the intersection points as the final intersection point O;

[0040] S626), estimate the upper surface area S1 of a single top layer box by calculating the product of the lengths of line segments OA and OE.

[0041] As a preferred, in step S625), first determine a first straight line, the first straight line passes through points D and E, and the slope of the first straight line is determined by the line connecting points A and C;

[0042] Secondly, determine a second straight line, the second straight line passes through points F and E, and the slope is determined by the line connecting points A and E;

[0043] Thirdly, determine a third straight line, the third straight line passes through points E and F, and the slope is determined by the line connecting points C and F;

[0044] The intersection points of the three straight lines are O1, O2 and O3 respectively; take the average position of O1, O2 and O3 as the final intersection point O.

[0045] As a preferred, in step S63), estimate the upper surface area S2 of a whole layer of goods boxes, which specifically includes the following steps:

[0046] S631), calculate the convex hulls y m_hull , y n_hull in set DH with the maximum y value and the minimum y value;

[0047] S632), calculate points A', B', C' and E' in set DH according to the method of steps S621)-S622); and when convex hulls y m_hull , y n_hull are different convex hulls, calculate the point closest to the right lower corner of the image in convex hull y n_hull , and mark it as point D';

[0048] S633), determine the intersection point position of the key straight line according to the slope relationship between the points, and calculate the final intersection point O; then estimate the upper surface area S2 of a whole layer of boxes by calculating the product of the lengths of line segments OA and OE.

[0049] As preferred, the application also provides a goods quantity identification inventory system based on instance segmentation, comprising:

[0050] An acquisition module, configured to acquire a goods image;

[0051] A labeling module, configured to mark each goods box on the top layer in the goods image as an upbox and as category 1, and mark each layer other than the top layer as a downbox and as category 0;

[0052] An image segmentation identification module, configured to identify and segment the goods image by using a pre-trained instance segmentation model to obtain a segmentation mask and mask category information of the goods image;

[0053] A goods quantity estimation module, configured to determine different scenarios according to the image mask and mask category information and estimate the goods quantity based on area ratio.

[0054] As preferred, the goods quantity estimation module preliminarily determines whether it is a legal case according to the mask quantity and mask category information of the goods image, and reports the data if it is illegal; otherwise, the convex hull of the mask is calculated, the geometric relationship of the points in the convex hull is determined, and whether the top layer of the goods is full is determined if it is full, the total goods quantity is directly calculated, otherwise, the number of boxes in any layer of the non-top layer of the goods is estimated based on the area ratio according to the geometric relationship of the convex hull of the mask, and then the total number of boxes of the goods is calculated.

[0055] The application has the following beneficial effects:

[0056] 1. The application realizes the identification of goods in a complex stacking scenario by using an instance segmentation model, generates a goods segmentation mask and mask type, and calculates the total goods quantity by identifying and distinguishing different layers of goods boxes;

[0057] 2. The application calculates the goods quantity based on the area ratio method for different stacking conditions, which is not only suitable for the case of scattered distribution of the top layer of goods, but also can directly and accurately count the quantity in the case of full stacking of the top layer of goods;

[0058] 3. The application has high accuracy in goods counting in a complex warehouse environment based on the area ratio estimation method of the mask, does not need to rely on electronic tags or sensor data, reduces the inventory cost, and is particularly suitable for dark warehouses and other warehouse environments with poor light conditions. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 The figure is a flowchart of the method of the application;

[0060] Figure 2Fig. 1 is a schematic diagram of the determination of whether the top layer of goods is full or not according to the present application, wherein (a) and (b) are schematic diagrams of the cases of not full and full, respectively;

[0061] Figure 3 Fig. 2 is a schematic diagram of the estimation of the upper surface area of a single top layer box according to the present application, wherein (a) is a mask combination schematic diagram of a real picture in an actual scene, and (b) is a selected upbox mask schematic diagram;

[0062] Figure 4 Fig. 3 is a schematic diagram of the estimation of the upper surface area of a whole layer of goods according to the present application, wherein (a) is a mask combination schematic diagram of a real picture in an actual scene, and (b) is a selected downbox mask schematic diagram. DETAILED DESCRIPTION

[0063] The specific embodiments of the present application will be further described below in conjunction with the accompanying drawings:

[0064] Embodiment 1

[0065] As shown in the drawings, the present embodiment provides a goods quantity identification and inventory method based on instance segmentation, comprising the following steps: Figure 1

[0066] S1), high-definition cameras are used to collect goods picture data sets, and the goods picture data sets are labeled and processed; and the labeled and processed goods picture data sets are divided into a training set and a test set according to a certain proportion;

[0067] In the present embodiment, the labeling and processing refers to marking each goods box on the top layer in the goods image as upbox and as class 1, and marking each layer other than the top layer as downbox and as class 0. S2), a plurality of instance segmentation models are trained using the training set in the goods picture data set in step S1);

[0068] In the present embodiment, the instance segmentation model uses a Mask2former model, or a SOLOv2 model, a QueryInst model, or a SparseInst model.

[0069] During the training process, 300 epochs are continuously used. The Mask2former model is optimized using an AdamW optimizer, the learning rate is initialized to 0.0001, the weight decay factor is 0.05, the momentum coefficient of the optimizer is (0.9, 0.999), and the batch size is 4.

[0070] ​The SOLOv2 model is optimized using a Stochastic Gradient Descent (SGD) optimizer with a learning rate of 0.01, a momentum factor of 0.9, a weight decay of 0.0001, and a batch size of 8.

[0071] The QueryInst model is optimized using an AdamW optimizer with a learning rate initialization of 0.0001, a weight decay factor of 0.0001, and a batch size of 8.

[0072] The SparseInst model is optimized using an AdamW optimizer with a learning rate initialization of 0.00005, a weight decay factor of 0.05, and a batch size of 8.

[0073] To improve generalization, during training, the image size is randomly adjusted while ensuring the image ratio, and random horizontal flipping is used for data augmentation.

[0074] S3), according to different scenarios, different trained instance segmentation models are used to recognize and segment the cargo image, to obtain the segmentation mask and mask category information of the cargo image;

[0075] In this embodiment, the instance segmentation model identifies the target in the cargo image and generates a mask for each object, and classifies each mask as upbox or downbox, and the mask matrix value is 1 or 0, 0 indicating that the pixel point has no mask, and 1 indicating that the pixel point has a mask.

[0076] S4), according to the mask quantity and mask category information of the cargo image, it is preliminarily determined whether it is a legal case, if not, report this data; in this embodiment, when the number of downbox masks is 0, it is considered as an illegal case, otherwise, step S5) is entered;

[0077] S5), calculate the convex hull of the mask, and determine whether the top layer of the cargo is full according to the geometric relationship of the points in the convex hull, if it is full, the total number of cargos is directly calculated, otherwise, step S6) is entered; whether the top layer of the cargo is full is determined, as shown in (a) and (b) of FIG. 1, which respectively shows the case of not full and the case of full, which specifically includes the following steps: Figure 2

[0078] S51), according to the mask and mask type of the cargo, calculate the convex hull HM corresponding to all masks, wherein the convex hull is the geometry of multiple two-dimensional points;

[0079] ​S52), select the mask of the upbox category whose convex hull point is closest to the bottom of the cargo image, and calculate the center point center_point of the convex hull corresponding to the mask;

[0080] S53), select the corresponding convex hull of the mask with the downbox category closest to the top of the image, and calculate the highest point ymin_point of the convex hull;

[0081] S54) Compare the y value of the center point center_point of the convex hull in step S53) with the y value of the highest point ymin_point of the convex hull in step S53). If the y value of the center point center_point of the convex hull in step S53) is less than the y value of the highest point ymin_point of the convex hull in step S53), it is determined that the current situation is that the top layer of the cargo is fully loaded, otherwise it is not fully loaded.

[0082] In addition, in this embodiment, if there is only a mask with the category of upbox, it is also considered that the top floor of the cargo is fully loaded.

[0083] If the top layer of cargo is fully loaded, the instance segmentation model identifies the segmented mask upbox as the number of cargo boxes on the entire layer, and then obtains the total number of cargo through the number of masks downbox.

[0084] S6) According to the geometric relationship of the mask convex hull, the number of boxes of any layer of goods other than the top layer is estimated based on the number of area ratios, and then the number of boxes of the total goods is calculated, which specifically includes the following steps:

[0085] S61), according to the type of mask, divide the convex hull set HM of all masks calculated in step S5) into UH and DH, where UH is the convex hull set of the mask upbox and DH is the convex hull set of the mask downbox;

[0086] S62), for a single top box, estimate its upper surface area S1; Figure 3 As shown, the specific steps include:

[0087] S621) Find the convex hull y with the maximum y value in the convex hull set UH m_HU , and the convex hull y m_HU Sort all points in ascending order by the x-axis value. If the x-values ​​are the same, sort them by the y-axis value and give priority to the point with the smaller y-value. Mark the first-ranked point as point A.

[0088] S622) Calculate the convex hull y using the Euclidean distance m_HU The point closest to the lower left corner of the product image is marked as B; the points closest to the lower right corner and upper right corner of the product image are marked as C and E respectively; at the same time, in the convex hull ym_HU Find the point with the smallest y value and the smallest x value and mark it as D;

[0089] S623), calculate the convex hull y m_HU The center point of y; and filter out the convex hull y m_HU The points whose y value is greater than the y value of the center point are filtered out and put into the set below_center;

[0090] S624), sort the points in the set below_center in descending order according to the x-axis value. If the x-values ​​are the same, sort them according to the y-axis value, and then mark the first-ranked point as point F;

[0091] S625) Calculate multiple intersection points based on the slope relationship between the points, and use the average position of the multiple intersection points as the final intersection point O; specifically:

[0092] First, determine a first straight line, which passes through points D and E. The slope of the first straight line is determined by the line connecting points A and C.

[0093] Next, determine a second straight line, which passes through points F and E, and whose slope is determined by the line connecting points A and E.

[0094] Furthermore, a third straight line is determined, the third straight line passing through points E and F, and the slope of the third straight line is determined by the line formed by points C and F;

[0095] The intersection points of the three straight lines are O1, O2 and O3 respectively; the average position of O1, O2 and O3 is taken as the final intersection point O.

[0096] S626) Estimate the upper surface area S1 of a single top box by calculating the product of the lengths of line segments OA and OE. Figure 3 (a) in the figure is the mask combination of the real picture in the actual scene. Figure 3 (b) shows the selected upbox mask; L1, L2, and L3 are lines calculated from the convex hull points. Line L1 is constructed through points D, E, and A by calculating the slope of segment DE while passing through point A. This determines line L1. Lines L2 and L3 are constructed similarly.

[0097] S63), estimate the upper surface area S2 of any whole layer of cargo boxes in the non-top layer; Figure 4 As shown, Figure 4 (a) in the figure is the mask combination of the real picture in the actual scene. Figure 4 (b) in the figure is the selected downbox mask;

[0098] The specific steps include:

[0099] S631), calculating the convex hull y with the maximum y value and the minimum y value in the convex hull set DH m_hull n_hull

[0100] S632), calculating the A', B', C', E' points in the convex hull set DH according to the method of steps S621)-S622); and when the convex hull y m_hull n_hull is a different convex hull, calculating the convex hull y n_hull closest to the right lower corner of the distance image and recording it as the D' point;

[0101] S633), determining the intersection position of the key straight line according to the slope relationship between the points, and calculating the most total intersection O; then estimating the upper surface area S2 of the whole layer of boxes by calculating the product of the lengths of the line segments OA and OE.

[0102] S64), according to the ratio of the upper surface area S2 of the whole layer of boxes to the upper surface area S1 of a single top layer of boxes, and then taking the integer part downward, thereby obtaining the estimated value N layer of the number of whole layer of goods boxes.

[0103] S65), calculating the total number of goods N total , that is:

[0104] N total = N downbox *N layer +N upbox

[0105] In the formula, N downbox , N upbox are the number of masks of categories 0 and 1, respectively.

[0106] Embodiment 2

[0107] The embodiment provides a goods quantity identification and inventory system based on instance segmentation, comprising:

[0108] A collection module is configured to collect a goods image; in the embodiment, the collection module is a high-definition camera or a camera.

[0109] A labeling module is configured to mark each goods box on the top layer in the goods image as upbox and as category 1, and mark each layer other than the top layer as downbox and as category 0.

[0110] An image segmentation and identification module is configured to identify and segment the goods image by using a pre-trained instance segmentation model to obtain the segmentation mask and the mask category information of the goods image.

[0111] ​​​The cargo quantity estimation module is used to estimate the cargo quantity according to the image mask and the mask category. The cargo quantity estimation module preliminarily determines whether it is a legal case according to the mask quantity and the mask category information of the cargo image, and reports the data if it is illegal. In the embodiment, when the mask quantity of the downbox mask is 0, it is regarded as an illegal case. Otherwise, the convex hull of the mask is calculated, and whether the top layer of the cargo is full is determined according to the geometric relationship of the points in the convex hull. If it is full, the total cargo quantity is directly calculated. Otherwise, the box quantity of any layer of the cargo in the non-top layer is estimated based on the area ratio according to the geometric relationship of the convex hull of the mask, and then the total cargo box quantity is calculated.

[0112] In the embodiment, whether the top layer of the cargo is full is determined, which specifically includes the following steps. Figure 2

[0113] S51), the convex hull HM corresponding to all masks of the cargo is calculated according to the mask and the mask type of the cargo, wherein the convex hull is the geometry of a plurality of two-dimensional points;

[0114] S52), the mask of the upbox type whose convex hull point is closest to the bottom of the cargo image is selected, and the center point center_point of the convex hull corresponding to the mask is calculated;

[0115] S53), the convex hull of the downbox type whose mask is closest to the top of the image is selected, and the highest point ymin_point of the convex hull is calculated;

[0116] S54), the y values of the center point center_point of the convex hull in step S53) and the highest point ymin_point of the convex hull in step S53) are compared. If the y value of the center point center_point of the convex hull in step S53) is less than the y value of the highest point ymin_point of the convex hull in step S53), it is determined that the current case is that the top layer of the cargo is full, otherwise it is not full.

[0117] In addition, in the embodiment, if only the mask of the upbox type exists, it is also regarded as that the top layer of the cargo is full.

[0118] In the embodiment, the box quantity of any layer of the cargo in the non-top layer is estimated based on the area ratio according to the geometric relationship of the convex hull of the mask, and then the total cargo box quantity is calculated, which specifically includes the following steps.

[0119] S61), the convex hull set HM of all masks calculated in step S5) is divided into UH and DH according to the type of the mask, wherein UH is the convex hull set of the mask upbox, and DH is the convex hull set of the mask downbox.​

[0120] S62), for a single top-level box, estimate its upper surface area S1; as Figure 3 shown, specifically comprising the following steps:

[0121] S621), find the convex hull y with the largest y value in the convex hull set UH m_HU , and sort all points in the convex hull y m_HU in ascending order according to the x-axis value, and if the x value is the same, sort according to the y-axis value and preferentially select the point with the smaller y value; mark the first sorted point as point A;

[0122] S622), calculate the point closest to the lower left corner of the goods image in the convex hull y m_HU and mark it as point B; calculate the points closest to the lower right corner and the upper right corner of the goods image in the convex hull y m_HU and mark them as points C and E, respectively; and find the point with the smallest y value and the smallest x value in the convex hull y

[0123] S623), calculate the center point of the convex hull y m_HU ; and filter out the points in the convex hull y m_HU with a y value greater than the y value of the center point, and put all the filtered points into the set below_center;

[0124] S624), sort the points in the set below_center in descending order according to the x-axis value, and if the x value is the same, sort according to the y-axis value, and then mark the first sorted point as point F;

[0125] S625), calculate a plurality of intersection points according to the slope relationship between the points, and take the average position of the plurality of intersection points as the final intersection point O; specifically:

[0126] First, determine the first straight line, the first straight line passes through points D and E, and the slope of the first straight line is determined by the line connecting points A and C;

[0127] Second, determine the second straight line, the second straight line passes through points F and E, and the slope is determined by the line connecting points A and E;

[0128] Third, determine the third straight line, the third straight line passes through points E and F, and the slope is determined by the line connecting points C and F;

[0129] The intersection points of the three straight lines are O1, O2 and O3, respectively; and the average position of O1, O2 and O3 is taken as the final intersection point O.

[0130] S626), estimate the upper surface area S1 of a single top-level box by calculating the product of the lengths of the line segments OA and OE.Figure 3 (a) is a mask combination of real pictures in the actual scene, Figure 3 (b) is a selected upbox mask; wherein L1, L2 and L3 are straight lines calculated by the convex hull points, the straight line L1 is constructed by points D, E and A, the construction method is to calculate the slope of DE line segment, and the straight line passes through point A, so that the straight line L1 can be determined. The construction of L2 and L3 straight lines is the same.

[0131] S63), estimate the upper surface area S2 of any whole layer of cargo boxes in the non-top layer; as shown in Figure 4 , wherein, Figure 4 (a) is a mask combination of real pictures in the actual scene, Figure 4 (b) is a selected downbox mask;

[0132] Specifically includes the following steps:

[0133] S631), calculate the convex hull y m_hull , y n_hull with the maximum y value and the minimum y value in the convex hull set DH;

[0134] S632), calculate the A', B', C', E' points in the convex hull set DH according to the method of steps S621)-S622); and when the convex hull y m_hull , y n_hull is a different convex hull, calculate the convex hull y n_hull The point closest to the right lower corner of the image is recorded as D' point;

[0135] S633), determine the intersection position of the key straight line according to the slope relationship between the points, and calculate the total intersection point O; then estimate the upper surface area S2 of a whole layer of boxes by calculating the product of the lengths of line segments OA and OE.

[0136] S64), according to the ratio of the upper surface area S2 of a whole layer of boxes to the upper surface area S1 of a single top layer of boxes, and then taking the integer part downward, thereby obtaining the estimated value N layer of the number of a whole layer of cargo boxes.

[0137] S65), calculate the total cargo quantity N total , that is:

[0138] N total = N downbox *N layer +N upbox

[0139] In the formula, N downbox , N upbox are the mask quantities of categories 0 and 1 respectively.

[0140] The foregoing embodiments and description of the application only illustrate the principles of the application and the best mode presently contemplated by the inventors. Nothing in this detailed description should be taken to imply that any aspect or feature of the application is essential. The scope of the application should be determined by the appended claims and their legal equivalents, and the full scope of replacement claims to be submitted to the Patent and Trademark Office (PTO) if appropriate.

Claims

1. A method for identifying and counting the quantity of goods based on instance segmentation, characterized in that: The following steps are involved: S1) Collecting a cargo image dataset and labeling the cargo image dataset; S2), using the cargo image dataset in step S1) to train multiple instance segmentation models; S3) using the trained instance segmentation model to identify and segment the cargo image, and obtain the segmentation mask and mask category information of the cargo image; S4) Preliminarily determine whether the product image is legal based on the number of masks and mask category information. If illegal, report the data. Otherwise proceed to step S5); S5) Calculate the convex hull of the mask, and determine whether the top layer of the cargo is fully loaded based on the geometric relationship of the points in the convex hull. If it is fully loaded, directly calculate the total quantity of cargo; otherwise, proceed to step S6); Determining whether the top layer of cargo is fully loaded specifically includes the following steps: S51), according to the mask and mask type of the goods, calculating the convex hull set HM corresponding to all masks, where the convex hull is the geometry of multiple two-dimensional points; S52), select the mask of the upbox category whose convex hull point is closest to the bottom of the cargo image, and calculate the center point center_point of the convex hull corresponding to the mask; S53), select the corresponding convex hull of the mask with the downbox category closest to the top of the image, and calculate the highest point ymin_point of the convex hull; S54), comparing the y value of the center point center_point of the convex hull in step S52) with the y value of the highest point ymin_point of the convex hull in step S53), if the y value of the center point center_point of the convex hull in step S52) is less than the y value of the highest point ymin_point of the convex hull in step S53), it is determined that the current situation is that the top layer of the cargo is fully loaded, otherwise it is not fully loaded; S6) According to the geometric relationship of the mask convex hull, the number of boxes of goods in any layer other than the top layer is estimated based on the number of area ratios, and then the number of boxes of total goods is calculated; specifically, the steps include: S61), according to the type of mask, divide the convex hull set HM of all masks calculated in step S5) into UH and DH, where UH is the convex hull set of the mask upbox and DH is the convex hull set of the mask downbox; S62), for a single top box, estimate its upper surface area S1; S63), estimating the upper surface area S2 of any entire layer of cargo boxes in the non-top layer; S64) Based on the ratio of the upper surface area S2 of the entire layer of boxes to the upper surface area S1 of the single top box, and then rounded down, the estimated value N of the number of cargo boxes in the entire layer is obtained. layer ; S65) Calculate the total quantity of goods N total ,Right now: N total =N downbox *N layer +N upbox Where N downbox 、N upbox The number of masks for category 0 and category 1 respectively.

2. The method for identifying and counting the quantity of goods based on instance segmentation according to claim 1, characterized in that: In step S1), the labeling process is to mark each cargo box on the top layer in the cargo image as upbox and classify it as category 1, and mark each layer other than the top layer as downbox and classify it as category 0.

3. The method for identifying and counting the quantity of goods based on instance segmentation according to claim 1, characterized in that: In step S5), if there is only a mask with the category of upbox, the top floor of the cargo is considered to be fully loaded.

4. The method for identifying and counting goods quantity based on instance segmentation according to claim 1, characterized in that: Step S62) specifically includes the following steps: S621) Find the convex hull y with the maximum y value in the convex hull set UH m_HU , and the convex hull y m_HU Sort all points in ascending order by the x-axis value. If the x-values ​​are the same, sort them by the y-axis value and give priority to the point with the smaller y-value. Mark the first-ranked point as point A. S622) Calculate the convex hull y using the Euclidean distance m_HU The point closest to the lower left corner of the product image is marked as B; the points closest to the lower right corner and upper right corner of the product image are marked as C and E respectively; at the same time, in the convex hull y m_HU Find the point with the smallest y value and the smallest x value and mark it as D; S623), calculate the convex hull y m_HU The center point of y; and filter out the convex hull y m_HU The points whose y value is greater than the y value of the center point are filtered out and put into the set below_center; S624), sort the points in the set below_center in descending order according to the x-axis value. If the x-values ​​are the same, sort them according to the y-axis value, and then mark the first-ranked point as point F; S625), calculating multiple intersection points based on the slope relationship between the points, and taking the average position of the multiple intersection points as the final intersection point O; S626) Estimate the upper surface area S1 of a single top box by calculating the product of the lengths of line segments OA and OE.

5. The method for identifying and counting the quantity of goods based on instance segmentation according to claim 4, characterized in that: In step S625), a first straight line is first determined, the first straight line passing through points D and E, and the slope of the first straight line is determined by the line connecting points A and C; Next, determine a second straight line, which passes through points F and E, and whose slope is determined by the line connecting points A and E. Furthermore, a third straight line is determined, the third straight line passing through points E and F, and the slope of the third straight line is determined by the line formed by points C and F; The intersection points of the three straight lines are O1, O2 and O3 respectively; the average position of O1, O2 and O3 is taken as the final intersection point O.

6. A goods quantity identification and inventory counting system based on instance segmentation, characterized in that: The system utilizes the method according to any one of claims 1 to 5 to realize the identification and inventory of the quantity of goods, and the system comprises: A collection module, used to collect cargo images; The labeling module is used to label each cargo box in the top layer of the cargo image as upbox and classify it as category 1, and each non-top layer as downbox and classify it as category 0; An image segmentation and recognition module is used to identify and segment cargo images using a pre-trained instance segmentation model to obtain segmentation masks and mask category information of cargo images; The cargo quantity estimation module is used to determine different scenes based on image masks and mask categories and estimate the cargo quantity based on area ratio.

7. The system for identifying and counting goods quantity based on instance segmentation according to claim 6, characterized in that: The cargo quantity estimation module preliminarily determines whether the cargo image is legal based on the mask quantity and mask category information, and reports the data if illegal; Otherwise, calculate the convex hull of the mask and determine whether the top layer of the cargo is fully loaded based on the geometric relationship of the points in the convex hull. If it is fully loaded, directly calculate the total quantity of cargo; Otherwise, according to the geometric relationship of the mask convex hull, the number of boxes of any layer of goods other than the top layer is estimated based on the number of area ratios, and then the number of boxes of the total goods is calculated.

Images