Cargo quantity identification and inventory method and system based on instance segmentation

Abstract

The invention provides a cargo quantity identification and inventory method and system based on instance segmentation. The method comprises the steps of collecting a cargo picture data set and performing labeling processing; training a plurality of instance segmentation models; identifying and segmenting the cargo image by using the instance segmentation model to obtain a segmentation mask and mask category information of the cargo image; judging whether the condition is legal or not according to the mask number and mask category information of the cargo image; and judging whether the top layer of the goods is fully loaded or not, if so, directly calculating the total goods quantity, otherwise, estimating the box quantity of the goods on any layer in the non-top layer based on the quantity of the area ratio, and then calculating to obtain the box quantity of the total goods. According to the method, identification of goods in a complex stacking scene is realized by using an instance segmentation model, a goods segmentation mask and a mask type are generated, and the total goods quantity is calculated by identifying and distinguishing different layers of goods boxes; and accurate quantity statistics can be directly carried out under the condition that the top layer of goods is full.

Description

Technical Field

[0001] The present invention relates to the technical field of cargo inventory counting, and in particular to a cargo quantity identification and inventory counting method and system based on instance segmentation. Background Art

[0002] With the rapid development of the logistics industry, warehouse management plays an increasingly important role in the supply chain. In order to improve warehouse utilization efficiency and reduce labor costs, automated cargo management and inventory technology has gradually become a research hotspot. In a warehouse environment, the number of cargo boxes directly affects the utilization of storage space and the efficiency of cargo allocation. Therefore, how to efficiently and accurately count the number of cargo boxes has become an important issue in current warehouse management.

[0003] At present, there are two main types of intelligent inventory counting methods for the quantity of stored goods. The first type of method is based on RFID (Radio Frequency Identification) recognition and counting technology. This method installs RFID tags on the goods, uses radio wave readers to identify each tag and obtains the quantity information of the goods in real time. RFID technology has high inventory efficiency and does not rely on human intervention. However, due to the high cost of implementing and replacing RFID tags, its popularity in large-scale storage environments is limited.

[0004] The second method is to use UAVs (Unmanned Aerial Vehicles) or AGVs (Automated Guided Vehicles) combined with identification codes for inventory. This method is cheaper than RFID, and can reduce labor costs to a certain extent by reading the barcode or QR code of the goods through automated equipment. However, this method also relies on electronic tags, has high requirements for the integrity of the identification code, and may be interfered by factors such as occlusion in complex storage environments, affecting the accuracy of identification.

[0005] In addition to label-based inventory counting methods, some factories also use sensor-based measurement methods to achieve inventory counting by comparing the status of goods before and after stacking, such as combining visual perception with weight sensors. Although these methods have certain effects in specific environments, their applicability is relatively limited and cannot meet the storage needs of various complex stacking forms.

[0006] In recent years, with the development of computer vision and deep learning technology, inventory counting tasks can gradually be solved by analyzing images of stacked goods in storage environments through visual algorithms to automatically count the number of goods. For example, some researchers have proposed an inventory counting method based on target detection and post-processing modules, which can accurately count the number of goods in a three-dimensional storage environment with good lighting conditions. However, in actual storage environments, the stacking structure of goods is complex and changeable, especially in dark warehouse environments, and existing visual algorithms are difficult to accurately identify and count goods. Dark warehouse storage usually only turns on the lighting near the camera during inventory counting, which leads to a significant increase in factors such as shadows and uneven lighting in the image, posing a huge challenge to the inventory counting algorithm.

[0007] Although existing vision-based cargo counting methods have solved the problem of label dependence to a certain extent, they still have obvious limitations in complex warehouse environments. First, the stacking of goods is diverse, especially when the goods on the top floor are scattered or there are gaps, traditional target detection algorithms are difficult to count accurately. In addition, in actual warehouses, interference from environmental factors such as shooting angle, lighting, and occlusion will affect the stability of segmentation and detection results, and thus affect the accuracy of counting. Summary of the invention

[0008] In view of the deficiencies in the prior art, the present invention provides a method and system for identifying and counting the quantity of goods based on instance segmentation. The present invention identifies the quantity of goods on the top layer and non-top layer according to different stacking situations.

[0009] The technical solution of the present invention is: a method for identifying and counting the quantity of goods based on instance segmentation, comprising the following steps:

[0010] S1), collect cargo image data sets, and label the cargo image data sets;

[0011] S2), using the cargo image dataset in step S1) to train multiple instance segmentation models;

[0012] S3), using the trained instance segmentation model to identify and segment the cargo image, and obtain the segmentation mask and mask category information of the cargo image;

[0013] S4), based on the number of masks and mask category information of the cargo image, preliminarily determine whether it is legal, if illegal, report the data; otherwise, go to step S5);

[0014] S5), calculate the convex hull of the mask, and determine whether the top layer of the cargo is fully loaded according to the geometric relationship of the points in the convex hull. If it is fully loaded, directly calculate the total quantity of cargo, otherwise, enter step S6);

[0015] S6), according to the geometric relationship of the mask convex hull, the number of boxes of any layer of goods other than the top layer is estimated based on the number of area ratios, and then the number of boxes of the total goods is calculated.

[0016] Preferably, in step S1), the labeling process refers to marking each cargo box on the top layer in the cargo image as upbox and as category 1, and marking each non-top layer as downbox and as category 0.

[0017] Preferably, in step S2), the instance segmentation model adopts a Mask2former model, or a SOLOv2 model, a QueryInst model, or a SparseInst model.

[0018] Preferably, in step S3), the instance segmentation model identifies the target in the cargo image and generates a mask for each object, and classifies each mask as upbox or downbox, and the value of the mask matrix is ​​1 or 0, 0 indicates that the pixel has no mask, and 1 indicates that the pixel has a mask.

[0019] Preferably, in step S4), when the number of masks with the mask category of downbox is 0, it is regarded as an illegal situation.

[0020] Preferably, in step S5), determining whether the top layer of the cargo is fully loaded specifically comprises the following steps:

[0021] S51), according to the mask and mask type of the goods, calculate the convex hull HM corresponding to all masks, where the convex hull is the geometry of multiple two-dimensional points;

[0022] S52), select the mask of the upbox category whose convex hull point is closest to the bottom of the cargo image, and calculate the center point center_point of the convex hull corresponding to the mask;

[0023] S53), select the corresponding convex hull of the mask with the category of downbox that is closest to the top of the image, and calculate the highest point ymin_point of the convex hull;

[0024] S54), compare the y value of the center point center_point of the convex hull in step S53) with the y value of the highest point ymin_point of the convex hull in step S53). If the y value of the center point center_point of the convex hull in step S53) is less than the y value of the highest point ymin_point of the convex hull in step S53), it is determined that the current situation is that the top layer of the cargo is fully loaded, otherwise it is not fully loaded.

[0025] Preferably, in step S5), if there is only a mask with the category of upbox, the top floor of the cargo is deemed to be fully loaded.

[0026] Preferably, in step S6), estimating the number of boxes of any whole layer of goods in the non-top layer by the number based on the area ratio specifically includes the following steps:

[0027] S61), according to the type of mask, divide the convex hull set HM of all masks calculated in step S5) into UH and DH, where UH is the convex hull set of mask upbox and DH is the convex hull set of mask downbox;

[0028] S62), for a single top-level box, estimate its upper surface area S1;

[0029] S63), estimating the upper surface area S2 of any entire layer of cargo boxes in the non-top layer;

[0030] S64), based on the ratio of the upper surface area S2 of a whole layer of boxes to the upper surface area S1 of a single top layer box, and then rounded down, the estimated value N of the number of cargo boxes in a whole layer is obtained. layer ;

[0031] S65) Calculate the total quantity of goods N total ,Right now:

[0032] N total =N downbox *N layer +N upbox

[0033] Where N downbox 、N upbox are the number of masks for category 0 and category 1 respectively.

[0034] Preferably, step S62) specifically includes the following steps:

[0035] S621), find the convex hull y with the maximum y value in the convex hull set UH m_HU , and for the convex hull y m_HU All points in the graph are sorted in ascending order according to the x-axis value. If the x-values ​​are the same, they are sorted according to the y-axis value and the points with smaller y-values ​​are selected first. The first-ranked point is marked as point A.

[0036] S622), use Euclidean distance to calculate the convex hull y m_HU The point closest to the lower left corner of the cargo image is marked as B; the points closest to the lower right corner and upper right corner of the cargo image are marked as C and E respectively; at the same time, in the convex hull y m_HU Find the point with the smallest y value and the smallest x value and mark it as D;

[0037] S623), calculate the convex hull y m_HU The center point ofm_HU The points whose y values ​​are greater than the y value of the center point are filtered out, and all the filtered points are put into the set below_center;

[0038] S624), sort the points in the set below_center in descending order according to the x-axis value. If the x-values ​​are the same, sort them according to the y-axis value, and then mark the first-ranked point as point F;

[0039] S625), calculating multiple intersection points according to the slope relationship between the points, and taking the average position of the multiple intersection points as the final intersection point O;

[0040] S626) Estimate the upper surface area S1 of a single top box by calculating the product of the lengths of line segments OA and OE.

[0041] Preferably, in step S625), a first straight line is first determined, the first straight line passes through points D and E, and the slope of the first straight line is determined by the line connecting points A and C;

[0042] Next, determine a second straight line, which passes through points F and E, and whose slope is determined by the line formed by points A and E;

[0043] Furthermore, a third straight line is determined, the third straight line passes through points E and F, and its slope is determined by the line formed by points C and F;

[0044] The intersection points of the three straight lines are O1, O2 and O3 respectively; the average position of O1, O2 and O3 is taken as the final intersection point O.

[0045] Preferably, in step S63), estimating the upper surface area S2 of a whole layer of cargo boxes specifically includes the following steps:

[0046] S631), calculate the convex hull y with the maximum y value and the minimum y value in the convex hull set DH m_hull ,y n_hull ;

[0047] S632), calculate the points A', B', C', E' in the convex hull set DH according to the method of steps S621)-S622); and when the convex hull y m_hull ,y n_hull When they are different convex hulls, calculate the convex hull y n_hull The point closest to the lower right corner of the image is recorded as point D′;

[0048] S633), determine the intersection position of the key straight lines according to the slope relationship between the points, and calculate the total intersection point O; then estimate the upper surface area S2 of the entire layer of boxes by calculating the product of the length of the line segment OA and OE.

[0049] Preferably, the present invention further provides a system for identifying and counting the number of goods based on instance segmentation, comprising:

[0050] A collection module, used for collecting cargo images;

[0051] The labeling module is used to label each cargo box in the top layer of the cargo image as upbox and classify it as category 1, and label each non-top layer as downbox and classify it as category 0;

[0052] An image segmentation and recognition module is used to identify and segment the cargo image using a pre-trained instance segmentation model to obtain the segmentation mask and mask category information of the cargo image;

[0053] The cargo quantity estimation module is used to determine different scenes according to image masks and mask categories and estimate the cargo quantity based on area ratio.

[0054] Preferably, the cargo quantity estimation module preliminarily determines whether the cargo image is legal based on the number of masks and mask category information. If illegal, the data is reported; otherwise, the convex hull of the mask is calculated, and whether the top layer of the cargo is fully loaded is determined based on the geometric relationship of the points in the convex hull. If it is fully loaded, the total number of cargo is directly calculated; otherwise, the number of boxes of cargo in any layer other than the top layer is estimated based on the number of area ratios according to the geometric relationship of the convex hull of the mask, and then the total number of boxes of cargo is calculated.

[0055] The beneficial effects of the present invention are:

[0056] 1. The present invention uses an instance segmentation model to realize the recognition of goods in complex stacking scenes, and generates a goods segmentation mask and mask type, and calculates the total number of goods by identifying and distinguishing different layers of goods boxes;

[0057] 2. The present invention calculates the quantity of goods based on the area ratio method for different stacking situations, which is not only applicable to the situation where the goods on the top floor are scattered, but also can directly perform accurate quantity statistics when the goods on the top floor are fully stacked;

[0058] 3. The mask-based area ratio estimation method of the present invention has high cargo counting accuracy in complex storage environments, does not need to rely on electronic tags or sensor data, reduces inventory costs, and is particularly suitable for storage environments with poor lighting conditions such as dark warehouses. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a schematic diagram of the process of the present invention;

[0060] Figure 2Schematic diagram of determining whether the top layer of cargo is fully loaded according to the present invention; wherein (a) and (b) are schematic diagrams of a fully loaded case and a fully loaded case, respectively;

[0061] Figure 3 Schematic diagram of estimating the upper surface area of ​​a single top box according to the present invention; wherein (a) is a schematic diagram of a mask combination of a real picture in an actual scene, and (b) is a schematic diagram of a selected upbox mask;

[0062] Figure 4 Schematic diagram of estimating the upper surface area of ​​a whole layer of goods according to the present invention, wherein (a) is a schematic diagram of the mask combination of a real picture in an actual scene, and (b) is a schematic diagram of the selected downbox mask. DETAILED DESCRIPTION

[0063] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings:

[0064] Example 1

[0065] like Figure 1 As shown, this embodiment provides a method for identifying and counting the quantity of goods based on instance segmentation, comprising the following steps:

[0066] S1), using a high-definition camera to collect a cargo image dataset, and labeling the cargo image dataset; and dividing the labeled cargo image dataset into a training set and a test set according to a certain ratio;

[0067] In this embodiment, the labeling process refers to marking each cargo box in the top layer of the cargo image as upbox and as category 1, and marking each layer other than the top layer as downbox and as category 0. S2), using the cargo image data set in step S1) to train multiple instance segmentation models;

[0068] In this embodiment, the instance segmentation model adopts the Mask2former model, or the SOLOv2 model, the QueryInst model, or the SparseInst model.

[0069] The training process lasted for 300 epochs. The Mask2former model was optimized using the AdamW optimizer, with the learning rate initialized to 0.0001, the weight decay factor to 0.05, the momentum coefficient of the optimizer to (0.9, 0.999), and the batch size to 4.

[0070] The SOLOv2 model is optimized using the Stochastic Gradient Descent (SGD) optimizer with a learning rate of 0.01, a momentum factor of 0.9, a weight decay of 0.0001, and a batch size of 8.

[0071] The QueryInst model is optimized using the AdamW optimizer, with the learning rate initialized to 0.0001, the weight decay factor to 0.0001, and the batch size to 8.

[0072] The SparseInst model is optimized using the AdamW optimizer with an initial learning rate of 0.00005, a weight decay factor of 0.05, and a batch size of 8.

[0073] In order to improve the generalization ability, during training, the image size is randomly adjusted while maintaining the image ratio, and random horizontal flipping is used for data enhancement.

[0074] S3), according to different scenarios, using different trained instance segmentation models to identify and segment the cargo image, and obtain the segmentation mask and mask category information of the cargo image;

[0075] In this embodiment, the instance segmentation model identifies the target in the cargo image and generates a mask for each object, and classifies each mask as upbox or downbox, and the value of the mask matrix is ​​1 or 0, 0 indicates that the pixel has no mask, and 1 indicates that the pixel has a mask.

[0076] S4), based on the number of masks and mask category information of the cargo image, preliminarily determine whether it is legal, if illegal, report the data; in this embodiment, when the number of masks with the mask category of downbox is 0, it is considered illegal, otherwise go to step S5);

[0077] S5), calculate the convex hull of the mask, and determine whether the top layer of the cargo is fully loaded according to the geometric relationship of the points in the convex hull. If it is fully loaded, directly calculate the total quantity of cargo. Otherwise, proceed to step S6); determine whether the top layer of the cargo is fully loaded. Figure 2 As shown in (a) and (b) in the figure, respectively, the unloaded state and the fully loaded state are shown, and the specific steps include the following:

[0078] S51), according to the mask and mask type of the goods, calculate the convex hull HM corresponding to all masks, where the convex hull is the geometry of multiple two-dimensional points;

[0079] S52), select the mask of the upbox category whose convex hull point is closest to the bottom of the cargo image, and calculate the center point center_point of the convex hull corresponding to the mask;

[0080] S53), select the corresponding convex hull of the mask with the category of downbox that is closest to the top of the image, and calculate the highest point ymin_point of the convex hull;

[0081] S54), compare the y value of the center point center_point of the convex hull in step S53) with the y value of the highest point ymin_point of the convex hull in step S53). If the y value of the center point center_point of the convex hull in step S53) is less than the y value of the highest point ymin_point of the convex hull in step S53), it is determined that the current situation is that the top layer of the cargo is fully loaded, otherwise it is not fully loaded.

[0082] In addition, in this embodiment, if there is only a mask with the category of upbox, it is also regarded that the top floor of the cargo is fully loaded.

[0083] If the top layer of cargo is fully loaded, the instance segmentation model identifies the segmented mask upbox as the number of cargo boxes on the entire layer, and then obtains the total number of cargo through the number of masks downbox.

[0084] S6), according to the geometric relationship of the mask convex hull, the number of boxes of any layer of goods other than the top layer is estimated based on the number of area ratios, and then the number of boxes of the total goods is calculated, which specifically includes the following steps:

[0085] S61), according to the type of mask, divide the convex hull set HM of all masks calculated in step S5) into UH and DH, where UH is the convex hull set of mask upbox and DH is the convex hull set of mask downbox;

[0086] S62), for a single top box, estimate its upper surface area S1; Figure 3 As shown, the specific steps include:

[0087] S621), find the convex hull y with the maximum y value in the convex hull set UH m_HU , and for the convex hull y m_HU All points in the graph are sorted in ascending order according to the x-axis value. If the x-values ​​are the same, they are sorted according to the y-axis value and the points with smaller y-values ​​are selected first. The first-ranked point is marked as point A.

[0088] S622), use Euclidean distance to calculate the convex hull y m_HU The point closest to the lower left corner of the cargo image is marked as B; the points closest to the lower right corner and upper right corner of the cargo image are marked as C and E respectively; at the same time, in the convex hull ym_HU Find the point with the smallest y value and the smallest x value and mark it as D;

[0089] S623), calculate the convex hull y m_HU The center point of m_HU The points whose y values ​​are greater than the y value of the center point are filtered out, and all the filtered points are put into the set below_center;

[0090] S624), sort the points in the set below_center in descending order according to the x-axis value. If the x-values ​​are the same, sort them according to the y-axis value, and then mark the first-ranked point as point F;

[0091] S625), multiple intersection points are calculated based on the slope relationship between the points, and the average position of the multiple intersection points is used as the final intersection point O; specifically:

[0092] First, determine the first straight line, the first straight line passes through points D and E, and the slope of the first straight line is determined by the line connecting points A and C;

[0093] Next, determine a second straight line, which passes through points F and E, and whose slope is determined by the line formed by points A and E;

[0094] Furthermore, a third straight line is determined, the third straight line passes through points E and F, and its slope is determined by the line formed by points C and F;

[0095] The intersection points of the three straight lines are O1, O2 and O3 respectively; the average position of O1, O2 and O3 is taken as the final intersection point O.

[0096] S626) Estimate the upper surface area S1 of a single top box by calculating the product of the lengths of line segments OA and OE. Figure 3 (a) in the figure is the mask combination of the real picture in the actual scene. Figure 3 (b) is the selected upbox mask; L1, L2 and L3 are the lines calculated by the convex hull points. Line L1 is constructed through points D, E and A by calculating the slope of segment DE. At the same time, the line passes through point A, so that line L1 can be determined. The construction of lines L2 and L3 is similar.

[0097] S63), estimate the upper surface area S2 of any whole layer of cargo boxes in the non-top layer; Figure 4 As shown, Figure 4 (a) in the figure is the mask combination of the real picture in the actual scene. Figure 4 (b) in the figure is the selected downbox mask;

[0098] The specific steps include:

[0099] S631), calculate the convex hull y with the maximum y value and the minimum y value in the convex hull set DH m_hull ,y n_hull ;

[0100] S632), calculate the points A', B', C', E' in the convex hull set DH according to the method of steps S621)-S622); and when the convex hull y m_hull ,y n_hull When they are different convex hulls, calculate the convex hull y n_hull The point closest to the lower right corner of the image is recorded as point D′;

[0101] S633), determine the intersection position of the key straight lines according to the slope relationship between the points, and calculate the total intersection point O; then estimate the upper surface area S2 of the entire layer of boxes by calculating the product of the length of the line segment OA and OE.

[0102] S64), based on the ratio of the upper surface area S2 of a whole layer of boxes to the upper surface area S1 of a single top layer box, and then rounded down, the estimated value N of the number of cargo boxes in a whole layer is obtained. layer ;

[0103] S65) Calculate the total quantity of goods N total ,Right now:

[0104] N total =N downbox *N layer +N upbox

[0105] Where N downbox 、N upbox are the number of masks for category 0 and category 1 respectively.

[0106] Example 2

[0107] This embodiment provides a system for identifying and counting the number of goods based on instance segmentation, including:

[0108] The acquisition module is used to acquire images of goods; in this embodiment, the acquisition module adopts a high-definition camera or a still camera.

[0109] The labeling module is used to label each cargo box in the top layer of the cargo image as upbox and classify it as category 1, and label each non-top layer as downbox and classify it as category 0;

[0110] An image segmentation and recognition module is used to identify and segment the cargo image using a pre-trained instance segmentation model to obtain the segmentation mask and mask category information of the cargo image;

[0111] The cargo quantity estimation module is used to determine different scenes according to the image mask and mask category and estimate the cargo quantity based on the area ratio. The cargo quantity estimation module preliminarily determines whether it is a legal situation based on the mask quantity and mask category information of the cargo image. If it is illegal, it reports this data; in this embodiment, when the number of masks with the mask category of downbox is 0, it is considered illegal. Otherwise, the convex hull of the mask is calculated, and according to the geometric relationship of the points in the convex hull, it is determined whether the top layer of the cargo is fully loaded. If it is fully loaded, the total number of cargo is directly calculated. Otherwise, according to the geometric relationship of the mask convex hull, the number of boxes of any layer of cargo in the non-top layer is estimated based on the number of area ratios, and then the total number of boxes of cargo is calculated.

[0112] In this embodiment, it is determined whether the top floor of the cargo is fully loaded. Figure 2 As shown, the specific steps include:

[0113] S51), according to the mask and mask type of the goods, calculate the convex hull HM corresponding to all masks, where the convex hull is the geometry of multiple two-dimensional points;

[0114] S52), select the mask of the upbox category whose convex hull point is closest to the bottom of the cargo image, and calculate the center point center_point of the convex hull corresponding to the mask;

[0115] S53), select the corresponding convex hull of the mask with the category of downbox that is closest to the top of the image, and calculate the highest point ymin_point of the convex hull;

[0116] S54), compare the y value of the center point center_point of the convex hull in step S53) with the y value of the highest point ymin_point of the convex hull in step S53). If the y value of the center point center_point of the convex hull in step S53) is less than the y value of the highest point ymin_point of the convex hull in step S53), it is determined that the current situation is that the top layer of the cargo is fully loaded, otherwise it is not fully loaded.

[0117] In addition, in this embodiment, if there is only a mask with the category of upbox, it is also regarded that the top floor of the cargo is fully loaded.

[0118] In this embodiment, according to the geometric relationship of the mask convex hull, the number of boxes of any layer of goods other than the top layer is estimated based on the number of area ratios, and then the number of boxes of the total goods is calculated, which specifically includes the following steps:

[0119] S61), according to the type of mask, divide the convex hull set HM of all masks calculated in step S5) into UH and DH, where UH is the convex hull set of mask upbox and DH is the convex hull set of mask downbox;

[0120] S62), for a single top box, estimate its upper surface area S1; Figure 3 As shown, the specific steps include:

[0121] S621), find the convex hull y with the maximum y value in the convex hull set UH m_HU , and for the convex hull y m_HU All points in the graph are sorted in ascending order according to the x-axis value. If the x-values ​​are the same, they are sorted according to the y-axis value and the points with smaller y-values ​​are selected first. The first-ranked point is marked as point A.

[0122] S622), use Euclidean distance to calculate the convex hull y m_HU The point closest to the lower left corner of the cargo image is marked as B; the points closest to the lower right corner and upper right corner of the cargo image are marked as C and E respectively; at the same time, in the convex hull y m_HU Find the point with the smallest y value and the smallest x value and mark it as D;

[0123] S623), calculate the convex hull y m_HU The center point of m_HU The points whose y values ​​are greater than the y value of the center point are filtered out, and all the filtered points are put into the set below_center;

[0124] S624), sort the points in the set below_center in descending order according to the x-axis value. If the x-values ​​are the same, sort them according to the y-axis value, and then mark the first-ranked point as point F;

[0125] S625), multiple intersection points are calculated based on the slope relationship between the points, and the average position of the multiple intersection points is used as the final intersection point O; specifically:

[0126] First, determine the first straight line, the first straight line passes through points D and E, and the slope of the first straight line is determined by the line connecting points A and C;

[0127] Next, determine a second straight line, which passes through points F and E, and whose slope is determined by the line formed by points A and E;

[0128] Furthermore, a third straight line is determined, the third straight line passes through points E and F, and its slope is determined by the line formed by points C and F;

[0129] The intersection points of the three straight lines are O1, O2 and O3 respectively; the average position of O1, O2 and O3 is taken as the final intersection point O.

[0130] S626) Estimate the upper surface area S1 of a single top box by calculating the product of the lengths of line segments OA and OE. Figure 3 (a) in the figure is the mask combination of the real picture in the actual scene. Figure 3 (b) is the selected upbox mask; L1, L2 and L3 are the lines calculated by the convex hull points. Line L1 is constructed through points D, E and A by calculating the slope of segment DE. At the same time, the line passes through point A, so that line L1 can be determined. The construction of lines L2 and L3 is similar.

[0131] S63), estimate the upper surface area S2 of any whole layer of cargo boxes in the non-top layer; Figure 4 As shown, Figure 4 (a) in the figure is the mask combination of the real picture in the actual scene. Figure 4 (b) in the figure is the selected downbox mask;

[0132] The specific steps include:

[0133] S631), calculate the convex hull y with the maximum y value and the minimum y value in the convex hull set DH m_hull ,y n_hull ;

[0134] S632), calculate the points A', B', C', E' in the convex hull set DH according to the method of steps S621)-S622); and when the convex hull y m_hull ,y n_hull When they are different convex hulls, calculate the convex hull y n_hull The point closest to the lower right corner of the image is recorded as point D′;

[0135] S633), determine the intersection position of the key straight lines according to the slope relationship between the points, and calculate the total intersection point O; then estimate the upper surface area S2 of the entire layer of boxes by calculating the product of the length of the line segment OA and OE.

[0136] S64), based on the ratio of the upper surface area S2 of a whole layer of boxes to the upper surface area S1 of a single top layer box, and then rounded down, the estimated value N of the number of cargo boxes in a whole layer is obtained. layer ;

[0137] S65) Calculate the total quantity of goods N total ,Right now:

[0138] N total =N downbox *N layer +N upbox

[0139] Where N downbox 、N upbox are the number of masks for category 0 and category 1 respectively.

[0140] The above embodiments and descriptions are only for illustrating the principles and best embodiments of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, all of which fall within the scope of the present invention to be protected.

Claims

1. A method for identifying and counting the number of goods based on instance segmentation, characterized in that: The following steps are involved: S1), collect cargo image data sets, and label the cargo image data sets; S2), using the cargo image dataset in step S1) to train multiple instance segmentation models; S3), using the trained instance segmentation model to identify and segment the cargo image, and obtain the segmentation mask and mask category information of the cargo image; S4), based on the number of masks and mask category information of the cargo image, preliminarily determine whether it is legal, and if illegal, report the data; Otherwise, proceed to step S5); S5), calculate the convex hull of the mask, and determine whether the top layer of the cargo is fully loaded according to the geometric relationship of the points in the convex hull. If it is fully loaded, directly calculate the total quantity of cargo, otherwise, enter step S6); S6), according to the geometric relationship of the mask convex hull, the number of boxes of any layer of goods other than the top layer is estimated based on the number of area ratios, and then the number of boxes of the total goods is calculated.

2. The method for identifying and counting the number of goods based on instance segmentation according to claim 1 is characterized in that: In step S1), the labeling process refers to marking each cargo box on the top layer in the cargo image as upbox and as category 1, and marking each layer other than the top layer as downbox and as category 0.

3. The method for identifying and counting the number of goods based on instance segmentation according to claim 1 is characterized in that: In step S5), determining whether the top floor of the cargo is fully loaded specifically includes the following steps: S51), according to the mask and mask type of the goods, calculate the convex hull HM corresponding to all masks, where the convex hull is the geometry of multiple two-dimensional points; S52), select the mask of the upbox category whose convex hull point is closest to the bottom of the cargo image, and calculate the center point center_point of the convex hull corresponding to the mask; S53), select the corresponding convex hull of the mask with the category of downbox that is closest to the top of the image, and calculate the highest point ymin_point of the convex hull; S54), compare the y value of the center point center_point of the convex hull in step S53) with the y value of the highest point ymin_point of the convex hull in step S53). If the y value of the center point center_point of the convex hull in step S53) is less than the y value of the highest point ymin_point of the convex hull in step S53), it is determined that the current situation is that the top layer of the cargo is fully loaded, otherwise it is not fully loaded.

4. The method for identifying and counting the number of goods based on instance segmentation according to claim 3 is characterized in that: In step S5), if there is only a mask with the category of upbox, the top floor of the cargo is considered to be fully loaded.

5. The method for identifying and counting the number of goods based on instance segmentation according to claim 1 is characterized in that: In step S6), the number of boxes of any whole layer of goods in the non-top layer is estimated based on the number of area ratios, and then the number of boxes of the total goods is calculated, which specifically includes the following steps: S61), according to the type of mask, divide the convex hull set HM of all masks calculated in step S5) into UH and DH, where UH is the convex hull set of mask upbox and DH is the convex hull set of mask downbox; S62), for a single top-level box, estimate its upper surface area S1; S63), estimating the upper surface area S2 of any entire layer of cargo boxes in the non-top layer; S64), based on the ratio of the upper surface area S2 of a whole layer of boxes to the upper surface area S1 of a single top layer box, and then rounded down, the estimated value N of the number of cargo boxes in a whole layer is obtained. layer ; S65) Calculate the total quantity of goods N total ,Right now: N total =N downbox *N layer +N upbox Where N downbox 、N upbox are the number of masks for category 0 and category 1 respectively.

6. The method for identifying and counting the number of goods based on instance segmentation according to claim 5 is characterized in that: Step S62) specifically includes the following steps: S621), find the convex hull y with the maximum y value in the convex hull set UH m_HU , and for the convex hull y m_HU All points in the graph are sorted in ascending order according to the x-axis value. If the x-values ​​are the same, they are sorted according to the y-axis value and the points with smaller y-values ​​are selected first. The first-ranked point is marked as point A. S622), use Euclidean distance to calculate the convex hull y m_HU The point closest to the lower left corner of the cargo image is marked as B; the points closest to the lower right corner and upper right corner of the cargo image are marked as C and E respectively; at the same time, in the convex hull y m_HU Find the point with the smallest y value and the smallest x value and mark it as D; S623), calculate the convex hull y m_HU The center point of m_HU The points whose y values ​​are greater than the y value of the center point are filtered out, and all the filtered points are put into the set below_center; S624), sort the points in the set below_center in descending order according to the x-axis value. If the x-values ​​are the same, sort them according to the y-axis value, and then mark the first-ranked point as point F; S625), calculating multiple intersection points according to the slope relationship between the points, and taking the average position of the multiple intersection points as the final intersection point O; S626) Estimate the upper surface area S1 of a single top box by calculating the product of the lengths of line segments OA and OE.

7. The method for identifying and counting the number of goods based on instance segmentation according to claim 6 is characterized in that: In step S625), first determine a first straight line, the first straight line passes through points D and E, and the slope of the first straight line is determined by the line connecting points A and C; Next, determine a second straight line, which passes through points F and E, and whose slope is determined by the line formed by points A and E; Furthermore, a third straight line is determined, the third straight line passes through points E and F, and its slope is determined by the line formed by points C and F; The intersection points of the three straight lines are O1, O2 and O3 respectively; the average position of O1, O2 and O3 is taken as the final intersection point O.

8. The method for identifying and counting the number of goods based on instance segmentation according to claim 7 is characterized in that: In step S63), the upper surface area S2 of a whole layer of cargo boxes is estimated, which specifically includes the following steps: S631), calculate the convex hull y with the maximum y value and the minimum y value in the convex hull set DH m_hull ,y n_hull ; S632), calculate the points A', B', C', E' in the convex hull set DH according to the method of steps S621)-S622); and when the convex hull y m_hull ,y n_hull When the convex hulls are different, calculate the convex hull y n_hull The point closest to the lower right corner of the image is recorded as point D′; S633), determining the intersection position of the key straight line according to the slope relationship between the points, and calculating the total intersection point O; Then, the upper surface area S2 of the entire layer of boxes is estimated by calculating the product of the lengths of line segments OA and OE.

9. A goods quantity identification and inventory counting system based on instance segmentation, characterized in that: The system uses the method described in any one of claims 1 to 8 to realize the identification and inventory of the quantity of goods, and the system includes: A collection module, used for collecting cargo images; The labeling module is used to label each cargo box in the top layer of the cargo image as upbox and classify it as category 1, and label each non-top layer as downbox and classify it as category 0; An image segmentation and recognition module is used to identify and segment the cargo image using a pre-trained instance segmentation model to obtain the segmentation mask and mask category information of the cargo image; The cargo quantity estimation module is used to determine different scenes according to image masks and mask categories and estimate the cargo quantity based on area ratio.

10. The system for identifying and counting the number of goods based on instance segmentation according to claim 9, characterized in that: The cargo quantity estimation module preliminarily determines whether the cargo image is legal based on the mask quantity and mask category information, and reports the data if it is illegal; Otherwise, the convex hull of the mask is calculated, and according to the geometric relationship of the points in the convex hull, it is determined whether the top layer of the cargo is fully loaded. If it is fully loaded, the total quantity of cargo is directly calculated; Otherwise, according to the geometric relationship of the mask convex hull, the number of boxes of any layer of goods other than the top layer is estimated based on the number of area ratios, and then the number of boxes of the total goods is calculated.

Images