Automatic cargo checking method and system for unmanned warehouse shelf

Abstract

The invention discloses an unmanned warehouse shelf-oriented automatic cargo checking method and system, and the method is characterized in that the method comprises the following steps: employing a target detection algorithm to extract a cargo boundary frame from captured continuous cargo images; for each cargo bounding box, calculating shape features, color features, texture features, size features and local features of the cargo in the cargo bounding box, and splicing feature vectors of the cargo according to the features; pre-defining cargo templates of different types of cargos, matching the cargo templates with the feature vectors of the cargos so as to obtain a category label of each cargo bounding box, and iteratively optimizing the target detection algorithm through a loss function; and tracking the goods extracted by the target detection algorithm by using a Kalman filtering target tracking algorithm, and obtaining the position and quantity information of the goods from the continuous goods images.

Description

Technical Field

[0001] The present invention relates to a field, and more specifically, to an automatic inventory method and system for goods on unmanned warehouse shelves. Background Art

[0002] With the booming development of global e-commerce, the logistics and warehousing industry is facing unprecedented challenges and opportunities. Traditional warehouse management methods, especially inventory counting, usually rely on manual work, which is not only inefficient but also prone to errors. It cannot meet the needs of modern warehouse management for efficiency, accuracy and real-time. Especially in large warehouses, manual inventory counting often requires a lot of manpower, material resources and time, and the inventory counting results often have errors, which causes inconvenience to warehouse management and logistics scheduling. During the manual inventory counting process, employees need to check the goods one by one, use handheld devices to scan barcodes or record data manually. Not only is the process cumbersome, but it is also prone to errors when recording and entering data, resulting in inaccurate inventory data.

[0003] In this context, the automatic inventory counting algorithm for unmanned warehouses based on computer vision technology came into being, bringing revolutionary changes to the warehouse management industry. Computer vision technology is a scientific technology that simulates human visual functions. Through image acquisition, processing and analysis, it can realize the functions of identifying, locating, tracking and measuring target objects. In the field of warehouse management, computer vision technology can realize fast and accurate inventory counting by automatically processing and analyzing the images of goods in the warehouse, greatly improving the efficiency and accuracy of warehouse management. In the field of warehouse management, the automatic inventory counting algorithm for unmanned warehouses based on computer vision technology has been widely used. For example, large e-commerce companies such as Amazon have adopted this technology to realize fast and accurate inventory counting of millions of items in the warehouse. Through this technology, enterprises can grasp the inventory situation in real time, optimize inventory management strategies, and improve logistics efficiency and customer satisfaction.

[0004] Specifically, the unmanned warehouse automatic inventory counting algorithm based on computer vision technology captures the image information of the goods through the visual sensors installed in the warehouse, uses image processing technology to preprocess and extract features of the images, and then uses advanced algorithms such as deep learning to identify and count the goods in the image. This process does not require human intervention and can achieve real-time monitoring and automatic inventory of goods in the warehouse, greatly improving the level of intelligent warehouse management. Through the automated system, warehouse managers can obtain accurate inventory data in real time, promptly discover inventory shortages or backlogs, and optimize warehousing and logistics scheduling. In addition, the unmanned warehouse automatic inventory counting algorithm based on computer vision technology has the following advantages: First, it can adapt to various complex warehousing environments, including light changes, goods occlusion, goods stacking, etc. Visual sensors and algorithms can work under different lighting conditions and handle recognition difficulties caused by irregular or overlapping goods. Second, it can quickly and accurately identify various types of goods, including items of different shapes, colors, and sizes. Whether it is goods of different shapes or packaging of various colors, the computer vision system can efficiently identify and classify them. Finally, it can update inventory information in real time and provide accurate data support for warehouse management and logistics scheduling. Real-time updated inventory data makes warehouse management more transparent and efficient, helps to formulate more scientific replenishment and distribution plans, reduces inventory holding costs, and improves operational efficiency. In short, the unmanned warehouse automatic inventory counting algorithm based on computer vision technology not only solves the problems of low efficiency and easy errors in traditional inventory counting methods, but also brings significant improvement and optimization to warehouse management and logistics industries through its high efficiency, accuracy, and intelligence.

[0005] The existing warehouse inventory counting methods mainly include manual inventory counting, handheld PDA inventory counting and RFID-based inventory counting. Each method has its specific application scenarios and advantages, but also has limitations and shortcomings.

[0006] The earliest warehouse inventory counting method was purely manual operation, where staff completed the inventory by manually recording and checking the cargo information. With the expansion of warehouse scale and the increase in management needs, manual inventory counting gradually showed problems such as low efficiency and high error rate. However, manual inventory counting is still widely used in some small warehouses or temporary inventory counting because it is simple and easy to do. Manual inventory counting requires a lot of manpower input, and warehouse operations may need to be suspended for inventory counting, resulting in production and service interruptions and extremely high costs. Human operations are prone to errors, such as incorrect records, counting errors or omissions, resulting in inaccurate inventory data. It is estimated that the error rate of manual inventory counting may be as high as more than 5%. Traditional methods cannot update inventory information in real time, cannot immediately reflect inventory changes, and are difficult to adjust and manage in a timely manner. When problems or errors occur, it is difficult to accurately track the specific links where the problems occur.

[0007] With the development of barcode technology, handheld PDA (Personal Digital Assistant) devices have been introduced into warehouse inventory. Handheld PDA combined with barcode scanning technology has greatly improved inventory efficiency and data accuracy. By scanning the barcode on the goods, staff can quickly complete data collection and transmit the data to the warehouse management system in real time. If there is no barcode on the goods or the barcode is not clear, handheld PDA inventory will not be able to proceed. Although faster than manual inventory, the efficiency of handheld PDA inventory is still limited in large-scale warehousing environments. The purchase, maintenance and update of handheld PDA equipment is also a considerable expense.

[0008] The emergence and development of RFID (Radio Frequency Identification) technology has brought new possibilities for warehouse management. RFID tags can store more information and can be read by wireless radio frequency identifiers without visual conditions. RFID inventory systems can complete a wide range of goods identification and data collection in a short time, greatly improving inventory efficiency and accuracy. RFID electronic tags are more expensive than ordinary barcode tags, generally dozens of times more expensive than ordinary barcode tags, which leads to high costs when used on a large scale. RFID technology has been around for a relatively short time, and although it has developed rapidly, it is still not mature enough in some aspects. For example, there are difficulties in the application of ultra-high frequency RFID electronic tags in commodities such as metals and liquids. The security issues faced by RFID technology mainly include illegal reading and malicious tampering of RFID electronic tag information. In addition, RFID technology can track and monitor items, but it may also leak personal privacy information.

[0009] In summary, the existing warehouse inventory methods have their own advantages and disadvantages, but they all have limitations and shortcomings to varying degrees under the demand for large-scale, high-efficiency, and high-precision warehouse inventory. Therefore, it is necessary to seek more efficient, accurate, and intelligent warehouse inventory solutions to meet the needs of modern warehouse management. Computer vision technology plays an increasingly important role in modern society, and its necessity is reflected in many aspects: computer vision is used in the automatic driving system of automobiles to identify road signs, pedestrians, and other vehicles to ensure safe driving. At the same time, traffic monitoring systems use computer vision technology to monitor traffic conditions in real time and optimize traffic flow and safety. Various places such as banks, stores, and public transportation systems widely use computer vision to detect suspicious behavior, identify facial features, and track the process of events, improve safety, and prevent crime. Computer vision systems on automated production lines can detect product surface defects, dimensional deviations, or assembly errors, and improve production efficiency and product quality. Computer vision technology has greatly improved all aspects of people's lives and work through automation, improved efficiency, and enhanced safety, and has also provided new technical support for warehouse management.

[0010] Researchers began to explore the possibility of applying computer vision technology to the field of warehouse inventory. They used cameras and image processing technology to try to identify and count goods in the warehouse. However, due to technical limitations and algorithm complexity, the results of this stage were not ideal, with problems such as low recognition accuracy and slow processing speed.

[0011] With the rapid development of deep learning technology, researchers have begun to apply deep learning algorithms to the field of computer vision and have achieved remarkable results. In the field of warehouse inventory, automatic inventory counting algorithms based on deep learning have also begun to appear. These algorithms can more accurately identify goods in the warehouse and count them quickly by training a large amount of image data.

[0012] With the continuous maturity of technology and continuous optimization of algorithms, unmanned warehouse automatic inventory counting algorithms based on computer vision technology have begun to be applied in actual warehousing environments. These systems can capture images of goods in the warehouse in real time, and provide accurate data support for warehouse management through automatic identification and counting. At the same time, these systems can also be integrated with warehouse management systems to achieve real-time update and sharing of inventory data.

[0013] The application of computer vision technology in warehouse management is becoming more and more extensive, and computer vision technology is playing an increasingly important role in warehouse management. The technical innovations based on computer vision technology include the ability to adapt to complex warehousing environments, real-time updating of inventory information, diversified cargo identification, intelligent horizontal drafts, and strong security and privacy protection capabilities.

[0014] Computer vision technology can adapt to various complex storage environments, including light changes, goods occlusion and stacking. This enables the system to accurately identify and count goods under different conditions. The system can collect and process image information in the warehouse in real time, update inventory data in a timely manner, and provide accurate data support for warehouse management and logistics scheduling. This real-time performance is not achievable with traditional methods. Through advanced image processing and deep learning algorithms, the system can identify various types of goods, including items of different shapes, colors and sizes, and has strong adaptability. Using deep learning and artificial intelligence technology, the system can continuously learn and optimize to improve the accuracy and efficiency of identification and counting. This level of intelligence has greatly improved the automation and modernization of warehouse management. Compared with the security and privacy issues that RFID technology may face, the computer vision-based system realizes inventory through image processing, avoiding the risk of illegal reading or tampering of tag information.

[0015] In view of the above problems, there is an urgent need for an automatic inventory method and system for goods on unmanned warehouse shelves. Summary of the invention

[0016] In order to solve the deficiencies in the prior art, the present invention provides a method and system for automatic inventory counting of goods for unmanned warehouse shelves, which detects goods through a target detection algorithm, and classifies and tracks the goods by identifying the characteristics of the goods.

[0017] The present invention adopts the following technical solution.

[0018] A first aspect of the present invention relates to an automatic inventory method for unmanned warehouse shelves, the method comprising the following steps: extracting a cargo boundary box from a captured continuous cargo image using a target detection algorithm; for each cargo boundary box, calculating the shape features, color features, texture features, size features and local features of the cargo in the cargo boundary box, and splicing a feature vector of the cargo based on the above features; pre-defining cargo templates of different types of cargo, matching the cargo templates with the feature vectors of the cargo to obtain a category label for each cargo boundary box, and iteratively optimizing the target detection algorithm through a loss function; using a Kalman filter target tracking algorithm to track the cargo extracted by the target detection algorithm, and obtaining the location and quantity information of the cargo from the continuous cargo image.

[0019] Preferably, extracting the cargo bounding box from the captured continuous cargo images using a target detection algorithm includes: extracting the cargo bounding box from the continuous cargo images using a YOLO or SSD algorithm.

[0020] Preferably, for each cargo boundary box, the shape features, color features, texture features, size features and local features of the cargo in the cargo boundary box are calculated, including: using a deep learning model to extract the contour of the cargo in the cargo boundary box, and generating a cargo contour map, calculating a Fourier descriptor sequence through the cargo contour map, and calculating the geometric moments and invariant moments of the cargo contour map; performing color space conversion on the image in the cargo boundary box, and calculating the first-order moment, second-order moment, and third-order moment of the color of the cargo boundary box in different color spaces; constructing a gray level co-occurrence matrix of the image in the cargo boundary box, and extracting feature quantities of the gray level co-occurrence matrix; calculating the length, width and height of the cargo boundary box; using MDNet to track CNN online, and inputting the image in the cargo boundary box into CNN to obtain the output of the CNN.

[0021] Preferably, the feature vector of the goods is spliced ​​out according to the above features, including: using the Fourier descriptor sequence, geometric moment, and invariant moment as the shape feature of the goods boundary box; using the first-order moment, second-order moment, and third-order moment of the color of the goods boundary box in the different color spaces as the color feature of the goods boundary box; using the feature quantity of the gray level co-occurrence matrix as the texture feature of the goods boundary box; using the length, width, and height of the goods boundary box as the size feature of the goods boundary box; using the output of the CNN as the local feature of the goods boundary box; encoding the shape feature, color feature, texture feature, size feature, and local feature to splice them into the feature vector of the goods.

[0022] Preferably, cargo templates of different types of cargo are predefined, and the cargo templates are matched with feature vectors of the cargo, including: the cargo templates of different types of cargo are standard feature vectors of different types of cargo; the standard feature vectors are obtained by pre-collecting standard images of different types of cargo and performing feature extraction, feature encoding and splicing on the standard images; and the cargo templates are matched with the feature vectors of the cargo by one or more of Euclidean distance, Manhattan distance, cosine similarity, and Pearson correlation coefficient.

[0023] Preferably, obtaining the category label of each cargo boundary box includes: analyzing the matching result between a current cargo boundary box and a type of cargo template; if the matching result is less than a certain classification threshold, identifying the current cargo frame boundary as a certain classification, and defining a category label for the current cargo frame boundary; if the matching result does not meet the classification requirements, marking the cargo category as an unknown category.

[0024] Preferably, the target detection algorithm is iteratively optimized through a loss function, including: the loss function is the sum of the MSE loss, confidence loss, and classification loss of the YOLO or SSD algorithm; the training parameters of the target detection algorithm are updated with the loss as the target to update the cargo boundary box output by the target detection algorithm.

[0025] Preferably, the goods extracted by the target detection algorithm are tracked using a Kalman filter target tracking algorithm, and the location and quantity information of the goods are obtained from the continuous goods images, including: tracking the goods using a Kalman filter target tracking algorithm, associating multiple goods extracted from the continuous goods images as the same target; calculating the target positions at different times in the continuous goods images, and the number of targets after associating them as the same target, thereby obtaining the location and quantity information of the goods.

[0026] The second aspect of the present invention relates to an automatic inventory system for unmanned warehouse shelves, and the system is implemented by using an automatic inventory method for unmanned warehouse shelves in the first aspect of the present invention; wherein the system includes a capture module, a calculation module, a matching module, and a tracking module; the capture module is used to extract a cargo boundary box from the captured continuous cargo image using a target detection algorithm; the calculation module is used to calculate the shape features, color features, texture features, size features and local features of the cargo in the cargo boundary box for each cargo boundary box, and splice the feature vector of the cargo according to the above features; the matching module is used to predefine cargo templates of different types of cargo, match the cargo templates with the feature vectors of the cargo, thereby obtaining a category label for each cargo boundary box, and iteratively optimize the target detection algorithm through a loss function; the tracking module is used to track the cargo extracted by the target detection algorithm using a Kalman filter target tracking algorithm, and obtain the location and quantity information of the cargo from the continuous cargo image.

[0027] A third aspect of the present invention relates to a terminal, comprising a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the method in the first aspect of the present invention.

[0028] The beneficial effect of the present invention is that, compared with the prior art, the method and system for automatic inventory counting of goods for unmanned warehouse shelves in the present invention detects goods through a target detection algorithm, and classifies and tracks the goods by identifying the characteristics of the goods. The method of the present invention is highly effective and accurate, greatly reduces labor costs, realizes automatic, fast and accurate inventory counting of goods in the warehouse, and improves the intelligent level of warehouse management.

[0029] The beneficial effects of the present invention also include:

[0030] The unmanned warehouse automatic inventory counting algorithm uses advanced computer vision technology to greatly improve the speed and efficiency of inventory counting. Traditional manual inventory counting usually takes a lot of time, while the unmanned inventory counting system using computer vision technology can run 24 / 7, scanning and recording inventory status in real time. Through automated image processing and analysis, a large amount of inventory counting work can be completed without human intervention, which greatly reduces the inventory counting cycle and improves the overall operational efficiency of the warehouse. The unmanned warehouse automatic inventory counting system can significantly reduce the inventory counting error rate through high-precision image recognition and data analysis.

[0031] Computer vision technology can accurately identify the characteristics of each product, whether it is a barcode, QR code or the shape of the item, ensuring the accuracy and reliability of the inventory data. The unmanned warehouse automatic inventory system significantly reduces the reliance on manual labor, thereby reducing labor costs. Traditional warehouses need to employ a large number of employees to conduct regular inventory, while the automatic inventory system can replace most of the manual work. By reducing manual participation, not only the salary and training costs are reduced, but also the potential health and safety risks caused by manual operation are reduced.

[0032] Modern warehouse environments are complex and ever-changing, with a wide variety of goods and various storage methods. The unmanned warehouse inventory counting system based on computer vision technology can adapt to various complex environments, accurately identify and record goods information through multi-angle and multi-light source image acquisition and processing. Even in low light or stacked goods, computer vision algorithms can still work efficiently to ensure the integrity and accuracy of inventory data.

[0033] The unmanned warehouse inventory counting system can update inventory data in real time, ensuring that warehouse managers are always aware of the latest inventory status. This is of great significance for quickly responding to changes in market demand and optimizing inventory management strategies. Real-time data updates can also be seamlessly integrated with other management systems (such as ERP, WMS) to form an integrated warehouse management solution, improving the transparency and responsiveness of the overall supply chain. Overall, the unmanned warehouse automatic inventory counting algorithm based on computer vision technology has brought significant efficiency improvements and cost savings to modern warehouse management. Through high-precision recognition and real-time data updates, the system can adapt to complex and changing warehouse environments and provide reliable and accurate inventory information. With the continuous advancement of technology, the unmanned warehouse inventory counting system will further promote the intelligence and automation of warehouse management and become an indispensable and important part of modern supply chain management.

[0034] By using advanced target detection algorithms (such as YOLO, SSD, etc.), the present invention can more accurately identify the type of goods in the image and determine its location and size, thereby reducing the possibility of misidentification or missed identification. Combined with advanced technologies such as deep learning, the model can continuously learn and optimize its own recognition capabilities to adapt to more diverse and complex cargo scenarios; real-time performance is crucial for cargo identification, especially in logistics, warehousing and other scenarios. The algorithm used in the present invention not only ensures accuracy, but also focuses on improving processing speed to ensure that a large amount of image data can be quickly responded to and processed. By optimizing the coordination of algorithms and hardware equipment, the present invention can achieve efficient and rapid cargo identification and quantity statistics to meet real-time requirements; the present invention greatly reduces the need for manual intervention by automatically identifying the type, location and quantity of goods, and improves work efficiency and accuracy. Combined with other intelligent technologies (such as the Internet of Things, big data analysis, etc.), the present invention can also achieve higher-level intelligent management, such as cargo tracking, inventory warning and other functions; the present invention adopts a modular design, which is easy to expand and customize, and can be flexibly adjusted according to different application scenarios and needs. By updating and replacing different algorithm modules or hardware devices, the present invention can adapt to cargo identification tasks of different scales and complexities; the present invention can generate a large amount of data on cargo flow and inventory, which can provide strong support for corporate decision-making. By analyzing this data, companies can better understand market demand, optimize supply chains, predict sales trends, etc., and thus make more informed business decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the deployment of an automatic inventory counting system in an automatic inventory counting method for unmanned warehouse shelves of the present invention;

[0036] Figure 2 A schematic diagram of image noise reduction in an automatic inventory method for unmanned warehouse shelves according to the present invention;

[0037] Figure 3 It is a schematic diagram of cargo template matching in an automatic cargo inventory method for unmanned warehouse shelves of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present invention clearer and more accurate, the technical solution of the present invention is described in detail below through multiple specific implementation methods. The embodiments adopted by the present invention are only used to explain the present invention and are not used to limit the content of the present invention.

[0039] This paper proposes an automatic inventory counting algorithm for unmanned warehouses based on computer vision technology, aiming to solve the problems of low efficiency, high error rate and high labor cost in the traditional warehouse inventory counting process. The algorithm combines advanced image acquisition, preprocessing, feature extraction, cargo identification, quantity statistics and result output technologies to achieve automatic, fast and accurate inventory counting of cargo in the warehouse, thereby greatly improving the intelligent level of warehouse management.

[0040] Figure 1 FIG. 1 is a schematic diagram of the deployment of an automatic inventory counting system in an automatic inventory counting method for unmanned warehouse shelves of the present invention. Figure 1 As shown, in order to solve the existing technical problems and achieve the above-mentioned purpose, the technical solution provided by the present invention is an automatic inventory method for unmanned warehouses, which is a new prediction framework. Specifically, the collected images are processed by image acquisition and preprocessing, and the key features of the goods are extracted from the preprocessed images using a deep learning algorithm, such as a convolutional neural network (CNN). These features may include shape, size, color, texture, etc., which can be used to distinguish different types of goods. The extracted feature vector is matched with a predefined cargo template to identify the type of goods. According to the identified type of goods, the number of each type of goods is counted. The target detection algorithm (such as YOLO, SSD, etc.) can be used to locate and count the goods in the image to improve the accuracy of statistics. Detect abnormal conditions in the warehouse, such as damage and missing goods, and perform data management on the inventory results, including storage, query, analysis, etc. Perform data management on the inventory results, including storage, query, analysis, etc. Establish a database to store inventory data, including information such as cargo type, quantity, location, time, etc. Provide a query function to facilitate users to view the inventory status and historical inventory records of the warehouse at any time. Use data analysis technology to conduct in-depth mining of inventory data to discover potential optimization points and improvement space.

[0041] A first aspect of the present invention relates to an automatic inventory method for goods on unmanned warehouse shelves, the method comprising steps 1 to 4.

[0042] Step 1: Use the object detection algorithm to extract the cargo bounding box from the captured continuous cargo images.

[0043] Image acquisition involves deploying cameras or other image acquisition devices in the warehouse to capture real-time images of goods in the warehouse.

[0044] Computer vision technology uses cameras and other image acquisition devices to capture the operating status of machines and equipment in unmanned warehouses in real time. Through image processing and feature extraction technology, the system can identify each machine and equipment and build spatial correspondence information between them. This information reflects the physical layout, workflow design, and data transmission path between the equipment, thereby revealing the degree of working correlation between them.

[0045] Choose suitable cameras or other image acquisition devices according to the size and structure of the warehouse. Choose different types of cameras, such as fixed cameras that continuously observe a key location for a long time, such as shelves, entrances and exits; dynamic cameras (PTZ) that cover multiple activity areas or require high-precision monitoring tasks; 3D cameras for stereo monitoring. Set the camera's acquisition parameters according to actual needs, such as frame rate, exposure time, white balance, etc. If multiple cameras are used for image acquisition, it is necessary to ensure synchronization between them for subsequent data processing and analysis. Calibrate the camera to eliminate distortion and error and improve image quality. Use a distributed and reasonably designed storage solution to ensure the security and accessibility of image data, and use HDFS for unified management to ensure automatic distribution and redundancy of data between nodes.

[0046] Preprocess the original image, image denoising eliminates the noise in the image and improves the image quality. Use median filtering to slide a window on the image, and replace the value of the central pixel of the window with the median of the pixel value in the window, thereby eliminating isolated noise points. Median filtering is a nonlinear filtering technology. Its basic principle is to replace the value of a point in a digital image or digital sequence with the median of the values ​​of each point in a neighborhood of the point, thereby eliminating isolated noise points.

[0047] Median filtering uses a fixed-size window (usually an odd size, such as 3x3, 5x5, etc.) to cover each pixel in the image. The choice of window size depends on the details and noise level of the image. Compared with linear filtering methods such as mean filtering, median filtering can better preserve the edge details of the image while removing noise. For each pixel value in the window, sort it according to its grayscale or color value. This can be achieved by various sorting algorithms such as quick sort, bubble sort, insertion sort, etc. The value in the middle of the sorted sequence is selected as the filtering result. If the window size is an odd number, the median is the value in the middle of the sorted sequence; if the window size is an even number, the median is usually the average of the two middle values. The calculated median is used as the new value of the center pixel of the window, replacing the original pixel value. Slide the window over the entire image and repeat the above steps until all pixels have been processed.

[0048] Figure 2The present invention is a schematic diagram of image denoising in an automatic inventory method for goods in unmanned warehouse shelves. After using median filtering, considering the process of using neural network RNN, an effective system based on batch normalization (BN) and recursive neural network or RNN technology is proposed to remove white noise, salt noise and pepper noise by incorporating long short-term memory (or LSTM). BN is applied to the update of LSTM units and the conversion from input to hidden and from hidden to hidden. Since the forget gate in the memory unit is a very important part, it plays the role of removing unimportant unit information, so it is critical and necessary to perform BN processing on it during the unit update process.

[0049] The BN transformation is defined as follows:

[0050]

[0051] where β and γ represent the mean and standard deviation of the normalized activations, and ∈ is a regularization hyperparameter used for smoothing to prevent is zero, the unit of the constructed enhanced LSTM model is as follows Figure 2 .

[0052] Convert the denoised color image into a grayscale image to simplify the image information. For each pixel (x, y) in the color image, its grayscale value G(x, y) can be obtained by the weighted average of the three components R, G, and B, that is, G(x, y) = wR*R(x, y)+wG*G(x, y)+wB*B(x, y), where wR, wG, and wB are weights, usually 0.299, 0.587, and 0.114, respectively.

[0053] Image enhancement can improve the visual effect of an image. Grayscale transformation is the most basic image enhancement method, which improves the contrast and brightness of an image by adjusting the grayscale value of the image. The present invention can adopt linear transformation, logarithmic transformation, power law transformation and the like. The grayscale histogram of the image is converted into a uniformly distributed form through a transformation function, thereby enhancing the contrast of the image and making the details more clearly visible.

[0054] During the image preprocessing process, the features of the original image can be collected, and these features can also be used as part of the feature vector of the goods in step 2.

[0055] Divide the image into gray levels and calculate the percentage of pixels at each gray level. Calculate the cumulative distribution for each gray level. For each pixel in the original image, replace its gray level value with the corresponding cumulative amount or distribution amount, map the original image, and obtain the equalized image.

[0056] Count the frequency of each gray level in the image and calculate the pixel proportion p of each gray level i, for each gray level i, calculate its cumulative proportion s i ,Right now For each pixel in the original image, replace its pixel value with the corresponding s i , each pixel value of the original image is mapped to a new pixel value to obtain the equalized image.

[0057] Detecting edge information in an image can prepare for subsequent image segmentation and feature extraction. Using Canny edge detection includes Gaussian filtering, calculating gradient strength and direction, non-maximum suppression, double threshold processing, and edge tracking. First, apply a Gaussian filter to the input image to reduce the impact of noise. The formula for the Gaussian filter is:

[0058]

[0059] Among them, (x, y) is the pixel coordinate in the image, σ is the standard deviation of the Gaussian function, which controls the degree of smoothing. By convolving the Gaussian filter with the image, a smoothed image can be obtained. On the Gaussian filtered image, the Sobel operator or other edge detection operator is used to calculate the gradient magnitude and direction of each pixel. The gradient magnitude calculation formula is:

[0060]

[0061] Among them, Gx and Gy are the gradients of the image in the x and y directions respectively. At the same time, the direction of the gradient can be obtained, usually expressed as an angle or radian. Non-maximum suppression: Non-maximum suppression is performed on the gradient amplitude image, the peak points of the local gradient amplitude are retained, and non-peak points are suppressed. This step determines whether the point is an edge point by comparing the gradient amplitude of the pixel with the pixels in its neighborhood. Double threshold processing: Threshold processing is performed on the image after non-maximum suppression using two thresholds (usually called high threshold and low threshold). Pixels with gradient amplitudes exceeding the high threshold are considered strong edge points, and pixels below the low threshold are considered non-edge points. For pixels with gradient amplitudes between the high and low thresholds, whether they are edge points is determined based on their connectivity with strong edge points. Edge tracking and connection: Through the above steps, we have obtained strong edge points and possible weak edge points. According to certain rules (such as 8 connectivity), the edge points are connected into complete edge lines. Image pyramid-based methods can perform edge detection at multiple scales by using image pyramids of different scales, and merge the results to obtain more comprehensive edge information.

[0062] Image augmentation is a technique used to increase the diversity of image datasets. It is often used in deep learning and computer vision tasks to improve the generalization ability of the model. The following are the main methods and steps for image augmentation: Identify the key features of goods on unmanned warehouse shelves, such as shape, size, color, texture, etc. Determine the augmentation target, such as improving the robustness of the model to changes in the position, angle, and illumination of goods. Flip the image left and right to simulate the different orientations of the goods on the shelf; randomly rotate the image within a certain angle range (such as ±15°) to simulate different angles of the goods on the shelf; randomly crop a part of the image to simulate different positions and sizes of the goods in the image; randomly move the image on the image plane to simulate slight changes in the position of the goods on the shelf; convert the image from one color space (such as RGB) to another color space (such as HSV). The above method can also be used to generate a cargo template.

[0063] Use the deep learning framework PyTorch to write code for image augmentation. During the training process, apply augmentation methods to the images of goods on the shelves of unmanned warehouses to generate augmented images as training data. Ensure that the training dataset contains sufficiently diverse augmented images to improve the generalization ability of the model. Use unaugmented test data to test the trained model to evaluate the accuracy and robustness of the model in identifying goods. The optional part of the above method is used to implement image preprocessing.

[0064] Preferably, the object detection algorithm is used to extract the cargo boundary box from the captured continuous cargo images, including: using YOLO or SSD algorithm to extract the cargo boundary box from the continuous cargo images.

[0065] Use target detection algorithms (such as YOLO, SSD, etc.) to complete target detection. It is widely used because of its fast speed and high accuracy. The core idea of ​​YOLO is to transform the target detection problem into a regression problem, and predict the bounding box and category probability directly on the image. The following is a detailed introduction to YOLO. The main idea of ​​YOLO is to divide the input image into S×S grids, and each grid cell is responsible for predicting the target inside it. Each grid cell predicts a fixed number of bounding boxes and their confidence scores, and predicts the category probabilities of the targets in these boxes. The network architecture of YOLO is based on convolutional neural networks (CNNs). The following are the specific steps of its architecture.

[0066] The input image is usually resized to a fixed size, such as 416x416 or 608x608; multiple convolutional layers are used to extract features of the image, usually followed by a maximum pooling layer to reduce the size of the feature map; the convolutional features are mapped to bounding box coordinates and category probabilities; the final output is a tensor of S×S×(B×5+C), where B is the number of bounding boxes predicted by each grid cell, 5 represents the 4 coordinates and 1 confidence score of the bounding box, and C is the number of categories; YOLO's prediction YOLO divides the entire image into an S×S grid, and each grid cell predicts B bounding boxes.

[0067] Each bounding box contains the following information: (x, y) is the offset of the center of the bounding box relative to the grid cell; w, h are the width and height of the bounding box, normalized relative to the entire image; confidence score: indicates the confidence that the bounding box contains an object and the accuracy of the bounding box; category probability: predicts the probability that the object belongs to each category.

[0068] Unmanned warehouses need to process a large amount of cargo information in real time, so the YOLO algorithm can effectively improve target detection efficiency. YOLO completes detection in a single forward propagation, with high speed and accuracy. The algorithm can detect multiple cargoes at the same time and maintain high detection performance in complex scenes such as occlusion and light changes.

[0069] Step 2: For each cargo boundary box, calculate the shape features, color features, texture features, size features and local features of the cargo in the cargo boundary box, and splice the feature vector of the cargo based on the above features.

[0070] Preferably, for each cargo boundary box, the shape features, color features, texture features, size features and local features of the cargo in the cargo boundary box are calculated, including: using a deep learning model to extract the contour of the cargo in the cargo boundary box, and generating a cargo contour map, calculating a Fourier descriptor sequence through the cargo contour map, and calculating the geometric moment and invariant moment of the cargo contour map; performing color space conversion on the image in the cargo boundary box, and calculating the first-order moment, second-order moment, and third-order moment of the color of the cargo boundary box in different color spaces; constructing a gray level co-occurrence matrix of the image in the cargo boundary box, and extracting the feature quantity of the gray level co-occurrence matrix; calculating the length, width and height of the cargo boundary box; using MDNet to track CNN online, and inputting the image in the cargo boundary box into CNN to obtain the output of CNN.

[0071] To extract shape features, you need to select a deep learning model suitable for edge detection, such as a convolutional neural network (CNN) or a fully convolutional network (FCN). Use a dataset with edge annotations to train the deep learning model so that it can learn the edge features in the image. Contour extraction is based on the detected edges to extract the contour of the goods. Shape description uses Fourier descriptors, moment descriptors and other methods to describe the shape. The basic idea of ​​the Fourier descriptor is to use the Fourier transform of the object boundary as a shape description. The formula of a commonly used Fourier descriptor is as follows:

[0072]

[0073] Among them, s(k) is the complex coordinate of the kth point on the boundary of the object (including the horizontal and vertical coordinate information); N is the number of boundary points; u is the order of Fourier transform. By calculating the Fourier coefficients S(u) of different orders, a Fourier descriptor sequence describing the shape of the object can be obtained. This sequence contains global and local information of the shape and is robust to noise and small shape changes.

[0074] Moment descriptors, especially geometric moments and invariant moments, are another commonly used shape description method. Here we take the Hu invariant moments as an example, which are a set of moment statistics with translation, rotation and scaling invariance. The formulas of Hu invariant moments are more complicated because they are a combination of lower-order moments (such as second-order moments, third-order moments, etc.) based on the regional grayscale distribution. The following is a part of the Hu invariant moments, starting with η pq Represents the original central moment:

[0075] θ1=η 20 +η 02

[0076] θ2=(η 20 -η 02 ) 2 +4η 11 2

[0077] 1.θ3=(η 30 -3η 12 ) 2 +(3η 21 -η 03 ) 2

[0078] η pq is the p+q order central moment, where p and q are non-negative integers. Hu invariant moments are invariant to translation, rotation, and scaling operations in the image, making them well suited for shape recognition and comparison.

[0079] Color feature extraction requires color space conversion, color histogram calculation, and color moment calculation. In color feature extraction, common color spaces include RGB, HSV, Lab, etc. RGB is a color space based on red, green, and blue, while HSV describes color based on hue, saturation, and value. The formula for color space conversion depends on the specific conversion method. The color histogram is a global color feature that describes the numerical distribution of pixel colors in an image. Suppose we have an image where the color of each pixel can be represented by a numerical value (such as a value in a certain color space). Color moment is a simple color feature representation method that uses moments of color distribution (such as mean, variance, and skewness) to describe color features.

[0080] For each color channel (such as RGB channel), we can calculate its color moment. The following are the calculation formulas for the first-order moment (mean), second-order moment (variance), and third-order moment (skewness): First-order moment (mean): mean = ∑ (p i *c i ) / ∑p i , where p_i is the probability (or number of pixels) of color c_i appearing in the image. Second-order moment (variance): variance = ∑((p i *c i -mean) 2 ) / ∑p i . Third-order moment (skewness): skewness = ∑((p i *c i -mean) 3 ) / ∑p i The first-order moment (mean) describes the central position of the data distribution, the second-order moment (variance) measures the degree of dispersion of the data points relative to the mean, and the third-order moment (skewness) reflects the asymmetry of the data distribution, that is, the direction and degree of deviation of the tail of the data.

[0081] The texture features of the image are calculated using the gray-level co-occurrence matrix (GLCM), where T(N×N), N is the number of gray-level divisions, and M(i, j) represents the frequency of pixels with gray levels i and j appearing simultaneously at a point and at a point with a span along the defined direction. M(i, j) = traverses all pixel pairs (p1, p2) in the image, if the gray value of p1 is i, the gray value of p2 is j, and the distance and direction between p1 and p2 meet the preset conditions, then M(i, j) is increased by 1; by analyzing the GLCM, a variety of texture features can be extracted, such as contrast, correlation, energy, and homogeneity, thereby comprehensively describing the texture information of the image.

[0082] Size feature extraction uses the contour information to obtain the bounding box of the goods and then calculates the width, height, area and other parameters of the bounding box. Assuming that the coordinates of the upper left corner of the bounding box are (x_min, y_min) and the coordinates of the lower right corner are (x_max, y_max), then the width width = x_max-x_min, the height height = y_max-y_min, and the area area = width*height; these size parameters provide basic information about the size of the object, which is helpful for further analysis and processing.

[0083] Construct a convolutional neural network (CNN) model and use multiple convolution kernels to perform convolution operations on the input image to extract local features in the image.

[0084] Pre-trained deep CNN, applying the network structure of multi-domain training (Multi-Domain Network, MDNet), MDNet refers to a learning method that applies training data from multiple domains and domain information to the learning process. Multi-domain learning has been widely used in natural language processing. By sharing cross-domain information and features, the generalization ability and accuracy of the model on specific tasks are improved, especially in those fields where the amount of data is small or difficult to collect. MDNet receives image patches of a specific size as input, usually 107×107 RGB images; the network contains three convolutional layers (Convolution 1-3), which are similar to the corresponding parts of the VGG-M network, but the size of the feature map is adjusted according to the size of the input image; followed by two fully connected layers with 512 output units (FC4-5), which combine ReLU activation functions and Dropout to prevent overfitting; the key part of MDNet is that it has multiple fully connected layer branches (FC61 to FC6K) corresponding to different domains (i.e. training sequences). Each branch contains a binary classification layer with Softmax cross entropy loss to distinguish between targets and backgrounds in each domain. The VGG-M network is a variant of the VGG-16 architecture. It builds a deep network by stacking multiple 3x3 small convolution kernels and introduces nonlinearity after each convolution layer using the ReLU activation function. At the same time, a maximum pooling layer is added to the network to reduce the spatial size of the feature map, and finally the classification output is performed through several fully connected layers. This design enables VGG-M to have high computational efficiency while maintaining good performance, and is suitable for a variety of computer vision tasks such as image classification and feature extraction.

[0085] Dropout is a technique used to prevent overfitting during neural network training. It reduces the complexity of the model by randomly discarding a portion of neurons at each iteration. Specifically, in each training batch, Dropout sets the output of some neurons to zero with a certain probability (such as 0.5), forcing the network to learn more robust feature representations. This avoids the network's over-reliance on specific combinations of neurons and improves the generalization ability of the model. In the testing phase, Dropout is usually canceled and all neurons are used, but their weights are scaled by the Dropout probability.

[0086] MDNet is first pre-trained on a large and diverse video dataset to learn common features across multiple domains. Then, when tracking an object in a specific video sequence, MDNet is fine-tuned for that sequence to learn features unique to that sequence.

[0087] The convolutional neural network (CNN) model uses multiple convolution kernels to perform convolution operations on the input image to extract local features in the image. For the input image I, the convolution kernel K, the value of a certain position (i, j) of the output feature map F is:

[0088]

[0089] An activation function, such as ReLU, is usually added after the convolution layer to increase the nonlinearity of the network f(x) = max(0, x). Pooling operations are used to reduce the dimension of the feature map, reduce the amount of calculation, and retain important features. Through the trained CNN model, the image of the goods is input into the network, and after processing by the convolution layer, activation function layer, and pooling layer, the characteristic information of the goods is finally extracted.

[0090] Preferably, the feature vector of the goods is spliced ​​out according to the above features, including: using the Fourier descriptor sequence, geometric moment, and invariant moment as the shape feature of the goods boundary box; using the first-order moment, second-order moment, and third-order moment of the color of the goods boundary box in the different color spaces as the color feature of the goods boundary box; using the feature quantity of the gray level co-occurrence matrix as the texture feature of the goods boundary box; using the length, width, and height of the goods boundary box as the size feature of the goods boundary box; using the output of the CNN as the local feature of the goods boundary box; encoding the shape feature, color feature, texture feature, size feature, and local feature to splice them into the feature vector of the goods.

[0091] Among them, shape features describe the overall shape or outline of the goods, such as circle, rectangle, irregular shape, etc.; color features describe the color information of the goods, which can be described using color spaces such as RGB, HSV, etc.; texture features describe the texture information of the surface of the goods, such as roughness, smoothness, directionality, etc.; size features describe the size or dimension information of the goods, such as length, width, height, etc.

[0092] After quantifying the above features, a feature vector can be formed to represent the feature information of the goods. This feature vector can be multi-dimensional, with each dimension corresponding to a feature attribute. For some features, it may also be necessary to define their thresholds or ranges in order to determine whether the goods to be identified belong to the category of goods represented by the template during the recognition process.

[0093] Step 3: predefine cargo templates for different types of cargo, match the cargo templates with the feature vectors of the cargo, thereby obtaining the category label of each cargo bounding box, and iteratively optimize the target detection algorithm through the loss function.

[0094] For each cargo that needs to be identified, a cargo template needs to be predefined. This template usually contains the standard feature vectors of the cargo, which can be manually defined or extracted and averaged from multiple sample images of the cargo by training a CNN model. The main purpose of defining a cargo template is to provide a standard or reference for comparison with the features extracted from the actual cargo images during cargo identification, so as to achieve accurate classification and identification of the cargo. Templates are obtained in a similar way to cargo feature vectors.

[0095] Preferably, cargo templates of different types of cargo are predefined, and the cargo templates are matched with feature vectors of the cargo, including: the cargo templates of different types of cargo are standard feature vectors of different types of cargo; the standard feature vectors are obtained by pre-collecting standard images of different types of cargo and performing feature extraction, feature encoding and splicing on the standard images; and the cargo templates are matched with the feature vectors of the cargo by one or more of Euclidean distance, Manhattan distance, cosine similarity, and Pearson correlation coefficient.

[0096] Feature matching matches the feature vector extracted from the image of the goods to be identified with the predefined goods template. The matching method usually includes calculating the distance or similarity between the feature vectors. Common distance calculation methods include Euclidean distance, Manhattan distance, etc.

[0097] Taking the Euclidean distance as an example, assuming that F1 is the feature vector of the goods to be identified and F2 is the feature vector of the goods template, the Euclidean distance d between them can be calculated as:

[0098]

[0099] Here, n is the dimension of the feature vector.

[0100] Taking cosine similarity as an example, the cosine similarity s between F1 and F2 can be calculated as:

[0101]

[0102] The purpose of the identification decision is to judge the similarity or difference between the goods to be identified and the predefined goods template based on the result of feature matching, so as to determine the category to which the goods to be identified belong.

[0103] Figure 3 Schematic diagram of cargo template matching in an automatic inventory method for unmanned warehouse shelves of the present invention. Figure 3 As shown, the steps of identification decision-making include collating feature matching results, matching the features extracted from the image of the goods to be identified with the predefined goods template to obtain a series of matching results. These matching results may include the distance between feature vectors, similarity scores, or other quantitative indicators. Set a threshold or standard. According to the actual application scenario and requirements, set a threshold or standard to determine whether the matching result meets the requirements. The threshold or standard can be determined based on historical data, expert experience, or actual testing. Compare the matching results, and compare each matching result with the set threshold or standard. If the matching result meets the threshold or standard, it is considered that the goods to be identified are successfully matched with the corresponding goods template.

[0104] Preferably, obtaining the category label of each cargo boundary box includes: analyzing the matching result between a current cargo boundary box and a type of cargo template; if the matching result is less than a certain classification threshold, identifying the current cargo frame boundary as a certain classification, and defining a category label for the current cargo frame boundary; if the matching result does not meet the classification requirements, marking the cargo category as an unknown category.

[0105] The category of the goods to be identified is determined based on the successfully matched goods template. If multiple templates are successfully matched, they can be sorted and selected based on the degree of matching or other factors. If the goods to be identified do not match any predefined templates, further processing can be performed based on actual needs. For example, the goods to be identified can be marked as unknown categories, or matched with the most similar template and possible category suggestions can be provided.

[0106] Step 4: Use the Kalman filter target tracking algorithm to track the goods extracted by the target detection algorithm, and obtain the location and quantity information of the goods from the continuous goods images.

[0107] Preferably, the target detection algorithm is iteratively optimized through a loss function, including: the loss function is the sum of the MSE loss, confidence loss, and classification loss of the YOLO or SSD algorithm; the training parameters of the target detection algorithm are updated with the loss as the target to update the cargo boundary box output by the target detection algorithm.

[0108] YOLO's loss function consists of three parts: positioning loss, confidence loss, and classification loss. Positioning loss measures the error between the predicted bounding box and the actual bounding box, and is calculated using mean square error (MSE):

[0109]

[0110] Confidence Loss, which measures the error in the bounding box confidence score. It distinguishes between grid cells that contain objects and grid cells that do not contain objects:

[0111]

[0112] Classification Loss, which measures the error in the probability of the target category within each grid cell:

[0113]

[0114] YOLO is a real-time target detection algorithm that can process dozens of images per second and is suitable for applications in real-time video processing and autonomous driving. YOLO makes predictions on the entire image and is not prone to missed detections or false detections. YOLO transforms the target detection problem into a single regression problem with a simple architecture that is easy to train and deploy. YOLO takes into account contextual information, so it can better understand objects in the image, thereby reducing false detections of the background. YOLO can predict multiple objects at the same time, reducing redundant calculations and has good adaptability to new scenes.

[0115] As the algorithm continues to run in practical applications and data accumulates, the algorithm can be continuously optimized and iterated to improve its performance and accuracy, which may include improving feature extraction algorithms, optimizing matching strategies, adding new cargo templates, etc.

[0116] After identifying the type of goods, use the target detection algorithm (such as YOLO, SSD, etc.) to detect the position and size of each item in the image, and count the number of each item detected in the image. This is usually achieved by traversing the detected target boxes and accumulating them according to the category labels of the target boxes.

[0117] Preferably, a Kalman filter target tracking algorithm is used to track the goods extracted by the target detection algorithm, and the location and quantity information of the goods are obtained from the continuous goods image, including tracking the goods using the Kalman filter target tracking algorithm, associating multiple goods extracted from the continuous goods image as the same target; calculating the target positions at different times in the continuous goods image, and the number of targets after associating them as the same target, thereby obtaining the location and quantity information of the goods.

[0118] The application of tracking algorithms in cargo identification mainly focuses on real-time and accurate tracking of the location, status and possible abnormal behavior of cargo, in order to improve logistics efficiency and ensure cargo safety. Kalman filtering is an algorithm that uses the linear system state equation to optimally estimate the system state through system input and output observation data. This algorithm plays a key role in the process of tracking the state of dynamic cargo in the warehousing system. By continuously predicting and updating the state, it ensures that the system can accurately track the location of cargo even in the presence of noise interference. The algorithm can estimate the state of a dynamic system from a series of data with measurement noise when the measurement variance is known. The Kalman filter target tracking algorithm mainly includes two steps: prediction and update: Prediction: Based on the system state estimate and system model at the previous moment, predict the system state at the current moment. During the prediction process, the uncertainty and noise of the system will be considered. State prediction equation:

[0119] x k|k-1 =Fx k|k -1+Bu k

[0120] Error covariance prediction equation:

[0121] P k|k-1 =FP k|k-1 F T +Q

[0122] Update equation:

[0123] K k =P k|k-1 H T (HP k|k-1 H T +R) -1

[0124] x k|k =x k|k-1 +K k (z k -Hx k|k-1 )

[0125] P k|k =(IK k H)Pk|k-1

[0126] Among them, x k|k-1 is the predicted state at time k. k|k-1 is the prediction error covariance at time k. K k is the Kalman gain, which enables the system to adjust the prediction results according to the measured values ​​and achieve dynamic tracking. k is the observed value. F, B, H, Q, R, I are known matrices.

[0127] After identifying the type of goods, use the target detection algorithm (such as YOLO, SSD, etc.) to detect the position and size of each item in the image, and count the number of each item detected in the image. This is usually achieved by traversing the detected target box and accumulating it according to the category label of the target box.

[0128] In the process of goods identification and quantity counting, once the algorithm has completed the identification and quantity counting of the goods in the image, the next step is to output these results to the warehouse management system or other related systems for inventory updates, order processing, etc. The following are the detailed steps and considerations for result output and data update:

[0129] Before outputting the results, it may be necessary to process the raw data generated by the algorithm, such as format conversion, data cleaning, etc., to ensure that the output data can be correctly read and understood by the warehouse management system or other systems. Define the format and content of the data output according to the data interface specifications of the warehouse management system or other systems. This usually includes the definition of data fields, data types, data formats, etc. Through the defined interface, the processed data is output to the warehouse management system or other system. The output can be in the form of file transfer (such as files in CSV, XML, JSON, etc.), database connection, API call, etc.

[0130] Before updating inventory or other related data, the output data needs to be verified to ensure the accuracy and completeness of the data. This can be achieved by comparing with existing data, verifying the data format and range, etc. Data update: Once the data is verified, the new data can be updated to the warehouse management system or other systems. The update method can be to directly modify the records in the database, call the system's update interface, etc.

[0131] During the data update process, relevant log information needs to be recorded for subsequent tracking and auditing. Log information can include update time, update content, operator, etc. Ensure the security of data transmission to prevent data leakage or tampering. Measures such as encrypted transmission and identity authentication can be adopted. During the data update process, a fault-tolerant mechanism is designed to ensure timely recovery and continued operation when errors occur. When transmitting and updating large amounts of data, batch processing methods are used to improve efficiency. For data processing and transmission tasks that can be operated in parallel, parallel processing technology is used to shorten the time. Use a cache mechanism to reduce repeated calculations and data transmission and improve system response speed.

[0132] During system operation, especially in systems involving complex algorithms and data processing, optimization and iteration are key steps to ensure long-term stable operation of the system and improve performance and accuracy. The following are some specific measures that may be taken during system optimization and iteration, especially for the aspects you mentioned, such as improving feature extraction algorithms, optimizing matching strategies, and adding new cargo templates. Optimize matching strategies, optimize matching algorithms, such as adjusting similarity measurement methods, optimizing search strategies, etc., to improve matching speed and accuracy. Preprocess input data, such as standardization, normalization, and denoising, to reduce the impact of data quality issues on matching results. Enhance the model's ability to identify difficult situations by mining difficult-to-distinguish negative samples. Add new cargo templates, and continuously expand the cargo template library according to actual application needs to accommodate more types of cargo identification. As the appearance, size, and other attributes of the cargo change, update the templates in the template library in a timely manner to ensure the accuracy of identification. Use machine learning technology to achieve automatic learning and updating of templates. For example, automatically update the templates in the template library through user feedback or system self-evaluation.

[0133] The algorithm outputs the inventory count results to the warehouse management system or other related systems for inventory updates, order processing, etc.

[0134] The second aspect of the present invention relates to an automatic inventory counting system for unmanned warehouse shelves, and the system is implemented by using an automatic inventory counting method for unmanned warehouse shelves in the first aspect of the present invention; wherein the system includes a capture module, a calculation module, a matching module, and a tracking module; the capture module is used to extract a cargo boundary box from a captured continuous cargo image using a target detection algorithm; the calculation module is used to calculate the shape features, color features, texture features, size features and local features of the cargo in the cargo boundary box for each cargo boundary box, and splice a feature vector of the cargo based on the above features; the matching module is used to predefine cargo templates of different types of cargo, match the cargo templates with the feature vectors of the cargo, thereby obtaining a category label for each cargo boundary box, and iteratively optimize the target detection algorithm through a loss function; the tracking module is used to track the cargo extracted by the target detection algorithm using a Kalman filter target tracking algorithm, and obtain the location and quantity information of the cargo from the continuous cargo image.

[0135] A third aspect of the present invention relates to a terminal, comprising a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the method according to the third aspect of the present invention.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit them. Although the present invention is described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the technical solutions of the present invention still include contents that can modify or replace the specific embodiments of the present invention. Any modification or replacement that does not depart from the spirit and scope of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A method for automatic inventory counting of goods for unmanned warehouse shelves, characterized in that: The method comprises the following steps: The object detection algorithm is used to extract the cargo bounding box from the captured continuous cargo images; For each cargo boundary box, calculate the shape features, color features, texture features, size features and local features of the cargo in the cargo boundary box, and splice the feature vector of the cargo according to the above features; Predefine cargo templates of different types of cargo, match the cargo templates with feature vectors of the cargo, thereby obtaining a category label of each cargo boundary box, and iteratively optimize the object detection algorithm through a loss function; The Kalman filter target tracking algorithm is used to track the goods extracted by the target detection algorithm, and the location and quantity information of the goods are obtained from the continuous goods images.

2. The method for automatic inventory counting of goods for unmanned warehouse shelves according to claim 1 is characterized by: The method of extracting a cargo boundary box from the captured continuous cargo images using a target detection algorithm includes: A YOLO or SSD algorithm is used to extract a cargo bounding box from the continuous cargo images.

3. The method for automatic inventory counting of goods for unmanned warehouse shelves according to claim 2 is characterized by: The step of calculating the shape features, color features, texture features, size features and local features of the goods in each goods boundary box includes: Extracting the contour of the cargo in the cargo boundary box by using a deep learning model, and generating a cargo contour map, calculating a Fourier descriptor sequence through the cargo contour map, and calculating a geometric moment and an invariant moment of the cargo contour map; Performing color space conversion on the image within the cargo boundary frame, and calculating the first-order moment, second-order moment, and third-order moment of the color of the cargo boundary frame in different color spaces; Constructing a gray level co-occurrence matrix of the image within the cargo boundary box, and extracting feature quantities of the gray level co-occurrence matrix; Calculate the length, width and height of the cargo boundary box; Using MDNet online tracking CNN, the image within the cargo boundary box is input into the CNN to obtain the output of the CNN.

4. The method for automatic inventory counting of goods for unmanned warehouse shelves according to claim 3 is characterized by: The step of splicing the feature vector of the goods according to the above features includes: Using the Fourier descriptor sequence, geometric moment, and invariant moment as shape features of the cargo boundary box; Using the first-order moment, second-order moment, and third-order moment of the color of the cargo boundary box in the different color spaces as the color features of the cargo boundary box; Using the feature quantity of the gray level co-occurrence matrix as the texture feature of the cargo boundary box; The length, width and height of the cargo boundary box are used as the size features of the cargo boundary box; Using the output of the CNN as the local feature of the cargo bounding box; The shape features, color features, texture features, size features, and local features are encoded and concatenated into a feature vector of the goods.

5. The method for automatic inventory counting of goods for unmanned warehouse shelves according to claim 4 is characterized by: The pre-defining cargo templates of different types of cargo, and matching the cargo templates with the feature vectors of the cargo, comprises: The cargo templates of different types of cargo are standard feature vectors of different types of cargo; The standard feature vector is obtained by pre-collecting standard images of different types of goods and performing feature extraction, feature encoding and splicing on the standard images; The matching of the cargo template with the feature vector of the cargo is achieved by one or more of Euclidean distance, Manhattan distance, cosine similarity, and Pearson correlation coefficient.

6. The method for automatic inventory counting of goods for unmanned warehouse shelves according to claim 5 is characterized by: The step of obtaining the category label of each cargo bounding box includes: Analyze the matching result between the current cargo boundary box and a type of cargo template, and if the matching result is less than a certain classification threshold, identify the current cargo boundary box as a certain classification, and define a category label for the current cargo boundary box; If the matching result does not meet the classification requirements, the cargo category is marked as unknown.

7. The method for automatic inventory counting of goods for unmanned warehouse shelves according to claim 6 is characterized by: The iterative optimization of the target detection algorithm by using a loss function includes: The loss function is the sum of the MSE loss, confidence loss, and classification loss of the YOLO or SSD algorithm; The training parameters of the target detection algorithm are updated with the loss as a target, so as to update the cargo boundary box output by the target detection algorithm.

8. The method for automatic inventory counting of goods for unmanned warehouse shelves according to claim 7 is characterized by: The method of tracking the cargo extracted by the target detection algorithm using the Kalman filter target tracking algorithm and obtaining the location and quantity information of the cargo from the continuous cargo images includes: The Kalman filter target tracking algorithm is used to track the goods, and multiple goods extracted from the continuous goods images are associated as the same target; The target positions at different times in the continuous cargo images are calculated, and the number of targets associated with the same target is obtained, thereby obtaining the position and quantity information of the cargo.

9. An automatic inventory system for unmanned warehouse shelves, characterized by: The system is implemented by using an automatic inventory method for unmanned warehouse shelves as described in any one of claims 1 to 8; wherein, The system includes a capture module, a calculation module, a matching module, and a tracking module; The capture module is used to extract the cargo boundary box from the captured continuous cargo images using a target detection algorithm; The calculation module is used to calculate the shape features, color features, texture features, size features and local features of the goods in each goods boundary box, and splice the feature vector of the goods according to the above features; The matching module is used to predefine cargo templates of different types of cargo, match the cargo templates with the feature vectors of the cargo, thereby obtaining a category label of each cargo boundary box, and iteratively optimize the target detection algorithm through a loss function; The tracking module is used to track the goods extracted by the target detection algorithm using the Kalman filter target tracking algorithm, and obtain the location and quantity information of the goods from the continuous goods images.

10. A terminal comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1-8.

Images