Warehouse goods image recognition method based on deep learning

Abstract

The invention discloses a deep learning-based warehouse cargo image recognition method, and particularly relates to the technical field of warehouse cargo recognition, and the method comprises the steps: simulating the day and night illumination difference through an image enhancement module, and constructing a training data set covering a real scene; carrying out feature extraction by using an improved ResNet101 model, and reconstructing a 256-dimensional full-connection layer to output a high-discrimination cargo exclusive feature vector; a double-branch classification structure is innovatively adopted, the first branch judges the goods category, the second branch judges the packaging state, and according to a confidence coefficient threshold value, intelligent decision making of high-confidence-coefficient result automatic output and low-confidence-coefficient triggering rechecking is achieved; the system integrates an error analysis module, a performance evaluation module and a model dynamic optimization module, and supports scene-based directional incremental updating. The method overcomes the defect that the prior art is poor in adaptability to complex scenes, illumination changes and package anomalies, remarkably improves the recognition precision and the operation efficiency, and effectively reduces the inconsistency of inventory accounts.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of warehouse goods identification, and more particularly to a warehouse goods image identification method based on deep learning. BACKGROUND

[0002] Under the background of the deep integration of intelligent manufacturing and logistics automation, warehouse goods identification, as the core link of inventory management, sorting and turnover, and inventory reconciliation, has become a key enabling technology to support modern warehouse efficient operation, and its application is widespread in e-commerce warehouses, industrial spare parts libraries, and pharmaceutical cold chain warehouses.

[0003] At present, the mainstream technical path in this field mainly falls into two categories: one is the traditional machine vision-based identification method, which usually relies on artificially designed color, shape and other shallow features, and combines a simple classifier to realize goods identification; the other is the fixed architecture-based deep learning model, which directly uses a general network that is not optimized for warehouse scenarios for feature extraction. In addition, some improved schemes also try to adapt to different warehouse environments by manually labeling a small number of scene samples or manually adjusting model parameters.

[0004] However, these technologies still have some shortcomings in actual application, which are as follows:

[0005] The traditional method relies on artificially designed features, which is difficult to accurately depict the deep visual patterns of goods in complex states such as stacking occlusion and dynamic transmission blur; and the general deep learning model is not optimized for single goods flat, multi-layer stacking, and pipeline dynamics due to its fixed architecture, resulting in inaccurate feature extraction, which easily leads to category misjudgment, such as misjudging stacked goods as single pieces, or misreading label information due to motion blur;

[0006] Existing schemes generally lack effective processing mechanisms for significant light differences in warehouse environments; models are mostly trained based on samples under a single light condition, resulting in blurred packaging textures under strong light during the day and missing detailed features in dim environments at night, which ultimately leads to dramatic fluctuations in day and night recognition accuracy, making it difficult to meet the robustness requirements of stable operations throughout the day. SUMMARY

[0007] Therefore, the embodiments of the present application provide a warehouse goods image identification method based on deep learning, which provides the following technical solutions:

[0008] S1, an image acquisition module is used to acquire warehouse goods image data sets of target goods and goods of the same category in a warehouse scene, and the warehouse goods image data sets are stored in a database;

[0009] S2, an image enhancement module is used to extract and process the warehouse goods image data sets to simulate the light difference between day and night in the warehouse, and generate an enhanced image data set containing day and night light scenes;

[0010] S3, the deep learning feature extraction module is used for receiving an enhanced image data set, loading a pre-trained ResNet101 model, training the model by using a stochastic gradient descent algorithm, and outputting a cargo-specific feature vector;

[0011] S4, the cargo classification and identification module is used for receiving the cargo-specific feature vector, constructing a double-branch classification structure, and setting a classification threshold;

[0012] S5, the identification error analysis module is used for comparing the identification result of the cargo classification and identification module with the actual result confirmed by manual review, and calculating core error indicators;

[0013] S6, the comprehensive identification performance evaluation module calculates a comprehensive identification performance index based on the indicator data of the identification error analysis module by using a weighted scoring method;

[0014] S7, the model dynamic optimization and update module is used for triggering an optimization mechanism according to the result of the comprehensive identification performance evaluation module;

[0015] S8, the visual monitoring module is used for displaying the identification result, performance indicators, model optimization logs and abnormal alarms through a mobile terminal, so that management personnel can control the warehouse cargo image recognition state in real time.

[0016] The technical effects and advantages of the present application are as follows:

[0017] The image acquisition module of the present application can collect three types of core scene images, namely single cargo flat, multi-layer stacking and pipeline dynamic, and associate with metadata such as category label and packaging size to build a database. Combined with the image enhancement module, the scene is simulated according to the difference in day and night light, and it is not limited to traditional single scene sample collection and general enhancement for simple brightness adjustment. On the one hand, it can realize the deep matching of warehouse cargo image data and actual operation scene, cover all scene characteristics such as static, dynamic and stacking, and on the other hand, it can avoid the problem of insufficient model generalization ability caused by single scene in traditional scheme, provide high-quality, multi-dimensional training samples for subsequent feature extraction, and improve the basic precision of cargo identification from the data source.

[0018] The application is based on the reconstruction of a 256-dimensional exclusive feature generation layer of a ResNet101 model, sets a confidence threshold according to scenes in combination with a double-branch classification structure, matches the hierarchical training of a deep learning feature extraction module with the multi-index quantification of an error analysis module, is not limited to traditional artificial design features and fixed model architectures for general classification, on the one hand, strengthens the distinguishability of core visual features such as warehouse goods packaging texture and specification identification through the exclusive feature layer, and on the other hand, through the cooperation of the double branches, misjudgments caused by packaging abnormalities, stacking occlusion and dynamic blur can be filtered, at the same time, high-confidence results are directly outputted and low-confidence results trigger review by means of scientific threshold division, which not only avoids the excessive dependence of traditional schemes on artificial review, but also reduces the inconsistency between inventory accounts and actual inventory caused by misjudgments, and realizes the balance between high precision and high efficiency.

[0019] The application triggers an optimization mechanism according to scenes through a model dynamic optimization and updating module, verifies in combination with exclusive test sets of daytime, nighttime and mixed scenes, is not limited to the deployment mode of only retraining and overall parameter adjustment of a fixed model in traditional schemes, on the one hand, directional optimization is made for scene short boards such as strong light reflection in daytime and dimness at night, and on the other hand, the model bottom layer general feature layer is frozen through incremental learning, scene related parameters are updated, and original parameter compatibility is retained, so that the performance backoff after optimization does not affect the work, and the warehouse environment dynamic change is quickly adapted. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 It is a whole structure schematic diagram of the application.

[0021] Figure 2 It is a whole structure flow chart of the application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0023] As shown in the accompanying Figure 1 As shown in the accompanying

[0024] A warehouse goods image recognition method based on deep learning, characterized in that it comprises an image acquisition module, a database, an image enhancement module, a deep learning feature extraction module, a goods classification and recognition module, a recognition error analysis module, a comprehensive recognition performance evaluation module, a model dynamic optimization and updating module, a visual monitoring module and a mobile terminal.

[0025] The image acquisition module is used to acquire goods image data sets.

[0026] The database is used to store the cargo image dataset, the enhanced image dataset, the daytime scene exclusive test set, the night scene exclusive test set and the mixed scene exclusive test set.

[0027] The image enhancement module is used to generate the enhanced image dataset.

[0028] The deep learning feature extraction module is used to output the cargo exclusive feature vector.

[0029] The cargo classification and recognition module is used to construct a double-branch classification structure and set a classification threshold.

[0030] The recognition error analysis module is used to calculate the core error index.

[0031] The comprehensive recognition performance evaluation module is used to calculate the comprehensive recognition performance index.

[0032] The model dynamic optimization and updating module is used to trigger the optimization mechanism in different scenes.

[0033] The visual monitoring module provides visual display for the management personnel through the mobile terminal.

[0034] The mobile terminal is used to provide visual display for the management personnel.

[0035] As shown in the accompanying drawings: Figure 2

[0036] A warehouse cargo image recognition method based on deep learning, the specific implementation includes the following steps:

[0037] S1, the image acquisition module is used to acquire the warehouse cargo image dataset of the target cargo and the same kind of cargo in the warehouse scene and store it in the database.

[0038] As a preferred feasible embodiment, it needs to be explained that the warehouse cargo image dataset includes three categories, specifically:

[0039] ​One is a single flat static image without shelter. A fixed collection platform with uniform illumination in the warehouse is selected. A light gray non-reflective background plate is laid on the platform to avoid background interference. A single flat is placed on the platform to ensure that the package is wrinkle-free, stain-free and completely exposed. A Basler acA2500-14gm industrial camera with a resolution of 2592x1942 is used to take pictures from four angles: 50cm vertically above the goods, 30cm horizontally to the left, 30cm horizontally to the right, and 60cm at a 45° angle from the top of the goods. Each image contains only one complete good. This type of image is mainly used for basic category labeling of the model, helping the model learn the standard visual features of a single product category, such as packaging texture, logo style, and specification identification, laying the foundation for subsequent complex scene recognition.

[0040] Two is a 2-5 layer stacked goods image. The collection site is selected at the standard goods location of the actual warehouse shelf. The goods location size is consistent with the conventional warehouse goods location. According to the actual warehouse stacking specification, the same category of goods is stacked with aligned edges to avoid tilting, and 2, 3, 4, and 5 layers of goods are stacked respectively. A Keyence IV2 series track industrial camera installed above the shelf is used. The track height is 200cm from the bottom of the goods location, and can move along the length of the goods location. It is precisely positioned to two shooting positions by track movement, which are 100cm in front of the goods location and 120cm at a 30° angle from the front of the goods location, respectively. This ensures that the image can clearly present the interlayer structure and exposed features of the stacked goods. This type of image is used to let the model learn the local features of the goods in the stacked state, such as the exposed packaging corners and interlayer gap features after stacking, to avoid misjudgment of the category due to stacking obstruction.

[0041] Three is a dynamic goods image of the assembly line. The collection site is set at the middle section of the warehouse assembly line. The conveyor belt is 80cm wide and runs at a speed of 0.5m / s, consistent with the actual warehouse assembly line speed. The goods are placed on the conveyor belt according to the actual transmission interval, with one piece every 10cm, and 5-8 pieces per transmission. A Hikvision MV-CS050-10GM-PRO industrial camera with a frame rate synchronized with the assembly line is used, with a frame rate of 25 frames per second. The camera is installed directly above the conveyor belt and is linked to the conveyor belt through an encoder to ensure that the camera shooting time is synchronized with the position of the goods. The image naturally contains slight motion blur consistent with the actual running speed of the conveyor belt, without the need for additional processing. This type of image is used to train the model's ability to recognize dynamic targets and ensure accurate classification during the transmission of goods.

[0042] Further explanation is needed that each image also needs to record metadata information, and the metadata is stored by binding with the image through a unique ID, which specifically includes four contents: first, category label, which accurately labels the product category level to which the goods belong; second, package size, which records the specific values of the length, width and height of the goods package, which is used to judge the actual placement state of the goods in combination with the image size in the subsequent, such as whether there is tilting or misplacement; third, material attribute, which labels the material type of the goods package, such as paper, plastic, metal and glass, which assists the model in learning the light and shadow reflection characteristics of different materials, such as the light transmission of glass packaging and the light reflection of metal packaging; fourth, storage location, which records the physical location of the goods in the warehouse when the image is collected, such as the shelf area: A1 shelf 3 layers 2 columns, and the assembly line area: middle section of No. 1 conveyor belt, which is convenient for subsequent tracing of the recognition error of a specific area; in order to ensure the learning effect of the model, the total number of images collected for each product category is not less than 500, and the proportion of stacked images and assembly line images is not less than 30%, of which the proportion of stacked images is 15%-20%, and the proportion of assembly line images is 10%-15%, so as to avoid the lack of model generalization ability due to single scene.

[0043] S2, the image enhancement module is used to extract the warehouse goods image data set for processing to simulate the light difference between daytime and nighttime in the warehouse, and generate an enhanced image data set containing daytime and nighttime light scenes;

[0044] As a preferred feasible embodiment, it needs to be explained that the processing includes multi-dimensional enhancement operation on the warehouse goods image data set, which simulates the light difference between daytime and nighttime in the warehouse, and simultaneously optimizes the image adaptability to meet the model training requirements, realizes data expansion, and specifically:

[0045] First, the brightness, contrast, and saturation of the original image are adjusted in a targeted manner. For daytime scenes, the brightness is increased by 10-20%, the contrast is enhanced by 5-15%, and the saturation is kept natural. This simulates the bright environment in a daytime warehouse with natural light and artificial lighting superimposed. For nighttime scenes, the brightness is reduced by 20-40%, the contrast is reduced by 10-20%, and the saturation is reduced by 5-10%. At the same time, a slight light source color bias is added to simulate the dim environment in a nighttime warehouse relying only on emergency lighting or local lighting. Through targeted adjustment of the lighting parameters for daytime and nighttime scenes, the model learns the core visual features of goods under different lighting conditions. On this basis, geometric transformation operations are performed simultaneously. The image is randomly rotated by -15° to 15° to simulate the angle deviation of goods stacking, and randomly scaled by 0.8 to 1.2 times to match the imaging size under different shooting distances. At the same time, horizontal flipping is supported to cover the left-right orientation difference of goods. Through pose diversity enhancement, the model's adaptability to changes in goods placement is improved. In addition, controllable Gaussian noise and motion blur are added to the image. The standard deviation of Gaussian noise is controlled between 0.01 and 0.05, and the motion blur kernel size is set to 3x3. This simulates the imaging interference caused by equipment vibration and air dust in the actual warehouse. Finally, all enhanced images are adjusted to 640x640 pixels in size, and the pixel values are standardized to ensure that the output enhanced image dataset meets the model input format requirements, providing high-quality training samples for subsequent deep learning feature extraction.

[0046] It needs to be further explained that the selection of simulating daytime and nighttime lighting scenes instead of directly collecting real images is due to the balance between data effectiveness and collection cost. On the one hand, direct collection needs to deal with the limitations of warehouse scenes, such as low frequency of nighttime operations, incomplete coverage of goods types, and difficulty in controlling natural lighting differences. For example, the light on a cloudy day may overlap with the light on a sunny night, leading to sample imbalance or feature confusion in real data. On the other hand, through enhancement processing, lighting parameters can be precisely controlled based on existing base images, ensuring the stability of daytime and nighttime feature differences, generating a large number of diversified samples in a short time, significantly reducing collection costs, ensuring that the lighting robustness learned by the model has more generalization value, and avoiding recognition bias caused by accidental fluctuations in real scene lighting.

[0047] S3, the deep learning feature extraction module is used to receive the enhanced image dataset, load the pre-trained ResNet101 model, train the model using the stochastic gradient descent algorithm, and output the goods-specific feature vector;

[0048] As a preferred embodiment, it needs to be explained that the ResNet101 model is a deep convolutional neural network belonging to the residual network series; the residual network introduces residual learning, effectively solving the problems of gradient disappearance and performance degradation in the training process of deep neural networks, so that the network can still be efficiently trained and achieve excellent performance when reaching a very deep layer. In this embodiment, a pre-trained ResNet101 model is obtained from a deep learning open source platform. The pre-trained model is trained on large-scale image datasets such as ImageNet and has strong general image feature extraction capability.

[0049] The steps of training the model using the stochastic gradient descent algorithm are as follows:

[0050] a1, data preparation and loading;

[0051] The enhanced image dataset is divided according to the proportion of 70% training set, 15% validation set and 15% test set, and the image data is loaded in batches through the data loader; at the same time, the data is preprocessed, and the preprocessing operation is connected with the pixel value standardization to ensure the consistency of the input data distribution and avoid affecting the model training effect due to data format differences.

[0052] a2, model initialization and configuration;

[0053] The pre-trained ResNet101 model is loaded by default to adapt to the 1000-class general object classification task of the ImageNet dataset, and the output dimension of the last fully connected layer strictly matches the classification task; the goal of this embodiment is to extract the exclusive features of warehouse goods, rather than directly complete general object classification, so the model structure needs to be adjusted: remove the last fully connected layer of the original model, and add a new fully connected layer with an output dimension of 256; the output layer is the generation layer of the exclusive features of the goods; the essence of this adjustment is to strip the network components in the pre-trained model that are strongly bound to the general classification task, while retaining the general feature extraction capability of the deep convolutional layer, and through structural reconstruction, it focuses on learning the exclusive visual features of goods in the warehouse scene, providing more accurate feature input for the subsequent goods classification and recognition module.

[0054] a3, define the loss function and the optimizer;

[0055] According to the task attributes of warehouse goods feature extraction, the cross-entropy loss function is selected, the stochastic gradient descent optimizer is used, the learning rate is set to 1e -3 ~1e -4 , and the momentum is set to 0.9, which is used to update the model weights during training to balance the training efficiency and convergence stability.

[0056] a4, model training iteration;

[0057] The training set data is input into the model in batches, forward propagation calculation is performed, and a 256-dimensional goods feature output is obtained; the loss value between the output and the actual category label of the goods is calculated through the cross-entropy loss function; then the gradient of each parameter of the model is calculated according to the loss value by using the back propagation algorithm; finally, the weight parameters of the model are updated by the stochastic gradient descent optimizer; the model is trained for multiple rounds according to the above process, and the model performance is evaluated using the validation set every 15 rounds; if the validation set accuracy does not improve for 3 consecutive rounds or the loss value starts to rise, the learning rate decay strategy is used to adjust the training process to avoid model overfitting.

[0058] a5, model evaluation and saving.

[0059] After training, the model is comprehensively evaluated using the test set, and the core evaluation indicators include the classification accuracy after goods feature extraction, the intra-class similarity of the feature vector, and the inter-class discriminability; when the model performance meets the requirements, the trained model is saved, including the model structure and weight parameters, which can be directly called for feature extraction of the input image in the future, and a 256-dimensional goods-specific feature vector is output.

[0060] The goods-specific feature vector is the output result of the newly added 256-dimensional fully connected layer of the model, and each dimension of the vector corresponds to a specific visual feature of the goods, such as packaging texture features, specification identification features, etc.; and the Euclidean distance difference of the feature vectors of different categories of goods is ≥0.8, and the similarity of the feature vectors of the same category of goods is ≥0.9.

[0061] S4, the goods classification and recognition module is used to receive the goods-specific feature vector, construct a double-branch classification structure, and set a classification threshold;

[0062] As a preferred feasible embodiment, it needs to be explained that in the double-branch classification structure, the first branch is the goods category classification branch, which is mainly used to determine the specific category to which the goods belongs; this branch uses the combination architecture of fully connected layer + Softmax classifier, takes the 256-dimensional goods-specific feature vector output by the deep learning feature extraction module as input, first completes feature dimension compression and key information integration through 2 layers of fully connected layer, the first layer of fully connected layer compresses the 256-dimensional feature vector to 128-dimensional, and the second layer further reduces to a dimension matching the number of warehouse goods categories, and then calculates the probability value of each category of goods corresponding to the feature vector through the Softmax classifier, forming a category probability distribution result, to quantify the possibility of the goods belonging to each category.

[0063] The second branch is a cargo package state classification branch, which is an auxiliary branch of category identification, used to determine whether the cargo package is intact to avoid category misjudgment caused by abnormal package. The branch uses a support vector machine classifier, also takes the cargo exclusive feature vector as input, divides the feature vector into three categories of intact package, damaged package and label ambiguity through the construction of a classification hyperplane, and outputs the confidence of the corresponding state. The recognition result of the label ambiguity category can directly trigger the subsequent manual review process to reduce the recognition error caused by unclear identification.

[0064] The setting of the classification threshold needs to be determined in combination with the warehouse recognition accuracy requirement and the actual scene tolerance rate. The specific threshold division is as follows: for the first branch, when the probability value of a certain category of cargo is greater than or equal to 0.9, it is determined as a high-confidence recognition result, and the category is directly output. According to the test results of the verification set, the accuracy is greater than or equal to 98.5% and the misjudgment rate is less than 1.5% under this threshold, which can cover more than 80% of the normal scene and does not need to be reviewed. When the probability value is between 0.7 and 0.9, it is determined as a medium-confidence, which needs to be verified in combination with the second branch. The specific determination logic is: if the second branch outputs intact package, the category is confirmed, and if it is damaged / label ambiguity, it is pending review. According to the test results of the verification set, the accuracy is 92%-98% in this interval, but 32% of the errors come from package abnormalities, and 85% of the errors can be filtered by combining the state judgment. The lower limit is set to 0.7 to avoid invalid review increase; when the probability value is less than 0.7, it is determined as a low-confidence, which is directly pending manual review. According to the test results of the verification set, the accuracy is less than 80% under this threshold, the misjudgment is random, and the error correction rate is only 40%, which is easy to lead to more than 10% of the inventory error.

[0065] For the second branch, when the confidence is greater than or equal to 0.8, it is determined as an effective state, which assists the category identification. According to the verification results of the labeled samples, the accuracy of the three states is greater than or equal to 91% under this threshold, and the confusion rate is less than 8%, which can effectively assist the first branch to correct the error. When the confidence is less than 0.8, manual review is triggered. According to the verification results of the labeled samples, the state judgment in this interval has high uncertainty, and the misjudgment probability is as high as 45%. If it directly assists the identification, the error correction rate will decrease to 52%.

[0066] S5, the recognition error analysis module is used to compare the recognition result of the cargo classification identification module with the actual result confirmed by manual review, and calculate the core error index.

[0067] As a preferred feasible embodiment, it needs to be explained that the core error index includes: category misjudgment rate, state misjudgment rate, missed recognition rate and review dependence rate.

[0068] The category misjudgment rate is used to measure the accuracy of the goods category identification, and reflects the distinguishing ability of the model for different goods categories; the state misjudgment rate is used to evaluate the reliability of the goods packaging state judgment, and embodies the effectiveness of the second branch auxiliary function; the missed identification rate is used to measure the completeness of the model for identifying goods in the image, and avoids missing inventory due to non-identification; the review dependency rate is used to analyze the necessity of artificial review, and balances the identification efficiency and accuracy.

[0069] The specific calculation formula is:

[0070] The category misjudgment rate is (the number of samples with incorrect category identification ÷ the total number of identified samples) × 100%,

[0071] The number of samples with incorrect category identification refers to the number of samples whose model output category is inconsistent with the manually reviewed confirmed category, such as misjudging braised beef noodles as spicy beef noodles, and the total number of identified samples is the total number of all goods samples identified by the model in a certain statistical period;

[0072] The state misjudgment rate is (the number of samples with incorrect packaging state identification ÷ the total number of state identification samples) × 100%; the number of samples with incorrect packaging state identification refers to the number of samples whose second branch output packaging state is inconsistent with the manual review result, and the total number of state identification samples is the total number of samples judged by the second branch in the same period;

[0073] The missed identification rate is (the number of goods samples not identified by the model ÷ the total number of actual goods samples) × 100%; the number of goods samples not identified by the model includes the number of image samples for which the model does not output any category result, and if there are some goods not identified in a single image, the number of such goods is also included in the number of goods samples not identified by the model; the total number of actual goods samples is the total number of images containing goods in the same period, confirmed by manual annotation;

[0074] The review dependency rate is (the number of samples triggering artificial review ÷ the total number of identified samples) × 100%; the number of samples triggering artificial review includes the number of samples with low confidence results of the first branch, medium confidence combined with state abnormalities, and low confidence state samples of the second branch; if the same sample triggers multiple review conditions, it is only counted once as the number of samples triggering artificial review to avoid repeated statistics; the total number of identified samples is the total number of samples identified in the same period; this index is used to evaluate the efficiency of the model's autonomous identification.

[0075] S6, the comprehensive identification performance evaluation module calculates a comprehensive identification performance index using a weighted scoring method based on the index data of the identification error analysis module;

[0076] As a preferred embodiment, it needs to be explained that the weighted scoring method converts multi-dimensional error indicators into a single comprehensive index by quantifying and then weighting the core indicators output by the identification error analysis module, and the specific implementation is as follows:

[0077] Firstly, the weight distribution of each core error indicator is determined, and the weight is based on the priority demand of the warehouse scene for category accuracy, state reliability, detection integrity and identification efficiency. The sum of the weight proportions of the four core indicators needs to meet 100%.

[0078] Secondly, the mapping rule of indicators and scores is established: the actual value of each indicator is converted into a standardized score of 0-100 points; the specific rule is:

[0079] The category misjudgment rate ≤1% gets 100 points, in the interval of 1%-3%, according to: score = 100-[(actual misjudgment rate-1%) ÷(3%-1%)]x100%, linearly decreasing; >3% gets 0 points;

[0080] The state misjudgment rate ≤2% gets 100 points, in the interval of 2%-5%, according to: score = 100-[(actual misjudgment rate-2%) ÷(5%-2%)]x100%, linearly decreasing; >5% gets 0 points;

[0081] The missed identification rate ≤0.5% gets 100 points, in the interval of 0.5%-2%, according to: score = 100-[(actual identification rate-0.5%) ÷(2%-0.5%)]x100%, linearly decreasing; >2% gets 0 points;

[0082] The review dependency rate ≤10% gets 100 points, in the interval of 10%-25%, according to: score = 100-[(actual dependency rate-10%) ÷(25%-10%)]x100%, linearly decreasing; >25% gets 0 points.

[0083] Finally, the comprehensive identification performance index is calculated: the sum of the product of each indicator standardized score and the corresponding weight is obtained, the formula is:

[0084] The comprehensive identification performance index = (category misjudgment rate score x b1) + (state misjudgment rate score x b2) + (missed identification rate score x b3) + (review dependency rate score x b4), wherein b1, b2, b3, b4 are respectively the weight of category misjudgment rate, the weight of state misjudgment rate, the weight of missed identification rate, and the weight of review dependency rate; the index range is 0-100 points, wherein ≥90 points is excellent, the identification performance is stable and reliable; 75-89 points is good, needs slight optimization; 60-74 points is qualified, there is obvious short board; <60 points is unqualified, needs overall rectification.

[0085] S7, the model dynamic optimization and updating module is used for triggering an optimization mechanism according to the result of the comprehensive identification performance evaluation module in different scenes;

[0086] As a preferred feasible embodiment, it needs to be explained that the scene-triggered optimization mechanism is divided into two major warehouse core lighting scenes, day and night, and the performance differences of the comprehensive identification performance index in the corresponding scenes are combined to match the optimization strategy. Since there are significant differences in lighting characteristics and imaging interference between day and night scenes, the model is prone to performance short boards in a single scene, so the problem needs to be located according to the scene and optimized accurately. The specific scene and corresponding mechanism are as follows:

[0087] When the overall comprehensive identification performance index meets the standard, but there is a local error in the daytime scene, the daytime scene feature enhancement optimization mechanism is triggered: first, filter the abnormal samples of the daytime scene from the database, supplement 300-500 samples of the same kind and label; then, the high-level convolutional layer of the ResNet101 model is trained, the bottom layer of the general feature extraction layer is frozen, and only the high-level capture ability of the strong light reflection, high contrast shadow and other daytime specific features is updated; At the same time, the daytime scene preprocessing parameters in the image enhancement module are fine-tuned; after optimization, the daytime scene exclusive test set is verified to ensure that the daytime scene error index is reduced to the same level as the night scene, and the daytime scene reasoning parameters are updated after meeting the standard, which does not affect the night identification process.

[0088] It needs to be further explained that the daytime scene exclusive test set is an evaluation set collected independently from the training set / verification set / test set, which only contains daytime lighting scene goods image set, and the sample size is not less than 200, covering main goods categories.

[0089] When the overall comprehensive identification performance index meets the standard, but there is a local error in the night scene, the night scene noise reduction and feature enhancement optimization mechanism is triggered: first, extract the abnormal samples of the night scene, supplement 200-400 goods samples of different night lighting intensities, and expand the training set after multi-scale Retinex noise reduction preprocessing of the samples; then, a lightweight attention module is added in front of the feature extraction layer of the ResNet101 model to enhance the focusing ability of the model on the key area of the goods in the night scene, and the bottom 60% convolutional layer of the model is trained using incremental learning algorithm; At the same time, adjust the night scene judgment threshold of the second branch SVM classifier; after optimization, the night scene exclusive test set is verified to ensure that the core indicators of the night scene meet the standard, and the night reasoning logic of the model is updated, which is stored and called independently of the daytime scene parameters.

[0090] It needs to be further explained that the night scene exclusive test set is an evaluation set collected independently from the training set / verification set / test set, which only contains night scene goods image set, and the sample size is not less than 200, covering main goods categories.

[0091] When the comprehensive recognition performance index does not meet the standard, and there are obvious errors in both daytime and nighttime scenes, the dual-scene cooperative optimization mechanism is triggered: first, the core error indicators of daytime and nighttime scenes are respectively counted, and the short board samples of the dual scenes are supplemented in a targeted manner, and then the training set is expanded after unified image enhancement processing; then the ResNet101 model is trained in layers, the bottom 50% convolutional layers are frozen, the middle layer feature fusion layer and the top layer classification layer are updated synchronously, and the random gradient descent algorithm is used for full incremental training; after optimization, the daytime, nighttime and mixed scene test sets are verified respectively to ensure that the dual-scene comprehensive index is greater than or equal to 75 points, and there is no obvious error fluctuation in the mixed scene, and then the model is updated in different periods to avoid affecting the all-weather operation of the warehouse.

[0092] It needs to be further explained that the mixed scene exclusive test set is an evaluation set collected independently from the training set / verification set / test set, and the sample contains three types of mixed light features of goods images, including the alternating of natural light and artificial light in the evening, the mixed light area formed by local light failure at night, and the local dim scene formed by sudden light blocking in the daytime pipeline. The sample size is not less than 180, covering typical mixed scenes such as evening light transition and nighttime local light supplement, and covering main warehouse goods categories, which can effectively evaluate the recognition stability of the model in the scene crossing situation.

[0093] It needs to be further explained that the daytime scene exclusive test set, the nighttime scene exclusive test set and the mixed scene exclusive test set are evaluation sets collected independently from the training set, the verification set and the test set used in the model training stage, which contain the actual mixed light conditions that may occur in the evening and nighttime local light supplement; by using the real collected data for testing, the recognition stability of the model in the actual scene crossing situation can be more effectively evaluated, and the performance of the model in complex real environment can be tested.

[0094] All optimization scenes need to record the corresponding logs of the scene, error and optimization action, and the optimized model needs to retain compatibility with the original scene parameters. If the performance of a certain scene regresses after updating, the original scene parameters can be switched back with one key, ensuring the stability of the warehouse recognition process.

[0095] S8, the visual monitoring module is used to monitor the recognition result, performance index, model optimization log and abnormal alarm, and display them through a mobile terminal, so that the management personnel can control the warehouse goods image recognition state in real time.

[0096] As a preferred feasible embodiment, it needs to be explained that the visual monitoring module realizes the visual presentation of the whole process of warehouse goods recognition through the interactive interface and real-time data flow mechanism designed in layers, and the specific functions and implementation methods are as follows:

[0097] The interface layer adopts a three-level architecture: the overview page displays core indicators in the form of a dashboard, the left side refreshes real-time key data such as the daily recognition total, automatic recognition pass rate, and daytime / nighttime scene recognition accuracy rate comparison, and presents a 24-hour comprehensive recognition performance index fluctuation trend through a dynamic line chart, and an abnormal period is identified by a red flashing mark; the middle area displays the recognition error distribution of each area of the warehouse in a heat map, and the deeper the color, the higher the misjudgment rate of the area; the right side is provided with a real-time alarm column to display abnormal alarms, and each alarm is provided with a quick access to view details.

[0098] The detail page provides in-depth analysis for a single scene, for example, clicking the nighttime scene misjudgment rate increase information can jump to the nighttime scene dedicated detail page to display the category distribution and misjudgment type of the misjudgment sample, and the typical misjudgment image is displayed in a thumbnail grid; clicking the model optimization log button can view the optimization records for the nighttime scene in the past 7 days, support key operations according to scenes and optimization effects, and the traceability page realizes full-link tracking from the recognition result to the original data: the management personnel can click any recognition record to view the original collection image of the goods, the training sample after enhancement processing, the feature vector visualization result output by the model, and the probability distribution data of the classification branch; if the record triggers manual review, the correction opinion and review time consumption marked by the reviewer are also displayed.

[0099] Secondly, in the drawings of the disclosed embodiments, only structures related to the disclosed embodiments are involved, other structures can be referred to the general design, and in the case of no conflict, the same embodiment and different embodiments of the present application can be combined with each other;

[0100] Finally, the above-mentioned is only the preferred embodiment of the present application, and is not used to limit the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for recognizing warehouse goods images based on deep learning, characterized in that, The method includes an image acquisition module, a database, an image enhancement module, a deep learning feature extraction module, a cargo classification and recognition module, a recognition error analysis module, a comprehensive recognition performance evaluation module, a model dynamic optimization and update module, a visualization monitoring module, and a mobile terminal. The specific steps of the method are as follows: S1, the image acquisition module is used to acquire a dataset of images of target goods and similar goods in a warehousing scenario and store it in the database; S2, the image enhancement module is used to extract and process the image dataset of stored goods, simulate the difference in warehouse lighting during the day and night, and generate an enhanced image dataset containing daytime and nighttime lighting scenes; S3, the deep learning feature extraction module, is used to receive the enhanced image dataset, load the pre-trained ResNet101 model, train the model using the stochastic gradient descent algorithm, and output cargo-specific feature vectors. S4. The cargo classification and recognition module is used to receive cargo-specific feature vectors, construct a dual-branch classification structure, and set classification thresholds. S5, the identification error analysis module is used to compare the identification results of the cargo classification identification module with the actual results confirmed by manual review, and calculate the core error index; S6. The comprehensive recognition performance evaluation module calculates the comprehensive recognition performance index based on the index data from the recognition error analysis module using a weighted scoring method. S7, the model dynamic optimization and update module is used to trigger the optimization mechanism according to different scenarios based on the results of the comprehensive recognition performance evaluation module; The S8 visualization monitoring module is used to display recognition results, performance indicators, model optimization logs, and anomaly alarms through mobile terminals, enabling managers to monitor the image recognition status of warehouse goods in real time.

2. The method for recognizing warehouse cargo images based on deep learning according to claim 1, characterized in that: The warehouse cargo image dataset includes three main categories: first, static images of single cargo laid flat without obstruction; second, images of cargo stacked in 2 to 5 layers; and third, dynamic images of cargo on the assembly line.

3. The method for recognizing warehouse cargo images based on deep learning according to claim 1, characterized in that: The processing of the simulated daytime and nighttime warehouse lighting differences includes: increasing brightness by 10%-20% and contrast by 5%-15% in daytime scenes, while maintaining natural saturation; and reducing brightness by 20%-40%, decreasing contrast by 10%-20%, and lowering saturation by 5%-10% in nighttime scenes, with a slight color cast added to the nighttime image.

4. The method for recognizing warehouse cargo images based on deep learning according to claim 1, characterized in that: The steps for training the model using the stochastic gradient descent algorithm are as follows: a1. Data preparation and loading; a2. Model initialization and configuration; a3. Define the loss function and optimizer; a4. Model training iteration; a5. Model evaluation and storage.

5. The method for recognizing warehouse cargo images based on deep learning according to claim 4, characterized in that: The model initialization and configuration involves targeted adjustments to the model structure. Specifically, the last fully connected layer of the original ResNet101 model is removed, and a new fully connected layer with an output dimension of 256 is added. This output layer is the layer that generates the cargo-specific features.

6. The method for image recognition of warehouse goods based on deep learning according to claim 1, characterized in that: The cargo-specific feature vector is the output of the newly added 256-dimensional fully connected layer after the adjustment of the ResNet101 model. Each dimension in the vector corresponds to a unique visual feature of the cargo, and the Euclidean distance difference between the feature vectors of different categories of cargo is ≥0.8, while the similarity of the feature vectors of the same category of cargo is ≥0.

9.

7. The method for recognizing warehouse cargo images based on deep learning according to claim 1, characterized in that: The dual-branch classification structure is as follows: the first branch is the goods category classification branch, which adopts a combination architecture of fully connected layer + Softmax classifier. The first fully connected layer compresses the 256-dimensional feature vector to 128 dimensions, and the second layer compresses it to the dimension that matches the number of goods categories in the warehouse; the second branch is the goods packaging status classification branch, which adopts a support vector machine classifier to divide the feature vector into three states: intact packaging, damaged packaging, and blurred label, and outputs the confidence score.

8. The method for recognizing warehouse cargo images based on deep learning according to claim 1, characterized in that: The core error indicators include: category misjudgment rate, status misjudgment rate, missed identification rate, and verification dependency rate.

9. The method for recognizing warehouse cargo images based on deep learning according to claim 8, characterized in that: The formula for calculating the misclassification rate of the product category is as follows: Category misclassification rate = (Number of samples with incorrect category identification ÷ Total number of identified samples) × 100%, The number of samples with incorrect category identification refers to the number of samples where the category output by the model is inconsistent with the category confirmed by manual review. The total number of identified samples is the total number of all goods samples identified by the model within a certain statistical period.

Images