A Smart Decision-Making Method and System for Warehouse Goods Quality Based on Multimodal Data Fusion

By integrating multimodal data and making intelligent decisions, the problems of information isolation and rigid decision-making in the quality inspection of goods in automated warehousing systems have been solved. This has enabled quantitative assessment and intelligent grading of goods status, thereby improving system efficiency and reliability.

CN123070848APending Publication Date: 2026-11-10SHANDONG XINHUA HEALTH BUSINESS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611291842.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-25
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

In existing automated warehousing systems, the quality inspection of goods suffers from problems such as isolated information, rigid decision-making, lack of quantitative evaluation, and confusion from multiple barcode data sources, resulting in low efficiency and frequent human intervention.

Method used

By employing a multimodal data fusion method, a baseline digital twin model is constructed, integrating weight, visual, and barcode data. Confidence is calculated using linear weighted fusion, and combined with a multi-level decision-making mechanism, quantitative assessment and intelligent grading of product status are achieved.

Benefits of technology

It enables objective and accurate assessment of cargo quality, improves the efficiency of automated warehouse operations, reduces human intervention, and ensures unmanned operation of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123070848A_ABST
    Figure CN123070848A_ABST
Patent Text Reader

Abstract

This invention belongs to the fields of artificial intelligence and automated warehousing technology, and particularly relates to an intelligent decision-making method and system for warehouse goods quality based on multimodal data fusion. First, for intact SKU samples, weight distribution parameters are calibrated, visual feature templates are constructed, and nominal barcodes are entered to generate a baseline digital twin model. Then, three types of modal data—barcode, weight, and multi-view appearance images—are simultaneously collected through an automated inspection platform on the warehouse conveyor line. After multi-barcode arbitration verification and reconstruction of damaged barcodes, barcode matching scores, weight normality scores, and appearance similarity scores are calculated respectively. A linear weighted fusion is used to obtain the overall confidence level of the goods. Based on high and low confidence level thresholds, a three-level decision-making process is implemented: release, manual review, and interception alarm. This invention solves the problems of traditional warehousing inspection being limited to a single dimension, rigid decision-making, and difficulty in handling critical anomalies. It can adapt to different goods characteristics, improve the accuracy of goods quality assessment, reduce manual intervention, and ensure automated closed-loop operation of warehousing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and automated warehousing technology, and in particular relates to an intelligent decision-making method and system for the quality of warehouse goods based on multimodal data fusion. Background Technology

[0002] In automated warehousing systems, quality inspection of incoming and outgoing goods is a crucial step in ensuring warehouse safety and accurate delivery. Existing technologies typically employ single-dimensional, Boolean logic-based (i.e., "pass" or "fail") inspection methods. For example, the system independently determines whether a barcode is correct or whether the weight is within a preset threshold.

[0003] This approach has the following limitations: Information isolation, inability to make comprehensive judgments: The system cannot utilize the correlation between information from different dimensions. For example, a product may have a correct barcode, but a slight weight deviation and minor scratches on the outer packaging. Traditional independent judgment logic might determine it as "qualified" because each individual item is within tolerance, or simply and crudely determine it as "abnormal" because one item is unqualified, lacking the ability to comprehensively assess the overall condition of the product. Rigid decision-making, inability to handle critical states: Faced with critical situations such as "borderlines" or "gray areas," traditional methods lack flexibility and intelligence. It cannot distinguish between "severe damage" and "minor defects," nor can it assess the risk level based on different combinations of abnormalities, resulting in a large number of "suspected abnormal products" requiring manual verification, reducing the operational efficiency of the entire automated system. Lack of quantitative assessment methods: Existing technology can only provide qualitative "qualified / unqualified" conclusions, unable to provide a quantitative "health" or "confidence" indicator, which is not conducive to refined quality management and data analysis.

[0004] Furthermore, existing automated scanning systems, even those employing multiple scanners to improve read rates, generally suffer from a technical blind spot: they typically assume that only one valid barcode related to the current business exists on the cargo box. However, in actual warehousing and logistics, a cargo box often bears multiple barcodes of different types and meanings, such as: the GS1-128 logistics code representing the entire box, the EAN-13 product code representing individual items, internal traceability codes used by suppliers during the production process, or even shipping codes left over from upstream logistics links.

[0005] When a multi-scanner system reads multiple different barcodes simultaneously, traditional processing logic can lead to data confusion, making it impossible to automatically determine which barcode is the primary barcode required for the current operation (such as inbound or outbound). This ambiguity usually causes system workflow interruptions, requiring manual intervention for identification and selection, thus breaking the closed loop of full automation and reducing the overall efficiency and reliability of the system.

[0006] Furthermore, in actual warehousing and distribution processes, due to handling, friction, or environmental factors, barcodes on product boxes often suffer from small-area printing defects, wear, stains, or glare. Traditional single-image-based decoding algorithms experience a sharp drop in decoding success rate when faced with these incomplete or damaged barcode information. This "code reading failure" is one of the most common reasons for automated production line interruptions requiring manual intervention, severely impacting the overall processing efficiency and automation level of the system. Simply improving hardware performance cannot fundamentally solve this problem; a new algorithm that can proactively "repair" damaged information and improve decoding robustness is urgently needed.

[0007] Therefore, how to integrate multi-dimensional detection data to make more accurate and intelligent quantitative assessments and decisions on the status of goods is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0008] To address the technical problems existing in the background art described above, this invention proposes an intelligent decision-making method and system for warehouse goods quality based on multimodal data fusion.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0010] S1. For intact product samples of the target new product SKU, respectively, the weight probability distribution parameters are calibrated, visual feature templates are constructed, nominal barcode data is entered, and a benchmark digital twin model is generated.

[0011] S2. Through an automated inspection platform integrated into the warehouse conveyor line, real-time barcode datasets, real-time weight data, and multi-view real-time appearance images of the goods to be inspected are collected simultaneously.

[0012] S3. Quantize the three modal data of barcode, weight and appearance respectively, and calculate the barcode matching score, weight normality score and appearance similarity score accordingly.

[0013] S4. Using linear weighted fusion, weight coefficients are assigned to the barcode matching score, weight normality score, and appearance similarity score, and the weighted sum is obtained to obtain the confidence level of the goods to be inspected.

[0014] S5. Compare the confidence level with the preset high confidence threshold and low confidence threshold, and generate and execute control commands for high confidence to allow passage, medium confidence to transfer to manual review, and low confidence to block alarms based on the comparison results.

[0015] As a preferred embodiment, step S1 is specifically implemented as follows:

[0016] First, calibrate the weight probability distribution parameters. Then, draw N samples (N ≥ 30) from the same batch of intact goods and collect the weight data of each sample using a calibrated high-precision weighing device. Assuming the weights of the goods follow a normal distribution, calculate the mean of the sample weights. with standard deviation The mean and standard deviation are used as the weight probability distribution parameters for this SKU. Store the baseline digital twin model;

[0017] Construct visual feature templates, select gold samples, and collect standard images of them from multiple perspectives. And perform size and pixel value normalization preprocessing;

[0018] A hierarchical collaborative convolution-attention hybrid model is used to extract image features. The hierarchical collaborative hybrid model includes a convolutional backbone, a tokenization layer and a transformer encoder connected in sequence.

[0019] The convolutional backbone uses a pre-trained convolutional neural network to extract local semantic features from image I and output a two-dimensional semantically rich feature map. , ,in, The convolutional backbone is defined by H and W, where H and W are the dimensions of image I, and C is the number of channels.

[0020] The tokenization layer segments the feature map X in spatial dimensions into There are 3 non-overlapping small blocks, each with a size of 1. Each small block is flattened and mapped to a token through a linear projection layer, thereby transforming the feature map X into a token sequence. ,in, It is a learnable [CLS] token used for final classification;

[0021] The transformer encoder consists of L identical transformer layers stacked together. Each transformer contains a multi-head self-attention (MSA) module and a feedforward network (FNN). The classification tokens output by the transformer encoder are extracted as the baseline feature vector for the corresponding viewpoint. For the , layer:

[0022] , ,in, Representing the normalization layer, the multi-head self-attention mechanism (MSA) models the global relationships between tokens by calculating the query, key, and value matrices.

[0023] All standard images are input into the hierarchical co-mixing model, and the transformer encoder outputs... [CLS] token As the final high-dimensional feature vector of the image; average the multiple [CLS] tokens for each viewpoint to obtain k baseline feature vectors. This set is the visual feature template;

[0024] By scanning the barcode of a gold sample or directly importing it from electronic data provided by an upstream supplier, the accurate barcode string is stored in the model as standard barcode information. The weight probability distribution parameters, visual feature templates, and nominal barcode data are then combined to generate a benchmark digital twin model.

[0025] Preferably, before the quantization process in step S3, multiple barcode arbitration and verification steps are included, specifically including:

[0026] When a set of barcodes is collected from the same product within a preset time window, the syntax rules are first checked through a preset barcode rule base, and barcodes that do not conform to the encoding system, data length and format rules are discarded to obtain a set of candidate barcodes.

[0027] Based on the business type of the current operation, associate the corresponding business document data, perform context association verification on the candidate barcode set, retain the barcodes that have matching records in the business documents, and obtain the final candidate set;

[0028] The barcodes in the final candidate set are sorted according to the preset barcode priority queue, and the barcode with the highest priority is selected as the main business barcode.

[0029] If the candidate set is empty after any verification step, or if there are multiple barcodes with the same highest priority, the arbitration is deemed to have failed, the corresponding goods are sorted to the abnormal inspection area, and an alarm message is sent.

[0030] As a preferred method, the system uses a unique primary business barcode for barcode matching and score calculation, specifically including:

[0031] The system first performs preliminary decoding on the acquired barcode image. If the decoder directly outputs a single, clear barcode... If the decoder outputs a set of barcodes, the process continues; This triggers a multi-barcode arbitration and verification sub-process to determine the unique primary business barcode. As ;

[0032] If the decoder fails to decode successfully or the output decoding confidence is lower than the preset threshold, it is determined to be a decoding failure, and the damaged barcode reconstruction process is started, which sequentially executes multi-view image fusion and virtual flattening, context-based predictive image repair, and hypothesis-driven decoding and verification.

[0033] If a unique and valid barcode is obtained after the reconstruction process, the barcode will be used as the business barcode to be compared for barcode matching score calculation; if the reconstruction process fails, the current goods inspection process will be terminated and exception handling will be performed.

[0034] The finalized valid barcode Compared with the benchmark digital twin model Perform a comparison, if = Then barcode matching score It is 1 if it is true, otherwise it is 0.

[0035] Preferably, the specific implementation of the weight normal distribution in step S3 includes:

[0036] First, calculate the real-time weight. Z-score standardized value relative to the baseline normal distribution ;

[0037] Map the z-value to the score using the exponential decay function. This reflects the degree of deviation in weight: ,in, The adjustable system sensitivity is used to control the rate at which the score decreases with weight deviation.

[0038] Preferably, the appearance similarity score in step S3 is obtained based on the feature space distance of the hierarchical collaborative hybrid model, specifically as follows:

[0039] Real-time captured images of the outer packaging The input is fed into the hierarchical collaborative fusion model, and the [CLS] token of the final output of the model is extracted as the feature vector of the real-time image: ,in, It is a hierarchical collaborative hybrid model;

[0040] Real-time feature vectors The baseline feature vectors from the corresponding perspectives stored in the digital twin model The comparison is performed using cosine similarity as the metric function, and then normalized to obtain the final appearance similarity score. .

[0041] As a preferred option, step S4 employs linear weighted fusion, assigning weight coefficients to the barcode matching score, weight normality score, and appearance similarity score, and then summing them by weight to obtain the confidence level of the product to be inspected.

[0042] By fusing multimodal individual scores, a linear weighted model is used to integrate three independent individual scores into a single, comprehensive confidence index. ,in, The weighting coefficients for the three modalities—barcode, weight, and appearance—satisfy the constraints. The weighting coefficient is a configurable parameter, and the weighting allocation can be adjusted according to the value level, fragility characteristics and business inspection requirements of the goods.

[0043] Preferably, the hierarchical decision-making in step S5 specifically includes:

[0044] When the confidence level is greater than or equal to the high confidence threshold, the goods are determined to be in good condition, a release control instruction is generated, and the normal warehousing or outbound process is executed.

[0045] When the confidence level is greater than or equal to the low confidence threshold and less than the high confidence threshold, it is judged as a suspected anomaly, a review control instruction is generated, and the transfer equipment is dispatched to transport the goods to the manual review station or the high-precision automatic re-inspection station.

[0046] When the confidence level is less than the low confidence level threshold, it is determined to be an anomaly, an interception alarm control command is generated, the goods are sorted to the isolation area and an alarm message containing all detection data and appearance images is pushed.

[0047] A warehouse goods quality intelligent decision-making system based on multimodal data fusion, characterized in that it includes:

[0048] The benchmark modeling module is used to establish a benchmark digital twin model of goods. For intact goods samples of the target SKU, it calibrates the weight probability distribution parameters, constructs visual feature templates, enters nominal barcode data, and generates and stores the benchmark digital twin model.

[0049] The data acquisition module interfaces with the automated inspection platform integrated into the warehouse conveyor line to synchronously collect real-time barcode datasets, real-time weight data, and multi-view real-time appearance images of the goods to be inspected.

[0050] The single-modal quantization module is used to quantize the three modal data of barcode, weight and appearance respectively, and calculate the barcode matching score, weight normality score and appearance similarity score respectively; the fusion calculation module is used to use a linear weighted fusion model to weight and sum the scores of the three categories to obtain the confidence level of the goods to be inspected.

[0051] The hierarchical decision-making module is used to compare the confidence level with preset high confidence thresholds and low confidence thresholds, and generate and execute corresponding hierarchical control instructions.

[0052] Preferably, the automated testing platform includes a transmission and positioning unit, a dynamic weighing unit, a multi-view visual acquisition array, an omnidirectional barcode reading system, and an edge computing unit.

[0053] The conveying and positioning unit is used to control the operation of the conveyor belt, and to detect the position of the goods and trigger the acquisition signal through photoelectric sensors;

[0054] The dynamic weighing unit is used to collect weight data in real time during the uniform speed passage of goods;

[0055] The multi-view visual acquisition array includes multiple industrial cameras installed at different angles and a synchronous strobe lighting system, used to acquire multi-view appearance images of goods.

[0056] The omnidirectional barcode reading system includes multiple fixed industrial barcode scanners at different angles, forming a scanning tunnel to collect barcode data from the surface of the goods from all angles;

[0057] The edge computing unit is used to receive various types of sensor data and run the algorithm logic of each module.

[0058] Compared with existing technologies, the advantages and positive effects of this invention are as follows: This invention upgrades traditional binary qualitative product quality inspection to a continuous quantitative confidence score based on probability and feature space distance, achieving objective, accurate, and traceable assessment of product quality. This invention employs a weight probability distribution statistical model combined with a convolutional-attention hybrid visual model, exhibiting good robustness to reasonable production tolerances for products. With configurable weights and thresholds, it can adapt to various product inspection scenarios. A three-level decision-making mechanism enables risk-level management, accurately focusing on suspected abnormal gray-area products, abandoning a one-size-fits-all approach, and improving the efficiency of automated warehouse operations. The convolutional-attention hybrid model balances detection performance and computational efficiency, meeting the high-throughput real-time detection requirements of industry. Simultaneously, a three-level barcode arbitration mechanism (syntax, context, and priority) solves the problem of confusion between multiple barcode data sources, and a damaged barcode reconstruction strategy enhances decoding robustness, reduces manual intervention, and ensures end-to-end closed-loop unmanned operation of warehouse operations. The rule base and priority queue are flexibly configurable, expanding the scenario adaptability of the solution. Attached Figure Description

[0059] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a schematic diagram illustrating the implementation process of an intelligent decision-making method for warehouse goods quality based on multimodal data fusion. Detailed Implementation

[0061] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0062] Numerous specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways than those described herein, and therefore the invention is not limited to the specific embodiments disclosed in the following specification.

[0063] In this embodiment, the present invention aims to overcome the shortcomings of existing technologies, such as single detection dimensions, rigid decision logic, and inability to handle critical states. It provides a method and system for intelligent decision-making on the quality of warehouse goods based on multimodal data fusion, enabling quantitative assessment and intelligent grading of goods integrity. To achieve the above objective, the present invention provides an intelligent decision-making method for the quality of warehouse goods based on multimodal data fusion. This method is executed by a computer program deployed on a server or edge computing device. Specific implementation steps are as follows: Figure 1 As shown.

[0064] Step 1: For intact product samples of the target new product SKU, respectively, the weight probability distribution parameters are calibrated, visual feature templates are constructed, nominal barcode data is entered, and a benchmark digital twin model is generated.

[0065] Specifically, this step is an offline modeling process performed when a new SKU is first introduced into the warehouse or during periodic calibration. Its purpose is to provide a high-precision, highly robust "gold standard" for subsequent online real-time inspection. This process can be broken down into three parallel sub-steps: calibrating weight probability distribution parameters, constructing visual feature templates, and entering nominal barcode data.

[0066] To overcome the slight weight variations in individual items caused by factors such as packaging and content density, we do not use a single nominal weight, but instead establish a statistical model. From the same batch of confirmed intact goods, we randomly select N samples (in industrial statistics, N ≥ 30 is typically required to ensure statistical significance). Then, we perform high-precision weighing using a calibrated static electronic scale with an accuracy of at least 0.1 grams, weighing each of these N samples individually and recording the weight of each sample. Assume the sample weights follow a normal distribution. The sample mean is calculated using the following formula. and sample standard deviation : , The calculated and As a parameter of the weight probability distribution of this SKU Store in digital twin model middle.

[0067] Next, visual feature templates are constructed based on a convolution-attention hybrid model, the core of which is a hierarchical collaborative convolution-attention hybrid model. The model consists of a convolutional backbone, a tokenization layer, and a transformer encoder connected in series. Standard sample preparation and preprocessing are performed, and high-quality samples are selected by acquiring a multi-view standard image set. And normalize the size and pixel values.

[0068] Hierarchical collaborative hybrid model Definition: a) Convolutional backbone A lightweight, pre-trained convolutional neural network is used as the feature extraction front-end. In some embodiments of the invention, the first three stages (Stage 1-3) of ResNet-34 are employed. The input to this backbone is an image I (size... The output is a two-dimensional semantically rich feature map. , Where C is the number of channels. b) Tokenization layer, which divides the feature map X into spatial dimensions. Non-overlapping small blocks: The size of each small piece is Each small block is flattened and mapped to a D-dimensional token through a linear projection layer, thereby transforming the feature map X into a token sequence. ,in, It is a learnable [CLS] token used for final classification. Transformer Encoder: This is the core of the model, consisting of L identical transformer layers stacked together. Each transformer layer contains a multi-head self-attention module (MSA) and a feedforward network (FFN). For the Lth... layer: , ,in, Representing the normalization layer, the Multi-Head Self-Attention (MSA) mechanism models the global relationships between tokens by computing query, key, and value matrices; the generation and storage of baseline feature vectors: all standard images { Input into the hybrid model In the middle. Take the encoder output of the converter. [CLS] token This serves as the final high-dimensional feature vector for the image. The average of multiple [CLS] tokens for each viewpoint is used to obtain k baseline feature vectors. This set is the visual feature template. .

[0069] The nominal barcode data ( The data entry step involves scanning the barcode of the gold sample or directly importing it from electronic materials provided by the upstream supplier, using the accurate barcode string as the data entry point. Store the data in the model. At this point, a complete benchmark digital twin model, including an accurate statistically weighted model and robust depth vision features, is complete. Build complete.

[0070] Step 2 involves simultaneously collecting real-time barcode datasets, real-time weight data, and multi-view real-time appearance images of the goods to be inspected through an automated inspection platform integrated into the warehouse conveyor system. Specifically, this step describes the online operation process, outlining the physical structure and workflow of the automated inspection platform to ensure efficient and accurate data collection corresponding to the baseline model in Step 1.

[0071] The automated inspection platform consists of an integrated physical workstation installed on the conveyor belt at the warehouse entrance or exit. Its core components include: a conveyor and positioning unit: a conveyor belt whose speed is precisely controlled by a PLC (Programmable Logic Controller). In the inspection area, photoelectric sensors or laser beam sensors are installed to accurately detect whether goods have reached the predetermined inspection position and trigger the acquisition signal. A dynamic weighing unit: an embedded in-motion weighing module integrated with the conveyor belt, capable of instantly completing high-precision weight measurement while goods pass through at a constant speed without stopping. Multi-view visual acquisition array: Top camera: A high-resolution industrial area array camera mounted directly above the goods to capture top surface images. Side cameras: Two or four industrial cameras symmetrically mounted on both sides of the conveyor belt to ensure simultaneous capture of images from all sides of the goods. Synchronous lighting system: A strobe LED light source linked to the camera array provides high-intensity, uniform illumination at the moment of camera exposure, eliminating ambient light interference and avoiding motion blur. Omnidirectional barcode reading system: Multiple (usually 3-5) industrial-grade fixed barcode scanners installed at different angles form a "scanning tunnel," ensuring that no matter which surface the barcode is affixed to, it can be successfully read by at least one scanner. Edge computing unit: An industrial PC (IPC) or edge server equipped with a high-performance GPU, responsible for receiving all sensor data and running the complex algorithms in step 3 in real time.

[0072] When a product to be inspected When goods enter the testing platform, the PLC will precisely schedule the process according to the following timing sequence: Entry and Trigger: Goods Upon reaching the conveyor belt detection area, its front end blocks the photoelectric sensor, triggering a "start acquisition" signal to the edge computing unit. Data frame synchronous capture: At the instant the trigger signal is received (millisecond-level response), the system synchronously executes: the dynamic weighing module locks and outputs the current weight reading. The visual acquisition array's cameras and lighting system synchronously trigger a flash and exposure, capturing a set of real-time images encompassing all viewpoints. All scanners in the omnidirectional barcode reading system are activated simultaneously, and the first successfully decoded barcode data is displayed. As a final result. Data transmission and queue processing: The complete data packets collected ( , , The goods are immediately timestamped and assigned a unique ID, then sent to the processing queue of the edge computing unit. Physical release: Simultaneously, the goods... Having passed the inspection area, the item continues moving downstream on the conveyor belt, making way for the next item to be inspected. This "on-the-fly" and pipeline-style processing method allows for parallel physical transport efficiency and data computation, significantly increasing the overall system throughput (UPH - Units Per Hour). The high-quality, multimodal data collected lays a solid foundation for accurate quantitative scoring in the subsequent step 3.

[0073] Step 3 involves quantifying the three modal data types—barcode, weight, and appearance—to calculate the corresponding barcode matching score, weight normality score, and appearance similarity score. This step aims to convert the raw data with different dimensions into a unified, normalized score that falls between these categories.

[0074] Before proceeding to step 3, multi-barcode arbitration and verification processing is required. This occurs when the omnidirectional barcode reading system in step 2 collects a set of barcodes from the same product within a preset time window. When (n>1), the system does not process immediately, but instead initiates the following multi-level arbitration and verification process to determine a unique, context-sensitive master business barcode. .

[0075] Level 1: Syntactic Validation. Specifically, the system has a built-in configurable "barcode rule library," which defines the syntactic rules that valid barcodes should follow in different business scenarios. Rules can include: Encoding system (Symbology): such as Code128, QRCode, DataMatrix, etc.; Data length (Length): such as a fixed length of 13 digits, or a variable length of 20-30 digits; Data format (FormatPattern): defined using regular expressions, for example, specifying that batch number barcodes must begin with "L" followed by 10 digits; product codes must be purely numeric. The system iterates through the barcode set. Each barcode The system uses a rule base for matching. Any barcode that does not conform to the preset syntax rules will be discarded, forming a candidate set that has passed the syntax check. .

[0076] Level 2: ContextualValidation. The system retrieves relevant business document data from the WMS system based on the current job type (e.g., "Inbound Process" or "Outbound Process"). For inbound, it associates the "Expected Arrival Notification (ASN)". For outbound, it associates the "Picking List". The system iterates through the candidate set. Each barcode The system then queries and matches the barcodes in the associated business document database. Only barcodes that can be matched in the current business document are considered "context-valid" and added to the final candidate set. For example, during the warehousing process, only goods with barcode information on their ASN are considered valid.

[0077] Level 3: Priority Queue Arbitration. In reality, multiple barcodes may pass the first two levels of verification (for example, a box may have both a full box code and individual item codes, and both types of items are on the ASN). In this case, the system initiates priority arbitration. The system has a built-in "barcode priority queue," which presets priorities for different types of barcodes based on business logic. For example, a typical pharmaceutical warehousing priority queue might be configured as follows: P1 (highest): Logistics Unit / Pallet Code (SSCC), P2: Full Box / Outer Box Code (GS1-128CaseCode), P3: Inner Packaging / Middle Packaging Code, P4 (lowest): Individual Sales Unit Code (EAN / UPC). The system will then finalize the candidate set. All barcodes are sorted according to this priority queue. The barcode with the highest priority is selected as the final, unique primary business barcode. .

[0078] Level 4: Anomaly Handling Process. If, after any level of arbitration, the candidate set is empty (i.e., no barcode is valid), or after level 3 arbitration, multiple barcodes with the same highest priority exist, the system will determine this scan as "arbitration failed." In this case, the system will automatically execute anomaly handling: physically sort the item to the "anomaly waiting area" and send an alarm to the administrator. The alarm information should include the set of all barcodes collected in this scan. And for any failed steps, we await human intervention.

[0079] Only after completing the arbitration and verification process for multiple barcodes does the system use this uniquely determined one. Proceed to the next steps.

[0080] The first step involves product identification and barcode matching calculation. Specifically, this includes initial decoding attempts and multi-barcode arbitration. The system first performs preliminary decoding on the acquired barcode images. If the decoder directly outputs a single, clear barcode... The process continues. If the decoder outputs a set of barcodes... If (n>1), then the "Multi-barcode Arbitration and Verification" sub-process is triggered to determine the unique primary business barcode. As .

[0081] If the decoder fails to decode successfully or the output decoding confidence level is lower than a preset threshold, it is judged as "decoding failure" and the next step of damaged barcode reconstruction and verification is triggered. The system starts this step. Level 1: Multi-view image fusion and virtual flattening; Level 2: Context-based predictive image inpainting; Level 3: Hypothesis-driven decoding and verification; Level 4: Anomaly handling. If this step successfully verifies a unique barcode, it is used as... If the process fails, the entire product inspection process is halted, and an exception is handled. The final barcode matching score is calculated, and the finally determined valid barcode is... In the digital twin model Compare the data. Barcode matching is performed. =1 (if == 0 or 0 (if not equal). This score reflects the correctness of the item's identity.

[0082] Calculate real-time weight Z-score standardized value relative to the baseline normal distribution Use the exponential decay function to map the z-value to the score. This reflects the degree of deviation in weight: ,in, To provide adjustable system sensitivity, the system controls the rate at which the score decreases with weight deviation. The appearance similarity score is obtained based on the feature space distance of a hierarchical collaborative hybrid model, specifically by using real-time acquired images of the outer packaging. The input is fed into the hierarchical collaborative fusion model, and the [CLS] token of the final output of the model is extracted as the feature vector of the real-time image: ,in, For a hierarchical collaborative hybrid model; real-time feature vectors The baseline feature vectors from the corresponding perspectives stored in the digital twin model The comparison is performed using cosine similarity as the metric function, and then normalized to obtain the final appearance similarity score. .

[0083] Step 4 involves using linear weighted fusion to assign weight coefficients to the barcode matching score, weight normality score, and appearance similarity score, and then summing them to obtain the confidence level of the product to be inspected. Specifically, this involves fusing the multimodal individual scores using a linear weighted model to integrate the three independent individual scores into a single, comprehensive confidence level index. ,in, The weighting coefficients for the three modalities—barcode, weight, and appearance—satisfy the constraints. These weighting coefficients are configurable parameters; for example, they can be appropriately increased for high-value, fragile medicines. The value of .

[0084] Finally, a tiered decision-making process is performed based on the confidence level; the final confidence score C is compared with two preset, dynamically adjustable decision thresholds (high confidence threshold and high confidence threshold). and low confidence threshold The comparison is performed, and the corresponding automated control instructions are executed: High confidence level: if C is greater than or equal to (For example, 0.95) If the system determines the goods are in good condition, it generates a "release" control command, which is then handed over to subsequent automated equipment to execute the normal inbound or outbound process. Medium confidence level verification: If C is greater than or equal to... (e.g., 0.75) and less than The system determines the item as "suspected anomaly" and generates a "transfer to review area" control command, dispatching the AGV to transport the item to a designated manual review station or a higher-precision automated review station. Low-confidence interception: If C is less than... The system determines that the problem is "confirmed as abnormal", generates a control command to "intercept and alarm", sorts the goods to the isolation area, and immediately pushes an alarm message containing all detection data and images to the administrator through the WMS system.

[0085] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments for application in other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A smart decision-making method for warehouse goods quality based on multimodal data fusion, characterized in that, Includes the following steps: S1. For intact product samples of the target new product SKU, respectively, the weight probability distribution parameters are calibrated, visual feature templates are constructed, nominal barcode data is entered, and a benchmark digital twin model is generated. S2. Through an automated inspection platform integrated into the warehouse conveyor line, real-time barcode datasets, real-time weight data, and multi-view real-time appearance images of the goods to be inspected are collected simultaneously. S3. Quantize the three modal data of barcode, weight and appearance respectively, and calculate the barcode matching score, weight normality score and appearance similarity score accordingly. S4. Linear weighted fusion is used to assign weight coefficients to the barcode matching score, weight normality score, and appearance similarity score, and the weighted sum is obtained to obtain the confidence level of the goods to be inspected. S5. Compare the confidence level with the preset high confidence threshold and low confidence threshold, and generate and execute control commands for high confidence to allow passage, medium confidence to transfer to manual review, and low confidence to block alarms based on the comparison results.

2. The intelligent decision-making method for warehouse goods quality based on multimodal data fusion according to claim 1, characterized in that, The specific implementation of step S1 is as follows: First, calibrate the weight probability distribution parameters. Then, draw N samples (N ≥ 30) from the same batch of intact goods and collect the weight data of each sample using a calibrated high-precision weighing device. Assuming the weights of the goods follow a normal distribution, calculate the mean of the sample weights. with standard deviation The mean and standard deviation are used as the weight probability distribution parameters for this SKU. Store the baseline digital twin model; Construct visual feature templates, select gold samples, and collect standard images of them from multiple perspectives. And perform size and pixel value normalization preprocessing; A hierarchical collaborative convolution-attention hybrid model is used to extract image features. The hierarchical collaborative hybrid model includes a convolutional backbone, a tokenization layer and a transformer encoder connected in sequence. The convolutional backbone uses a pre-trained convolutional neural network to extract local semantic features from image I and output a two-dimensional semantically rich feature map. , ,in, The convolutional backbone is defined by H and W, where H and W are the dimensions of image I, and C is the number of channels. The tokenization layer segments the feature map X in spatial dimensions into There are 3 non-overlapping small blocks, each with a size of 1. Each small block is flattened and mapped to a token through a linear projection layer, thereby transforming the feature map X into a token sequence. ,in, It is a learnable [CLS] token used for final classification; The transformer encoder consists of L identical transformer layers stacked together. Each transformer contains a multi-head self-attention (MSA) module and a feedforward network (FNN). The classification tokens output by the transformer encoder are extracted as the baseline feature vector for the corresponding viewpoint. For the , layer: , ,in, Representing the normalization layer, the multi-head self-attention mechanism (MSA) models the global relationships between tokens by calculating the query, key, and value matrices. All standard images are input into the hierarchical co-mixing model, and the transformer encoder outputs... [CLS] token As the final high-dimensional feature vector of the image; average the multiple [CLS] tokens for each viewpoint to obtain k baseline feature vectors. This set is the visual feature template; By scanning the barcode of a gold sample or directly importing it from electronic data provided by an upstream supplier, the accurate barcode string is stored in the model as standard barcode information. The weight probability distribution parameters, visual feature templates, and nominal barcode data are then combined to generate a benchmark digital twin model.

3. The intelligent decision-making method for warehouse goods quality based on multimodal data fusion according to claim 1, characterized in that, Before the quantization process in step S3, multiple barcode arbitration and verification steps are included, specifically: When a set of barcodes is collected from the same product within a preset time window, the syntax rules are first checked through a preset barcode rule base, and barcodes that do not conform to the encoding system, data length and format rules are discarded to obtain a set of candidate barcodes. Based on the business type of the current operation, associate the corresponding business document data, perform context association verification on the candidate barcode set, retain the barcodes that have matching records in the business documents, and obtain the final candidate set; The barcodes in the final candidate set are sorted according to the preset barcode priority queue, and the barcode with the highest priority is selected as the main business barcode. If the candidate set is empty after any verification step, or if there are multiple barcodes with the same highest priority, the arbitration is deemed to have failed, the corresponding goods are sorted to the abnormal inspection area, and an alarm message is sent.

4. The intelligent decision-making method for warehouse goods quality based on multimodal data fusion according to claim 3, characterized in that, The system uses a unique primary business barcode for barcode matching and score calculation, specifically including: The system first performs preliminary decoding on the acquired barcode image. If the decoder directly outputs a single, clear barcode... If the decoder outputs a set of barcodes, the process continues; This triggers a multi-barcode arbitration and verification sub-process to determine the unique primary business barcode. As ; If the decoder fails to decode successfully or the output decoding confidence is lower than the preset threshold, it is determined to be a decoding failure, and the damaged barcode reconstruction process is started, which sequentially executes multi-view image fusion and virtual flattening, context-based predictive image repair, and hypothesis-driven decoding and verification. If a unique and valid barcode is obtained after the reconstruction process, the barcode will be used as the business barcode to be compared for barcode matching score calculation; if the reconstruction process fails, the current goods inspection process will be terminated and exception handling will be performed. The finalized valid barcode Compared with the benchmark digital twin model Perform a comparison, if = Then barcode matching score It is 1 if it is true, otherwise it is 0.

5. The intelligent decision-making method for warehouse goods quality based on multimodal data fusion according to claim 1, characterized in that, The specific implementation of the weight normality in step S3 includes: First, calculate the real-time weight. Z-score standardized value relative to the baseline normal distribution ; Map the z-value to the score using the exponential decay function. This reflects the degree of deviation in weight: ,in, The adjustable system sensitivity is used to control the rate at which the score decreases with weight deviation.

6. The intelligent decision-making method for warehouse goods quality based on multimodal data fusion according to claim 1, characterized in that, In step S3, the appearance similarity score is obtained based on the feature space distance of the hierarchical collaborative hybrid model, specifically as follows: Real-time captured images of the outer packaging The input is fed into the hierarchical collaborative fusion model, and the [CLS] token of the final output of the model is extracted as the feature vector of the real-time image: ,in, It is a hierarchical collaborative hybrid model; Real-time feature vectors The baseline feature vectors corresponding to the viewpoints stored in the digital twin model The comparison is performed using cosine similarity as the metric function, and then normalized to obtain the final appearance similarity score. .

7. The intelligent decision-making method for warehouse goods quality based on multimodal data fusion according to claim 1, characterized in that, Step S4 employs linear weighted fusion, assigning weight coefficients to the barcode matching score, weight normality score, and appearance similarity score, and then summing them by weight to obtain the confidence level of the product to be inspected. The specific implementation of this method is as follows: By fusing multimodal individual scores, a linear weighted model is used to integrate three independent individual scores into a single, comprehensive confidence index. ,in, The weighting coefficients for the three modalities—barcode, weight, and appearance—satisfy the constraints. The weighting coefficient is a configurable parameter, and the weighting allocation can be adjusted according to the value level, fragility characteristics and business inspection requirements of the goods.

8. The intelligent decision-making method for warehouse goods quality based on multimodal data fusion according to claim 1, characterized in that, The hierarchical decision-making in step S5 specifically includes: When the confidence level is greater than or equal to the high confidence threshold, the goods are determined to be in good condition, a release control instruction is generated, and the normal warehousing or outbound process is executed. When the confidence level is greater than or equal to the low confidence threshold and less than the high confidence threshold, it is judged as a suspected anomaly, a review control instruction is generated, and the transfer equipment is dispatched to transport the goods to the manual review station or the high-precision automatic re-inspection station. When the confidence level is less than the low confidence level threshold, it is determined to be an anomaly, an interception alarm control command is generated, the goods are sorted to the isolation area and an alarm message containing all detection data and appearance images is pushed.

9. A smart decision-making system for warehouse goods quality based on multimodal data fusion, characterized in that, include: The benchmark modeling module is used to establish a benchmark digital twin model of goods. For intact goods samples of the target SKU, it calibrates the weight probability distribution parameters, constructs visual feature templates, enters nominal barcode data, and generates and stores the benchmark digital twin model. The data acquisition module interfaces with the automated inspection platform integrated into the warehouse conveyor line to synchronously collect real-time barcode datasets, real-time weight data, and multi-view real-time appearance images of the goods to be inspected. The single-modal quantization module is used to quantize the three modal data of barcode, weight and appearance respectively, and calculate the barcode matching score, weight normality score and appearance similarity score accordingly. The fusion calculation module is used to perform a weighted summation of the three categories of individual scores using a linear weighted fusion model to obtain the confidence level of the product to be tested. The hierarchical decision-making module is used to compare the confidence level with preset high confidence thresholds and low confidence thresholds, and generate and execute corresponding hierarchical control instructions.

10. The intelligent decision-making system for warehouse goods quality based on multimodal data fusion according to claim 9, characterized in that, The automated detection platform includes a transmission and positioning unit, a dynamic weighing unit, a multi-view vision acquisition array, an omnidirectional barcode reading system, and an edge computing unit. The conveying and positioning unit is used to control the operation of the conveyor belt, and to detect the position of the goods and trigger the acquisition signal through photoelectric sensors; The dynamic weighing unit is used to collect weight data in real time during the uniform speed passage of goods; The multi-view visual acquisition array includes multiple industrial cameras installed at different angles and a synchronous strobe lighting system, used to acquire multi-view appearance images of goods. The omnidirectional barcode reading system includes multiple fixed industrial barcode scanners at different angles, forming a scanning tunnel to collect barcode data from the surface of the goods from all angles; The edge computing unit is used to receive various types of sensor data and run the algorithm logic of each module.