Intelligent liquidation receipt management method based on multi-modal data fusion

Abstract

The invention discloses an intelligent management method for liquidation receipts based on multi-modal data fusion, and particularly relates to the field of data analysis. Comprising the steps of S1, multispectral data acquisition in a limited illumination environment, S2, cross-modal feature decoupling and recombination, S3, space-time heterograph neural network analysis, S4, multi-scale attention decision fusion, S5, resistance enhancement verification, and S6, incremental management based on knowledge distillation. According to the method, the physical anti-counterfeiting capability is remarkably improved, the paper material, the ink components and the surface structure are deeply analyzed through a multispectral sequence acquisition mechanism, and hidden tampering behaviors such as color fading and chemical altering of the thermo-sensitive paper are accurately identified. Cross-modal deep correlation analysis is achieved in a breakthrough mode, a physical-semantic decoupling technology and a space-time heterogeneous graph network are adopted for modeling, and non-dominant laws such as commodity position offset and tax rate anomaly are effectively captured.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and more specifically, to a method for intelligent management of clearing receipts based on multimodal data fusion. Background Art

[0002] With the advancement of electronic payments and financial digitization, intelligent management of clearing receipts has become a core component of corporate financial systems. This area primarily implements automated accounting processing through data collection, content recognition, and anomaly detection on paper or electronic receipts. Existing technologies typically use standard scanning devices to capture receipt images, extract text information through optical character recognition (OCR), and finally verify the data using a rules engine or simple machine learning model. The entire process relies on visible light imaging and single-modal data processing, forming a linear "collection-recognition-verification" processing chain.

[0003] The current mainstream solutions have three key flaws: First, they rely solely on ordinary optical imaging during the data collection phase, and are unable to obtain physical property information such as paper material and ink characteristics, making it difficult to effectively identify physical tampering behaviors such as alteration and forgery; second, during the data processing process, each modal information is analyzed in isolation, and features such as text, images, and numbers lack deep correlation, making it impossible to capture implicit abnormal patterns across modalities; finally, the system lacks the ability to adapt in adversarial environments, and requires frequent retraining to face new attack methods. In addition, the incremental learning efficiency is low, making it difficult to adapt to rapidly changing business scenarios. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides an intelligent management method for clearing receipts based on multimodal data fusion, which solves the problems raised in the above-mentioned background technology through the following scheme.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for intelligent management of liquidation receipts based on multimodal data fusion, comprising: S1: Multispectral data acquisition in restricted light environments: Construct a special physical acquisition environment to obtain high-resolution multimodal data at the source of the data, including environment configuration, synchronous acquisition, and data preprocessing; S2: Cross-modal feature decoupling and reorganization: Extract features from multimodal data and separate physical and semantic attributes, including physical attribute decoupling, semantic attribute decoupling, and reorganization of fusion tensors; S3: Spatiotemporal Heterogeneous Graph Neural Network Analysis: Modeling receipt elements as a spatiotemporal evolutionary system to capture implicit associations, including graph structure construction, heterogeneous message passing, and dynamic evolution prediction; S4: Multi-scale attention decision fusion: integrating local details with global context to generate management decisions, including scale separation, attention gating, and decision tree integration; S5: Adversarial Enhanced Verification: Improving system robustness through adversarial mechanisms, including generating adversarial examples, defense training, and closed-loop verification; S6: Incremental management based on knowledge distillation: Enables continuous model evolution without forgetting historical knowledge, including knowledge extraction, distillation training, and edge deployment.

[0006] Preferably, the environmental configuration includes building a closed collection room with a length, width and height of 2 meters and covering its inner walls with light-absorbing materials with a reflectivity of less than 5%; installing three sets of adjustable spectrum light source systems in orthogonal directions in the collection room, and the wavelength adjustment range of each set of light sources needs to cover 300 nanometers to 2500 nanometers; synchronously configuring two sensing devices including an industrial-grade visible light camera with a resolution of 20 million pixels and a near-infrared sensor supporting dual bands of 850 nanometers and 940 nanometers. All devices are fixed by mechanical brackets, and the clock synchronization error is within 1 millisecond.

[0007] Preferably, the synchronous acquisition is performed when the trigger light source switches in a predetermined sequence. First, in sequence 1, a combination of white light and ultraviolet light is turned on to excite the fluorescent ink features on the receipt. Then, in sequence 2, it is switched to a combination of 0-degree and 90-degree polarized light to suppress mirror reflection interference. Finally, in sequence 3, a 940nm near-infrared light source is started to penetrate surface stains and obtain underlying information. The entire process uses a hardware clock synchronizer to control the capture time error between the industrial camera and the near-infrared sensor to within 1 millisecond.

[0008] Preferably, the data preprocessing first performs optical distortion correction, and repairs the geometric distortion of the collected image through a non-uniform B-spline model. The model calculates the corrected pixel coordinates based on the control point matrix and basis function to eliminate the influence of lens distortion; then multispectral registration is performed, and the SURF feature point matching algorithm is used to align images of different spectral bands, and the registration accuracy is improved to the sub-pixel level of 0.2px, forming standardized multimodal data.

[0009] Preferably, the physical property decoupling uses a physical model to process material properties and geometric properties separately, including a material classification branch and a geometric reconstruction branch. In the material classification branch, ResNet-18 is used to extract image features, while the spectral reflectance curve is introduced as a physical prior constraint. The feature alignment loss function is used to force the network to learn the optical response characteristics of the material. In the geometric reconstruction branch, a surface reflection model is constructed based on a differentiable physical rendering engine, and a surface normal map is generated through ray tracing simulation. This normal map can accurately represent the three-dimensional geometric deformation characteristics of the data surface. Ultimately, the material classification result and the surface normal map are jointly output as the decoupled physical properties. The entire process achieves the coordinated optimization of data-driven features and physical laws through end-to-end training, ultimately generating an interpretable physical property representation.

[0010] Preferably, the semantic attribute decoupling adopts an adversarial feature separation framework, first constructing a two-way structure of a generator and a discriminator; the generator receives the physical features obtained by physical attribute decoupling, including material classification results and geometric normal maps, as input, and reconstructs the semantic features; the discriminator simultaneously receives the real semantic features and the generator output, and drives adversarial training by minimizing the expected value of the error rate of the discriminator in identifying the real semantic features and the generator's forged semantic features.

[0011] Preferably, the graph structure is constructed by defining each element in the receipt as a node and establishing multiple edge types: first, the nodes correspond to specific elements in the receipt; second, the edge types are divided into three categories: spatial edges calculate the Euclidean distance based on the physical position of the elements in the receipt image to capture spatial proximity relationships; semantic edges measure the strength of semantic associations between elements based on cosine similarity using word embedding technology; and temporal edges establish cross-time point connections for the same product items appearing in multiple historical receipts to track product change trends.

[0012] Preferably, the heterogeneous message passing distinguishes the interaction modes between different nodes by defining three independent relationship types: spatial relationship, semantic relationship and temporal relationship. During the message passing process, an exclusive trainable weight matrix is ​​assigned to each relationship type. When the node updates its own features, it first selects the corresponding weight matrix according to the currently processed edge type, and then aggregates the feature vectors of all neighbor nodes connected to the edge type. The neighbor features of each relationship type are linearly transformed and then summed up. Finally, a new feature representation of the node is generated through a nonlinear activation function.

[0013] Preferably, the dynamic evolution prediction uses a time series consistency loss method to identify abnormal changes in commodity prices. This method is implemented through a spatiotemporal heterogeneous graph neural network analysis process. First, the historical price data of the commodity is collected and its expected price is calculated as a benchmark value. Then, the deviation value between the commodity price value at the current time point and the expected price is calculated. At the same time, a time change parameter is introduced to amplify or reduce the impact of the deviation. Then, the inverse covariance matrix of the historical price data is used to evaluate the statistical significance of the deviation, thereby obtaining a weighted distance measurement value. Finally, the distance measurement value is compared with a preset dynamic threshold to determine whether an abnormality occurs. Once the threshold is exceeded, the commodity price is marked as abnormal and a management system alarm is triggered.

[0014] Preferably, the generative adversarial samples test the robustness of the system by constructing a dual attack mechanism. First, at the physical attack level, a generative adversarial network is used to generate interference texture patterns that can be actually printed from a random noise vector z. These textures are optimized to produce visual deception effects when fused with the original receipt image at the pixel level. The generator network is constrained to operate under sparse conditions to ensure that the generated interference textures are not obvious after actual printing. Specifically, the constraint is achieved by controlling the number of non-zero pixels. Secondly, at the digital attack level, the system adds perturbation vectors to the high-dimensional vector space of text embedding. These perturbations are obtained by solving an optimization problem, that is, while keeping the perturbation amplitude to a minimum, the model produces an incorrect classification result for the perturbed input. The perturbation vector is calculated using a gradient ascent method to search around the original sample for the minimum change that can cause the classification boundary to flip.

[0015] Preferably, the defense training adopts a maximum margin adversarial training strategy to enhance the robustness of the model. First, the physical and digital adversarial samples generated in the adversarial sample generation step are mixed with normal training samples to form a training set. Then, the original samples and the corresponding adversarial samples are input simultaneously in each training batch. The maximum margin criterion is adopted when calculating the loss function during the training process, that is, the model is required to maintain a safety margin between the prediction score of the correct category and the highest prediction score of the wrong category. The safety margin parameter is adjusted according to the specific application scenario. When the model's prediction confidence of the adversarial sample is lower than 0.8 or the classification is wrong, the loss function will be activated and guide the model parameter update. The training process adopts an alternating optimization strategy, that is, the model parameters are first fixed to generate stronger adversarial samples, and then the adversarial samples are fixed to update the model parameters.

[0016] Preferably, the closed-loop verification establishes an adaptive feedback mechanism from error detection results to dynamic adjustment of acquisition parameters. When the system detects an anomaly in the spatiotemporal heterogeneous graph neural network analysis of S3 or generates a low-confidence prediction in S4, the system automatically records the multispectral image features, light source configuration parameters and specific type of detection failure of the receipt, and then inputs the feature vectors of these failure cases into the light source optimization module. The module uses a Bayesian optimization algorithm to calculate the optimal spectral combination strategy. Specifically, by analyzing the reflectivity differences of failed samples at different wavelengths, the light source sequence that can maximize the contrast between the target features and the background noise is found. The system will update the switching timing and intensity ratio of the three groups of light sources in S1 in real time.

[0017] Technical effects and advantages of the present invention: 1. This invention significantly enhances physical anti-counterfeiting capabilities. By constructing a multi-spectral sequence acquisition mechanism in a restricted light environment, the system simultaneously acquires multi-dimensional physical characteristics such as ultraviolet excitation, polarization suppression, and infrared penetration, enabling in-depth analysis of paper material, ink composition, and surface structure. This allows for accurate identification of physical tampering traces that are undetectable by traditional methods, such as concealed fraudulent activities such as thermal paper fading and chemical alteration. 2. This invention achieves a breakthrough in cross-modal deep correlation analysis, using physical-semantic adversarial decoupling technology to separate the carrier attributes and commercial information of receipts. It also models elements such as products, prices, and locations as a dynamically evolving system through a spatiotemporal heterogeneous graph network. This effectively captures non-explicit correlation patterns such as "product location offset and tax rate anomalies," significantly improving the ability to detect anomalies in complex scenarios. 3. This invention has the advantages of continuous evolution and high robustness. The innovative physical-digital joint adversarial mechanism generates printable adversarial samples for closed-loop defense training, significantly enhancing the system's anti-interference ability. At the same time, combined with the multi-teacher knowledge distillation framework, it achieves efficient incremental learning while ensuring that historical knowledge is not lost, and can quickly adapt to new fraud methods and changes in business rules. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the overall structure of the present invention.

[0019] Figure 2 It is a schematic diagram of the S1 structure of the present invention.

[0020] Figure 3 It is a schematic diagram of the S2 structure of the present invention.

[0021] Figure 4 It is a schematic diagram of the S3 structure of the present invention.

[0022] Figure 5 It is a schematic diagram of the S5 structure of the present invention. DETAILED DESCRIPTION

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0024] As attached Figure 1-Figure 5 The method for intelligent management of clearing receipts based on multimodal data fusion includes: S1: Multispectral data acquisition in restricted light environments: Construct a special physical acquisition environment to obtain high-resolution multimodal data at the source of the data, including environment configuration, synchronous acquisition, and data preprocessing.

[0025] The environmental configuration includes building a closed collection room with a length, width and height of 2 meters and covering its inner walls with light-absorbing materials with a reflectivity of less than 5%; installing three sets of adjustable spectrum light source systems in orthogonal directions in the collection room, and the wavelength adjustment range of each light source must cover 300 nanometers to 2500 nanometers; and synchronously configuring two sensing devices including an industrial-grade visible light camera with a resolution of 20 million pixels and a near-infrared sensor supporting dual bands of 850 nanometers and 940 nanometers. All devices are fixed by mechanical brackets, and the clock synchronization error is within 1 millisecond.

[0026] The synchronized acquisition process is executed when the trigger light source switches in a predetermined sequence. First, in sequence 1, a combination of white and ultraviolet light is activated to excite the fluorescent ink features on the receipt. Then, in sequence 2, it switches to a combination of 0- and 90-degree polarized light to suppress specular reflection interference. Finally, in sequence 3, a 940nm near-infrared light source is activated to penetrate surface stains and acquire underlying information. Throughout this process, a hardware clock synchronizer is used to control the capture time error between the industrial camera and the near-infrared sensor to within 1 millisecond.

[0027] The data preprocessing first performs optical distortion correction, and the geometric distortion of the collected image is repaired through the non-uniform B-spline model. The model calculates the corrected pixel coordinates based on the control point matrix and basis functions to eliminate the influence of lens distortion; then multispectral registration is performed, and the SURF feature point matching algorithm is used to align images of different spectral bands, and the registration accuracy is improved to the sub-pixel level of 0.2px, forming standardized multimodal data. Let the two-dimensional coordinate position of the target image plane be s, t, and the elements of the control point matrix be , Store predefined correction parameters to determine the magnitude and direction of distortion repair. , Indicates that the corrected image is at coordinates The pixel value at and Represents B-spline basis functions, which act in the s and t directions respectively and are used for smooth interpolation calculations. Indicates traversal of the 4×4 local control point grid ( Each takes 4 values), and the final correction result is obtained through weighted combination. By locally weighting the basis functions and control points, lens distortion (such as barrel / pincushion distortion) is mapped to the ideal imaging plane.

[0028] S2: Cross-modal feature decoupling and reorganization: Extract features from multimodal data and separate physical and semantic attributes, including physical attribute decoupling, semantic attribute decoupling, and reorganization of fusion tensors.

[0029] The proposed physical property decoupling employs a physical model to separately process material and geometric properties, including a material classification branch and a geometric reconstruction branch. In the material classification branch, ResNet-18 is used to extract image features, while spectral reflectance curves are introduced as physical prior constraints. A feature alignment loss function is used to force the network to learn the optical response characteristics of the material. In the geometric reconstruction branch, a surface reflection model is constructed based on a differentiable physical rendering engine, and a surface normal map is generated through ray tracing simulation. This normal map accurately represents the three-dimensional geometric deformation characteristics of the data surface. Ultimately, the material classification result and the surface normal map are combined as the decoupled physical property output. The entire process achieves the coordinated optimization of data-driven features and physical laws through end-to-end training, ultimately generating an interpretable representation of the physical properties.

[0030] The semantic attribute decoupling adopts an adversarial feature separation framework. First, a two-way structure of generator and discriminator is constructed. The generator receives physical features obtained by physical attribute decoupling, including material classification results and geometric normal maps, as input and reconstructs semantic features. The discriminator receives both real semantic features and generator outputs. Adversarial training is driven by minimizing the expected error rate of the discriminator identifying real semantic features and the generator's forged semantic features. Let S be the semantic features, P be the physical features, and G be the generator. Construct the adversarial feature separator: During the training process, the underlying information of the physical features, such as material reflectivity and surface geometric properties, is forcibly stripped away, and ultimately, abstract semantic expressions unrelated to the physical carrier, such as pure text recognition confidence and product name semantic encoding, are extracted.

[0031] The recombined fusion tensor is achieved by splicing four feature layers: first, the eight-dimensional material coding feature layer output by the material classification branch is extracted, which contains the physical property classification information of the receipt material; second, the three-dimensional geometric coding feature layer output by the geometric reconstruction branch is obtained, which carries the surface normal vector data; then, the one-dimensional confidence feature layer output by the optical character recognition model is superimposed, which quantifies the reliability of text recognition; finally, the one-dimensional infrared penetration feature layer collected by the near-infrared sensor is fused, which characterizes the validity of the data under stain occlusion; the four feature layers are spliced ​​along the fourth dimension, and finally a fourth-order fusion tensor structure with a height of H, a width of W, a number of channels of C, and a number of feature layers of M is constructed.

[0032] S3: Spatiotemporal Heterogeneous Graph Neural Network Analysis: Modeling receipt elements as a spatiotemporal evolving system to capture implicit associations, including graph structure construction, heterogeneous message passing, and dynamic evolution prediction.

[0033] The graph structure is constructed by defining each element in the receipt as a node and establishing multiple edge types. First, the nodes correspond to specific elements in the receipt, including key entities such as product items, tax rates, and total prices. Second, edge types are divided into three categories: spatial edges calculate the Euclidean distance based on the physical position of elements in the receipt image to capture spatial proximity relationships; semantic edges measure the strength of semantic associations between elements based on cosine similarity using word embedding technology; and temporal edges establish cross-time connections for the same product items appearing in multiple historical receipts to track product change trends.

[0034] The implementation of the spatial edge is achieved by calculating the two-dimensional coordinates of the center points of the bounding boxes of each element in the receipt image, using the Euclidean distance formula to calculate the physical distance between any two elements, and establishing a spatial edge connection when the distance is less than a preset threshold. The implementation of the semantic edge first uses a pre-trained word embedding model to convert text elements such as product names and merchant categories into vector representations, then calculates the cosine similarity of each pair of element vectors, and establishes semantic edges for element pairs with similarity above the set threshold. The implementation of the temporal edge requires the establishment of a cross-receipt product index library. When the system detects the presence of the same product code or the same product confirmed by fuzzy matching in receipts at different time points, a temporal edge is established between the corresponding product nodes, and the transaction timestamp information is recorded to support subsequent temporal analysis.

[0035] The heterogeneous message passing method distinguishes the interaction modes between different nodes by defining three independent relationship types: spatial relationship, semantic relationship, and temporal relationship. During the message passing process, a unique trainable weight matrix is ​​assigned to each relationship type. When a node updates its own features, it first selects the corresponding weight matrix based on the edge type currently being processed. Then, it aggregates the feature vectors of all neighbor nodes connected to the edge type, performs linear transformation on the neighbor features of each relationship type, and then sums them up. Finally, a new feature representation of the node is generated through a nonlinear activation function. is the set of all predefined relationship types, namely {spatial relationship, semantic relationship, temporal relationship}, For a specific edge relationship type, select from the predefined relationship set If you select , is the feature vector of the current node v after the k-th layer neural network is updated, which is the output result of message passing. Represents a nonlinear activation function (such as ReLU or Sigmoid), which is used to perform nonlinear transformation on the aggregated features. Refers to the set of all neighbor nodes connected to the current node v through τ type edges, Represents the trainable weight matrix dedicated to the τ relation type in the k-th layer of the neural network. Different relation types (spatial / semantic / temporal) have independent weight matrices. The feature vector representing neighbor node u at the k-1th layer (the previous layer) serves as the input feature for the current layer. This process allows spatially adjacent receipt elements to exchange location and layout information, semantically similar items to share textual features, and temporally related items to convey price fluctuation trends. The resulting output node features simultaneously encode physical layout, textual associations, and historical change patterns, providing a fused representation foundation for subsequent temporal consistency anomaly detection.

[0036] What needs to be specifically explained in this embodiment is that in a heterogeneous graph neural network, each edge must belong to a predefined relationship type. Therefore: a spatial edge is a connection that carries a spatial relationship, a semantic edge is a connection that carries a semantic relationship, and a temporal edge is a connection that carries a temporal relationship.

[0037] The dynamic evolution prediction uses the time series consistency loss method to identify abnormal changes in commodity prices. This method is implemented through a spatiotemporal heterogeneous graph neural network analysis process. First, the historical price data of the commodity is collected and its expected price is calculated as a benchmark value. Then, the deviation value between the commodity price value at the current time point and the expected price is calculated. At the same time, a time variation parameter is introduced to amplify or reduce the impact of the deviation. Then, the inverse covariance matrix of the historical price data is used to evaluate the statistical significance of the deviation, thereby obtaining a weighted distance measurement value. Finally, the distance measurement value is compared with a preset dynamic threshold to determine whether an abnormality occurs. Once the threshold is exceeded, the commodity price is marked as abnormal and the management system alarm is triggered. is the time function of commodity price, then the temporal consistency loss , represents the time variation, is the observed value of the commodity price function at the current time t, that is, the actual price of the target commodity at time point t, It indicates the historical expected value of commodity prices, which is calculated by averaging the commodity's long-term price data in the past and represents the historical normal price level. The Mahalanobis distance operator applies the inverse covariance matrix to the parenthesized vector (price deviation multiplied by time variance) to eliminate the impact of price fluctuations across commodities. This entire process is based entirely on the dynamic evolution of historical price series, enabling automated detection without manual intervention.

[0038] S4: Multi-scale attention decision fusion: Integrating local details with global context to generate management decisions, including scale separation, attention gating, and decision tree integration.

[0039] The scale separation extracts features from the input receipt image through three parallel processing branches. First, in the micro-scale branch, a 3×3 convolution kernel is used to scan the original image pixel by pixel to extract texture details such as paper fiber orientation and ink penetration depth. The output feature map F_micro maintains the original image size but has 128 channels to preserve fine-grained information. Simultaneously, in the meso-scale branch, an object detection algorithm is used to identify the product item regions on the receipt. Then, a RoIAlign operation is used to align and pool features for each detected rectangular region, ensuring that product items of different sizes are extracted into a feature vector F_meso of uniform dimension. Each item corresponds to a 256-dimensional feature vector. Finally, in the macro-scale branch, the OCR-recognized text of the entire receipt is arranged into a sequence from top to bottom and input into a bidirectional LSTM network for encoding. The hidden state dimension of the LSTM is set to 512. Through forward and backward information propagation, the context of the entire receipt is captured to generate sequence features F_macro. The feature extraction processes of the three branches are executed completely in parallel. Finally, the features of three different scales are spliced ​​and fused at the feature level to form a complete multi-scale feature representation for subsequent attention gating processing.

[0040] The attention gate is first linearly transformed through three independent fully connected layers to generate a query vector, a key vector and a value vector, where the query vector is used to represent the feature content that needs to be paid attention to, the key vector is used to calculate the similarity matching degree between features, and the value vector carries the actual feature information. Then the cosine similarity score between each pair of features is calculated. Specifically, the query vector is dot-producted with all key vectors and divided by the product of the vector length for normalization. The similarity score is then input into the softmax function for normalization to obtain the attention weight so that the sum of all weights is equal to 1. The larger the weight value, the higher the importance of the corresponding feature. The attention weight is then multiplied by the corresponding value vector and summed to obtain the weighted fusion feature representation. Assuming i and j are different scales, the multi-scale decision model , represents the attention weight coefficient of the i-th scale to the j-th scale, with a value range of [0,1] and satisfying , the weight represents the contribution of features from scale j in the feature calculation of the current scale i, Represents the feature representation vector of the i-th scale, including micro-scale features , mesoscale characteristics , macro-scale characteristics , is the query weight matrix, which is used to transform the features Mapped to the query space, the matrix is ​​learned through back propagation to determine "what to focus on" at the current scale. is the key weight matrix, which is used to transform the features Mapped to the key space, the matrix learns the "attention attributes" of other scale features, The similarity calculation function, cosine similarity, measures the degree of match between the query vector and the key vector in semantic space. This process dynamically allocates weights between features at different scales, enabling the system to automatically adjust its focus on microscopic texture details, mesoscopic target areas, or macroscopic overall context based on the needs of the current task. The resulting fused features capture both local details and global semantic information, providing high-quality, attention-filtered feature input for subsequent decision tree integration.

[0041] The decision tree ensemble adopts a hybrid architecture design of random forest and neural network. First, the multi-scale features extracted in the scale separation step are input into a pre-trained 10-layer convolutional neural network for feature encoding to obtain a 512-dimensional high-level semantic representation vector. This vector is then distributed as input to a random forest module consisting of 100 independently trained decision trees. Each decision tree uses a different feature subset and sample subset during training to ensure diversity. Specifically, each tree randomly selects 70% of the feature dimensions and 80% of the training samples for construction. The maximum depth of the tree is limited to 15 layers to prevent overfitting. Each leaf node outputs a probability distribution vector representing the confidence level for each category. 100 trees The outputs of the tree's leaf nodes are aggregated using a weighted average, with weights dynamically assigned based on each tree's accuracy on the validation set. The aggregated probability vector is then fed into a two-layer fully connected neural network for the final classification decision. The first fully connected layer contains 256 neurons and uses the ReLU activation function. The second fully connected layer has an output dimension equal to the final number of categories, and uses the Softmax activation function to generate the final classification probability. The entire hybrid model is trained end-to-end, using a loss function that combines cross-entropy loss with an entropy regularization term on the decision tree output. Training utilizes the Adam optimizer, with a learning rate set to 0.001, and 64 random samples for gradient updates at each iteration. This design effectively combines the interpretability of decision trees with the nonlinear fitting capabilities of neural networks.

[0042] S5: Adversarial Enhanced Verification: Improve system robustness through adversarial mechanisms, including generating adversarial samples, defense training, and closed-loop verification.

[0043] The proposed generative adversarial example system tests the robustness of the system by constructing a dual attack mechanism. First, at the physical attack level, a generative adversarial network (GAN) is used to generate printable interference texture patterns from a random noise vector z. These textures are optimized to produce visual deception when fused with the original receipt image at the pixel level. The generator network is constrained to operate under sparse conditions to ensure that the generated interference textures are not noticeable after printing. This constraint is achieved by controlling the number of non-zero pixels. Second, at the digital attack level, the system adds perturbation vectors to the high-dimensional vector space of text embeddings. These perturbations are obtained by solving an optimization problem: minimizing the perturbation amplitude while causing the model to misclassify the perturbed input. The perturbation vectors are calculated using a gradient ascent method around the original sample to search for the smallest change that will flip the classification boundary. The entire adversarial example generation process encompasses both physically feasible and purely digital attack methods. This dual mechanism comprehensively tests the intelligent receipt management system's defense capabilities against malicious interference.

[0044] The defense training adopts a maximum margin adversarial training strategy to enhance the robustness of the model. First, the physical and digital adversarial samples generated in the adversarial sample generation step are mixed with normal training samples to form a training set. Then, the original samples and the corresponding adversarial samples are input simultaneously in each training batch. The maximum margin criterion is adopted when calculating the loss function during training, that is, the model is required to maintain a safety margin between the prediction score of the correct category and the highest prediction score of the wrong category. The safety margin parameter is adjusted according to the specific application scenario. When the model's prediction confidence of the adversarial sample is lower than 0.8 or the classification is wrong, the loss function will be activated and guide the update of the model parameters. The training process adopts an alternating optimization strategy, that is, first fixing the model parameters to generate stronger adversarial samples, and then fixing the adversarial samples to update the model parameters.

[0045] set up is the adversarial sample set, is a safety boundary, then the maximum margin adversarial training , L is the adversarial training loss function value, which indicates the risk level of the current sample under adversarial attack, It defines the minimum decision interval between the correct category and the wrong category, x is the original input sample, and in the clearing receipt scenario, it refers to the receipt data containing multimodal information. is a set of adversarial samples centered on x, including physical adversarial samples and digital adversarial samples, is a specific sample in the adversarial sample set, For neural networks in adversarial examples The output vectors on include "Compliance", "Amount Abnormal", "Supplier Abnormal", etc. is the model’s prediction confidence for the true label y, Predict labels for the model The confidence level, It means that among all possible adversarial samples, we look for the sample that minimizes the difference between the true category confidence and the predicted category confidence. is the ReLU activation function, ensuring that the loss value is non-negative. When an attacker attempts to deceive the system by modifying the amount or product name on the receipt, this loss function ensures that even under the worst adversarial examples, the model's confidence in judging "abnormal receipts" is at least higher than its confidence in misjudging "normal receipts." For example, if , the system is required to have a confidence level of at least 0.2 higher than that of misjudging a carefully forged receipt as abnormal, thus avoiding financial risks. This adversarial training method forces the model to learn more stable feature representations, enabling it to maintain good classification performance in the face of unknown attacks. The entire defense training process gradually improves the model's anti-interference capabilities through multiple rounds of iteration, ultimately enabling the receipt intelligent management system to maintain high recognition and classification accuracy in adversarial environments.

[0046] The closed-loop verification establishes an adaptive feedback mechanism from error detection results to dynamic adjustment of acquisition parameters. When the system detects an anomaly in the spatiotemporal heterogeneous graph neural network analysis of S3 or generates a low-confidence prediction in S4, the system automatically records the multispectral image features, light source configuration parameters, and specific type of detection failure of the receipt, and then inputs the feature vectors of these failure cases into the light source optimization module. The module uses a Bayesian optimization algorithm to calculate the optimal spectral combination strategy. Specifically, by analyzing the reflectivity differences of failed samples at different wavelengths, the module finds the light source sequence that can maximize the contrast between the target features and the background noise. The system will update the switching timing and intensity ratio of the three groups of light sources in S1 in real time. For example, when it detects a decrease in the text recognition accuracy of a certain type of thermal paper receipt, the system will automatically increase the exposure time of 940nm near-infrared light and reduce the white light intensity. At the same time, it adjusts the angle of polarized light from the original 0 degrees and 90 degrees to 45 degrees and 135 degrees. This parameter adjustment will be immediately applied to the collection process of the next receipt, forming a closed-loop mechanism of real-time learning and improvement, ensuring that the collection quality continues to improve with the increase in the number of processed samples. The response time of this feedback process is controlled within 100 milliseconds, ensuring the real-time performance of the system.

[0047] S6: Incremental management based on knowledge distillation: Enables continuous model evolution without forgetting historical knowledge, including knowledge extraction, distillation training, and edge deployment.

[0048] The knowledge extraction adopts the feature mapping and attention weight extraction method of multiple teacher networks. First, the node embedding vector and the weight matrix of the graph convolution layer are extracted from the spatiotemporal graph neural network teacher model. The 128-dimensional feature representation of each receipt element is obtained by traversing all graph nodes. At the same time, the attention weight distribution of the three relationship types in the heterogeneous message transmission process is recorded. These weights are used as soft labels to guide the training of the student model. Secondly, feature maps of three scales are extracted from the multi-scale decision model. Specifically, the micro-scale 3×3 convolution feature map is globally average pooled to obtain the texture descriptor, the meso-scale RoIAlign feature is reduced to 64 dimensions through principal component analysis to retain the main information, and the macro-scale Bi-LSTM hidden state sequence is subjected to maximum pooling operation to extract key temporal features. The knowledge of the two teacher models is then fused through feature alignment. The linear transformation matrix of the feature space is calculated using the least squares method to achieve alignment of the outputs of different teacher models. Finally, a knowledge distillation loss function is constructed to simultaneously transfer the soft prediction distribution and intermediate layer features of the teacher model to the student model, and the smoothness of the soft label is controlled by the temperature parameter. Ensure that the student model can learn the decision logic of the teacher model rather than just the final classification results.

[0049] The distillation training is achieved by combining multi-teacher knowledge extraction and feature-level distillation loss. First, the spatiotemporal graph neural network in S3 and the multi-scale decision model in S4 are used as two teacher models, and then a lightweight student model is constructed as the target network. During the training process, the student model must not only learn the correct prediction results, but also imitate the intermediate feature representation of the teacher model. The learning process of the student model is constrained by calculating the feature-level distillation loss function. The loss function contains two parts. The first part is the L2 distance loss of the feature map, that is, the square of the Euclidean distance between the teacher model and the student model is calculated for the feature output of each layer of the network. The second part is the KL divergence loss of the predicted distribution, which is used to ensure that the output probability distribution of the student model is consistent with the teacher model. An alternating optimization strategy is adopted during training. First, the teacher model parameters are fixed to update the student model, and then the teacher model is periodically fine-tuned using the newly collected receipt data. Assume is the feature map of the lth layer, p is the predicted distribution, then the distillation loss function , represents the overall distillation loss function, represents the feature output of the teacher model layer l, represents the feature output of the lth layer of the student model, Represents the square of the L2 norm, which is used to measure the distance between the feature outputs of the teacher and student models at each layer. Indicates the accumulation of feature outputs of all L network layers, that is, constraining the feature expression of the entire model. Represents the weight coefficient between feature-level loss and probability distribution loss, which is used to balance the relative importance of the two. Represents the output probability distribution of the teacher model And the output probability distribution of the student model The KL divergence between the two is used to ensure that the output of the student model is consistent with the teacher model. This ensures knowledge transfer while avoiding catastrophic forgetting. The entire distillation process is optimized using a gradient descent algorithm. The learning rate is gradually decayed from 0.01 to 0.0001 using a cosine annealing schedule. The number of training rounds is set to 50 epochs, and a validation set evaluation is performed every 10 epochs to monitor model performance.

[0050] The edge deployment includes three core links. The first is the neural architecture search to generate a lightweight quantum model. By setting computing resource constraints, such as memory not exceeding 512MB and inference delay less than 100ms, the network structure is automatically optimized using a differentiable architecture search algorithm. The search space covers convolution kernel sizes from 1x1 to 7x7, number of channels from 16 to 512, and activation function types including ReLU and Swish variants. The search process uses gradient descent to optimize the super network weights, and finally outputs the optimal sub-network architecture that meets the edge device constraints. The second is the implementation of the mixed precision quantization scheme, which quantizes the weights of the convolution layer and the fully connected layer in the original FP32 model to FP16 format to reduce storage overhead, and at the same time converts the activation function output and batch normalization parameter quantity It is converted to INT8 format to accelerate calculations. During the quantization process, the entropy calibration method is used to determine the optimal quantization threshold, that is, by statistically analyzing the entropy changes in the activation value distribution on the calibration data set, the truncation point that minimizes information loss is selected as the quantization boundary. Finally, the model is compressed and deployed for optimization. The quantized model is used to remove redundant connections using model pruning technology. The pruning strategy is based on weight amplitude and gradient sensitivity analysis, and retains network connections that contribute highly to the final output. After compression, the model is converted to an inference engine format supported by edge devices, such as TensorRT or OpenVINO, and memory layout optimization and computational graph fusion are performed on the target hardware platform to ensure that the model can run stably in resource-constrained edge environments and maintain the expected inference performance.

[0051] The present invention starts with multispectral data collection in a restricted lighting environment, constructs a special physical collection environment to obtain high-resolution multimodal data, including environmental configuration such as building a closed collection room covered with light-absorbing materials and installing three sets of adjustable spectral light source systems. Synchronous collection controls the light source sequence switching to capture images through a hardware clock synchronizer. Data preprocessing performs optical distortion correction and multispectral registration to form standardized multimodal data. Cross-modal feature decoupling and reorganization are then performed. Material attributes and geometric attributes are separated by physical attribute decoupling. ResNet-18 is used to extract features and combined with physical prior constraints. The geometric reconstruction branch generates a surface normal map. Semantic attribute decoupling uses an adversarial feature separation framework to strip physical information and extract pure semantic expressions. The recombined fusion tensor is constructed by splicing the material coding feature layer, the geometric coding feature layer, the confidence feature layer and the infrared penetration feature layer. The fourth-order fusion tensor is constructed. Then the spatiotemporal heterogeneous graph neural network analysis stage is entered. The receipt elements are modeled as nodes and spatial edges, semantic edges and temporal edges are established. Heterogeneous message transmission is achieved through trainable weights. The matrix processes different relationship types to dynamically update node features, and the dynamic evolution prediction uses the temporal consistency loss method to detect abnormal commodity prices and trigger alarms; then multi-scale attention decision fusion is implemented, and features are extracted in parallel through micro-scale, meso-scale and macro-scale branches. The attention gating mechanism dynamically calculates weights and weightedly fuses features, and the decision tree integration uses a hybrid architecture of random forest and neural network to generate classification decisions; then adversarial reinforcement verification is performed, and adversarial samples are generated including physical attack and digital attack dual mechanisms to test robustness. Defense training uses the maximum margin adversarial training strategy to mix samples and optimize the model. Closed-loop verification establishes an adaptive feedback mechanism to dynamically adjust acquisition parameters such as light source sequence; finally, in the incremental management stage based on knowledge distillation, features and weights are extracted from the teacher model through knowledge extraction, and distillation training combines feature-level loss and prediction distribution loss to update the student model. Edge deployment includes neural architecture search to generate lightweight quantum models, mixed precision quantization schemes, and model compression and deployment optimization to ensure that the system continues to evolve and runs stably in resource-constrained environments.

[0052] Secondly: The drawings of the embodiments disclosed in the present invention only involve structures related to the embodiments disclosed in the present invention. Other structures may refer to conventional designs. The same embodiment and different embodiments of the present invention may be combined with each other without conflict. Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. The intelligent management method of liquidation receipts based on multimodal data fusion is characterized by: include: S1: Multispectral data acquisition in restricted light environments: Construct a special physical acquisition environment to obtain high-resolution multimodal data at the source of the data, including environment configuration, synchronous acquisition, and data preprocessing; S2: Cross-modal feature decoupling and reorganization: Extract features from multimodal data and separate physical and semantic attributes, including physical attribute decoupling, semantic attribute decoupling, and reorganization of fusion tensors; S3: Spatiotemporal Heterogeneous Graph Neural Network Analysis: Modeling receipt elements as a spatiotemporal evolutionary system to capture implicit associations, including graph structure construction, heterogeneous message passing, and dynamic evolution prediction; S4: Multi-scale attention decision fusion: integrating local details with global context to generate management decisions, including scale separation, attention gating, and decision tree integration; S5: Adversarial Enhanced Verification: Improving system robustness through adversarial mechanisms, including generating adversarial examples, defense training, and closed-loop verification; S6: Incremental management based on knowledge distillation: Enables continuous model evolution without forgetting historical knowledge, including knowledge extraction, distillation training, and edge deployment.

2. The method for intelligent management of liquidation receipts based on multimodal data fusion according to claim 1, characterized in that: The environmental configuration includes building a closed collection chamber with a length, width, and height of 2 meters, and covering its inner walls with a light-absorbing material with a reflectivity of less than 5%; installing three sets of tunable spectral light source systems in orthogonal directions in the collection chamber, with each set of light sources required to cover a wavelength adjustment range of 300 nanometers to 2500 nanometers; and synchronously configuring two sensing devices, including an industrial-grade visible light camera with a resolution of 20 million pixels and a near-infrared sensor supporting dual bands of 850 nanometers and 940 nanometers. All devices are fixed with mechanical brackets, and the clock synchronization error is within 1 millisecond. The synchronous acquisition process is executed when the trigger light source switches in a predetermined sequence. First, in sequence 1, a combination of white and ultraviolet light is activated to excite the fluorescent ink features on the receipt. Then, in sequence 2, a combination of 0- and 90-degree polarized light is switched to suppress specular reflection interference. Finally, in sequence 3, a 940nm near-infrared light source is activated to penetrate surface stains and acquire underlying information. Throughout this process, a hardware clock synchronizer is used to control the capture time error between the industrial camera and the near-infrared sensor to within 1 millisecond. The data preprocessing first performs optical distortion correction, repairing the geometric distortion of the collected images using a non-uniform B-spline model. The model calculates the corrected pixel coordinates based on a control point matrix and basis functions to eliminate the effects of lens distortion. Multispectral registration is then performed, using the SURF feature point matching algorithm to align images of different spectral bands, improving the registration accuracy to a sub-pixel level of 0.2px, forming standardized multimodal data.

3. The method for intelligent management of liquidation receipts based on multimodal data fusion according to claim 1, characterized in that: The physical property decoupling adopts a physical model to process material properties and geometric properties separately, including a material classification branch and a geometric reconstruction branch. In the material classification branch, ResNet-18 is used to extract image features, and the spectral reflectance curve is introduced as a physical prior constraint. The feature alignment loss function is used to force the network to learn the optical response characteristics of the material. In the geometric reconstruction branch, a surface reflection model is constructed based on a differentiable physical rendering engine, and a surface normal map is generated through ray tracing simulation. The semantic attribute decoupling adopts an adversarial feature separation framework. First, a two-way structure of a generator and a discriminator is constructed. The generator receives the physical features obtained by physical attribute decoupling as input, including material classification results and geometric normal maps, and reconstructs semantic features. The discriminator receives both real semantic features and generator outputs, and drives adversarial training by minimizing the expected error rate of the discriminator in identifying real semantic features and the generator in forging semantic features.

4. The method for intelligent management of liquidation receipts based on multimodal data fusion according to claim 1, characterized in that: The graph structure is constructed by defining each element in the receipt as a node and establishing multiple edge types. First, the nodes correspond to specific elements in the receipt. Second, the edge types are divided into three categories: spatial edges calculate the Euclidean distance based on the physical position of the elements in the receipt image to capture spatial proximity relationships; semantic edges measure the strength of semantic associations between elements based on cosine similarity using word embedding technology; and temporal edges establish cross-time connections for the same product items appearing in multiple historical receipts to track product change trends.

5. The method for intelligent management of liquidation receipts based on multimodal data fusion according to claim 1, characterized in that: The heterogeneous message passing method distinguishes the interaction modes between different nodes by defining three independent relationship types: spatial relationship, semantic relationship and temporal relationship. During the message passing process, a unique trainable weight matrix is ​​assigned to each relationship type. When a node updates its own features, it first selects the corresponding weight matrix according to the currently processed edge type, then aggregates the feature vectors of all neighbor nodes connected to the edge type, performs linear transformation on the neighbor features of each relationship type and then sums them up, and finally generates a new feature representation of the node through a nonlinear activation function.

6. The method for intelligent management of liquidation receipts based on multimodal data fusion according to claim 1, characterized in that: The dynamic evolution prediction uses a temporal consistency loss method to identify abnormal changes in commodity prices. This method is implemented through a spatiotemporal heterogeneous graph neural network analysis process. First, historical price data of commodities are collected and their expected prices are calculated as a benchmark value. Then, the deviation between the commodity price value at the current time point and the expected price is calculated. At the same time, a time variation parameter is introduced to amplify or reduce the impact of the deviation. Then, the inverse covariance matrix of the historical price data is used to evaluate the statistical significance of the deviation, thereby obtaining a weighted distance measurement value. Finally, the distance measurement value is compared with a preset dynamic threshold to determine whether an anomaly occurs. Once the threshold is exceeded, the commodity price is marked as abnormal and a management system alarm is triggered.

7. The method for intelligent management of liquidation receipts based on multimodal data fusion according to claim 1, characterized in that: The generative adversarial samples described herein test the robustness of the system by constructing a dual attack mechanism. First, at the physical attack level, a generative adversarial network is used to generate printable interference texture patterns from a random noise vector z. These textures are optimized to produce visual deception when fused with the original receipt image at the pixel level. The generator network is constrained to operate under sparse conditions to ensure that the generated interference texture is not obvious after actual printing. Specifically, the constraint is achieved by controlling the number of non-zero pixels. Second, at the digital attack level, the system adds perturbation vectors to the high-dimensional vector space of text embedding. These perturbations are obtained by solving an optimization problem, that is, making the model misclassify the perturbed input while minimizing the perturbation amplitude. The perturbation vector is calculated using a gradient ascent method to search around the original sample for the minimum change that can cause the classification boundary to flip.

8. The method for intelligent management of liquidation receipts based on multimodal data fusion according to claim 1, characterized in that: The defense training adopts a maximum margin adversarial training strategy to enhance the robustness of the model. First, the physical and digital adversarial samples generated in the adversarial sample generation step are mixed with normal training samples to form a training set. Then, the original samples and the corresponding adversarial samples are input simultaneously in each training batch. The maximum margin criterion is adopted when calculating the loss function during training, that is, the model is required to maintain a safety margin between the prediction score of the correct category and the highest prediction score of the wrong category. The safety margin parameter is adjusted according to the specific application scenario. When the model's prediction confidence of the adversarial sample is lower than 0.8 or the classification is wrong, the loss function will be activated and guide the update of the model parameters. The training process adopts an alternating optimization strategy, that is, first fixing the model parameters to generate stronger adversarial samples, and then fixing the adversarial samples to update the model parameters.

9. The method for intelligent management of liquidation receipts based on multimodal data fusion according to claim 1, characterized in that: The closed-loop verification establishes an adaptive feedback mechanism from error detection results to dynamic adjustment of acquisition parameters. When the system detects an anomaly in the spatiotemporal heterogeneous graph neural network analysis of S3 or generates a low-confidence prediction in S4, the system automatically records the multispectral image features, light source configuration parameters, and specific type of detection failure of the receipt, and then inputs the feature vectors of these failure cases into the light source optimization module. The module uses a Bayesian optimization algorithm to calculate the optimal spectral combination strategy. Specifically, by analyzing the reflectivity differences of failed samples at different wavelengths, the module finds the light source sequence that can maximize the contrast between the target features and the background noise. The system will update the switching timing and intensity ratio of the three groups of light sources in S1 in real time.

Images