A customs clearance document automatic processing method, system and device based on multi-source data fusion and a medium

Abstract

The application discloses a kind of based on multi-source data fusion's customs clearance document automation processing method, system, equipment and medium, belong to computer technology and artificial intelligence technical field, including: acquisition customs clearance business in multi-source heterogeneous data, structured processing is generated multimodal information flow;Multimodal information flow is input into pre-trained multimodal fusion model and carries out cross-modal semantic analysis, outputs local processing result, generates multi-target optimal decision scheme set based on local processing result, receives the final decision instruction generated after artificial review to complete customs clearance business processing;Collect the processing record containing multimodal information flow, local processing result and final decision instruction, filter high-value training sample, establish negative deviation sample library;Based on negative deviation sample library executes asynchronous federal aggregation, to update multimodal fusion model, realizes from accurate perception, intelligent decision to global knowledge collaborative evolution Dynamic risk control closed loop.

Description

Technical Field

[0001] This invention relates to the intersection of computer technology and artificial intelligence, and in particular to a method, system, device and medium for automated processing of customs clearance documents based on multi-source data fusion. Background Technology

[0002] As a core hub of international trade, customs clearance directly impacts trade facilitation levels through the accuracy and timeliness of its data processing. With the deepening development of cross-border e-commerce and global supply chains, customs processes massive amounts of data daily, including customs declarations, accompanying commercial invoices, packing list images, and logistics tracking data. Current technologies typically employ an architecture combining hierarchical federated learning and multimodal fusion models. This involves feature alignment through a learning mechanism and reinforcement learning for customs clearance decisions. The initial design intent of this type of architecture is to break down data silos between ports of entry and leverage the complementarity of multimodal data to improve the accuracy of automated document review.

[0003] However, as customs operations become increasingly complex, existing technologies, when processing multimodal data, generally tend to pursue absolute alignment between different modalities. During model training and inference, systems often treat contradictory data that is difficult to align or data with semantic discrepancies between modalities as noise and smooth or remove them. For example, when there is a logical discrepancy between the weight of goods shown in the customs declaration text and the volume features extracted from the packing list image, traditional fusion networks will mask this discrepancy through feature averaging or dimensionality reduction projection to output a stable classification result. Such centralized or strongly aligned scheduling mechanisms inevitably become bottlenecks for performance and risk control: all generation requests or risk assessments must sequentially pass through a single alignment logic, which not only causes task queuing delays but also limits the system's scalability to abnormal features due to strong coupling. More importantly, such systems typically adopt a "full smoothing, global averaging" strategy, that is, regardless of whether there is a potential risk of false declarations or concealment in the current data, global feature fusion is performed. This results in a large amount of computing resources being wasted on redundant calculations in risk-free areas when processing massive amounts of heterogeneous data, severely restricting the system's response speed and throughput to hidden risks.

[0004] Therefore, how to construct an automated customs clearance document processing architecture that can decouple routine data processing from abnormal risk capture, support dynamic amplification of conflict features on demand, and possess natural parallel expansion and global anomaly sharing capabilities, while simultaneously establishing a feedback channel between local risk assessment and global model evolution, has become a key challenge and an urgent technical problem for those skilled in the art. Summary of the Invention

[0005] This invention provides a method, system, device, and medium for automated processing of customs clearance documents based on multi-source data fusion. This addresses the problems of existing customs clearance systems based on multimodal fusion, which overwhelm risk features due to the forced pursuit of feature alignment, and whose centralized and static architecture leads to low processing efficiency, serious resource waste, and difficulty in achieving global collaborative evolution in response to new risks.

[0006] In a first aspect, the present invention provides a method for automated processing of customs clearance documents based on multi-source data fusion, comprising: Collect multi-source heterogeneous data from customs clearance operations, and perform structured processing on the multi-source heterogeneous data to generate a multimodal information flow; The multimodal information stream is input into a pre-trained multimodal fusion model for cross-modal semantic analysis, and the local processing result is output. The multimodal fusion model includes a feature alignment stream and a conflict capture stream. Based on the local processing results, a set of multi-objective optimal decision schemes is generated through reinforcement learning decision-making, and the final decision instruction generated after manual review is received and executed to complete the customs clearance business processing. Collect processing records containing the multimodal information stream, the local processing results, and the final decision instruction; use an active learning mechanism to filter high-value training samples and establish a negative bias sample library. Asynchronous federated aggregation is performed based on the negative bias sample library to update the multimodal fusion model.

[0007] Optionally, the step of inputting the multimodal information stream into a pre-trained multimodal fusion model for cross-modal semantic analysis and outputting local processing results includes: calculating the modal deviation of each modal feature representation in the multimodal information stream in the shared latent space; When the modal deviation is lower than or equal to a preset first threshold, the local processing result is output through the feature alignment stream; When the modal deviation exceeds a preset first threshold, the conflict capture stream is activated; The conflict capture stream uses an attention mechanism combined with a local rule knowledge graph to perform preliminary risk assessment and output local processing results.

[0008] Optionally, calculating the modal deviation of each modal feature representation in the multimodal information stream within the shared latent space includes: The text modality feature vectors, image modality feature vectors, and temporal feature vectors in the multimodal information stream are respectively mapped to the shared latent space; The bulldozer distance algorithm based on optimal transmission theory is used to calculate the distribution differences between the text modal feature vector and the image modal feature vector, the text modal feature vector and the temporal feature vector, and the image modal feature vector and the temporal feature vector in the shared latent space, respectively, and the distribution differences are used as the modal deviation.

[0009] Optionally, the conflict capture stream performs preliminary risk assessment using an attention mechanism combined with a local rule knowledge graph, and outputs local processing results, including: Construct a cross-attention matrix, using the non-aligned conflict features in the multimodal information flow as the query vector, and the historical violation feature entities in the local rule knowledge graph as the key vector and value vector; Calculate the similarity score between the query vector and the key vector, and normalize the similarity score using a normalized exponential function; The normalized similarity score is weighted and summed with the value vector to obtain the enhanced conflict feature representation. This is then combined with the local rule knowledge graph to perform a preliminary risk assessment and output the local processing result.

[0010] Optionally, the step of generating a multi-objective optimal decision set based on the local processing results through reinforcement learning, receiving a final decision instruction generated after manual review, and executing the final decision instruction to complete the customs clearance process includes: A Markov decision process model is constructed, in which the local processing result is defined as the state space, and customs clearance, manual inspection, and supplementary declaration are defined as the action space. Design a multi-objective reward function, which includes a customs clearance efficiency reward item, a tax security reward item, and a risk prevention and control reward item; The Markov decision process model is iteratively solved using a proximal policy optimization algorithm, and the set of multi-objective optimal decision schemes that maximize the cumulative expected reward in the current state is output. Receive the final decision instruction generated after manual review, and execute the final decision instruction to complete the customs clearance process.

[0011] Optionally, the collection includes the processing records of the multimodal information stream, the local processing results, and the final decision instruction, and establishes a negative bias sample library by screening high-value training samples through an active learning mechanism, including: Collect processing records containing the multimodal information stream, the local processing results, and the final decision instruction; Extract records from the processing records that have a classification difference from the final decision instruction as first candidate samples; Records containing the non-alignment conflict features and causing the prediction entropy value output by the multimodal fusion model to be greater than a preset uncertainty threshold are extracted as second candidate samples. The first candidate sample and the second candidate sample are deduplicated and their features are concatenated, and then stored in the negative bias sample library.

[0012] Optionally, the step of performing asynchronous federated aggregation based on the negative bias sample library to update the multimodal fusion model includes: Based on the negative deviation sample library, asynchronous federated aggregation is performed periodically to obtain the aggregated global model and global anomaly pattern map; The multimodal fusion model is updated based on the global model and the global anomaly pattern map.

[0013] Secondly, the present invention provides an automated customs clearance document processing system based on multi-source data fusion, the system being used to execute any of the automated customs clearance document processing methods based on multi-source data fusion described in the present invention, including: The data acquisition module is used to collect multi-source heterogeneous data in customs clearance operations and to perform structured processing on the multi-source heterogeneous data to generate a multimodal information stream. The output module is used to input the multimodal information stream into a pre-trained multimodal fusion model for cross-modal semantic analysis and output the local processing results. The multimodal fusion model includes a feature alignment stream and a conflict capture stream. The execution module is used to generate a set of multi-objective optimal decision schemes through reinforcement learning based on the local processing results, and to receive the final decision instruction generated after manual review, and execute the final decision instruction to complete the customs clearance business processing; The filtering module is used to collect processing records containing the multimodal information stream, the local processing results, and the final decision instruction, and to filter high-value training samples through an active learning mechanism to establish a negative bias sample library. An update module is used to perform asynchronous federated aggregation based on the negative bias sample library to update the multimodal fusion model.

[0014] Thirdly, the present invention provides an electronic device, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to execute the automated customs clearance document processing method based on multi-source data fusion as described in any embodiment of the present invention.

[0015] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions, which are used to cause a processor to execute the automated processing method for customs clearance documents based on multi-source data fusion as described in any embodiment of the present invention.

[0016] Beneficial effects: (1) The technical solution of the present invention introduces a dual-stream multimodal fusion model that includes feature alignment stream and conflict capture stream, and dynamically schedules data processing paths based on real-time calculated modal deviation, enabling the system to perform smooth fusion on conventional data with consistent modalities, while specifically extracting and amplifying the semantic conflicts represented by high deviation. Compared with the prior art, the present invention achieves logical decoupling between conventional data processing and abnormal risk capture, avoids the waste of computing resources in traditional forced alignment, focuses computing power on potential risk points, and improves processing efficiency and risk prevention and control. Furthermore, this on-demand scheduling mechanism enables the system to complete the judgment of suspicious documents within millisecond-level latency, significantly improving the real-time detection and interception capabilities of concealed violations such as false reporting and concealment.

[0017] (2) The technical solution of this invention quantifies modal deviation by using a bulldozer distance algorithm based on optimal transmission theory, and uses this as a criterion to activate the conflict capture stream. Then, it uses an attention mechanism combined with a local rule knowledge graph to enhance the judgment of non-aligned conflict features. This process first realizes a closed loop from perception, quantification to recognition of semantic contradictions between multimodal information, transforming the intermodal discrepancies, which are regarded as noise in traditional methods, into calculable and interpretable risk quantification signals, and outputs concrete risk warnings through knowledge association; thus realizing the targeted amplification and in-depth analysis of risk features. Furthermore, this mechanism enables the system not only to identify existing contradictions, but also to understand what historical violations the contradictions are similar to, thereby improving the accuracy, interpretability and decision support of the initial risk assessment.

[0018] (3) The technical solution of this invention establishes a continuous model evolution closed loop by constructing a negative bias sample library and performing asynchronous federated aggregation. This mechanism actively collects samples containing classification differences and high uncertainty conflict features, and performs specialized clustering and knowledge distillation on abnormal gradients under the federated aggregation framework. By transforming the rare abnormal risk patterns captured locally by each edge node into a globally shareable global abnormal pattern map, and driving all node models to update synchronously, it achieves rapid aggregation and collaborative immunity against new and hidden risk patterns. Furthermore, it drives the entire customs risk prevention and control network to evolve collaboratively from passively responding to local risks to actively immunizing against new global risks, greatly enhancing the adaptability and foresight of the risk control system.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1 is a flowchart of a method for automating customs clearance document processing based on multi-source data fusion provided in Embodiment 1 of the present invention; Figure 2 is a framework diagram of a customs clearance document automation processing system based on multi-source data fusion provided in Embodiment 3 of the present invention; Figure 3 shows a schematic diagram of the structure of an electronic device that can be used to implement an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0023] It should be understood that the data used in this way can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in a sequence other than those illustrated or described herein. Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] Application Overview: The inherent characteristics of existing technologies at the principle level have gradually revealed deep-seated contradictions when addressing new challenges. Fundamentally, this lies in the structural mismatch between their static data alignment paradigm and the dynamic need for identifying hidden risks. On the one hand, customs clearance data itself possesses strong multimodal correlations and logical rigor. In real-world risk identification processes, even minor semantic conflicts between modalities often characterize high-dimensional risks such as potential false declarations or disguised smuggling. On the other hand, existing systems treat all multimodal data in a "one-size-fits-all" manner, lacking the intelligent perception and on-demand scheduling capabilities for feature conflicts. This crude "alignment-driven everything" approach not only results in a significant waste of computing resources but also makes the system highly susceptible to underreporting or even failure of risk control defenses when facing new and concealed customs violations. Furthermore, because the local model training process is disconnected from the subsequent global federated aggregation stage, rare anomaly conflict features captured by individual edge nodes are often smoothed out as local noise by conventional parameter-weighted averaging strategies. This prevents them from being effectively integrated into the global network's defense loop, limiting their value release in cutting-edge applications such as intelligent forecasting and cross-border collaboration. In this context, if the traditional approach of forced feature alignment and full parameter averaging continues, even with hardware stacking to improve single-point computing power, the inherent performance bottlenecks of the strong alignment architecture make it difficult to guarantee processing efficiency while ensuring risk control accuracy.

[0025] This application breaks through the logical bottleneck of traditional forced feature alignment by introducing a collaborative processing mechanism that combines dual-stream feature extraction with commonality-deviation dual-track federated aggregation. By constructing a dual-stream architecture that includes feature alignment and conflict capture streams, it decouples regular data streams from abnormal risk logic, supports parallel responses to multi-dimensional features, and avoids the overwhelming of risk features. Based on real-time computation of modal deviation, it dynamically generates feature extraction strategies and combines them with an attention mechanism to amplify conflict features on demand, accurately focusing on potential violation areas and eliminating the resource waste of smooth processing of all data. It combines local processing results with final decision instructions to establish a negative deviation sample library and uses an active learning mechanism to screen high-value samples, significantly improving the throughput and convergence speed of model iteration. This not only reduces risk identification latency to the millisecond level and supports high-concurrency interaction, but also reduces the computational overhead of invalid parameter transmission through the dual-track federated aggregation mechanism. At the same time, event-driven abnormal pattern clustering provides a foundation for feedback loop, enabling the global abnormal pattern map to be dynamically integrated into the forecasting decisions of various ports, and promoting the improvement of the risk control network from local passive response to global active defense.

[0026] Example 1: Figure 1 is a flowchart of a method for automated processing of customs clearance documents based on multi-source data fusion, provided in Example 1 of the present invention. This example is applicable to the automation of customs clearance documents, and the method can be executed by a customs clearance document automation processing system based on multi-source data fusion. As shown in Figure 1, the method includes: S110. Collect multi-source heterogeneous data from customs clearance operations, and perform structured processing on the multi-source heterogeneous data to generate a multimodal information flow.

[0027] S120. Input the multimodal information stream into a pre-trained multimodal fusion model for cross-modal semantic analysis and output the local processing result. The multimodal fusion model includes a feature alignment stream and a conflict capture stream.

[0028] S130. Based on the local processing results, generate a set of multi-objective optimal decision schemes through reinforcement learning decision-making, and receive the final decision instruction generated after manual review, and execute the final decision instruction to complete the customs clearance business processing.

[0029] S140. Collect processing records containing the multimodal information stream, the local processing results, and the final decision instruction; filter high-value training samples through an active learning mechanism; and establish a negative bias sample library.

[0030] S150. Perform asynchronous federated aggregation based on the negative bias sample library to update the multimodal fusion model.

[0031] In this embodiment, a dual-stream architecture of feature alignment and conflict capture is introduced to logically decouple conventional data processing from anomaly risk feature capture. This allows the system to schedule data on demand, efficiently and smoothly fusing most risk-free modal-consistent data, while specifically extracting and amplifying semantic conflict features characterized by high deviations. This avoids the waste of computing resources caused by traditional one-size-fits-all forced alignment, precisely focusing computing power on potential risk points, thereby completing the assessment within millisecond latency and significantly improving the ability to detect covert violations such as false reports and concealment. This breaks down the conflict between performance and risk control at the architectural level. Furthermore, by constructing a negative deviation sample library and an asynchronous federated aggregation mechanism, locally captured anomaly patterns are clustered to form a global anomaly map and fed back to each node. This drives the entire risk control network to achieve a collaborative evolution from passive response to proactive defense, enabling new violations to be quickly transformed into network-wide shared defensive knowledge once discovered at a single point.

[0032] Example 2: The technical solution in this example is a further refinement based on the above examples.

[0033] In step S110, multi-source heterogeneous data from customs clearance operations are collected, and the multi-source heterogeneous data is processed in a structured manner to generate a multimodal information flow. The customs clearance process refers to the entire legal administrative and logistical operation process of declaring to customs, submitting documents, undergoing inspection, paying taxes and fees, and finally obtaining release permission when importing or exporting goods, articles, or means of transport. The multi-source heterogeneous data refers to a set of digital signals with different underlying encoding formats, spatial resolutions, and time sampling frequencies generated within a specific time window by physical sensors, electronic declaration terminals, and third-party logistics information systems distributed in different geographical locations. In storage media, it is represented as discrete text string sequences, continuous two-dimensional pixel matrices, and one-dimensional floating-point arrays with timestamps. The multi-source heterogeneous data of customs clearance involved in this invention includes, but is not limited to, customs declarations, accompanying document images, and logistics time-series data, the collection and use of which strictly comply with laws, regulations, and customs supervision provisions. The structured processing refers to the process of applying specific parsing engines and mathematical transformation matrices to raw binary data streams of different formats, transforming their irregularly arranged information units into standardized tensor formats with fixed dimensions, unified data types, and direct readability by deep learning frameworks. The multimodal information flow refers to a continuous data sequence that, after preliminary structuring, achieves strict alignment in timestamps and business serial numbers, and simultaneously contains multiple modal representations such as text feature vectors, image feature vectors, and time-series feature vectors. It is typically stored continuously in memory in the form of a multidimensional array.

[0034] Specifically, by binding to specific network ports and utilizing asynchronous non-blocking network communication protocols, it can monitor and receive message data from different business systems, such as customs declaration systems, scanning devices, and logistics platforms, in real time. Data is decoded from the physical link through protocol parsing and temporarily stored in a circular memory buffer to ensure data integrity and processing efficiency under high concurrency. All collected multi-source heterogeneous data is clearly divided into three types: structured customs declaration data, images of paper documents, and IoT time-series data, and then subjected to parallel and targeted vectorization processing. Structured customs declaration data refers to data entered and submitted by enterprises in standard field formats through systems such as electronic ports during the customs declaration process. Its essence is key-value pairs or tabular data with a fixed schema. Structured customs declaration data includes, but is not limited to, commodity number, commodity name, specifications, quantity, weight, value, country of origin, trade method, and consignor / consignee information. This data exists in the system in the form of database records or XML / JSON messages, and is naturally structured information that can be directly read by machines. Structured data from customs declarations can be directly converted into discrete feature vectors through field mapping.

[0035] Paper document images refer to image files of various accompanying documents, either in paper or scanned form, that need to be submitted during customs clearance. Essentially, they are unstructured pixel matrices. Paper document images include photographs or scans of commercial invoices, packing lists, certificates of origin, bills of lading, contracts, etc. These images contain crucial textual and layout information, but they need to be recognized to be converted into calculable data. Paper document images can be processed using OCR to convert the text in the image into a string. Then, natural language processing techniques, such as word embedding, are used to convert this into a text feature vector. This vector is then combined with a convolutional neural network (CNN) to analyze the image pixels, extracting visual features such as seals, signature positions, table structures, and company logos, forming a visual feature vector. Finally, the two are combined to represent all the information in the document image.

[0036] IoT time-series data refers to time-stamped sequential data continuously collected and reported by various sensing devices during logistics and transportation. Essentially, it is a one-dimensional or multi-dimensional signal that changes over time. IoT time-series data includes container GPS positioning trajectories, temperature and humidity sensor readings in cold chain transportation, vehicle vibration data, and container door opening and closing status records. This data reflects the real-time status and physical flow of goods. IoT time-series data can be used with models such as Long Short-Term Memory (LSTM) networks to extract dynamic features and transform them into time-series feature vectors.

[0037] After the above processing is completed, using the customs declaration number as the primary key, the generated feature vectors of different dimensions such as text, image, and time series are aligned in time and space, and then the aligned multi-dimensional feature vectors are concatenated. That is, address space is dynamically allocated in memory, and text feature vectors, image feature vectors, time series feature vectors, etc., are concatenated into a unified, continuous data sequence according to preset rules, such as timestamps and business serial numbers, which is the multimodal information stream. This multimodal information stream is a data entity that is strictly aligned in time and business logic, containing multiple modal (text, visual, time series) feature representations. It is usually stored in a high-speed shared memory area in multi-dimensional array (tensor) format for efficient reading by downstream edge fusion modules.

[0038] In this embodiment, by converting raw data in different formats, such as scattered customs declarations, paper images, and IoT time series, into unified and aligned standardized feature vectors in parallel, the predicament of fragmented multi-source data is effectively solved. This provides high-quality, directly computable structured input for downstream multimodal deep semantic fusion and intelligent risk assessment, and is the cornerstone for realizing fully automated processing.

[0039] In step S120, the multimodal information stream is input into a pre-trained multimodal fusion model for cross-modal semantic analysis, and the local processing result is output; wherein, the multimodal fusion model includes a feature alignment stream and a conflict capture stream, including: Calculate the modal deviation of each modal feature representation in the multimodal information stream within the shared latent space; When the modal deviation exceeds a preset first threshold, the conflict capture stream is activated; The conflict capture stream uses an attention mechanism combined with a local rule knowledge graph to perform preliminary risk assessment and output local processing results.

[0040] The multimodal fusion model can refer to a pre-trained deep learning model, whose core feature is its dual-stream architecture, comprising a "feature alignment stream" and a "conflict capture stream." This model is used for deep semantic analysis and understanding of the input multimodal information stream. The feature alignment stream can be a processing branch or data stream within the multimodal fusion model. Its main responsibility, like traditional multimodal models, is to learn and align the semantically consistent parts of different modalities, such as text and image data, for feature fusion and task inference in conventional, conflict-free scenarios. The conflict capture stream can be another processing branch in the multimodal fusion model, running parallel to the feature alignment stream. Its responsibility is specifically to handle data with semantic discrepancies or contradictions between modalities. When significant conflicts are detected, the conflict capture stream is activated, aiming to capture and amplify potential abnormal or risky features, rather than smoothing them. Local processing results can refer to the structured output of the multimodal fusion model, typically including a risk level assessment of the current customs clearance document, specific risk point alerts (e.g., suspected discrepancies between declared product name and image features), and preliminary inspection or review suggestions.

[0041] Modal feature representation refers to a mathematical vector representing the semantic information of different modalities such as text, images, and time series data after processing by a feature extraction network. A shared latent space is an abstract, high-dimensional mathematical space learned by the model. Raw data from different modalities (such as text, images, and time series) are mapped into this unified space after passing through their respective encoders. Within this space, data points from different modalities but with consistent semantics will be close to each other, enabling cross-modal semantic comparison and computation. Modal deviation refers to the final scalar value obtained after quantizing the distribution difference. This value directly reflects the degree of semantic inconsistency between specific modal pairs and is a key criterion for determining whether to activate the conflict capture stream. The first threshold can be a preset numerical boundary used to determine whether modal conflicts are significant. When the calculated modal deviation exceeds this threshold, it is determined that the current data has a noteworthy semantic conflict, thus triggering the anomaly handling process, i.e., activating the conflict capture stream. The attention mechanism specifically refers to the cross-attention mechanism, which allows the model to dynamically and selectively focus on different parts of the input features; it is used in the conflict capture stream to enhance attention to non-aligned conflict features and suppress irrelevant information.

[0042] A local rule knowledge graph can refer to a structured knowledge base stored locally on each edge node (such as a port server). It organizes a large number of entities related to customs operations in the form of a graph, such as commodities, enterprises, and regulatory clauses; and relationships, such as "belongs to" and "violates." It also includes feature patterns from historical violation cases, providing interpretable domain knowledge support for the risk assessment of the multimodal fusion model. Preliminary risk assessment refers to the initial risk evaluation process completed locally. The multimodal fusion model combines features learned from data and relevant knowledge retrieved from the local rule knowledge graph to generate a judgment result containing the probability of various risks (such as product name mismatch or price anomaly). Non-aligned conflict features refer to semantic contradictions or disagreements extracted from the multimodal information flow when the modal deviation exceeds a threshold. These contradictions exist between different modalities and cannot be smoothed out by conventional fusion logic. They are not predefined fixed features but rather abnormal data fragments dynamically marked based on real-time calculation results during data processing. For example, the text describes it as a chip, but the image shows it as plastic granules.

[0043] Specifically, the multimodal fusion model maps different modal features from the multimodal information flow to the same shared latent space through their respective fully connected layers or projection layers. In this space, the semantics of different modalities are expected to be comparable. The multimodal fusion model employs a bulldozer distance algorithm based on optimal transport theory to calculate the distribution difference between two feature vectors in the shared latent space. This algorithm calculates the minimum workload required to transform the distribution of one modality feature into the distribution of another. This workload, or distance, is the modality deviation, a scalar value that quantifies the degree of consistency between the two modalities in describing the same thing; a larger value indicates a more significant semantic conflict.

[0044] The calculated modality deviation is compared in real time with a pre-set first threshold. This threshold, established through extensive experimentation or business experience, distinguishes between acceptable modality differences and suspected semantic conflicts. If the modality deviation is lower than or equal to the pre-set first threshold, the feature alignment stream learns and aligns the semantically consistent parts of the different modal data. The aligned and fused multimodal features are then input into one or more fully connected layers or other subsequent networks for final classification or regression calculations, outputting structured local processing results.

[0045] If the modal deviation exceeds a first threshold, this invention will not smooth or force alignment of conflicting features as in traditional methods. Instead, it will specifically extract the data regions in the multimodal information stream that cause high deviations—for example, a product name field and its corresponding image region in a customs declaration, or a specific field in text—and mark them as non-aligned conflicting features. Simultaneously, a control signal is generated to activate the conflict capture stream's processing channel. That is, during the model's forward propagation, the relevant network layers of the conflict capture stream, such as the attention layer, will be given higher computational weights or fully activated, ready to process the aforementioned "non-aligned conflicting features." The processing results of the feature alignment stream may be downweighted or used only as background information.

[0046] The extracted non-aligned conflict features are used as the query vector (Q). Simultaneously, feature entities related to historical violation cases such as the current product, company, and flight route are retrieved from the local rule knowledge graph and used as the key vector (K) and value vector (V). The similarity between the query vector Q and all key vectors K is calculated, for example, by using a dot product, resulting in a set of similarity scores. These scores are normalized using a normalized exponential function to obtain a set of weight coefficients (attention weights). These weights reflect the correlation between the current conflict features and various historical violation patterns in the graph. The value vectors (i.e., historical violation features) are then weighted and summed using these weights to obtain an enhanced conflict feature representation. Essentially, this process focuses the model on the historical risk knowledge most relevant to the current conflict.

[0047] The resulting enhanced conflict feature representation integrates the original data features and relevant knowledge graph information. This representation is then fed into one or more fully connected layers for final analysis and classification. After processing by fully connected layers and other network components, the model outputs a structured, locally processed result. This result is typically a vector or list containing probability scores for various risk categories, such as classification risk, price risk, and security risk, along with text descriptions generated based on these scores and knowledge graph associations, such as "Please carefully verify the consistency between the certificate of origin and invoice information," and suggested verification methods, such as "Suggest opening the package for inspection."

[0048] In this embodiment, a dual-stream architecture and modal deviation detection are used to accurately identify and extract non-aligned conflict features, such as document discrepancies, which are traditionally used as noise filters. Then, an attention mechanism combined with a local knowledge graph is used for enhanced analysis. This effectively solves the problem of risk feature overload caused by forced feature alignment, significantly improving the ability to detect and assess hidden and novel customs clearance violations.

[0049] Optionally, calculating the modal deviation of each modal feature representation in the multimodal information stream within the shared latent space includes: The text modality feature vectors, image modality feature vectors, and temporal feature vectors in the multimodal information stream are respectively mapped to the shared latent space; The bulldozer distance algorithm based on optimal transmission theory is used to calculate the distribution differences between the text modal feature vector and the image modal feature vector, the text modal feature vector and the temporal feature vector, and the image modal feature vector and the temporal feature vector in the shared latent space, respectively, and the distribution differences are used as the modal deviation.

[0050] Among them, text modal feature vectors refer to high-dimensional numerical vectors that can represent the semantics of text, extracted from structured data such as customs declarations and invoices through natural language processing models such as BERT and word embeddings. Image modal feature vectors refer to visual vectors that can represent the visual content of images, such as text layout, extracted from images of paper documents such as packing lists and certificates of origin through visual models such as convolutional neural networks. High-dimensional numerical vectors representing information related to seals, logos, and page layout. Temporal feature vectors refer to high-dimensional numerical vectors extracted from time-series data such as IoT devices and logistics trajectories using time-series models (such as LSTM) to characterize dynamic changes. Optimal transport theory refers to the mathematical framework used to calculate the minimum conversion cost between two probability distributions. Earth Mover's Distance (EMD) algorithm. The bulldozer distance, a specific metric in optimal transport theory, calculates the minimum amount of work required to transform one distribution into another. Distributional differences arise in the shared latent space from the inconsistencies between distributions represented by different sets of modal eigenvectors. The value calculated by the bulldozer distance algorithm is its quantification.

[0051] Specifically, during the training phase of the multimodal fusion model, the system trains a separate encoder network for each of the three modalities: a text encoder, typically using structures such as Transformer or BERT, encodes text modality feature vectors into deep semantic vectors; an image encoder, typically using Convolutional Neural Networks (CNNs) or Vision Transformers, encodes image modality feature vectors into deep visual semantic vectors; and a temporal encoder, typically using Long Short-Term Memory (LSTM), Transformers, or temporal convolutional networks, encodes temporal feature vectors into semantic vectors representing dynamic patterns. During training, a joint loss function (such as contrastive loss) is designed to encourage these three encoders to output semantically related multimodal data, such as consistent product text, images, and logistics temperature records, to adjacent positions in a shared latent space. When processing new data, the preprocessed feature vectors of the three modalities are input in parallel into their respective trained encoders.

[0052] Each encoder acts as a fixed function, mapping the input to a shared latent space and outputting a comparable vector representation within that common space.

[0053] For a pair of modalities to be computed, such as text A and image B, their corresponding vector sets are obtained from the shared latent space, treated as two distributions. The Euclidean distance or other distances between all pairwise sample points in the two distributions are calculated to form a cost matrix. This is achieved by calling a library that implements the bulldozer distance algorithm, such as linear programming or approximate algorithms like Sinkhorn iteration. The core of this algorithm is solving an optimization problem: finding a transportation scheme that minimizes the total cost of transforming the text distribution into the image distribution. This is the weighted sum of the cost matrices. The minimum total cost is the bulldozer distance, and this distance value is used as the modal deviation between the two modes.

[0054] The distributional differences between different modal pairs have different business implications. For example, the text-image modal pair is used to detect whether the text description on the customs declaration is consistent with the content of the accompanying document image. For example, the text declares a branded mobile phone, but the image recognition shows an unbranded white-box communication device. Specifically, the calculation process involves substituting the vector sets output by the text encoder and the image encoder into the Bulldozer distance algorithm. The text-time sequence modal pair is used to detect whether the text declaration information is consistent with the logistics sensor data in terms of time logic. For example, the text declares perishable goods transported at a constant temperature; however, time sequence sensor data shows prolonged abnormal temperature fluctuations during transportation. Specifically, the calculation process involves substituting the vector sets output by the text encoder and the time sequence encoder into the Bulldozer distance algorithm. This measures the difference between the descriptive commitment and the actual dynamic record. The image-time sequence modal pair is used to detect whether the visual state of the goods contradicts the sensor records. For example, the image shows sealed packaging, but time sequence humidity sensor data shows extremely high ambient humidity before loading, indicating a potential risk of moisture damage. The calculation process involves substituting the vector sets output by the image encoder and the temporal encoder into the bulldozer distance algorithm. This measures the difference between the visual representation and the physical environment recording.

[0055] In this embodiment, heterogeneous data is mapped to a commensurable space through three independent encoders, and then a unified bulldozer distance tool is used to quantify the degree of semantic divergence between any two modalities, providing quantitative and multi-dimensional input signals for subsequent threshold-based conflict detection, attention amplification, and risk assessment.

[0056] Optionally, the conflict capture stream performs preliminary risk assessment using an attention mechanism combined with a local rule knowledge graph, and outputs local processing results, including: Construct a cross-attention matrix, using the non-aligned conflict features in the multimodal information flow as the query vector, and the historical violation feature entities in the local rule knowledge graph as the key vector and value vector; Calculate the similarity score between the query vector and the key vector, and normalize the similarity score using a normalized exponential function; The normalized similarity score is weighted and summed with the value vector to obtain the enhanced conflict feature representation. This is then combined with the local rule knowledge graph to perform a preliminary risk assessment and output the local processing result.

[0057] The cross-attention matrix refers to a specific structure used to implement the attention mechanism. In this matrix, non-aligned conflict features are used as query vectors, and historical violation entities from the local rule knowledge graph are used as key and value vectors. Conflict features are associated and enhanced by calculating similarity. The three basic components of the attention mechanism are the query vector, key vector, and value vector. The query vector represents the content that needs attention, i.e., the conflict feature; the key vector represents the matching entries in the knowledge base, i.e., historical violation features; and the value vector is the information content corresponding to the key vector. By calculating the similarity between the query and the key, the values ​​are weighted and summed to obtain the enhancement information most relevant to the query. The normalized exponential function (Softmax) is a mathematical function that transforms a set of values, such as similarity scores, into a probability distribution, such that all output values ​​are between 0 and 1 and sum to 1. It is used to convert the similarity scores between the query and each key into attention weights. Prediction entropy is a metric that measures the uncertainty of a model's prediction; the higher the entropy, the more uncertain the model's judgment of the current input.

[0058] Specifically, labeled non-aligned conflict features are extracted from the conflict capture stream, such as a feature vector indicating a significant discrepancy between the declared weight and the visual volume. This feature vector is transformed into a query vector that meets the dimensional requirements of attention calculation through a trainable linear projection layer. Based on the basic information of the current document, such as the product code and country of origin, a set of related historical violation feature entities is retrieved, such as historical case 1: the product name is plastic granules, but the image is of a metal ingot; historical case 2: the price is 30% lower than that of similar products in the same period. These entities have been pre-encoded as vectors when constructing the knowledge graph. These entity vectors can be directly used as key vectors and value vectors, respectively. Typically, the key and value can be the same, or the value vector can be obtained through another projection layer. These constitute the three core inputs required for cross-attention matrix calculation: Q (query), K (key), and V (value).

[0059] Perform matrix operations, multiplying the query vector Q (shape [1, d]) and the key vector matrix K (shape [n, d], where n is the number of historical violations) using a dot product or other similarity calculations, to obtain a vector containing n scores, i.e., the similarity scores. Each score represents the initial association strength between the current conflict and a certain historical violation pattern. Input the n similarity scores obtained in the previous step into a normalized exponential function. Assume the scores are [ , , ..., ], Softmax is calculated as: attention weights Where i = 1, 2, 3, ..., n; This represents the similarity score between the query vector and the key vector of the i-th historical violation entity. This represents the attention weight assigned to the i-th historically violating entity. exp( ) indicates that the score index of the current i-th entity is calculated. Σ(exp( )) represents summing the score indices of all entities (j iterates from 1 to n). a=[ , ,..., ],and After this processing, the n scores are transformed into a set of attention weights that sum to 1. , ,..., The weight of the largest value is [ ]. The corresponding historical violation patterns are considered the most relevant patterns for explaining the current conflict.

[0060] The normalized attention weights [ , ,..., ] as coefficients, respectively corresponding to each vector in the value vector matrix V. , ,..., Multiply the results and then sum them to obtain the enhanced conflict feature representation. ;in, This represents the value vector corresponding to the i-th historical violation entity. The result is a comprehensive feature vector that integrates the current specific conflict context and the most relevant historical knowledge. The enhanced conflict feature representation is used for initial risk assessment. Further, this enhanced feature may be tensor-concatenated with the original entity information retrieved from the knowledge graph to form a richer feature. This feature is then input into subsequent fully connected neural network layers for classification or regression calculations. The model outputs a structured assessment conclusion, i.e., the local processing result. This is typically a multi-dimensional vector that may contain: the probability of various risk labels, such as "Classification error: 0.85", "False price: 0.10"; specific review prompts, such as "The conflict feature is highly similar to historical 'false product name' cases; it is recommended to focus on verifying the physical item"; risk level and inspection suggestions, etc.

[0061] In this embodiment, the cross-attention mechanism is used to accurately associate and compare the currently captured modal semantic conflicts, i.e., non-aligned features, with historical violation patterns in the knowledge base. This enables the system not only to identify existing contradictions, but also to understand which known violations the contradictions are similar to, thereby outputting more interpretable and targeted risk assessment results, which greatly improves the accuracy of primary risk assessment and decision support.

[0062] In step S130, based on the local processing results, a multi-objective optimal decision set is generated through reinforcement learning decision-making, and a final decision instruction generated after manual review is received. The final decision instruction is then executed to complete the customs clearance process, including: A Markov decision process model is constructed, in which the local processing result is defined as the state space, and customs clearance, manual inspection, and supplementary declaration are defined as the action space. Design a multi-objective reward function, which includes a customs clearance efficiency reward item, a tax security reward item, and a risk prevention and control reward item; The Markov decision process model is iteratively solved using a proximal policy optimization algorithm, and the set of multi-objective optimal decision schemes that maximize the cumulative expected reward in the current state is output. Receive the final decision instruction generated after manual review, and execute the final decision instruction to complete the customs clearance process.

[0063] In this context, reinforcement learning decision-making can refer to a core component of the system. It uses an intelligent agent, specifically reinforcement learning decision-making, to interactively learn from the customs clearance business environment and make sequential decisions. Its goal is to learn to generate decision schemes that balance multiple, sometimes conflicting, business objectives. The multi-objective optimal decision scheme set can refer to a set of structured decision suggestions output by the reinforcement learning decision-making module. This set, calculated by the policy network in the current state space, represents one or more actions that maximize the long-term cumulative expected reward, such as release, inspection, and their related assessment values. This scheme set is provided to human auditors as decision support. The final decision instruction can refer to the operation command ultimately selected and issued by the human auditor through a human-computer interaction interface after reviewing the generated multi-objective optimal decision scheme set, combining their own experience and judgment. For example, it could be to execute an unpacking inspection. This instruction carries a digital signature and serves as the final authoritative basis for driving the business system's execution. The Markov decision process model can refer to a standard mathematical framework for formalizing sequential decision problems, containing core elements such as state space, action space, state transition probabilities, and reward functions, used to rigorously model the decision-making process of customs clearance and document review.

[0064] In a Markov decision process model, the state space refers to the set of all possible environmental situations that an agent can perceive. The local processing results, i.e., the structured vectors containing various risk probabilities, document review prompts, etc., are defined as the state space, which is the complete information summary upon which the current decision is based. The action space, in a Markov decision process model, refers to the set of all possible actions that an agent can take in any given state. This space is defined as three discrete actions: clearance, manual inspection, and supplementary declaration. The multi-objective reward function refers to the core mathematical function used to quantitatively evaluate the quality of the agent's decisions. Because it needs to simultaneously optimize three objectives—clearance efficiency, tax security, and risk control—it is called multi-objective. The scalar value output by the function is the reward, and the fundamental learning objective of the agent is to maximize the long-term cumulative sum of all rewards obtained from the environment.

[0065] Customs clearance efficiency rewards can refer to a component of a multi-objective reward function, designed to incentivize rapid processing and improved port throughput. For example, making a release decision may yield a positive reward, while a decision that causes prolonged process delays will result in a negative reward. Tax security rewards can refer to a component of a multi-objective reward function. For example, successfully identifying and intercepting a price underreporting incident and collecting back taxes will yield a high positive reward. Risk prevention rewards can refer to a component of a multi-objective reward function, designed to prevent security risks such as smuggling and the entry of prohibited or restricted goods. For example, making an inspection decision on high-risk goods and successfully identifying the problem will yield a high reward. Proximal policy optimization algorithms can refer to advanced and stable reinforcement learning algorithms, particularly suitable for training complex policy networks. PPO, through cleverly designed objective functions, limits the magnitude of policy updates in each iteration, thereby ensuring the stability and efficiency of the training process, making it very suitable for high-dimensional, continuous decision problems such as customs clearance decisions. Cumulative expected reward can refer to the expected value obtained by summing all possible rewards that an agent can obtain from the current state, considering all possible future state development paths. Maximizing this value means pursuing optimal long-term, global performance, rather than short-term gains.

[0066] Specifically, the system receives the local processing results. These results are semantically rich structured vectors, potentially containing: [probability of risk A, probability of risk B, risk level, commodity coding features, enterprise credit score, etc.]. The constructor directly, or after standardization and dimensionality reduction, defines this vector as the state space S of the MDP model. Therefore, the state st ∈ S at any given time completely encapsulates all known information and risk assessment conclusions for the current document review process. Based on the core customs supervision process, the action space A is explicitly defined as three discrete, executable basic operations: a1: customs clearance, a2: manual inspection, and a3: supplementary declaration. The agent (i.e., the policy network) is responsible for selecting one action from this action space to execute when it observes state st.

[0067] The reward for customs clearance efficiency is negatively correlated with processing time. For example, Re = -α * processing time. Choosing "release" usually takes the shortest time and yields a relatively high reward; choosing "manual inspection" introduces significant delays and results in a negative reward. The reward for tax security is positively correlated with tax preservation effectiveness. For example, if the agent's decision, such as inspection, ultimately leads to the collection of supplementary tax T, then Rt = +β * T; if the decision, such as mistakenly releasing the goods, results in a large negative reward. The reward for risk prevention is positively correlated with the severity of the identified risk. For example, if the inspection action actually uncovers prohibited or restricted items or major violations, with a risk level of L, then Rr = +γ * L; if high-risk goods are mistakenly released, a large negative reward is obtained.

[0068] The final immediate reward is Rto = Re + Rt + Rr. Here, α, β, and γ are hyperparameters used to fine-tune the relative weights of the three objectives: efficiency, taxation, and security. The agent learns how to take actions to maximize the cumulative expected reward of future Rto by interacting with the environment or utilizing historical data. Before system deployment, a simulated environment is constructed using historical interaction data, such as states, actions, next states, and rewards obtained, and a proximal policy optimization (PPO) algorithm is run. PPO iterates through multiple iterations, allowing the agent to try different policies, collecting interaction data, and then updating the policy network (deep neural network) parameters by optimizing an alternative objective function with confidence region constraints. This process is repeated continuously, causing the policy to gradually tend towards selecting action sequences that yield higher cumulative expected rewards, until the policy network performance converges.

[0069] During online execution, the pre-trained policy network is loaded. When the state space builder inputs the current state st (i.e., the local processing result), the policy network inferencer performs forward computation, outputting the probability distribution π(ai|st) of each action ai in the action space A. Typically, the action with the highest probability is selected as the primary recommendation. Simultaneously, the value function (Q-value) or advantage function for each action can be calculated to evaluate its expected value. Finally, the system encapsulates the recommended actions, their probabilities, and value assessments into a structured set of multi-objective optimal decision solutions, which is then pushed to the front-end review interface via an application programming interface (such as a REST API) as an intelligent aid to customs officers' decision-making.

[0070] The instruction routing gateway, acting as a microservice gateway, continuously listens to the message bus or specific API endpoints. Once the front-end auditor makes a final decision and confirms delivery, a final decision instruction message containing a digital signature and operator identification information arrives. The gateway first verifies the signature to ensure the instruction is complete, tamper-proof, and of reliable origin; then it performs authorization verification to confirm that the operator is authorized to issue such instructions for the goods in this invoice. After successful verification, the gateway translates the business-level instruction—for example, actions like "manual inspection," and parameters like "inspection type - open box," and "inspection focus - verify product name"—into standardized control commands or messages that downstream business execution systems, such as intelligent inspection systems, checkpoint release systems, and customs declaration status management systems, can recognize. Subsequently, through the Enterprise Service Bus (ESB), a messaging infrastructure for system integration, these control commands are accurately routed and distributed to the appropriate target systems.

[0071] The target business system executes the received commands, driving field equipment such as illuminating inspection lights, opening gates, or updating business status, such as suspending customs declarations or sending supplementary material notifications, thereby completing the customs clearance operation in both the physical world and business logic. The complete record of this decision-making and execution is saved for subsequent auditing and may be included as a high-value sample in the negative bias sample library, participating in the continuous evolution of the model.

[0072] In this embodiment, by constructing a multi-objective Markov decision process and using proximal strategy optimization to solve the problem, the experience-dependent customs clearance decision is transformed into a quantifiable and optimizable sequence learning problem. This allows for the automatic generation of an optimal decision set that balances efficiency, taxation, and risk, providing precise intelligent assistance for manual review. Ultimately, while ensuring the effectiveness of supervision, the scientific nature and consistency of the overall customs clearance decision are significantly improved.

[0073] In step S140, processing records containing the multimodal information stream, the local processing results, and the final decision instruction are collected. High-value training samples are screened through an active learning mechanism to establish a negative bias sample library, including: Collect processing records containing the multimodal information stream, the local processing results, and the final decision instruction; Extract records from the processing records that have a classification difference from the final decision instruction as first candidate samples; Records containing the non-alignment conflict features and causing the prediction entropy value output by the multimodal fusion model to be greater than a preset uncertainty threshold are extracted as second candidate samples. The first candidate sample and the second candidate sample are deduplicated and their features are concatenated, and then stored in the negative bias sample library.

[0074] The processing records refer to a set of data generated during a complete customs clearance process, exhibiting strict causal and temporal correlation. This includes multimodal information flow (original features), local processing results (the model's initial assessment), and final decision instructions (human final review). These constitute the original data source for model evolution. Classification discrepancies refer to the inconsistencies between the model's risk assessment conclusions in the local processing results (e.g., "low risk, recommend release") and the actual business conclusions represented by the final decision instructions (e.g., "human instruction: open for inspection"). These discrepancies are key signals for identifying model errors or deficiencies. First candidate samples refer to the training samples selected based on the aforementioned classification discrepancy criteria. These are case records where the machine made a mistake but it was corrected manually, possessing high corrective value. These features themselves may indicate potential risks. Prediction entropy is a quantitative indicator derived from information theory, used to measure the uncertainty of model predictions. When the model's judgment of the input data is ambiguous or lacks confidence (e.g., the probabilities of judging high and low risk are not equal), it indicates a lack of confidence. Nearly 50% of the time, the output prediction entropy value will be very high. The uncertainty threshold can refer to a preset scalar value used to judge whether the model's prediction is sufficiently uncertain. When the prediction entropy value is greater than this threshold, the model's understanding of the sample is considered ambiguous and confused. Secondary candidate samples are training samples selected based on both the non-aligned conflict features and the prediction entropy value. These are case records that inherently possess contradictory features and confuse the model, possessing high exploration and learning value. The negative bias sample library can refer to a local database specifically used to store high-value training samples. Negative bias refers to storing samples where the model's judgment is biased (erroneous or uncertain), with the aim of specifically strengthening the model's performance in its weak areas.

[0075] Specifically, it continuously runs and subscribes to specific event messages on the enterprise service bus related to the completion of customs clearance business, such as the completion of the document review process. Once such an event is detected, the process is triggered. Based on the business transaction number carried in the event, it retrieves the time-aligned multimodal information stream, local processing results, and final decision instructions corresponding to the business from intermediate storage such as high-speed shared memory and temporary databases, and binds them into a complete processing record.

[0076] The active learning evaluator reads processing records, analyzes risk classification conclusions (e.g., low risk) from local processing results, and compares them with final decision instructions (e.g., manual inspection). If the machine suggests release but manual inspection is required, or the machine suggests inspection but manual release is initiated directly, a classification discrepancy is identified. These records are extracted as first candidate samples, which clearly reveal the gap between the model's current judgment and actual business needs.

[0077] The active learning evaluator simultaneously checks two pieces of information: whether the processed record generated misaligned conflict features; and the entropy value of the output predicted probability distribution when processing the record. The evaluator calculates the entropy value of the prediction, for example, using the formula H(p) = -Σpi log (pi), where pi is the model's predicted probability for each risk category. The calculated predicted entropy value is then compared to a pre-set uncertainty threshold. Only when the processed record simultaneously meets both conditions—"contains misaligned conflict features" and "its predicted entropy value is greater than the uncertainty threshold"—is it extracted as a second candidate sample. This type of sample represents difficult cases with inherent contradictions that the current model cannot accurately handle.

[0078] The union of the first and second candidate sample sets is taken, and then compared based on key features of the samples, such as the hash value of the feature vector, to remove completely duplicate samples. For each deduplicated sample, i.e., a processing record, its original multimodal information stream as input features, local processing results as intermediate features or auxiliary labels, and the final decision instruction as the feature vector corresponding to the final target supervision signal of that sample are concatenated in a fixed order along the feature dimension to form a unified whole sample that can be used for model retraining. This concatenated complete sample is serialized into binary format and stored in a local dedicated negative bias sample library through a database operation interface.

[0079] In this embodiment, a high-value sample library is constructed by accurately screening high-value samples that are misjudged or confused by the machine through an active learning mechanism. The results of manual review and model uncertainty are transformed into targeted optimization signals, thereby providing high-quality and targeted training data for subsequent federated learning. This drives the model to continuously focus on its own weaknesses and new risk patterns, realizing a key shift from passive processing to active evolution.

[0080] In step S150, asynchronous federated aggregation is performed based on the negative bias sample library to update the multimodal fusion model, including: Based on the negative deviation sample library, asynchronous federated aggregation is performed periodically to obtain the aggregated global model and global anomaly pattern map; The multimodal fusion model is updated based on the global model and the global anomaly pattern map.

[0081] Asynchronous federated aggregation can refer to a federated learning model update strategy. Asynchronous means that after each edge node completes its local multimodal fusion training, it can upload updates to the central node at any time without waiting for other nodes, thus improving the overall system efficiency and flexibility. The aggregated global model can refer to the baseline model maintained by the central layer. It is generated by weighted aggregation of the basic parameter updates of the multimodal fusion model uploaded by each edge node and trained based on regular aligned samples, incorporating the network's consensus knowledge on routine business. It represents a baseline model that integrates the consensus knowledge of all participating nodes on routine business, used to improve and standardize the accuracy and stability of each edge node when processing ordinary customs clearance business.

[0082] A global anomaly pattern graph can refer to a structured risk knowledge base generated and maintained by the central layer of the system, such as a cloud server. It does not store raw data, but rather a collection of knowledge representing common new risk patterns across the entire network, formed by specifically aggregating and abstracting anomaly feature information uploaded by edge nodes, such as customs ports. When each edge node trains its model using its local negative bias sample library, it extracts anomaly gradient vectors that reflect the specificity of the data. These gradient vectors are then uploaded to the central layer, where clustering algorithms (such as DBSCAN) are used to analyze them, grouping similar gradient vectors into the same category. The center of each cluster is abstracted and defined as a global anomaly pattern. All these anomaly patterns and their associated information are structured and organized to form the global anomaly pattern graph. The information encapsulated in the graph includes at least: anomaly feature prototypes, associated knowledge, and pattern labels. An anomaly feature prototype is a risk feature pattern that repeatedly appears in different nodes and documents, such as a specific vector representation of "the declared product name is A, but the image feature points to B". Associated knowledge can refer to the characteristics, risk category labels, and risk levels of historical violations related to the anomaly pattern. For example, risk category labels could include misreporting product names, price concealment, and risk levels. Pattern labels are identifiers defined for each common pattern, facilitating their reference and distribution within the system. This global anomaly pattern map is typically distributed periodically to each edge node as a negative constraint for knowledge distillation, guiding each node to update its local multimodal fusion model. This allows a newly identified violation at a port to be quickly transformed into a risk pattern that can be recognized by all port models, greatly enhancing the foresight and adaptability of the entire risk control network.

[0083] Specifically, asynchronous federated aggregation can employ a commonality-deviation dual-track mechanism. The system triggers the aggregation process according to a preset cycle, for example, daily. Each edge node, i.e., the local edge computing server deployed at various customs ports, works in parallel within this cycle: It trains a local multimodal fusion model using regular data to obtain updates to the basic parameters of the multimodal fusion model, corresponding to commonality knowledge. It also trains the multimodal fusion model using a local negative deviation sample library and extracts anomalous gradient vectors generated during training that reflect the specificity of the data, corresponding to deviation knowledge. Each node asynchronously uploads both types of data to the central server, i.e., the specific hardware or cloud service carrier of the central layer functions, through a secure channel. The upload channel is divided into two logical tracks: a commonality aggregation track, which receives updates to the basic model parameters from each edge node; and a deviation aggregation track, which receives anomalous gradient vectors from each edge node.

[0084] The central layer, i.e., the central cloud server or central server cluster, employs a data-volume-based weighted average algorithm, FedAvg. When each edge node uploads updates to the basic parameters of its multimodal fusion model trained on regular data, it must simultaneously report the effective data volume or number of samples used in this round of local training. The central layer uses the proportion of each node's effective data volume to the total reported data volume as its weight. A weighted average is applied to all parameter updates to obtain a new, optimized set of multimodal fusion model parameters, i.e., the aggregated global model. This process enhances the model's commonality capabilities. The central layer collects all anomalous gradient vectors, high-dimensional vectors carrying locally discovered anomalous signals. Cluster analysis is performed on all anomalous gradient vectors, for example, using the DBSCAN algorithm. Similar anomalous gradients are grouped into the same cluster, with each cluster center representing a recurring anomalous pattern across the network. The cluster centers and their associated feature information are structured to generate a global anomalous pattern map. This map systematically answers the question, "What types of new risks currently exist?"

[0085] The global collaboration module client, deployed at various edge nodes such as port servers, synchronously receives data packets from the central aggregation server via a content delivery network or dedicated line. This data packet contains two core parts: an aggregated global model, i.e., a new set of neural network parameter files; and a global anomaly pattern graph, a structured knowledge representation file. The client performs integrity checks and hash signature verification on the received data packets to ensure that the data has not been tampered with during transmission. After successful verification, the system logically decouples the two components for subsequent different update processes.

[0086] Edge nodes receive the parameter file of the aggregated global model and use it to completely replace or initialize their local multimodal fusion model. This involves loading the new parameter file into the model's inference engine, overwriting the old parameters, and enabling the local multimodal fusion model to instantly acquire the latest general knowledge that incorporates the experience of all nodes. This directly improves its accuracy and robustness when processing most routine aligned samples. It solves the data silo problem between different data points and enables rapid sharing of common knowledge.

[0087] Edge nodes incorporate the received global anomaly pattern map as a negative constraint term for knowledge distillation into the training objective function of the local multimodal fusion model. This results in the construction of the fusion loss function: ; in, For the total loss, L t It is the original task loss, such as classification loss; L d λ is the distillation loss associated with the anomalous spectrum, and λ is a hyperparameter controlling the constraint strength. dThe core idea is to teach the multimodal fusion model to be alert to anomalous patterns defined in the graph. For example, if the feature f extracted by the multimodal fusion model has a high similarity to a prototype p of an anomalous pattern in the graph, and the final output of the multimodal fusion model has a high probability of being "risk-free," then L... d It will output a large penalty value, forcing the model parameters to be updated, thereby learning to make high-risk judgments for such features.

[0088] Edge nodes use a local negative bias sample library, with L t To optimize the objective, the model with completed parameter replacements undergoes several rounds of fine-tuning training. This process is known as guided updates, which infuses the neural network with abstracted anomaly knowledge from the graph.

[0089] In this embodiment, by establishing a negative deviation sample library and a dual-track federated aggregation mechanism, the local and rare abnormal risk features captured by each edge node (rather than being discarded as noise) are specifically aggregated and clustered to form a globally shareable abnormal pattern map. This solves the problem of long-tail abnormal pattern loss caused by the traditional federated averaging method, drives the global network to quickly identify and warn of new and hidden violation patterns, and realizes the collaborative evolution of risk control capabilities from local discovery to global defense.

[0090] Example 3: Figure 2 is a framework diagram of a customs clearance document automation processing system based on multi-source data fusion provided in Example 3 of the present invention. The system is used to execute the customs clearance document automation processing method based on multi-source data fusion described in any embodiment of the present invention. Figure 2 As shown, the system includes: The data acquisition module 210 is used to acquire multi-source heterogeneous data in customs clearance operations and perform structured processing on the multi-source heterogeneous data to generate a multimodal information stream. The output module 220 is used to input the multimodal information stream into a pre-trained multimodal fusion model for cross-modal semantic analysis and output the local processing results. The multimodal fusion model includes a feature alignment stream and a conflict capture stream. The execution module 230 is used to generate a set of multi-objective optimal decision schemes through reinforcement learning based on the local processing results, and to receive the final decision instruction generated after manual review, and execute the final decision instruction to complete the customs clearance business processing; The filtering module 240 is used to collect processing records containing the multimodal information stream, the local processing results and the final decision instruction, and to filter high-value training samples through an active learning mechanism to establish a negative bias sample library. The update module 250 is used to perform asynchronous federated aggregation based on the negative bias sample library to update the multimodal fusion model.

[0091] The automated customs clearance document processing system based on multi-source data fusion provided in this embodiment of the invention can execute the automated customs clearance document processing method based on multi-source data fusion provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0092] Example 4: Figure 3 shows a schematic diagram of an electronic device that can be used to implement an embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0093] like Figure 3 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor 11 to enable the at least one processor 11 to perform the method provided by the present invention.

[0094] The processor 11 can perform various appropriate actions and processes based on a computer program stored in the read-only memory (ROM) 12 or a computer program loaded from the storage unit 18 into the random access memory (RAM) 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0095] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0096] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as a method for automating customs clearance document processing based on multi-source data fusion.

[0097] In some embodiments, the automated customs clearance document processing method based on multi-source data fusion can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the automated customs clearance document processing method based on multi-source data fusion described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the automated customs clearance document processing method based on multi-source data fusion by any other suitable means (e.g., by means of firmware).

[0098] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a memory system, at least one input device, and at least one output device, and transferring data and instructions to the memory system, the at least one input device, and the at least one output device.

[0099] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0100] In the context of this invention, a computer-readable storage medium stores computer instructions that, when executed by a processor, implement the automated customs clearance document processing method based on multi-source data fusion provided by this invention. The computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0101] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD monitor)) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0102] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0103] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0104] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0105] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for automated processing of customs clearance documents based on multi-source data fusion, characterized in that, include: Collect multi-source heterogeneous data from customs clearance operations, and perform structured processing on the multi-source heterogeneous data to generate a multimodal information flow; The multimodal information stream is input into a pre-trained multimodal fusion model for cross-modal semantic analysis, and the local processing result is output. The multimodal fusion model includes a feature alignment stream and a conflict capture stream. Based on the local processing results, a set of multi-objective optimal decision schemes is generated through reinforcement learning decision-making, and the final decision instruction generated after manual review is received and executed to complete the customs clearance business processing. Collect processing records containing the multimodal information stream, the local processing results, and the final decision instruction; use an active learning mechanism to filter high-value training samples and establish a negative bias sample library. Asynchronous federated aggregation is performed based on the negative bias sample library to update the multimodal fusion model.

2. The method according to claim 1, characterized in that, The multimodal information stream is input into a pre-trained multimodal fusion model for cross-modal semantic analysis, and the local processing results are output, including: Calculate the modal deviation of each modal feature representation in the multimodal information stream within the shared latent space; When the modal deviation is lower than or equal to a preset first threshold, the local processing result is output through the feature alignment stream; When the modal deviation is higher than the preset first threshold, the conflict capture stream is activated; The conflict capture stream uses an attention mechanism combined with a local rule knowledge graph to perform preliminary risk assessment and output local processing results.

3. The method according to claim 2, characterized in that, Calculating the modal deviation of each modal feature representation in the multimodal information stream within the shared latent space includes: The text modality feature vectors, image modality feature vectors, and temporal feature vectors in the multimodal information stream are respectively mapped to the shared latent space; The bulldozer distance algorithm based on optimal transmission theory is used to calculate the distribution differences between the text modal feature vector and the image modal feature vector, the text modal feature vector and the temporal feature vector, and the image modal feature vector and the temporal feature vector in the shared latent space, respectively, and the distribution differences are used as the modal deviation.

4. The method according to claim 2, characterized in that, The conflict capture stream uses an attention mechanism combined with a local rule knowledge graph to perform preliminary risk assessment and outputs local processing results, including: Construct a cross-attention matrix, using the non-aligned conflict features in the multimodal information flow as the query vector, and the historical violation feature entities in the local rule knowledge graph as the key vector and value vector; Calculate the similarity score between the query vector and the key vector, and normalize the similarity score using a normalized exponential function; The normalized similarity score is weighted and summed with the value vector to obtain the enhanced conflict feature representation. This is then combined with the local rule knowledge graph to perform a preliminary risk assessment and output the local processing result.

5. The method according to claim 1, characterized in that, The process of generating a multi-objective optimal decision set based on the local processing results through reinforcement learning, receiving a final decision instruction generated after manual review, and executing the final decision instruction to complete the customs clearance process includes: A Markov decision process model is constructed, in which the local processing result is defined as the state space, and customs clearance, manual inspection, and supplementary declaration are defined as the action space. Design a multi-objective reward function, which includes a customs clearance efficiency reward, a tax security reward, and a risk prevention and control reward. The Markov decision process model is iteratively solved using a proximal policy optimization algorithm, and the set of multi-objective optimal decision schemes that maximize the cumulative expected reward in the current state is output. Receive the final decision instruction generated after manual review, and execute the final decision instruction to complete the customs clearance process.

6. The method according to claim 4, characterized in that, The collection includes the processing records containing the multimodal information stream, the local processing results, and the final decision instruction. High-value training samples are selected through an active learning mechanism to establish a negative bias sample library, including: Collect processing records containing the multimodal information stream, the local processing results, and the final decision instruction; Extract the records in the processing records that have a classification difference from the final decision instruction as the first candidate samples; Records containing non-aligned conflict features and causing the predicted entropy value output by the multimodal fusion model to be greater than a preset uncertainty threshold are extracted as second candidate samples. The first candidate sample and the second candidate sample are deduplicated and their features are concatenated, and then stored in the negative bias sample library.

7. The method according to claim 1, characterized in that, The step of performing asynchronous federated aggregation based on the negative bias sample library to update the multimodal fusion model includes: Based on the negative deviation sample library, asynchronous federated aggregation is performed periodically to obtain the aggregated global model and global anomaly pattern map; The multimodal fusion model is updated based on the global model and the global anomaly pattern map.

8. A customs clearance document automation processing system based on multi-source data fusion, characterized in that, The system is used to execute the automated customs clearance document processing method based on multi-source data fusion as described in any one of claims 1-7, including: The data acquisition module is used to collect multi-source heterogeneous data in customs clearance operations and to perform structured processing on the multi-source heterogeneous data to generate a multimodal information stream. The output module is used to input the multimodal information stream into a pre-trained multimodal fusion model for cross-modal semantic analysis and output the local processing results. The multimodal fusion model includes a feature alignment stream and a conflict capture stream. The execution module is used to generate a set of multi-objective optimal decision schemes through reinforcement learning based on the local processing results, and to receive the final decision instruction generated after manual review, and execute the final decision instruction to complete the customs clearance business processing; The filtering module is used to collect processing records containing the multimodal information stream, the local processing results, and the final decision instruction, and to filter high-value training samples through an active learning mechanism to establish a negative bias sample library. An update module is used to perform asynchronous federated aggregation based on the negative bias sample library to update the multimodal fusion model.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the customs clearance document automation processing method based on multi-source data fusion as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the automated processing method for customs clearance documents based on multi-source data fusion as described in any one of claims 1-7.

Images