Method and device for multi-modal commodity recognition and incremental generalization of customs inspection robot dog

Abstract

This application provides a method and apparatus for multimodal commodity recognition and incremental generalization using a customs inspection robot, relating to the fields of customs intelligent supervision and embodied intelligent robot visual recognition. The method includes: acquiring multi-view visual feature data and auxiliary declaration text of target goods; classifying visual features level by level using a multi-level cascaded classification model to obtain target classification results, with the current-level classification head input being the tariff semantic vector of the previous-level prediction node, and the candidate set limited to the child nodes of that node to achieve semantic space constraints of the parent node; independently encoding visual features and declaration text and determining single-goods consistency indicators; responding to situations where confidence or consistency conditions are not met and a new category instruction is received, reusing the parent-level feature prototype and combining it with semantic embedding to generate a new feature prototype, and performing incremental fine-tuning through comparative learning. This application ensures that the classification conforms to the tariff hierarchy logic, guarantees the ability to detect false declarations, and achieves compliant generalization of new categories.

Description

Technical Field

[0001] This application relates to the field of customs intelligent supervision and embodied intelligent robot visual recognition, specifically to a method and apparatus for multimodal commodity recognition and incremental generalization by customs inspection robot dogs. Background Technology

[0002] Currently, quadruped robot dogs are increasingly being used for cargo photography and inspection in on-site supervision scenarios such as customs container inspection and cargo accompanying inspection. Intelligent commodity recognition serves as a core capability supporting single-item comparison and classification verification. However, existing robot dog-based recognition solutions often directly apply general image classification models or simply fuse customs declaration text for cross-modal alignment, resulting in the following technical problems: General classification models employ a flat, fully labeled architecture, lacking tariff-legal hierarchical constraints, making them prone to regulatory-level logical errors across broad categories; existing multimodal solutions fuse and align declaration text with visual features, guiding recognition results closer to the declared information, masking the risk of discrepancies between single and cargo items, and deviating from the regulatory objective of detecting false declarations and concealments; incremental learning lacks prior knowledge of the tariff system, making adaptation to new categories prone to catastrophic forgetting, and the classification results are not constrained by classification rules. Therefore, there is an urgent need for a commodity recognition method that can adapt to the customs tariff hierarchy logic, ensure the ability to detect false declarations, and support compliant incremental generalization. Summary of the Invention

[0003] To address the problems of existing technologies, such as the lack of tariff-level constraints in general classification models leading to cross-category regulatory-level logical errors, the concealment of false declarations and concealment risks by cross-modal fusion schemes, and the lack of prior knowledge of the tariff system in incremental learning leading to non-compliance in the adaptation of new categories, this application proposes a multimodal commodity recognition and incremental generalization method and device for customs inspection robot dogs. This method achieves classification results that naturally conform to the statutory tariff level, ensures independent detection of false declarations and concealment risks, and enables rapid generalization of compliance for new categories.

[0004] To achieve the above objectives, this application adopts the following technical solution:

[0005] The process involves acquiring multi-view visual feature data of the target cargo and auxiliary declaration text; classifying the multi-view visual feature data level by level using a multi-level cascaded classification model to obtain the target classification result; the multi-level cascaded classification model includes multiple cascaded classification heads that correspond sequentially to the customs tariff hierarchical classification system, with the input of the current level classification head concatenated with the tariff semantic vector corresponding to the predicted node output by the previous level classification head, and the classification candidate set of the current level classification head limited to all child nodes of the predicted node in the customs tariff hierarchical classification system to achieve semantic space constraints of the parent node; and independently encoding the multi-view visual feature data and the auxiliary declaration text to obtain visual semantic data based on the same customs tariff semantic space. The system uses visual semantic features and textual semantic features; it determines a single-item consistency index based on the visual semantic features and textual semantic features; in response to the target classification result's confidence level not meeting the preset confidence condition or the single-item consistency index not meeting the preset consistency condition, and receives a new product category trigger instruction, it obtains the labeled samples and corresponding tariff node information of the new category, reuses the trained feature prototypes of the corresponding parent level in the customs tariff hierarchical classification system, and generates a feature prototype vector of the new category by combining the semantic embedding of the new tariff node; it constructs positive and negative sample pairs with the tariff semantic vector of the corresponding parent node as the feature space benchmark, and incrementally fine-tunes the multi-level cascaded classification model through comparative learning to complete the generalization update of the new category.

[0006] Preferably, based on the prediction node output by the previous level classification head, the target classification rule semantic vector is matched from the preset classification rule library; cross-attention calculation is performed on the target classification rule semantic vector and the multi-view visual feature data to generate a spatial attention weight map; the key region features in the multi-view visual feature data are enhanced based on the spatial attention weight map, and the enhanced features are input into the current level classification head for classification.

[0007] Preferably, the multi-view visual feature data is concatenated with the tariff semantic vector corresponding to the predicted node output by the previous level classification head, and then input into the current level classification head; in the current level classification head, the similarity between the multi-view visual feature data and the semantic vector of each candidate node corresponding to the current level classification head is calculated; based on the similarity, the initial classification probability output by the current level classification head is weighted and calibrated to obtain the final classification probability of the current level classification head.

[0008] Preferably, the multi-level cascaded classification model is trained using a loss function that includes multi-level consistency constraints. This loss function comprises: a hierarchical cross-entropy constraint, used to calculate the cross-entropy loss for each level's classification results and weighting them according to increasing hierarchical depth; a hierarchical subordination constraint, used to verify whether the parent node affiliation of each prediction result is consistent with the previous level's prediction result and assigning tiered penalty weights based on the depth of the errors; and a semantic distance constraint, used to ensure that the distance between the multi-view visual feature data and the correct tariff semantics is less than the distance with the incorrect tariff semantics, based on the semantic vectors of the customs tariff hierarchical classification system, and assigning higher penalty weights to cross-level errors.

[0009] Preferably, based on the visual semantic features and the text semantic features, a category consistency determination is performed at the coarse-grained level of the customs tariff hierarchical classification system to obtain a first consistency score; in response to the first consistency score satisfying a preset category consistency condition, a semantic distance determination is performed at the fine-grained level of the customs tariff hierarchical classification system based on the visual semantic features and the text semantic features to obtain a second consistency score; based on the first consistency score and the second consistency score, the single-cargo consistency index is generated.

[0010] Preferably, the method is applied to a verification system comprising an end-side device and a cloud server. The method further includes: in the end-side device, in response to the target classification result having a confidence level that meets a preset confidence level condition and the single-item consistency index meeting a preset consistency condition, the target classification result is locally archived; in response to the target classification result having a confidence level that does not meet the preset confidence level condition, or the single-item consistency index not meeting the preset consistency condition, or the multi-view visual feature data belonging to a preset complex scene data, the multi-view visual feature data and the target classification result are uploaded to the cloud server; in the cloud server, the uploaded data is subjected to full-dimensional compliance verification based on a full-scale classification network and a complete hierarchical knowledge graph, and a final judgment result is output.

[0011] As a preferred approach, during incremental training, a tariff hierarchy regularization term is introduced to constrain the distance distribution between newly added category features and historical category features to conform to the semantic hierarchy distribution of the customs tariff classification system. The compliance of the hierarchical affiliation and classification rules for the newly added commodity categories is verified. After the verification is passed, the newly added commodity categories are synchronously integrated into the customs tariff classification system.

[0012] Preferably, the customs inspection robot dog is controlled to plan multiple differentiated operating poses for the target goods, and visible light images and depth images are acquired simultaneously in each differentiated operating pose; before classifying the multi-view visual feature data step by step through the multi-level cascaded classification model, the method further includes: extracting overall appearance features from the multi-view visual feature data through a global feature extraction stream to obtain global features; extracting micro-region features from the multi-view visual feature data through a local detail feature extraction stream to obtain local features; fusing the global features and the local features across scales to obtain multi-scale fused visual features of a unified dimension, and using the multi-scale fused visual features as input to the multi-level cascaded classification model.

[0013] In addition, this application also provides a customs inspection robot dog multimodal commodity recognition and incremental generalization device, including: a processor; a memory for storing a computer program; when the processor executes the computer program, it implements the customs inspection robot dog multimodal commodity recognition and incremental generalization method as described above.

[0014] In addition, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the customs inspection robot dog multimodal commodity recognition and incremental generalization method as described above.

[0015] Beneficial effects:

[0016] This application leverages the synergistic effect of parent node semantic space constraints and decoupled cross-modal verification in a multi-level cascaded classification model to force the classification process to follow the legal hierarchy logic of tariffs at the network structure level, eliminating cross-category regulatory errors. Simultaneously, after independent encoding of visual and textual data, consistency of individual goods is determined based on semantic distance, avoiding interference from declared information with visual recognition and ensuring the independent detection capability of false and concealed declaration risks. In the incremental generalization stage, parent-level feature prototypes are reused, and comparative learning sample pairs are constructed using the parent node semantic vector as the feature space benchmark, enabling the new category features to be embedded and inherited from the existing hierarchical structure, achieving compliant, low-forgetting, and rapid adaptation.

[0017] By using a dynamic attention mechanism guided by classification rules, the model automatically matches the semantic vector of the corresponding classification rule based on the results of the upper-level classification, and generates a cross-attention weight map to enhance the features of key areas. This enables the model to focus on fine-grained discrimination areas such as core components and material identifiers corresponding to the legal basis of tariff classification, significantly improving the accuracy of tariff code-level differentiation of similar-looking goods.

[0018] By employing a multi-level consistency constraint loss function, the training phase utilizes a triple mechanism of hierarchical cross-entropy, step-penalty subordination constraints, and semantic distance comparison constraints to force the model to learn the hierarchical subordination logic of tariff rules rather than simply fitting the data distribution. This ensures the consistency of the hierarchical distribution between the visual feature space and the tariff rule semantic space, fundamentally improving the regulatory compliance of the classification results.

[0019] By employing a tiered verification strategy, a rapid screening for broad-based consistency is first conducted at a coarse-grained level. Once this screening is successful, a precise semantic distance determination is performed at a fine-grained level, balancing verification efficiency with regulatory accuracy. Combined with a closed-loop edge-cloud tiered verification system, the edge processes high-confidence routine data, while the cloud performs full-dimensional compliance review on low-confidence, consistency anomalies, or complex scenario data, achieving optimized resource allocation and comprehensive risk coverage.

[0020] By constraining the distance distribution between newly added and historical category features during incremental training using regularization terms of tariff hierarchy relationships, the features are made to conform to the semantic hierarchy structure of the tariff, thus suppressing catastrophic forgetting at the feature space level. Simultaneously, compliance checks on hierarchical attribution and classification rules are performed to ensure that newly added categories are naturally incorporated into the statutory tariff system and can be integrated into regulatory business processes without additional manual review.

[0021] By controlling the robot dog to plan multiple differentiated working poses and simultaneously acquire visible light and depth images, the occlusion and low-light information blind spots under a single viewpoint are physically supplemented. Combined with global-local dual-stream feature extraction and cross-scale fusion, the overall appearance and micro-region details are captured at the same time, providing a complete multi-scale feature foundation for progressive classification and significantly improving the recognition robustness in complex inspection environments. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the customs inspection robot dog multimodal commodity recognition and incremental generalization method according to an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0025] Example 1

[0026] like Figure 1 As shown, this embodiment provides a multimodal commodity recognition and incremental generalization method for customs inspection robot dogs. This method relies on the collaborative execution of a quadruped robot terminal with motion perception capabilities and backend computing resources, aiming to solve technical problems such as poor compliance of commodity classification logic, difficulty in detecting false declaration risks, and difficulty in adapting to new categories in customs supervision scenarios. The method flow of this embodiment mainly includes the following steps.

[0027] Step S1: Obtain multi-view visual feature data of the target cargo and auxiliary declaration text.

[0028] Specifically, the multi-view visual feature data is actively collected by the customs inspection robot dog during on-site perception of the target cargo. In this embodiment, the customs inspection robot dog is a quadrupedal robot platform with autonomous mobility and multi-degree-of-freedom pose adjustment capabilities. It is equipped with various sensing devices such as visible light cameras, depth sensors, and infrared supplementary lighting modules, enabling it to autonomously plan inspection paths and perform close-range observation tasks in unstructured environments such as inside containers and cargo yards. Unlike traditional fixed cameras or manual handheld shooting methods, the robot dog can actively adjust its travel route, standing height, and shooting angle according to the size of the target cargo, its stacking status, and the ambient lighting conditions, thereby acquiring the appearance, texture details, and depth information of the cargo from different observation postures. For example, the robot dog can use a crouching posture to photograph the details of the bottom pallet of the cargo, a standing posture to look down at the top markings of the cargo, or a sideways posture to approach the side of the cargo to capture depth information of obscured areas. Through this proactive multi-view physical completion, the robot dog can effectively overcome the adverse effects of low lighting inside the container and cargo stacking obstructing visual acquisition, obtaining multi-view visual feature data containing rich discriminative information, providing a complete data foundation for subsequent classification and recognition. The auxiliary declaration text refers to structured or unstructured text information such as customs declarations, manifests, or lists associated with this batch of goods, which records declarations such as the declared product name, specifications, and intended use. In this embodiment, visual data and text data are acquired separately as two independent information sources. This separation at the data acquisition level lays the physical foundation for decoupling verification in subsequent processing logic, avoiding the introduction of prior bias during the data acquisition stage.

[0029] Step S2: The multi-view visual feature data is classified step by step using a multi-level cascaded classification model to obtain the target classification result. The multi-level cascaded classification model contains multiple cascaded classification heads that correspond sequentially to the customs tariff hierarchical classification system. The input of the current level classification head is concatenated with the tariff semantic vector corresponding to the predicted node output by the previous level classification head. The classification candidate set of the current level classification head is limited to all child nodes of the predicted node in the customs tariff hierarchical classification system to achieve semantic space constraints of the parent node.

[0030] Specifically, the semantic space constraint of the parent node is the core mechanism that distinguishes this application from general image classification schemes. In general classification models, all categories are usually located in the same flat label space, and the model makes probability predictions across all labels, which is prone to logical errors that cross legal levels. In this embodiment, however, each level of the multi-level cascaded classification model does not work independently, nor does it simply search globally. Instead, the effective classification range of the subsequent level is strictly limited to the semantic subspace determined by the prediction results of the previous level. For example, when the previous level predicts a specific parent node, the next level only makes a judgment within the set of child nodes under that parent node, and other irrelevant nodes are automatically masked or set to zero at the current level. At the same time, the tariff semantic vector corresponding to the predicted node of the previous level is concatenated into the input of the current level. This not only provides contextual semantic guidance, but more importantly, establishes the legal boundaries of the current level classification. Through this hierarchically convergent semantic space constraint, this embodiment forces the classification process to follow the established hierarchical logic at the network structure level, fundamentally eliminating the possibility of cross-class regulatory level errors and ensuring the regulatory compliance of the classification results.

[0031] Step S3: The multi-view visual feature data and auxiliary declaration text are independently encoded to obtain visual semantic features and text semantic features based on the same customs tariff semantic space.

[0032] Specifically, this step emphasizes the combination of independent encoding and the same semantic space. Common multimodal fusion schemes in existing technologies often directly integrate text features into the visual feature extraction network, attempting to leverage textual information to improve recognition accuracy. However, in customs inspection scenarios, the declaration text itself may be forged or erroneous. Using it as prior input for visual recognition can lead to confirmation bias in the model, forcing the visual recognition result to align with incorrect declaration information, thus masking the risk of discrepancies between the goods and the declaration. This embodiment employs a fully decoupled encoding architecture. The visual branch extracts features solely based on the pixel information of the image itself, unaffected by any textual content, ensuring the objectivity and independence of visual features; the text branch performs semantic parsing of the declaration content separately. Although isolated during the encoding stage, both are ultimately mapped to the same pre-constructed customs tariff semantic space, making visual and textual features comparable. This design allows subsequent verification stages to truly play a supervisory role, rather than simply an auxiliary one, providing a reliable technical prerequisite for accurately identifying false declarations and concealment. It should be noted that the customs tariff semantic space is the infrastructure upon which all steps of multi-level cascaded classification, decoupling verification, and incremental generalization depend. Its construction process includes two stages: graph ontology construction and semantic representation learning. In the graph ontology construction stage, a knowledge graph containing four-level hierarchical entities is constructed using legitimate data sources. The four-level hierarchical entities are, in order, tariff chapters, headings, subheadings, and statutory tariff numbers, with strict subordinate relationships between each level. At the same time, classification rule entities such as general classification rules, category notes, chapter notes, and subheading notes, as well as classification attribute entities such as material, function, use, and composition, are entered. Three types of core relationships are defined between entities: hierarchical subordinate relationships, used to represent the legal attribution between superior and subordinate nodes; rule constraint relationships, used to represent the applicability of classification rules to specific nodes; and attribute association relationships, used to represent the correspondence between a node and its classification attribute. Through the organization of entities and relationships, a customs tariff hierarchical knowledge graph with a legal hierarchical structure is formed. In the semantic representation learning stage, a graph attention network with hierarchical weights is used to embed the customs tariff hierarchical knowledge graph. During message passing in the graph attention network, the attention weight between parent and child nodes is configured to be significantly higher than that between neighboring nodes to strengthen the vertical transmission of hierarchical semantics. A hard constraint is applied during training: the semantic vector of a child node must fall within the neighborhood space of its parent node's semantic vector, thus ensuring that the semantic hierarchy is completely consistent with the legal hierarchical structure. After training, the graph attention network outputs a unified-dimensional semantic vector for all tariff entities and classification rule entities. These semantic vectors constitute the customs tariff semantic space. In this semantic space, the semantic vectors of child nodes belonging to the same parent node are geometrically clustered in the same neighborhood, while the semantic vectors of different parent nodes are spatially separated. The hierarchical hierarchy is fully represented through the spatial distribution of the semantic vectors.

[0033] Regarding the dimension of semantic vectors, the output layer dimension of the graph attention network is determined based on the total number of nodes and relation density of the customs tariff hierarchical knowledge graph, ensuring that each tariff entity and classification rule entity is mapped to a dense vector of the same dimension. The selection of the dimension balances expressive power and computational efficiency: too low a dimension leads to insufficient semantic distinguishability between nodes at different levels, while too high a dimension increases the overhead of subsequent classification head concatenation and distance calculation. In this embodiment, the dimension of the semantic vectors is selected within a reasonable range based on the graph size, ensuring that all nodes at the chapter, item, sub-item, and tariff number levels receive sufficient semantic representation. Regarding the topological alignment of the semantic space with the tariff tree hierarchy, the hierarchical subordination hard constraint applied during training causes the output semantic vector space to form a nested geometric topology corresponding to the tariff tree hierarchy structure. Specifically, the semantic vector of each parent node defines a semantic subspace, and the semantic vectors of all its child nodes are constrained within this semantic subspace; the subspace itself is nested within a larger semantic subspace defined by its parent node. Taking a four-level hierarchical structure as an example, the semantic vector of a chapter-level node defines the outermost semantic subspace. The semantic vector of a category-level node is located within its corresponding chapter-level subspace and defines a smaller category-level semantic subspace. The semantic vector of a subcategory-level node is located within its corresponding category-level subspace and defines a smaller subcategory-level semantic subspace. The semantic vector of a tariff code-level node is located within its corresponding subcategory-level subspace. Within the same level, the semantic subspaces defined by different parent nodes are geometrically separated and do not overlap. The semantic vectors of sibling nodes under the same parent node maintain a distinguishable distance within their corresponding semantic subspaces. Through this layer-by-layer nested, layer-separated geometric structure, the semantic vector space fully represents the tree-like hierarchical relationship of the customs tariff. This distance distribution characteristic allows subsequent steps such as classification decisions, consistency checks, and incremental generalization based on semantic distance to naturally follow the legal hierarchical logic of the tariff.

[0034] It should be understood that although this embodiment uses graph attention network as the implementation method for semantic representation learning, in other embodiments, graph convolutional network or knowledge graph embedding methods such as translation embedding can also be used, as long as the training process applies hierarchical subordination constraints so that the output semantic vector can maintain a spatial distribution consistent with the statutory hierarchical structure.

[0035] Step S4: Determine the consistency index of goods based on visual semantic features and textual semantic features.

[0036] Specifically, the consistency index for individual goods is a quantitative measure of whether the actual goods match the declared information. Since both visual and textual semantic features reside within the same customs tariff semantic space, the system can determine consistency by calculating their geometric distance or similarity within that space. This index is not a simple label comparison, but rather based on the alignment of deep semantics. For example, even if the declared product name and the visual recognition result are not entirely identical in text, if they point to the same classification node or adjacent area in the tariff semantic space, they can still be considered consistent; conversely, if they are too far apart in the semantic space, even if there is some overlap in superficial keywords, they will be considered inconsistent. This consistency determination mechanism based on semantic space can effectively handle complex situations such as non-standard declarations and the misuse of aliases, while maintaining high sensitivity to substantial classification differences.

[0037] Step S5: In response to the target classification result's confidence level not meeting the preset confidence level condition or the single-item consistency index not meeting the preset consistency condition, and receiving a new commodity category trigger instruction, the labeled samples and corresponding tariff node information of the new category are obtained. The trained feature prototypes of the corresponding parent level in the customs tariff hierarchical classification system are reused, and the feature prototype vector of the new category is generated by combining the semantic embedding of the new tariff node. Positive and negative sample pairs are constructed using the tariff semantic vector of the corresponding parent node as the feature space benchmark. The multi-level cascaded classification model is incrementally fine-tuned through comparative learning to complete the generalization update of the new category.

[0038] Specifically, this step defines the triggering mechanism and execution logic for incremental generalization. Incremental updates are not triggered automatically without conditions; they require both performance or consistency anomalies and explicit triggering instructions to be met simultaneously. When the system finds that the existing model cannot accurately identify a certain type of goods or frequently issues warnings of discrepancies between goods and orders, it indicates that there may be new categories that are not covered. In this case, the incremental learning process can only be started after receiving a clear update instruction. At the execution level, this embodiment does not use full retraining but reuses the trained feature prototypes of the corresponding parent level. This means that the learning of new categories is based on inheriting the existing hierarchical knowledge structure, rather than starting from scratch. By constructing positive and negative sample pairs with the semantic vector of the parent node as the feature space benchmark for comparative learning, the feature distribution of the new categories is constrained within a reasonable semantic region, ensuring both the distinguishability between the new categories and historical categories and maintaining the topological stability of the entire tariff hierarchy. This mechanism effectively alleviates the catastrophic forgetting problem common in incremental learning while ensuring that the new categories naturally conform to the hierarchical classification rules of customs tariffs, achieving compliant and efficient knowledge expansion.

[0039] Through the coordinated execution of steps S1 to S5 above, this embodiment constructs a complete method for customs inspection robot dog commodity recognition and incremental generalization. This method solves the compliance problem of classification logic through parent node semantic space constraints, solves the problem of false declaration risk detection through independent encoding and decoupling verification, and solves the problems of new category adaptation and knowledge forgetting through a controlled incremental generalization mechanism. The modules are logically tightly coupled, jointly supporting intelligent recognition capabilities that adapt to the special regulatory needs of customs.

[0040] Example 2

[0041] Building upon Example 1, this example further elaborates on the internal operating mechanism of the multi-level cascaded classification model, specifically involving feature enhancement guided by classification rules, probability calibration of semantic space constraints, and training strategies for multi-level consistency constraints, in order to construct a defense depth for classification accuracy and compliance.

[0042] Based on the prediction nodes output by the previous-level classification head, the system matches the target classification rule semantic vector from a preset classification rule library. It then performs cross-attention calculations based on the target classification rule semantic vector and multi-view visual feature data to generate a spatial attention weight map. Finally, it enhances the key region features in the multi-view visual feature data using the spatial attention weight map and inputs the enhanced features into the current-level classification head for classification. Specifically, this mechanism solves the problem of fine-grained differentiation of similar-looking but heterogeneous goods in customs tariff classification. In customs operations, different categories of goods often follow completely different classification logics. For example, electromechanical products are usually classified based on core functional components, while textiles are classified based on material composition. If only general visual features are relied upon, the model cannot automatically focus on these legally valid discriminative regions. In this embodiment, the system dynamically matches the corresponding classification rule semantic vector based on the previous-level classification result. This vector carries the legal classification basis information for a specific category. Through cross-attention calculations, the abstract rule semantics are transformed into a concrete spatial attention weight map, enabling the model to adaptively focus its attention on key regions in the image related to the classification rules, while suppressing interference from background or non-critical components. This dynamic adaptation mechanism ensures that the classification process can not only see the goods, but also understand the identification points required by the tariff rules, thereby significantly improving the accuracy of tariff code-level differentiation of goods with similar appearances.

[0043] The system concatenates multi-view visual feature data with the tariff semantic vector corresponding to the predicted node output by the previous-level classification head, and inputs this concatenation into the current-level classification head. Within the current-level classification head, the similarity between the multi-view visual feature data and the semantic vectors of each candidate node corresponding to the current-level classification head is calculated. Based on this similarity, the initial classification probability output by the current-level classification head is weighted and calibrated to obtain the final classification probability. Specifically, the tariff semantic vector of the previous-level predicted node is concatenated into the current-level input, which not only serves to convey contextual information but, more importantly, establishes a legitimate semantic boundary for the current-level classification. Based on this, the system calculates the similarity between the visual features and the semantic vectors of all candidate nodes in the current level, and uses this similarity to weight and calibrate the initial probability output by the classification head. The physical significance of this calibration mechanism is that when visual features become blurred due to lighting, occlusion, or angle issues, the tariff semantic vector, as strong prior knowledge, can guide the model to shift the probability distribution towards semantically more reasonable nodes, rather than simply relying on potentially damaged visual pixel information. For example, in item-level classification, if a visual feature is highly similar to the semantic vector of a sub-item, even if its original visual confidence is slightly low, the probability after calibration will be increased, thereby effectively avoiding misjudgment within the hierarchy caused by visual noise and ensuring that the classification result conforms to both visual facts and the semantic structure of the tariff.

[0044] The multi-level cascaded classification model is trained using a loss function that includes multiple levels of consistency constraints. This loss function comprises: a hierarchical cross-entropy constraint, which calculates the cross-entropy loss for each level's classification results and assigns weights according to increasing hierarchical depth for weighted summation; a hierarchical dependency constraint, which verifies whether the parent node affiliation of each level's prediction is consistent with the previous level's prediction and assigns tiered penalty weights based on the depth of the errors; and a semantic distance constraint, which, based on the semantic vectors of the customs tariff hierarchical classification system, ensures that the distance between multi-view visual feature data and correct tariff semantics is less than the distance with incorrect tariff semantics, and assigns higher penalty weights to cross-level errors. Specifically, this loss function is designed to force the model to learn and internalize the hierarchical logic of customs tariffs from the training level, rather than merely fitting the statistical distribution of image data. The hierarchical cross-entropy constraint ensures the basic accuracy of each classification level, and the increasing weights with deeper levels reflect the business characteristic of requiring finer-grained distinctions at lower levels. The hierarchical subordination constraint explicitly conveys the regulatory logic that cross-chapter errors are more serious than cross-category errors through tiered penalty weights. This forces the model to strictly adhere to the tree-like hierarchical structure in the feature space, fundamentally eliminating regulatory-level logical errors across major categories. The semantic distance constraint aims to align the visual feature space with the tariff semantic space, ensuring that visual samples belonging to the same tariff node are tightly clustered in the feature space, while samples from different nodes are far apart. It also imposes a higher penalty on cross-level errors, further strengthening the compliance of the feature representation. The synergistic effect of these three loss functions results in a multi-level cascaded classification model that not only performs excellently in statistical metrics but also naturally conforms to the legal hierarchical system of customs tariffs in its intrinsic mechanism, laying a solid structural foundation for subsequent incremental generalization and practical regulatory applications.

[0045] Example 3

[0046] Building upon Examples 1 and 2, this example further constructs a system-level defense line of layered verification and edge-cloud collaboration, aiming to solve the technical challenge of balancing inspection efficiency and risk control in customs supervision scenarios. Specifically, based on visual and textual semantic features, a first consistency score is obtained by determining the consistency of major categories at the coarse-grained level of the customs tariff classification system. In response to the first consistency score satisfying a preset major category consistency condition, a second consistency score is obtained by determining the semantic distance at the fine-grained level of the customs tariff classification system based on visual and textual semantic features. Based on the first and second consistency scores, a single-cargo consistency index is generated. This layered verification logic is a detailed refinement of the single-cargo consistency index determination steps in Example 1. In actual customs operations, the cargo throughput is enormous. Performing the finest-grained tariff code-level semantic comparison directly on each shipment would be computationally expensive and time-consuming. This embodiment employs a coarse-to-fine progressive verification strategy: First, a rapid screening for broad-category consistency is performed at the coarse-grained level. This level has a large semantic space, enabling the rapid elimination of abnormal data that clearly does not match the actual goods with extremely low computational cost. Only after the coarse-grained judgment passes is the precise semantic distance judgment at the fine-grained level triggered. This design ensures real-time interception of obvious false reporting while avoiding the waste of valuable edge computing resources on low-risk data, achieving a dynamic balance between regulatory efficiency and accuracy. It should be understood that although this embodiment uses a two-layer verification as an example, in other implementations, it can be extended to three or more layers of progressive verification according to business needs, as long as the coarse-to-fine screening logic is followed.

[0047] Furthermore, this method is applied to a verification system comprising edge devices and a cloud server. On the edge device, if the confidence level of the target classification result meets a preset confidence condition and the single-item consistency index meets a preset consistency condition, the target classification result is archived locally. If the confidence level of the target classification result does not meet the preset confidence condition, or the single-item consistency index does not meet the preset consistency condition, or the multi-view visual feature data belongs to a preset complex scene, the multi-view visual feature data and the target classification result are uploaded to the cloud server. On the cloud server, the uploaded data undergoes full-dimensional compliance verification based on a full-scale classification network and a complete hierarchical knowledge graph, and the final judgment result is output. Specifically, this edge-cloud hierarchical mechanism fully leverages the differentiated advantages of the edge robot dog and the cloud server in terms of computing power, knowledge base completeness, and response time. Edge devices, constrained by the power consumption and heat dissipation limitations of embedded platforms, typically deploy lightweight models suitable for handling routine inspection tasks requiring high confidence and consistent shipments. They achieve millisecond-level customs clearance responses through local archiving. Cloud servers, on the other hand, are not limited by real-time performance or power consumption. They are equipped with a larger, comprehensive classification network and a full-dimensional knowledge base including a complete tariff-level knowledge graph, a regulatory negative list, and a historical violation case database. When the edge device detects insufficient confidence, abnormal shipment consistency, or identifies complex scenarios such as severe occlusion or extremely low lighting, it automatically triggers the data uplink, which is then taken over by the cloud for in-depth verification. It is particularly important to note that the edge-cloud traffic splitting decision in this embodiment is tightly coupled with the decoupled verification mechanism in Embodiment 1: the shipment consistency index obtained from decoupled verification is not only the basis for judging whether goods are compliant but also the core routing factor determining the data processing path. This design ensures that only data with genuine regulatory risks or difficult identification will occupy cloud resources, thereby achieving optimal allocation of computing resources and full coverage of regulatory risks at the system level. For example, when the robot dog collects a set of low-light images deep inside a container, and the initial identification results show a significant semantic deviation from the declaration text at the coarse-grained level, the system will immediately mark the set of data as high-risk and upload it to the cloud. The cloud will then conduct a secondary assessment by combining the classification annotations and negative list in the complete map. If it is confirmed to be a new type of false declaration method, the risk warning will be updated in real time and fed back to the inspection terminal, forming a complete closed loop from perception to decision-making to feedback.

[0048] Example 4

[0049] Building upon the incremental generalization triggering mechanism described in Example 1, this example further details the model stability assurance and compliance control during incremental fine-tuning. Specifically, during incremental training, a tariff hierarchy regularization term is introduced to constrain the distance distribution between newly added category features and historical category features to conform to the semantic hierarchy distribution of the customs tariff classification system. This regularization term is designed to address the catastrophic forgetting problem commonly found in traditional incremental learning. In common few-shot learning or incremental learning schemes, the model often focuses only on the classification loss of new samples, leading to a sharp deviation of parameter updates from the original optimal solution, resulting in a significant decrease in the recognition accuracy of old categories. In this example, the tariff hierarchy regularization term acts as a stabilizer for the feature space. Since customs tariffs themselves have a strict tree-like hierarchical structure, different child nodes belonging to the same parent node should maintain a relatively stable topological relationship in the semantic space. This regularization term calculates the geometric distance between the feature vector of the newly added category, the feature vectors of historical categories at the same level, and the feature vector of the parent node, and compares it with the predefined hierarchical distribution of the tariff schedule. It enforces that the embedding position of the new feature must not disrupt the existing hierarchical topology. For example, when a specific model of industrial valve is added, the regularization term constrains its feature vector to fall within the semantic neighborhood of its category node and to maintain a reasonable distance from the features of other existing valves under that category, preventing it from drifting far from the category or overlapping with features of unrelated categories. This structured constraint based on prior business knowledge ensures, in principle, that the model retains the feature representation capabilities of old knowledge to the greatest extent while learning new knowledge, thus effectively suppressing catastrophic forgetting.

[0050] Furthermore, the newly added commodity categories undergo compliance verification of hierarchical affiliation and classification rules. Once verification is successful, the new commodity categories are synchronously integrated into the customs tariff hierarchical classification system. This step reflects the unique characteristics of incremental learning in customs supervision scenarios, where technical recognizability must yield to business compliance. In general image recognition, a model is considered to have successfully learned a new category as long as it can distinguish it; however, in customs inspection, if a new category is not correctly attached to a legal node in the tariff tree, or does not conform to the statutory classification rules for that type of commodity, its recognition result is legally invalid. Therefore, this embodiment sets compliance verification as a prerequisite or synchronous step for incremental updates. The system automatically verifies whether the annotation information of the newly added category fully matches the attribute constraints of the target parent node and checks whether it violates any exclusive classification annotations. Only when all verification items pass will the feature prototype of the new category be formally written into the model parameter library and synchronously updated to the client and cloud. This mechanism ensures that each incremental update produces a model version that is inherently compliant, avoiding the risk of illegal classification results due to automatic algorithm expansion. This allows the updated model to be directly and seamlessly integrated into actual regulatory business processes without the need for additional manual compliance review.

[0051] It should be understood that although this embodiment uses the regularization term of the tariff hierarchy as an example to illustrate the anti-forgetting mechanism, in other implementations, other continuous learning strategies such as elastic weight consolidation and knowledge distillation can also be combined to help maintain model performance, as long as its core constraint logic still follows the hierarchical semantic distribution of the customs tariff. Meanwhile, the specific rule base for compliance verification should be dynamically maintained with the annual updates of the tariff version to ensure the timeliness of the verification basis. Through the dual protection of the above regularization constraints and compliance verification, the incremental generalization achieved in this embodiment not only maintains a balanced recognition accuracy between new and old categories in terms of technical indicators, but also ensures in terms of business logic that the model evolution process always remains within the legal regulatory framework, truly achieving the goal of technology serving compliance regulation.

[0052] Example 5

[0053] Building upon the aforementioned embodiments, this embodiment further details the acquisition method and preprocessing flow of multi-view visual feature data. Specifically, the control system plans multiple differentiated operational poses for the target cargo, and simultaneously acquires visible light images and depth images in each differentiated operational pose. This feature reflects the design concept of fully utilizing the unstructured terrain adaptability of quadruped robots. In customs container inspection scenarios, goods are often densely stacked and the lighting conditions inside the container are poor. Fixed-viewpoint or single-height shooting can easily produce visual blind spots or missing information. The differentiated operational poses in this embodiment are not random movements, but rather targeted observation strategies generated based on the estimated size and stacking status of the target cargo. For example, the robot dog can plan a crouching low-position posture to obtain details of the bottom pallet or lower packaging of the cargo, a standing high-position posture to view the top markings or upper stacking of the cargo, and a left-right lateral posture to capture depth information of side gaps or occluded areas of the cargo. Through this proactive multi-view physical completion, the system can fundamentally solve the problem of feature incompleteness caused by occlusion or single angle, providing a complete data foundation for subsequent recognition. It should be understood that although specific postures such as crouching, standing, and tilting are listed here, in actual applications, the number and specific forms of working postures can be dynamically adjusted according to the confinement of the inspection site, the type of goods, and the kinematic constraints of the robot dog itself, as long as multi-angle information complementarity can be achieved.

[0054] After acquiring the raw image data, to simultaneously meet the dual requirements of coarse-grained category classification and fine-grained tariff code differentiation, this embodiment designs a dual-stream feature extraction architecture. Specifically, a global feature extraction stream extracts overall appearance features from the multi-view visual feature data to obtain global features; a local detail feature extraction stream extracts minute region features from the multi-view visual feature data to obtain local features; and cross-scale fusion of global and local features yields a unified-dimensional multi-scale fused visual feature, which is then used as input to a multi-level cascaded classification model. The dual-stream architecture, rather than a single network, is adopted because customs tariff classification has significant cross-scale characteristics: chapter-level or item-level classification relies on macro-level semantics such as the overall outline of the goods and packaging form, while subheading-level or tariff code-level classification often depends on micro-level details such as nameplate text, component interfaces, and fabric texture. If only the global feature stream is used, high-resolution details will be lost during downsampling, leading to failure in fine-grained differentiation; if only the local feature stream is used, there is a lack of global context guidance, making it susceptible to background noise interference and resulting in significant computational redundancy. Therefore, this embodiment captures high-level semantics through a global flow to support the initial decision-making stages of the cascaded classification in Embodiment 1, while retaining high-frequency details through a local flow to support the fine-grained discrimination in later stages. In the cross-scale fusion stage, the system utilizes a channel attention mechanism or a feature pyramid structure to align and integrate features from two different scales, ensuring that the final output multi-scale fused visual features contain both macroscopic judgment criteria and microscopic discriminative evidence. These fused features are directly used as input to the multi-level cascaded classification model, achieving tight coupling between the perception module and the cognitive module.

[0055] Example 6

[0056] This embodiment provides a multimodal commodity recognition and incremental generalization device for a customs inspection robot, including a processor and a memory. The memory stores a computer program, and the processor executes the computer program to implement the multimodal commodity recognition and incremental generalization method for a customs inspection robot as described in any of Embodiments 1 to 5. Specifically, this device is the physical hardware implementation of the aforementioned method embodiments. The processor can adopt different hardware architectures depending on the actual deployment location. When the device is deployed on the edge of a customs inspection robot, the processor can be an embedded artificial intelligence chip, a field-programmable gate array, or a dedicated neural network acceleration unit integrated in the robot's main control unit to meet the stringent requirements of low power consumption, real-time inference, and synchronous processing of multi-sensor data. When the device is deployed on a cloud server, the processor can be a high-performance graphics processing unit cluster or a tensor processing unit to support large-scale parallel computing of the full classification network and complex knowledge graph retrieval tasks. The memory includes high-speed random access memory and non-volatile storage devices for persistently storing the program code, tariff-level knowledge graph data, and model parameter weights that implement the above-mentioned multi-level cascaded classification, decoupling verification, and incremental generalization logic. This hardware and software integrated device form can transform the abstract algorithm flow in the aforementioned method embodiments into a manufacturable and marketable physical product, thereby providing sufficient intellectual property protection basis in the hardware manufacturing and whole machine sales process.

[0057] This embodiment also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the multimodal commodity recognition and incremental generalization method for customs inspection robot dogs, as described in any of Embodiments 1 to 5. Specifically, this storage medium is the specific carrier form of the aforementioned method embodiments at the software product level, covering various commercial implementation scenarios such as software distribution, licensing, and cloud deployment. The computer-readable storage medium can be a tangible, non-transitory storage medium, such as a read-only memory, random access memory, disk, optical disk, flash drive, or solid-state drive, or it can be a carrier wave or propagation medium capable of transmitting program code via electrical signals, optical signals, or other signals. When the storage medium is loaded onto the end-side control system or cloud monitoring platform of the customs inspection robot dog, the computer program stored therein is activated and drives the processor to execute the corresponding recognition and verification logic. It should be understood that regardless of whether the storage medium is sold as a standalone software product, pre-installed as firmware within the inspection device, or updated to the terminal via network download, as long as the program instructions stored therein can implement the technical solutions of any of the aforementioned method embodiments when executed, it falls within the protection scope of this application. This separate protection of the storage medium effectively covers potential infringements in the pure software distribution process, constructing a complete three-dimensional protection system from method to device to medium.

[0058] Example 7

[0059] To more clearly illustrate the application value of the technical solution of the present application in actual supervision services, this embodiment takes the fine-grained distinction of electromechanical commodities that are extremely easy to confuse in the customs inspection scenario as an example to comprehensively verify the core technical features in the foregoing embodiments. In this application scenario, the target goods to be inspected are industrial fluid conveying equipment with highly similar appearances, which may belong to the tariff number of centrifugal water pumps under a certain heading, or may belong to the tariff number of motors for liquid pumps under another heading. These two types of commodities are often extremely close in overall outline, shell coating and mounting interfaces, but their core functional components and legal classification bases are completely different, making them typical test objects for verifying the effectiveness of the multi-stage cascade classification model and decoupling verification mechanism of the present application.

[0060] First, in the data acquisition stage, the customs inspection robot dog executes the multi-view linked acquisition strategy described in Embodiment 5 for the target goods. Specifically, the robot dog automatically plans and executes differentiated working postures according to the size and stacking state of the target goods: adopting a squatting low posture to shoot the details of the bottom base and the inlet and outlet flanges of the goods to obtain the features of the installation method and pipeline connection; adopting a standing high posture to overlook the top of the goods to capture the overall layout and heat dissipation structure information; adopting left and right side tilting postures to get close to the side of the goods to clearly collect nameplate parameters, junction box positions and shell joint processes. Through this active physical completion, the system obtains multi-view visual feature data containing rich discriminative information, which effectively overcomes the adverse effects of low illumination and stacking occlusion in the container on subsequent fine-grained classification.

[0061] Subsequently, the multi-stage cascade classification model performs stepwise classification processing on the acquired multi-view visual feature data. In the chapter-level classification stage, the model preliminarily determines the target goods as the chapter of mechanical appliances or the chapter of electrical equipment based on global features. Assuming that the prediction node output by the current-level classification head is the chapter of mechanical appliances, according to the parent node semantic space constraint mechanism of Embodiment 1, the candidate set of the next-level heading classification head is strictly limited to all heading nodes under this chapter, completely eliminating the interference of the chapter of electrical equipment and other unrelated chapters, and structurally ensuring the legal compliance of the classification path.

[0062] Building upon this foundation, this embodiment focuses on demonstrating how a dynamic attention mechanism guided by classification rules addresses the challenge of fine-grained differentiation. When the item-level classification head makes a judgment within the semantic space of the chapter, the system automatically matches a rule semantic vector from a preset classification rule library, applicable to the category of goods and categorized by core functional components. This rule vector undergoes cross-attention calculation with multi-view visual feature data to generate a targeted spatial attention weight map. Specifically, for goods suspected of being centrifugal water pumps, the attention weight map automatically highlights key areas such as the impeller structure, volute flow channel, and liquid inlet / outlet seals inside the pump body, while suppressing feature responses from non-discriminatory areas such as the outer casing and base. For goods suspected of being motors for liquid pumps, even if their appearance is similar to a water pump, the rule-guided attention will focus on electrical feature areas such as the stator winding ends, rotor shaft extension ends, terminal blocks, and cooling fan covers. Based on the enhanced key area features of this spatial attention weight map, the current-level classification head is input, enabling the model to accurately capture the essential differences that determine the tariff code classification, thereby accurately outputting the classification results for centrifugal water pumps or motors for liquid pumps under the corresponding item.

[0063] Meanwhile, the decoupled cross-modal verification mechanism played a crucial role in risk control in this scenario. Suppose the goods are actually motors for liquid pumps, but the auxiliary declaration text falsely declares them as centrifugal water pumps. Because this application uses an independent coding architecture, the feature extraction of the visual branch is completely unaffected by the erroneous text information about centrifugal water pumps, and can still identify it as a motor based on objective image features. When determining the consistency index of individual goods, the distance between the visual semantic features and the textual semantic features calculated by the system in the customs tariff semantic space significantly exceeds a preset threshold, thus immediately triggering a warning of discrepancies between individual goods and goods. In contrast, if a traditional feature fusion scheme is used, the visual model is highly likely to be misled by the erroneous declaration text, forcibly aligning the visual features of the motor to the semantic space of the water pump. This embodiment fully demonstrates that the decoupled verification mechanism, while retaining the ability to utilize multimodal information, effectively ensures the customs' independent detection capability against false declarations and concealment.

[0064] Furthermore, if a new type of special fluid transport equipment not yet covered by the existing model is discovered during the inspection process, and the confidence level or single-item consistency index of the equipment does not meet the preset conditions, the inspectors can issue a trigger command to add a new commodity category. The system will initiate the incremental generalization process as described in Example 4: reusing the trained feature prototype of the parent level to which the new category belongs as a basis, and fine-tuning it by combining newly collected labeled samples and corresponding tariff node information. During the incremental training process, the tariff hierarchy regularization term will constrain the features of the new category to maintain a reasonable semantic distance distribution with the historical categories at the same level, ensuring the identifiability of the new category while avoiding catastrophic forgetting of the recognition accuracy of existing categories. After verification, the new category is synchronously integrated into the customs tariff hierarchy classification system, realizing the dynamic and compliant evolution of regulatory capabilities.

[0065] It should be understood that although this embodiment uses the distinction between motors for centrifugal water pumps and liquid pumps as a specific example, the technical solution of this application is also applicable to other categories with complex classification rules, such as textiles, chemical products, and optical instruments. As long as the classification logic of the target product depends on specific attributes or components, and there are cases where products are similar in appearance but classified differently, the rule-guided attention mechanism and decoupling verification mechanism described in this application can both achieve the corresponding technical effects. This embodiment is merely illustrative and is not intended to limit the scope of protection of this application.

[0066] The above description is merely a specific implementation of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. For example, although this application uses a quadruped robot dog as a perception carrier and customs tariffs as a hierarchical classification system for illustration, those skilled in the art should understand that the core of this technical solution lies in achieving compliance of classification logic through semantic space constraints of parent nodes, ensuring independent detection of false report risks through independent visual and textual encoding, and maintaining the stability of model evolution through controlled incremental generalization. Any substitutions based on the above core technical concepts, such as using other mobile robot platforms, adapting to different legal hierarchical classification standards, or adjusting the implementation form of specific feature extraction and verification modules, are equivalent to this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the claims.

Claims

1. A method for multimodal commodity recognition and incremental generalization using a customs inspection robot dog, characterized in that, Includes the following steps: Acquire multi-view visual feature data of the target cargo and auxiliary declaration text; The multi-view visual feature data is classified step by step using a multi-level cascaded classification model to obtain the target classification result. The multi-level cascaded classification model contains multiple cascaded classification heads that correspond sequentially to the customs tariff hierarchical classification system. The input of the current level classification head is concatenated with the tariff semantic vector corresponding to the predicted node output by the previous level classification head. The classification candidate set of the current level classification head is limited to all child nodes of the predicted node in the customs tariff hierarchical classification system to achieve semantic space constraints of the parent node. The multi-view visual feature data and the auxiliary declaration text are independently encoded to obtain visual semantic features and text semantic features based on the same customs tariff semantic space; The consistency index of goods and items is determined based on the visual semantic features and the text semantic features. In response to the target classification result's confidence level not meeting the preset confidence level condition or the single-item consistency index not meeting the preset consistency condition, and upon receiving a new commodity category trigger instruction, the system obtains the labeled samples and corresponding tariff node information of the new category, reuses the trained feature prototypes of the corresponding parent level in the customs tariff hierarchical classification system, and generates a feature prototype vector of the new category by combining the semantic embedding of the new tariff node; constructs positive and negative sample pairs using the tariff semantic vector of the corresponding parent node as the feature space benchmark, and incrementally fine-tunes the multi-level cascaded classification model through comparative learning to complete the generalization update of the new category.

2. The multimodal commodity recognition and incremental generalization method for customs inspection robot dogs according to claim 1, characterized in that, The step-by-step classification of the multi-view visual feature data using a multi-level cascaded classification model to obtain the target classification result includes: Based on the prediction nodes output by the previous level classification head, the target classification rule semantic vector is matched from the preset classification rule base; Based on the target classification rule semantic vector and the multi-view visual feature data, cross-attention calculation is performed to generate a spatial attention weight map; The key region features in the multi-view visual feature data are enhanced based on the spatial attention weight map, and the enhanced features are input into the current level classification head for classification.

3. The multimodal commodity recognition and incremental generalization method for customs inspection robot dogs according to claim 1, characterized in that, The method for implementing semantic space constraints of the parent node includes: The multi-view visual feature data is concatenated with the tariff semantic vector corresponding to the predicted node output by the previous level classification head, and then input into the current level classification head. In the current level classification head, the similarity between the multi-view visual feature data and the semantic vector of each candidate node corresponding to the current level classification head is calculated. Based on the similarity, the initial classification probability output by the current level classification head is weighted and calibrated to obtain the final classification probability of the current level classification head.

4. The multimodal commodity recognition and incremental generalization method for customs inspection robot dogs according to claim 3, characterized in that, The multi-level cascaded classification model is trained using a loss function that includes multi-level consistency constraints. The loss function includes: The hierarchical cross-entropy constraint part is used to calculate the cross-entropy loss for the classification results of each level, and to perform weighted summation by allocating weights according to the principle of increasing hierarchical depth. The hierarchical subordination constraint part is used to verify whether the parent node of each level prediction result is consistent with the prediction result of the previous level, and to allocate the step penalty weight according to the depth of the level crossed by the error. The semantic distance constraint part is used to constrain the distance between the multi-view visual feature data and the correct tariff semantics to be less than the distance with the incorrect tariff semantics by contrastive loss, and to assign higher penalty weights to cross-level errors.

5. The multimodal commodity recognition and incremental generalization method for customs inspection robot dogs according to claim 1, characterized in that, The determination of the single-item consistency index based on the visual semantic features and the text semantic features includes: Based on the visual semantic features and the text semantic features, a class consistency determination is performed at the coarse-grained level of the customs tariff hierarchical classification system to obtain a first consistency score; in response to the first consistency score satisfying the preset class consistency condition, a semantic distance determination is performed at the fine-grained level of the customs tariff hierarchical classification system based on the visual semantic features and the text semantic features to obtain a second consistency score. The single-item consistency index is generated based on the first consistency score and the second consistency score.

6. The multimodal commodity recognition and incremental generalization method for customs inspection robot dogs according to claim 5, characterized in that, The method is applied to a verification system that includes end-side devices and a cloud server, and the method further includes: In the terminal device, in response to the target classification result satisfying the preset confidence condition and the single-item consistency index satisfying the preset consistency condition, the target classification result is archived locally; In response to the fact that the confidence level of the target classification result does not meet the preset confidence level condition, or the single-item consistency index does not meet the preset consistency condition, or the multi-view visual feature data belongs to the preset complex scene data, the multi-view visual feature data and the target classification result are uploaded to the cloud server. In the cloud server, the uploaded data is subjected to full-dimensional compliance verification based on a full-scale classification network and a complete hierarchical knowledge graph, and the final judgment result is output.

7. The method for multimodal commodity recognition and incremental generalization of customs inspection robot dogs according to claim 1, characterized in that, The response is that the confidence level of the target classification result does not meet the preset confidence level condition or the single-item consistency index does not meet the preset consistency condition, and a new commodity category trigger instruction is received. The system obtains the labeled samples and corresponding tariff node information of the new category, reuses the trained feature prototypes of the corresponding parent level in the customs tariff hierarchical classification system, and generates a feature prototype vector of the new category by combining the semantic embedding of the new tariff node. Positive and negative sample pairs are constructed using the tariff semantic vector of the corresponding parent node as the feature space benchmark. Incremental fine-tuning of the multi-level cascaded classification model is performed through comparative learning to complete the generalization update of the new category, including: During incremental training, a tariff hierarchy regularization term is introduced to constrain the distance distribution between new category features and historical category features to conform to the semantic hierarchy distribution of the customs tariff classification system. The new commodity categories are subjected to compliance verification of hierarchical affiliation and classification rules. After the verification is passed, the new commodity categories are synchronously connected to the customs tariff classification system.

8. The method for multimodal commodity recognition and incremental generalization of customs inspection robot dogs according to claim 1, characterized in that, The acquisition of multi-view visual feature data of the target cargo includes: The customs inspection robot dog is controlled to plan multiple differentiated operating poses for the target goods, and simultaneously acquire visible light images and depth images in each of the differentiated operating poses; Before classifying the multi-view visual feature data step by step using a multi-level cascaded classification model, the method further includes: Global features are obtained by extracting overall appearance features from the multi-view visual feature data using a global feature extraction stream. Local features are obtained by extracting minute region features from the multi-view visual feature data using a local detail feature extraction stream. The global features and the local features are fused across scales to obtain a unified-dimensional multi-scale fused visual feature, which is then used as the input to the multi-level cascaded classification model.

9. A multimodal commodity recognition and incremental generalization device for customs inspection robot dogs, characterized in that, include: processor; Memory, used to store computer programs; When the processor executes the computer program, it implements the customs inspection robot dog multimodal commodity recognition and incremental generalization method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the customs inspection robot dog multimodal commodity recognition and incremental generalization method as described in any one of claims 1 to 8.

Images