A method and apparatus for intelligent material identification based on multimodal perception fusion
By employing multimodal perception fusion and adaptive learning techniques, the problems of single modality and difficulty in expansion in traditional material identification are solved, achieving efficient, accurate, and interpretable material identification, adapting to complex environments and new material identification, and reducing system maintenance costs.
Patent Information
- Application Number
- CN202611239584.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-17
- Publication Date
- 2026-11-17
AI Technical Summary
Traditional material identification technologies suffer from limited modalities, poor adaptability, difficulty in model expansion, high maintenance costs, heavy reliance on data annotation, lack of interpretability and zero-shot identification capability, resulting in low material identification efficiency and insufficient accuracy.
A multimodal perception fusion method is adopted, which combines visible light images, depth images, infrared thermal images, barcode/QR code information and physical attribute data. Through cross-modal attention fusion and adaptive weighting, multi-dimensional identification of material features is achieved. Furthermore, by utilizing self-supervised pre-training, few-shot incremental learning and zero-shot recognition techniques, an interference-resistant, adaptive and interpretable material identification system is constructed.
It improves the accuracy and scalability of material identification, reduces system maintenance costs, reduces labeling requirements, enhances the interpretability of identification results, enables the identification of new materials, and improves the level of intelligent material management.
Smart Images

Figure CN123116128A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and more specifically, to a method and apparatus for intelligent material identification based on multimodal perception fusion. Background Technology
[0002] As enterprises continue to expand their material management scale, material warehouses handle a large number of material inbound and outbound operations every day. Traditional material inbound methods rely on operators manually entering material information and then matching it in a database. This method is cumbersome, inefficient, and prone to errors, severely restricting the level of intelligent warehouse management.
[0003] In recent years, deep learning-based image recognition technology has been attempted to be applied to the field of material identification. However, existing technical solutions still have the following shortcomings: (i) Limited recognition modality and poor adaptability Existing solutions primarily rely on visible light images for material identification. In real-world warehousing environments, factors such as varying lighting conditions, partial material occlusion, and packaging differences can significantly reduce identification accuracy. This is especially true when two materials are highly similar in appearance (e.g., different types of screws or different specifications of electronic components), where a single visual modality is insufficient for effective differentiation.
[0004] (ii) The model is difficult to expand and has high maintenance costs. Traditional deep learning models have fixed parameters after training, making them unable to adapt to scenarios where material appearance changes over time (such as packaging redesigns and batch differences) or new material models are dynamically added. Whenever a new material type is added to the material library, a large amount of labeled data needs to be collected again and the model retrained, resulting in extremely high system maintenance costs and severely limiting the system's scalability.
[0005] (iii) Heavy reliance on data annotation and long deployment cycle Existing solutions require a large number of manually labeled material images for model training. Labeling is costly and time-consuming, and the quality of the labeling directly impacts model performance, becoming a major bottleneck restricting the rapid deployment and expansion of the system.
[0006] (iv) The identification results lack interpretability Existing deep learning models only output material category labels, failing to provide identification criteria or confidence level assessments. When identification errors occur, operators struggle to trace the cause, impacting the system's credibility and user acceptance.
[0007] (v) Lack of ability to identify unknown materials Existing models can only identify material categories that have appeared in the training set. They are completely unable to handle novel materials that have never been seen before and lack the ability to perform zero-shot identification using prior knowledge such as material attribute descriptions. Summary of the Invention
[0008] To address the shortcomings of existing technologies, this invention provides a method and apparatus for intelligent material identification based on multimodal perception fusion.
[0009] According to one aspect of the present invention, a method for intelligent material identification based on multimodal perception fusion is provided, comprising: Multimodal sensing data of the target material is collected synchronously through terminal scanning equipment. The multimodal sensing data includes at least two of the following: visible light images, depth images, infrared thermal images, barcode / QR code information, and physical property data. Feature extraction is performed on the multimodal perception data to obtain the modal features corresponding to each modality. The modal features are then adaptively weighted and fused through a cross-modal attention fusion mechanism to obtain the multimodal fused features. The multimodal fusion features are matched with the prototype features of each category in the preset material feature library to determine the category identifier of the target material. The system retrieves the corresponding material information from the material information database based on the category identifier, outputs the identification result, and triggers the warehousing operation.
[0010] According to another aspect of the present invention, a material intelligent identification device based on multimodal perception fusion is provided, comprising: The acquisition module is used to synchronously acquire multimodal sensing data of the target material through a terminal scanning device. The multimodal sensing data includes at least two of the following: visible light images, depth images, infrared thermal images, barcode / QR code information, and physical property data. The extraction module is used to extract features from the multimodal perception data to obtain the modal features corresponding to each modality, and to perform adaptive weighted fusion of the modal features through a cross-modal attention fusion mechanism to obtain multimodal fused features; The matching module is used to perform similarity matching between the multimodal fusion features and the prototype features of each category in the preset material feature library to determine the category identifier of the target material. The output module is used to retrieve the corresponding material information from the material information database based on the category identifier, output the identification result, and trigger the warehousing operation.
[0011] According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the methods described in any of the above aspects of the present invention.
[0012] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the method described in any of the preceding aspects of the present invention.
[0013] Therefore, this invention provides a material intelligent identification method based on multimodal perception fusion. By fusing multimodal data such as visible light images, depth images, infrared thermal images, barcode / QR code information and physical attributes, and combining self-supervised pre-training, few-sample incremental learning, zero-sample identification and other technical means, an interference-resistant, adaptive, interpretable and easily scalable material identification scheme is constructed to realize intelligent material management of "scan and put into storage". Attached Figure Description
[0014] Exemplary embodiments of the present invention can be more fully understood by referring to the following figures: Figure 1 This is a flowchart illustrating a material intelligent identification method based on multimodal perception fusion provided in an exemplary embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of a material intelligent identification device based on multimodal perception fusion provided in an exemplary embodiment of the present invention; Figure 3 This is the structure of an electronic device provided in an exemplary embodiment of the present invention. Detailed Implementation
[0015] Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments of the present invention; it should be understood that the present invention is not limited to the exemplary embodiments described herein.
[0016] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps described in these embodiments do not limit the scope of the invention.
[0017] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of the present invention are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.
[0018] It should also be understood that in the embodiments of the present invention, "multiple" can refer to two or more, and "at least one" can refer to one, two or more.
[0019] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more unless explicitly defined or given contrary instructions in the context.
[0020] Furthermore, the term "and / or" in this invention is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this invention generally indicates that the preceding and following related objects have an "or" relationship.
[0021] It should also be understood that the description of the various embodiments in this invention emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.
[0022] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.
[0023] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.
[0024] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.
[0025] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.
[0026] The embodiments of this invention can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Well-known examples of terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.
[0027] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.
[0028] Exemplary methods Figure 1 This is a schematic flowchart of a material intelligent identification method based on multimodal sensing fusion provided by an exemplary embodiment of the present invention. This embodiment can be applied to electronic devices, such as... Figure 1 As shown, the intelligent material identification method 100 based on multimodal perception fusion includes the following steps: Step 101: Synchronously collect multimodal sensing data of the target material through a terminal scanning device. The multimodal sensing data includes at least two of the following: visible light images, depth images, infrared thermal images, barcode / QR code information, and physical property data. Step 102: Extract features from the multimodal perception data to obtain modal features corresponding to each modality, and perform adaptive weighted fusion of the modal features through a cross-modal attention fusion mechanism to obtain multimodal fusion features; Step 103: Perform similarity matching between the multimodal fusion features and the prototype features of each category in the preset material feature library to determine the category identifier of the target material; Step 104: Retrieve the corresponding material information from the material information database based on the category identifier, output the identification result, and trigger the warehousing operation.
[0029] Specifically, in response to the technical problems existing in the background art, this invention proposes a material intelligent identification method based on multimodal perception fusion, which is implemented as follows: 1. A material multi-dimensional feature fusion and recognition method based on multimodal perception Objective: To solve the problem of difficulty in distinguishing similar materials due to the limitation of single-modal information. Technical solution: Five-dimensional sensing data of materials are collected synchronously via a terminal scanner: Visual modality (RGB image): Material appearance, color, texture → Visual features Fv extracted via CNN / ViT; Depth modality (depth image): Material 3D contour, thickness, and shape → Geometric feature Fd extracted via PointNet / Depth CNN; Physical property modality: Material size and weight are obtained by scanning → encoded into physical feature Fp by MLP; Encoded information modality: Barcode / QR code decoding information, RFID tag data → encoded into identification feature Fc through the embedding layer; Thermal radiation mode (infrared image): Material temperature distribution characteristics → Thermal feature Ft extracted using a lightweight CNN; Cross-modal attention fusion mechanism: Using visual feature Fv as the query and other modal features as the key / value pair, calculate the cross-modal attention weights: F fused = Concat(Fv, Σα i ·V i ) Where α i The attention weights for each modality are dynamically calculated using a learnable gating network, V. i Let be the feature of the i-th mode.
[0030] Environment-adaptive weighted strategy: Based on the output of the light sensor of the terminal device, dynamically adjust the fusion weights of each modality: Good lighting (illuminance > 500 lux): visual feature weight 0.6, other modalities total 0.4. Insufficient illumination (illuminance <200 lux): The weights of infrared and depth features are increased to 0.7. 2. Incremental learning method based on prototype network and dynamic feature space expansion Objective: To overcome the fixed-category dilemma and achieve incremental expansion without retraining for new materials. Technical solution: Prototype network architecture: Maintain a prototype vector (the average value of the features of the samples in that category) for each material category in the embedding space. During inference, the distance between the query sample and all class prototypes is calculated, and the closest one is used as the prediction result; Dynamic feature space expansion mechanism: Maintain a dynamically expandable categorical feature library E = {(c1, p1,σ1), (c2, p2,σ2), ...}, where p is the prototype vector, σ is the intra-class variance, and c is the class label.
[0031] When adding new materials, only 1-3 sample images are needed → extract fusion features → calculate the new category prototype p_new → add to the feature library; The loss is L_reg = Σ ||p_new - p_old|| through feature space regularization. 2 This avoids distortion of the feature space caused by adding new categories.
[0032] Rapid generation of prototypes with few samples: For new material categories, data augmentation (rotation, scaling, color dithering, etc.) is used to expand 1-3 samples into 20-50 augmented samples. The mean of the features of the augmented samples is calculated as the prototype of the category to ensure the stability of the prototype estimation.
[0033] 3. Zero-shot identification method based on attribute semantic mapping Objective: To solve the problem of identifying novel materials that have never been seen before and reduce labeling costs. Technical solution: Building a material attribute semantic library: Establish a structured attribute description vector A for each material. i ∈ ℝᴹ, which includes the following attribute dimensions: Shape attributes: cuboid / cylinder / irregular shape / sheet / tubular; Color attributes: Red / Orange / Yellow / Green / Blue / Purple / Black / White / Gray / Silver / Gold; Material properties: Metal / Plastic / Rubber / Glass / Paper / Wood / Composite materials; Size ratings: Micro / Small / Medium / Large / Extra Large; Weight class: Ultralight / Light / Medium / Heavy / Extra Heavy; Surface characteristics: smooth / rough / textured / matte / glossy; Functional categories: Electronic components / Hardware / Fasteners / Packaging materials / Chemicals / Tools.
[0034] Visual-semantic mapping networks: Train a mapping network M: Fv → A pred This maps visual features to the attribute space.
[0035] The mapping network uses a two-branch structure: Branch 1 (Global Branch): Extract global attributes from the entire image. Branch 2 (Local Branch): Extract local attributes from the detected material region. Zero-shot inference logic: New material image → Visual feature extraction → Attribute prediction vector A pred Calculate A pred With respect to the attribute vectors A of each material in the attribute semantic library i Cosine similarity; The result with the highest similarity is the zero-sample identification result.
[0036] 4. Fine-grained feature extraction network based on attention enhancement Objective: To address the insufficient feature representation capabilities and enhance the focus on local discriminative features. Technical solution: Hybrid Attention Module (HAM): F out = F in + F in ⊗CAM(F in ) + F in ⊗SAM(F in ) CAM stands for Channel Attention (Squeeze-and-Excitation), and SAM stands for Spatial Attention (Deformable Convolution Enhancement).
[0037] Multi-scale dilated convolution pyramid: Multi-scale features are extracted in parallel using dilated convolutions with different dilation rates: Expansion ratio d=1: Extract detailed features (small-sized materials); Expansion rate d=3: Extracts mesoscale features; Expansion ratio d=5: Extract large-scale features; Expansion rate d=7: Extract global contextual features.
[0038] Deformable Convolution v2: Replacing standard convolution with deformable convolution allows the convolution kernel to adaptively adjust the sampling position, better adapting to changes in the shape and posture of materials, especially for feature extraction of irregularly shaped materials.
[0039] 5. Hierarchical Material Classification and Adaptive Retrieval Strategy Objective: To solve the problem of identification efficiency and accuracy in large-scale material databases. Technical solution: Construction of a three-level classification system: Primary categories (major categories): Electronic components, hardware, fasteners, packaging materials, chemicals, tools, and others; Secondary categories (subcategories): Resistors / Capacitors / Inductors / ICs, Bolts / Nuts / Washers / Screws, etc.; Third-level classification (specific material): unique material code (such as RS-001, CAP-100μF, etc.).
[0040] Hierarchical recognition process: Feature fusion → First-level classifier (7 major categories) → Output major category labels; Based on the major category label, activate the corresponding secondary classifier → output the subclass label; Based on the subclass label, activate the corresponding three-level classifier (specific material) → output the material code.
[0041] Adaptive retrieval strategy: High confidence level (>0.95): Directly matches specific materials and automatically completes warehousing. Medium confidence level (0.75-0.95): Returns the top-3 candidate materials for quick operator confirmation. Low confidence (<0.75): Trigger the active learning process, requesting manual annotation and recording. Six: Lightweight Edge Deployment Solution Based on Knowledge Distillation Objective: To address the balance between limited computing power of terminal devices and model performance. Technical solution: Teacher-student network architecture: Teacher network: Large models (such as ResNet101+ cross-modal attention) run on cloud servers and are responsible for high-precision recognition.
[0042] Student network: Lightweight model (such as MobileNetV3+ improved attention module), running on terminal scanner.
[0043] Online knowledge distillation strategy: When the confidence level of the terminal device is low, the image is automatically uploaded to the cloud for secondary confirmation by the teacher network; The output of the teacher's network serves as a soft label, used to guide the online fine-tuning of the student's network. Once more than 10 teacher-labeled samples of the same type of material are accumulated, a local incremental update of the student network is triggered.
[0044] Model quantization and pruning: Post-training quantization: FP32 → INT8, reducing model size by 4 times; Channel pruning: Remove convolutional kernels whose contribution is less than a threshold to further compress the model.
[0045] 7. Interpretable methods for identification results based on causal reasoning Objective: To address the lack of interpretability in "black box" models. Technical solution: Multi-level interpretation output: Heatmap visualization: Grad-CAM++ is used to generate a category activation heatmap, highlighting key regions used for decision-making in the model, which are then overlaid on the original image for presentation. Structured Attribute Matching Report: Along with the recognition results, it provides detailed matching scores for each attribute, forming a complete chain of evidence. Counterfactual Explanation: Generate counterfactual analysis to help understand the model's decision boundary by asking "what would happen if certain features were removed?" Cause-effect graph modeling: Construct a causal graph for material identification: visual features → attribute inference → category decision, decomposing end-to-end black-box prediction into an interpretable causal chain, with each link being auditable and verifiable.
[0046] Therefore, compared with the prior art, the present invention has the following beneficial effects: Significantly improved recognition accuracy: Through multimodal perception fusion, the recognition accuracy is significantly improved compared to single-modal solutions in complex scenarios such as insufficient light, material occlusion, and similar appearance; through joint encoding of physical attribute features, the mismatch rate of similar-looking materials is greatly reduced.
[0047] The system's scalability is greatly enhanced: Through the prototype network and dynamic feature library architecture, the addition of new materials does not require retraining the model. Only 1 to 5 example images are needed to complete the expansion of new categories, and the system maintenance cost is greatly reduced.
[0048] Significantly reduced labeling costs: Through a self-supervised pre-training strategy, the model can achieve high recognition accuracy with a small number of labeled samples, significantly reducing the reliance on manually labeled data.
[0049] It has zero-sample recognition capability: Through a visual-semantic mapping network, the system can identify novel materials that have never been seen before, effectively meeting the needs of scenarios where the material library is dynamically growing.
[0050] The identification results are highly interpretable: Through category activation heatmaps and structured attribute matching reports, users can understand the decision-making basis of the model, enhancing the credibility of the system and user acceptance.
[0051] Exemplary device Figure 2 This is a schematic diagram of the structure of a material intelligent identification device based on multimodal perception fusion provided in an exemplary embodiment of the present invention. Figure 2 As shown, the device 200 includes: The acquisition module 210 is used to synchronously acquire multimodal sensing data of the target material through a terminal scanning device. The multimodal sensing data includes at least two of the following: visible light images, depth images, infrared thermal images, barcode / QR code information, and physical property data. The extraction module 220 is used to extract features from the multimodal perception data to obtain the modal features corresponding to each modality, and to perform adaptive weighted fusion of the modal features through a cross-modal attention fusion mechanism to obtain multimodal fusion features; The matching module 230 is used to perform similarity matching between the multimodal fusion features and the prototype features of each category in the preset material feature library to determine the category identifier of the target material. The output module 240 is used to retrieve the corresponding material information from the material information database according to the category identifier, output the identification result and trigger the warehousing operation.
[0052] Exemplary electronic devices Figure 3 This is the structure of an electronic device provided in an exemplary embodiment of the present invention. For example... Figure 3 As shown, the electronic device 30 includes one or more processors 31 and memory 32.
[0053] The processor 31 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.
[0054] The memory 32 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 31 may execute the program instructions to implement the methods of the software programs of the various embodiments of the present invention described above, and / or other desired functions. In one example, the electronic device may also include an input device 33 and an output device 34, these components being interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0055] In addition, the input device 33 may also include, for example, a keyboard, a mouse, etc.
[0056] The output device 34 can output various information to the outside. The output device 34 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.
[0057] Of course, for the sake of simplicity, Figure 3 Only some of the components of the electronic device relevant to the present invention are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.
[0058] Exemplary computer program products and computer-readable storage media In addition to the methods and apparatus described above, embodiments of the present invention may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of the present invention described in the "Exemplary Methods" section above.
[0059] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of the present invention. The programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0060] Furthermore, embodiments of the present invention may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the methods for information mining of historical change records according to various embodiments of the present invention as described in the "Exemplary Methods" section above.
[0061] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0062] The basic principles of the present invention have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in the present invention are merely examples and not limitations, and should not be considered as essential features of each embodiment of the present invention. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the present invention to the necessity of employing the aforementioned specific details.
[0063] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0064] The block diagrams of devices, systems, devices, and systems involved in this invention are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, systems, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.
[0065] The methods and systems of the present invention may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of the present invention are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, the present invention may also be implemented as a program recorded on a recording medium, the program comprising machine-readable instructions for implementing the methods according to the present invention. Thus, the present invention also covers recording media storing programs for performing the methods according to the present invention.
[0066] It should also be noted that in the systems, apparatus, and methods of the present invention, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered equivalents of the present invention. The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the invention. Therefore, the invention is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.
[0067] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the invention to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.
Claims
1. A method for intelligent material identification based on multimodal perception fusion, characterized in that, include: Multimodal sensing data of the target material is collected synchronously by a terminal scanning device. The multimodal sensing data includes at least two of the following: visible light images, depth images, infrared thermal images, barcode / QR code information, and physical property data. Feature extraction is performed on the multimodal perception data to obtain modal features corresponding to each modality. Then, the modal features are adaptively weighted and fused through a cross-modal attention fusion mechanism to obtain multimodal fused features. The multimodal fusion features are matched with the prototype features of each category in the preset material feature library to determine the category identifier of the target material; Based on the category identifier, retrieve the corresponding material information from the material information database, output the identification result, and trigger the warehousing operation.
2. The method according to claim 1, characterized in that, The cross-modal attention fusion mechanism dynamically adjusts the fusion weights of each modality feature according to the ambient lighting conditions: when the light intensity is lower than a preset threshold, the fusion weights of infrared thermal image features and depth image features are increased. When the light intensity is higher than a preset threshold, the fusion weight of visible light image features is increased.
3. The method according to claim 1, characterized in that, The preset material feature library is a dynamically expandable feature library. When a new material type is added, the fused feature vector of the new material is added to the feature library and the classifier weights are updated through few-shot incremental learning. The few-shot incremental learning is based on a prototype network architecture. For a new material type, only 1 to 3 example images and their corresponding category labels are required.
4. The method according to claim 1, characterized in that, Also includes: For unknown materials not registered in the preset material feature library, extract the visual features of the unknown materials and map them into attribute prediction vectors; The attribute prediction vector is matched with a preset material attribute semantic library for similarity, and the candidate category of the unknown material is determined based on the matching result; wherein the material attribute semantic library contains at least one attribute description among the shape, color, material, size class, weight class and functional category of each material.
5. The method according to claim 1, characterized in that, The feature extraction employs a hybrid attention module, which sequentially performs channel attention weighting and spatial attention weighting. The spatial attention uses deformable convolution enhancement, which enables the convolution kernel to adaptively adjust the sampling position to adapt to changes in the shape and orientation of the material.
6. The method according to claim 1, characterized in that, The category identification adopts a hierarchical retrieval strategy: first, a primary classifier identifies the major category to which the target material belongs; then, based on the major category result, a corresponding secondary classifier is activated to identify the subclass; finally, based on the subclass result, a corresponding tertiary classifier is activated to identify the specific material code. The adaptive retrieval strategy outputs different results based on the identification confidence level: when the confidence level is higher than the first threshold, the specific material is directly matched; when the confidence level is lower than the first threshold but higher than the second threshold, multiple candidate materials are returned; when the confidence level is lower than the second threshold, an active learning request is triggered.
7. The method according to claim 6, characterized in that, The terminal scanning device runs a lightweight student network, while the cloud server runs a high-precision teacher network. When the recognition confidence of the lightweight student network is lower than a preset threshold, the image is uploaded to the cloud for secondary recognition by the teacher network. The output of the teacher network is used to guide the online incremental updates of the student network.
8. The method according to claim 1, characterized in that, Also includes: Generate a category activation heatmap of the target material, which is used to mark key areas for decision-making in the multimodal sensing data; Simultaneously, a structured attribute matching report is generated, which includes a list of matched attribute items and a matching score for each attribute item.
9. A material intelligent identification device based on multimodal perception fusion, characterized in that, include: The acquisition module is used to synchronously acquire multimodal sensing data of the target material through a terminal scanning device. The multimodal sensing data includes at least two of the following: visible light images, depth images, infrared thermal images, barcode / QR code information, and physical property data. The extraction module is used to extract features from the multimodal perception data to obtain modal features corresponding to each modality, and to perform adaptive weighted fusion of the modal features through a cross-modal attention fusion mechanism to obtain multimodal fusion features; The matching module is used to perform similarity matching between the multimodal fusion features and the prototype features of each category in the preset material feature library to determine the category identifier of the target material; The output module is used to retrieve the corresponding material information from the material information database according to the category identifier, output the identification result, and trigger the warehousing operation.
10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for performing the method described in any one of claims 1-8.
