Tobacco leaf foreign matter intelligent identification method based on multi-modal fusion

Abstract

The invention relates to the technical field of foreign matter recognition, in particular to an intelligent tobacco foreign matter recognition method based on multi-modal fusion, which comprises the following steps: acquiring historical tobacco foreign matter data in the production process of a cigarette factory, and constructing a tobacco foreign matter labeling data set and a tobacco foreign matter recognition logical reasoning rule set based on the historical tobacco foreign matter data; constructing a tobacco foreign matter intelligent identification model based on the tobacco foreign matter identification logical reasoning rule set and a machine learning algorithm; identifying the current tobacco foreign matter data based on the tobacco foreign matter intelligent identification model; and strategy optimization is carried out on the tobacco leaf foreign matter intelligent identification model based on the identification result and the optimization algorithm. The problems of low identification accuracy and low identification efficiency in the prior art are solved.

Description

Technical Field

[0001] This invention relates to the field of foreign object recognition technology, and in particular to an intelligent method for identifying foreign objects in tobacco leaves based on multimodal fusion. Background Technology

[0002] Intelligent foreign object identification methods for tobacco leaves are key to improving cigarette production quality. Tobacco leaves come from diverse sources and inevitably contain foreign objects such as hemp rope, wood chips, foam, paper scraps, and feathers. If these cannot be removed, they will pose significant safety and quality risks to cigarette processing and production. Tobacco leaf impurity removal is an important part of cigarette production, and foreign object identification is the key to the impurity removal process.

[0003] Currently, the main methods for removing foreign objects from tobacco leaves in the tobacco processing stage include wind separation, photoelectric, and machine vision recognition. Although these methods can remove foreign objects from tobacco leaves to a large extent, they still generally suffer from problems such as limited supporting technologies for foreign object recognition, limited recognition capacity, resulting in low recognition accuracy and low recognition efficiency, which cannot meet the needs for higher accuracy and efficiency in recognition. Summary of the Invention

[0004] The purpose of this invention is to provide a method for intelligent identification of foreign objects in tobacco leaves based on multimodal fusion, which solves the problems of low identification accuracy and low identification efficiency in existing technologies.

[0005] To achieve the above objectives, this invention provides a method for intelligent identification of foreign objects in tobacco leaves based on multimodal fusion, comprising the following steps: S1. Obtain historical tobacco foreign object data during the cigarette factory production process, and construct a tobacco foreign object annotation dataset and a tobacco foreign object identification logic reasoning rule set based on the historical tobacco foreign object data; S2. Construct an intelligent identification model for foreign objects in tobacco leaves based on a set of logical reasoning rules for foreign object identification in tobacco leaves and machine learning algorithms; S3. Identify foreign objects in the current tobacco leaves based on the intelligent identification model for foreign objects in tobacco leaves; S4. Optimize the strategy of the intelligent foreign object identification model for tobacco leaves based on the identification results and optimization algorithm.

[0006] In some embodiments of this application, in S1, constructing a tobacco foreign object annotation dataset based on historical tobacco foreign object data includes: The foreign matter data in tobacco leaves were manually labeled using Labelme to obtain the basic dataset of foreign matter in tobacco leaves; An automatic labeling algorithm for tobacco foreign objects is generated based on transfer learning algorithm and basic dataset of tobacco foreign objects, and the automatic labeling of tobacco foreign objects is performed. A tobacco foreign object annotation dataset was constructed based on the basic dataset of tobacco foreign objects and the results of automatic labeling of tobacco foreign objects.

[0007] In some embodiments of this application, in S1, constructing the logical reasoning rule set for tobacco foreign object identification includes: logical rules Defined as a triple, including ,in This indicates a judgment on the rule category. For activation functions with logical knowledge, It is passed on and used as supervision information for subsequent models to describe rule information; Introducing a similarity factor T The output probability of each logical rule category is obtained. : ; in, for As Input; Based on preset similarity factors T This yields the output probabilities of the logical rule categories for the computational model and subsequent models: ; in, Given respectively T The output probabilities of the logical rule categories of the computational model and subsequent models; The model loss function is expressed as follows: ; in, These represent the cross-entropy loss of the calculation model and the cross-entropy loss of the subsequent model, respectively. N The number of sub-features.

[0008] In some embodiments of this application, in S2, constructing an intelligent identification model for tobacco foreign objects based on a set of logical reasoning rules for tobacco foreign object identification and a machine learning algorithm includes: S21. Extract from the set of logical reasoning rules for foreign object identification in tobacco leaves N Individual features are used to construct a set of logical reasoning rules. O Sub-feature categories; S22, Based on logical rules and logical rule set Obtain sub-features and obtain the reasoning results. Based on sub-features The corresponding classifier obtains the corresponding classification result. ; S23, Based on reasoning results and classification results Make joint decisions.

[0009] In some embodiments of this application, in S23, according to and Making decisions based on the level of trust includes: when and If the states are identical and all are trustworthy, proceed directly to the next iteration; when... and When the states are different but both are trustworthy, a preset decision formula is used for fusion; when and If the values ​​are different and there is an unreliable state, the calculation will be recalculated immediately.

[0010] In some embodiments of this application, S2 further includes: S24. Conduct a credibility assessment of the intelligent foreign object recognition model for tobacco leaves and establish a system that includes... and Knowledge graph of key parameters and relationships and establish respectively With sub-feature classifier results, With logical rule set, With related parameters and logical rule sets With credibility assessment value The association model, and the calculation based on the association model. .

[0011] S25. Introduce a clustering algorithm into the self-organizing memory module of the model to calculate the representational values ​​of tobacco leaves and each type of foreign matter, and obtain... ROI Correlation with characterization quantities and determination of unlabeled foreign object types; S26. Feed back the unlabeled foreign object types to the dataset, and update the tobacco leaf foreign object intelligent identification model based on the fed-back dataset.

[0012] In some embodiments of this application, in step S4, the strategy optimization of the intelligent identification model for foreign objects in tobacco leaves based on the identification results and optimization algorithms includes: S41. The intelligent identification model for foreign objects in tobacco leaves is compressed based on the embedded optimization algorithm, so that the intelligent identification model for foreign objects in tobacco leaves can be deployed in embedded devices. S42. Optimize the data transmission process based on the GigEVision conversion protocol and SPI serial port protocol.

[0013] The advantages and beneficial effects of this invention compared to the prior art are: 1. By fusing data from multiple modalities, this invention can more comprehensively capture the characteristics of foreign objects in tobacco leaves, thereby improving the accuracy and reliability of identification; through rapid and accurate identification, it reduces equipment failures and downtime caused by foreign objects, and improves equipment utilization and production efficiency.

[0014] 2. This invention utilizes historical tobacco foreign matter data from the cigarette factory production process to construct a dataset and rule set, enabling the model to learn the characteristics of foreign matter under different environments and conditions, thereby improving its applicability in different scenarios.

[0015] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0016] Figure 1 This is a schematic diagram illustrating the steps of a multimodal fusion-based intelligent identification method for foreign objects in tobacco leaves according to an embodiment of the present invention. Detailed Implementation

[0017] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product is in use. They are used only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," and "connect" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0018] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0019] In one embodiment, the technical implementation involves three steps: First, collect foreign matter data of tobacco leaves from cigarette factories, manually annotate the basic foreign matter data of tobacco leaves, design an automatic annotation algorithm to annotate the foreign matter data of tobacco leaves, and extract logical knowledge of foreign matter of tobacco leaves to form a set of reasoning rules; Secondly, a multimodal tobacco foreign object recognition model is designed based on the integration of reasoning and learning. A self-organizing memory module is designed, clustering algorithm is used to discover unlabeled foreign object types, and then the intelligent recognition model of tobacco foreign objects is realized through fusion. Finally, the model was optimized and adapted, the location information of foreign objects was located at high speed, and an embedded optimization algorithm for a multimodal fusion intelligent identification model of tobacco foreign objects was designed to improve the model's adaptability to the environment, increase the model's running speed, and enhance the model's generalization ability.

[0020] like Figure 1 As shown, this invention provides a method for intelligent identification of foreign objects in tobacco leaves based on multimodal fusion, comprising the following steps: S1. Obtain historical tobacco foreign object data during the cigarette factory production process, and construct a tobacco foreign object annotation dataset and a set of logical reasoning rules for tobacco foreign object identification based on the historical tobacco foreign object data.

[0021] S2. Construct an intelligent identification model for foreign objects in tobacco leaves based on a set of logical reasoning rules for foreign object identification and machine learning algorithms.

[0022] S3. Based on the intelligent identification model for foreign objects in tobacco leaves, identify the current foreign object data in tobacco leaves.

[0023] S4. Optimize the strategy of the intelligent foreign object identification model for tobacco leaves based on the identification results and optimization algorithm.

[0024] The beneficial effects of this invention are: 1. By fusing data from multiple modalities, this invention can more comprehensively capture the characteristics of foreign objects in tobacco leaves, thereby improving the accuracy and reliability of identification; through rapid and accurate identification, it reduces equipment failures and downtime caused by foreign objects, and improves equipment utilization and production efficiency.

[0025] 2. This invention utilizes historical tobacco foreign matter data from the cigarette factory production process to construct a dataset and rule set, enabling the model to learn the characteristics of foreign matter under different environments and conditions, thereby improving its applicability in different scenarios.

[0026] In some embodiments of this application, in S1, constructing a tobacco foreign object annotation dataset based on historical tobacco foreign object data includes: The foreign matter data in tobacco leaves were manually labeled using Labelme to obtain the basic dataset of foreign matter in tobacco leaves; An automatic labeling algorithm for tobacco foreign objects is generated based on transfer learning algorithm and basic dataset of tobacco foreign objects, and the automatic labeling of tobacco foreign objects is performed. A tobacco foreign object annotation dataset was constructed based on the basic dataset of tobacco foreign objects and the results of automatic labeling of tobacco foreign objects.

[0027] Specifically, a basic foreign object dataset is obtained by manually labeling foreign objects. With this basic dataset, a transfer learning method is used to improve the designed automatic labeling algorithm for automatic labeling of foreign objects in tobacco leaves. The manually labeled data and the automatically labeled data form the tobacco leaf foreign object labeling dataset, which serves as the data foundation for this invention.

[0028] In some embodiments of this application, in S1, constructing the logical reasoning rule set for tobacco foreign object identification includes: logical rules Defined as a triple, including ,in This indicates a judgment on the rule category. For activation functions with logical knowledge, It is passed on and used as supervision information for subsequent models to describe rule information; Introducing a similarity factor T The output probability of each logical rule category is obtained. : ; in, for As Input; Similarity factor in this method T The selection of the appropriate rule is crucial for successful logical rule extraction, and cross-entropy loss can be used to measure and optimize the objective. Assume a given... T First, calculate the output probabilities of the logical rule categories of the model and subsequent models respectively: ; in, Given respectively T The output probabilities of the logical rule categories of the computational model and subsequent models; The model loss function is expressed as follows: ; in, These represent the cross-entropy loss of the calculation model and the cross-entropy loss of the subsequent model, respectively. N This represents the number of sub-features. By balancing the two values, an approximate degree of similarity can be obtained.

[0029] Specifically, the automatic extraction of logical rules is the process of acquiring "logical knowledge" from data such as training models, data labels, model parameters, and image structures. These rules can then be used for subsequent model training or application. In a narrow sense, "logical knowledge" refers to a certain similarity contained in the data output, which can be used to assist in subsequent model training. In a broader sense, it refers to all usable knowledge and representations within the data, such as features, parameters, and information. Therefore, the extraction process needs to inherit information such as the data's "category," "similarity," and "rule description." Logical rules It can be defined as a triple. >. Among them: This represents the "judgment" of the rule category. These judgments need to be processed by the activation function of the model to obtain the predicted probability of the category, and then the loss of the model is directly calculated. It is an activation function with "logical knowledge". The similarity information in the data is usually the most valuable. In order to better characterize this similarity, a similarity factor is introduced. T Describe it; Used to describe rule information, it can be passed on and used as supervision information for subsequent models.

[0030] Based on the aforementioned set of rules for identifying foreign objects in tobacco leaves, a fusion model is developed using traditional machine learning. Pre-inference and post-inference are incorporated into the machine learning algorithm. Pre-inference uses the rules as input constraints, while post-inference determines whether the identified foreign object matches the rule knowledge, which is then used for the next machine learning iteration. Model construction includes key steps such as rule feature extraction, pre-inference, post-inference, reliability evaluation of inference results, and a self-organizing memory module. The specific steps are as follows: In some embodiments of this application, in S2, constructing an intelligent identification model for tobacco foreign objects based on a set of logical reasoning rules for tobacco foreign object identification and a machine learning algorithm includes: S21. Rule Feature Extraction Process: Logical Rule Set This includes sub-features, which are then connected by building corresponding sub-feature classifiers in subsequent machine learning modules. Sub-features are the correlation features between logical knowledge and labeled data related to the decision objective. Frequently mentioned concepts in the logical knowledge are knowledge features, and the features of the labeled data itself are data features. The features jointly constructed by associating and mapping knowledge features and data features are called sub-features. Suppose we extract from the data Sub-characteristics Then the set of logical rules Sub-feature class based on Build and label data The annotation attributes and categories are respectively... Using this as the classification standard, subsequent classifiers also use it. The data to be classified is used as a standard classification tool, thereby completing the task of extracting rule features; S22, Joint Decision Making Based on Reasoning and Learning: This refers to reasoning outcomes based on rule features. Compared with data-based model training results Joint decision-making. Based on logical rules. and logical rule set Obtain sub-features and obtain the reasoning results. Sub-features Corresponding classifier Obtain the corresponding classification results. Reasoning results and classification results Both results are based on the target features, and joint decision-making can be made based on the two results. S23, Based on reasoning results and classification results Make joint decisions.

[0031] In some embodiments of this application, in S23, according to and Making decisions based on the level of trust includes: when and If the states are identical and all are trustworthy, proceed directly to the next iteration; when... and When the states are different but both are trustworthy, a preset decision formula is used for fusion; when and If the values ​​are different and there is an unreliable state, the calculation will be recalculated immediately.

[0032] In some embodiments of this application, S2 further includes: S24. Since the factors determining whether the reasoning result is credible are tree-structured, knowledge graphs and DS evidence theory are used for evaluation.

[0033] First, construct a knowledge graph. , includes and Key parameters and relationships. Establish them separately. With sub-feature classifier results, With logical rule set, With related parameters and logical rule sets With credibility assessment value Direct formal relationships form a complete logical chain. Secondly, the results of the reasoning are evaluated, i.e., calculated. The process involves identifying a framework that contains all the hypotheses, assigning a probability to each hypothesis, and then performing rule fusion and result determination. DS evidence theory is an imprecise reasoning theory widely used in evidence (data) synthesis.

[0034] S25. Self-organizing memory module: Clustering algorithm is incorporated into the self-organizing memory module of the model to calculate the most representative and distinguishable representations of tobacco leaves and each type of foreign matter. Calculating the relationship between ROI and these representations can discover unlabeled foreign matter types. S26. The self-organizing memory module feeds newly discovered unlabeled foreign objects back into the dataset, enabling timely updates of the dataset and model. In some embodiments of this application, in step S4, the strategy optimization of the intelligent identification model for foreign objects in tobacco leaves based on the identification results and optimization algorithms includes: S41. The intelligent identification model for foreign objects in tobacco leaves is compressed based on the embedded optimization algorithm, so that the intelligent identification model for foreign objects in tobacco leaves can be deployed in embedded devices. S42. Optimize the data transmission process based on the GigEVision conversion protocol and SPI serial port protocol.

[0035] Specifically, the multimodal fusion intelligent identification model for foreign objects in tobacco leaves is optimized to quickly locate the positional information of images, thereby improving the model's operating speed. A high-speed hardware interaction protocol is designed to enhance the model's adaptability to the environment. After optimization, the model can run on embedded devices, facilitating its deployment in different cigarette production impurity removal processes, thus improving the model's applicability.

[0036] The advantages and beneficial effects of this invention compared to the prior art are as follows: The multimodal fusion intelligent identification method for foreign objects in tobacco leaves proposed in this invention can not only efficiently and accurately remove impurities from tobacco leaves, but also extend to other key links in cigarette manufacturing, comprehensively improving the production efficiency and quality of tobacco products.

[0037] In this application, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. In case of any inconsistency, the meaning set forth in this specification or derived from the content described herein shall prevail. Furthermore, the terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit the scope of this application.

[0038] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for intelligent identification of foreign objects in tobacco leaves based on multimodal fusion, characterized in that, Includes the following steps: S1. Obtain historical tobacco foreign object data during the cigarette factory production process, and construct a tobacco foreign object annotation dataset and a tobacco foreign object identification logic reasoning rule set based on the historical tobacco foreign object data; S2. Construct an intelligent identification model for foreign objects in tobacco leaves based on a set of logical reasoning rules for foreign object identification in tobacco leaves and machine learning algorithms; S3. Identify foreign objects in the current tobacco leaves based on the intelligent identification model for foreign objects in tobacco leaves; S4. Optimize the strategy of the intelligent foreign object identification model for tobacco leaves based on the identification results and optimization algorithm.

2. The intelligent identification method for foreign objects in tobacco leaves based on multimodal fusion according to claim 1, characterized in that, In step S1, constructing a tobacco foreign object annotation dataset based on historical tobacco foreign object data includes: The foreign matter data in tobacco leaves were manually labeled using Labelme to obtain the basic dataset of foreign matter in tobacco leaves; An automatic labeling algorithm for tobacco foreign objects is generated based on transfer learning algorithm and basic dataset of tobacco foreign objects, and the automatic labeling of tobacco foreign objects is performed. A tobacco foreign object annotation dataset was constructed based on the basic dataset of tobacco foreign objects and the results of automatic labeling of tobacco foreign objects.

3. The intelligent identification method for foreign objects in tobacco leaves based on multimodal fusion according to claim 2, characterized in that, In step S1, the set of logical reasoning rules for identifying foreign objects in tobacco leaves includes: logical rules Defined as a triple, including ,in This indicates a judgment on the rule category. For activation functions with logical knowledge, It is passed on and used as supervision information for subsequent models to describe rule information; Introducing a similarity factor T The output probability of each logical rule category is obtained. : ； in, for As Input; Based on preset similarity factors T This yields the output probabilities of the logical rule categories for the computational model and subsequent models: ； in, Given respectively T The output probabilities of the logical rule categories of the computational model and subsequent models; The model loss function is expressed as follows: ； in, These represent the cross-entropy loss of the calculation model and the cross-entropy loss of the subsequent model, respectively. N The number of sub-features.

4. The intelligent identification method for foreign objects in tobacco leaves based on multimodal fusion according to claim 3, characterized in that, In step S2, the intelligent identification model for tobacco foreign objects, constructed based on the logical reasoning rule set for foreign object identification and machine learning algorithms, includes: S21. Extract from the set of logical reasoning rules for foreign object identification in tobacco leaves N Individual features are used to construct a set of logical reasoning rules. O Sub-feature categories; S22, Based on logical rules and logical rule set Obtain sub-features and obtain the reasoning results. Based on sub-features The corresponding classifier obtains the corresponding classification result. ; S23, Based on reasoning results and classification results Make joint decisions.

5. The intelligent identification method for foreign objects in tobacco leaves based on multimodal fusion according to claim 4, characterized in that, In S23, according to and Making decisions based on the level of trust includes: when and If the states are identical and all are trustworthy, proceed directly to the next iteration; when... and When the states are different but both are trustworthy, a preset decision formula is used for fusion; when and If the values ​​are different and there is an unreliable state, the calculation will be recalculated immediately.

6. The intelligent identification method for foreign objects in tobacco leaves based on multimodal fusion according to claim 5, characterized in that, S2 also includes: S24. Conduct a credibility assessment of the intelligent foreign object recognition model for tobacco leaves and establish a system that includes... and Knowledge graph of key parameters and relationships and establish respectively With sub-feature classifier results, With logical rule set, With related parameters and logical rule sets With credibility assessment value The association model, and the calculation based on the association model. ; S25. Introduce a clustering algorithm into the self-organizing memory module of the model to calculate the representational values ​​of tobacco leaves and each type of foreign matter, and obtain... ROI Correlation with characterization quantities and determination of unlabeled foreign object types; S26. Feed back the unlabeled foreign object types to the dataset, and update the tobacco leaf foreign object intelligent identification model based on the fed-back dataset.

7. The intelligent identification method for foreign objects in tobacco leaves based on multimodal fusion according to claim 6, characterized in that, In step S4, the strategy optimization of the intelligent identification model for foreign objects in tobacco leaves based on the identification results and optimization algorithm includes: S41. The intelligent identification model for foreign objects in tobacco leaves is compressed based on the embedded optimization algorithm, so that the intelligent identification model for foreign objects in tobacco leaves can be deployed in embedded devices. S42. Optimize the data transmission process based on the GigEVision conversion protocol and SPI serial port protocol.

Images