Cigarette equipment data deduplication method, device, equipment, medium and product

Abstract

The invention discloses a cigarette equipment data deduplication method and device, equipment, a medium and a product. The method comprises the following steps: acquiring original monitoring data through a heterogeneous data source of cigarette industrial equipment, and converting the original monitoring data into Bloom adaptation characteristics; selecting a target window matched with the Bloom adaptation feature in a counting type Bloom filter for multi-window parallel processing, and determining a first duplicate removal result of the Bloom adaptation feature; based on the original monitoring data, deep learning adaptation features corresponding to the Bloom adaptation features which are indicated by the first deduplication result and need to be judged continuously are extracted; determining a second duplicate removal result of the deep learning adaptive feature based on pre-judgment of a lightweight deep learning model and secondary verification of a cigarette production business rule; and fusing the first duplicate removal result and the second duplicate removal result to obtain a target duplicate removal result. According to the scheme, through bidirectional optimization of Bloom coarse screening and deep learning fine judgment, the data de-duplication efficiency and de-duplication precision are improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology, and in particular to a method, apparatus, equipment, medium and product for deduplicating data in cigarette equipment. Background Technology

[0002] Currently, deduplication techniques for cigarette manufacturing equipment data can be implemented using a single Bloom filter, but this method suffers from a high false positive rate due to hash collisions when processing high-frequency heterogeneous data. Alternatively, deduplication can be achieved through complex feature extraction using a single deep learning model, but its inference latency on edge devices is too high, failing to meet the real-time deduplication requirements of cigarette production. In other words, existing deduplication technologies cannot balance deduplication efficiency and accuracy in the cigarette manufacturing industry. Summary of the Invention

[0003] This invention provides a method, apparatus, equipment, medium, and product for deduplication of data in cigarette manufacturing equipment, which can balance the efficiency and accuracy of data deduplication in the cigarette manufacturing industry.

[0004] In a first aspect, embodiments of the present invention provide a method for deduplicating data in cigarette manufacturing equipment, comprising:

[0005] Raw monitoring data is obtained from heterogeneous data sources in the cigarette industry equipment, and the raw monitoring data is converted into Bloom adapt features.

[0006] In a counting Bloom filter that processes multiple windows in parallel, a target window matching the Bloom adaptation feature is selected, and the first deduplication result of the Bloom adaptation feature is determined.

[0007] Based on the original monitoring data, extract the deep learning adaptation features corresponding to the Bloom adaptation features that need to be further judged as indicated by the first deduplication result.

[0008] Based on the preliminary judgment of the lightweight deep learning model and the secondary verification of the cigarette production business rules, the second deduplication result of the deep learning adaptation feature is determined.

[0009] The first deduplication result and the second deduplication result are combined to obtain the target deduplication result.

[0010] Secondly, embodiments of the present invention provide a data deduplication device for cigarette manufacturing equipment, comprising:

[0011] The first processing module is used to acquire raw monitoring data through heterogeneous data sources of cigarette manufacturing equipment and convert the raw monitoring data into Bloom adapt features.

[0012] The second processing module is used to select the target window that matches the Bloom adapt feature in the counting Bloom filter that processes multiple windows in parallel, and to determine the first deduplication result of the Bloom adapt feature.

[0013] The third processing module is used to extract the deep learning adaptation features corresponding to the Bloom adaptation features that need to be further judged based on the original monitoring data indicated by the first deduplication result.

[0014] The fourth processing module is used to determine the second deduplication result of the deep learning adaptation feature based on the pre-judgment of the lightweight deep learning model and the secondary verification of the cigarette production business rules.

[0015] The fifth processing module is used to merge the first deduplication result and the second deduplication result to obtain the target deduplication result.

[0016] Thirdly, embodiments of the present invention provide an electronic device, including:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method as described in the first aspect.

[0020] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions that cause a processor to execute the method described in the first aspect.

[0021] Fifthly, embodiments of the present invention provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the method described in the first aspect.

[0022] The technical solution of this invention obtains raw monitoring data from heterogeneous data sources in the cigarette industry and converts the raw monitoring data into Bloom adaptive features. In a multi-window parallel processing counting Bloom filter, a target window matching the Bloom adaptive features is selected to determine the first deduplication result of the Bloom adaptive features. Based on the raw monitoring data, deep learning adaptive features corresponding to the Bloom adaptive features that require further judgment according to the first deduplication result are extracted. Based on the pre-judgment of a lightweight deep learning model and the secondary verification of cigarette production business rules, a second deduplication result of the deep learning adaptive features is determined. The first deduplication result and the second deduplication result are fused to obtain the target deduplication result. This solution achieves rapid coarse screening of explicit duplicate data through a multi-window counting Bloom filter, reducing the processing load of deep learning. Through the semantic analysis capabilities of lightweight deep learning, it can identify implicit duplicate data that is difficult to process with rapid coarse screening under limited computing power. Combined with the secondary verification of cigarette production business rules to adapt to actual business needs, the false judgment rate is reduced. Through the bidirectional optimization of Bloom coarse screening and deep learning fine judgment, the deduplication efficiency and accuracy are improved.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a method for deduplicating data in cigarette making equipment according to Embodiment 1 of the present invention;

[0026] Figure 2 This is a flowchart of a method for deduplicating data in cigarette making equipment according to Embodiment 2 of the present invention;

[0027] Figure 3 This is a schematic diagram of the structure of a data deduplication device for cigarette equipment according to Embodiment 3 of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of an electronic device that implements an embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] Example 1

[0032] Figure 1 This is a flowchart of a method for deduplicating data in cigarette manufacturing equipment according to Embodiment 1 of the present invention. This embodiment is applicable to situations involving deduplication of data in cigarette manufacturing equipment. The method can be executed by a data deduplication device for cigarette manufacturing equipment, which can be implemented in software and / or hardware and integrated into an electronic device. Furthermore, the electronic device includes, but is not limited to, computers, laptops, smartphones, servers, etc.

[0033] like Figure 1 As shown, the method includes:

[0034] S110. Obtain raw monitoring data through heterogeneous data sources of cigarette manufacturing equipment, and convert the raw monitoring data into Bloom adapt features.

[0035] Heterogeneous data sources for cigarette manufacturing equipment can be a collection of IoT devices of different types and protocols used in the cigarette production process, including but not limited to cigarette making machines, packaging machines, process sensors, and programmable logic controllers (PLCs). Raw monitoring data can be collected from these IoT devices, as shown in Table 1 below.

[0036]

[0037] Among them, Modbus is a widely used industrial serial communication protocol; Modbus Transmission Control Protocol (TCP) is an Ethernet-based transmission protocol; Modbus Remote Terminal Unit (RTU) is a binary transmission protocol based on a serial bus; OPC UA is an Open Platform Communications Unified Architecture; Profinet can be understood as a process field network; LoRa can be understood as a long-distance wireless communication technology.

[0038] The acquisition of raw monitoring data can be achieved through a multi-protocol compatible acquisition gateway. The acquisition strategy can be directional acquisition based on "device identifier + measuring point identifier". For high-frequency data, a "batch acquisition + frame synchronization" mechanism is adopted, with batch reading once every 10ms to avoid data loss. Optionally, the acquired data can be encapsulated into structured data frames of "device identifier - measuring point identifier - timestamp - data content - protocol type - acquisition location", with timestamp accuracy down to the millisecond level to meet the real-time monitoring needs of cigarette production.

[0039] The raw monitoring data undergoes preprocessing, including: noise filtering: removing transmission errors such as ASCII garbled characters and abnormal extreme values; removing invalid values ​​based on the three-standard-deviation principle, such as values ​​where the winding machine temperature exceeds the normal operating range of 50℃±10℃; missing value completion: for occasional missing data from sensors, interpolation using the same type of measuring points in the equipment is used to complete the missing data. For example, if a winding machine is configured with 3 temperature sensors, when one is missing, the average of the other 2 is used to fill the gap, avoiding deviations caused by single interpolation; format standardization: converting numerical data from different protocols to 64-bit floating-point type, converting text data to 8-bit universal multi-byte encoding, and unifying the timestamp to the system timestamp.

[0040] The preprocessed raw monitoring data is converted into Bloom adapt features, which are features adapted to the counting Bloom filter for rapid coarse screening. A Bloom adapt feature is a triple consisting of "device identifier + protocol type + core data field hash value," with a total length of less than 64 bytes, ensuring low computational overhead for the counting Bloom filter. The core data fields for each device can be fields from Table 1 or fields further filtered from Table 1; there are no restrictions here. Hash calculation is performed on the combined values ​​of each core data field to avoid misjudgments caused by hash collisions of a single field. For example, if the device identifier is JZJ-01 (i.e., roll-to-roll machine 01), the protocol type is OPC UA, and the core data field hash value is 0x7A3F2D1E, the corresponding Bloom adapt feature is "JZJ-01-OPCUA-7A3F2D1E".

[0041] S120. In the counting Bloom filter with multi-window parallel processing, select the target window that matches the Bloom adaptation feature and determine the first deduplication result of the Bloom adaptation feature.

[0042] Based on practical application requirements, the counting Bloom filter is designed with multiple windows that process independently and in parallel. The division of multiple windows can be based on factors such as data duplication rate, and is not limited here. Based on practical application requirements, the number of hash functions used for mapping is set for each window of the counting Bloom filter, and is not limited here. The execution logic of each window is basically the same, that is, for any window, if the counters corresponding to multiple hash mapping bits of the Bloom adapt feature are all greater than or equal to 1, then the Bloom adapt feature is determined to be suspected duplicate data, and deep learning is required to continue the determination; otherwise, it is determined to be non-duplicate data, the data is directly retained and the counter value of the window is updated, that is, the counter is incremented by 1.

[0043] Based on the Bloom fit features and the multi-window partitioning criteria, the Bloom fit features are distributed to the target windows that match them within the multi-window partitioning. Within the target windows, the Bloom fit features are repeatedly checked according to the aforementioned processing logic to obtain the first deduplication result. The first deduplication result indicates the Bloom fit features that need to proceed to deep learning for further evaluation, and also indicates the Bloom fit features that do not need to proceed to deep learning for further evaluation.

[0044] S130. Based on the original monitoring data, extract the deep learning adaptation features corresponding to the Bloom adaptation features that need to be further determined according to the first deduplication result.

[0045] For the Bloom fitting features that the first deduplication result indicates need to proceed to deep learning for further judgment, deep learning fitting features are extracted from the data corresponding to these Bloom fitting features in the original monitoring data and some production-related basic parameters. Deep learning fitting features are features that are adapted to deep learning processing and are used for fine judgment, as shown in Table 2 below.

[0046]

[0047] The three types of features in Table 2—structured features, temporal features, and semantic features—can be concatenated to form a unified input feature vector for deep learning. It should be noted that numerical features can be mapped to the [0,1] interval using max-min normalization to avoid the impact of dimensional differences on model training; temporal features can be processed using difference (current value - previous value) to highlight data change trends; and semantic features can be encoded using one-hot encoding or embedding encoding.

[0048] S140. Based on the pre-judgment of the lightweight deep learning model and the secondary verification of the cigarette production business rules, determine the second deduplication result of the deep learning adaptation feature.

[0049] The lightweight deep learning model can be a deep learning model based on deep separable convolution + attention mechanism. Its model optimization includes: quantization: using 8-bit integer data type (INT8) for integer quantization, converting 32-bit floating-point parameters to 8-bit integers, reducing the number of model parameters and improving inference speed; pruning optimization: removing redundant channels in the convolutional layers with weights having an absolute value less than 0.01, with a pruning rate of 30%, further reducing computational overhead; inference performance: inference latency for a single edge data point is ≤30ms, meeting real-time requirements. The model structure of the lightweight deep learning model is shown in Table 3 below.

[0050]

[0051] A lightweight deep learning model is used for preliminary judgment. The deep learning-adapted features are input into the lightweight deep learning model, which outputs a multi-dimensional feature vector. The similarity between the output multi-dimensional feature vector and the historical feature vector is calculated to determine whether it is suspected duplicate data. If so, the cigarette production business rules are superimposed for secondary verification to avoid semantic misjudgment. For data that is suspected of being duplicate data based on similarity, if the secondary verification of the cigarette production business rules passes, it is confirmed as duplicate data and discarded. Otherwise, it is retained as valid data, resulting in the second deduplication result.

[0052] It should be noted that the secondary verification of cigarette production business rules involves embedding business logic into the production scenario to perform a semantic-level secondary verification on suspected duplicate data identified through similarity checks. This aims to distinguish between valid data with similar values ​​but different semantics and duplicate data with differing values ​​but identical semantics, fundamentally avoiding semantic misjudgments inherent in purely algorithmic decisions. A successful secondary verification confirms the data as duplicate. Cigarette production business rules can be dynamically adjusted as cigarette production operations change; this is not limited here.

[0053] S150. Merge the first deduplication result and the second deduplication result to obtain the target deduplication result.

[0054] The data corresponding to the Bloom fitting features indicated by the first deduplication result that do not need to be further determined by deep learning, and the valid data indicated by the second deduplication result, are retained as the final data after deduplication; the duplicate data indicated by the second deduplication result are retained as the duplicate data in the final deduplication determination; the target deduplication result is obtained by combining the above results.

[0055] The technical solution of this invention obtains raw monitoring data from heterogeneous data sources in the cigarette industry and converts the raw monitoring data into Bloom adaptive features. In a multi-window parallel processing counting Bloom filter, a target window matching the Bloom adaptive features is selected to determine the first deduplication result of the Bloom adaptive features. Based on the raw monitoring data, deep learning adaptive features corresponding to the Bloom adaptive features that require further judgment according to the first deduplication result are extracted. Based on the pre-judgment of a lightweight deep learning model and the secondary verification of cigarette production business rules, a second deduplication result of the deep learning adaptive features is determined. The first deduplication result and the second deduplication result are fused to obtain the target deduplication result. This solution achieves rapid coarse screening of explicit duplicate data through a multi-window counting Bloom filter, reducing the processing load of deep learning. Through the semantic analysis capabilities of lightweight deep learning, it can identify implicit duplicate data that is difficult to process with rapid coarse screening under limited computing power. Combined with the secondary verification of cigarette production business rules to adapt to actual business needs, the false judgment rate is reduced. Through the bidirectional optimization of Bloom coarse screening and deep learning fine judgment, the deduplication efficiency and accuracy are improved.

[0056] Example 2

[0057] Figure 2This is a flowchart of a data deduplication method for cigarette manufacturing equipment according to Embodiment 2 of the present invention. This embodiment, based on Embodiment 1, further refines the first deduplication result of the Bloom adaptor feature by selecting the target window matching the Bloom adaptor feature in a multi-window parallel processing counting Bloom filter; and further refines the second deduplication result of the deep learning adaptor feature by performing pre-judgment based on a lightweight deep learning model and secondary verification of cigarette production business rules. Figure 2 As shown, the method includes:

[0058] S110. Obtain raw monitoring data through heterogeneous data sources of cigarette manufacturing equipment, and convert the raw monitoring data into Bloom adapt features.

[0059] S121. In the counting Bloom filter that processes multiple windows in parallel, select the window that matches the data source type and data repetition rate of the Bloom adaptation feature as the target window.

[0060] To address the differences in data repetition cycles across different devices, three independent windows are set up for parallel processing. For example, data with a repetition rate > 60% is assigned to the high repetition rate window, data with a repetition rate < 20% is assigned to the low repetition rate window, and the rest are assigned to the medium repetition rate window.

[0061] Based on the window division criteria, the data source type (i.e., device type) of the Bloom adapter feature, and the data repetition rate, the target window for Bloom adapter feature matching is determined. For example, the target window corresponding to PLC control instructions is a high repetition rate window, and the target window corresponding to temperature sensor data is a low repetition rate window.

[0062] S122. In the target window, determine whether the counters corresponding to the multiple hash mapping bits of the Bloom adapt feature are all greater than or equal to 1. If so, continue to determine; otherwise, no further determination is needed. Determine the first deduplication result of the Bloom adapt feature.

[0063] For example, if the target window has 5 hash functions, and the counters corresponding to the 5 hash mapping bits of the Bloom adapt feature are all greater than or equal to 1, then further judgment is required; otherwise, further judgment is not required.

[0064] In one embodiment, the counting Bloom filter is configured as follows:

[0065] Each window adopts a dual structure of bit array and counter array. The length of the bit array is determined according to the expected amount of data in the window and the preset initial screening false judgment rate. The number of hash mapping bits is determined according to the length of the bit array and the expected amount of data in the window.

[0066] Each window dynamically adjusts the counter decay rate according to the data repetition rate, and periodically decreases the counter value according to the counter decay rate.

[0067] The length of the bit array is dynamically adjusted for each window based on the average occupancy rate of the counter.

[0068] The combined memory usage of all windows must be lower than the set memory threshold.

[0069] Each window employs a dual structure of bit array and counter array. The bit array length is m (dynamically adjustable), and each bit corresponds to a 16-bit counter (supporting a maximum repetition count of 65535, meeting the high-frequency repetition requirements of cigarette-making equipment). The bit array length m is determined by... Determine that n is the expected amount of data within the window, and p is the preset initial screening false positive rate, such as 3%. The number of hash functions k (i.e., the number of hash mapping bits) is determined by... Confirmed. The initial value of the counter can be 3. This addresses the limitation of traditional Bloom filters in deleting expired data by achieving implicit data deletion through counter decay, thus adapting to the time-series nature of data in the cigarette industry.

[0070] Each window dynamically adjusts its counter decay rate based on the data duplication rate. For example, if the data duplication rate of a window is >60%, the counter decay rate is adjusted to 0.8 times / second (to accelerate the elimination of expired duplicate data); if the data duplication rate of a window is <20%, the counter decay rate is adjusted to 1.2 times / second (to prevent valid data from being accidentally deleted). The edge device (i.e., the electronic device in this invention) has a built-in system clock that periodically decrements the counter value according to the counter decay rate of each window. When the counter value drops to 0, the flag of the corresponding bit group is released, indicating that the data has expired.

[0071] Each window dynamically adjusts its bit array length based on the average occupancy rate of its counter. If the average occupancy rate of a window's counter exceeds 80%, it indicates a surge in data volume. The bit array length of that window is automatically doubled to avoid an increase in the probability of hash collisions.

[0072] The combined memory usage of each window must be lower than the set memory threshold, such as the total memory usage of 3 windows being ≤80MB, so that the edge can be easily handled.

[0073] S130. Based on the original monitoring data, extract the deep learning adaptation features corresponding to the Bloom adaptation features that need to be further determined according to the first deduplication result.

[0074] S141. Input the deep learning adaptation features into the lightweight deep learning model and output a multi-dimensional feature vector. The multi-dimensional feature vector is the 128-dimensional feature vector output by the output layer.

[0075] S142. Determine the similarity between the multidimensional feature vector and the historical feature vector in the vector cache pool.

[0076] The edge can maintain a 10-minute valid feature vector cache pool, storing 128-dimensional feature vectors of recently valid data. The cache pool capacity is ≤100,000 entries (occupying ≤128×100,000×4 bytes=51.2MB of memory). It adopts the Least Recently Used (LRU) eviction policy to automatically delete expired vectors.

[0077] Cosine similarity (strong anti-interference capability, adaptable to small fluctuations in industrial data) is used to calculate the similarity S between the newly output multidimensional feature vector and the historical feature vectors in the vector cache pool. The formula is as follows: , where a is the new output multidimensional feature vector and b is the historical feature vector.

[0078] S143. Determine whether the determined similarity exceeds the corresponding similarity threshold. If so, perform a second verification using the cigarette production business rules. Otherwise, no second verification is required, and the second deduplication result is obtained.

[0079] The similarity threshold can be dynamically set based on data importance. For example, the similarity threshold for fault alarm-related data is 0.95, for process control-related data it is 0.92, and for ordinary sensor data it is 0.90. No specific limit is set here. If the determined similarity exceeds the corresponding similarity threshold, it indicates suspected duplicate data, requiring secondary verification using cigarette production business rules; otherwise, secondary verification is unnecessary.

[0080] The rules for cigarette production operations may include the following: Rule 1: Equipment identification + data collection location consistency verification, that is, similar data with different equipment identifications or different data collection locations are considered valid (e.g., the temperature data of the cigarette making machine of JZJ-01 and JZJ-02 are similar and are not considered duplicates); Rule 2: Production batch verification, that is, similar data of different production batches are considered valid (e.g., the packaging quality data of batches B1 and B2 are similar and need to be retained for batch traceability); Rule 3: Equipment operation status verification, that is, the first similar data reported after the equipment changes from "stop" to "run" is considered valid (to avoid accidentally deleting the startup initialization data).

[0081] Data with similarity exceeding the threshold and passing the cigarette production business rule verification (i.e. invalid data) are identified as duplicate data and discarded; otherwise, they are identified as valid data, retained, and added to the cache pool to obtain the second deduplication result.

[0082] S150. Merge the first deduplication result and the second deduplication result to obtain the target deduplication result.

[0083] In one embodiment, the method further includes:

[0084] If the number of new deep learning adaptation features exceeds a set threshold, or if the misjudgment rate of the pre-judgment using the lightweight deep learning model exceeds a set misjudgment rate threshold, the lightweight deep learning model will be locally updated through an incremental training mechanism.

[0085] Deduplication logs are periodically uploaded to the cloud to enable global updates of the lightweight deep learning model in the cloud.

[0086] The lightweight deep learning model local update is as follows: When the edge accumulates ≥1000 new samples or the misclassification rate of deep learning precision judgment is ≥0.5% (misclassification rate = misclassified data volume / total processed data volume), a local update of the model is triggered; the edge automatically collects "valid data + misclassified data" (labeled as positive and negative samples) and divides them into training set and validation set in a 7:3 ratio; mini-batch gradient descent (batch size = 32), learning rate = 0.001, iterates for 10 rounds, and only updates the parameters of the model's attention module, global average pooling layer and output layer (to avoid the high computational cost of full training); after training is completed, the edge replaces the old model with the new model parameters. The whole process takes ≤30 minutes and does not affect production operation.

[0087] The lightweight deep learning model global update is as follows: During daily off-peak production periods, the edge device uploads a full deduplication log (including original data, feature vectors, judgment results, misclassification labels, etc.) to the cloud. The cloud uses the TensorFlow framework to train the model offline based on the data contained in the full deduplication log, optimizing network weights and attention module parameters to improve the model's generalization ability. The cloud distributes optimized model parameters to the edge device once a month. The edge device updates the parameters during production breaks (such as equipment maintenance) to ensure that the model is adaptable to all scenarios. After receiving the optimized parameters from the cloud, the edge device verifies the effect through small-batch verification (selecting 1000 real-time data). If the deduplication accuracy improves by ≥1% and the latency does not increase, the new parameters are officially enabled. If the accuracy decreases or the latency exceeds 50ms, the device automatically rolls back to the original parameters and reports the anomaly to the cloud, forming a closed loop of distribution-verification-feedback-re-optimization.

[0088] In one embodiment, the method further includes:

[0089] Based on the false positive rate of the pre-judgment using the lightweight deep learning model, the bit array length and the number of hash mapping bits of the counting Bloom filter are dynamically adjusted.

[0090] The parameters of the counting Bloom filter are dynamically adjusted. Specifically: based on the deep learning-based fine-judgment period (e.g., every 5 minutes), the coarse screening false positive rate (i.e., the proportion of data that the counting Bloom filter classifies as duplicates but is actually valid) is fed back to the counting Bloom filter. If the coarse screening false positive rate > 5%, the bit array length is doubled or the number of hash functions is increased (i.e., the number of hash mapping bits, +1, maximum 8) to reduce the probability of hash collisions. If the coarse screening false positive rate < 1% and memory usage > 80MB, the bit array length is halved or the number of hash functions is reduced (-1, minimum 4) to optimize memory resources. The adjustment constraints are that the bit array length ranges from 2^16 to 2^20 bits, and the number of hash functions ranges from 4 to 8, to avoid performance degradation caused by extreme parameter settings.

[0091] In one embodiment, the method further includes: monitoring CPU utilization, memory usage, and network bandwidth. If CPU utilization > 80% or memory usage > 90%, incremental training is temporarily disabled to prioritize real-time deduplication; if network bandwidth < 1 Mbps, the frequency of parameter adjustment for the counting Bloom filter is reduced (from once every 5 minutes to once every 15 minutes) to reduce data transmission overhead; the above degradation strategy does not affect the core deduplication function, but only suspends non-real-time optimization tasks to ensure production continuity.

[0092] The technical solution of this invention uses a combination of counting-type and multi-window Bloom filters for coarse screening, filtering 80%-90% of explicit duplicate data (such as network retransmission packets and high-frequency repeated sensor sampling values) with low memory and low latency, reducing the processing pressure on deep learning. Through a lightweight deep learning model and business rule fusion, it accurately determines the suspected duplicate data (approximately 10%-20% of the total data) that requires further judgment after being filtered by the counting-type Bloom filter, identifying implicit duplicate data (such as protocol field variations and data with similar temporal characteristics but different semantics), controlling the overall false positive rate to ≤0.5%. Through bidirectional feedback between the edge and cloud, it dynamically adjusts the parameters of the counting-type Bloom filter and the deep learning model to adapt to the dynamic changes in cigarette industry data.

[0093] This invention effectively solves the problem of the trade-off between real-time performance, accuracy, and dynamic adaptability in data deduplication. The measured edge processing latency is ≤38ms, the deduplication accuracy is ≥98.7%, and the false positive rate is ≤0.32%. This method is applicable to real-time data deduplication in core cigarette equipment such as cigarette making machines and packaging machines, and can be extended to other process industries.

[0094] Example 3

[0095] Figure 3 This is a schematic diagram of a data deduplication device for cigarette equipment according to Embodiment 3 of the present invention. This embodiment is applicable to situations where data deduplication of cigarette equipment is implemented, such as... Figure 3As shown, the specific structure of the device includes:

[0096] The first processing module 31 is used to acquire raw monitoring data through heterogeneous data sources of cigarette industrial equipment and convert the raw monitoring data into Bloom adapt features.

[0097] The second processing module 32 is used to select the target window that matches the Bloom adapt feature in the counting Bloom filter that processes multiple windows in parallel, and to determine the first deduplication result of the Bloom adapt feature.

[0098] The third processing module 33 is used to extract, based on the original monitoring data, the deep learning adaptation features corresponding to the Bloom adaptation features that need to be further judged according to the first deduplication result;

[0099] The fourth processing module 34 is used to determine the second deduplication result of the deep learning adaptation feature based on the pre-judgment of the lightweight deep learning model and the secondary verification of the cigarette production business rules.

[0100] The fifth processing module 35 is used to merge the first deduplication result and the second deduplication result to obtain the target deduplication result.

[0101] The deduplication device for cigarette equipment data provided in this embodiment obtains original monitoring data from heterogeneous data sources of cigarette industrial equipment through a first processing module and converts the original monitoring data into Bloom adaptive features. A second processing module selects a target window matching the Bloom adaptive features from a multi-window parallel processing counting Bloom filter to determine the first deduplication result of the Bloom adaptive features. A third processing module extracts deep learning adaptive features corresponding to the Bloom adaptive features that require further evaluation based on the original monitoring data, as indicated by the first deduplication result. A fourth processing module determines the second deduplication result of the deep learning adaptive features based on the pre-judgment of a lightweight deep learning model and the secondary verification of cigarette production business rules. A fifth processing module merges the first deduplication result and the second deduplication result to obtain the target deduplication result. This solution uses a multi-window counting Bloom filter to quickly screen explicit duplicate data, reducing the processing load of deep learning. By leveraging the semantic analysis capabilities of lightweight deep learning, it can identify implicit duplicate data that is difficult to process with rapid screening, even with limited computing power. Combined with secondary verification of cigarette production business rules, it adapts to actual business needs and reduces the false positive rate. Through bidirectional optimization of Bloom screening and deep learning for fine-tuning, it improves deduplication efficiency and accuracy.

[0102] Furthermore, the second processing module 32 is specifically used for:

[0103] In a counting Bloom filter that processes multiple windows in parallel, a window that matches the data source type and data repetition rate of the Bloom adaptation features is selected as the target window.

[0104] In the target window, it is determined whether the counters corresponding to the multiple hash mapping bits of the Bloom adapt feature are all greater than or equal to 1. If so, further determination is required; otherwise, no further determination is required, and the first deduplication result of the Bloom adapt feature is determined.

[0105] Furthermore, the counting type Bloom filter is configured as follows:

[0106] Each window adopts a dual structure of bit array and counter array. The length of the bit array is determined according to the expected amount of data in the window and the preset initial screening misjudgment rate. The number of hash mapping bits is determined according to the length of the bit array and the expected amount of data in the window.

[0107] Each window dynamically adjusts the counter decay rate according to the data repetition rate, and periodically decreases the counter value according to the counter decay rate.

[0108] The length of the bit array is dynamically adjusted for each window based on the average occupancy rate of the counter.

[0109] The combined memory usage of all windows must be lower than the set memory threshold.

[0110] Furthermore, the fourth processing module 34 is specifically used for:

[0111] The deep learning adaptation features are input into the lightweight deep learning model, and a multi-dimensional feature vector is output.

[0112] Determine the similarity between the multidimensional feature vector and the historical feature vectors in the vector cache pool;

[0113] Determine whether the determined similarity exceeds the corresponding similarity threshold. If so, perform a second verification using the cigarette production business rules; otherwise, no second verification is required, and the second deduplication result is obtained.

[0114] Furthermore, the device also includes:

[0115] The local update module is used to locally update the lightweight deep learning model through an incremental training mechanism when the number of new deep learning adaptation features exceeds a set threshold, or the misjudgment rate of the pre-judgment using the lightweight deep learning model exceeds a set misjudgment rate threshold.

[0116] The global update module is used to periodically upload deduplication logs to the cloud so that the lightweight deep learning model can be globally updated in the cloud.

[0117] Furthermore, the device also includes:

[0118] The filter parameter update module is used to dynamically adjust the bit array length and the number of hash mapping bits of the counting Bloom filter based on the false positive rate of the pre-judgment of the lightweight deep learning model.

[0119] The deduplication device for cigarette equipment data provided in this embodiment of the invention can execute the deduplication method for cigarette equipment data provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0120] Example 4

[0121] Figure 4 This is a schematic diagram of the structure of an electronic device implementing embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0122] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 performs various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0123] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0124] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the deduplication method for cigarette equipment data.

[0125] In some embodiments, the cigarette device data deduplication method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the cigarette device data deduplication method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the cigarette device data deduplication method by any other suitable means (e.g., by means of firmware).

[0126] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0128] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0130] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0131] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0132] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and no limitation is imposed herein.

[0133] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for deduplicating data in cigarette manufacturing equipment, characterized in that, include: Raw monitoring data is obtained from heterogeneous data sources in the cigarette industry equipment, and the raw monitoring data is converted into Bloom adapt features. In a counting Bloom filter that processes multiple windows in parallel, a target window matching the Bloom adaptation feature is selected, and the first deduplication result of the Bloom adaptation feature is determined. Based on the original monitoring data, extract the deep learning adaptation features corresponding to the Bloom adaptation features that need to be further judged as indicated by the first deduplication result. Based on the preliminary judgment of the lightweight deep learning model and the secondary verification of the cigarette production business rules, the second deduplication result of the deep learning adaptation feature is determined. The first deduplication result and the second deduplication result are combined to obtain the target deduplication result.

2. The method according to claim 1, characterized in that, In a counting Bloom filter with multi-window parallel processing, the target window for matching the Bloom adaptation features is selected, and the first deduplication result of the Bloom adaptation features is determined, including: In a counting Bloom filter that processes multiple windows in parallel, a window that matches the data source type and data repetition rate of the Bloom adaptation features is selected as the target window. In the target window, it is determined whether the counters corresponding to the multiple hash mapping bits of the Bloom adapt feature are all greater than or equal to 1. If so, further determination is required; otherwise, no further determination is required, and the first deduplication result of the Bloom adapt feature is determined.

3. The method according to claim 2, characterized in that, The counting type Bloom filter is configured as follows: Each window adopts a dual structure of bit array and counter array. The length of the bit array is determined according to the expected amount of data in the window and the preset initial screening false judgment rate. The number of hash mapping bits is determined according to the length of the bit array and the expected amount of data in the window. Each window dynamically adjusts the counter decay rate according to the data repetition rate, and periodically decreases the counter value according to the counter decay rate. The length of the bit array is dynamically adjusted for each window based on the average occupancy rate of the counter. The combined memory usage of all windows must be lower than the set memory threshold.

4. The method according to claim 1, characterized in that, Based on the pre-judgment of the lightweight deep learning model and the secondary verification of the cigarette production business rules, the second deduplication result of the deep learning adaptation features is determined, including: The deep learning adaptation features are input into the lightweight deep learning model, and a multi-dimensional feature vector is output. Determine the similarity between the multidimensional feature vector and the historical feature vectors in the vector cache pool; Determine whether the determined similarity exceeds the corresponding similarity threshold. If so, perform a second verification using the cigarette production business rules; otherwise, no second verification is required, and the second deduplication result is obtained.

5. The method according to claim 1, characterized in that, Also includes: If the number of new deep learning adaptation features exceeds a set threshold, or if the misjudgment rate of the pre-judgment using the lightweight deep learning model exceeds a set misjudgment rate threshold, the lightweight deep learning model will be locally updated through an incremental training mechanism. Deduplication logs are periodically uploaded to the cloud to enable global updates of the lightweight deep learning model in the cloud.

6. The method according to claim 1, characterized in that, Also includes: Based on the false positive rate of the pre-judgment using the lightweight deep learning model, the bit array length and the number of hash mapping bits of the counting Bloom filter are dynamically adjusted.

7. A data deduplication device for cigarette manufacturing equipment, characterized in that, include: The first processing module is used to acquire raw monitoring data through heterogeneous data sources of cigarette manufacturing equipment and convert the raw monitoring data into Bloom adapt features. The second processing module is used to select the target window that matches the Bloom adapt feature in the counting Bloom filter that processes multiple windows in parallel, and to determine the first deduplication result of the Bloom adapt feature. The third processing module is used to extract the deep learning adaptation features corresponding to the Bloom adaptation features that need to be further judged based on the original monitoring data indicated by the first deduplication result. The fourth processing module is used to determine the second deduplication result of the deep learning adaptation feature based on the pre-judgment of the lightweight deep learning model and the secondary verification of the cigarette production business rules. The fifth processing module is used to merge the first deduplication result and the second deduplication result to obtain the target deduplication result.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-6.

Images