Multi-source data processing method and system of cigarette detection instrument

Abstract

The application discloses a multi-source data processing method and system of a cigarette detection instrument, relates to the technical field of tobacco, and realizes non-discriminatory access to heterogeneous data sources through a unified interface protocol and automatic connection, avoids the huge workload of customized development and the problem of system rigidity, provides stable real-time data flow for subsequent processing, uniformly converts unstructured original data into standard structured records by using a formatted description template and automatic analysis technology, automatically and efficiently associates multi-instrument data of the same detection event by using a similarity matching algorithm based on local sensitive hashing, applies a weighting aggregation algorithm based on an attention mechanism to give differentiated weights to time series data and fusion, generates a fusion feature matrix that more comprehensively represents the quality state of a sample, performs automatic feature extraction and dimension reduction by using a stacked denoising autoencoder, and finally outputs results in the form of a standardized data packet with complete check information, thereby providing a ready-to-use unified high-quality data basis for upper-layer quality analysis and the like applications.

Description

Technical Field

[0001] This application relates to the field of tobacco technology, and in particular to a multi-source data processing method and system for a cigarette testing instrument. Background Technology

[0002] In the cigarette production process, comprehensive quality testing is required on finished cigarettes and tobacco materials (such as cigarette paper, filter rods, and packaging cartons). This involves a wide variety of testing instruments, including air permeability meters, coefficient of friction meters, colorimeters, electronic balances, and integrated testing platforms. These instruments come from different manufacturers, and their data output methods and protocols vary significantly. Some send data directly via serial ports, some store data in standalone databases (such as Access or FoxPro), some generate test report files in specific formats (such as PDF or XML), and others connect to network databases (such as SQL Server or Oracle).

[0003] Currently, the mainstream solution is to write dedicated data acquisition programs for each specific type of instrument. This approach has significant drawbacks: enormous development workload, high repetition, difficult maintenance, system rigidity, and poor scalability. When a new instrument model is introduced or the acquisition logic needs adjustment, the program code often needs to be modified and redeployed, making it difficult to meet the real-time, flexible, consistent, and highly reliable data acquisition requirements of modern cigarette factories. Summary of the Invention

[0004] The main objective of this application is to provide a multi-source data processing method and system for cigarette testing instruments, in order to solve the problem that the existing technology has a wide variety of tobacco-related testing data and it is difficult to adapt to multiple different data sources.

[0005] To achieve the above objectives, this application provides the following technical solution: A multi-source data processing method for a cigarette testing instrument, wherein the multi-source data processing method is applied to several different cigarette testing instruments, and the multi-source data processing method includes: Step S1: Access several heterogeneous raw data from different cigarette testing instruments through network interface, database connection, serial communication, and file interface protocol; Step S2: For each multi-source heterogeneous raw data, perform parsing and format recognition based on context-free grammar, and convert it into structured time-series data records with timestamps and device identifiers; Step S3: Input the current structured time series data record into the similarity matching algorithm based on locality-sensitive hashing, and match and associate the structured time series data records from different cigarette testing instruments according to the proximity of timestamps and the correlation of device identifiers; Step S4: All structured time series data records are fused into a multidimensional feature matrix with each detection sample as a unit using a weighted aggregation algorithm based on an attention mechanism. Step S5: Input the fused multidimensional feature matrix into the stacked noise reduction autoencoder for feature extraction and dimensionality reduction to obtain a dimensionality-reduced dense feature vector set; Step S6: Output the dimensionality-reduced dense feature vector set as the result of standardized multi-source data processing.

[0006] Beneficial effects of steps S1 to S6: To address the challenges of chaotic multi-source instrument data access, difficult parsing, weak correlation, data silos, and complex subsequent analysis and processing in cigarette production quality inspection, this method constructs a complete and flexible data processing pipeline from heterogeneous access to standardized output, reducing the technical complexity and maintenance costs of multi-source data integration. Specifically, step S1, through a unified interface protocol and automated connection, achieves seamless access to heterogeneous data sources, avoiding the enormous workload and system rigidity issues associated with developing custom acquisition programs for each instrument, providing a stable and real-time multi-source data stream for subsequent processing. Step S2 utilizes predefined formatted description templates and automated parsing technology to uniformly transform unstructured raw data into structured records with standard timestamps and device identifiers, overcoming the difficulties in data interpretation caused by protocol and format differences, and laying the foundation for data correlation. Step S3, through a similarity matching algorithm based on locality-sensitive hashing, automatically and efficiently identifies and correlates data records from different instruments belonging to the same detection event, solving the problems of data fragmentation and missing correlations caused by dispersed data sources, and achieving logical correlation based on the test sample. Step S4 employs a weighted aggregation algorithm based on an attention mechanism to assign differentiated weights to the correlated time-series data and fuse them, effectively integrating multidimensional measurement information from multiple instruments and different time points to generate a fusion feature matrix that can more comprehensively characterize the quality status of individual samples. Step S5 uses a stacked noise reduction autoencoder to automatically extract features and reduce the dimensionality of the high-dimensional fusion matrix, removing redundancy and noise while retaining core information, compressing complex multi-source mixed data into low-dimensional, dense, and more representative feature vectors, significantly improving the analyzability of the data. Step S6 finally outputs the processing results in the form of a standardized data package with complete verification information, ensuring the integrity, consistency, and high reliability of the data processing results, providing a ready-to-use, unified, and high-quality data foundation for upper-level quality analysis, process control, and other applications.

[0007] As a further improvement to this application, step S1 involves accessing several heterogeneous raw data sets from different cigarette testing instruments via network interface, database connection, serial communication, and file interface protocol, including: Step S1.1: Detect and determine the type and connection parameters of the data source to be connected by using the external device instrument list, and generate a data source connection configuration set; Step S1.2: Based on the type identifier in the data source connection configuration set, call the standard database connection driver for the data source identified as a relational database to establish a network database connection; Step S1.3: Based on the type identifier in the data source connection configuration set, for data sources identified as serial communication devices, establish a serial communication connection according to the baud rate and parity parameters in the data source connection configuration set; Step S1.4: Based on the type identifier in the data source connection configuration set, for data sources identified as file services, establish a secure file transfer protocol connection according to the address and authentication information in the data source connection configuration set; Step S1.5: Perform connectivity verification tests on the network database connection, the serial communication connection, and the secure file transfer protocol connection to obtain a verified valid connection pool; Step S1.6: After verification, the valid connection pool concurrently executes data query instructions, serial port data read instructions, and file pull instructions to obtain a multi-source data stream in the form of an initial byte stream or character stream; Step S1.7: Encapsulate the multi-source data stream and add source device identifier and timestamp to obtain raw data units in standard packet format; Step S1.8: Continuously output the original data units along the time process to obtain the multi-source heterogeneous original data.

[0008] Beneficial effects of steps S1.1 to S1.8: Overall, the sub-steps of this section construct an automated and universal data access process. Through centralized configuration and unified connection management, it reduces the workload of developing dedicated acquisition programs for different cigarette testing instruments, lowers system maintenance complexity, and enhances the flexibility and scalability of data acquisition, laying a standardized data foundation for subsequent multi-source data processing. Specifically, step S1.1 automatically detects the list of external devices and instruments and determines the data source type and connection parameters, generating a data source connection configuration set to achieve unified identification and configuration management of different data sources; based on the type identification of this configuration set; step S1.2 calls the standard database connection driver to establish a network database connection; step S1.3 establishes a serial communication connection based on baud rate and parity parameters; step S1.4 establishes a secure file transfer protocol connection based on address and authentication information, thus covering multiple data source access methods such as relational databases, serial devices, and file services; step S1.5 processes the established connections for various types of data. The connection performs connectivity verification tests to form a valid connection pool after verification, ensuring connection reliability and stability. Step S1.6 uses this valid connection pool to concurrently execute data query commands, serial port data read commands, and file pull commands to obtain multi-source data streams in the form of initial byte streams or character streams, improving data acquisition efficiency. Step S1.7 encapsulates the multi-source data streams and adds source device identifiers and timestamps to generate raw data units in standard packet format, giving data traceability and time consistency. Step S1.8 continuously outputs these raw data units along the time process, providing stable and real-time multi-source heterogeneous raw data.

[0009] As a further improvement to this application, step S2 involves parsing and format recognition of each multi-source heterogeneous raw data based on context-free grammar, and converting it into structured time-series data records with timestamps and device identifiers, including: Step S2.1: Assign the original data unit to the corresponding parsing channel according to the source device identifier of the original data unit; Step S2.2: Load a preset data format description template corresponding to the device model for each parsing channel. The preset data format description template includes a syntax structure for defining data in the extended Backus paradigm. Step S2.3: Based on the preset data format description template, perform syntax parsing on the payload part of each original data unit using the recursive descent method to obtain a parsed dataset based on each original data unit; Step S2.4: Apply key-value pair extraction rules based on a preset pattern to the parsed dataset of the current original data unit to map unstructured text fields or binary segments into a set of key-value pairs. Step S2.5: Merge the key-value pair set of the current original data unit with the corresponding timestamp and device identifier to obtain a preliminary associated tuple sequence; Step S2.6: Based on the IEEE floating-point standard and string conversion, normalize the numerical values ​​in each preliminary associated tuple sequence respectively; Step S2.7: Sort each normalized preliminary associated tuple sequence in ascending order by timestamp and encapsulate it into a structured time series data record with a fixed field structure.

[0010] Beneficial effects of steps S2.1 to S2.7: Overall, these sub-steps constitute an automated, rule-driven data parsing and standardization pipeline that transforms raw data from diverse sources and with varied formats into high-quality, structurally consistent inputs that can be directly used for subsequent similarity matching and feature aggregation. This fundamentally avoids the enormous development and maintenance burden of writing hard-coded parsers for each data format. In this process, step S2.1 assigns raw data units to corresponding parsing channels based on the source device identifier, achieving preliminary divide-and-conquer and directional processing of data flow, laying the foundation for the categorized parsing of heterogeneous data. Step S2.2 loads a data format description template corresponding to the device model and defined by the extended Backus paradigm into each parsing channel, transforming the device-specific non-standard data format into a standardized description rule that can be recognized and parsed by machines, solving the problem of inconsistent parsing logic caused by the diversity of instrument output formats. Step S2.3, based on this description template, performs syntax parsing based on recursive descent on the payload portion of the raw data units. This method can systematically identify and decompose the inherent syntax structure of the data, accurately extract discrete fields and values, thereby transforming the raw byte or character stream into a data set with preliminary semantics. Step S2.4 applies key-value pair extraction rules based on a preset pattern to the parsed data, transforming unstructured text or binary data into a more coherent and meaningful structure. The process of mapping fragments into a unified set of key-value pairs further eliminates the remaining differences in data representation and establishes a standardized intermediate data structure. Step S2.5 merges this set of key-value pairs with the original timestamps and device identifiers in the data units to generate a preliminary associated tuple sequence. This establishes an inseparable link between data content and metadata (time, source), ensuring data traceability and spatiotemporal attribute integrity. Step S2.6 performs normalization processing on the numerical data in the tuple sequence based on the IEEE floating-point standard and string conversion, unifying the numerical representation methods and precision that different instruments may use, eliminating data quality risks caused by inconsistencies in units, number systems, or formats, and providing a reliable foundation for subsequent accurate calculations and comparisons. Step S2.7 arranges the normalized tuple sequence in ascending order by timestamp and encapsulates it into a record object with a fixed field structure, ultimately producing a completely unified, time-series-clear, and structurally sound structured time-series data record.

[0011] As a further improvement to this application, step S3 involves inputting the current structured time-series data record into a similarity matching algorithm based on locality-sensitive hashing. This algorithm matches and associates structured time-series data records from different cigarette testing instruments based on timestamp proximity and device identifier correlation, including: Step S3.1: Extract the absolute time value of the timestamp and the original string of the device identifier from the current structured time series data record; Step S3.2: Convert the original string of the device identifier into a semantically related numerical device code through a preset mapping table; Step S3.3: The absolute time value and the numerical device code are concatenated and normalized, and combined into a high-dimensional feature vector representing the spatiotemporal attributes of the record. Step S3.4: Input the high-dimensional feature vector into the locality-sensitive hash algorithm based on random projection for hash operation, and allocate all structured time series data records to the corresponding hash buckets according to the calculated hash value; Step S3.5: Within the same hash bucket, calculate the absolute difference of absolute time values ​​between any two structured time series data records, and define structured time series data records whose absolute difference is less than the preset time series window threshold as having a definite matching relationship; Step S3.6: Attach a globally unique matching group identifier to each pair of structured time series data records that match the relationship, and integrate them into a set of matched records with matching association information.

[0012] Beneficial effects of steps S3.1 to S3.6: For structured time-series data records from different cigarette testing instruments, an automated and efficient data matching and association mechanism was constructed. Overall, these steps work together to solve the core problem of scattered and difficult-to-collect data of the same test sample caused by parallel testing by multiple instruments. By replacing the traditional hard-coded matching method that relies on manual rules or fixed logic with an algorithmic approach, the mechanism achieves automated, intelligent and highly scalable data association, providing a structured input for subsequent data fusion and feature analysis on a sample-by-sample basis. Step S3.1 extracts the absolute time value and the original string of the device identifier from each record, providing the core comparison dimension for subsequent similarity calculations. Step S3.2 converts the original string of the device identifier into a numerical device code using a preset mapping table, unifying the discrete, unstructured device names into semantic codes that can be used for mathematical operations, laying a formal foundation for the association between devices. Step S3.3 concatenates and normalizes the time value and the device code to form a high-dimensional feature vector representing the spatiotemporal attributes of the record, transforming the multidimensional matching problem into a nearest neighbor search problem in the vector space. Step S3.4 applies a locality-sensitive hashing algorithm based on random projection to perform a hash operation on this high-dimensional feature vector, the core effect of which is... This involves mapping potentially matching records to the same hash bucket with a high probability, thereby transforming the global, costly, and precise comparison into a local, fast intra-bucket comparison, significantly improving the efficiency of finding related records in massive amounts of data. Step S3.5 involves calculating the absolute difference of timestamps between records within the same hash bucket and comparing it with a preset time-series window threshold, while also verifying it in conjunction with the association rules of the device encoding. This process achieves a precise determination from "potentially related" to "confirmed as related," ensuring the accuracy of the matching relationship and the rationality of the business. Step S3.6 involves attaching a globally unique matching group identifier to each pair of confirmed matching records, thereby integrating scattered and isolated single records into a set of matched records with clear association information.

[0013] As a further improvement to this application, step S4 involves fusing all structured time-series data records into a fused multidimensional feature matrix based on the detection sample using a weighted aggregation algorithm based on an attention mechanism, including: Step S4.1: Based on the matching group identifier of each structured time series data record in the matched record set, divide all structured time series data records into different record groups, with each record group corresponding to a unique detection sample. Step S4.2: Extract all numerical data fields except timestamps and device identifiers from each structured time series data record in the current record group to obtain the original feature fragments of all structured time series data records in the current record group; Step S4.3: Arrange all original feature fragments in the current record group in chronological order and combine them into the initial serialized feature matrix corresponding to the current unique detection sample; Step S4.4: Calculate the attention weight distribution of each structured time series data record in the current initial serialized feature matrix based on the scaling dot product attention mechanism; Step S4.5: Based on the attention weight distribution, perform a weighted summation of the original feature segments at different time steps in the current initial serialization feature matrix to obtain the weighted aggregated feature vector corresponding to the current record group; Step S4.6: Arrange the weighted aggregated feature vectors of all record groups according to the sequence number of the matching group identifier, and assemble them into a fused multidimensional feature matrix. Each row of the fused multidimensional feature matrix corresponds to a unique matching group identifier.

[0014] Beneficial effects of steps S4.1 to S4.6: For data records from multiple detection sources that have been matched and associated, intelligent data fusion with time-series awareness is achieved on a per-sample basis. Overall, these steps, by introducing an attention mechanism, endow the system with the ability to distinguish the primary and secondary information and the focus of attention during the data fusion process. This effectively overcomes the information dilution or noise amplification problems that may be caused by the equal weighting of time-series data in traditional methods. It transforms the scattered, multi-point raw observation data into high-quality and information-rich sample-level feature representations that can be directly used for subsequent deep feature extraction. In step S4.1, the discrete structured time-series data records are divided into different record groups logically belonging to the same test sample based on the matching group identifier, establishing clear data boundaries and aggregation units for subsequent sample-level comprehensive analysis. Step S4.2 extracts all numerical data fields (excluding metadata) from each record within each record group to obtain the original feature fragments constituting the multidimensional quality profile of the sample. Step S4.3 arranges and combines these original feature fragments according to their inherent time order to construct the initial serialized feature matrix corresponding to each sample. This matrix not only retains multidimensional measurement information but also depicts the possible changes in detection parameters over time. Step S4.4 calculates the following based on the scaling dot product attention mechanism: The attention weight distribution recorded at different time steps in each initial serialized feature matrix is ​​used to enable the model to automatically learn and evaluate the relative importance of data records from different times or instruments to the final sample feature representation during the same detection process, rather than simply averaging or adding them with equal weights. Step S4.5 uses the dynamically generated attention weight distribution to perform differential weighted summation on the features at different time steps in the serialized feature matrix, thereby generating a single weighted aggregated feature vector that can more accurately and concisely reflect the overall state and key features of the detected sample. Step S4.6 finally arranges and assembles the weighted aggregated feature vectors corresponding to all samples according to their identifiers to form a complete fused multidimensional feature matrix.

[0015] As a further improvement to this application, step S5 involves inputting the fused multidimensional feature matrix into a stacked noise reduction autoencoder for feature extraction and dimensionality reduction, resulting in a dimensionality-reduced dense feature vector set, including: Step S5.1: Perform Z-score normalization preprocessing on the fused multidimensional feature matrix to obtain a normalized feature matrix; Step S5.2: Add random noise conforming to a Gaussian distribution to the standardized feature matrix to obtain a noisy input feature matrix; Step S5.3: Input the noisy input feature matrix into the first layer encoder of the stacked denoising autoencoder, and obtain the first layer encoded features through linear transformation and ReLU activation function; Step S5.4: The first layer encoded features are passed sequentially through several subsequent encoding layers of the stacked noise reduction autoencoder to reduce the feature dimension of each row of the noisy input feature matrix layer by layer, and low-dimensional encoded features are obtained at the bottleneck layer. Step S5.5: The low-dimensional encoded features are used as input to the decoder part of the stacked noise reduction autoencoder, and the data is reconstructed through a symmetric decoding layer structure to obtain the reconstructed feature matrix. Step S5.6: Discard the decoder part of the stacked noise reduction autoencoder, retain the path of the stacked noise reduction autoencoder from the first encoder layer to the bottleneck layer, input the normalized feature matrix into the path, and extract the dimensionality-reduced dense feature vector set.

[0016] Beneficial effects of steps S5.1 to S5.6: Deep feature learning and information compression are performed on the fused multidimensional feature matrix generated in the previous steps, which is based on the detection samples, to extract more representative and robust data representations. Overall, these steps automatically learn the mapping relationship from the high-dimensional, possibly redundant fused matrix to the low-dimensional, dense feature vector during adversarial training by stacking a noise-reducing autoencoder encoder-decoder framework. This not only achieves data dimensionality reduction and facilitates subsequent storage and computation, but more importantly, it removes noise, discovers robust intrinsic patterns and deep features in the data, and improves the representational ability and generalization of the feature vectors. In this process, step S5.1 performs Z-score standardization preprocessing on the fused multidimensional feature matrix to eliminate biases caused by differences in dimensions and numerical ranges between different feature dimensions, ensuring that all features are on the same order of magnitude, thus laying the foundation for stable training and convergence of the subsequent model. Step S5.2 adds Gaussian-distributed random noise to the standardized feature matrix to generate a noisy input feature matrix. This operation simulates the interference and incompleteness that may exist in real-world data, forcing the model to learn to denoise and capture the robust underlying structure of the data. Step S5.3 inputs the noisy input feature matrix into the first layer of the stacked denoising autoencoder, and performs preliminary nonlinear mapping and feature abstraction through linear transformation and the ReLU activation function to obtain the first layer of encoded features. 4. The encoded features are passed through multiple subsequent encoding layers, undergoing higher-order nonlinear transformations and feature compression layer by layer. Redundant information is gradually discarded and core patterns are extracted. Finally, a low-dimensional encoded feature with significantly reduced dimensionality is obtained at the bottleneck layer, achieving efficient data compression and explicit expression of latent features. Step S5.5. The low-dimensional encoded features are reconstructed using a symmetrical decoder structure to attempt to restore the structure of the original input. This reconstruction process and the encoding process together constitute self-supervised training, driving the model to learn low-dimensional encoding that must contain enough key information to reconstruct the original data. Step S5.6. After the model training converges, the encoder parameters are fixed, and the standardized feature matrix without added noise is directly input into the encoder to the bottleneck layer path to extract the dimensionality-reduced dense feature vector set.

[0017] As a further improvement to this application, step S6, outputting the dimensionality-reduced dense feature vector set as the result of standardized multi-source data processing, includes: Step S6.1: Bind each row of the dimensionality-reduced dense feature vector set to the corresponding matching group identifier to generate feature data pairs with sample identifiers; Step S6.2: For the dimensionality-reduced dense feature vector set portion of the feature data pair, convert all feature data pairs into binary byte streams using the data serialization operation specified in the Apache Avro format. Step S6.3: Append a fixed-format message header, including data version, processing timestamp and data mode identifier, to each binary byte stream, and integrate them into a complete data message; Step S6.4: Calculate the cyclic redundancy check code of the complete data message and append it to the end of the complete data message to obtain the standardized multi-source data processing result.

[0018] Beneficial effects of steps S6.1 to S6.4: A complete, reliable, and standardized data output and encapsulation mechanism is constructed for the dimensionality-reduced dense feature vector set produced in the preceding steps. Overall, these steps collectively transform abstract data features into data messages with clear semantics, standardized formats, complete metadata, and integrity protection. This completes the final transformation from internal processing to providing standardized services externally, providing a ready-to-use, reliable, and efficient data interface for upper-layer applications. Specifically, step S6.1 binds each row of data in the dimensionality-reduced dense feature vector set with its corresponding matching group identifier, generating feature data pairs with sample identification. This operation establishes a robust and unambiguous association between the core feature data and the original physical detection sample it represents, ensuring the traceability of the data processing results and the clarity of the business meaning. Step S6.2 applies Apache-based technology to the feature vector portion of the feature data pairs. The Avro format specification's data serialization operation converts the data into a compact binary byte stream. This process not only compresses the data volume to improve transmission and storage efficiency but also achieves self-descriptive data structure through Avro's built-in schema description, providing a standardized, programming language-independent intermediate format for cross-system and cross-platform data exchange. Step S6.3 appends a fixed-format message header containing data version, processing timestamp, and data schema identifier to each serialized binary byte stream and integrates them into a complete data message. This step encapsulates crucial metadata information into the original data content, enabling the receiving end to accurately identify the data format version, determine data freshness, and correctly parse the data body based on the schema identifier, thereby ensuring the consistency and understandability of the data in subsequent transmission and application. Step S6.4 calculates the cyclic redundancy check (CRC) code of the complete data message and appends it to the end of the message. This operation provides end-to-end integrity verification capability for the entire data packet, effectively detecting any unexpected errors or tampering that may occur during data transmission or storage, and is a key step in ensuring the high reliability of data processing results delivery.

[0019] To achieve the above objectives, this application also provides the following technical solutions: A multi-source data processing system for a cigarette testing instrument, the multi-source data processing system being applied to the multi-source data processing method described above, the multi-source data processing system comprising: The heterogeneous raw data acquisition module is used to access several heterogeneous raw data from different cigarette testing instruments through network interface, database connection, serial communication, and file interface protocol. The heterogeneous raw data conversion module is used to parse and identify the format of each multi-source heterogeneous raw data based on context-free grammar, and convert it into structured time series data records with timestamps and device identifiers. The time series data record association module is used to input the current structured time series data record into the similarity matching algorithm based on locality-sensitive hashing, and to match and associate the structured time series data records from different cigarette testing instruments according to the timestamp proximity and device identifier association. The time series data recording fusion module is used to fuse all structured time series data records into a fused multidimensional feature matrix based on the detection sample through a weighted aggregation algorithm based on an attention mechanism. The fused multidimensional feature matrix processing module is used to input the fused multidimensional feature matrix into the stacked noise reduction autoencoder for feature extraction and dimensionality reduction, to obtain a dimensionality-reduced dense feature vector set. The multi-source data processing result output module is used to output the dimensionality-reduced dense feature vector set as the standardized multi-source data processing result.

[0020] To achieve the above objectives, this application also provides the following technical solutions: An electronic device includes a processor and a memory coupled to the processor, the memory storing program instructions executable by the processor; when the processor executes the program instructions stored in the memory, it implements the multi-source data processing method of the cigarette detection instrument described above.

[0021] To achieve the above objectives, this application also provides the following technical solutions: A computer-readable storage medium storing program instructions, which, when executed by a processor, enable the implementation of the multi-source data processing method for a cigarette testing instrument as described above. Attached Figure Description

[0022] Figure 1 This is a schematic flowchart of one embodiment of a multi-source data processing method for a cigarette testing instrument according to this application; Figure 2 This is a functional module diagram of an embodiment of a multi-source data processing system for a cigarette testing instrument according to this application; Figure 3 This is a schematic diagram of the structure of an embodiment of the electronic device of this application; Figure 4 This is a schematic diagram of the structure of one embodiment of the storage medium of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0024] The terms "first," "second," and "third" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationships and movements between components in a specific orientation (as shown in the figures). If the specific orientation changes, the directional indications also change accordingly. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0025] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0026] like Figure 1 As shown, this embodiment provides an example of a multi-source data processing method for cigarette testing instruments. In this embodiment, the multi-source data processing method is applied to several different cigarette testing instruments, such as an air permeability meter, a friction coefficient meter, a colorimeter, an electronic balance, and a comprehensive testing platform.

[0027] Specifically, this multi-source data processing method includes the following steps: Step S1 involves accessing several heterogeneous raw data sources from different cigarette testing instruments via network interface, database connection, serial communication, and file interface protocol.

[0028] Furthermore, step S1 specifically includes the following steps: Step S1.1: Detect and determine the type and connection parameters of the data source to be connected by the external device instrument list, and generate a data source connection configuration set.

[0029] Preferably, this can be achieved by reading a predefined external device instrumentation configuration file (e.g., instrument_config.yaml).

[0030] Specifically, this file uses YAML format to declare the metadata of all instruments to be connected, and it is automatically parsed when the system starts. The parsing process is based on regular expression matching and key-value pair mapping, automatically identifying the type field of each data source, such as "database", "serial_port", "sftp", etc., and extracting the corresponding connection parameters, such as the jdbc_url, username, and password of the database; the port_name, baud_rate, and parity of the serial port; and the host, port, username, and private_key_path of the file service.

[0031] Preferably, all the parsed configuration information is stored in a hash table data structure in memory, forming a data source connection configuration set. The key of this set is the device identifier, and the value is a structured object containing the type and complete connection parameters.

[0032] Step S1.2: Based on the type identifier in the data source connection configuration set, call the standard database connection driver for the data source identified as a relational database to establish a network database connection.

[0033] Preferably, for items in the data source connection configuration set whose type is identified as "database", the corresponding standard JDBC driver (e.g., mysql-connector-java-xxxx.jar) can be loaded according to its subtype (e.g., "mysql", "sqlserver", "oracle"). Then, the connection string (JDBCURL), username, and password in the configuration are used as parameters.

[0034] Preferably, if using the Java technology stack, the following can be called: The java.sql.DriverManager.getConnection() method, or the equivalent API for the corresponding language / framework, such as Python's sqlalchemy.create_engine().

[0035] Preferably, when a connection is established, connection pool parameters are set, such as minimum idle connections minIdle=2 and maximum wait time maxWait=5000ms. Successfully established connection objects are encapsulated and temporarily stored, ready for subsequent connectivity verification.

[0036] Step S1.3: Based on the type identifier in the data source connection configuration set, establish a serial communication connection for the data source identified as a serial communication device according to the baud rate and parity parameters in the data source connection configuration set.

[0037] Preferably, for the item in the data source connection configuration set whose type is identified as "serial_port", the underlying serial communication API of the operating system can be called according to the configured port_name (e.g., "COM3" or " / dev / ttyUSB0"), baud_rate (e.g., 9600, 115200), data_bits (default 8), stop_bits (default 1), parity (e.g., "NONE") and other parameters.

[0038] Preferably, taking Java as an example, the gnu.io.RXTXCommDriver or jSerialComm library can be used to establish a connection by using SerialPort.getCommPort(portName).openPort() and setting various parameters. After a successful connection, a read timeout of readTimeout=2000ms and a write timeout will be configured, and the established serial port object will be placed in the verification queue.

[0039] Step S1.4: Based on the type identifier in the data source connection configuration set, for data sources identified as file services, establish a secure file transfer protocol connection according to the address and authentication information in the data source connection configuration set.

[0040] Preferably, for items in the data source connection configuration set whose type is identified as "sftp", the host, port (default 22), username, and authentication information (password or private_key_path) in the configuration can be used to establish an SSH / SFTP connection.

[0041] For example, using Java's JSch library, the core pseudocode is as follows: JSchjsch = new JSch(); if (privateKeyPath != null) { jsch.addIdentity(privateKeyPath); } Sessionsession=jsch.getSession(username, host, port); session.setPassword(password); / / If a password is used session.setConfig("StrictHostKeyChecking", "no"); / / Can be adjusted according to security requirements session.connect(timeout); ChannelSftpchannel=(ChannelSftp)session.openChannel("sftp"); channel.connect(timeout); The successfully created ChannelSftp object will be used for subsequent file list retrieval and file content fetching operations.

[0042] Step S1.5: Perform connectivity verification tests on network database connection, serial communication connection, and secure file transfer protocol connection to obtain a verified valid connection pool.

[0043] Preferably, a connectivity verification test can be performed on each connection object established in steps S1.2, S1.3, and S1.4, with a timeout set to 5000 milliseconds. Specifically, the database connection can execute a simple test query, such as "SELECT1"; the serial port connection can send a device-recognizable query command (e.g., "*IDN?\n") and wait for a response in the expected format; the SFTP connection attempts to list the files under a specified root directory (e.g., " / ").

[0044] Preferably, only connections that receive the expected response or successfully execute an operation are considered valid. All valid connection objects are registered with a connection pool manager (e.g., using HikariCP to manage database connections, and using a custom pool to manage serial and SFTP connections), forming a verified valid connection pool. Invalid connections will be logged in the error log and trigger warnings.

[0045] Step S1.6: After verification, the valid connection pool concurrently executes data query instructions, serial port data read instructions, and file pull instructions to obtain a multi-source data stream in the form of an initial byte stream or character stream.

[0046] Preferably, data retrieval requests can be initiated concurrently to each data source through a valid connection pool after verification (e.g., using a thread pool with the core thread count set to twice the number of CPU cores).

[0047] Specifically, for database connections, the SELECT query SQL statements defined in the configuration can be executed, and the ResultSet can be obtained through java.sql.Statement.executeQuery() and converted into a JSON string stream; for serial port connections, the serial port buffer can be continuously read or query commands can be sent periodically according to instructions and the returned byte stream can be read; for SFTP connections, the configured remote directory can be polled, and newly added detection report files (such as xml, csv, pdf, etc.) can be pulled into local memory through channelSftp.get(remoteFilePath) and their contents can be read out as byte streams or character streams.

[0048] Step S1.7: Encapsulate the multi-source data stream and attach the source device identifier and timestamp to obtain the original data unit in the standard packet format.

[0049] Preferably, each piece of data in a multi-source data stream can be encapsulated using a unified data structure. For example, the ProtocolBuffers syntax defines the data structure: messageRawDataPacket{ stringdevice_id=1; / / Source device identifier, taken from configuration int64timestamp_ns=2; / / Timestamp of data acquisition, precision in nanoseconds bytespayload=3; / / Payload, i.e., the original data stream content stringdata_format = 4; / / Data format hints, such as "json", "binary", "text" } Step S1.8: Continuously output raw data units along the time process to obtain multi-source heterogeneous raw data.

[0050] Preferably, the original data units are placed in a thread-safe bounded blocking queue (e.g., LinkedBlockingQueue, with a configurable capacity of 10,000).

[0051] Preferably, a separate output thread or stream processing pipeline (such as Apache KafkaProducer or Apache FlinkSource) continuously pulls data units from the queue and outputs them as multi-source heterogeneous raw data.

[0052] Beneficial effects of steps S1.1 to S1.8: Overall, the sub-steps of this section construct an automated and universal data access process. Through centralized configuration and unified connection management, it reduces the workload of developing dedicated acquisition programs for different cigarette testing instruments, lowers system maintenance complexity, and enhances the flexibility and scalability of data acquisition, laying a standardized data foundation for subsequent multi-source data processing. Specifically, step S1.1 automatically detects the list of external devices and instruments and determines the data source type and connection parameters, generating a data source connection configuration set to achieve unified identification and configuration management of different data sources; based on the type identification of this configuration set; step S1.2 calls the standard database connection driver to establish a network database connection; step S1.3 establishes a serial communication connection based on baud rate and parity parameters; step S1.4 establishes a secure file transfer protocol connection based on address and authentication information, thus covering multiple data source access methods such as relational databases, serial devices, and file services; step S1.5 processes the established connections for various types of data. The connection performs connectivity verification tests to form a valid connection pool after verification, ensuring connection reliability and stability. Step S1.6 uses this valid connection pool to concurrently execute data query commands, serial port data read commands, and file pull commands to obtain multi-source data streams in the form of initial byte streams or character streams, improving data acquisition efficiency. Step S1.7 encapsulates the multi-source data streams and adds source device identifiers and timestamps to generate raw data units in standard packet format, giving data traceability and time consistency. Step S1.8 continuously outputs these raw data units along the time process, providing stable and real-time multi-source heterogeneous raw data.

[0053] Step S2: Each multi-source heterogeneous raw data is parsed and format identified based on context-free grammar, and then converted into structured time-series data records with timestamps and device identifiers.

[0054] Furthermore, step S2 specifically includes the following steps: Step S2.1: Assign the raw data unit to the corresponding parsing channel according to the source device identifier of the raw data unit.

[0055] Preferably, a resolution channel registry can be maintained, which is a mapping table (e.g., ConcurrentHashMap in Java) with device identifiers (DeviceID) as keys and corresponding resolution processor instances as values.<String,Parser> Upon receiving a raw data unit in the standard packet format from step S1, the source device identifier is extracted from its encapsulated device_id field and searched in the registry. If a corresponding parsing processor is found, the data unit is placed in the processor's dedicated input queue (e.g., LinkedBlockingQueue); otherwise, based on the prefix or model keyword of the device identifier (e.g., an identifier starting with "KC-" triggers a colorimeter parser, and one starting with "DT-" triggers an air permeability meter parser), a new parsing processor is dynamically instantiated and registered using the factory pattern, creating a new parsing channel. Each parsing channel operates independently, using a producer-consumer model to process the data units in its queue.

[0056] Step S2.2: Load a preset data format description template corresponding to the device model for each parsing channel. The preset data format description template includes a syntax structure for defining data in an extended Backus paradigm.

[0057] Preferably, each parsing channel, upon initialization, loads a data format description template from a centralized template repository based on its corresponding device model. This template is a JSON or YAML configuration file, the core of which uses data syntax rules defined using Extended Backus Normative Form (EBNF).

[0058] For example, for an instrument that sends colon-separated text via serial port, its template fragment is shown in the following YAML statement: parser_type: "text_delimited" grammar: start:record record:field(':'field)* field:NUMBER|STRING NUMBER:[0-9]+('.'[0-9]+) STRING:[A-Za-z0-9_]+ encoding: "ASCII" delimiter: ":" field_names:["sample_id","value","unit","status"] Step S2.3: Based on the preset data format description template, perform syntax parsing on the payload part of each original data unit using the recursive descent method to obtain a parsed dataset based on each original data unit.

[0059] Preferably, the parser processor retrieves a raw data unit from the input queue, reading its payload and data_format field. Based on the data_format (e.g., "text") and the parser_type defined in the template, the appropriate parser is invoked. For text format, the core parser is a recursive descent parser. This parser transforms the byte stream or character stream into an abstract syntax tree (AST) according to the grammar rules defined in EBNF.

[0060] For example, the core pseudocode of the parsing process is shown below: defparse_record(tokens,grammar): ast={} forfield_defingrammar['field_names']: token = tokens.consume_next() # Validate the token type (NUMBER or STRING) according to the syntax rules. ifnotmatches_grammar(token,grammar['rules'][field_def]): raiseParseError(f "Token{token}doesnotmatchrulefor{field_def}") ast[field_def]=token.value returnast Preferably, for binary or specific format files (such as PDF, XML), a dedicated parsing library (such as PDFBox for PDF text extraction, and DOM4J for XML parsing) is used to first convert the file content into an intermediate text representation before applying the above-mentioned syntax for parsing. The parsing result is a dictionary containing the names of each field and their original values, constituting the parsed dataset for that data unit.

[0061] Step S2.4: Apply key-value pair extraction rules based on a preset pattern to the parsed dataset of the current original data unit to map unstructured text fields or binary segments into a set of key-value pairs.

[0062] Preferably, after obtaining the parsed dataset (original dictionary), the system applies the data extraction rules defined in the template. These rules are usually a set of predefined regular expressions or XPath expressions used to further extract structured information from unstructured text fields.

[0063] For example, a raw field "Result:12.5mg;Pass" can be extracted into key-value pairs {"measured_value":12.5, "judgment":"Pass"} by performing regular expression matching with the rule "Result:([0-9.]+)mg;(Pass|Fail)". For already structured data, this step may simply involve renaming or direct mapping. Ultimately, all fields are transformed into a unified, standardized set of key-value pairs, with the data structure being a Map.<String,Object> .

[0064] Step S2.5: Merge the key-value pair set of the current original data unit with the corresponding timestamp and device identifier to obtain a preliminary associated tuple sequence.

[0065] Preferably, the key-value pair set generated in step S2.4 is merged with the device_id and timestamp_ns metadata that are inherent in the original data unit, and the merging operation creates a new data structure.

[0066] Step S2.6: Based on the IEEE floating-point standard and string conversion, normalize the numerical values ​​in each preliminary associated tuple sequence.

[0067] Preferably, the kvPairs of each tuple in the initial associated tuple sequence can be traversed. For each value, the following steps are performed: ①Type recognition and conversion: First, determine its string representation.

[0068] If the floating-point regular expression "^[-+]?[0-9]*\.?[0-9]+([eE][-+]?[0-9]+)?$" is matched, then use Double.parseDouble() to convert it to a double-precision floating-point number conforming to the IEEE 754 standard. If it is a pure integer, it is converted to Long. Otherwise, it is preserved as a string.

[0069] ② Unit Standardization: Check if the key name is in the "Unit Mapping Table". For example, the value "1.2 lbs" for the key "weight" will be matched and converted to a floating-point number of 0.5443104 in kilograms (kg) by looking up the mapping table {"lbs":0.453592}. The conversion formula is: Standard value = Original value * Conversion factor.

[0070] ③ Precision uniformity: Apply uniform precision rules to all floating-point numbers, for example, use BigDecimal.setScale(6, RoundingMode.HALF_UP) to retain 6 decimal places.

[0071] Step S2.7: Sort each normalized preliminary associated tuple sequence in ascending order by timestamp and encapsulate it into a structured time series data record with a fixed field structure.

[0072] Preferably, all normalized preliminary associative tuple sequences are collected into a list. First, the list is sorted in ascending order based on the timestampNs field of each tuple using a quicksort algorithm (such as Collections.sort() with a custom comparator). Then, each sorted tuple is encapsulated as a record object with a fixed field structure. The class definition of this record object enforces structural uniformity, for example: messageStructuredRecord{ stringdevice_id=1; int64timestamp_ns=2; stringsample_id=3; / / Core business ID extracted from kvPairs map<string,double> numerical_fields=4; / / All numeric fields map<string,string> textual_fields=5; / / All text fields stringrecord_type=6; / / Record type, such as "measurement", "alarm" } Beneficial effects of steps S2.1 to S2.7: Overall, these sub-steps constitute an automated, rule-driven data parsing and standardization pipeline that transforms raw data from diverse sources and with varied formats into high-quality, structurally consistent inputs that can be directly used for subsequent similarity matching and feature aggregation. This fundamentally avoids the enormous development and maintenance burden of writing hard-coded parsers for each data format. In this process, step S2.1 assigns raw data units to corresponding parsing channels based on the source device identifier, achieving preliminary divide-and-conquer and directional processing of data flow, laying the foundation for the categorized parsing of heterogeneous data. Step S2.2 loads a data format description template corresponding to the device model and defined by the extended Backus paradigm into each parsing channel, transforming the device-specific non-standard data format into a standardized description rule that can be recognized and parsed by machines, solving the problem of inconsistent parsing logic caused by the diversity of instrument output formats. Step S2.3, based on this description template, performs syntax parsing based on recursive descent on the payload portion of the raw data units. This method can systematically identify and decompose the inherent syntax structure of the data, accurately extract discrete fields and values, thereby transforming the raw byte or character stream into a data set with preliminary semantics. Step S2.4 applies key-value pair extraction rules based on a preset pattern to the parsed data, transforming unstructured text or binary data into a more coherent and meaningful structure. The process of mapping fragments into a unified set of key-value pairs further eliminates the remaining differences in data representation and establishes a standardized intermediate data structure. Step S2.5 merges this set of key-value pairs with the original timestamps and device identifiers in the data units to generate a preliminary associated tuple sequence. This establishes an inseparable link between data content and metadata (time, source), ensuring data traceability and spatiotemporal attribute integrity. Step S2.6 performs normalization processing on the numerical data in the tuple sequence based on the IEEE floating-point standard and string conversion, unifying the numerical representation methods and precision that different instruments may use, eliminating data quality risks caused by inconsistencies in units, number systems, or formats, and providing a reliable foundation for subsequent accurate calculations and comparisons. Step S2.7 arranges the normalized tuple sequence in ascending order by timestamp and encapsulates it into a record object with a fixed field structure, ultimately producing a completely unified, time-series-clear, and structurally sound structured time-series data record.

[0073] Step S3: Input the current structured time series data record into the similarity matching algorithm based on locality-sensitive hashing, and match and associate the structured time series data records from different cigarette testing instruments according to the timestamp proximity and device identifier correlation.

[0074] Furthermore, step S3 specifically includes the following steps: Step S3.1: Extract the absolute time value of the timestamp and the original string of the device identifier from the current structured time series data record.

[0075] Preferably, step S3.1 receives a stream of structured time-series data records from step S2, where each record contains fixed fields such as timestamp_ns (nanosecond-level timestamp) and device_id (device identifier string). This step first extracts these two core fields from the record: timestamp_ns is directly read as a 64-bit long integer value, denoted as the absolute time value T; device_id is read as a string, denoted as the raw device identifier string D_raw.

[0076] Step S3.2: Convert the original string of the device identifier into a semantically related numerical device code through a preset mapping table.

[0077] Preferably, a device semantic mapping table can be maintained. This mapping table is a predefined collection of key-value pairs, such as stored in Redis or an in-memory HashMap, and its function is to map device identifiers with business meaning to integer codes with spatial or logical association.

[0078] For example, one JSON form of a device semantic mapping table is as follows: { “DETECTOR_A_01”:101, “DETECTOR_A_02”:102, “SCANNER_B_01”:201, “WEIGHER_C_01”:301 } Specifically, the mapping logic uses coding design to ensure that devices on the same production line, measuring the same indicator, or physically located nearby have numerically similar code values. For example, multiple air permeability meters of the same model are coded as 101, 102, and 103; while completely different colorimeters start coding from 201. This mapping table is then consulted to convert the original string "D_raw" of the device identifier into an integer numeric device code "D_code". If "D_raw" is not in the mapping table, an alarm is triggered, a reserved default code (such as 0) is assigned, and a log is logged for future updates to the mapping table.

[0079] Step S3.3 involves concatenating and normalizing the absolute time value and the numerical device code, and combining them into a high-dimensional feature vector representing the spatiotemporal attributes of the record.

[0080] Preferably, time normalization normalizes all absolute time values ​​T. First, the minimum timestamp T_min and the maximum timestamp T_max within a time window are found. Then, T is normalized to the [0,1] interval using the formula T_norm=(T-T_min) / (T_max-T_min). This ensures that time differences are not affected by the magnitude of the absolute time values.

[0081] Preferably, device code normalization can normalize all numerical device codes D_code to the same order of magnitude. For example, the formula D_norm = D_code / Max_Device_Code can be used, where Max_Device_Code is the largest device code value in the system.

[0082] Preferably, vector concatenation can combine the normalized time value T_norm and the device-encoded value D_norm into a two-dimensional vector V=[T_norm, D_norm]. This two-dimensional vector is the high-dimensional feature vector (2-dimensional in this embodiment) representing the spatiotemporal attributes of the record. In more complex scenarios, other normalized features (such as detection values) can be added to increase the vector dimension.

[0083] Step S3.4: Input the high-dimensional feature vector into the locality-sensitive hashing algorithm based on random projection for hash operation, and allocate all structured time series data records to the corresponding hash buckets according to the calculated hash value.

[0084] Preferably, a Locality Sensitive Hashing with Random Projection algorithm can be used to map similar vectors to the same hash bucket with a high probability.

[0085] Specifically, the hash function family is defined as follows: k hash functions are pre-generated, and each function h_i(v) is defined as: h_i(v) = sign(r_i·v + b_i). Here, v is the input feature vector, r_i is a random vector with the same dimension as v (its components are usually sampled from a standard normal distribution), · represents the dot product, b_i is a bias term uniformly sampled in the range [0, w], w is the bucket width parameter, and sign() is the sign function (the result is usually 0 or 1).

[0086] The next step is to construct a composite hash key: For a vector V, calculate its k hash values ​​to obtain a k-bit binary string (or integer), i.e., g(V)=[h_1(V),h_2(V),...,h_k(V)], which serves as the composite hash key of the vector.

[0087] Next, bucketing: The system maintains a hash table (dictionary) with g(V) as the key. Vector V and its corresponding original record index are added to the bucket (list) with key g(V). Finally, all records are grouped into different hash buckets based on the similarity of their high-dimensional feature vectors; similar vectors (those close in time, strongly related to the same device) have a higher probability of having the same g(V).

[0088] Step S3.5: Within the same hash bucket, calculate the absolute difference of absolute time values ​​between any two structured time series data records, and define structured time series data records whose absolute difference is less than the preset time series window threshold as having a definite matching relationship.

[0089] Preferably, precise pairing comparisons are performed within the same hash bucket to confirm the final matching relationship.

[0090] The temporal proximity determination is as follows: For any two records i and j within the bucket, calculate the absolute difference Δt = |T_i - T_j| between their original absolute time values. A preset time window threshold W_t is set (e.g., W_t = 5000 milliseconds, indicating that the same detection event is completed within 5 seconds). If Δt <= W_t, the record passes the initial temporal proximity screening.

[0091] The equipment association rule verification involves predefining a set of equipment association rules. This rule set can be a list of allowed pairings or a rule based on codes. For example, one possible rule is: "Allow equipment codes 101 and 102 to pair" (meaning data from different machines of the same model can be associated); "Allow pairing of any two machines within the equipment code range [101, 105]"; "Prohibit pairing of equipment codes 101 and 301" (meaning data from machines in different processes cannot be associated). The system checks whether the equipment codes D_code_i and D_code_j for records i and j satisfy at least one allowed pairing rule. If they do, the equipment association verification passes.

[0092] Among them, determining the matching relationship: Records i and j are defined as having a definite matching relationship and belonging to the same detection event only when both Δt≤W_t are satisfied and the device association rule is verified.

[0093] Step S3.6: Attach a globally unique matching group identifier to each pair of structured time series data records that match the relationship, and integrate them into a set of matched records with matching association information.

[0094] Preferably, a globally unique MatchGroupID is assigned to each pair or group of records that determines a matching relationship (note that more than two records may be matched within a bucket through transitivity). This identifier is typically an incrementing integer or a UUID.

[0095] Specifically, identifier generation can use a thread-safe counter or `UUID.randomUUID().toString()` to generate a unique ID; identifier appending can iterate through all matching relationships, appending the same matching group ID to the metadata of all records within the group. For example, a `match_group_id` field can be added to the record object; the integrated output can form a set of matched records with matching association information, consisting of all records appended with `match_group_id`. A single record that does not match any other record can be assigned an independent matching group ID containing only itself, or output depending on business rules. This set is the direct input for the subsequent step S4, which performs sample-based data fusion.

[0096] Beneficial effects of steps S3.1 to S3.6: For structured time-series data records from different cigarette testing instruments, an automated and efficient data matching and association mechanism was constructed. Overall, these steps work together to solve the core problem of scattered and difficult-to-collect data of the same test sample caused by parallel testing by multiple instruments. By replacing the traditional hard-coded matching method that relies on manual rules or fixed logic with an algorithmic approach, the mechanism achieves automated, intelligent and highly scalable data association, providing a structured input for subsequent data fusion and feature analysis on a sample-by-sample basis. Step S3.1 extracts the absolute time value and the original string of the device identifier from each record, providing the core comparison dimension for subsequent similarity calculations. Step S3.2 converts the original string of the device identifier into a numerical device code using a preset mapping table, unifying the discrete, unstructured device names into semantic codes that can be used for mathematical operations, laying a formal foundation for the association between devices. Step S3.3 concatenates and normalizes the time value and the device code to form a high-dimensional feature vector representing the spatiotemporal attributes of the record, transforming the multidimensional matching problem into a nearest neighbor search problem in the vector space. Step S3.4 applies a locality-sensitive hashing algorithm based on random projection to perform a hash operation on this high-dimensional feature vector, the core effect of which is... This involves mapping potentially matching records to the same hash bucket with a high probability, thereby transforming the global, costly, and precise comparison into a local, fast intra-bucket comparison, significantly improving the efficiency of finding related records in massive amounts of data. Step S3.5 involves calculating the absolute difference of timestamps between records within the same hash bucket and comparing it with a preset time-series window threshold, while also verifying it in conjunction with the association rules of the device encoding. This process achieves a precise determination from "potentially related" to "confirmed as related," ensuring the accuracy of the matching relationship and the rationality of the business. Step S3.6 involves attaching a globally unique matching group identifier to each pair of confirmed matching records, thereby integrating scattered and isolated single records into a set of matched records with clear association information.

[0097] Step S4: All structured time series data records are fused into a multidimensional feature matrix with each detection sample as a unit by using a weighted aggregation algorithm based on an attention mechanism.

[0098] Furthermore, step S4 specifically includes the following steps: Step S4.1: Based on the matching group identifier of each structured time series data record in the matched record set, divide all structured time series data records into different record groups, with each record group corresponding to a unique detection sample.

[0099] Preferably, step S4.1 receives a set of matched records with matching association information from step S3. Each record in this set has been appended with a match_group_id field.

[0100] Specifically, the grouping structure can be initialized by creating a list of records with match_group_id as the key. <structuredrecord>A hash table (e.g., HashMap) where values ​​are the hash table's values. <String,List <structuredrecord>>), is called the record group mapping table.

[0101] Specifically, traversal and dispatch can be performed by iterating through each record in the input set. For each record, the record group mapping table is queried using its match_group_id as the key. If the key does not exist, an empty list is created and stored in the mapping table as the value; then the current record is appended to the corresponding list.

[0102] Specifically, after the output grouping is completed, each key-value pair (i.e., a match_group_id and its corresponding list of records) in the record group mapping table constitutes a record group, representing all the associated data generated by a unique test sample on multiple instruments.

[0103] Step S4.2: Extract all numerical data fields except timestamps and device identifiers from each structured time series data record in the current record group to obtain the original feature fragments of all structured time series data records in the current record group.

[0104] Preferably, for each record group (i.e., a list of records) in the record group mapping table, the original features can be extracted through the following operation: ① Field Filtering: Iterate through each StructuredRecord in the group. For each record, exclude predefined metadata fields (such as timestamp_ns, device_id itself, and possibly sample_id, etc.) from its numerical_fields and textual_fields. The remaining key-value pairs constitute the effective observational features for describing the sample quality of that record.

[0105] ② Vectorization: All features (assuming there are m features) of each record after filtering are arranged in a predefined, globally uniform field order (e.g., ["breathability", "weight", "length"...]). For missing fields, they are filled with 0.0 (numeric) or an empty string (text, but usually only numeric fields are processed later). Thus, each record is converted into a feature vector of length m, called the original feature fragment f_i of that record (where i represents the record's index within the group).

[0106] ③ Assembly: The original feature fragments [f_1, f_2, ..., f_n] of all records in a record group (n is the number of records in the group) are collected as the feature fragment set of the group.

[0107] Step S4.3: Arrange all original feature fragments in the current record group in chronological order and combine them into the initial serialized feature matrix corresponding to the current unique detection sample.

[0108] Preferably, for each set of feature segments [f_1, f_2, ..., f_n] of a record group, time sorting and matrix stacking are performed respectively.

[0109] The temporal sorting involves sorting the set [f_1, f_2, ..., f_n] in ascending order based on the timestamp_ns field of the original records corresponding to the feature segments. This ensures that f_1 corresponds to the earliest timestamp and f_n corresponds to the latest timestamp, forming a temporally evolving observation sequence.

[0110] The matrix stacking process involves stacking the sorted n feature vectors (each with dimension m) vertically to form an n×m matrix X. This matrix X is the initial serialized feature matrix corresponding to the detected sample. Here, the row dimension (n) represents the time step (different observation times or different instruments within the same detection event), and the column dimension (m) represents the feature dimension (different detection metrics).

[0111] Step S4.4: Calculate the attention weight distribution for each structured time series data record in the current initial serialized feature matrix based on the scaling dot product attention mechanism.

[0112] Preferably, for each initial serialized feature matrix X (size n×m), the attention weight for each time step (i.e., each row) is calculated using the ScaledDot-ProductAttention mechanism. The specific process is as follows: ① Linear Projection: For each row of X (i.e., each feature vector), a trainable projection matrix is ​​learned or defined to generate query, key, and value vectors. For simplicity, in this implementation, X can be directly considered as the projected representation, i.e., let Q=K=V=X. More generally, this can be achieved through three independent linear transformation layers: Q = XW^Q, K = XW^K, V = XW^V. Where W^Q, W^K, and W^V are learnable parameter matrices of dimensions m×d_k, m×d_k, and m×d_v, respectively. For simplicity, we assume d_k = d_v = m.

[0113] ② Calculate the attention score: Calculate the dot product of the query vector Q and all key vectors K, and divide by the scaling factor sqrt(d_k) to prevent the dot product value from being too large and causing gradient vanishing. The formula is: Scores = (Q * K^T) / sqrt(d_k). Here, * denotes matrix multiplication, and K^T is the transpose of K. The resulting Scores is an n×n matrix, where the element in the i-th row and j-th column represents the attention score of the i-th time step to the j-th time step.

[0114] ③ Obtain the weight distribution using Softmax: Apply the Softmax function to each row of the Scores matrix, making the sum of all weights in each row equal to 1, thus obtaining the formal attention weight distribution matrix A: A = softmax(Scores, dim = -1). Here, the i-th row of matrix A represents the attention weight distribution of the i-th time step across all n time steps (including itself). In practical applications, to allow the model to focus more on the information of the sequence itself, self-attention is often used, where Q, K, and V all originate from X.

[0115] Step S4.5: Based on the attention weight distribution, perform a weighted summation of the original feature segments at different time steps in the current initial serialization feature matrix to obtain the weighted aggregated feature vector corresponding to the current record group.

[0116] Preferably, the original feature sequences are weighted and fused using the attention weight distribution matrix A obtained in step S4.4: ① Weighted summation: Calculate the attention output matrix O = A * V. Here, V is the value vector matrix (i.e., X or XW^V). The effect of matrix multiplication A * V is that, for the i-th row of the output matrix O (i.e., the context vector at the i-th time step), it is a weighted sum of the value vectors V_j at all time steps, with the weights given by the i-th row of A.

[0117] ② Generating the Aggregated Vector: In simple aggregation scenarios, to obtain a single vector representing the entire sequence, we can take the mean of the attention output matrix O along the time step dimension (rows), or take the output of a specific position (such as the last row). More commonly, we can directly treat each row of O as an augmented representation for each time step. To obtain a single vector at the sample level, we can perform global average pooling or max pooling on all rows of O again. Assuming global average pooling is used, the weighted aggregated feature vector v_agg is calculated as follows: v_agg = (1 / n) * sum(O[i,:] for i in 1 ton). Where v_agg is a vector of dimension m (or d_v).

[0118] Step S4.6: Arrange the weighted aggregated feature vectors of all record groups according to the sequence number of the matching group identifier, and assemble them into a fused multidimensional feature matrix. Each row of the fused multidimensional feature matrix corresponds to a unique matching group identifier.

[0119] Preferably, steps S4.1 to S4.5 are repeated for all detected samples in the system (i.e., all match_group_ids). Each sample generates a corresponding weighted aggregated feature vector v_agg.

[0120] The sorting and indexing can be achieved by sorting all samples' match_group_id according to their intrinsic sequence number or generation order.

[0121] The assembly of the fused multidimensional feature matrix can be achieved by stacking all rows, with each sample's v_agg as a row and sorted by its match_group_id, to form a p×m matrix M. Here, p represents the total number of uniquely detected samples, and m is the dimension of the feature vectors. This matrix M is the fused multidimensional feature matrix, where each row uniquely corresponds to a detected sample and contains the comprehensive feature representation of that sample after multi-source data alignment and temporal weighted fusion, providing input for subsequent deep feature extraction.

[0122] Beneficial effects of steps S4.1 to S4.6: For data records from multiple detection sources that have been matched and associated, intelligent data fusion with time-series awareness is achieved on a per-sample basis. Overall, these steps, by introducing an attention mechanism, endow the system with the ability to distinguish the primary and secondary information and the focus of attention during the data fusion process. This effectively overcomes the information dilution or noise amplification problems that may be caused by the equal weighting of time-series data in traditional methods. It transforms the scattered, multi-point raw observation data into high-quality and information-rich sample-level feature representations that can be directly used for subsequent deep feature extraction. In step S4.1, the discrete structured time-series data records are divided into different record groups logically belonging to the same test sample based on the matching group identifier, establishing clear data boundaries and aggregation units for subsequent sample-level comprehensive analysis. Step S4.2 extracts all numerical data fields (excluding metadata) from each record within each record group to obtain the original feature fragments constituting the multidimensional quality profile of the sample. Step S4.3 arranges and combines these original feature fragments according to their inherent time order to construct the initial serialized feature matrix corresponding to each sample. This matrix not only retains multidimensional measurement information but also depicts the possible changes in detection parameters over time. Step S4.4 calculates the following based on the scaling dot product attention mechanism: The attention weight distribution recorded at different time steps in each initial serialized feature matrix is ​​used to enable the model to automatically learn and evaluate the relative importance of data records from different times or instruments to the final sample feature representation during the same detection process, rather than simply averaging or adding them with equal weights. Step S4.5 uses the dynamically generated attention weight distribution to perform differential weighted summation on the features at different time steps in the serialized feature matrix, thereby generating a single weighted aggregated feature vector that can more accurately and concisely reflect the overall state and key features of the detected sample. Step S4.6 finally arranges and assembles the weighted aggregated feature vectors corresponding to all samples according to their identifiers to form a complete fused multidimensional feature matrix.

[0123] Step S5: Input the fused multidimensional feature matrix into the stacked noise reduction autoencoder for feature extraction and dimensionality reduction to obtain a set of dense feature vectors after dimensionality reduction.

[0124] Furthermore, step S5 specifically includes the following steps: Step S5.1: Perform Z-score normalization preprocessing on the fused multidimensional feature matrix to obtain the normalized feature matrix.

[0125] Preferably, step S5.1 receives the fused multidimensional feature matrix M from step S4, which has dimensions of p×m (p samples, m features). Z-score normalization is performed independently on each column of the matrix M (i.e., each feature dimension).

[0126] Specifically, for the j-th feature column, calculate its mean μ_j and standard deviation σ_j across all p samples in the batch.

[0127] μ_j=(1 / p)*Σ_{i=1}^{p}M_{i,j}.

[0128] σ_j=sqrt((1 / (p-1))*Σ_{i=1}^{p}(M_{i, j}-μ_j)^2).

[0129] To prevent division by zero errors, σ_j is set to 1 when it is close to zero (e.g., less than 1e-8).

[0130] Specifically, for each element M_{i, j} in the matrix, the standardized value Z_{i, j} is obtained through the following transformation formula: Z_{i,j}=(M_{i,j}-μ_j) / σ_j. Performing this operation on all i,j yields the standardized feature matrix Z. Each column of matrix Z has a mean of 0 and a standard deviation of 1, eliminating the influence of different feature dimensions and numerical ranges.

[0131] Step S5.2: Add random noise that conforms to a Gaussian distribution to the standardized feature matrix to obtain a noisy input feature matrix.

[0132] Preferably, noise is applied to the normalized feature matrix Z by constructing a noise perturbation input using a stacked noise-reducing autoencoder.

[0133] Specifically, noise sampling can be achieved by generating a noise matrix N with the same dimensions as matrix Z, where each element N_{i,j} is independently sampled from a Gaussian (normal) distribution with a mean of 0 and a standard deviation of σ_noise, i.e., N_{i,j} ~ N(0, σ_noise^2). The noise standard deviation σ_noise is an adjustable hyperparameter, typically set to 0.1 or determined using a validation set.

[0134] Specifically, noise can be added element-wise by summing the noise matrix N with the normalized matrix Z to obtain the noisy input feature matrix ...

[0135] Step S5.3: Input the noisy input feature matrix into the first layer encoder of the stacked denoising autoencoder, and obtain the first layer encoded features through linear transformation and ReLU activation function.

[0136] Preferably, the noisy input feature matrix \(\tilde{Z}\) can be input into the first-layer encoder of the stacked denoising autoencoder by performing an affine transformation (linear transformation plus bias). Let the weight matrix of the first-layer encoder be \(W_1\) (dimension \(m\times h1\)), and the bias vector be \(b_1\) (dimension \(h1\)), where \(h1\) is the number of the first-layer hidden units (\(h1 < m\), achieving dimensionality reduction). The transformation formula is: \(H1\_raw = \tilde{Z}*W_1 + b_1\), where \(*\) represents matrix multiplication.

[0137] Then, apply the ReLU (Rectified Linear Unit) activation function to the result \(H1\_raw\) of the linear transformation through a non-linear activation to introduce non-linearity. The definition of the ReLU function is \(f(x)=max(0, x)\), which is an element-wise operation. Finally, the first-layer encoded feature \(H1 = ReLU(H1\_raw)\) is obtained, and its dimension is \(p\times h1\).

[0138] Step S5.4, successively pass the first-layer encoded feature through several subsequent encoding layers of the stacked denoising autoencoder, layer by layer reducing the feature dimension of each row of the noisy input feature matrix, and obtaining the low-dimensional encoded feature at the bottleneck layer.

[0139] Preferably, take the first-layer encoded feature \(H1\) as the input and sequentially pass it through multiple subsequent encoding layers of the stacked denoising autoencoder. Assume there are a total of \(L\) encoding layers (including the first layer), and the operations of the \(l\)-th layer (\(l\) ranges from 2 to \(L\)) can be uniformly described as: \(H_l = ReLU(H_{l - 1}*W_l + b_l)\). Among them, \(W_l\) is the weight matrix of the \(l\)-th layer (dimension \(h_{l - 1}\times h_l\)), \(b_l\) is the bias vector (dimension \(h_l\)), and \(h_l\) is the number of units in the \(l\)-th layer. The network design follows the principle of dimensionality reduction, that is, \(h_1 > h_2 > \cdots > h_L\). The output \(H_L\) of the last encoding layer (the \(L\)-th layer) has the smallest dimension \(h_L\), and this layer is the bottleneck layer. \(H_L\) is the low-dimensional encoded feature, which is a highly compressed and abstract representation of the original input data, with a dimension of \(p\times h_L\) (\(h_L\) is much smaller than the initial feature dimension \(m\)).

[0140] Step S5.5, take the low-dimensional encoded feature as the input of the decoder part of the stacked denoising autoencoder, and perform data reconstruction through a symmetric decoding layer structure to obtain the reconstructed feature matrix.

[0141] Preferably, input the low-dimensional encoded feature \(H_L\) into the decoder part to reconstruct the original, non-noisy, standardized input \(Z\).

[0142] Specifically, for the symmetric decoding structure, the decoder is usually symmetric with the encoder and contains \(L\) layers. The operations of the \(l\)-th layer decoder (\(l\) ranges from \(L + 1\) to \(2L\)) can be described as: D_{l} = ReLU(D_{l-1}*W'_{l}+b'_{l}). Here, D_{L} = H_L is the initial input to the decoder. The dimension design of the decoding layer weights W'_{l} causes the feature dimension to increase layer by layer, ultimately restoring the dimension of the last decoding layer D_{2L} to m.

[0143] Specifically, for the output layer, the last decoding layer typically does not use ReLU activation, but instead uses linear or sigmoid activation (if the input data is normalized to between 0 and 1) to generate reconstructed values. The final reconstructed feature matrix \hat{Z} has the same dimensions as the original Z, which is p × m. The goal of model training is to minimize the reconstruction error, for example, using the mean squared error (MSE) loss function: Loss = (1 / (p*m)) * Σ(Z - \hat{Z})^2.

[0144] Step S5.6: Discard the decoder part of the stacked noise reduction autoencoder, retain the path from the first encoder layer to the bottleneck layer of the stacked noise reduction autoencoder, input the standardized feature matrix into the path, and extract the dimensionality-reduced dense feature vector set.

[0145] Preferably, after the model training phase is completed, the parameters {W_1, b_1, ..., W_L, b_L} of all encoder layers are fixed. The normalized feature matrix Z (not tilde{Z}) without added noise is used as input, and forward propagation is performed only through the encoder part (from the first layer to the bottleneck layer of the Lth layer).

[0146] Preferably, the feature extraction calculation process is H1=ReLU(Z*W_1+b_1), and then sequentially calculated up to the bottleneck layer H_L=ReLU(H_{L-1}*W_L+b_L). The output H_L of the bottleneck layer is the required dense feature vector set after dimensionality reduction.

[0147] Preferably, the feature vector set is a p×h_L matrix, where each row corresponds to an original detection sample, representing its core feature representation after deep nonlinear transformation and dimensionality reduction. This output will be passed to step S6 for further processing.

[0148] Beneficial effects of steps S5.1 to S5.6: Deep feature learning and information compression are performed on the fused multidimensional feature matrix generated in the previous steps, which is based on the detection samples, to extract more representative and robust data representations. Overall, these steps automatically learn the mapping relationship from the high-dimensional, possibly redundant fused matrix to the low-dimensional, dense feature vector during adversarial training by stacking a noise-reducing autoencoder encoder-decoder framework. This not only achieves data dimensionality reduction and facilitates subsequent storage and computation, but more importantly, it removes noise, discovers robust intrinsic patterns and deep features in the data, and improves the representational ability and generalization of the feature vectors. In this process, step S5.1 performs Z-score standardization preprocessing on the fused multidimensional feature matrix to eliminate biases caused by differences in dimensions and numerical ranges between different feature dimensions, ensuring that all features are on the same order of magnitude, thus laying the foundation for stable training and convergence of the subsequent model. Step S5.2 adds Gaussian-distributed random noise to the standardized feature matrix to generate a noisy input feature matrix. This operation simulates the interference and incompleteness that may exist in real-world data, forcing the model to learn to denoise and capture the robust underlying structure of the data. Step S5.3 inputs the noisy input feature matrix into the first layer of the stacked denoising autoencoder, and performs preliminary nonlinear mapping and feature abstraction through linear transformation and the ReLU activation function to obtain the first layer of encoded features. 4. The encoded features are passed through multiple subsequent encoding layers, undergoing higher-order nonlinear transformations and feature compression layer by layer. Redundant information is gradually discarded and core patterns are extracted. Finally, a low-dimensional encoded feature with significantly reduced dimensionality is obtained at the bottleneck layer, achieving efficient data compression and explicit expression of latent features. Step S5.5. The low-dimensional encoded features are reconstructed using a symmetrical decoder structure to attempt to restore the structure of the original input. This reconstruction process and the encoding process together constitute self-supervised training, driving the model to learn low-dimensional encoding that must contain enough key information to reconstruct the original data. Step S5.6. After the model training converges, the encoder parameters are fixed, and the standardized feature matrix without added noise is directly input into the encoder to the bottleneck layer path to extract the dimensionality-reduced dense feature vector set.

[0149] Step S6: Output the dimensionality-reduced dense feature vector set as the result of standardized multi-source data processing.

[0150] Furthermore, step S6 specifically includes the following steps: Step S6.1: Bind each row of the dimensionality-reduced dense feature vector set to the corresponding matching group identifier to generate feature data pairs with sample identifiers.

[0151] Preferably, step S6.1 receives the dimensionality-reduced dense feature vector set from step S5. This set is typically represented in memory as a two-dimensional array or matrix F with dimensions p×h_L (p samples, each with h_L dimensional features). Simultaneously, the system inherits or maintains a list of matching group identifiers G=[g_1,g_2,…,g_p] that strictly corresponds to the row order of F from the upstream processing steps (steps S3 / S4).

[0152] The next step is to establish a mapping: ensure that the i-th row of the vector set F corresponds one-to-one with the i-th element g_i of the identifier list G, representing the same detection sample.

[0153] The next step is to construct data pairs: by iterating through i from 1 to p, F[i] (the feature vector of the i-th sample) and g_i are combined into a structured data unit, namely the feature data pair Pair_i={"match_group_id":g_i, "feature_vector":F[i]}. In implementation, this can be a custom class instance, a tuple, or a dictionary (such as dict in Python or Map in Java).

[0154] The next step is to output the set: all p pairs of data form an ordered set, which is the generated set of feature data pairs with sample identifiers, P = {Pair_1, Pair_2, ..., Pair_p}. This step ensures that the association between core feature data and business entities is never lost.

[0155] Step S6.2: For the dimensionality-reduced dense feature vector set part of the feature data pair, convert all feature data pairs into binary byte streams through data serialization operations according to the Apache Avro format specification.

[0156] Preferably, for each feature data pair Pair_i in set P, data serialization based on Apache Avro is applied to its feature_vector portion.

[0157] First, an Avro pattern needs to be predefined to describe the structure of the feature vector. This pattern is usually defined in JSON format and embedded in the program or stored in a pattern registry.

[0158] For example, the following JSON statement: { "type": "record", "name": "ProcessedFeature", "fields": [ { "name": "match_group_id", "type": "string"}, { "name": "feature_dimension", "type": "int"}, { "name": "features", "type": { "type": "array", "items": "float"}}, { "name": "generation_timestamp", "type": "long", "logicalType":"timestamp-millis"} ] } Secondly, a specific Avro compiler (e.g., avro-tools) is used to generate the corresponding programming language classes (e.g., Java classes) based on the above pattern, or a generic reader / writer (GenericDatumWriter) is used. The core Java code snippet for the serialization process is as follows: / / Let `featureRecord` be a record object generated based on a schema and already populated with data. GenericRecord avroRecord = new GenericData.Record(schema); avroRecord.put("match_group_id", pair.getMatchGroupId()); avroRecord.put("feature_dimension", h_L); avroRecord.put("features", pair.getFeatureVector()); / / Set List <float> avroRecord.put("generation_timestamp", System.currentTimeMillis()); ByteArrayOutputStream outputStream = new ByteArrayOutputStream(); BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(outputStream, null); DatumWriter <genericrecord>writer = new GenericDatumWriter<>(schema); writer.write(avroRecord, encoder); encoder.flush(); byte[] serializedBytes = outputStream.toByteArray(); / / Binary byte stream Step S6.3: Append a fixed-format message header, including data version, processing timestamp and data mode identifier, to each binary byte stream, and integrate them into a complete data message.

[0159] Preferably, a fixed message header can be defined, which typically includes: ① Protocol Version: 1 byte, used to identify the version of the data format (e.g., 0x01).

[0160] ② Timestamp: 8 bytes, representing the Unix timestamp (millisecond precision) when the data packet was created.

[0161] ③ Data Schema ID: 4 bytes, a unique identifier hash value corresponding to the Avro schema, used to find the correct schema during deserialization.

[0162] ④ Payload Length: 4 bytes, representing the number of bytes in the subsequent Avro binary data portion.

[0163] Step S6.4: Calculate the cyclic redundancy check code of the complete data message and append it to the end of the complete data message to obtain the standardized multi-source data processing result.

[0164] Preferably, to ensure data integrity during transmission or storage, a checksum can be calculated for the complete data message. For example, using the standard Cyclic Redundancy Check (CRC-32) or CRC-32C algorithm, the calculated 32-bit (4-byte) checksum value is appended to the end of the complete data message in little-endian or big-endian byte order (which must be agreed upon by the receiver).

[0165] Beneficial effects of steps S6.1 to S6.4: This paper constructs a complete, reliable, and standardized data output and encapsulation mechanism for the dimensionality-reduced dense feature vector set produced in the preceding steps. Overall, these steps collectively transform abstract data features into data messages with clear semantics, standardized formats, complete metadata, and integrity protection. This completes the final transformation from internal processing to providing standardized services externally, offering a ready-to-use, reliable, and efficient data interface for upper-layer applications. Specifically, step S6.1 binds each row of data in the dimensionality-reduced dense feature vector set with its corresponding matching group identifier, generating feature data pairs with sample identification. This operation establishes a robust and unambiguous association between the core feature data and the original physical detection sample it represents, ensuring the traceability of the data processing results and the clarity of the business meaning. Step S6.2 applies data serialization operations based on the Apache Avro format specification to the feature vector portion of the feature data pairs, converting it into a compact binary byte stream. This process not only compresses the data volume to improve transmission and storage efficiency but also achieves self-descriptive data structure through Avro's built-in schema description, providing a standardized, programming language-independent intermediate frame for cross-system and cross-platform data exchange. Step S6.3: Append a fixed-format message header containing data version, processing timestamp, and data pattern identifier to each serialized binary byte stream, and integrate them into a complete data message. This step encapsulates crucial metadata information for the original data content, enabling the receiving end to accurately identify the data format version, determine data freshness, and correctly parse the data body according to the pattern identifier, thereby ensuring the consistency and understandability of the data in subsequent transmission and application. Step S6.4: Calculate the cyclic redundancy check (CRC) code of the complete data message and append it to the end of the message. This operation provides end-to-end integrity verification capability for the entire data packet, effectively detecting any unexpected errors or tampering that may occur during data transmission or storage, and is a key link in ensuring the high reliability of data processing results delivery.

[0166] Beneficial effects of steps S1 to S6: To address the challenges of chaotic multi-source instrument data access, difficult parsing, weak correlation, data silos, and complex subsequent analysis and processing in cigarette production quality inspection, this method constructs a complete and flexible data processing pipeline from heterogeneous access to standardized output, reducing the technical complexity and maintenance costs of multi-source data integration. Specifically, step S1, through a unified interface protocol and automated connection, achieves seamless access to heterogeneous data sources, avoiding the enormous workload and system rigidity issues associated with developing custom acquisition programs for each instrument, providing a stable and real-time multi-source data stream for subsequent processing. Step S2 utilizes predefined formatted description templates and automated parsing technology to uniformly transform unstructured raw data into structured records with standard timestamps and device identifiers, overcoming the difficulties in data interpretation caused by protocol and format differences, and laying the foundation for data correlation. Step S3, through a similarity matching algorithm based on locality-sensitive hashing, automatically and efficiently identifies and correlates data records from different instruments belonging to the same detection event, solving the problems of data fragmentation and missing correlations caused by dispersed data sources, and achieving logical correlation based on the test sample. Step S4 employs a weighted aggregation algorithm based on an attention mechanism to assign differentiated weights to the correlated time-series data and fuse them, effectively integrating multidimensional measurement information from multiple instruments and different time points to generate a fusion feature matrix that can more comprehensively characterize the quality status of individual samples. Step S5 uses a stacked noise reduction autoencoder to automatically extract features and reduce the dimensionality of the high-dimensional fusion matrix, removing redundancy and noise while retaining core information, compressing complex multi-source mixed data into low-dimensional, dense, and more representative feature vectors, significantly improving the analyzability of the data. Step S6 finally outputs the processing results in the form of a standardized data package with complete verification information, ensuring the integrity, consistency, and high reliability of the data processing results, providing a ready-to-use, unified, and high-quality data foundation for upper-level quality analysis, process control, and other applications.

[0167] like Figure 2 As shown, this embodiment provides an example of a multi-source data processing system for a cigarette testing instrument. In this embodiment, the multi-source data processing system is applied to the multi-source data processing method described in the above embodiment.

[0168] Specifically, the multi-source data processing system includes a heterogeneous raw data acquisition module 1, a heterogeneous raw data conversion module 2, a time series data recording association module 3, a time series data recording fusion module 4, a fusion multi-dimensional feature matrix processing module 5, and a multi-source data processing result output module 6, which are electrically or communicatively connected in sequence.

[0169] The system comprises the following modules: Heterogeneous Raw Data Acquisition Module 1, which accesses several heterogeneous raw data sources from different cigarette testing instruments via network interface, database connection, serial communication, and file interface protocol; Heterogeneous Raw Data Conversion Module 2, which performs context-free grammar-based parsing and format recognition on each multi-source heterogeneous raw data source and converts it into structured time-series data records with timestamps and device identifiers; Time-Series Data Record Association Module 3, which inputs the current structured time-series data record into a locality-sensitive hash-based similarity matching algorithm to match and associate structured time-series data records from different cigarette testing instruments based on timestamp proximity and device identifier association; Time-Series Data Record Fusion Module 4, which fuses all structured time-series data records into a fused multidimensional feature matrix based on the tested sample using a weighted aggregation algorithm based on an attention mechanism; Fusion Multidimensional Feature Matrix Processing Module 5, which inputs the fused multidimensional feature matrix into a stacked noise reduction autoencoder for feature extraction and dimensionality reduction to obtain a dimensionality-reduced dense feature vector set; and Multi-Source Data Processing Result Output Module 6, which outputs the dimensionality-reduced dense feature vector set as the standardized multi-source data processing result.

[0170] It should be noted that this embodiment is a functional module embodiment based on the above method embodiment. For additional content such as extensions, optimizations, limitations, examples, principle explanations, and beneficial effects of this embodiment, please refer to the above embodiments. This embodiment will not repeat them here.

[0171] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Figure 3 As shown, the electronic device 7 includes a processor 71 and a memory 72 coupled to the processor 71.

[0172] The memory 72 stores program instructions for implementing the federated learning-based collaborative energy-saving method for government data clusters in any of the above embodiments.

[0173] The processor 71 is used to execute program instructions stored in the memory 72 for collaborative energy saving of government data clusters based on federated learning.

[0174] The processor 71 can also be referred to as a CPU (Central Processing Unit). The processor 71 may be an integrated circuit chip with signal processing capabilities. The processor 71 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor.

[0175] Furthermore, Figure 4 This is a schematic diagram of the structure of a storage medium according to an embodiment of this application. See also: Figure 4 The storage medium 8 in this embodiment stores program instructions 81 capable of implementing all the above methods. These program instructions 81 can be stored in the storage medium as a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in each embodiment of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets.

[0176] In the several embodiments provided in this application, it should be understood that the disclosed systems, methods, and approaches can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, signal, or other forms.

[0177] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.< / genericrecord> < / float> < / structuredrecord> < / structuredrecord>

Claims

1. A multi-source data processing method for cigarette testing instruments, wherein the multi-source data processing method is applied to several different cigarette testing instruments, characterized in that, The multi-source data processing method includes: Step S1: Access several heterogeneous raw data from different cigarette testing instruments through network interface, database connection, serial communication, and file interface protocol; Step S2: For each multi-source heterogeneous raw data, perform parsing and format recognition based on context-free grammar, and convert it into structured time-series data records with timestamps and device identifiers; Step S3: Input the current structured time series data record into the similarity matching algorithm based on locality-sensitive hashing, and match and associate the structured time series data records from different cigarette testing instruments according to the proximity of timestamps and the correlation of device identifiers; Step S4: All structured time series data records are fused into a multidimensional feature matrix with each detection sample as a unit using a weighted aggregation algorithm based on an attention mechanism. Step S5: Input the fused multidimensional feature matrix into the stacked noise reduction autoencoder for feature extraction and dimensionality reduction to obtain a dimensionality-reduced dense feature vector set; Step S6: Output the dimensionality-reduced dense feature vector set as the result of standardized multi-source data processing.

2. The multi-source data processing method according to claim 1, characterized in that, Step S1 involves accessing several heterogeneous raw data sets from different cigarette testing instruments via network interface, database connection, serial communication, and file interface protocol, including: Step S1.1: Detect and determine the type and connection parameters of the data source to be connected by using the external device instrument list, and generate a data source connection configuration set; Step S1.2: Based on the type identifier in the data source connection configuration set, call the standard database connection driver for the data source identified as a relational database to establish a network database connection; Step S1.3: Based on the type identifier in the data source connection configuration set, for data sources identified as serial communication devices, establish a serial communication connection according to the baud rate and parity parameters in the data source connection configuration set; Step S1.4: Based on the type identifier in the data source connection configuration set, for data sources identified as file services, establish a secure file transfer protocol connection according to the address and authentication information in the data source connection configuration set; Step S1.5: Perform connectivity verification tests on the network database connection, the serial communication connection, and the secure file transfer protocol connection to obtain a verified valid connection pool; Step S1.6: After verification, the valid connection pool concurrently executes data query instructions, serial port data read instructions, and file pull instructions to obtain a multi-source data stream in the form of an initial byte stream or character stream; Step S1.7: Encapsulate the multi-source data stream and add source device identifier and timestamp to obtain raw data units in standard packet format; Step S1.8: Continuously output the original data units along the time process to obtain the multi-source heterogeneous original data.

3. The multi-source data processing method according to claim 1, characterized in that, Step S2 involves parsing and format recognition of each multi-source heterogeneous raw data based on context-free grammar, and converting it into structured time-series data records with timestamps and device identifiers, including: Step S2.1: Assign the original data unit to the corresponding parsing channel according to the source device identifier of the original data unit; Step S2.2: Load a preset data format description template corresponding to the device model for each parsing channel. The preset data format description template includes a syntax structure for defining data in the extended Backus paradigm. Step S2.3: Based on the preset data format description template, perform syntax parsing on the payload part of each original data unit using the recursive descent method to obtain a parsed dataset based on each original data unit; Step S2.4: Apply key-value pair extraction rules based on a preset pattern to the parsed dataset of the current original data unit to map unstructured text fields or binary segments into a set of key-value pairs. Step S2.5: Merge the key-value pair set of the current original data unit with the corresponding timestamp and device identifier to obtain a preliminary associated tuple sequence; Step S2.6: Based on the IEEE floating-point standard and string conversion, normalize the numerical values ​​in each preliminary associated tuple sequence respectively; Step S2.7: Sort each normalized preliminary associated tuple sequence in ascending order by timestamp and encapsulate it into a structured time series data record with a fixed field structure.

4. The multi-source data processing method according to claim 1, characterized in that, Step S3 involves inputting the current structured time-series data record into a similarity matching algorithm based on locality-sensitive hashing. This algorithm matches and associates structured time-series data records from different cigarette testing instruments based on timestamp proximity and device identifier correlation, including: Step S3.1: Extract the absolute time value of the timestamp and the original string of the device identifier from the current structured time series data record; Step S3.2: Convert the original string of the device identifier into a semantically related numerical device code through a preset mapping table; Step S3.3: The absolute time value and the numerical device code are concatenated and normalized, and combined into a high-dimensional feature vector representing the spatiotemporal attributes of the record. Step S3.4: Input the high-dimensional feature vector into the locality-sensitive hash algorithm based on random projection for hash operation, and allocate all structured time series data records to the corresponding hash buckets according to the calculated hash value; Step S3.5: Within the same hash bucket, calculate the absolute difference of absolute time values ​​between any two structured time series data records, and define structured time series data records whose absolute difference is less than the preset time series window threshold as having a definite matching relationship; Step S3.6: Attach a globally unique matching group identifier to each pair of structured time series data records that match the relationship, and integrate them into a set of matched records with matching association information.

5. The multi-source data processing method according to claim 4, characterized in that, Step S4: All structured time-series data records are fused into a multi-dimensional feature matrix based on the detection sample using a weighted aggregation algorithm based on an attention mechanism. This includes: Step S4.1: Based on the matching group identifier of each structured time series data record in the matched record set, divide all structured time series data records into different record groups, with each record group corresponding to a unique detection sample. Step S4.2: Extract all numerical data fields except timestamps and device identifiers from each structured time series data record in the current record group to obtain the original feature fragments of all structured time series data records in the current record group; Step S4.3: Arrange all the original feature fragments in the current record group in chronological order and combine them into the initial serialized feature matrix corresponding to the current unique detection sample; Step S4.4: Calculate the attention weight distribution of each structured time series data record in the current initial serialized feature matrix based on the scaling dot product attention mechanism; Step S4.5: Based on the attention weight distribution, perform a weighted summation of the original feature segments at different time steps in the current initial serialization feature matrix to obtain the weighted aggregated feature vector corresponding to the current record group; Step S4.6: Arrange the weighted aggregated feature vectors of all record groups according to the sequence number of the matching group identifier, and assemble them into a fused multidimensional feature matrix. Each row of the fused multidimensional feature matrix corresponds to a unique matching group identifier.

6. The multi-source data processing method according to claim 1, characterized in that, Step S5: Input the fused multidimensional feature matrix into the stacked denoising autoencoder for feature extraction and dimensionality reduction to obtain a dimensionality-reduced dense feature vector set, including: Step S5.1: Perform Z-score normalization preprocessing on the fused multidimensional feature matrix to obtain a normalized feature matrix; Step S5.2: Add random noise conforming to a Gaussian distribution to the standardized feature matrix to obtain a noisy input feature matrix; Step S5.3: Input the noisy input feature matrix into the first layer encoder of the stacked denoising autoencoder, and obtain the first layer encoded features through linear transformation and ReLU activation function; Step S5.4: The first layer encoded features are passed sequentially through several subsequent encoding layers of the stacked noise reduction autoencoder to reduce the feature dimension of each row of the noisy input feature matrix layer by layer, and low-dimensional encoded features are obtained at the bottleneck layer. Step S5.5: The low-dimensional encoded features are used as input to the decoder part of the stacked noise reduction autoencoder, and the data is reconstructed through a symmetric decoding layer structure to obtain the reconstructed feature matrix. Step S5.6: Discard the decoder part of the stacked noise reduction autoencoder, retain the path of the stacked noise reduction autoencoder from the first encoder layer to the bottleneck layer, input the normalized feature matrix into the path, and extract the dimensionality-reduced dense feature vector set.

7. The multi-source data processing method according to claim 1, characterized in that, Step S6, output the dimensionality-reduced dense feature vector set as the result of standardized multi-source data processing, including: Step S6.1: Bind each row of the dimensionality-reduced dense feature vector set to the corresponding matching group identifier to generate feature data pairs with sample identifiers; Step S6.2: For the dimensionality-reduced dense feature vector set portion of the feature data pair, convert all feature data pairs into binary byte streams using the data serialization operation specified in the Apache Avro format. Step S6.3: Append a fixed-format message header, including data version, processing timestamp and data mode identifier, to each binary byte stream, and integrate them into a complete data message; Step S6.4: Calculate the cyclic redundancy check code of the complete data message and append it to the end of the complete data message to obtain the standardized multi-source data processing result.

8. A multi-source data processing system for a cigarette testing instrument, wherein the multi-source data processing system is applied to the multi-source data processing method as described in any one of claims 1 to 7, characterized in that, The multi-source data processing system includes: The heterogeneous raw data acquisition module is used to access several heterogeneous raw data from different cigarette testing instruments through network interface, database connection, serial communication, and file interface protocol. The heterogeneous raw data conversion module is used to parse and identify the format of each multi-source heterogeneous raw data based on context-free grammar, and convert it into structured time series data records with timestamps and device identifiers. The time series data record association module is used to input the current structured time series data record into the similarity matching algorithm based on locality-sensitive hashing, and to match and associate the structured time series data records from different cigarette testing instruments according to the timestamp proximity and device identifier association. The time series data recording fusion module is used to fuse all structured time series data records into a fused multidimensional feature matrix based on the detection sample through a weighted aggregation algorithm based on an attention mechanism. The fused multidimensional feature matrix processing module is used to input the fused multidimensional feature matrix into the stacked noise reduction autoencoder for feature extraction and dimensionality reduction, to obtain a dimensionality-reduced dense feature vector set. The multi-source data processing result output module is used to output the dimensionality-reduced dense feature vector set as the standardized multi-source data processing result.

9. An electronic device, characterized in that, The method includes a processor and a memory coupled to the processor, the memory storing program instructions executable by the processor; when the processor executes the program instructions stored in the memory, it implements the multi-source data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that, when executed by a processor, enable the implementation of the multi-source data processing method as described in any one of claims 1 to 7.

Images