A tea feature data management method and system based on cloud-edge collaboration

Abstract

The application relates to the technical field of cloud-edge collaborative data processing, and discloses a tea feature data management method and system based on cloud-edge collaboration. The method obtains tea initial description data uploaded by an edge node and parses the tea initial description data into tea initial feature data; each key-value pair sequence is labeled and matched with a preset tea feature professional dictionary library through a preset conditional random field model, and field definition and numerical standards are extracted; the field definition and the numerical standards are compared with a preset metadata specification benchmark library to obtain a comprehensive difference value, a heterogeneous characteristic is determined and standardized conversion is carried out, and a standard description data set is obtained; semantic offset is calculated by extracting text description information and noise records are removed, and origin identification information and processing technology information are obtained to perform cross-region quality correlation analysis, and preset evaluation rule library is mapped to generate an analysis result. The application unifies multi-source heterogeneous tea feature data and suppresses semantic noise, and the accuracy of cross-region tea quality correlation analysis is improved.

Description

Technical Field

[0001] This invention relates to the field of cloud-edge collaborative data processing technology, and in particular to a method and system for managing tea feature data based on cloud-edge collaboration. Background Technology

[0002] In the tea quality traceability scenario of edge computing and cloud collaboration, edge acquisition nodes are deployed in tea gardens and tea processing workshops to collect various characteristic data such as tea origin and processing. Relying on the edge-cloud collaborative architecture, local data preprocessing and cloud-based global joint analysis are realized. The core goal of this data management process is to complete the standardized cleaning of multi-source heterogeneous tea data and accurately explore the intrinsic relationship between origin, processing technology and tea quality, so as to realize intelligent evaluation of tea quality across the entire domain.

[0003] However, existing edge-cloud collaborative solutions have not established a unified metadata benchmark to quantify heterogeneous differences in multi-source tea heterogeneous data. Due to differences in the deployment time of edge devices in various production areas, supplier differences, and localization customization, the uploaded feature data have significant differences in field naming, numerical units, and encoding formats. These differences directly prevent the cloud from processing data from different nodes in a unified manner. During the data collection process, edge nodes may mix in noisy records due to factors such as sensor failure, manual input errors, and network transmission interference. These noisy records manifest as semantic deviations at the text description level. This semantic deviation will distort the true quality correlation between different production areas and different processes in subsequent cross-regional quality analysis, making the quality evaluation results based on multi-source data fusion unreliable. Ultimately, the entire traceability system cannot provide accurate basis for tea quality judgment.

[0004] Therefore, existing technologies are limited by the strong heterogeneity and high noise interference of tea characteristic data collected from edge nodes, making it difficult to complete cross-regional tea quality correlation analysis. This further makes it difficult for the tea product quality traceability system to output accurate and reliable tea quality judgment criteria. Summary of the Invention

[0005] This invention provides a cloud-edge collaborative tea feature data management method and system to solve the problems of existing cloud-edge collaborative solutions that fail to establish a unified metadata benchmark for multi-source heterogeneous tea feature data, lack automatic field definition extraction and semantic denoising capabilities, resulting in inaccurate cross-regional tea quality correlation analysis results, and consequently making it difficult for the tea product quality traceability system to output accurate and reliable tea quality judgment criteria.

[0006] Firstly, to address the aforementioned technical problems, this invention provides a method for managing tea feature data based on cloud-edge collaboration, comprising:

[0007] Get the initial description data of tea uploaded by the edge node, parse the initial description data of tea to obtain structured initial feature data of tea;

[0008] The key-value pairs in the initial feature data of tea are traversed, and the key name and numerical text in each key-value pair are sequence-annotated by a preset conditional random field model. The sequence annotation results are matched with a preset professional dictionary of tea features, and the field definition and numerical standard of each key-value pair are extracted based on the matching confidence.

[0009] The field definition and the numerical standard are compared with the benchmark field definition and benchmark numerical standard in the preset metadata specification benchmark library to obtain a comprehensive difference value. Based on the comparison result of the comprehensive difference value and the preset heterogeneity threshold, the heterogeneity characteristics of the initial feature data of tea are determined, and a standardization transformation is performed to obtain the standard description dataset of tea.

[0010] Textual description information is extracted from the standard description dataset, semantic offset is calculated and noisy records are removed to obtain the tea's origin identification information and processing technology information. Cross-regional quality correlation analysis is then performed, and the results are generated by mapping the quality correlation analysis results to a preset evaluation rule base.

[0011] Secondly, the present invention provides a tea feature data management system based on cloud-edge collaboration, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described above.

[0012] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] (1) This invention uses a preset conditional random field model to perform sequence labeling on key-value pairs of initial tea feature data, and then extracts field definitions and numerical standards after matching with a preset professional dictionary library of tea features. This automatically maps the original fields with different names and dimensions of each edge node to a unified standard field and numerical standard. Since the field definitions are derived from the standard entries of the domain dictionary rather than the local naming of the nodes, the heterogeneity of field naming and encoding format of multi-source nodes is eliminated, and a unified metadata benchmark is established for the consistency processing of cross-node data.

[0015] (2) The present invention compares the field definition and numerical standard with the metadata specification benchmark library in terms of structural dimension and numerical distribution dimension, respectively, to obtain the first difference value and the second difference value, and integrates them into a comprehensive difference value and compares it with the preset heterogeneity threshold. Since this judgment simultaneously describes two independent dimensions, field structure mapping deviation and numerical distribution deviation, it can more comprehensively quantify the degree of heterogeneity compared with single-dimensional comparison, thereby accurately identifying the truly heterogeneous data that needs to be cleaned before standardization conversion, and avoiding unnecessary conversion of already standardized data and introducing errors.

[0016] (3) The present invention first vectorizes the text description information and calculates its semantic similarity value with the preset standard feature vector. Based on this, the semantic offset is determined. Then, based on the distribution statistical characteristics of the semantic offset, anomaly classification calculation is performed and noisy records are removed according to the anomaly confidence level. Since noise is manifested as a offset relative to the standard semantics at the text semantic level, the statistical distribution of the semantic offset can be used as an anomaly criterion to identify records with normal dimensions but distorted semantics. Thus, semantic noise that will distort the association results is removed before cross-regional quality association analysis, making the final cross-regional quality association analysis results more accurate and the quality judgment basis more reliable. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the tea feature data management method based on cloud-edge collaboration provided in an embodiment of the present invention;

[0018] Figure 2 This is a schematic diagram of CRF sequence labeling matching confidence provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] like Figure 1 As shown, an embodiment of the present invention provides a tea feature data management method based on cloud-edge collaboration, including steps S11 to S14:

[0021] S11, Obtain the initial description data of tea uploaded by the edge node, parse the initial description data of tea to obtain structured initial feature data of tea;

[0022] S12, traverse each key-value pair in the initial feature data of tea, perform sequence labeling on the key name and numerical text in each key-value pair through a preset conditional random field model, match the sequence labeling results with a preset professional dictionary of tea features, and extract the field definition and numerical standard of each key-value pair according to the matching confidence.

[0023] S13, compare the field definition and the numerical standard with the benchmark field definition and benchmark numerical standard in the preset metadata specification benchmark library to obtain a comprehensive difference value. Based on the comparison result of the comprehensive difference value and the preset heterogeneity threshold, determine the heterogeneity characteristics of the initial feature data of tea and perform standardization transformation to obtain the standard description dataset of tea.

[0024] S14, extract the text description information from the standard description dataset, calculate the semantic offset and remove noisy records to obtain the tea's origin identification information and processing technology information, and perform cross-regional quality correlation analysis. Based on the quality correlation analysis results, map the preset evaluation rule base to generate analysis results.

[0025] This invention applies to the field of cloud-edge collaborative tea feature data management. For ease of explanation, this embodiment uses a scenario where edge acquisition nodes deployed in three tea gardens and two tea processing workshops upload tea feature data to the cloud as an illustrative example. The edge acquisition nodes and their uploaded encrypted data packets serve as the source of the initial tea description data, thus providing an exemplary description of the method of this invention.

[0026] In step S11, the initial tea description data uploaded by the edge node is obtained, and the initial tea description data is parsed to obtain structured initial tea feature data, including:

[0027] Decrypt the encrypted byte stream corresponding to the initial description data of the tea leaves to obtain a memory data object;

[0028] Perform an integrity check on the memory data object, and determine the memory data object that passes the check as the structured initial feature data of tea leaves.

[0029] The initial tea description data consists of multiple features collected by edge acquisition nodes, including tea origin, processing technology, physicochemical indicators, and sensory evaluation. These features are then serialized, encrypted, and uploaded via a cloud-edge communication link. It should be noted that after the cloud reads the encrypted byte streams uploaded by each edge node from the message queue, it decrypts them using the symmetric key agreed upon with the edge node during encapsulation, and then deserializes them according to the agreed serialization protocol to obtain memory data objects organized in key-value pairs. The keys are feature names (text) and the values ​​are the corresponding observation content. For example, a memory data object can be represented as a record containing key-value pairs such as "Dry tea appearance: tightly rolled," "Tea polyphenol content: 18.6%," and "Firing temperature: 220℃."

[0030] It should be noted that the integrity verification involves performing a hash check on the memory data object: a hash value is calculated on the original data during encapsulation at the edge node and uploaded along with the data packet. After decryption at the cloud, the hash value of the memory data object is recalculated and compared with the hash value uploaded with the packet. If they match, it is determined that the memory data object has not been tampered with or damaged during transmission, and it is identified as structured initial tea feature data. If they do not match, the data packet is discarded and the edge node is requested to retransmit. For example, the hash check can use a message digest algorithm with a hash value length of 128 bits. The structured initial tea feature data obtained after successful verification serves as the input for step S12.

[0031] In step S12, each key-value pair in the initial tea feature data is traversed, and the key name and numerical text in each key-value pair are sequence-annotated using a preset conditional random field model. The sequence annotation results are matched with a preset tea feature professional dictionary, and the field definitions and numerical standards of each key-value pair are extracted based on the matching confidence, including:

[0032] When the matching confidence is not lower than the preset confidence threshold, the matched dictionary entry is determined as the field definition of the corresponding key-value pair, and the numerical content associated with the key-value pair is determined as the corresponding numerical standard.

[0033] It should be noted that the preset conditional random field model is a pre-trained named entity recognition model used to identify the semantic categories of fields in key-value pair text. The feature template of the preset conditional random field model consists of character-level n-gram features, part-of-speech tagging features, and domain dictionary matching features from the tea feature professional dictionary library. The training data is a sample set of tea feature description texts pre-labeled with field semantic categories. During the training phase, the labeled sequences are used as labels, and the observation sequences extracted from the above feature templates are used as input. The weights of the feature transition matrix and state features are estimated by maximizing the conditional log-likelihood. Training terminates when the log-likelihood increment between two adjacent iterations is less than a preset convergence value or reaches a preset maximum number of iterations. For example, the preset convergence value is 1 × 10⁻⁻⁴. 4 The maximum number of iterations is preset to 200. During the usage phase, observation sequences are extracted from the key names and numerical text of each key-value pair according to the feature template. The optimal labeled sequence is calculated using the trained feature transition matrix and label sequence scoring function. For example, character-level n-grams are combined with binary and ternary pairs, with a preset maximum number of iterations of 200. The semantic categories of the fields include six categories of labels: tea type, physicochemical indicators, processing parameters, sensory evaluation terms, planting management parameters, and numerical information. Each label is labeled in the training sample set according to the BIO labeling scheme. For example, the tea feature description text sample set contains no less than 3000 manually independently labeled key-value pair text records, covering the six semantic categories of the fields. The distribution ratio of each category is consistent with the distribution ratio of the field categories in the actual reported data of the edge nodes. The training sample set is divided into training set, validation set, and test set in a 7:1.5:1.5 ratio. During the training phase, an F1 score of at least 0.85 for the sequence labels on the validation set is used as a condition for model training success. In CRF training, 1×10⁻ 4 It is the conventional convergence threshold of the log-likelihood increment, which, together with the maximum of 200 iterations, forms a complementary termination condition. Those skilled in the art can fully reproduce the training process once this value is known.

[0034] The entries in the tea characteristic professional dictionary database are organized into five categories: tea category names, including standard names and alternative names of tea varieties; tea physicochemical index names, including standard names of physicochemical components such as tea polyphenols, amino acids, and caffeine; key processing parameters names, including standard names of processing parameters such as fixation temperature, rolling time, and fermentation temperature; sensory evaluation terminology names, including terms based on tea sensory evaluation standards such as liquor color, aroma, taste, and leaf appearance; and planting management parameter names, including standard names of management parameters such as planting methods, pest control methods, and harvesting methods. Each entry in these five categories is equipped with a corresponding set of synonyms, including synonyms, alternative names, and English abbreviations. The sequence labeling results are matched with the standard names and synonym sets of the five categories of entries, and the edit similarity between the labeled entities and the dictionary entries is used as the matching confidence level.

[0035] Furthermore, the preset confidence threshold is used to control the accuracy of field definition extraction. Its value is determined by a pre-labeled validation sample set: the precision and recall of field extraction are statistically analyzed under different confidence values ​​on the validation sample set, and the confidence value corresponding to the equilibrium point of precision and recall is taken as the preset confidence threshold, exemplarily set to 0.90. When the matching confidence is not lower than the preset confidence threshold, the standard name of the matched dictionary entry is determined as the field definition of the corresponding key-value pair, and the numerical content associated with the key-value pair is determined as the corresponding numerical standard after dimensional conversion. In one implementation, when the matching confidence is lower than the preset confidence threshold, the original key name of the key-value pair is retained and marked as pending verification, and it is not included in subsequent field definitions and numerical standards to avoid low-confidence erroneous mappings polluting the standard field system.

[0036] like Figure 2As shown in the figure, this figure illustrates the distribution of matching confidence scores between key-value pairs in the initial tea feature data, after sequence labeling using a pre-defined conditional random field model, and a professional dictionary of tea features. The comparison clearly shows that the matching confidence scores of each field exhibit a significant stratified distribution within the range of 0.60 to 0.98. Specifically, the matching confidence scores of fields such as brewability, color, roasting temperature, aroma type, rolling time, and tea polyphenol content reach above 0.96 (blue, green, and red bars). The confidence levels for fields such as aftertaste, liquor brightness, leaf tenderness, purity, appearance, amino acid content, variety, prevention and control methods, taste, aroma, and liquor color are concentrated in the range of 0.80 to 0.95 (orange, purple, and brown bars), which is in the medium confidence level range. However, the confidence levels for fields such as storage humidity, water extract, place of origin, province of origin, leaf appearance, fermentation temperature, fixation temperature, and dry tea appearance are below 0.70 (various colored bars), which is in the low confidence level range. After sequence labeling and dictionary matching processing using the pre-defined conditional random field model, the originally inconsistent and varied multi-source heterogeneous fields (such as "dry tea appearance," "shape," and "color") are effectively mapped to standard entries in the tea feature professional dictionary. High-confidence fields (confidence ≥ 0.90) are directly identified as standard field definitions, while low-confidence fields (confidence < 0.90) are marked as pending verification. This automatically normalizes the localized field names of each edge node into unified domain standard terminology. This result intuitively demonstrates that the field definition extraction method based on the conditional random field model and tea feature professional dictionary described in this scheme can effectively eliminate the heterogeneity of field naming and encoding formats among multi-source edge nodes, providing a solid data foundation for subsequent difference comparison and standardization conversion with the metadata specification benchmark library.

[0037] In step S13, the field definition and the numerical standard are compared with the benchmark field definition and benchmark numerical standard in the preset metadata specification benchmark library to obtain a comprehensive difference value. Based on the comparison result of the comprehensive difference value and the preset heterogeneity threshold, the heterogeneity characteristics of the initial feature data of tea are determined, and a standardization transformation is performed to obtain the standard description dataset of tea.

[0038] In one implementation, the field definition and the numerical standard are compared with the benchmark field definition and benchmark numerical standard in a preset metadata specification benchmark library to obtain a comprehensive difference value. The heterogeneity characteristics of the initial characteristic data of tea are determined based on the comparison result of the comprehensive difference value and a preset heterogeneity threshold, including: comparing the field definition with the benchmark field definition to obtain a first difference value; comparing the numerical standard with the benchmark numerical standard to obtain a second difference value; and integrating the first difference value and the second difference value to obtain a comprehensive difference value.

[0039] If the overall difference value is greater than the preset heterogeneity threshold, then the initial feature data of the tea leaves is determined to have heterogeneous characteristics.

[0040] In one implementation, the standardization transformation to obtain a standard description dataset for tea includes:

[0041] When it is determined that the initial tea feature data has heterogeneous characteristics, the initial tea feature data is subjected to structured decomposition processing to obtain intermediate reconstructed data;

[0042] The intermediate reconstructed data is subjected to outlier filtering and standardization to obtain a standard description dataset.

[0043] It should be noted that the preset metadata specification benchmark library is a standard field system pre-constructed based on standards in the tea industry. The benchmark field definitions specify the names and hierarchical relationships of each standard field, while the benchmark numerical standards specify the dimensions, value ranges, and distribution characteristics of each standard field's values. The first difference value characterizes the deviation of the field definition from the benchmark field definition in terms of structural hierarchy, and the second difference value characterizes the deviation of the numerical standard from the benchmark numerical standard in terms of value distribution.

[0044] In this embodiment, the metadata specification benchmark library is constructed based on national standards such as GB / T 14487-2017 "Tea Sensory Evaluation Terminology", GB / T 8302-2013 "Tea Sampling", and GB / T 23776-2018 "Tea Sensory Evaluation Methods". The benchmark field definitions cover standard fields such as tea category, province of origin, city of origin, physicochemical index name, processing parameter name, and sensory evaluation terminology. The benchmark numerical standards specify the dimensions, value ranges, and distribution characteristics of the above standard fields. The distribution characteristics include the mean, standard deviation, and upper and lower quartile ranges of each standard field value. For the P and Q distributions of the relative entropy formula in the calculation of the second difference value, those skilled in the art can extract specific distribution parameters from the benchmark library based on this.

[0045] The comparison of the first difference value exemplarily employs graph edit distance: the structural hierarchy of the field definitions and the structural hierarchy of the baseline field definitions are represented as labeled attribute graphs, where nodes represent fields and edges represent the dependencies between fields. The sum of the minimum costs of node insertion, node deletion, node replacement, and edge insertion and deletion required to transform the attribute graph defined by the field definitions into the attribute graph defined by the baseline field definitions is calculated as the first difference value. The cost of each type of editing operation is pre-set according to the hierarchical weight of the field in the system; the higher the level of the field, the greater its editing cost. The formula for calculating the first difference value is as follows:

[0046]

[0047] in, The first difference value is dimensionless and its range is greater than or equal to 0. In this embodiment, the example value is 800. The attribute graph defined for the field. The attribute graph defined for the baseline field; To be Transform into The set of all editing operation sequences; For the first Each editing operation is derived from node insertion, node deletion, node replacement, and simultaneous insertion and deletion. The length of the edit operation sequence, which is a positive integer; For editing operations The cost is pre-set according to the field's hierarchical weight in the system; the higher the level of the field, the greater its editing cost. For example, the node replacement cost is 1, and the edge editing cost is 0.5.

[0048] Furthermore, the comparison of the second difference value exemplarily employs relative entropy: The distributions of the numerical standard and the baseline numerical standard are statistically analyzed separately, and the relative entropy of the numerical standard's distribution relative to the baseline numerical standard's distribution is calculated as the second difference value. The formula for calculating relative entropy is as follows:

[0049]

[0050] in, The second difference value is the relative entropy of the distribution of numerical standard values ​​relative to the distribution of benchmark numerical standard values. , dimensionless, with a value range greater than or equal to 0, and an example value of 0.12 in this embodiment; The value distribution of the numerical standard is composed of the frequency of the numerical standard in each value interval, and is dimensionless. The value distribution of the benchmark numerical standard is composed of the frequencies of the benchmark numerical standard in the same value interval, and is dimensionless. The value is the number of intervals to be divided, which is a positive integer. In this example, the value is 10. The index of the range of values; For the numerical standard to fall into the first The frequency of each value interval The benchmark value falls into the first The frequency of each value interval. The larger the second difference value, the greater the deviation of the numerical standard's value distribution from the benchmark numerical standard.

[0051] It should be noted that the integration is a linear weighted sum of the first difference value and the second difference value according to preset weighting coefficients. The formula for calculating the comprehensive difference value is as follows:

[0052]

[0053] in, The value is a dimensionless value for the overall difference, and its range is greater than or equal to 0. In this embodiment, the example value is 0.392. The first difference value after normalization is dimensionless and its value ranges from 0 to 0. The second difference value after normalization is dimensionless and its value ranges from 0 to 0. The weighting coefficients for the first difference value after normalization. These are the weighting coefficients for the normalized second difference value; both are dimensionless and range from 0 to 1. The weighting coefficients are determined based on the degree of influence of field structure deviation and numerical distribution deviation on subsequent standardization processing. For example, numerical distribution deviation has a greater impact on quality correlation analysis, therefore, a weighting coefficient of 1 is used. .

[0054] To ensure that the first and second difference values ​​have comparable numerical scales, before calculating the composite difference value, min-max normalization is performed using the minimum and maximum values ​​of the corresponding difference values ​​in the historical calibration samples, respectively. and When the current difference value is equal to or less than the minimum value, the corresponding normalized value is 0; when the current difference value is equal to or greater than the maximum value, the corresponding normalized value is 1. The weighting coefficients are used to characterize the relative impact of the normalized field structure bias and numerical distribution bias on subsequent standardization processing, and satisfy w1 + w2 = 1. In this embodiment... and The values ​​are 0.80 and 0.12 respectively, therefore the overall difference value D is 0.392; the preset heterogeneity threshold is also determined by historical calibration samples under the same normalization scale, and in this embodiment it is taken as 0.35.

[0055] It should be noted that the preset heterogeneity threshold is used to define the level to which the comprehensive difference value is considered to indicate heterogeneity requiring standardization and cleaning. Its value is determined by historical multi-source data: a comprehensive difference value is calculated for each of the pre-collected historical tea feature data (which has been manually determined to be heterogeneous). The midpoint between the lower quantile of the comprehensive difference value distribution for heterogeneous samples and the upper quantile of the comprehensive difference value distribution for non-heterogeneous samples is taken as the preset heterogeneity threshold, exemplarily set to 0.35. If the comprehensive difference value is greater than the preset heterogeneity threshold, the initial tea feature data is determined to have heterogeneous characteristics and enters the structured decomposition process. In one implementation, if the comprehensive difference value is not greater than the preset heterogeneity threshold, the initial tea feature data is determined not to have heterogeneous characteristics. This initial tea feature data already conforms to the metadata specification benchmark and does not require structured decomposition and format conversion, serving as a component of the standard description dataset.

[0056] It should be noted that the structured decomposition process involves extracting the data type and encoding rules of the initial tea feature data, matching the corresponding structure parsing rules in a preset structure parsing rule library based on the data type and encoding rules, and decomposing nested or composite fields in the initial tea feature data into atomic fields according to the structure parsing rules to obtain intermediate reconstructed data. For example, "Origin: Fujian-Wuyishan" is decomposed into two atomic fields, "Province of Origin: Fujian" and "City of Origin: Wuyishan," according to the structure parsing rules.

[0057] It should be noted that the outlier filtering is to remove atomic field values ​​in the intermediate reconstructed data that exceed the corresponding benchmark value range; the standardization process includes timestamp alignment, unit unification and data type standardization, that is, aligning the collection timestamps of each atomic field to a unified time benchmark, converting the numerical units to the units specified by the benchmark numerical standard, and converting the data types to the types specified by the benchmark field definition, so as to obtain a standard description dataset that conforms to the metadata specification benchmark, which serves as the input for step S14.

[0058] In step S14, the text description information in the standard description dataset is extracted, the semantic offset is calculated and noisy records are removed to obtain the tea's origin identification information and processing technology information, and cross-regional quality correlation analysis is performed. The results of the quality correlation analysis are mapped to a preset evaluation rule base to generate the analysis results.

[0059] In one implementation, the step of obtaining the tea's origin identification information and processing technology information, performing cross-regional quality correlation analysis, and generating analysis results by mapping the quality correlation analysis results to a preset evaluation rule base includes:

[0060] Extract origin identification information and processing technology information from the standard descriptive data after noise removal;

[0061] Cluster analysis is performed based on the place of origin identification information and the processing technology information to construct a cross-regional data set;

[0062] The cross-regional data set is subjected to feature fusion and classification comparison to determine the quality correlation analysis results within the cross-regional data set and map them to a preset evaluation rule base to generate analysis results.

[0063] In one implementation, when the anomaly confidence level is greater than a preset judgment threshold, the corresponding data record is judged as a noise record and removed. It also includes retaining the corresponding data record and proceeding to the next analysis step when the anomaly confidence level is not greater than the preset judgment threshold.

[0064] It should be noted that the text description information refers to the field values ​​expressed in natural language text from the standard description dataset, such as the values ​​of the sensory evaluation terminology field. The vectorization mapping involves mapping the text description information into a fixed-dimensional real-number vector: after segmenting the text description information into words, a pre-trained word vector model is used to map each word into a word vector, and then a weighted average of the word vectors is taken to obtain the text feature vector. For example, the text feature vector has a dimension of 128. In this embodiment, the word vector model is a Word2Vec model based on the Skip-gram architecture, and the training corpus comes from standard terminology texts and academic literature in the tea industry, with a corpus size of no less than 500,000 words.

[0065] It should be noted that the preset standard feature vector is a reference vector calculated and pre-stored using the same vectorization mapping method for the standard semantic sample text. The semantic similarity value is exemplarily based on cosine similarity, and its calculation formula is as follows: Wherein, the standard semantic sample text is a representative descriptive statement selected according to the standard terms of various sensory evaluation dimensions specified in GB / T 14487-2017 "Tea Sensory Evaluation Terminology", covering four sensory dimensions: liquor color, aroma, taste, and infused leaf, with no less than 20 standard descriptive statements selected for each dimension.

[0066]

[0067] in, This is the semantic similarity value, which is dimensionless and ranges from -1 to 1. In this embodiment, the example value is 0.82. The first feature vector of the text One portion, The i-th component of the predefined standard feature vector is dimensionless. is the dimension of the feature vector, which is a positive integer; in this example, the value is 128. This represents the component index. The closer the semantic similarity value is to 1, the closer the text description information is to the standard semantics.

[0068] Furthermore, the semantic offset of the text description information is determined based on the difference between the semantic similarity value and a preset similarity threshold. The formula for calculating the semantic offset is as follows:

[0069]

[0070] in, This is a semantic offset, dimensionless, with a value ranging from -1 to 2. In this embodiment, the example value is 0.03. The preset similarity threshold is dimensionless and ranges from 0 to 1. Its value is determined by the standard semantic sample set and the known noise sample set. The known noise sample set consists of texts whose semantics deviate significantly from the terminology of the tea sensory evaluation standard. This includes standard terminology descriptions containing typos or pinyin replacements, mismatched descriptions between different sensory dimensions, and irrelevant text descriptions from non-tea fields. The intersection of the semantic similarity values ​​of the two is taken. In this embodiment, the example value is 0.85. This represents the semantic similarity value. The larger the semantic offset, the further the text description deviates from the standard semantics.

[0071] It should be noted that the distribution statistical features are statistics extracted from the set of semantic offsets of various textual description information within the same data record, including the distribution density and extreme values ​​of the semantic offsets. The anomaly classification calculation exemplarily uses a random forest: the distribution statistical features are used as input features and fed into a pre-trained random forest model. The random forest model consists of multiple decision trees. The training data consists of historical data records pre-labeled as noise records and their distribution statistical features. During the training phase, for each decision tree, splitting features and splitting points are selected based on the node splitting gain according to the bootstrap sampling samples until the node sample purity reaches a preset condition or the preset tree depth is reached. During the usage phase, the judgment results of each decision tree on the current data record are summarized according to the voting ratio to obtain the forest voting value, and the forest voting value is used as the anomaly confidence of the corresponding data record. For example, the number of decision trees in the random forest is 100, and the preset tree depth is 10 layers. The node splitting gain exemplarily uses the Gini index decrease, and the feature and splitting point that maximizes the Gini index decrease are taken as the splitting basis. The labeling criteria for noise records are as follows: if the semantic offset of any text description field in a data record exceeds the preset upper limit threshold for that field type, or if any key field in the origin identification information or processing technology information of the data record is missing, the data record is labeled as a noise record. For example, at least 2000 historical data records are used to train the random forest model, where the ratio of noise records to normal records is consistent with the ratio of noise records appearing in the actual operation of edge nodes, and they are divided into training set, validation set, and test set in a 7:1.5:1.5 ratio. During the training phase, a classification accuracy of at least 0.90 on the validation set is used as the model training achievement condition.

[0072] It should be noted that the preset judgment threshold is used to define at what level of anomaly confidence is a corresponding data record considered a noise record. Its value is determined by the labeled noise sample set, and the anomaly confidence that minimizes the sum of the misclassification rates of noise samples and normal samples is taken as the preset judgment threshold; for example, it is set to 0.70. When the anomaly confidence is greater than the preset judgment threshold, the corresponding data record is judged as a noise record and removed from the standard descriptive dataset. Missing value imputation and smoothing are then performed on the remaining valid data after removing the noise records to obtain the standard descriptive data after noise removal. In one implementation, when the anomaly confidence is not greater than the preset judgment threshold, the corresponding data record is retained and proceeds to the next analysis step, i.e., it is used as the standard descriptive data after noise removal in subsequent cross-regional quality correlation analysis.

[0073] It should be noted that the origin identification information is obtained by splicing the geographical coordinate information in the standard description data after noise removal with the tea production area code, and is used to uniquely identify the origin of the tea; the processing technology information is the value of process parameters such as fixation temperature, rolling time, and fermentation temperature extracted after parsing and associating the processing technology parameter records based on the origin identification information.

[0074] It should be noted that the cluster analysis involves grouping data records according to features composed of origin identification information and processing technology information, so that records with similar origin-process characteristics are grouped together. For example, the cluster analysis can first use principal component analysis to reduce the dimensionality of the multidimensional correlation matrix composed of origin-process characteristics, retaining principal components whose cumulative variance contribution rate reaches a preset proportion, and then use mean clustering algorithm to cluster the dimensionality-reduced features to construct a cross-regional dataset. For example, the number of principal components retained is 10, and the number of cluster centers is 4. The number of cluster centers is determined based on the number of main categories in the tea-producing areas to be analyzed.

[0075] It should be noted that the feature fusion and classification comparison involves extracting and fusing the origin and processing characteristics of each sample in the cross-regional dataset to obtain fused features. Then, the classification criteria for different quality grades are solved in the fused feature space, thereby determining the quality correlation analysis results between different origin-process combinations and tea quality within the cross-regional dataset. For example, feature fusion and classification comparison can use a radial basis function (RBF) to map origin and processing characteristics to a high-dimensional space. The formula for calculating the RBF is as follows:

[0076]

[0077] in, For the sample With sample The kernel function value after mapping to a higher-dimensional space is dimensionless and ranges from 0 to 1. , These are the origin-process fusion feature vectors for the two samples, respectively. It represents the Euclidean distance between two eigenvectors, with dimensions consistent with the eigenvalues; The width parameter of the kernel function is dimensionless and ranges from 0 to 0. Its value is determined based on the median of the Euclidean distance between feature vectors, taking the reciprocal of the square of the median Euclidean distance. In this embodiment, the example value is 0.5. The optimal classification hyperplane is solved in the high-dimensional space. When the gradient rate of change of the edge support vectors in continuous iterations meets the preset convergence condition, the quality decision boundary is obtained. The high-dimensional space features are fused and compared using the quality decision boundary to obtain the quality correlation analysis results within the cross-regional dataset.

[0078] Furthermore, the analysis results are generated by mapping the quality correlation analysis results to a pre-defined evaluation rule base. The pre-defined evaluation rule base is a set of quality evaluation rules pre-constructed based on tea industry standards and physicochemical indicators, specifying the judgment rules and scoring weights for quality grades under different origin-process combinations. The analysis results include tea quality assessment reports for each origin-process combination, serving as the basis for tea quality judgment output by the agricultural product quality traceability system.

[0079] Furthermore, the evaluation rule base is constructed according to the physicochemical indicators, sensory evaluation terminology, and processing technology indicators in tea industry standards. For each origin-process combination, each indicator participating in the evaluation is first normalized to an indicator value of 0 to 1 according to its corresponding benchmark value standard. Then, the scoring weight of each indicator is calculated based on a pre-collected historical tea sample set with manually labeled quality grades. Let the normalized value of the q-th historical sample on the r-th indicator be... If the total number of historical samples is N, then the sample proportion of the r-th indicator is calculated according to the following formula: In this embodiment, the minimum-maximum normalization of each indicator is based on the minimum and maximum values ​​of each indicator in the historical samples.

[0080]

[0081] in, Let q be the percentage of the q-th historical sample on the r-th indicator, and let information entropy be the percentage of the sample on the r-th indicator. Calculate using the following formula:

[0082]

[0083] Calculate the information entropy of the r-th indicator based on the sample proportion. And obtain the scoring weights according to the following formula. The sum of the weights of each score is 1.

[0084]

[0085] Where m represents the number of indicators involved in the scoring. For each normalized indicator value zr of the current origin-process combination, the quality score Q is calculated using the following formula:

[0086]

[0087] The quality score Q is compared with the pre-defined quality grade range for the corresponding tea category in the evaluation rule base to determine the quality grade. The quality grade range is determined by the score distribution of historical labeled sample sets grouped by quality grade, and is relabeled using newly added manually reviewed samples when the rule base is updated. The initial quality grade is based on the percentage-based scoring standard specified in GB / T23776-2018 "Sensory Evaluation Method for Tea," and is determined by independent manual labeling of each historical sample by no fewer than three qualified tea tasters; the consistency of the labeling results is taken as the quality grade.

[0088] In summary, this invention discloses a cloud-edge collaborative method and system for managing tea feature data. This invention obtains structured initial feature data by parsing encrypted initial tea description data uploaded by edge nodes. It then uses a pre-defined conditional random field model and a tea feature-specific dictionary to extract field definitions and numerical standards. Combined with a metadata standard benchmark library, it performs multi-dimensional difference comparisons to obtain comprehensive difference values, achieving heterogeneous characteristic determination and standardization transformation. Noisy records are then removed through text vectorization and semantic offset analysis. Finally, cross-regional clustering and feature fusion comparisons are performed on origin identification information and processing technology information to generate quality correlation analysis results. This achieves automated cleaning and accurate cross-regional quality assessment of multi-source heterogeneous tea feature data in the absence of a unified metadata benchmark.

[0089] The second embodiment of the present invention provides a tea feature data management system based on cloud-edge collaboration, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described above.

[0090] It should be noted that the cloud-edge collaborative tea feature data management system provided in this embodiment of the invention is used to execute all the process steps of the cloud-edge collaborative tea feature data management method in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0091] A third embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described above.

[0092] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A method for managing tea feature data based on cloud-edge collaboration, characterized in that, include: Get the initial description data of tea uploaded by the edge node, parse the initial description data of tea to obtain structured initial feature data of tea; The key-value pairs in the initial feature data of tea are traversed, and the key name and numerical text in each key-value pair are sequence-annotated by a preset conditional random field model. The sequence annotation results are matched with a preset professional dictionary of tea features, and the field definition and numerical standard of each key-value pair are extracted based on the matching confidence. The field definition and the numerical standard are compared with the benchmark field definition and benchmark numerical standard in the preset metadata specification benchmark library to obtain a comprehensive difference value. Based on the comparison result of the comprehensive difference value and the preset heterogeneity threshold, the heterogeneity characteristics of the initial feature data of tea are determined, and a standardization transformation is performed to obtain the standard description dataset of tea. Textual description information is extracted from the standard description dataset, semantic offset is calculated and noisy records are removed to obtain the tea's origin identification information and processing technology information. Cross-regional quality correlation analysis is then performed, and the results are generated by mapping the quality correlation analysis results to a preset evaluation rule base.

2. The tea feature data management method based on cloud-edge collaboration according to claim 1, characterized in that, The process of parsing the initial description data of the tea leaves to obtain structured initial feature data of the tea leaves includes: Decrypt the encrypted byte stream corresponding to the initial description data of the tea leaves to obtain a memory data object; Perform an integrity check on the memory data object, and determine the memory data object that passes the check as the structured initial feature data of tea leaves.

3. The tea feature data management method based on cloud-edge collaboration according to claim 1, characterized in that, The extraction of field definitions and numerical standards for each key-value pair based on matching confidence includes: When the matching confidence is not lower than the preset confidence threshold, the matched dictionary entry is determined as the field definition of the corresponding key-value pair, and the numerical content associated with the key-value pair is determined as the corresponding numerical standard.

4. The tea feature data management method based on cloud-edge collaboration according to claim 1, characterized in that, The step of comparing the defined fields and numerical standards with the benchmark field definitions and benchmark numerical standards in a preset metadata specification benchmark library to obtain a comprehensive difference value, and determining the heterogeneity characteristics of the initial characteristic data of tea based on the comparison result of the comprehensive difference value and a preset heterogeneity threshold, includes: By comparing the field definition with the benchmark field definition, a first difference value is obtained; by comparing the numerical standard with the benchmark numerical standard, a second difference value is obtained. The first difference value and the second difference value are integrated to obtain a comprehensive difference value; If the overall difference value is greater than the preset heterogeneity threshold, then the initial feature data of the tea leaves is determined to have heterogeneous characteristics.

5. The tea feature data management method based on cloud-edge collaboration according to claim 1, characterized in that, The standardization transformation yields a standard descriptive dataset for tea, including: When it is determined that the initial tea feature data has heterogeneous characteristics, the initial tea feature data is subjected to structured decomposition processing to obtain intermediate reconstructed data; The intermediate reconstructed data is subjected to outlier filtering and standardization to obtain a standard description dataset.

6. The tea feature data management method based on cloud-edge collaboration according to claim 1, characterized in that, The step of extracting text description information from the standard description dataset, calculating semantic offsets, and removing noisy records includes: Extract the text description information from the standard description dataset and perform vector mapping to obtain text feature vectors; Calculate the semantic similarity value between the text feature vector and the preset standard feature vector, and determine the semantic offset of the text description information based on the difference between the semantic similarity value and the preset similarity threshold; Anomaly classification is performed based on the distribution statistical characteristics of the semantic offset to obtain the anomaly confidence level of the corresponding data record; When the anomaly confidence level is greater than the preset judgment threshold, the corresponding data record is judged as a noise record and removed.

7. The tea feature data management method based on cloud-edge collaboration according to claim 1, characterized in that, The process involves obtaining the tea's origin identification information and processing technology information, performing cross-regional quality correlation analysis, mapping the quality correlation analysis results to a preset evaluation rule base, and generating analysis results, including: Extract origin identification information and processing technology information from the standard descriptive data after noise removal; Cluster analysis is performed based on the origin identification information and the processing technology information to construct a cross-regional data set; The cross-regional data set is subjected to feature fusion and classification comparison to determine the quality correlation analysis results within the cross-regional data set, and mapped to a preset evaluation rule base to generate analysis results.

8. The tea feature data management method based on cloud-edge collaboration according to claim 6, characterized in that, The step of classifying and removing the corresponding data record as a noise record when the anomaly confidence level is greater than a preset judgment threshold also includes: When the anomaly confidence level is not greater than the preset judgment threshold, the corresponding data record is retained and proceeds to the next analysis step.

9. A cloud-edge collaborative tea feature data management system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the tea feature data management method based on cloud-edge collaboration as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images