Cigarette brand data processing method based on data blood relationship

Abstract

The application discloses a cigarette brand data processing method based on data blood relationship, aiming at the problem that the distribution of cigarette brand data is disordered and the characteristics are difficult to extract, by introducing data blood relationship technology, the space-time distribution and type tracing of cigarette brand data are realized, the data blood relationship between the cigarette brand data as the starting point and all related metadata is established, and the cigarette brand influence degree quantified by data flow frequency is determined accordingly, the prediction model of LSTM architecture is constructed by using the time sequence information of cigarette brand data obtained after blood relationship analysis and the prediction target, thereby solving the problem that it is difficult to compare different cigarette brands at different time sequences in traditional deep learning, by simultaneously inputting the influence degree characteristics of different cigarette brands at different time, the influence degree of different brands is longitudinally predicted and horizontally compared, and thus the influence degree of different cigarette brands can be accurately predicted and sorted by the prediction model.

Description

Technical Field

[0001] This invention relates to the field of cigarette product data processing technology, and in particular to a method for processing cigarette brand data based on data lineage. Background Technology

[0002] In recent years, thanks to abundant data and computing resources, deep learning has been widely applied in many technological fields, such as traffic prediction and image recognition, due to its superior performance. However, the large amount of multimodal data limits data governance, source tracing, and correlation analysis.

[0003] In this field, for example, a cigarette company obtains information on various cigarette brands from multiple sources. The company wants to reallocate resources among these brands, but the data from these multiple sources is often unrelated and disorganized. The technical challenge lies in extracting the relationships between these disorganized data sets. To address this problem, data lineage technology has been proposed.

[0004] Data lineage is a key technology in data governance, identifying the connections between related data during the process of tracing their origins. It's a means of ensuring data fusion, enabling traceability of data fusion processing through lineage analysis. In big data, lineage refers to the data's generation chain, i.e., its source and the operations it underwent. During data processing, from the data source to the final data generation, data quality issues can arise at each stage. For example, if the data source itself is of low quality, and subsequent processing steps lack data quality checks and processing, the data reaching its destination will also be of low quality. Therefore, regarding data lineage, it's crucial to ensure that data quality checks and processing are performed at each stage to guarantee high-quality data in subsequent processing.

[0005] In the cigarette industry, cigarette companies can use data lineage technology to predict the information dissemination rate and scope of cigarette brands (which can also be understood as the data flow frequency; for ease of explanation, this invention refers to "information dissemination rate and scope" collectively as influence). This process typically involves three steps: First, collecting the lineage information of various data in the dataset and processing and writing the lineage information into a database; second, evaluating the data quality and classifying the data lineage, and preprocessing the data; and finally, using the collected time-series data and data from different modalities to train a time-series prediction model to predict the influence of cigarette brands at the next time point.

[0006] However, the main problem with existing deep learning methods for cigarette brand influence is that common deep learning approaches are only applicable to single structured data modalities. Taking cigarette brand ranking as an example, the data types of factors influencing brand influence in reality are complex and diverse, including text data modalities of cigarette brand reviews, video and audio data modalities, and structured data. A single data modality limits the performance of temporal network tasks, while multiple data modalities are difficult to utilize in deep learning methods. Therefore, exploring a deep learning mechanism for predicting brand influence based on data lineage can often uncover more data value and play a crucial role in improving machine task processing performance. Summary of the Invention

[0007] In view of the above, the present invention aims to provide a data processing method for cigarette brand based on data lineage, so as to solve the problem of poor predictive ability of cigarette brand information flow frequency.

[0008] The technical solution adopted in this invention is as follows:

[0009] This invention provides a method for processing cigarette brand data based on data lineage, including:

[0010] Record and dynamically update data resources for cigarette brands;

[0011] Based on the aforementioned data resources, cigarette brand data is tracked to determine the data lineage between the cigarette brand data, which serves as the starting point, and all related metadata objects.

[0012] The quantitative representation of cigarette brand influence is determined by the data lineage, wherein the cigarette brand influence is related to the data flow frequency determined based on the lineage.

[0013] Based on the lineage of cigarette brands with accompanying time-series information and the influence of those brands, a prediction model with an LSTM architecture is constructed.

[0014] The prediction model is used to predict and rank the influence of different cigarette brands.

[0015] In at least one possible implementation, the quantitative representation of the cigarette brand influence is as follows:

[0016]

[0017] Where, φ i，j Indicates the degree of influence, fre i，j This represents the data flow frequency of the i-th cigarette brand at time j.

[0018] In at least one possible implementation, the tracking of cigarette brand data includes: analyzing the data flow, source tracing, and impact of changes on the data resources.

[0019] In at least one possible implementation, the data processing method further includes: constructing a lineage map for obtaining the lineage relationship of cigarette brand data based on the data flow analysis results.

[0020] In at least one possible implementation, the data processing method further includes: combining the lineage map with preset data cleaning standards and data quality requirements, performing cold data archiving and issuing an alarm for cigarette brand data that still exists beyond a preset period and has no subordinate processes.

[0021] In at least one of the possible implementations, the data flow frequency includes the frequency of cigarette brand data citation or update used to characterize the application popularity of the preset data lineage classification.

[0022] In at least one possible implementation, the preset data lineage classification includes: system level, table level, and field level.

[0023] In at least one of the possible implementations, the data resources include: preset upstream and downstream data resource codes for cigarette brands, data item codes, time item codes, and data resource conversion rules.

[0024] In at least one possible implementation, the data processing method further includes:

[0025] After integrating the impact prediction results with the historical sales data of the corresponding brand of cigarettes, the sales prediction results are obtained again using the prediction model.

[0026] Different cigarette brands are ranked comprehensively based on the impact prediction results and sales prediction results.

[0027] In at least one possible implementation, the step of fusing the influence prediction result with the historical sales data of the corresponding brand of cigarettes, and then using the prediction model again to obtain the sales prediction result includes:

[0028] The impact prediction results are horizontally combined with the historical sales data of the corresponding brand of cigarettes to form a new feature;

[0029] The new features are input into the prediction model for processing, and the model outputs the predicted sales data of different cigarette brands at different times.

[0030] The main design concept of this invention is to address the unique problem of the chaotic distribution and difficulty in extracting features from cigarette brand data. By introducing data lineage technology, it realizes the spatiotemporal distribution and type tracing of cigarette brand data, establishes the data lineage relationship between the starting cigarette brand data and all related metadata, and determines the influence of cigarette brands as quantified by data flow frequency. Using the time series information of cigarette brand data obtained after lineage analysis and the prediction target, a prediction model with an LSTM architecture is constructed. This solves the problem of the difficulty in simultaneously comparing different cigarette brands at different time series in traditional deep learning. By simultaneously inputting the influence characteristics of different cigarette brands at different times, the influence of different brands can be predicted both vertically and compared horizontally. Thus, the prediction model can accurately predict and rank the influence of different cigarette brands. Attached Figure Description

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings, wherein:

[0032] Figure 1 A flowchart illustrating a cigarette brand data processing method based on data lineage, provided in an embodiment of the present invention. Detailed Implementation

[0033] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0034] Before elaborating on the specific embodiments of this invention, it is necessary to reiterate that in the cigarette manufacturing industry, there is a lack of strategies for deep learning data feature extraction under a large amount of messy data. Therefore, this invention proposes a deep learning method for cigarette brand data processing based on data lineage. This method mainly consists of two levels: First, deep learning methods are difficult to apply to feature extraction and prediction tasks from the large amount of irregular data in the cigarette industry. In addition, the temporal information in the data related to different cigarette brands is inconsistent, which may interfere with the temporal information of the deep learning model. In order to better sort out the information in the cigarette brand data, the following embodiments design a data processing mechanism based on data lineage, specifically, to perform temporal and task-based source tracing of cigarette brand data. Second, in real-world scenarios, the dissemination of cigarette brand information is affected not only by longitudinal comparisons of time series but also by horizontal comparisons between different cigarette brand data. In order to better handle this bidirectional comparison, the following embodiments propose to use the temporal model LSTM to process temporal data and construct a multi-channel model architecture to perform the aforementioned horizontal comparisons of different cigarette brand data.

[0035] Accordingly, an embodiment of a cigarette brand data processing method based on data lineage is provided, such as... Figure 1 As shown, the following steps may be included:

[0036] Step S1: Record and dynamically update the data resources of cigarette brands; specifically, this may include: recording the preset upstream and downstream data resource codes, data item codes, time item codes, and data resource conversion rules for cigarette brands.

[0037] Step S2: Based on the data resources, track the cigarette brand data and determine the data lineage between the cigarette brand data (as the starting point) and all related metadata objects;

[0038] Specific tracking methods can include analyzing the data flow, source tracing, and impact of changes on the data resources. In some embodiments, cigarette brand data with traceable lineage (such as cigarette videos and images, multimodal advertising data, and review data) can be queried according to data category, data item, and conversion rules, and a service interface can be provided to the data resources. Preferably, in some preferred embodiments, a formalized display of data lineage can also be provided, for example, a lineage graph can be constructed according to the data flow (which can be automatically generated using mature AI technology in the industry), thereby improving the efficiency of backtracking cigarette brand data. Based on this concept, it is also possible to combine the lineage graph with a preset data cleaning standard list and data quality requirements to perform cold data archiving and issue alerts for cigarette brand data that still exists beyond a preset period and has no subordinate processes.

[0039] Step S3: Through the data lineage, determine the quantitative representation of the cigarette brand influence as the prediction target, wherein the cigarette brand influence is related to the data flow frequency determined based on the lineage.

[0040] In practice, the data flow frequency can be specifically reflected in the frequency of referencing or updating cigarette brand data, which can characterize the application popularity of data lineage grading for cigarette brands. The data lineage grading mentioned here can include three preset levels: system level, table level, and field level. The granularity of data lineage analysis can be determined based on actual business needs and application scenarios. For example, taking video data related to cigarette brands as an example, which includes target cigarette brand image data, cigarette brand data feature attributes, and cigarette brand data feature vectors, the data lineage level of the cigarette brand video data can be at least set at the "table level" according to the analysis requirements.

[0041] To better adapt to subsequent data prediction and ranking prediction processing, some preferred embodiments propose that the quantitative representation of the cigarette brand influence is to calculate the cigarette brand influence related to the data flow frequency for different cigarette brands and at different times. The calculation formula is as follows:

[0042]

[0043] Where, fre i，j Let represent the turnover frequency of the i-th cigarette brand at time j, and the denominator is the maximum turnover frequency of the cigarette brand at time j.

[0044] Step S4: Based on the lineage of cigarette brand data with accompanying time-series information and the influence of the cigarette brands, construct a prediction model with an LSTM architecture;

[0045] Step S5: Use the prediction model to predict and rank the influence of different cigarette brands at future times.

[0046] Specifically, to better utilize the cigarette brand data with accompanying time-series information processed by the aforementioned data lineage technology, this invention employs LSTM as the main architecture of the prediction model. Thus, based on the aforementioned data lineage technology, the influence of different cigarette brands at different times can be traced, and corresponding lineage features with time-series information can be extracted. These features are then input into the LSTM model to predict the influence of different cigarette brands in subsequent times related to data flow frequency. The quantified results of these predicted influences are then ranked. In practice, by tracing the lineage relationships of cigarette brand data, the value and quality of the cigarette brand data can be evaluated. The higher-quality cigarette brand data is then structured and used to train the prediction model, thereby ensuring that the deep learning model meets the main task requirements.

[0047] Beyond the primary task requirements, those skilled in the art will understand that LSTM-based prediction models are sufficient to predict the influence of different cigarette brands with time-series characteristics, and that further utilization of this influence φ... i，j Performing more diverse ranking tasks allows for the correlation between impact prediction results and historical sales data of the corresponding cigarette brands. Horizontal splicing forms new features The data is then fed back into the LSTM model, enabling it to simultaneously predict sales data for different cigarette brands at different times. Based on the predicted impact and sales data, a comprehensive ranking is then performed. The specific ranking process is shown below:

[0048]

[0049] in This represents the predicted impact of the i-th cigarette brand in the j-th month. This represents the predicted sales data for the i-th cigarette brand in the j-th month.

[0050] Based on the above embodiments, a complete technical implementation outline is summarized here: Dynamically collect external data such as video, image, text, and voice data of cigarette brands; trace the lineage relationships between data and establish a lineage hierarchy; then evaluate the data quality and value; summarize the evaluated cigarette brand data; perform singular spectrum analysis on the summarized cigarette brand data; train an LSTM model and use the model to predict influence (and sales volume); finally, rank different cigarette brands based on the predicted quantitative results. Based on this complete cigarette brand data processing flow, the following introduction is provided for practical implementation reference:

[0051] (1) Singularity Spectrum Analysis of Cigarette Brand Data

[0052] Singular spectrum analysis is a method for processing nonlinear time series data. It extracts different component sequences (long-term trend, seasonal trend, noise, etc.) from the trajectory analysis of the time series under study by performing operations such as decomposition and reconstruction, thereby analyzing or denoising the time series.

[0053] The embedded element involved is that the object of singular spectral analysis is a finite-length one-dimensional time series [φ]. i，1 φ i，2 , ..., φ i，N [], where N is the sequence length. First, a suitable window length L needs to be selected, and the original time series is then rearranged to obtain the trajectory matrix:

[0054]

[0055] Typically, L < N / 2. Let K = N - L + 1, then the trajectory matrix X is an L × K matrix.

[0056]

[0057] Decomposition stage: In this stage, singular value decomposition is performed on the trajectory matrix, that is, X is decomposed into the following form:

[0058] X=U∑V T

[0059] Where U is the left matrix; ∑ has values ​​only on the main diagonal, i.e., singular values, and all other elements are zero; V is the right matrix. Furthermore, U and V are both identity orthogonal matrices, satisfying UU T =I,VV T =I.

[0060] Since direct trajectory decomposition is difficult, we first calculate the covariance matrix of the trajectory matrix:

[0061] S = XX T

[0062] Next, we perform eigenvalue decomposition on S to obtain eigenvalues ​​λ1>λ2>…>λ L ≥0 and the corresponding eigenvectors U1, U2, ..., U L . At this time U=[U1, U2,…,U L ], This is the singular spectrum of the original sequence. And it has the following characteristics:

[0063]

[0064] Where λ i The corresponding eigenvector U i It reflects the evolution of the time series.

[0065] Grouping: Divide all L components into c disjoint groups, representing different trend components. The components of each group are: [X l1 X l2 , ...X lc The time series X can then be represented as:

[0066] X = X l1 +…+X lc

[0067] in

[0068] (2) LSTM Model Construction

[0069] Compared to the hidden layers in a traditional RNN, LSTM adds a cell state. At time t, LSTM has three inputs: cell state C. t-1 Hidden state h t-1 The input vector X at time t t The output has two parts: cell state C. t and hidden state h t In addition, h t It is also used as the output at time t.

[0070] Furthermore, LSTM consists of a forget gate layer, an update gate layer, and an output gate layer. The formula for the forget gate layer can be expressed as:

[0071] f t =σ(W f ·[h t-1 x t ]+b f )

[0072] Where [] denotes the horizontal concatenation of two vectors, f t For network output, W f For input calculation parameters, b f To correct the parameters, σ is the activation function.

[0073] Updating the gate layer consists of two parts: This can be seen as information brought by new input; i t This indicates which parts of the information are stored. The calculation process is as follows:

[0074]

[0075] Among them W C Let denot be the parameter matrix, tanh be the activation function, and b be the parameter matrix. C To correct the parameters.

[0076] i t The calculation process is as follows:

[0077] i t =σ(W i ·[h t-1 x t ]+b i )

[0078] Among them W i Indicate i t The parameter matrix, b i This indicates the parameter to be corrected.

[0079] Therefore, the overall output of the updated gate can be expressed as:

[0080]

[0081] The output gate layer represents the output of the LSTM, at which point the cell state C... t It has been updated, when C t After scaling with tanh, and compared with o t Multiplication represents the output of a stage. Where o t and output h t The calculation formula is as follows:

[0082] o t =σ(W o [h t-1 x t ]+b o )

[0083] h t =o t *tanh(C t )

[0084] In summary, the main design concept of this invention is to address the unique problem of the chaotic distribution and difficulty in extracting features from cigarette brand data. By introducing data lineage technology, it realizes the spatiotemporal distribution and type tracing of cigarette brand data, establishes the data lineage relationship between the starting cigarette brand data and all related metadata, and determines the influence of cigarette brands as quantified by data flow frequency. Using the time series information of cigarette brand data obtained after lineage analysis and the prediction target, a prediction model with an LSTM architecture is constructed. This solves the problem of the difficulty in simultaneously comparing different cigarette brands at different time series in traditional deep learning. By simultaneously inputting the influence characteristics of different cigarette brands at different times, the influence of different brands can be predicted both vertically and compared horizontally. Thus, the prediction model can accurately predict and rank the influence of different cigarette brands.

[0085] In this embodiment of the invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0086] The above description of the structure, features, and effects of the present invention is based on the embodiments shown in the figures. However, the above are only preferred embodiments of the present invention. It should be noted that the technical features involved in the above embodiments and their preferred methods can be reasonably combined and matched by those skilled in the art to form a variety of equivalent solutions without departing from or changing the design concept and technical effects of the present invention. Therefore, the present invention is not limited to the scope of implementation shown in the figures. Any changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments, that do not exceed the spirit covered by the specification and figures, should be within the protection scope of the present invention.

Claims

1. A method for processing cigarette brand data based on data lineage, characterized in that, include: Record and dynamically update data resources for cigarette brands; Based on the aforementioned data resources, cigarette brand data is tracked to determine the data lineage between the cigarette brand data, which serves as the starting point, and all related metadata objects. The quantitative representation of cigarette brand influence is determined through the aforementioned data lineage, wherein the cigarette brand influence is related to the data flow frequency determined based on lineage. The quantitative representation of cigarette brand influence is as follows: ； in, Indicates the degree of influence. Indicates the first The cigarette brand in the first Data flow frequency at any given moment; Based on the lineage of cigarette brands with accompanying time-series information and the influence of those brands, a prediction model with an LSTM architecture is constructed. The prediction model is used to predict and rank the influence of different cigarette brands.

2. The cigarette brand data processing method based on data lineage according to claim 1, characterized in that, The tracking of cigarette brand data includes: analyzing the data flow, source tracing, and impact of changes on the data resources.

3. The cigarette brand data processing method based on data lineage according to claim 1, characterized in that, The data processing method further includes: constructing a lineage map based on the data flow analysis results to obtain the lineage relationship of cigarette brand data.

4. The cigarette brand data processing method based on data lineage according to claim 3, characterized in that, The data processing method further includes: combining the pedigree chart with preset data cleaning standards and data quality requirements, cold data archiving and alarming for cigarette brand data that still exists beyond a preset period and has no lower-level processes.

5. The cigarette brand data processing method based on data lineage according to claim 1, characterized in that, The data flow frequency includes the frequency of cigarette brand data citation or update, which is used to characterize the application popularity of the preset data lineage classification.

6. The cigarette brand data processing method based on data lineage according to claim 5, characterized in that, The preset data lineage classification includes: system level, table level, and field level.

7. The cigarette brand data processing method based on data lineage according to claim 1, characterized in that, The data resources include: preset upstream and downstream data resource codes for cigarette brands, data item codes, time item codes, and data resource conversion rules.

8. The cigarette brand data processing method based on data lineage according to any one of claims 1 to 7, characterized in that, The data processing method further includes: After integrating the impact prediction results with the historical sales data of the corresponding brand of cigarettes, the sales prediction results are obtained again using the prediction model. Different cigarette brands are ranked comprehensively based on the impact prediction results and sales prediction results.

9. The cigarette brand data processing method based on data lineage according to claim 8, characterized in that, The step of integrating the impact prediction results with the historical sales data of the corresponding brand of cigarettes, and then using the prediction model again to obtain the sales prediction results includes: The impact prediction results are horizontally combined with the historical sales data of the corresponding brand of cigarettes to form a new feature; The new features are input into the prediction model for processing, and the predicted sales data of different cigarette brands at different times are output.

Images