Wide table data model replacement method based on table structure similarity and related device

Abstract

The invention discloses a wide table data model replacement method based on table structure similarity and a related device. When the method provided by the embodiment of the invention is executed, firstly, table structure information of a stock data model to be replaced is extracted, and detailed description information is generated through text splicing; and meanwhile, performing text splicing on table structure information of each replaceable wide table data model to form respective description information. Performing word segmentation processing on the description information to generate a word segmentation sequence; a TF-IDF cosine similarity algorithm is utilized to calculate the similarity between the stock data model to be replaced and each wide table data model, and the similarity values have corresponding relations with the wide table data models; and finally, selecting a wide table data model corresponding to the similarity meeting the condition according to the screening condition, and taking the wide table data model as a reference and guidance for subsequent data model replacement. According to the method, the most suitable wide table can be automatically identified for replacement, and the overall data processing efficiency and accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of computer data processing, and in particular to a method for replacing a wide table data model based on table structure similarity and a related device. Background Art

[0002] The field of big data is constantly developing and evolving. For data warehouses, data model design is increasingly being updated, and wide tables are gradually replacing traditional bazaar stock model tables. Wide tables are a standardized, business-oriented, spliced, and tied organization of open source basic data, using a combination of dimensional modeling and three-paradigm modeling. Building wide tables can not only meet the requirements of model versatility, scalability, and stability, but also improve business readability and shorten iteration cycles. Therefore, it is very meaningful to design excellent wide tables to replace the original bazaar model.

[0003] After the construction of a large number of wide tables is completed, it is particularly important to allow the downstream to quickly replace the original data mart model and use the new wide table. The design specifications of the wide table are different from the stock model, and the table name and field name are more standardized. The table structure and table name of the old and new models are different. The field name and quantity may also be different. The data model is classified according to business themes (such as loans, deposits, etc.), and then the wide table with the same business theme as the data model is replaced by the data model. In most cases, the classification and replacement of the model still rely on manual work one by one, with a very low degree of automation, resulting in low analysis efficiency and time-consuming and labor-intensive.

[0004] Therefore, how to automatically replace the original data mart model with a large number of wide tables to improve analysis efficiency and reduce manual work is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] Based on the above problems, the present application provides a wide table data model replacement method based on table structure similarity and related devices, which can automatically replace the original data mart model with a large number of wide tables to improve analysis efficiency and reduce manual work.

[0006] The embodiments of the present application disclose the following technical solutions:

[0007] A method for replacing a wide table data model based on table structure similarity, the method comprising:

[0008] Determine the existing data model to be replaced and multiple replaceable wide table data models;

[0009] Extracting multiple table structure information of the stock data model to be replaced, and performing text splicing to obtain stock description information of the stock data model to be replaced; extracting multiple table structure information of each replaceable wide table data model, and performing text splicing to obtain wide table description information of each replaceable wide table data model;

[0010] Segmenting the existing description information to obtain an existing segmentation sequence, and segmenting the wide table description information of each replaceable wide table data model to obtain a plurality of wide table segmentation sequences;

[0011] Based on the term frequency TF-inverse document frequency IDF cosine similarity algorithm, the similarity between the to-be-replaced stock data model and each of the replaceable wide table data models is calculated using the stock word segmentation sequence and the multiple wide table word segmentation sequences; the similarity has a corresponding relationship with the replaceable wide table data model;

[0012] The replaceable wide table data model corresponding to the similarity that meets the screening condition is determined as the target replacement wide table data model of the stock data model to be replaced.

[0013] In a possible implementation, the calculating the similarity between the to-be-replaced stock data model and each of the replaceable wide table data models by using the stock word segmentation sequence and the multiple wide table word segmentation sequences based on the TF-IDF cosine similarity algorithm includes:

[0014] Calculating the TF-IDF value of each word in the existing word segmentation sequence, and calculating the TF-IDF value of each word in the multiple wide table word segmentation sequences;

[0015] Using the TF-IDF value of each word in the stock word segmentation sequence, the stock word segmentation sequence is vectorized to obtain a stock word segmentation vector, and using the TF-IDF value of each word in the multiple wide-table word segmentation sequences, the multiple wide-table word segmentation sequences are respectively vectorized to obtain multiple wide-table word segmentation vectors;

[0016] The cosine similarity formula is used to calculate the similarity between the to-be-replaced existing data model and each of the replaceable wide table data models based on the existing word segmentation vectors and the multiple wide table word segmentation vectors.

[0017] In a possible implementation, calculating the TF-IDF value of each word in the existing word sequence includes:

[0018] Calculate the TF value of each word in the existing word segmentation sequence according to the TF value calculation formula, and calculate the IDF value of each word in the existing word segmentation sequence according to the IDF value calculation formula;

[0019] The TF value and the IDF value of each word in the existing word segmentation sequence are multiplied to obtain the TF-IDF value of each word in the existing word segmentation sequence.

[0020] Wherein, the TF value calculation formula is:

[0021]

[0022] The IDF value calculation formula is:

[0023]

[0024] In a possible implementation, determining the replaceable wide table data model corresponding to the similarity that meets the screening condition as the target replacement wide table data model of the to-be-replaced stock data model includes:

[0025] Sorting the corresponding replaceable wide table data models according to the magnitude of the similarity, and determining the replaceable wide table data models corresponding to the similarity that rank and meet the screening condition as target replacement wide table data models for the stock data model to be replaced;

[0026] The screening condition is a rule for filtering and selecting replaceable wide table data models whose similarity rankings meet specific standards.

[0027] In a possible implementation, the screening condition includes:

[0028] Determine the replaceable wide table data model corresponding to the similarity greater than the similarity threshold as the target replacement wide table data model of the stock data model to be replaced; or,

[0029] The replaceable wide table data models corresponding to the similarities ranked in the top N are determined as target replacement wide table data models for the stock data model to be replaced.

[0030] A wide table data model replacement device based on table structure similarity, the device comprising:

[0031] A first determining unit, configured to determine a stock data model to be replaced and a plurality of replaceable wide table data models;

[0032] A first extraction and splicing unit is used to extract multiple table structure information of the stock data model to be replaced, and perform text splicing to obtain stock description information of the stock data model to be replaced;

[0033] A second extraction and splicing unit is used to extract multiple table structure information of each replaceable wide table data model, and perform text splicing to obtain wide table description information of each replaceable wide table data model;

[0034] A word segmentation unit, configured to segment the existing description information to obtain an existing word segmentation sequence, and segment the wide table description information of each replaceable wide table data model to obtain a plurality of wide table word segmentation sequences;

[0035] A first calculation unit is used to calculate the similarity between the to-be-replaced stock data model and each of the replaceable wide table data models by using the stock word segmentation sequence and the multiple wide table word segmentation sequences based on a TF-IDF cosine similarity algorithm; the similarity has a corresponding relationship with the replaceable wide table data model;

[0036] The screening unit is used to determine the replaceable wide table data model corresponding to the similarity that meets the screening condition as the target replacement wide table data model of the stock data model to be replaced.

[0037] In a possible implementation manner, the first computing unit specifically includes:

[0038] A second calculation unit is used to calculate the TF-IDF value of each word in the existing word sequence;

[0039] A third calculation unit is used to calculate the TF-IDF value of each word in the plurality of wide table word sequences;

[0040] A first vectorization unit, configured to vectorize the existing word segmentation sequence by using the TF-IDF value of each word in the existing word segmentation sequence to obtain an existing word segmentation vector;

[0041] A second vectorization unit is used to vectorize the plurality of wide-table word segmentation sequences respectively by using the TF-IDF value of each word in the plurality of wide-table word segmentation sequences to obtain a plurality of wide-table word segmentation vectors;

[0042] The fourth calculation unit is used to calculate the similarity between the to-be-replaced existing data model and each of the replaceable wide table data models based on the existing word segmentation vectors and the multiple wide table word segmentation vectors by using a cosine similarity formula.

[0043] In a possible implementation manner, the second computing unit specifically includes:

[0044] A fifth calculation unit, used to calculate the TF value of each word in the existing word segmentation sequence according to the TF value calculation formula, and calculate the IDF value of each word in the existing word segmentation sequence according to the IDF value calculation formula;

[0045] The sixth calculation unit is used to multiply the TF value and the IDF value of each word in the existing word segmentation sequence to obtain the TF-IDF value of each word in the existing word segmentation sequence.

[0046] Wherein, the TF value calculation formula is:

[0047]

[0048] The IDF value calculation formula is:

[0049]

[0050] A wide table data model replacement device based on table structure similarity includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the wide table data model replacement method based on table structure similarity as described above is implemented.

[0051] A computer-readable storage medium having instructions stored therein, which, when executed on a terminal device, causes the terminal device to execute the wide table data model replacement method based on table structure similarity as described above

[0052] Compared with the prior art, this application has the following beneficial effects:

[0053] The present application provides a method for replacing a wide table data model based on table structure similarity and a related device. Specifically, when executing the method for replacing a wide table data model based on table structure similarity provided in an embodiment of the present application, first, determine the stock data model to be replaced and multiple replaceable wide table data models. Then, extract a variety of table structure information from the stock data model to be replaced, and generate description information of the stock data model by text splicing. At the same time, extract a variety of table structure information from each replaceable wide table data model, and perform text splicing to generate respective wide table description information.

[0054] Next, these description information are segmented to generate stock segmentation sequences and multiple wide table segmentation sequences respectively. The cosine similarity algorithm combining term frequency (TF) and inverse document frequency (IDF) is used to calculate the similarity between the stock data model to be replaced and each replaceable wide table data model. Each similarity value corresponds to a corresponding replaceable wide table data model. Finally, according to the preset screening conditions, the replaceable wide table data model corresponding to the similarity value that meets the requirements is selected, and it is determined as the target replacement wide table data model for the stock data model to be replaced. The present application realizes automatic similarity calculation of stock data models and replaceable wide table data models by using the term frequency TF-IDF cosine similarity algorithm, which reduces the need for manual intervention and improves replacement efficiency. At the same time, through word segmentation processing and similarity calculation, it can intelligently match the most suitable wide table data model for replacement, thereby improving the accuracy and precision of replacement. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 A method flow chart of a wide table data model replacement method based on table structure similarity provided in an embodiment of the present application;

[0057] Figure 2 A method flow chart of a similarity calculation method provided in an embodiment of the present application;

[0058] Figure 3 A schematic diagram of the structure of a wide table data model replacement device based on table structure similarity provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] To facilitate understanding of the technical solutions provided by the embodiments of the present application, the background technology involved in the embodiments of the present application will be described below.

[0060] As the big data field continues to develop and evolve, the data model design in the data warehouse is also increasingly updated. The traditional market stock model table is gradually replaced by a wide table. A wide table is an organizational method that standardizes, commercializes, splices, and binds open source basic data. It uses a combination of dimensional modeling and three-paradigm modeling. By building a wide table, not only can the model's universality, scalability, and stability requirements be met, but it can also improve business readability and shorten the iteration cycle. Therefore, it is of great significance to design an excellent wide table to replace the original market model.

[0061] However, after the construction of a large number of wide tables is completed, it becomes particularly critical to allow downstream systems to quickly replace the original data mart model and use the new wide table. The wide table is different from the stock model in design specifications, and the table name and field name are more standardized. Due to the different table structures of the new and old models, the table names and field names may also be different. When classifying the data model according to business themes (such as loans, deposits, etc.) and replacing the original model with a wide table with the same business theme, in most cases, it still relies on manual classification and replacement of the model one by one, resulting in an extremely low degree of automation. This not only makes the analysis inefficient, but also consumes a lot of time and manpower.

[0062] In order to solve this problem, a method and related device for replacing a wide table data model based on table structure similarity are provided in an embodiment of the present application. First, a stock data model to be replaced and a plurality of replaceable wide table data models are determined. Subsequently, a plurality of table structure information is extracted from the stock data model to be replaced, and its text is spliced ​​to form the description information of the stock data model. Similarly, similar table structure information extraction and text splicing are performed on each replaceable wide table data model to obtain corresponding wide table description information. Then, a word segmentation operation is performed on the stock description information and each wide table description information, and the word segmentation is converted into a corresponding word segmentation sequence, and a stock word segmentation sequence and a plurality of wide table word segmentation sequences are obtained respectively. Furthermore, by comparing the stock word segmentation sequence with the plurality of wide table word segmentation sequences based on the TF-IDF cosine similarity algorithm, the similarity between the stock data model to be replaced and each replaceable wide table data model is calculated. These similarities correspond to the replaceable wide table data models, and finally the wide table data model corresponding to the similarity that meets the conditions is screened out as the target replacement wide table data model. This application can more accurately identify wide tables suitable for replacement and reduce human errors by calculating similarity based on word frequency TF-IDF and cosine similarity algorithm. The automated matching process can significantly shorten the time for model replacement, thereby completing the wide table replacement task faster and improving overall analysis efficiency.

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0064] See also Figure 1 , which is a method flow chart of a wide table data model replacement method based on table structure similarity provided by an embodiment of the present application, such as Figure 1 As shown, the wide table data model replacement method based on table structure similarity may include steps S101-S105:

[0065] S101: Determine an existing data model to be replaced and multiple replaceable wide table data models.

[0066] Determining the existing data models to be replaced and multiple replaceable wide table data models means identifying the old data models (existing data models) that need to be replaced during the data warehouse transformation process, and selecting multiple new wide table data models as potential alternatives. This process usually involves a comprehensive evaluation and comparison of existing data models and new data models to ensure that the new wide table can better meet business needs, improve data processing efficiency, and enhance system flexibility.

[0067] For example, suppose there is a bank's data warehouse, which contains multiple traditional market stock data models, such as "loan model". These traditional models usually store and query data through much more complex table associations. As the system develops, in order to improve data processing efficiency and enhance business flexibility, it is decided to introduce the concept of wide tables to replace these traditional models.

[0068] Stock data model:

[0069] Loan Model: Contains the lender’s basic information, loan amount, repayment plan, etc.

[0070] Alternative wide table data model:

[0071] Customer Comprehensive Info Wide Table: Integrates the customer's loan information and deposit information and stores it in a wide table for quick query and analysis.

[0072] Funding Activity Wide Table: Focuses on the customer's financing activities, including information related to loans and deposits, to facilitate specific types of business analysis.

[0073] In this example, the "loan model" is determined as the stock data model to be replaced, while the "customer comprehensive information broad table" and "financing activity broad table" are selected as candidate replaceable broad table data models. This replacement can not only simplify the data model structure, but also improve the efficiency of data query and analysis, and better support business decision-making.

[0074] S102: extracting multiple table structure information of the stock data model to be replaced, and performing text splicing to obtain the stock description information of the stock data model to be replaced; extracting multiple table structure information of each replaceable wide table data model, and performing text splicing to obtain the wide table description information of each replaceable wide table data model.

[0075] Extract various table structure information, such as table name, field name, etc., from the existing data model to be replaced, and generate description information of the existing data model through text concatenation. Similarly, extract relevant information from all replaceable wide table data models and generate their respective description information.

[0076] Specifically, we first extract various structural features of the "stock data model to be replaced", that is, the old data model that is about to be updated or replaced, such as Chinese table names, field names and other detailed information, and then combine this information to form a complete "stock description information". This process helps to fully understand the characteristics and status of the existing data model.

[0077] Next is the processing of "replaceable wide table data models". These new models are alternatives to the old models. Similarly, it is also necessary to extract various table structure information from these new models, including but not limited to their Chinese table names, various field names and other elements, and then summarize them into "wide table description information" for each new model through text splicing. This is done to systematically compare the differences between the new and old models, so as to facilitate the evaluation and selection of the most suitable data model for replacement.

[0078] For example, suppose an e-commerce company is using an old data model to store product information. This model contains a table called "Order Table" with fields such as "Order Number", "Customer Identifier (ID)", "Product ID", etc. Now, the company decides to upgrade the data model, and there are three new wide table data models as candidates. The "Order Table" in each new model has different field configurations, such as one model adds a "Shipping Address" field, and another introduces a "Payment Method" field.

[0079] Extract the table structure information of the stock data model to be replaced:

[0080] Chinese table name: "Order Table"; Field name: For the original model, there may be a description information like this: "Order number (unique identifier), customer ID (purchaser ID), product ID (purchased product ID).".

[0081] Concatenate inventory description information: The generated description information example is: "The order table includes the order number, and the customer ID includes the product ID field"

[0082] Extract the table structure information of each replaceable wide table data model:

[0083] Model A: Chinese table name: "Order Table"; field names: "Order Number, Customer ID, Product ID, Shipping Address".

[0084] Model B: Chinese table name: "Order Table"; field names: "Order Number, Customer ID, Product ID, Payment Method".

[0085] Splice wide table description information: Model A description information: "The order table includes order number, customer ID, product ID and delivery address fields";

[0086] Model B describes the information: "The order table includes order number, customer ID, product ID and payment method fields."

[0087] Through the above steps, you can clearly see the specific structural differences of each model, so that you can choose the most appropriate wide table data model for replacement according to actual business needs.

[0088] S103: Segmenting the existing description information to obtain an existing segmentation sequence, and segmenting the wide table description information of each replaceable wide table data model to obtain a plurality of wide table segmentation sequences.

[0089] Segmenting the existing description information means segmenting the description information of the existing data model into words or phrases according to certain rules for subsequent calculation and comparison. Similarly, the description information of the replaceable wide table data model also needs to go through the same process, splitting it into sequences of words or phrases for subsequent similarity calculation.

[0090] For example, it is assumed that the concatenated inventory description information is as follows: the account information table includes the following fields: account ID, user ID, account type, and account balance.

[0091] The description information of the spliced ​​wide table is: The user account comprehensive information table includes the fields: user ID, name, ID number, telephone number, account ID, account type, and account balance.

[0092] By performing word segmentation processing on the inventory description information, the following word segmentation sequence can be obtained:

[0093] "The account information table includes the fields account ID user ID account type account balance".

[0094] By performing word segmentation processing on the wide table description information, the following word segmentation sequence can be obtained:

[0095] "User account comprehensive information table includes fields User ID Name ID Number Phone Number Account ID Account Type Account Balance"

[0096] S104: Based on the TF-IDF cosine similarity algorithm, the similarity between the to-be-replaced existing data model and each of the replaceable wide table data models is calculated using the existing word segmentation sequence and the multiple wide table word segmentation sequences.

[0097] The cosine similarity algorithm combining term frequency (TF) and inverse document frequency (IDF) is used to calculate the similarity between the existing data model to be replaced and each replaceable wide table data model based on the existing word segmentation sequence and multiple wide table word segmentation sequences. Each similarity value corresponds to a replaceable wide table data model.

[0098] See also Figure 2 , Figure 2 A method flow chart of a similarity calculation method provided in an embodiment of the present application. Accordingly, step S104 calculates the similarity between the to-be-replaced stock data model and each of the replaceable wide table data models using the stock word segmentation sequence and the multiple wide table word segmentation sequences based on the term frequency TF-inverse document frequency IDF cosine similarity algorithm, which can be implemented by steps A1-A3:

[0099] A1: Calculate the TF-IDF value of each word in the existing word segmentation sequence, and calculate the TF-IDF value of each word in the multiple wide table word segmentation sequences.

[0100] Calculating the TF-IDF value of each word in the stock word sequence and multiple wide table word sequences is a method for measuring the importance of words in a text. TF refers to the frequency of a word in a text, while IDF represents the inverse of the importance of a word in the entire document collection. By combining these two pieces of information, the TF-IDF value can help determine the importance of a word in a specific text, which in turn affects the result of the overall similarity calculation.

[0101] Specifically, the TF-IDF value of each word in the stock word sequence and multiple wide table word sequences is calculated to quantify the importance of each word to the text description. Words with high TF-IDF values ​​usually appear frequently in specific texts but rarely in the entire document collection, so they are more representative. This calculation process helps to accurately evaluate the contribution of keywords or phrases to the description model, thereby providing a more accurate basis for subsequent similarity calculations.

[0102] By calculating the TF-IDF values ​​of words in existing word segmentation sequences and multiple wide-table word segmentation sequences, we can better understand the content and characteristics of text data, provide a deeper analysis basis for the automatic matching process of model replacement tasks, and ensure that the most suitable wide-table data model for replacement is selected to improve analysis efficiency and accuracy.

[0103] In a possible implementation, calculating the TF-IDF value of each word in the existing word sequence includes:

[0104] Calculate the TF value of each word in the existing word segmentation sequence according to the TF value calculation formula, and calculate the IDF value of each word in the existing word segmentation sequence according to the IDF value calculation formula;

[0105] The TF value and the IDF value of each word in the existing word segmentation sequence are multiplied to obtain the TF-IDF value of each word in the existing word segmentation sequence.

[0106] Wherein, the TF value calculation formula is:

[0107]

[0108] The IDF value calculation formula is:

[0109]

[0110] Example 1: Assume the following word segmentation sequence:

[0111] Stock word sequence: [data, management, data, model]

[0112] Wide table segmentation sequence 1: [data, processing, wide table, model]

[0113] Wide table word segmentation sequence 2: [data, optimization, data, management]

[0114] First, calculate the frequency of each word in its respective sequence:

[0115] TF values ​​of existing word segmentation sequences: data: 2 / 4 = 0.5; management: 1 / 4 = 0.25; model: 1 / 4 = 0.25.

[0116] TF values ​​of wide table word segmentation sequence 1: data: 1 / 4 = 0.25; processing: 1 / 4 = 0.25; wide table: 1 / 4 = 0.25; model: 1 / 4 = 0.25.

[0117] TF values ​​of wide table word segmentation sequence 2: Data: 2 / 4 = 0.5; Optimization: 1 / 4 = 0.25; Management: 1 / 4 = 0.25.

[0118] Then the IDF value of each word is calculated based on the number of documents containing a word in all documents (that is, all word sequences):

[0119] In 3 sequences, "data" appears 3 times; in 3 sequences, "management" appears 2 times; in 2 sequences, "model" appears 2 times; in 1 sequence, "processing" appears 1 time; in 1 sequence, "wide table" appears 1 time; in 1 sequence, "optimization" appears 1 time. The total number of sequences is 3.

[0120] The IDF value of the data: log(3 / 3)=log(1)=0; the IDF value of management: log(3 / 2)≈0.176; the IDF value of the model: log(3 / 2)≈0.176; the IDF value of the processing: log(3 / 1)≈0.477; the IDF value of the wide table: log(3 / 1)≈0.477; the optimized IDF value: log(3 / 1)≈0.477.

[0121] Finally, multiply the TF value and the IDF value to get the TF-IDF value:

[0122] TF-IDF values ​​of existing word segmentation sequences: data: 0.5*0=0; management: 0.25*0.176≈0.044; model: 0.25*0.176≈0.044.

[0123] TF-IDF value of wide table word sequence 1: data: 0.25*0=0; processing: 0.25*0.477≈0.119; wide table: 0.25*0.477≈0.119; model: 0.25*0.176≈0.044.

[0124] TF-IDF values ​​of wide table word sequence 2: Data: 0.5*0=0; Optimization: 0.25*0.477≈0.119; Management: 0.25*0.176≈0.044.

[0125] It should be noted that the calculation method of the TF-IDF value of each word in the wide table word sequence is similar to the calculation method of the TF-IDF value of each word in the existing word sequence, which will not be repeated here.

[0126] A2: Using the TF-IDF value of each word in the existing word segmentation sequence, the existing word segmentation sequence is vectorized to obtain an existing word segmentation vector, and using the TF-IDF value of each word in the multiple wide-table word segmentation sequences, the multiple wide-table word segmentation sequences are respectively vectorized to obtain multiple wide-table word segmentation vectors.

[0127] Using the TF-IDF values ​​of the existing word segmentation sequences and wide-table word segmentation sequences for vectorization can help represent the similarity or relevance between different texts in a high-dimensional space. This vectorization method represents each text as a multidimensional vector, where each dimension corresponds to the TF-IDF value of a word segmentation. By comparing these vectors, the similarity between texts can be evaluated.

[0128] Example 2, taking Example 1 as an example to continue the explanation, according to the TF-IDF value calculated in Example 1, each sequence can be converted into a vector form. For example:

[0129] The vector representation of the existing word segmentation sequence is: [0, 0.044, 0.25, 0.044];

[0130] Vector representation of wide table word sequence 1: [0, 0.119, 0.119, 0.044];

[0131] Vector representation of wide table word sequence 2: [0, 0.119, 0.044].

[0132] A3: Using the cosine similarity formula, based on the existing word segmentation vectors and the multiple wide table word segmentation vectors, calculate the similarity between the to-be-replaced existing data model and each of the replaceable wide table data models.

[0133] The cosine similarity formula can be used to measure the similarity between two vectors. Based on the existing word segmentation vectors and multiple wide table word segmentation vectors, the similarity between the existing data model to be replaced and each replaceable wide table data model can be calculated. This similarity measurement can help understand the correlation and similarity between different data models.

[0134] For example, assume the following vector representation:

[0135] The vector representation of the existing word segmentation sequence: A = [0, 0.044, 0.25, 0.044];

[0136] Vector representation of wide table word sequence 1: B1 = [0, 0.119, 0.119, 0.044];

[0137] The vector representation of the wide table word segmentation sequence 2 is: B2 = [0, 0.119, 0.044, 0].

[0138] The formula for cosine similarity is as follows: Among them, A and B represent two vectors respectively, A·B represents the dot product of the vectors; ||A|| and ||B|| represent the modulus of the vectors respectively.

[0139] Similarity calculation between the existing word segmentation sequence and the wide table word segmentation sequence 1:

[0140] Compute the dot product:

[0141] A·B1=0*0+0.044*0.119+0.25*0.119+0.044*0.044=0+0.005236+0.02975+0.001936=0.036922.

[0142] Calculate the modulus length:

[0143] The modulus length of the existing word segmentation sequence is: ||A||≈0.2576; the modulus length of the wide table word segmentation sequence 1 is: ||B1||≈0.171.

[0144] Compute cosine similarity:

[0145]

[0146] Similarity calculation between the existing word segmentation sequence and the wide table word segmentation sequence 2:

[0147] Compute the dot product:

[0148] A·B1=0.036922.

[0149] Calculate the modulus length:

[0150] The modulus length of the existing word segmentation sequence is: ||A||≈0.2576; the modulus length of the wide table word segmentation sequence 2 is: ||B2||≈0.127.

[0151] Compute cosine similarity:

[0152]

[0153] S105: Determine the replaceable wide table data model corresponding to the similarity that meets the screening condition as the target replacement wide table data model of the stock data model to be replaced.

[0154] According to the preset screening conditions, the wide table data model corresponding to the similarity value that meets the requirements is selected and determined as the target replacement wide table data model for the existing data model to be replaced. This step ensures that the selected replacement model not only has high similarity, but also meets specific business requirements or standards.

[0155] In a possible implementation, determining the replaceable wide table data model corresponding to the similarity that meets the screening condition as the target replacement wide table data model of the to-be-replaced stock data model includes:

[0156] Sorting the corresponding replaceable wide table data models according to the magnitude of the similarity, and determining the replaceable wide table data models corresponding to the similarity that rank and meet the screening condition as target replacement wide table data models for the stock data model to be replaced;

[0157] The screening condition is a rule for filtering and selecting replaceable wide table data models whose similarity rankings meet specific standards.

[0158] After obtaining the similarity between the stock data model to be replaced and each replaceable wide table data model, the wide table segmentation sequence needs to be sorted according to these similarity values. The sorting criterion is the size of the similarity value, that is, the larger the similarity value, the better the corresponding wide table segmentation sequence matches the stock segmentation sequence. In this way, you can intuitively see which wide table segmentation sequence is closest to the stock segmentation sequence. Then, select a specific replacement model based on pre-set filtering conditions. These filtering conditions are used to filter and select those wide table segmentation sequences whose similarity rankings meet specific standards. For example, the filtering condition can be "the similarity must be greater than a certain threshold" or "the top N models with the highest similarity must be selected." By applying these rules, models that meet the requirements can be selected from the sorted candidate models as the final replacement options.

[0159] In this process, it is necessary to ensure that the selected model is not only highly similar to the existing word segmentation sequence, but also meets specific business or technical requirements, so as to ensure that the replaced data model can effectively replace the original existing data model while meeting the performance and functional requirements of the system.

[0160] In a possible implementation, the screening conditions include but are not limited to:

[0161] The replaceable wide table data model corresponding to the similarity greater than the similarity threshold is determined as the target replacement wide table data model of the stock data model to be replaced. This screening condition means that if the similarity of a replaceable wide table data model to the stock data model is greater than the set threshold, the replaceable wide table data model will be determined as the target replacement wide table data model. This means that only data models with similarity higher than a certain threshold will be selected

[0162] Or, the replaceable wide table data model corresponding to the similarity ranked in the top N is determined as the target replacement wide table data model for the stock data model to be replaced. The screening condition refers to selecting the target replacement wide table data model according to the ranking. For example, it can be set to select only the replaceable wide table data models ranked in the top N similarities as the target replacement wide table data model. This method is suitable for situations where it is necessary to select several most relevant models from a group of alternative data models.

[0163] These filter conditions can be adjusted and customized according to specific needs and scenarios. By setting filter conditions reasonably, the most suitable replacement options can be accurately selected to ensure that the selected target replacement wide table data model meets the established standards and requirements. Such a screening process helps improve the efficiency of data analysis and model replacement, and ensures that the selected data model has a high degree of relevance and matching with the original data model.

[0164] Based on the contents of S101-S105, it can be known that the stock data model that needs to be replaced is first identified, and multiple potential replaceable wide table data models are listed as candidates. A variety of table structure information is extracted from the stock data model to be replaced, and text splicing is performed to generate the stock description information of the model. At the same time, a variety of table structure information is also extracted for each replaceable wide table data model, and text splicing is performed to generate respective wide table description information. Then, the generated stock description information is segmented to obtain a stock segmentation sequence. Similarly, the wide table description information of each replaceable wide table data model is segmented to generate multiple wide table segmentation sequences. Using the TF-IDF cosine similarity algorithm, the similarity between the stock data model to be replaced and each replaceable wide table data model is calculated using the stock segmentation sequence and multiple wide table segmentation sequences. These similarity values ​​have a corresponding relationship with the corresponding replaceable wide table data models. Then, according to the preset screening conditions, the similarity values ​​that meet the conditions are screened out from the calculated similarities, and the replaceable wide table data models corresponding to these similarity values ​​are determined as the target replacement wide table data models. This application uses the word frequency TF-IDF and cosine similarity algorithms to calculate similarity, which can accurately find the most suitable wide table for replacement and reduce human errors. The automatic matching process greatly shortens the model replacement time, enables the wide table replacement work to be completed faster, and improves the overall analysis efficiency.

[0165] See also Figure 3 , Figure 3 A schematic diagram of a wide table data model replacement device based on table structure similarity provided in an embodiment of the present application. Figure 3 As shown, the wide table data model replacement device based on table structure similarity includes:

[0166] A first determining unit 301 is used to determine a stock data model to be replaced and a plurality of replaceable wide table data models;

[0167] A first extraction and splicing unit 302 is used to extract multiple table structure information of the stock data model to be replaced, and perform text splicing to obtain stock description information of the stock data model to be replaced;

[0168] A second extraction and splicing unit 303 is used to extract multiple table structure information of each replaceable wide table data model, and perform text splicing to obtain wide table description information of each replaceable wide table data model;

[0169] A word segmentation unit 304 is used to segment the existing description information to obtain an existing word segmentation sequence, and segment the wide table description information of each replaceable wide table data model to obtain multiple wide table word segmentation sequences;

[0170] A first calculation unit 305 is used to calculate the similarity between the to-be-replaced existing data model and each of the replaceable wide table data models using the existing word segmentation sequence and the multiple wide table word segmentation sequences based on a TF-IDF cosine similarity algorithm; the similarity has a corresponding relationship with the replaceable wide table data model;

[0171] The screening unit 306 is configured to determine the replaceable wide table data model corresponding to the similarity that meets the screening condition as the target replacement wide table data model of the stock data model to be replaced.

[0172] In a possible implementation, the first calculating unit 305 specifically includes:

[0173] A second calculation unit is used to calculate the TF-IDF value of each word in the existing word sequence;

[0174] A third calculation unit is used to calculate the TF-IDF value of each word in the plurality of wide table word sequences;

[0175] A first vectorization unit, configured to vectorize the existing word segmentation sequence by using the TF-IDF value of each word in the existing word segmentation sequence to obtain an existing word segmentation vector;

[0176] A second vectorization unit is used to vectorize the plurality of wide-table word segmentation sequences respectively by using the TF-IDF value of each word in the plurality of wide-table word segmentation sequences to obtain a plurality of wide-table word segmentation vectors;

[0177] The fourth calculation unit is used to calculate the similarity between the to-be-replaced existing data model and each of the replaceable wide table data models based on the existing word segmentation vectors and the multiple wide table word segmentation vectors by using a cosine similarity formula.

[0178] In a possible implementation manner, the second computing unit specifically includes:

[0179] A fifth calculation unit, used to calculate the TF value of each word in the existing word segmentation sequence according to the TF value calculation formula, and calculate the IDF value of each word in the existing word segmentation sequence according to the IDF value calculation formula;

[0180] The sixth calculation unit is used to multiply the TF value and the IDF value of each word in the existing word segmentation sequence to obtain the TF-IDF value of each word in the existing word segmentation sequence.

[0181] Wherein, the TF value calculation formula is:

[0182]

[0183] The IDF value calculation formula is:

[0184]

[0185] In a possible implementation, the screening unit 306 specifically includes:

[0186] A sorting unit, used for sorting corresponding replaceable wide table data models according to the similarity;

[0187] A second determining unit, configured to determine a replaceable wide table data model corresponding to the similarity that ranks in accordance with the screening condition as a target replacement wide table data model for the stock data model to be replaced;

[0188] The screening condition is a rule for filtering and selecting replaceable wide table data models whose similarity rankings meet specific standards.

[0189] In a possible implementation, the screening condition includes:

[0190] Determine the replaceable wide table data model corresponding to the similarity greater than the similarity threshold as the target replacement wide table data model of the stock data model to be replaced; or,

[0191] The replaceable wide table data models corresponding to the similarities ranked in the top N are determined as target replacement wide table data models for the stock data model to be replaced.

[0192] In addition, an embodiment of the present application also provides a wide table data model replacement device based on table structure similarity, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the wide table data model replacement method based on table structure similarity as described above is implemented.

[0193] In addition, an embodiment of the present application further provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes the wide table data model replacement method based on table structure similarity as described above.

[0194] The embodiment of the present application provides a wide table data model replacement device based on table structure similarity, first using a first determination unit 301 to determine the stock data model to be replaced and multiple replaceable wide table data models, and using a first extraction splicing unit 302 and a second extraction splicing unit 303 to extract multiple table structure information of the stock data model to be replaced and each replaceable wide table data model, and perform text splicing to obtain the stock description information of the stock data model to be replaced and the wide table description information of each replaceable wide table data model. The word segmentation unit 304 performs word segmentation on the stock description information to obtain a stock word segmentation sequence, and at the same time, performs word segmentation on the wide table description information of each replaceable wide table data model to obtain multiple wide table word segmentation sequences. Then, the first calculation unit 305 uses the TF-IDF cosine similarity algorithm to calculate the similarity between the stock data model to be replaced and each replaceable wide table data model using the stock word segmentation sequence and the multiple wide table word segmentation sequences, so that the screening unit 306 determines the replaceable wide table data model corresponding to the similarity that meets the screening condition as the target replacement wide table data model of the stock data model to be replaced. This application can more accurately identify wide tables suitable for replacement by using the word frequency TF-IDF and cosine similarity algorithms to calculate similarity, thereby reducing human errors. In addition, the automated matching process significantly shortens the time for model replacement, speeds up the completion of wide table replacement tasks, and improves the overall analysis efficiency.

[0195] The above is a detailed introduction to a wide table data model replacement method based on table structure similarity and related devices provided by the present application. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can refer to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

[0196] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0197] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

Claims

1. A wide table data model replacement method based on table structure similarity, characterized in that: The method comprises: Determine the existing data model to be replaced and multiple replaceable wide table data models; Extracting multiple table structure information of the stock data model to be replaced, and performing text splicing to obtain stock description information of the stock data model to be replaced; extracting multiple table structure information of each replaceable wide table data model, and performing text splicing to obtain wide table description information of each replaceable wide table data model; Segmenting the existing description information to obtain an existing segmentation sequence, and segmenting the wide table description information of each replaceable wide table data model to obtain a plurality of wide table segmentation sequences; Based on the term frequency TF-inverse document frequency IDF cosine similarity algorithm, the similarity between the to-be-replaced stock data model and each of the replaceable wide table data models is calculated using the stock word segmentation sequence and the multiple wide table word segmentation sequences; the similarity has a corresponding relationship with the replaceable wide table data model; The replaceable wide table data model corresponding to the similarity that meets the screening condition is determined as the target replacement wide table data model of the stock data model to be replaced.

2. The method according to claim 1, characterized in that The method of calculating the similarity between the to-be-replaced existing data model and each of the replaceable wide table data models by using the existing word segmentation sequence and the multiple wide table word segmentation sequences based on the TF-IDF cosine similarity algorithm includes: Calculating the TF-IDF value of each word in the existing word segmentation sequence, and calculating the TF-IDF value of each word in the multiple wide table word segmentation sequences; Using the TF-IDF value of each word in the stock word segmentation sequence, the stock word segmentation sequence is vectorized to obtain a stock word segmentation vector, and using the TF-IDF value of each word in the multiple wide-table word segmentation sequences, the multiple wide-table word segmentation sequences are respectively vectorized to obtain multiple wide-table word segmentation vectors; The cosine similarity formula is used to calculate the similarity between the to-be-replaced existing data model and each of the replaceable wide table data models based on the existing word segmentation vectors and the multiple wide table word segmentation vectors.

3. The method according to claim 2, characterized in that The calculating the TF-IDF value of each word in the existing word sequence includes: Calculate the TF value of each word in the existing word segmentation sequence according to the TF value calculation formula, and calculate the IDF value of each word in the existing word segmentation sequence according to the IDF value calculation formula; The TF value and the IDF value of each word in the existing word segmentation sequence are multiplied to obtain the TF-IDF value of each word in the existing word segmentation sequence. Wherein, the TF value calculation formula is: The IDF value calculation formula is:

4. The method according to claim 1, characterized in that: The step of determining the replaceable wide table data model corresponding to the similarity that meets the screening condition as the target replacement wide table data model of the stock data model to be replaced includes: Sorting the corresponding replaceable wide table data models according to the magnitude of the similarity, and determining the replaceable wide table data models corresponding to the similarity that rank and meet the screening condition as target replacement wide table data models for the stock data model to be replaced; The screening condition is a rule for filtering and selecting replaceable wide table data models whose similarity rankings meet specific standards.

5. The method according to claim 1 or 4, characterized in that: The screening conditions include: Determine the replaceable wide table data model corresponding to the similarity greater than the similarity threshold as the target replacement wide table data model of the stock data model to be replaced; or, The replaceable wide table data models corresponding to the similarities ranked in the top N are determined as target replacement wide table data models for the stock data model to be replaced.

6. A wide table data model replacement device based on table structure similarity, characterized in that: The device comprises: A first determining unit, configured to determine a stock data model to be replaced and a plurality of replaceable wide table data models; A first extraction and splicing unit is used to extract multiple table structure information of the stock data model to be replaced, and perform text splicing to obtain stock description information of the stock data model to be replaced; A second extraction and splicing unit is used to extract multiple table structure information of each replaceable wide table data model, and perform text splicing to obtain wide table description information of each replaceable wide table data model; A word segmentation unit, configured to segment the existing description information to obtain an existing word segmentation sequence, and segment the wide table description information of each replaceable wide table data model to obtain a plurality of wide table word segmentation sequences; A first calculation unit is used to calculate the similarity between the to-be-replaced stock data model and each of the replaceable wide table data models by using the stock word segmentation sequence and the multiple wide table word segmentation sequences based on a TF-IDF cosine similarity algorithm; the similarity has a corresponding relationship with the replaceable wide table data model; The screening unit is used to determine the replaceable wide table data model corresponding to the similarity that meets the screening condition as the target replacement wide table data model of the stock data model to be replaced.

7. The device according to claim 6, characterized in that The first calculation unit specifically includes: A second calculation unit is used to calculate the TF-IDF value of each word in the existing word sequence; A third calculation unit is used to calculate the TF-IDF value of each word in the plurality of wide table word sequences; A first vectorization unit, configured to vectorize the existing word segmentation sequence by using the TF-IDF value of each word in the existing word segmentation sequence to obtain an existing word segmentation vector; A second vectorization unit is used to vectorize the plurality of wide-table word segmentation sequences respectively by using the TF-IDF value of each word in the plurality of wide-table word segmentation sequences to obtain a plurality of wide-table word segmentation vectors; The fourth calculation unit is used to calculate the similarity between the to-be-replaced existing data model and each of the replaceable wide table data models based on the existing word segmentation vectors and the multiple wide table word segmentation vectors by using a cosine similarity formula.

8. The method according to claim 7, characterized in that The second calculation unit specifically includes: A fifth calculation unit, used to calculate the TF value of each word in the existing word segmentation sequence according to the TF value calculation formula, and calculate the IDF value of each word in the existing word segmentation sequence according to the IDF value calculation formula; The sixth calculation unit is used to multiply the TF value and the IDF value of each word in the existing word segmentation sequence to obtain the TF-IDF value of each word in the existing word segmentation sequence. Wherein, the TF value calculation formula is: The IDF value calculation formula is:

9. A wide table data model replacement device based on table structure similarity, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for replacing a wide table data model based on table structure similarity as described in any one of claims 1 to 5 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the wide table data model replacement method based on table structure similarity as described in any one of claims 1 to 5.

Images