Wide table data model replacement method based on table structure similarity and related device

Abstract

The application discloses a wide table data model replacement method based on table structure similarity and related devices. When the method provided by the application is executed, first, the table structure information of the inventory data model to be replaced is extracted and text splicing is performed to generate detailed description information. At the same time, the table structure information of each replaceable wide table data model is text spliced to form respective description information. Then, the description information is subjected to word segmentation processing to generate a word segmentation sequence. The TF-IDF cosine similarity algorithm is used to calculate the similarity between the inventory data model to be replaced and each wide table data model, and the similarity values have a corresponding relationship with the wide table data models. Finally, the wide table data models corresponding to the similarity that meet the conditions are selected as the reference and guidance for subsequent data model replacement according to the screening conditions. The application can not only automatically identify the most suitable wide table for replacement, but also improve the overall data processing efficiency and accuracy.

Description

Technical Field

[0001] This application relates to the field of computer data processing technology, specifically to a method and related apparatus for replacing wide table data models based on table structure similarity. Background Technology

[0002] As the big data field continues to evolve, data warehouses are experiencing rapid updates and replacements in data model design, with wide tables gradually replacing traditional stock mart models. Wide tables are standardized, business-oriented, concatenated, and consolidated organizations of open-source foundational data, employing a combination of dimensional modeling and third-normalization modeling. Building wide tables not only meets the requirements of model universality, scalability, and stability but also improves business readability and shortens iteration cycles. Therefore, designing excellent wide tables to replace existing mart models is highly meaningful.

[0003] Once a large number of wide tables are built, the process of enabling downstream users to quickly replace the existing data mart models with the new wide tables becomes particularly important. The design specifications for wide tables differ from those of existing models; table names and field names are more standardized. The table structures, names, and field names and numbers may differ between the old and new models. Classifying data models according to business themes (such as loans, deposits, etc.) and then replacing the data models with wide tables that share the same business themes often still relies on manual classification and replacement, resulting in very low automation, low analytical efficiency, and significant time and effort required.

[0004] Therefore, how to automatically replace the original data mart model with a large number of wide tables to improve analysis efficiency and reduce manual work is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a method and related apparatus for replacing wide table data models based on table structure similarity. This method can automatically replace the original data mart model with a large number of wide tables, thereby improving analysis efficiency and reducing manual work.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] A method for replacing wide table data models based on table structure similarity, the method comprising:

[0008] Identify the existing data model to be replaced and several replaceable wide table data models;

[0009] Extract various table structure information of the existing data model to be replaced, and concatenate the text to obtain the existing description information of the existing data model to be replaced; extract various table structure information of each replaceable wide table data model, and concatenate the text to obtain the wide table description information of each replaceable wide table data model.

[0010] The existing description information is segmented into words to obtain an existing word segmentation sequence, and the wide table description information of each replaceable wide table data model is segmented into words to obtain multiple wide table word segmentation sequences.

[0011] Based on the cosine similarity algorithm of word frequency TF-inverse document frequency IDF, the similarity between the existing word segmentation sequence and the multiple wide table word segmentation sequences is calculated using the existing data model to be replaced and each of the replaceable wide table data models; the similarity has a corresponding relationship with the replaceable wide table data models;

[0012] The replaceable wide table data models corresponding to the similarity that meet the screening criteria are determined as the target replacement wide table data models of the existing data models to be replaced.

[0013] In one possible implementation, the TF-IDF cosine similarity algorithm, which calculates the similarity between the existing word segmentation sequence and each of the replaceable wide table data models using the existing word segmentation sequence and the multiple wide table word segmentation sequences, includes:

[0014] Calculate the TF-IDF value of each word in the existing word segmentation sequence, and calculate the TF-IDF value of each word in the multiple wide-table word segmentation sequences;

[0015] The existing word segmentation sequence is vectorized using the TF-IDF value of each word in the existing word segmentation sequence to obtain an existing word segmentation vector. The multiple wide-table word segmentation sequences are vectorized using the TF-IDF value of each word in the multiple wide-table word segmentation sequences to obtain multiple wide-table word segmentation vectors.

[0016] Using the cosine similarity formula, the similarity between the existing word segmentation vector and each of the replaceable wide table word segmentation vectors is calculated based on the existing word segmentation vector and the multiple wide table word segmentation vectors.

[0017] In one possible implementation, calculating the TF-IDF value of each word in the existing word segmentation sequence includes:

[0018] The TF value of each word in the existing word segmentation sequence is calculated according to the TF value calculation formula, and the IDF value of each word in the existing word segmentation sequence is calculated according to the IDF value calculation formula.

[0019] The TF and IDF values ​​of each word in the existing word segmentation sequence are multiplied together to obtain the TF-IDF value of each word in the existing word segmentation sequence.

[0020] The formula for calculating the TF value is as follows:

[0021]

[0022] The formula for calculating the IDF value is:

[0023]

[0024] In one possible implementation, determining the replaceable wide table data model corresponding to the similarity that meets the screening criteria as the target replacement wide table data model for the existing data model to be replaced includes:

[0025] The corresponding replaceable wide table data models are sorted according to the similarity, and the replaceable wide table data models whose ranking meets the filtering conditions are determined as the target replaceable wide table data models of the existing data models to be replaced.

[0026] The filtering criteria are rules used to filter and select replaceable wide table data models whose similarity rankings meet specific criteria.

[0027] In one possible implementation, the filtering criteria include:

[0028] The replaceable wide table data model corresponding to the similarity score greater than the similarity threshold is determined as the target replacement wide table data model for the existing data model to be replaced; or,

[0029] The replaceable wide table data models corresponding to the top N similarity rankings are determined as the target replacement wide table data models of the existing data models to be replaced.

[0030] A wide table data model replacement device based on table structure similarity, the device comprising:

[0031] The first determining unit is used to determine the existing data model to be replaced and multiple replaceable wide table data models;

[0032] The first extraction and splicing unit is used to extract various table structure information of the existing data model to be replaced, and perform text splicing to obtain the existing description information of the existing data model to be replaced.

[0033] The second extraction and splicing unit is used to extract various table structure information of each replaceable wide table data model and perform text splicing to obtain the wide table description information of each replaceable wide table data model.

[0034] The word segmentation unit is used to segment the existing description information to obtain an existing word segmentation sequence, and to segment the wide table description information of each replaceable wide table data model to obtain multiple wide table word segmentation sequences.

[0035] The first calculation unit is used to calculate the similarity between the existing data model to be replaced and each of the replaceable wide table data models based on the TF-IDF cosine similarity algorithm, using the existing word segmentation sequence and the multiple wide table word segmentation sequences; the similarity has a corresponding relationship with the replaceable wide table data models;

[0036] The filtering unit is used to determine the replaceable wide table data model corresponding to the similarity that meets the filtering conditions as the target replacement wide table data model of the existing data model to be replaced.

[0037] In one possible implementation, the first computing unit specifically includes:

[0038] The second calculation unit is used to calculate the TF-IDF value of each word in the existing word segmentation sequence;

[0039] The third calculation unit is used to calculate the TF-IDF value of each word in the multiple wide-table word segmentation sequences;

[0040] The first vectorization unit is used to vectorize the existing word segmentation sequence using the TF-IDF values ​​of each word in the existing word segmentation sequence to obtain the existing word segmentation vector.

[0041] The second vectorization unit is used to vectorize the multiple wide-table word segmentation sequences respectively using the TF-IDF values ​​of each word in the multiple wide-table word segmentation sequences to obtain multiple wide-table word segmentation vectors;

[0042] The fourth calculation unit is used to calculate the similarity between the existing data model to be replaced and each of the replaceable wide table data models based on the cosine similarity formula, using the existing word segmentation vectors and the multiple wide table word segmentation vectors.

[0043] In one possible implementation, the second computing unit specifically includes:

[0044] The fifth calculation unit is used to calculate the TF value of each word in the existing word segmentation sequence according to the TF value calculation formula, and to calculate the IDF value of each word in the existing word segmentation sequence according to the IDF value calculation formula.

[0045] The sixth calculation unit is used to multiply the TF value and IDF value of each word in the existing word segmentation sequence to obtain the TF-IDF value of each word in the existing word segmentation sequence.

[0046] The formula for calculating the TF value is as follows:

[0047]

[0048] The formula for calculating the IDF value is:

[0049]

[0050] A wide table data model replacement device based on table structure similarity includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the wide table data model replacement method based on table structure similarity as described above.

[0051] A computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform the wide table data model replacement method based on table structure similarity as described above.

[0052] Compared with the prior art, this application has the following beneficial effects:

[0053] This application provides a method and related apparatus for replacing wide table data models based on table structure similarity. Specifically, when executing the wide table data model replacement method based on table structure similarity provided in this application embodiment, firstly, the existing data model to be replaced and multiple replaceable wide table data models are determined. Next, various table structure information is extracted from the existing data model to be replaced, and a description of the existing data model is generated by text concatenation. Simultaneously, various table structure information is also extracted from each replaceable wide table data model, and text concatenation is performed to generate their respective wide table description information.

[0054] Next, these descriptive information are segmented into word sequences, generating both existing word sequences and multiple wide-table word sequences. A cosine similarity algorithm combining Term Frequency (TF) and Inverse Document Frequency (IDF) is used to calculate the similarity between the existing data model to be replaced and each replaceable wide-table data model. Each similarity value corresponds to a specific replaceable wide-table data model. Finally, based on preset filtering criteria, the replaceable wide-table data models corresponding to the required similarity values ​​are selected and designated as the target replacement wide-table data models for the existing data model to be replaced. This application achieves automatic similarity calculation between existing data models and replaceable wide-table data models by using the TF-IDF cosine similarity algorithm, reducing the need for manual intervention and improving replacement efficiency. Simultaneously, through word segmentation and similarity calculation, the most suitable wide-table data model for replacement can be intelligently matched, improving the accuracy and precision of the replacement. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 A flowchart illustrating a method for replacing a wide table data model based on table structure similarity, provided in an embodiment of this application;

[0057] Figure 2 A flowchart illustrating a similarity calculation method provided in this application embodiment;

[0058] Figure 3 This is a schematic diagram of a wide table data model replacement device based on table structure similarity provided in an embodiment of this application. Detailed Implementation

[0059] To facilitate understanding of the technical solutions provided in the embodiments of this application, the background technology involved in the embodiments of this application will be described below.

[0060] As the field of big data continues to evolve, the design of data models in data warehouses is also constantly being updated. Traditional mart inventory models are gradually being replaced by wide tables. Wide tables are an organizational method that standardizes, business-orientedizes, concatenates, and consolidates open-source foundational data, employing a combination of dimensional modeling and third-normalization modeling. By building wide tables, not only can the requirements for model universality, scalability, and stability be met, but business readability can also be improved, and iteration cycles shortened. Therefore, designing excellent wide tables to replace the original mart model is of great significance.

[0061] However, after a large number of wide tables are built, enabling downstream systems to quickly replace the original data mart models with the new wide tables becomes crucial. Wide tables differ from existing models in design specifications, with more standardized table and field names. Because the table structures of the old and new models differ, table and field names may also differ. When categorizing data models according to business themes (such as loans, deposits, etc.) and replacing the original models with wide tables matching these themes, in most cases, the categorization and replacement still rely on manual work, resulting in extremely low automation. This not only leads to low analytical efficiency but also consumes a significant amount of time and manpower.

[0062] To address this issue, this application provides a method and related apparatus for replacing wide table data models based on table structure similarity. First, the existing data model to be replaced and multiple replaceable wide table data models are determined. Then, various table structure information is extracted from the existing data model to be replaced, and its text is concatenated to form a description of the existing data model. Similarly, similar table structure information extraction and text concatenation are performed on each replaceable wide table data model to obtain corresponding wide table description information. Next, word segmentation is performed on the existing description information and each wide table description information, converting them into corresponding word segmentation sequences, resulting in existing word segmentation sequences and multiple wide table word segmentation sequences. Furthermore, using a TF-IDF cosine similarity algorithm, the similarity between the existing word segmentation sequence and each replaceable wide table data model is calculated by comparing the existing word segmentation sequence and the multiple wide table word segmentation sequences. These similarities correspond to replaceable wide table data models, and finally, wide table data models with similarity scores that meet the criteria are selected as the target replacement wide table data models. This application calculates similarity using word frequency-based TF-IDF and cosine similarity algorithms, enabling more accurate identification of suitable wide tables for replacement and reducing human error. Furthermore, the automated matching process significantly shortens model replacement time, thus completing the wide table replacement task faster and improving overall analysis efficiency.

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0064] See Figure 1 The figure is a flowchart of a wide table data model replacement method based on table structure similarity provided in an embodiment of this application. Figure 1 As shown, this wide table data model replacement method based on table structure similarity may include steps S101-S105:

[0065] S101: Determine the existing data model to be replaced and multiple replaceable wide table data models.

[0066] Identifying the existing data model to be replaced and multiple alternative wide-table data models refers to the process of identifying the old data model (existing data model) that needs to be replaced during data warehouse transformation, and selecting multiple new wide-table data models as potential alternatives. This process typically involves a comprehensive evaluation and comparison of the existing and new data models to ensure that the new wide tables better meet business needs, improve data processing efficiency, and enhance system flexibility.

[0067] For example, suppose there is a bank's data warehouse containing multiple traditional mart stock data models, such as a "loan model." These traditional models typically store and query data through complex multi-table joins. As the system evolves, in order to improve data processing efficiency and enhance business flexibility, it is decided to introduce the concept of wide tables to replace these traditional models.

[0068] Existing data model:

[0069] Loan Model: Includes basic information about the borrower, loan amount, repayment plan, etc.

[0070] Replaceable wide table data model:

[0071] Customer Comprehensive Info Wide Table: This table integrates customer loan and deposit information, storing it in a wide table for easy and quick querying and analysis.

[0072] Funding Activity Wide Table: Focuses on a client's funding activities, including information on loans and deposits, facilitating specific types of business analysis.

[0073] In this example, the "Loan Model" is identified as the existing data model to be replaced, while the "Customer Comprehensive Information Wide Table" and the "Financing Activity Wide Table" are considered as candidate replacement wide table data models. This replacement not only simplifies the data model structure but also improves the efficiency of data querying and analysis, better supporting business decisions.

[0074] S102: Extract various table structure information of the existing data model to be replaced, and perform text concatenation to obtain the existing description information of the existing data model to be replaced; extract various table structure information of each replaceable wide table data model, and perform text concatenation to obtain the wide table description information of each replaceable wide table data model.

[0075] Extract various table structure information, such as table names and field names, from the existing data model to be replaced, and generate a descriptive description of the existing data model through text concatenation. Similarly, extract relevant information from all replaceable wide table data models and generate their respective descriptive descriptions.

[0076] Specifically, the first step is to extract the various structural features of the "existing data model to be replaced"—that is, the old data model that is about to be updated or replaced—such as detailed information like Chinese table names and field names. Then, this information is combined to form a complete "existing data description." This process helps to fully understand the characteristics and status of the existing data model.

[0077] The next step is to process the "replaceable wide table data models." These new models are alternatives to replace the old models. Similarly, it's necessary to extract various table structure information from these new models, including but not limited to their Chinese table names, various field names, etc., and then summarize this information into a "wide table description" for each new model through text concatenation. This is done to systematically compare the differences between the old and new models, facilitating the evaluation and selection of the most suitable data model for replacement.

[0078] For example, suppose an e-commerce company is using an old data model to store product information. This model contains a table called the "Order Table," whose fields include "Order Number," "Customer Identifier (ID)," and "Product ID." Now, the company decides to upgrade its data model, with three new wide-table data models as candidates. Each new model's "Order Table" has different field configurations; for example, one model adds a "Delivery Address" field, while another introduces a "Payment Method" field.

[0079] Extract the table structure information of the existing data model to be replaced:

[0080] Chinese table name: "Order Table"; Field names: For the original model, there might be a description like this: "Order Number (Unique Identifier), Customer ID (Buyer Identifier), Product ID (Purchased Product Identifier)."

[0081] Concatenate inventory description information: An example of the generated description information is: "The order table includes the order number, and the customer ID includes the product ID field."

[0082] Extract the table structure information for each replaceable wide table data model:

[0083] Model A: Chinese table name: "Order Table"; Field names: "Order Number, Customer ID, Product ID, Delivery Address".

[0084] Model B: Chinese table name: "Order Table"; Field names: "Order Number, Customer ID, Product ID, Payment Method".

[0085] Description of the concatenated wide table: Model A description: "The order table includes order number, customer ID, product ID, and delivery address fields";

[0086] Model B description: "The order table includes the order number, customer ID, product ID, and payment method fields."

[0087] By following the steps above, the specific structural differences of each model can be clearly seen, allowing you to select the most suitable wide table data model for replacement based on actual business needs.

[0088] S103: The existing description information is segmented to obtain an existing segmentation sequence, and the wide table description information of each replaceable wide table data model is segmented to obtain multiple wide table segmentation sequences.

[0089] Word segmentation of existing descriptive information refers to dividing the descriptive information of the existing data model into words or phrases according to certain rules for subsequent calculation and comparison. Similarly, the descriptive information of replaceable wide table data models also needs to undergo the same processing, splitting it into sequences of words or phrases for subsequent similarity calculation.

[0090] For example, suppose the concatenated inventory description information is as follows: The account information table includes the fields: account ID, user ID, account type, and account balance.

[0091] The concatenated wide table description information is as follows: The user account comprehensive information table includes the following fields: user ID, name, ID card number, phone number, account ID, account type, and account balance.

[0092] By segmenting the existing descriptive information into words, the following segmentation sequence can be obtained:

[0093] "The account information table includes fields such as account ID, user ID, account type, and account balance."

[0094] By segmenting the wide table description information into words, the following segmentation sequence can be obtained:

[0095] The "User Account Comprehensive Information Table" includes the following fields: User ID, Name, ID Number, Phone Number, Account ID, Account Type, and Account Balance.

[0096] S104: Based on the TF-IDF cosine similarity algorithm, the similarity between the existing word segmentation sequence and the multiple wide table word segmentation sequences is calculated.

[0097] A cosine similarity algorithm combining Term Frequency (TF) and Inverse Document Frequency (IDF) is used to calculate the similarity between the existing segmented word sequence and each replaceable wide-table data model, based on the existing segmented word sequence and multiple wide-table segmented word sequences. Each similarity value corresponds to a replaceable wide-table data model.

[0098] See Figure 2 , Figure 2 This application provides a flowchart of a similarity calculation method. Accordingly, step S104, based on the TF-IDF cosine similarity algorithm, calculates the similarity between the existing segmented word sequence and each of the replaceable wide-table segmented word sequences. This can be achieved through steps A1-A3.

[0099] A1: Calculate the TF-IDF value of each word in the existing word segmentation sequence, and calculate the TF-IDF value of each word in the multiple wide-table word segmentation sequences.

[0100] Calculating the TF-IDF value of each word in the existing word segmentation sequence and multiple wide-table word segmentation sequences is a method for measuring the importance of words in text. TF refers to the frequency of a word's occurrence in the text, while IDF represents the inverse of a word's importance in the entire document set. By combining these two pieces of information, the TF-IDF value can help determine the importance of a word in a specific text, thus affecting the overall similarity calculation results.

[0101] Specifically, calculating the TF-IDF value of each word in the existing word segmentation sequence and multiple wide-table word segmentation sequences is to quantify the importance of each word to the text description. Words with high TF-IDF values ​​usually appear frequently in a specific text but less frequently in the entire document set, and are therefore more representative. This calculation process helps to accurately assess the contribution of keywords or phrases to the description model, thus providing a more accurate basis for subsequent similarity calculations.

[0102] By calculating the TF-IDF values ​​of words in existing word segmentation sequences and multiple wide-table word segmentation sequences, we can better understand the content and features of text data, providing a deeper analytical foundation for the automatic matching process of model replacement tasks, and ensuring that the most suitable wide-table data model for replacement is selected to improve analytical efficiency and accuracy.

[0103] In one possible implementation, calculating the TF-IDF value of each word in the existing word segmentation sequence includes:

[0104] The TF value of each word in the existing word segmentation sequence is calculated according to the TF value calculation formula, and the IDF value of each word in the existing word segmentation sequence is calculated according to the IDF value calculation formula.

[0105] The TF and IDF values ​​of each word in the existing word segmentation sequence are multiplied together to obtain the TF-IDF value of each word in the existing word segmentation sequence.

[0106] The formula for calculating the TF value is as follows:

[0107]

[0108] The formula for calculating the IDF value is:

[0109]

[0110] Example 1, suppose we have the following word segmentation sequence:

[0111] Existing word segmentation sequence: [data, management, data, model]

[0112] Wide table word segmentation sequence 1: [data, processing, wide table, model]

[0113] Wide table word segmentation sequence 2: [data, optimization, data, management]

[0114] First, calculate the frequency of each word in its respective sequence:

[0115] TF values ​​of existing word segmentation sequences: Data: 2 / 4 = 0.5; Management: 1 / 4 = 0.25; Model: 1 / 4 = 0.25.

[0116] TF values ​​for wide-table word segmentation sequence 1: Data: 1 / 4 = 0.25; Processing: 1 / 4 = 0.25; Wide table: 1 / 4 = 0.25; Model: 1 / 4 = 0.25.

[0117] TF values ​​for wide-table word segmentation sequence 2: Data: 2 / 4 = 0.5; Optimization: 1 / 4 = 0.25; Management: 1 / 4 = 0.25.

[0118] Then, the IDF value of each word is calculated based on the number of documents containing a certain word in all documents (i.e., all word segments):

[0119] In 3 sequences, "data" appeared 3 times; in 3 sequences, "management" appeared 2 times; in 2 sequences, "model" appeared 2 times; in 1 sequence, "processing" appeared 1 time; in 1 sequence, "wide table" appeared 1 time; and in 1 sequence, "optimization" appeared 1 time. The total number of sequences is 3.

[0120] The IDF value of the data is: log(3 / 3) = log(1) = 0; the IDF value of the management is: log(3 / 2) ≈ 0.176; the IDF value of the model is: log(3 / 2) ≈ 0.176; the IDF value of the processing is: log(3 / 1) ≈ 0.477; the IDF value of the wide table is: log(3 / 1) ≈ 0.477; the IDF value of the optimization is: log(3 / 1) ≈ 0.477.

[0121] Finally, multiply the TF value and IDF value to obtain the TF-IDF value:

[0122] TF-IDF values ​​of existing word segmentation sequences: Data: 0.5*0=0; Management: 0.25*0.176≈0.044; Model: 0.25*0.176≈0.044.

[0123] TF-IDF values ​​of wide table segmentation sequence 1: Data: 0.25*0=0; Processing: 0.25*0.477≈0.119; Wide table: 0.25*0.477≈0.119; Model: 0.25*0.176≈0.044.

[0124] TF-IDF values ​​of wide table segmentation sequence 2: Data: 0.5*0=0; Optimization: 0.25*0.477≈0.119; Management: 0.25*0.176≈0.044.

[0125] It should be noted that the calculation method for the TF-IDF value of each word in the wide word segmentation sequence is similar to that of the calculation method for the TF-IDF value of each word in the stock word segmentation sequence, and will not be repeated here.

[0126] A2: The existing word segmentation sequence is vectorized using the TF-IDF values ​​of each word in the existing word segmentation sequence to obtain an existing word segmentation vector. The multiple wide-table word segmentation sequences are vectorized using the TF-IDF values ​​of each word in the multiple wide-table word segmentation sequences to obtain multiple wide-table word segmentation vectors.

[0127] Vectorization using the TF-IDF values ​​of existing word segmentation sequences and wide-table word segmentation sequences can help represent the similarity or correlation between different texts in a high-dimensional space. This vectorization method represents each text as a multi-dimensional vector, where each dimension corresponds to the TF-IDF value of the word segmentation. By comparing these vectors, the degree of similarity between texts can be assessed.

[0128] Example 2, continuing from Example 1, uses the TF-IDF values ​​calculated in Example 1 to convert each sequence into vector form. For example:

[0129] Vector representation of the existing word segmentation sequence: [0, 0.044, 0.25, 0.044];

[0130] The vector representation of the wide-table word segmentation sequence 1 is: [0, 0.119, 0.119, 0.044];

[0131] The vector representation of the wide-table word segmentation sequence 2 is: [0, 0.119, 0.044].

[0132] A3: Using the cosine similarity formula, calculate the similarity between the existing word segmentation vector and each of the replaceable wide table data models based on the existing word segmentation vector and the multiple wide table word segmentation vectors.

[0133] The cosine similarity formula can be used to measure the degree of similarity between two vectors. Based on existing word segmentation vectors and multiple wide-table word segmentation vectors, the similarity between the existing data model to be replaced and each replaceable wide-table data model can be calculated. This similarity metric can help understand the correlation and degree of similarity between different data models.

[0134] For example, suppose we have the following vector representation:

[0135] Vector representation of the existing word segmentation sequence: A = [0, 0.044, 0.25, 0.044];

[0136] The vector representation of the wide-table word segmentation sequence 1: B1 = [0, 0.119, 0.119, 0.044];

[0137] The vector representation of the wide-table word segmentation sequence 2 is: B2 = [0, 0.119, 0.044, 0].

[0138] The formula for cosine similarity is as follows: Where A and B represent two vectors, A·B represents the dot product of the vectors; ||A|| and ||B|| represent the magnitudes of the vectors.

[0139] Similarity calculation between the existing word segmentation sequence and the wide-table word segmentation sequence 1:

[0140] Calculate the dot product:

[0141] A·B1=0*0+0.044*0.119+0.25*0.119+0.044*0.044=0+0.005236+0.02975+0.001936=0.036922.

[0142] Calculate the modulus:

[0143] The modulus of the existing word segmentation sequence is: ||A||≈0.2576; the modulus of the wide word segmentation sequence 1 is: ||B1||≈0.171.

[0144] Calculate cosine similarity:

[0145]

[0146] Similarity calculation between the existing word segmentation sequence and the wide-table word segmentation sequence 2:

[0147] Calculate the dot product:

[0148] A·B1=0.036922.

[0149] Calculate the modulus:

[0150] The modulus of the existing word segmentation sequence is: ||A||≈0.2576; the modulus of the wide word segmentation sequence 2 is: ||B2||≈0.127.

[0151] Calculate cosine similarity:

[0152]

[0153] S105: The replaceable wide table data model corresponding to the similarity that meets the screening criteria is determined as the target replacement wide table data model of the existing data model to be replaced.

[0154] Based on preset filtering criteria, the wide table data model corresponding to the required similarity value is selected and identified as the target wide table data model to be replaced in the existing data model. This step ensures that the selected replacement model not only has high similarity but also meets specific business needs or standards.

[0155] In one possible implementation, determining the replaceable wide table data model corresponding to the similarity that meets the screening criteria as the target replacement wide table data model for the existing data model to be replaced includes:

[0156] The corresponding replaceable wide table data models are sorted according to the similarity, and the replaceable wide table data models whose ranking meets the filtering conditions are determined as the target replaceable wide table data models of the existing data models to be replaced.

[0157] The filtering criteria are rules used to filter and select replaceable wide table data models whose similarity rankings meet specific criteria.

[0158] After obtaining the similarity scores between the existing data model to be replaced and each of the replaceable wide-table data models, the wide-table segmentation sequences need to be sorted based on these similarity values. The sorting criterion is the magnitude of the similarity value; that is, the higher the similarity value, the better the corresponding wide-table segmentation sequence matches the existing segmentation sequence. In this way, it is possible to intuitively see which wide-table segmentation sequence is closest to the existing segmentation sequence. Then, specific replacement models are selected according to pre-defined filtering conditions. These filtering conditions are used to filter and select wide-table segmentation sequences whose similarity ranking meets specific criteria. For example, the filtering conditions could be "similarity must be greater than a certain threshold" or "the top N models with the highest similarity must be selected." By applying these rules, models that meet the requirements can be selected from the sorted candidate models as the final replacement options.

[0159] In this process, it is necessary to ensure that the selected model is not only highly similar to the existing word segmentation sequence, but also meets specific business or technical requirements, so as to ensure that the replaced data model can effectively replace the original existing data model, while meeting the system's performance and functional requirements.

[0160] In one possible implementation, the filtering conditions include, but are not limited to:

[0161] Replaceable wide table data models with similarity scores exceeding a certain threshold are identified as target replacement wide table data models for the existing data models to be replaced. This selection criterion means that if the similarity score of a replaceable wide table data model to the existing data model exceeds a set threshold, then that replaceable wide table data model will be identified as the target replacement wide table data model. This implies that only data models with similarity scores above a specific threshold will be selected.

[0162] Alternatively, the top N most relevant wide table data models corresponding to the similarity scores can be identified as the target replacement wide table data models for the existing data model to be replaced. This selection criterion refers to choosing the target replacement wide table data model based on ranking. For example, it can be set to select only the top N most relevant wide table data models as the target replacement wide table data models. This method is suitable for situations where it is necessary to select several of the most relevant models from a set of candidate data models.

[0163] These filtering criteria can be adjusted and customized according to specific needs and scenarios. By setting appropriate filtering criteria, the most suitable replacement options can be accurately selected, ensuring that the chosen target replacement wide table data model meets the established standards and requirements. This filtering process helps improve the efficiency of data analysis and model replacement, and ensures that the selected data model has a high degree of relevance and matching with the original data model.

[0164] Based on the content of S101-S105, we first identify the existing data model that needs to be replaced and list several potential replacement wide-table data models as candidates. We then extract various table structure information from the existing data model to be replaced and concatenate the text to generate its existing description information. Simultaneously, we extract various table structure information for each replacement wide-table data model and concatenate the text to generate its own wide-table description information. Next, we perform word segmentation on the generated existing description information to obtain an existing word segmentation sequence. Similarly, we perform word segmentation on the wide-table description information of each replacement wide-table data model to generate multiple wide-table word segmentation sequences. Using the TF-IDF cosine similarity algorithm, we calculate the similarity between the existing data model to be replaced and each of the replacement wide-table data models using the existing word segmentation sequence and the multiple wide-table word segmentation sequences. These similarity values ​​have a corresponding relationship with the corresponding replacement wide-table data models. Then, based on preset filtering conditions, similarity values ​​that meet the conditions are selected from the calculated similarities, and the replaceable wide table data models corresponding to these similarity values ​​are determined as the target replacement wide table data models. This application uses word frequency TF-IDF and cosine similarity algorithms to calculate similarity, which can accurately find the most suitable wide table for replacement and reduce human error. The automatic matching process significantly shortens the model replacement time, enabling the wide table replacement work to be completed faster and improving the overall analysis efficiency.

[0165] See Figure 3 , Figure 3 This is a schematic diagram of a wide table data model replacement device based on table structure similarity, provided as an embodiment of this application. Figure 3 As shown, the wide table data model replacement device based on table structure similarity includes:

[0166] The first determining unit 301 is used to determine the existing data model to be replaced and multiple replaceable wide table data models;

[0167] The first extraction and splicing unit 302 is used to extract various table structure information of the existing data model to be replaced, and perform text splicing to obtain the existing description information of the existing data model to be replaced.

[0168] The second extraction and splicing unit 303 is used to extract various table structure information of each replaceable wide table data model and perform text splicing to obtain the wide table description information of each replaceable wide table data model.

[0169] The word segmentation unit 304 is used to segment the existing description information to obtain an existing word segmentation sequence, and to segment the wide table description information of each replaceable wide table data model to obtain multiple wide table word segmentation sequences.

[0170] The first calculation unit 305 is used to calculate the similarity between the existing word segmentation sequence and each of the replaceable wide table data models based on the TF-IDF cosine similarity algorithm; the similarity has a corresponding relationship with the replaceable wide table data models.

[0171] The filtering unit 306 is used to determine the replaceable wide table data model corresponding to the similarity that meets the filtering conditions as the target replacement wide table data model of the existing data model to be replaced.

[0172] In one possible implementation, the first computing unit 305 specifically includes:

[0173] The second calculation unit is used to calculate the TF-IDF value of each word in the existing word segmentation sequence;

[0174] The third calculation unit is used to calculate the TF-IDF value of each word in the multiple wide-table word segmentation sequences;

[0175] The first vectorization unit is used to vectorize the existing word segmentation sequence using the TF-IDF values ​​of each word in the existing word segmentation sequence to obtain the existing word segmentation vector.

[0176] The second vectorization unit is used to vectorize the multiple wide-table word segmentation sequences respectively using the TF-IDF values ​​of each word in the multiple wide-table word segmentation sequences to obtain multiple wide-table word segmentation vectors;

[0177] The fourth calculation unit is used to calculate the similarity between the existing data model to be replaced and each of the replaceable wide table data models based on the cosine similarity formula, using the existing word segmentation vectors and the multiple wide table word segmentation vectors.

[0178] In one possible implementation, the second computing unit specifically includes:

[0179] The fifth calculation unit is used to calculate the TF value of each word in the existing word segmentation sequence according to the TF value calculation formula, and to calculate the IDF value of each word in the existing word segmentation sequence according to the IDF value calculation formula.

[0180] The sixth calculation unit is used to multiply the TF value and IDF value of each word in the existing word segmentation sequence to obtain the TF-IDF value of each word in the existing word segmentation sequence.

[0181] The formula for calculating the TF value is as follows:

[0182]

[0183] The formula for calculating the IDF value is:

[0184]

[0185] In one possible implementation, the filtering unit 306 specifically includes:

[0186] A sorting unit is used to sort the corresponding replaceable wide table data models according to the magnitude of the similarity.

[0187] The second determining unit is used to determine the replaceable wide table data model corresponding to the similarity that meets the ranking conditions as the target replacement wide table data model of the existing data model to be replaced.

[0188] The filtering criteria are rules used to filter and select replaceable wide table data models whose similarity rankings meet specific criteria.

[0189] In one possible implementation, the filtering criteria include:

[0190] The replaceable wide table data model corresponding to the similarity score greater than the similarity threshold is determined as the target replacement wide table data model for the existing data model to be replaced; or,

[0191] The replaceable wide table data models corresponding to the top N similarity rankings are determined as the target replacement wide table data models of the existing data models to be replaced.

[0192] In addition, this application embodiment also provides a wide table data model replacement device based on table structure similarity, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the wide table data model replacement method based on table structure similarity as described above.

[0193] In addition, embodiments of this application also provide a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform the wide table data model replacement method based on table structure similarity as described above.

[0194] This application provides a wide table data model replacement device based on table structure similarity. First, a first determining unit 301 determines the existing data model to be replaced and multiple replaceable wide table data models. Then, a first extraction and splicing unit 302 and a second extraction and splicing unit 303 extract various table structure information from the existing data model to be replaced and each replaceable wide table data model, respectively, and perform text splicing to obtain the existing description information of the existing data model to be replaced and the wide table description information of each replaceable wide table data model. A word segmentation unit 304 segments the existing description information to obtain an existing word segmentation sequence, and simultaneously segments the wide table description information of each replaceable wide table data model to obtain multiple wide table segmentation sequences. Then, a first calculation unit 305 uses the TF-IDF cosine similarity algorithm to calculate the similarity between the existing data model to be replaced and each replaceable wide table data model using the existing word segmentation sequence and the multiple wide table segmentation sequences. This allows a filtering unit 306 to determine the replaceable wide table data model corresponding to the similarity that meets the filtering criteria as the target replacement wide table data model for the existing data model to be replaced. This application utilizes similarity calculation algorithms based on word frequency TF-IDF and cosine similarity to more accurately identify suitable wide tables for replacement, thereby reducing human error. Furthermore, the automated matching process significantly shortens the model replacement time, accelerates the completion of wide table replacement tasks, and improves overall analysis efficiency.

[0195] The foregoing has provided a detailed description of a wide table data model replacement method and related apparatus based on table structure similarity provided in this application. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0196] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0197] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for replacing wide table data models based on table structure similarity, characterized in that, The method includes: Identify the existing data model to be replaced and several replaceable wide table data models; Extract various table structure information of the existing data model to be replaced, and concatenate the text to obtain the existing description information of the existing data model to be replaced; extract various table structure information of each replaceable wide table data model, and concatenate the text to obtain the wide table description information of each replaceable wide table data model. The existing description information is segmented into words to obtain an existing word segmentation sequence, and the wide table description information of each replaceable wide table data model is segmented into words to obtain multiple wide table word segmentation sequences. Based on the cosine similarity algorithm of word frequency TF-inverse document frequency IDF, the similarity between the existing word segmentation sequence and the multiple wide table word segmentation sequences is calculated using the existing data model to be replaced and each of the replaceable wide table data models; the similarity has a corresponding relationship with the replaceable wide table data models; The replaceable wide table data models corresponding to the similarity that meet the screening criteria are determined as the target replacement wide table data models of the existing data models to be replaced.

2. The method according to claim 1, characterized in that, The TF-IDF cosine similarity algorithm calculates the similarity between the existing word segmentation sequence and each of the replaceable wide table word segmentation sequences, including: Calculate the TF-IDF value of each word in the existing word segmentation sequence, and calculate the TF-IDF value of each word in the multiple wide-table word segmentation sequences; The existing word segmentation sequence is vectorized using the TF-IDF value of each word in the existing word segmentation sequence to obtain an existing word segmentation vector. The multiple wide-table word segmentation sequences are vectorized using the TF-IDF value of each word in the multiple wide-table word segmentation sequences to obtain multiple wide-table word segmentation vectors. Using the cosine similarity formula, the similarity between the existing word segmentation vector and each of the replaceable wide table word segmentation vectors is calculated based on the existing word segmentation vector and the multiple wide table word segmentation vectors.

3. The method according to claim 2, characterized in that, The calculation of the TF-IDF value of each word in the existing word segmentation sequence includes: The TF value of each word in the existing word segmentation sequence is calculated according to the TF value calculation formula, and the IDF value of each word in the existing word segmentation sequence is calculated according to the IDF value calculation formula. The TF and IDF values ​​of each word in the existing word segmentation sequence are multiplied together to obtain the TF-IDF value of each word in the existing word segmentation sequence. The formula for calculating the TF value is as follows: The formula for calculating the IDF value is:

4. The method according to claim 1, characterized in that, The step of determining the replaceable wide table data model corresponding to the similarity that meets the screening criteria as the target replacement wide table data model for the existing data model to be replaced includes: The corresponding replaceable wide table data models are sorted according to the similarity, and the replaceable wide table data models whose ranking meets the filtering conditions are determined as the target replaceable wide table data models of the existing data models to be replaced. The filtering criteria are rules used to filter and select replaceable wide table data models whose similarity rankings meet specific criteria.

5. The method according to claim 1 or 4, characterized in that, The filtering criteria include: The replaceable wide table data model corresponding to the similarity score greater than the similarity threshold is determined as the target replacement wide table data model for the existing data model to be replaced; or, The replaceable wide table data models corresponding to the top N similarity rankings are determined as the target replacement wide table data models of the existing data models to be replaced.

6. A wide table data model replacement device based on table structure similarity, characterized in that, The device includes: The first determining unit is used to determine the existing data model to be replaced and multiple replaceable wide table data models; The first extraction and splicing unit is used to extract various table structure information of the existing data model to be replaced, and perform text splicing to obtain the existing description information of the existing data model to be replaced. The second extraction and splicing unit is used to extract various table structure information of each replaceable wide table data model and perform text splicing to obtain the wide table description information of each replaceable wide table data model. The word segmentation unit is used to segment the existing description information to obtain an existing word segmentation sequence, and to segment the wide table description information of each replaceable wide table data model to obtain multiple wide table word segmentation sequences. The first calculation unit is used to calculate the similarity between the existing data model to be replaced and each of the replaceable wide table data models based on the TF-IDF cosine similarity algorithm, using the existing word segmentation sequence and the multiple wide table word segmentation sequences; the similarity has a corresponding relationship with the replaceable wide table data models; The filtering unit is used to determine the replaceable wide table data model corresponding to the similarity that meets the filtering conditions as the target replacement wide table data model of the existing data model to be replaced.

7. The apparatus according to claim 6, characterized in that, The first computing unit specifically includes: The second calculation unit is used to calculate the TF-IDF value of each word in the existing word segmentation sequence; The third calculation unit is used to calculate the TF-IDF value of each word in the multiple wide-table word segmentation sequences; The first vectorization unit is used to vectorize the existing word segmentation sequence using the TF-IDF values ​​of each word in the existing word segmentation sequence to obtain the existing word segmentation vector. The second vectorization unit is used to vectorize the multiple wide-table word segmentation sequences respectively using the TF-IDF values ​​of each word in the multiple wide-table word segmentation sequences to obtain multiple wide-table word segmentation vectors; The fourth calculation unit is used to calculate the similarity between the existing data model to be replaced and each of the replaceable wide table data models based on the cosine similarity formula, using the existing word segmentation vectors and the multiple wide table word segmentation vectors.

8. The apparatus according to claim 7, characterized in that, The second calculation unit specifically includes: The fifth calculation unit is used to calculate the TF value of each word in the existing word segmentation sequence according to the TF value calculation formula, and to calculate the IDF value of each word in the existing word segmentation sequence according to the IDF value calculation formula. The sixth calculation unit is used to multiply the TF value and IDF value of each word in the existing word segmentation sequence to obtain the TF-IDF value of each word in the existing word segmentation sequence. The formula for calculating the TF value is as follows: The formula for calculating the IDF value is:

9. A wide table data model replacement device based on table structure similarity, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the wide table data model replacement method based on table structure similarity as described in any one of claims 1-5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the wide table data model replacement method based on table structure similarity as described in any one of claims 1-5.

Images