Ship multi-source data integration method based on Word2Vec model

Abstract

The invention discloses a ship multi-source data integration method based on a Word2Vec model. The method comprises the following steps: 1) collecting ship spare part data from different sources; 2) preprocessing the collected ship spare part data; 3) arranging the preprocessed multi-source ship spare part data, and forming a standard ship target database in combination with a ship database standard; 4) constructing a ship spare part word source according to the spare part name information extracted from the ship target database in the step 3), and constructing a word vector model based on the standardized ship spare part word source; 5) extracting ship spare part names needing to be compared from spare part data from different sources, comparing the ship spare part names with spare part names in the target database, and calculating cosine similarity; and 6) fusing the spare part names to be compared with the spare part names in the target database, and finally generating uniform data representation. The word vector model is used for data matching and fusion, manual intervention can be reduced, and the data integration efficiency is improved.

Description

Technical Field

[0001] The present invention relates to natural language processing, and in particular to a method for integrating multi-source ship data based on a Word2Vec model. Background Art

[0002] With the continuous progress and development of the shipping industry, the types and quantities of ship spare parts are also increasing dramatically. This has led to the diversity of data forms and contents from different sources, making it difficult for traditional data integration methods to cope with the processing and integration of these multi-source heterogeneous data. This data processing challenge further affects the accuracy and efficiency of data analysis and decision-making, which has a negative impact on the operation and development of the entire ship. Integrating multi-source data can make data more usable and analyzable, thereby providing more resources for decision-making, analysis and insight. Through integration, data from different sources can be associated, combined and analyzed to help reveal the correlation and trends between data, thereby providing organizations and individuals with a more comprehensive information basis to support better decision-making and action. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a method for integrating multi-source ship data based on the Word2Vec model in view of the defects in the prior art.

[0004] The technical solution adopted by the present invention to solve the technical problem is: a method for integrating multi-source ship data based on the Word2Vec model, comprising the following steps:

[0005] 1) Collect ship spare parts data from different sources, including spare parts name, specification model, delivery time, spare parts code, demand quantity, and delivery unit information;

[0006] 2) Preprocessing the collected ship spare parts data to obtain preprocessed multi-source ship spare parts data;

[0007] 3) Arrange the pre-processed multi-source ship spare parts data, combine it with the ship database standard, form a standard ship target database, and extract the spare parts name information in the ship target database;

[0008] 4) Construct word vector model Word2Vec;

[0009] Extract the spare parts name information in the ship target database in step 3), construct a ship spare parts word source, and construct a word vector model based on the standardized ship spare parts word source;

[0010] 5) Extract the names of ship spare parts that need to be compared from spare parts data from different sources and compare them with the spare parts names in the target database. Generate word vectors for the spare parts names through the word vector model Word2Vec and calculate the cosine similarity;

[0011] 6) Select the spare part names to be compared and the spare part names in the target database whose cosine similarity is above the set threshold, calculate the edit distance between their specifications, spare part codes and the information of the outgoing unit, and directly merge them when the edit distance is 0; when the edit distance is less than the set value, send it to manual comparison to determine whether to merge, and finally generate a unified data representation;

[0012] The edit distance refers to the minimum number of edit operations required to convert one string into another.

[0013] According to the above scheme, in step 2), the preprocessing includes data cleaning, format conversion and normalization processing, converting the English spare part names into Chinese, deleting English letters, and eliminating missing values ​​and abnormal data in the data.

[0014] According to the above scheme, in step 5), the cosine similarity is calculated as follows:

[0015] 5.1) Determine the keywords in the spare parts names that need to be compared based on the names of the ship spare parts extracted from the spare parts data from different sources;

[0016] 5.2) Extract several keywords from the spare parts names in the target database and merge them into a set;

[0017] 5.3) Find the keywords in the spare part name that needs to be compared and the word vector of the spare part corresponding to a keyword in the set in the pre-trained Word2Vec model in step 4);

[0018] 5.4) Calculate the cosine similarity of two vectors; the range of cosine similarity value is [-1,1], -1 means completely dissimilar, 1 means completely similar, and the larger the value, the more similar it is;

[0019] 5.5) Traverse the spare part names in the target database to obtain the cosine similarity between the spare part name to be compared and the spare part name in the target database.

[0020] According to the above scheme, in step 5.4), the cosine similarity of the two vectors is calculated

[0021]

[0022] Among them, cos(θ) represents the cosine similarity between spare part vector A and spare part vector B, A i and B i Represent the keyword components of spare parts vectors A and B respectively.

[0023] According to the above scheme, in step 6), the text preprocessing is included before the edit distance calculation: the specification model, spare part code, and the information of the shipping unit are converted to lowercase, punctuation marks are removed, extra spaces are removed, stop words are removed, and text is normalized;

[0024] According to the above scheme, in step 6), the edit distance is calculated as follows:

[0025] Assume that the lengths of characters A and B are m and n respectively, and define D(i,j) as the minimum number of edit operations required to convert the first i characters of string A into the first j characters of string B. Then the edit distance D(m,n) is calculated by the following recursive formula:

[0026] 1) Initial conditions:

[0027] D(i,0)=ii=0,1,...,m;

[0028] D(0,j)=jj=0,1,...,m;

[0029] 2) Recursive relationship:

[0030]

[0031] Among them, when min(i,j)=0, D(i,j)=max(i,j); when min(i,j)≠0, D(i-1,j)+1 represents a deletion operation, D(i,j-1)+1 represents an insertion operation, and D(i-1,j-1)+cost(A[i-1],B[j-1]) represents a replacement operation;

[0032] That is, if the last character is different, a replacement operation is required, with a cost of 1; if the last character of string A is deleted, the edit distance is equal to D(i-1,j) plus 1; if a character is inserted at the end of string A, the edit distance is equal to D(i,j-1) plus 1.

[0033] The beneficial effects produced by the present invention are:

[0034] The present invention uses a word vector model for data matching and fusion, which can efficiently process multi-source heterogeneous data, reduce manual intervention, and improve data integration efficiency.

[0035] The present invention ensures the consistency and accuracy of the integrated data through normalization processing and similarity calculation, and avoids information loss and errors caused by data heterogeneity. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0037] Figure 1is a method flow chart of an embodiment of the present invention;

[0038] Figure 2 is a schematic diagram of deleting a character according to an embodiment of the present invention;

[0039] Figure 3 is a schematic diagram of a character inserting operation according to an embodiment of the present invention;

[0040] Figure 4 4 is a schematic diagram of a character replacement operation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0042] like Figure 1 As shown, a method for integrating multi-source ship data based on the Word2Vec model includes the following steps:

[0043] 1) Collect ship spare parts data from different sources, including spare parts name, specification model, delivery time, spare parts code, demand quantity, and delivery unit information;

[0044] 2) Preprocessing the collected ship spare parts data to obtain preprocessed multi-source ship spare parts data;

[0045] Preprocessing includes data cleaning, format conversion and normalization. English spare parts names are converted into Chinese, English letters are deleted, and missing values ​​and abnormal data are eliminated.

[0046] 3) Arrange the pre-processed multi-source ship spare parts data, combine it with the ship database standard, form a standard ship target database, and extract the spare parts name information in the ship target database;

[0047] 4) Construct word vector model Word2Vec;

[0048] Extract the spare parts name information in the ship target database in step 3), construct a ship spare parts word source, and construct a word vector model based on the standardized ship spare parts word source;

[0049] The names of ship spare parts are extracted from spare parts data from different sources, and the entries are processed by text extraction and traditional Chinese simplification. The appropriate window size and vector dimension are selected, and the vector representation of each word is obtained through training.

[0050] 5) Extract the names of the ship spare parts that need to be compared from the spare parts data from different sources and compare them with the spare parts names in the target database. Generate word vectors for the spare parts names through the word vector model Word2Vec and calculate the cosine similarity;

[0051] The calculation of cosine similarity is as follows:

[0052] 5.1) Determine the keywords in the spare parts names that need to be compared based on the names of the ship spare parts extracted from the spare parts data from different sources;

[0053] 5.2) Extract several keywords from the spare parts names in the target database and merge them into a set;

[0054] 5.3) Find the keywords in the spare part name that needs to be compared and the word vector of the spare part corresponding to a keyword in the set in the pre-trained Word2Vec model in step 4);

[0055] 5.4) Calculate the cosine similarity of two vectors; the range of cosine similarity value is [-1,1], -1 means completely dissimilar, 1 means completely similar, and the larger the value, the more similar it is;

[0056]

[0057] Among them, cos(θ) represents the cosine similarity between spare part vector A and spare part vector B, A i and B i Represent the keyword components of spare parts vectors A and B respectively;

[0058] 5.5) Traverse the spare part names in the target database to obtain the cosine similarity between the spare part name to be compared and the spare part name in the target database;

[0059] 6) Select the spare part name to be compared and the spare part name in the target database whose cosine similarity is above the set threshold, calculate the edit distance between their specification model, spare part code and outbound unit information, and directly merge them when the edit distance is 0; when the edit distance is less than the set value, send it to manual comparison to determine whether to merge, and finally generate a unified data representation; when the edit distance is large, skip the spare part and form a new spare part data.

[0060] Update the ship target database based on the fusion results

[0061] The edit distance refers to the minimum number of editing operations required to convert two character strings from one to another, where the editing operations include inserting characters, deleting characters, and replacing characters.

[0062] Before the edit distance calculation, text preprocessing is included: converting the specification model, spare part code, and delivery unit information into lowercase, removing punctuation marks, removing redundant spaces, removing stop words, and normalizing the text;

[0063] The edit distance is calculated as follows:

[0064] Assume that the lengths of characters A and B are m and n respectively, and define D(i,j) as the minimum number of edit operations required to convert the first i characters of string A into the first j characters of string B. Then the edit distance D(m,n) is calculated by the following recursive formula:

[0065] 1) Initial conditions:

[0066] D(i,0)=ii=0,1,...,m;

[0067] D(0,j)=jj=0,1,...,m;

[0068] 2) Recursive relationship:

[0069]

[0070] Among them, when min(i,j)=0, D(i,j)=max(i,j); when min(i,j)≠0, D(i-1,j)+1 represents a deletion operation, D(i,j-1)+1 represents an insertion operation, and D(i-1,j-1)+cost(A[i-1],B[j-1]) represents a replacement operation; Figures 2 to 4 As shown;

[0071] That is, if the last character is different, a replacement operation is required, with a cost of 1; if the last character of string A is deleted, the edit distance is equal to D(i-1,j) plus 1; if a character is inserted at the end of string A, the edit distance is equal to D(i,j-1) plus 1.

[0072] A specific embodiment;

[0073] By collecting spare parts from multiple distribution plants of a certain shipyard and summarizing them, the cosine similarity of the spare parts name and the edit distance of information such as specification model and spare parts code of each spare part are calculated based on the word vector model trained by Word2Vec, and the spare parts to be matched are matched and sorted with the spare parts in the target database.

[0074] (1) Collect and organize multi-source ship spare parts data: Collect and organize ship spare parts data from different sources. Specific information includes but is not limited to spare parts name, specification model, delivery time, spare parts code, demand, delivery unit, etc. Ensure the diversity and breadth of data sources to ensure the comprehensiveness of subsequent processing and analysis.

[0075] (2) Data preprocessing: The collected data is preprocessed comprehensively, including data cleaning, format conversion, and normalization. All English spare parts names must be translated into Chinese, and all English letters must be deleted. In addition, missing values ​​and abnormal data in the data are deleted to ensure the integrity and consistency of the data. Some information is shown in Table 1.

[0076] Table 1 Spare parts information

[0077]

[0078] (3) Arrange the preprocessed data: Arrange the preprocessed multi-source ship spare parts data. Combined with the standards of the ship database, a unified and standardized ship target database is formed, and spare parts name information is extracted from it, mainly including: sealing rings, sealing rings, plates, plugs, and bolts.

[0079] (4) Build a Word2Vec model: extract the names of ship spare parts from spare parts databases from different sources, perform text extraction and traditional Chinese character simplification on the entries, select the appropriate window size and vector dimension, and obtain the vector representation of each word through training. Ensure that the model can accurately reflect the lexical relationship in the Chinese context and provide a reliable basis for subsequent similarity calculations.

[0080] (5) Cosine similarity calculation based on the Word2Vec model: The sealing ring is taken as the target spare part, and the sealing ring, sealing ring, board, plug and bolt are taken as the comparison spare part information. The cosine similarity is calculated based on the Word2Vec model as shown in Table 2, where the sealing ring, sealing ring and bolt have a cosine similarity greater than 0.8.

[0081] Table 2 Calculation results based on Word2Vec cosine similarity

[0082] Spare part name Cosine similarity Sealing ring 1.00 Sealing ring 0.90 Board 0.55 Screw plug 0.73 bolt 0.85

[0083] (6) Calculation of edit distance of representative information: Select the spare parts to be processed (seals, sealing rings and bolts) and the target spare parts (seals) with a cosine similarity of more than 0.8, convert the specifications, models, spare parts codes, and shipping units into lowercase, remove punctuation, remove extra spaces, remove stop words, and normalize the text, and calculate the edit distance between the representative information such as specifications, models, spare parts codes, and shipping units. As shown in Table 3, the edit distance between the seal and the sealing ring is 0, indicating that they are the same spare parts; the edit distance between the seal and the bolt is 20, which is manually identified as different spare parts.

[0084] Table 3 Edit distance between spare parts with cosine similarity greater than 0.8

[0085] Spare part name Sealing ring Sealing ring bolt Sealing ring 0 0 20 Sealing ring 0 0 20 bolt 20 20 0

[0086] (7) Data verification and update: Perform comprehensive manual verification of the integrated data to ensure data accuracy and consistency. Perform regular data updates and maintenance based on actual needs to ensure that the database is always up to date and accurate.

[0087] Through the above steps, the present invention realizes the efficient integration of multi-source heterogeneous ship spare parts data, and provides accurate and reliable data support for ship maintenance and supply chain management.

[0088] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A method for integrating multi-source ship data based on the Word2Vec model, characterized in that: The following steps are involved: 1) Collect ship spare parts data from different sources, including spare parts name, specification model, delivery time, spare parts code, demand quantity, and delivery unit information; 2) Preprocessing the collected ship spare parts data to obtain preprocessed multi-source ship spare parts data; 3) Arrange the pre-processed multi-source ship spare parts data, combine it with the ship database standard, form a standard ship target database, and extract the spare parts name information in the ship target database; 4) Construct word vector model Word2Vec; Extract the spare parts name information in the ship target database in step 3), construct a ship spare parts word source, and construct a word vector model based on the standardized ship spare parts word source; 5) Extract the names of ship spare parts that need to be compared from spare parts data from different sources and compare them with the spare parts names in the target database. Generate word vectors for the spare parts names through the word vector model Word2Vec and calculate the cosine similarity; 6) Select the spare part names to be compared and the spare part names in the target database whose cosine similarity is above the set threshold, calculate the edit distance between their specifications, spare part codes and the information of the outgoing unit, and directly merge them when the edit distance is 0; when the edit distance is less than the set value, send it to manual comparison to determine whether to merge, and finally generate a unified data representation; The edit distance refers to the minimum number of edit operations required to convert one string into another.

2. The method for integrating multi-source ship data based on the Word2Vec model according to claim 1, characterized in that: In step 2), the preprocessing includes data cleaning, format conversion and normalization processing, converting the English spare part names into Chinese, deleting English letters, and eliminating missing values ​​and abnormal data in the data.

3. The method for integrating multi-source ship data based on the Word2Vec model according to claim 1, characterized in that: In step 5), the cosine similarity is calculated as follows: 5.1) Determine the keywords in the spare parts names that need to be compared based on the names of the ship spare parts extracted from the spare parts data from different sources; 5.2) Extract several keywords from the spare parts names in the target database and merge them into a set; 5.3) Find the keywords in the spare part name that needs to be compared and the word vector of the spare part corresponding to a keyword in the set in the pre-trained Word2Vec model in step 4); 5.4) Calculate the cosine similarity of two vectors; 5.5) Traverse the spare part names in the target database to obtain the cosine similarity between the spare part name to be compared and the spare part name in the target database.

4. The method for integrating multi-source ship data based on the Word2Vec model according to claim 3 is characterized in that: In step 5.4), the cosine similarity of the two vectors is calculated Among them, cos(θ) represents the cosine similarity between spare part vector A and spare part vector B, A i and B i Represent the keyword components of spare parts vectors A and B respectively.

5. The method for integrating multi-source ship data based on the Word2Vec model according to claim 1, characterized in that: In the step 6), the text preprocessing is included before the edit distance calculation: converting the specification model, spare part code, and shipping unit information into lowercase, removing punctuation marks, removing redundant spaces, removing stop words, and normalizing the text.

6. The method for integrating multi-source ship data based on the Word2Vec model according to claim 1, characterized in that: Editing operations include inserting characters, deleting characters, and replacing characters.

7. The method for integrating multi-source ship data based on the Word2Vec model according to claim 6 is characterized in that: In step 6), the edit distance is calculated as follows: Assume that the lengths of characters A and B are m and n respectively, and define D(i,j) as the minimum number of edit operations required to convert the first i characters of string A into the first j characters of string B. Then the edit distance D(m,n) is calculated by the following recursive formula: 1) Initial conditions: D(i,0)=ii=0,1,...,m; D(0,j)=jj=0,1,...,m; 2) Recursive relationship: Among them, when min(i,j)=0, D(i,j)=max(i,j); when min(i,j)≠0, D(i-1,j)+1 represents a deletion operation, D(i,j-1)+1 represents an insertion operation, and D(i-1,j-1)+cost(A[i-1],B[j-1]) represents a replacement operation.

8. An electronic device, characterized in that: include: one or more processors; as well as a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images