Traffic technology data indexing method and system

Abstract

This invention discloses a method and system for indexing traffic technology data. The method includes: loading data to be indexed; determining the data type, converting the data type according to a standard database structure, and importing it into a processing database; performing word segmentation on the imported data to obtain N first keywords; and automatically indexing based on M first keywords; where M and N are positive integers, and M≤N. This method and system for indexing traffic technology data can quickly, efficiently, and accurately assist in the indexing of traffic technology data. By segmenting, identifying, and extracting keywords and classification numbers from large volumes of traffic technology data, it can automatically index or assist manual indexing personnel, significantly reducing the workload and intensity of manual indexing.

Description

Technical Field

[0001] This invention relates to a data indexing method, and more particularly to a data indexing method for transportation technology. This invention also relates to... Background Technology

[0002] A key step in establishing a document data warehouse is document indexing, and the quality of indexing directly affects the efficiency of computer retrieval. Currently, each document database has dedicated indexing personnel, meaning that manual indexing generally requires a large number of people. However, due to the low efficiency and error-prone nature of manual indexing, the current trend is to gradually transition from manual indexing to automated machine indexing.

[0003] Generally, the construction of a data warehouse involves more than just the direct use of existing data. It focuses on the integration, summarization, and clustering of data from multiple sources, enabling the data to be thematically and tagged within a unified framework. This allows for the uncovering of hidden value within scientific and technological data, serving more intelligent and personalized needs. Extracting core categories and key points from large amounts of text data is a crucial aspect of system construction. Current data indexing systems still require significant manual intervention, resulting in low efficiency and accuracy.

[0004] Therefore, the aforementioned problems with existing technologies urgently need to be improved. Summary of the Invention

[0005] In view of the aforementioned problems with existing technologies, one objective of this invention is a method for indexing transportation technology data. This method can significantly reduce the workload of manual indexing and greatly improve data indexing efficiency.

[0006] To achieve the above objectives, one embodiment of the present invention provides a method for indexing traffic technology data, comprising:

[0007] Load the data that needs to be indexed;

[0008] Determine the data type, convert the data type according to the standard database structure, and import it into the processing database;

[0009] The data imported into the database is segmented into words to obtain N first keywords;

[0010] Automatic indexing is performed based on M of the first keywords;

[0011] Where M and N are positive integers, and M ≤ N.

[0012] Preferably, before automatic indexing, the N first keywords that have undergone word segmentation are clustered to obtain a clustered vocabulary. The product of the prior probability and the corresponding TF-IDF value of each word in the clustered vocabulary is calculated, and the results are sorted. The first M first keywords in the sequence are extracted and automatically indexed.

[0013] Preferably, M is a positive integer selected from 3 to 8.

[0014] Preferably, the method further includes:

[0015] Calculate the co-occurrence combinations of keywords, compare them with existing standard data, and extract the Chinese Library Classification number with the highest frequency.

[0016] Search the Chinese Library Classification Thesaurus and extract the classification numbers of valid matching data;

[0017] The classification number data of the preset professional terminology is added to the cluster terminology as the standard point. The combination feature vector of each keyword is calculated, and the standard point that is closest to it is calculated. The classification number of the standard point data is extracted as the subject similarity classification number.

[0018] Use the China Library Classification-Subject Classification Number Mapping Table to obtain the subject mapping classification number;

[0019] The intersection of the subject similarity classification number and the subject mapping classification number is taken as the subject classification number result.

[0020] As a preferred method, the following priority is applied when calculating subject classification numbers:

[0021] Search the subject-specific thesaurus and extract the classification numbers of valid matching data;

[0022] Add the classification numbers of the valid matching data from the professional thesaurus to the clustering thesaurus.

[0023] Another aspect of the present invention provides a traffic technology data indexing system, the system comprising:

[0024] A data input unit configured to load data that needs to be indexed;

[0025] A data processing unit is configured to determine the data type of the data, convert the data type according to a standard database structure and import it into a processing database; perform word segmentation on the imported data in the processing database to obtain N first keywords;

[0026] The data indexing unit performs automatic indexing based on M of the first keywords; where M and N are positive integers, and M≤N.

[0027] Preferably, the data processing unit is further configured to: before automatic indexing, cluster the N first keywords that have undergone word segmentation to obtain a clustered word list, calculate the product of the prior probability and the corresponding TF-IDF value of each word in the clustered word list, sort according to the results, extract the first M first keywords in the sequence, and perform automatic indexing.

[0028] Preferably, M is a positive integer selected from 3 to 8.

[0029] Preferably, the data processing unit is further configured as follows:

[0030] Calculate the co-occurrence combinations of keywords, compare them with existing standard data, and extract the Chinese Library Classification number with the highest frequency.

[0031] Search the Chinese Library Classification Thesaurus and extract the classification numbers of valid matching data;

[0032] The classification number data of the preset professional terminology is added to the cluster terminology as the standard point. The combination feature vector of each keyword is calculated, and the standard point that is closest to it is calculated. The classification number of the standard point data is extracted as the subject similarity classification number.

[0033] Use the China Library Classification-Subject Classification Number Mapping Table to obtain the subject mapping classification number;

[0034] The intersection of the subject similarity classification number and the subject mapping classification number is taken as the subject classification number result.

[0035] As a preferred method, the following priority is applied when calculating subject classification numbers:

[0036] Search the subject-specific thesaurus and extract the classification numbers of valid matching data;

[0037] Add the classification numbers of the valid matching data from the professional thesaurus to the clustering thesaurus.

[0038] Compared with existing technologies, the traffic technology data indexing method and system of the present invention can quickly, efficiently and accurately assist in the indexing of traffic technology data. It can automatically index or assist manual indexing personnel by segmenting, identifying and extracting keywords and classification numbers from a large amount of traffic technology data, greatly reducing the workload and intensity of manual indexing.

[0039] It should be understood that the foregoing general description and the following detailed description are exemplary and illustrative only, and are not intended to limit this disclosure.

[0040] This application provides an overview of various implementations or examples of the technology described in this disclosure, and is not a full disclosure of the entire scope or all features of the disclosed technology. Attached Figure Description

[0041] Figure 1 This is a flowchart of the traffic technology data indexing method of the present invention.

[0042] Figure 2 This is a schematic diagram of the database structure of the traffic technology data indexing method of the present invention.

[0043] Figure 3 This is a structural block diagram of the word segmentation tool in the traffic technology data indexing method of the present invention.

[0044] Figure 4 This is a basic structural block diagram of the traffic technology data indexing system of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0046] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "comprising" or "including," and similar terms used in this disclosure, mean that an element or object preceding the term encompasses the elements or objects listed following the term and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but may also include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0047] To keep the following description of the embodiments of this disclosure clear and concise, detailed descriptions of known functions and known components are omitted.

[0048] like Figure 1 As shown, one embodiment of the present invention provides a method for indexing transportation technology data, including:

[0049] Load the data that needs to be indexed;

[0050] Determine the data type, convert the data type according to the standard database structure, and import it into the processing database;

[0051] The data imported into the database is segmented into words to obtain N first keywords;

[0052] Automatic indexing is performed based on M of the first keywords;

[0053] Where M and N are positive integers, and M ≤ N.

[0054] Preferably, before automatic indexing, the N first keywords that have undergone word segmentation are clustered to obtain a clustered vocabulary. The product of the prior probability and the corresponding TF-IDF value of each word in the clustered vocabulary is calculated, and the results are sorted. The first M first keywords in the sequence are extracted and automatically indexed.

[0055] Preferably, M is a positive integer selected from 3 to 8.

[0056] Preferably, the method further includes:

[0057] Calculate the co-occurrence combinations of keywords, compare them with existing standard data, and extract the Chinese Library Classification number with the highest frequency.

[0058] Search the Chinese Library Classification Thesaurus and extract the classification numbers of valid matching data;

[0059] The classification number data of the preset professional terminology is added to the cluster terminology as the standard point. The combination feature vector of each keyword is calculated, and the standard point that is closest to it is calculated. The classification number of the standard point data is extracted as the subject similarity classification number.

[0060] Use the China Library Classification-Subject Classification Number Mapping Table to obtain the subject mapping classification number;

[0061] The intersection of the subject similarity classification number and the subject mapping classification number is taken as the subject classification number result.

[0062] As a preferred method, the following priority is applied when calculating subject classification numbers:

[0063] Search the subject-specific thesaurus and extract the classification numbers of valid matching data;

[0064] Add the classification numbers of the valid matching data from the professional thesaurus to the clustering thesaurus.

[0065] The core data for the auxiliary indexing tool consists of four categories: scientific and technological personnel, scientific and technological projects, scientific and technological achievements, and scientific and technological literature. Therefore, the original data table is the basic information table for these four categories. Based on the original table, an indexing information table is generated for subsequent indexing processing. The indexing information is processed using a TF-IDF weight table and a classification number clustering table, and then integrated to generate keyword indexing result tables and classification number indexing result tables.

[0066] (1) The main tables for scientific and technological personnel, scientific and technological projects, scientific and technological achievements, and scientific and technological literature are the original data, and their structure is consistent with the existing data storage structure of the company's data warehouse. When some scientific and technological personnel information is missing, relevant scientific and technological literature data can be used to appropriately fill it in;

[0067] (2) The automatic indexing process requires the support of a self-built thesaurus. Since the technical personnel are grouped for processing at the end, the more accurate and complete the self-built thesaurus is, the greater the improvement in the accuracy of the automatic indexing results.

[0068] (3) Establish an effective calling interface. In order to improve the effectiveness of data indexing and avoid excessive contact between core data and the outside world during the data indexing process, it is necessary to connect with the company's data warehouse data management platform. All related processed data are stored in the public storage area. During the connection process, it is necessary to unify the input and output data structures of the two parties and establish an appropriate data indexing processing request submission and data processing result return mechanism. When processing data, the internal data of the management platform is called to automatically calculate and monitor the data processing status, so that the data indexing results can be reflected in science and technology information-related products and services more quickly.

[0069] The purpose of developing the automated indexing system is to solve the problems of indexing quality and efficiency in the company's data warehouse. This system can significantly reduce the workload of manual indexing, greatly improve data indexing efficiency, and at the same time, ensure the accuracy of indexing results, while providing effective support for data development and processing personnel.

[0070] In the company's data warehouse basic data resources, data such as scientific and technological projects, scientific and technological personnel, and scientific and technological achievements in the system should be classified and indexed by category and keywords in accordance with classification standards such as the "Chinese Library Classification (Fourth Edition)" and the "Classification and Code of Subjects of the People's Republic of China (National Standard GB / T 13745-2009)" to achieve the requirements of standardized and scientific indexing, accurate indexing results, and high indexing reusability.

[0071] Based on the development tasks of the data processing and management software, assistance will be provided to three user groups: system administrators, data developers, and data processing personnel.

[0072] (1) System administrator, responsible for ensuring the overall operation of the system, including data import and processing, database management, system users, setting user roles, etc.

[0073] (2) Data developers can call relevant software interfaces to perform batch indexing of keywords and classification numbers for target data;

[0074] (3) Data processing personnel can use the UI to index the target data using keywords and classification numbers.

[0075] All staff involved in this project will receive confidentiality and security training organized by the tendering party, strictly abide by the "Confidentiality Law", and assume confidentiality obligations for the data content involved in the data collection, sorting, indexing, and association services, ensuring the security and integrity of the purchaser's data and materials, and not filming, reprinting, copying, or disclosing or publishing the purchaser's data content to the outside world in any way;

[0076] 1.3 Referenced Documents

[0077]

[0078] 1.4 Identification

[0079]

[0080]

[0081] 2. Tool Overview

[0082] 2.1 Construction Goals

[0083] Establish a fast, efficient, and accurate scientific and technological data indexing system to automatically identify and extract related terms and classification numbers from large volumes of scientific and technological data. This system will assist manual indexing personnel in their indexing work, greatly reducing the workload and intensity of manual indexing.

[0084] It supports statistical analysis of various indexing results. When there are no high requirements for classification accuracy, it supports fully automatic extraction of related words and classification numbers from large batches of scientific and technological data, providing data support for functions such as user profiling and intelligent recommendation.

[0085] 2.2 Scope and Content of Construction

[0086] The construction of auxiliary indexing tools includes thesaurus construction, Chinese word segmentation, keyword extraction, and classification number calculation.

[0087] 2.2.1 Vocabulary Construction

[0088] The thesaurus data includes three categories: self-built thesaurus, professional thesaurus, and mapped thesaurus.

[0089] Building a self-built thesaurus involves screening core journal articles and extracting effective data containing titles, abstracts, keywords, and Chinese Library Classification numbers. The keywords are then statistically analyzed, and multiple parameters such as character count, frequency, dominant frequency, standard fit, classification aggregation degree, and time trend are calculated.

[0090] Each parameter is standardized using the sigmoid function, and the parameters are determined based on the specific data distribution to ensure that the median standardized value is around 0.5. Weights are assigned to each standardized parameter, and the goodness of fit with the original keywords is calculated. The optimal weight allocation scheme is then calculated using gradient descent. This optimal weight allocation is used as the prior probability of each word in subsequent calculations.

[0091] The professional thesaurus is divided into:

[0092] Table of Correspondence between Mechanical Industry Classification Numbers and Keywords

[0093] Classification Numbers and Keywords for Radio Electronics and Telecommunications Technology

[0094] Table of Correspondence between Electrical Engineering Classification Numbers and Keywords

[0095] Classification number and keyword correspondence table for metallurgy and metalworking

[0096] Correspondence table of building science classification numbers and keywords

[0097] Classification Number and Keyword Correspondence Table for Environmental Science and Safety Science

[0098] Engineering and technical classification numbers and keywords correspondence table

[0099] Classification Numbers and Keywords for Automation Technology and Computer Technology

[0100] Correspondence table of economic classification numbers and keywords

[0101] Classification numbers and keywords for light industry and handicrafts

[0102] Table of Classification Numbers and Keywords for Water Conservancy Projects

[0103] Medical Classification Number and Keyword Correspondence Table

[0104] Agricultural classification number and keyword correspondence table

[0105] Chemical Industry Classification Number and Keyword Correspondence Table

[0106] This part of the vocabulary needs to be compiled.

[0107] The mapping terminology is divided into three parts: a terminology-China Library Classification mapping terminology, a terminology-subject classification mapping terminology, and a China Library Classification-subject classification number mapping terminology. This part of the terminology needs to be compiled.

[0108] 2.2.2 Chinese word segmentation

[0109] The indexing data is segmented using IKanalyzer, and the segmentation results are entered into the database. The segmentation mode employs intelligent segmentation, which intelligently extracts possible combinations of words from the metadata.

[0110] Meanwhile, the word segmenter can be plugged in with a pre-compiled professional thesaurus as high-priority available information during the word segmentation process.

[0111] 2.2.3 Keyword Indexing

[0112] Automatically index data related to technology projects, personnel, and achievements in the company's data warehouse using keywords. Supports customizable keyword counts, with multiple keywords separated by semicolons.

[0113] Before keyword indexing, the self-built thesaurus needs to be clustered. After iterating through the word segmentation results of the collected documents and excluding invalid words (less than 2 consecutive Chinese characters or less than 4 consecutive English characters), the remaining word segmentation results are clustered using the word2vec open-source package. The core parameters such as size, min_count, window, and n_cluster are determined based on system resources and data volume to obtain the final clustered thesaurus. The natural language corpus used in the word vector process is in txt format, with a space as the delimiter; the data used for clustering is in csv format, ensuring that the processed keywords do not contain commas or semi-commas.

[0114] By calculating the TF-IDF value of each word in the word segmentation results and incorporating the prior probabilities from the vocabulary into the Bayesian formula, and since the keyword extraction values ​​for the same data segment can be considered constant, the product of the prior probability of each word and its corresponding TF-IDF value can be directly calculated. The word segmentation results are then sorted, and the top 3-8 words are extracted as keywords.

[0115] 2.2.1 Classification number indexing

[0116] In accordance with classification standards such as the "Chinese Library Classification (Fourth Edition)" and the "Classification and Codes of Subjects of the People's Republic of China (National Standard GB / T 13745-2009)," the data of scientific and technological projects, scientific and technological personnel, and scientific and technological achievements in the company's data warehouse are automatically classified and indexed, and the number of custom classification numbers is supported.

[0117] When calculating the Chinese Library Classification number, the following priority levels are used:

[0118] a) Calculate the co-occurrence combinations in the keywords, compare them with existing standard data, and extract the most frequent Chinese Library Classification number (data without subject classification numbers) as the result.

[0119] b) Search the Chinese Library Classification Thesaurus and extract the classification numbers of valid matching data.

[0120] c) Add the data from the professional terminology list as standard points to the cluster terminology list, calculate the combined feature vector of each keyword, calculate the standard point that is closest to it, and extract the classification number of the standard point data.

[0121] a) When calculating the subject classification number, the following priority is applied.

[0122] b) Search the subject-specific thesaurus and extract the classification numbers of valid matching data.

[0123] c) Add the data from the professional terminology list as standard points to the clustering terminology list, calculate the combined feature vector of each keyword, and calculate the standard point that is closest to it. Extract the classification number of the standard point data as the subject similarity classification number. At the same time, use the China Library Classification Number Mapping Table to obtain the subject mapping classification number. Take the intersection of the two classification results as the subject classification number result.

[0124] 2.2.3 Classification Number Mapping

[0125] Based on the mapping relationship between the Chinese Library Classification (CLC) and other classification standards provided by the purchaser, the default automatically indexed CLC number is mapped to the target classification number.

[0126] 3 Overall Design

[0127] 3.1 Project Deployment Design

[0128] 3.5.1 Hardware Environment

[0129] Windows 7 / 10 system, 32GB or more of RAM, and 2TB or more of storage space;

[0130] 3.5.2 Software Environment

[0131] For SQL Server 2008 R2, you need to enable environment configurations such as advanced options, Ole Automation Procedures, CMD_SHEEL, and clr enabled.

[0132] MySQL 5.7

[0133] Python 3 requires the inclusion of related packages such as pyodbc, pymysql, numpy, pandas, and sklearn.

[0134] Navicat or a related database management tool.

[0135] 3.5.3 Network Environment

[0136] Ensure that TCP / IP ports 1433 and 3306 are open.

[0137] 3.2 Functional Structure Design

[0138] Establishing a stable, efficient, and accurate auxiliary indexing system for automatic indexing of the company's data warehouse data improves data processing efficiency and reduces data maintenance cycles; it also reduces the workload of indexing staff, assisting them in better completing indexing tasks; and it provides indexing data support for the company's data warehouse data, enabling effective classification of data and providing strong data support. Based on the above standards, this tool will have the following functions:

[0139]

[0140] 3.3 Performance Design

[0141] For the company's data warehouse literature data, the indexing efficiency is approximately 100,000 records / day, and for the company's data warehouse non-literature data, the indexing efficiency is approximately 50,000 records / day.

[0142] 4 System Functions

[0143] 4.1 Common UI Components

[0144] This tool provides auxiliary indexing services as a way to access the data management platform. All processing is performed in the background and does not involve any front-end operations, so there are no public UI components.

[0145] 4.2 Public Data Operations

[0146] 1) Overview

[0147] The auxiliary indexing tool is directly invoked by the management platform; the tool itself does not store the basic data required for indexing. During auxiliary indexing processing, the indexing tool retrieves the corresponding data from the public storage area to the appropriate processing library for subsequent indexing. After the indexing processing is complete, the result data is also written to the designated location in the corresponding public storage area for use by the management platform.

[0148] 2) Business Logic

[0149] The auxiliary indexing service primarily performs indexing processing on data from technical personnel. During the indexing calculation process, the data used by the indexing tool in the public storage area varies depending on the processing request. The following is a description of the data required during the indexing process:

[0150] The parameters specify the data to be processed, the basic information table of scientific and technical personnel, and the resume table of scientific and technical personnel (which can be empty).

[0151] Upon receiving an indexing request, the tool will invoke the corresponding processing module based on the request type, simultaneously reading the data to be processed and related basic data tables for indexing. Due to the lengthy indexing cycle, a "processing in progress" signal will be returned to the calling interface upon successful processing. During indexing, the tool will track the processing progress and status, recording it in the tool's runtime log. Administrators can view this log to track the system's processing status. After processing is complete, the tool will automatically write the indexing results back to a designated location in the public storage area.

[0152] 3) Public Data Management

[0153] The data used in indexing tools is divided into basic data, calculated data, process data, log data, and result data.

[0154] The foundational data consists of various metadata required for processing. This data is stored in a public storage area and is updated and maintained by the management platform. The indexing tool does not involve the maintenance and management of this data.

[0155] The calculated data is the data that the indexing process depends on, including data processing rules, mapping dictionaries, etc. This part of the data is stored in the indexing calculation library and does not require management or maintenance during use. When the tool is upgraded, we will perform the version upgrade.

[0156] Process data refers to temporary data generated during indexing. This data has little value and consumes significant system resources. This data is stored in a computational database and is automatically cleaned up by the indexing tool after each indexing process.

[0157] Log data records the indexing process, allowing users to view the current indexing status and processing history. This data is stored in the indexing calculation library and requires no management or maintenance during use; it is automatically updated during tool operation.

[0158] The result data represents the indexing process. This data is stored in a public storage area, and its version management and maintenance are the responsibility of the control platform. The indexing tool does not handle the maintenance and management of this data.

[0159] 4.3 Inputs and Outputs

[0160] 4) Overview

[0161] The auxiliary indexing tool is directly invoked by the management platform. A unified standard is needed for various aspects of data integration; otherwise, it will negatively impact the stability of the processing. The specific requirements will be explained in several aspects below.

[0162] 5) Business Logic

[0163] The management platform invokes the auxiliary indexing tool, submitting relevant data processing parameters. The tool then calls the background processing script to index the data, updating the processing log table according to the progress and returning the current data processing progress. The auxiliary indexing tool primarily processes personnel, project, and outcome data. It can also process other types of data that meet the data requirements, but their data structure and content must be similar to the three types of resources. Therefore, a unified standard for the data structure requiring indexing is essential to maximize the completeness and accuracy of the indexing results during calculation.

[0164] 6) Input Data Structure

[0165] The API calls control the execution of the background processing script through parameters, and call the relevant data through MySQL ODBC, which avoids the risk of transmitting data from the API and improves the processing efficiency to a certain extent.

[0166] The parameters are explained below:

[0167]

[0168] 2. The Type type is as follows:

[0169]

[0170]

[0171] 5) Processing procedure

[0172] The system imports the data to be processed using MySQL ODBC, performs structural validity checks on the data submitted by the purchaser that requires indexing, confirms the specific quantity of each data item, the fill rate of each field, and other information, and provides feedback on any problematic or questionable data to ensure that the data structure used by both parties is consistent and that the data quality basically meets the requirements. After system review, subsequent indexing processing is automatically performed.

[0173] Indexing can be divided into three parts: data parsing layer, data logic processing layer, and data management layer.

[0174] The parsing layer is responsible for analyzing and standardizing the data.

[0175] A preliminary analysis is conducted on the structure and completeness of the received data, and some abnormal data is standardized according to the agreed data structure. Without modifying the original data, the standardization process is used to make the data have a unified standard.

[0176] For the standardized data, the word segmentation module is called to perform word segmentation, and the processing results are stored in the indexing processing database.

[0177] The logical layer is the core module for indexing processing. Its function is to perform keyword annotation on the company's data warehouse data by comprehensively weighting the words in the basic thesaurus and custom thesaurus with the words related to the data to be processed. It then performs classification number annotation on the clustering results of the labeled keywords and maps them to specified categories according to requirements using a classification number mapping table.

[0178] The logic layer indexes the received data and obtains the processing results. For data with the same ID, historical processing results are directly extracted, avoiding repeated calculations to improve overall indexing efficiency.

[0179] The management layer's functions are divided into data export and data processing log management. The data export function allows exporting processed indexing data results to the target data source location. After integration with the data management platform, customized exports can also be performed according to different needs. Log management primarily records the historical processing status of indexing data, while also recording a small amount of abnormal data, retaining processing logs to facilitate future optimization and maintenance of the indexing results by system administrators.

[0180] 6) Output data structure

[0181]

[0182]

[0183] 4.4 Log

[0184] 1) Overview

[0185] The auxiliary indexing log is divided into the processing log and the indexing result log.

[0186] The processing log is used to record the running status of the auxiliary indexing tool. Users can view the current running status of the tool through the processing log, and also view the historical records. When an exception occurs during the processing, it will be recorded in the log, which will facilitate the optimization of the tool's processing status.

[0187] The indexing results log records the specific processing details of the auxiliary indexing tools. It records information such as the number of indexed records and the number of valid indexed records. If you need to generate auxiliary indexing-related statistical reports, you can view them through the indexing results log.

[0188] 2) Applicable users

[0189] Data management platform administrator.

[0190] 3) Input data structure

[0191] Logs are automatically loaded during the indexing process based on the indexing tool parameters and specific processing details; log generation does not involve external parameters. Logs can be viewed through the relevant methods in the API.

[0192] 4) Processing procedure

[0193] The logs automatically obtain information such as the current time, parameters used by the indexing tool, current execution progress, various statistical indicators of metadata, and various statistical indicators of processing results, and automatically compile the relevant log tables without manual intervention.

[0194] 5) Output data structure

[0195] Log processing:

[0196]

[0197]

[0198] Indexing results log:

[0199]

[0200] 6) Physical storage

[0201] Log data will be stored in the data indexing processing database, and can also be accessed or exported through relevant methods.

[0202] 5 Business Functions

[0203] 5.1 Full Data Indexing Processing

[0204] 1) Overview

[0205] Full data indexing is used relatively infrequently. This function indexes all target data as agreed upon. Since indexing information is mostly descriptive text, modifications are minimal, resulting in little change to the corresponding indexing results, and the processing cycle is relatively long.

[0206] 2) Restrictions

[0207] When indexing data on all scientific and technological personnel, projects, and achievements in the database, it is necessary to ensure that the data used and the scientific and technological literature data are not in the process of being updated.

[0208] 3) Applicable Scenarios

[0209] It is recommended to index all data only when there are large changes in the content of the data, or when there are changes to the basic thesaurus, custom thesaurus, or related indexing algorithms on which the indexing depends.

[0210] 4) Business Logic

[0211] Load all data requiring indexing into the indexing processing database. After structural integration and classification, sequentially call the corresponding data indexing tools for calculation and return the results. Simultaneously, record the processing status through a log. This business logic is equivalent to sequentially calling the indexing processing programs corresponding to personnel, projects, and results.

[0212] 5) Input Data Structure

[0213] The API calls control the execution of the background processing script through parameters, and call the relevant data through MySQL ODBC, which avoids the risk of transmitting data from the API and improves the processing efficiency to a certain extent.

[0214] The parameters are explained below:

[0215] parameter Parameter type illustrate DataBase string Tbname string Type int 9

[0216] 6) Input data field restrictions and generation rules

[0217] When the parameter "Type" is 0, no other parameters are retrieved, and the full amount of scientific and technological personnel, project, and achievement data at the agreed location is directly loaded and the indexing results are calculated.

[0218] 7) Processing procedure

[0219] The system loads the full dataset from MySQL ODBC, invokes the automatic indexing module to index it, and then writes the resulting data to the specified database.

[0220] 8) Output data structure

[0221]

[0222] 9) Physical storage

[0223] The indexing process has a relatively long overall cycle, so instead of directly returning the processing result data, it returns the current progress status of the processing in the log. Once processing is complete, the data will be written directly back to the specified database.

[0224] 5.2 Personnel (Designated / Undesignated) Data Indexing Processing

[0225] 1) Overview

[0226] The personnel data indexing tool is specifically designed for indexing data of scientific and technical personnel. When a data source is specified, only a designated portion of the data is indexed. When no data source is specified, the entire set of personnel data is loaded for indexing.

[0227] 2) Restrictions

[0228] When performing personnel data indexing, it is necessary to ensure that the target data and scientific literature data are not being updated.

[0229] 3) Applicable Scenarios

[0230] The indexing of designated personnel data is relatively flexible and can be called when only a small amount of personnel data needs to be indexed. Indexing all personnel data is time-consuming and is recommended only when there are large changes in the data content, or when there are modifications to the underlying thesaurus, custom thesaurus, or related indexing algorithms.

[0231] 4) Business Logic

[0232] Load all data that needs to be indexed into the indexing processing database. After the data is structurally integrated, call the personnel data automatic indexing module to perform calculations and return the results. At the same time, record the processing status through logs.

[0233] 5) Input Data Structure

[0234] The API calls control the execution of the background processing script through parameters, and call the relevant data through MySQL ODBC, which avoids the risk of transmitting data from the API and improves the processing efficiency to a certain extent.

[0235] The parameters are explained below:

[0236] parameter Parameter type illustrate DataBase string Tbname string Type int 9

[0237] 6) Input data field restrictions and generation rules

[0238] When the parameter "Type" is 1, no other parameters are retrieved, and the full amount of scientific and technical personnel data at the agreed location is directly loaded and the indexing results are calculated.

[0239] When the parameter "Type" is 2, the corresponding personnel data is loaded from the database table at the specified location and the data structure is checked. After the data legality is verified, the data is loaded and the indexing result is calculated.

[0240] 7) Processing procedure

[0241] The system loads personnel data from MySQL ODBC, calls the automatic personnel indexing module to index the data, and then writes the results to the specified database.

[0242] 8) Output data structure

[0243]

[0244]

[0245] 9) Physical storage

[0246] The indexing process has a relatively long overall cycle, so instead of directly returning the processing result data, it returns the current progress status of the processing in the log. Once processing is complete, the data will be written directly back to the specified database.

[0247] 5.3 Project (Specified / Unspecified) Data Indexing Processing

[0248] 1) Overview

[0249] The indexing tool for project data is specifically designed for indexing scientific and technological project data. When a data source is specified, only a designated portion of the data is indexed. When no data source is specified, the entire project's data is loaded for indexing.

[0250] 2) Restrictions

[0251] When performing project data indexing, it is necessary to ensure that the target data and scientific literature data are not being updated.

[0252] 3) Applicable Scenarios

[0253] Specific project data indexing is relatively flexible and can be called when only a small amount of project data needs to be indexed. Full project data indexing is time-consuming and is recommended only when there are large changes in the content of the data, or when there are changes to the underlying thesaurus, custom thesaurus, or related indexing algorithms.

[0254] 4) Business Logic

[0255] Load all data that needs to be indexed into the indexing processing database. After the data is structurally integrated, call the project data automatic indexing module to perform calculations and return the results. At the same time, record the processing status through logs.

[0256] 5) Input Data Structure

[0257] The API calls control the execution of the background processing script through parameters, and call the relevant data through MySQL ODBC, which avoids the risk of transmitting data from the API and improves the processing efficiency to a certain extent.

[0258] The parameters are explained below:

[0259]

[0260]

[0261] 9) Physical storage

[0262] The indexing process has a relatively long overall cycle, so instead of directly returning the processing result data, it returns the current progress status of the processing in the log. Once processing is complete, the data will be written directly back to the specified database.

[0263] 5.4 Output (Specified / Unspecified) Data Indexing Processing

[0264] 1) Overview

[0265] The indexing tool targets the indexing module of scientific and technological achievement data during the data indexing process. When a data source is specified, only a specified portion of the data is indexed. When no data source is specified, the entire dataset of the achievement is loaded for indexing.

[0266] 2) Restrictions

[0267] When indexing research data, it is necessary to ensure that the target data and scientific literature data are not being updated.

[0268] 3) Applicable Scenarios

[0269] The specified output data indexing process is relatively flexible and can be called when only a small amount of output data needs to be indexed. Indexing the full output data is time-consuming, and it is recommended to only perform full output data indexing when there are large changes in the data content, or when there are modifications to the underlying thesaurus, custom thesaurus, or related indexing algorithms.

[0270] 4) Business Logic

[0271] Load all data that needs to be indexed into the indexing processing database. After the data is structurally integrated, call the automatic indexing module for results to perform calculations and return the results. At the same time, record the processing status through a log.

[0272] 5) Input Data Structure

[0273] The API calls control the execution of the background processing script through parameters, and call the relevant data through MySQL ODBC, which avoids the risk of transmitting data from the API and improves the processing efficiency to a certain extent.

[0274] The parameters are explained below:

[0275] parameter Parameter type illustrate DataBase string Tbname string Type int 9

[0276] 6) Input data field restrictions and generation rules

[0277] When the parameter "Type" is 5, no other parameters are retrieved, and the full amount of scientific and technological achievement data at the agreed location is directly loaded and the indexing results are calculated.

[0278] When the parameter "Type" is 6, the corresponding result data is loaded from the database table at the specified location and the data structure is checked. After the data legality is verified, the data is loaded and the indexing result is calculated.

[0279] 7) Processing procedure

[0280] The system loads the output data read from MySQL ODBC, calls the automatic output indexing module to perform indexing processing, and then writes the result data to the specified database.

[0281] 8) Output data structure

[0282]

[0283] 9) Physical storage

[0284] The indexing process has a relatively long overall cycle, so instead of directly returning the processing result data, it returns the current progress status of the processing in the log. Once processing is complete, the data will be written directly back to the specified database.

[0285] 6-interface design

[0286] 6.1 External Interface

[0287] The `dataTrans()` method allows you to set parameters, and the program automatically loads the corresponding SQL processing script. Data processing is completed in SQL Server, and the result table is obtained. Simultaneously, the `queryDataExePro()` interface is used to query log information and provide real-time feedback on the processing status.

[0288] 6.2 Internal Interface

[0289] During indexing, a word segmentation tool interface is required. Here, IK Analyzer 2012 is used. IKAnalyzer is an open-source, lightweight Chinese word segmentation toolkit developed in Java. Since its initial release as version 1.0 in December 2006, IKAnalyzer has released four major versions. Initially, it was a Chinese word segmentation component based on the open-source project Lucene, combining dictionary-based segmentation and grammatical analysis algorithms. Starting with version 3.0, IK evolved into a public word segmentation component for Java, independent of the Lucene project, while providing a default optimized implementation for Lucene. In version 2012, IK implemented a simple word segmentation ambiguity removal algorithm, marking the evolution of the IK word segmenter from simple dictionary-based segmentation to simulated semantic segmentation.

[0290] 7 System Error Handling Design

[0291] 7.1 Interface call exception

[0292] When an API call fails, communication between the management platform and the API becomes obstructed. In this case, the front-end can directly learn about the problem and maintain the transmission between the two parties.

[0293] 7.2 Data source anomaly

[0294] However, if incorrect parameters or other reasons prevent the processing of target data, a message indicating no valid data will be thrown and recorded in the log table. Processing personnel can carefully check the accuracy of each parameter in the log information and adjust any erroneous parameters.

[0295] 7.3 Abnormalities in data processing

[0296] When the processing crashes due to uncontrollable reasons, the log table records the most recent processing progress that was executing normally. The log also periodically scans the processing status, and when it detects that processing has stopped, it marks the relevant log entries as having been terminated. Processors can use the logs to query relevant anomalies and contact the developers for tool maintenance.

[0297] 8 Database Design

[0298] 8.1 Database Logical Design

[0299] See Figure 2 As shown.

[0300] The core data for the auxiliary indexing tool consists of three categories: scientific and technological personnel, scientific and technological projects, and scientific and technological achievements. Therefore, the original data table is the basic information table for these three categories. Based on the original table, an indexing information table is generated for subsequent indexing processing. The indexing information is processed using a TF-IDF weight table and a classification number clustering table, and then indexed using keywords and classification numbers to generate indexing result tables.

[0301] (1) The main tables for scientific and technological personnel, scientific and technological projects, and scientific and technological achievements are the original data, and their structure is consistent with the existing data storage structure of the company's data warehouse. When some scientific and technological personnel information is missing, relevant scientific and technological literature data can be used to appropriately fill it in;

[0302] (2) The automatic indexing process requires the support of a self-built thesaurus. The more accurate and complete the self-built thesaurus, the greater the improvement in the accuracy of the automatic indexing results. This is because the technical personnel are ultimately grouped for processing.

[0303] Of course, the above description is the preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. Traffic technology data indexing methods, including: Load the data that needs to be indexed; Determine the data type, convert the data type according to the standard database structure, and import it into the processing database; The data imported into the database is segmented into words to obtain N first keywords; Automatic indexing is performed based on M of the first keywords; Where M and N are positive integers, and M ≤ N; The method also includes: Calculate the co-occurrence combinations of keywords, compare them with existing standard data, and extract the Chinese Library Classification number with the highest frequency. Search the Chinese Library Classification Thesaurus and extract the classification numbers of valid matching data; The classification number data of the preset professional terminology is added to the cluster terminology as the standard point. The combination feature vector of each keyword is calculated, and the standard point that is closest to it is calculated. The classification number of the standard point data is extracted as the subject similarity classification number. Use the China Library Classification-Subject Classification Number Mapping Table to obtain the subject mapping classification number; The intersection of the subject similarity classification number and the subject mapping classification number is taken as the subject classification number result; In this method, before automatic indexing, the N first keywords that have undergone word segmentation are clustered to obtain a clustered vocabulary. The prior probability of each word in the clustered vocabulary is calculated and multiplied by the corresponding TF-IDF value. The results are then sorted, and the top M first keywords in the sequence are extracted for automatic indexing. The subject-specific vocabulary is retrieved, and the classification numbers of the valid matching data are extracted and added to the clustered vocabulary. The data includes data on scientific and technological personnel, scientific and technological projects, scientific and technological achievements, and scientific and technological literature. It includes using a basic information table of scientific and technological personnel, scientific and technological projects, scientific and technological achievements, and scientific and technological literature as the original data table. Based on the original data table, an indexing information table is generated for subsequent indexing processing. The indexing information is processed through a TF-IDF weight table and a classification number clustering table, and integrated to generate a keyword indexing result table and a classification number indexing result table.

2. The traffic technology data indexing method as described in claim 1, wherein M is a positive integer selected from 3 to 8.

3. A transportation technology data indexing system, which includes: A data input unit configured to load data that needs to be indexed; The data processing unit is configured as follows: Determine the data type, convert the data type according to the standard database structure, and import it into the processing database; The data imported into the database is segmented into words to obtain N first keywords; Automatic indexing is performed based on M of the first keywords; Where M and N are positive integers, and M ≤ N; The method also includes: Calculate the co-occurrence combinations of keywords, compare them with existing standard data, and extract the Chinese Library Classification number with the highest frequency. Search the Chinese Library Classification Thesaurus and extract the classification numbers of valid matching data; The classification number data of the preset professional terminology is added to the cluster terminology as the standard point. The combination feature vector of each keyword is calculated, and the standard point that is closest to it is calculated. The classification number of the standard point data is extracted as the subject similarity classification number. Use the China Library Classification-Subject Classification Number Mapping Table to obtain the subject mapping classification number; The intersection of the subject similarity classification number and the subject mapping classification number is taken as the subject classification number result; In this method, before automatic indexing, the N first keywords that have undergone word segmentation are clustered to obtain a clustered vocabulary. The prior probability of each word in the clustered vocabulary is calculated and multiplied by the corresponding TF-IDF value. The results are then sorted, and the top M first keywords in the sequence are extracted for automatic indexing. The subject-specific vocabulary is retrieved, and the classification numbers of the valid matching data are extracted and added to the clustered vocabulary. The data includes data on scientific and technological personnel, scientific and technological projects, scientific and technological achievements, and scientific and technological literature. It includes using a basic information table of scientific and technological personnel, scientific and technological projects, scientific and technological achievements, and scientific and technological literature as the original data table. Based on the original data table, an indexing information table is generated for subsequent indexing processing. The indexing information is processed through a TF-IDF weight table and a classification number clustering table, and integrated to generate a keyword indexing result table and a classification number indexing result table.

4. The traffic technology data indexing system as described in claim 3, wherein M is a positive integer selected from 3 to 8.

Images