Public resource transaction data-oriented automatic classification method and system

By preprocessing and feature extraction of public resource transaction data and classifying it in combination with associated features, the problems of data source heterogeneity and format diversity are solved, and the accuracy of data classification and model performance are improved.

CN119025995AInactive Publication Date: 2024-11-26SHANXI JINCHAN E-COMMERCE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411211540.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In the classification of public resource transaction data, the existing technology is difficult to process uniformly due to different data sources and different formats, resulting in inaccurate extraction of data-related features, affecting the accuracy of classification.

Method used

By collecting structured and unstructured data, pre-processing is performed to remove duplication and noise, extract structured and unstructured features, and classify them in combination with associated features, optimize the feature set, and improve the accuracy of data classification.

Benefits of technology

It realizes high-quality processing and classification of public resource transaction data, improves data integrity and analysis efficiency, and enhances the prediction ability and classification accuracy of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025995A_ABST
    Figure CN119025995A_ABST
Patent Text Reader

Abstract

The invention discloses an automatic classification method and system for public resource transaction data, and relates to the technical field of data processing. The method comprises the following steps: collecting public resource transaction data including first structured data and first unstructured data; preprocessing the plurality of pieces of first structured data and the first unstructured data to obtain a plurality of pieces of preprocessed structured data and a plurality of pieces of preprocessed unstructured data; extracting a first feature from each piece of pre-processed structured data, and extracting a second feature from each piece of pre-processed unstructured data; extracting associated features from the plurality of first features and the plurality of second features; removing associated features from the plurality of first features and the plurality of second features to obtain a plurality of third features and a plurality of fourth features; and classifying the plurality of third features, the plurality of fourth features and the plurality of associated features to obtain a plurality of classification results. By implementing the technical scheme of the invention, the accuracy of data classification can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an automatic classification method and system for public resource transaction data. Background Art

[0002] In the field of public resource transactions, there are many channels for publishing bidding announcements and various data sources. These data usually include announcement title, release time, project amount, announcement content, project description, etc. Effectively processing and analyzing these data is crucial for government departments and enterprises in the decision-making process. In particular, accurate classification of these data can greatly improve the accuracy and efficiency of decision-making.

[0003] However, the classification methods in the prior art have obvious technical defects. In the data analysis and classification stage, due to different data sources and different data formats, it is difficult for the existing analysis and classification methods to uniformly process various types of data, resulting in inaccurate extraction of data-related features, thus affecting the accuracy of the final data classification.

[0004] Therefore, how to improve the accuracy of data classification has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a method and system for automatic classification of public resource transaction data, which can improve the accuracy of data classification.

[0006] In a first aspect, the present application provides an automatic classification method for public resource transaction data, the method comprising: collecting public resource transaction data from various bidding announcement release channels, the public resource transaction data comprising first structured data and first unstructured data; preprocessing multiple first structured data to obtain multiple preprocessed structured data; preprocessing multiple first unstructured data to obtain multiple preprocessed unstructured data; extracting a first feature from each of the preprocessed structured data, and extracting a second feature from each of the preprocessed unstructured data; extracting associated features from multiple first features and multiple second features; removing the associated features from multiple first features and multiple second features, respectively, to obtain multiple third features and multiple fourth features, respectively; classifying multiple third features, multiple fourth features, and multiple associated features to obtain multiple classification results.

[0007] By adopting the above technical solution, public resource transaction data is collected from various bidding announcement release channels, and the public resource transaction data includes first structured data and first unstructured data, so as to comprehensively obtain the diversity and richness of bidding information, ensure the integrity and wide coverage of data, and provide high-quality basic data. By preprocessing multiple first structured data, multiple preprocessed structured data are obtained, thereby improving data quality, eliminating duplicate and redundant information, and providing a reliable basis for subsequent feature extraction and classification. By preprocessing multiple first unstructured data, multiple preprocessed unstructured data are obtained, thereby removing noise and irrelevant information, improving data quality and analysis effectiveness, and enhancing data diversity and integrity. By extracting the first feature from each preprocessed structured data and extracting the second feature from each preprocessed unstructured data, the original data is converted into a representative feature vector, retaining the key information in the data, improving the analyzability of the data and the input quality of the model. By extracting associated features from multiple first features and multiple second features, the information in structured data and unstructured data is comprehensively utilized to identify the most representative features, enhance the richness and information content of the feature set, and improve the prediction ability and accuracy of the model. By removing the associated features from the multiple first features and the multiple second features, multiple third features and multiple fourth features are obtained respectively, thereby optimizing the feature set, retaining the most informative features, removing redundant and repeated features, improving the quality of the data set and the performance of the model, and providing high-quality input for subsequent classification. By classifying multiple third features, multiple fourth features, and multiple associated features, multiple classification results are obtained, thereby fully utilizing the information in different feature sets, optimizing the input of the classification model, and improving the prediction accuracy of the classification model, thereby ensuring the accuracy of data classification.

[0008] Optionally, the preprocessed structured data includes preprocessed numerical type structured data and preprocessed non-numerical type structured data; the preprocessing of multiple first structured data to obtain multiple preprocessed structured data specifically includes: deduplicating multiple first structured data to obtain multiple second structured data; performing missing value processing on multiple second structured data to obtain multiple third structured data; the third structured data includes numerical type structured data and non-numerical type structured data; standardizing multiple numerical type structured data to obtain multiple preprocessed numerical type structured data; encoding multiple non-numerical type structured data to obtain multiple preprocessed non-numerical type structured data.

[0009] By adopting the above technical solution, multiple first structured data are deduplicated to obtain multiple second structured data, thereby eliminating duplicate records that may exist in the data set, ensuring the uniqueness and accuracy of the data, improving the quality of the data, and reducing the interference of redundant information on subsequent analysis. By processing missing values ​​for multiple second structured data, multiple third structured data are obtained, thereby filling or processing incomplete parts of the data set, improving the integrity of the data, ensuring the availability of the data, and avoiding the impact of missing values ​​on the data analysis results. By standardizing multiple numerical type structured data, multiple pre-processed numerical type structured data are obtained, thereby converting numerical data of different dimensions to the same scale, eliminating the impact of dimensional differences on data analysis, enhancing the comparability and consistency of the data, and providing high-quality numerical data for subsequent data analysis and model training. By encoding multiple non-numerical type structured data, multiple pre-processed non-numerical type structured data are obtained, thereby converting categorical data into numerical data, so that it can be effectively processed by machine learning algorithms, improving the expressiveness of data, enhancing the analyzability of data, and laying the foundation for subsequent feature extraction and classification.

[0010] Optionally, the first unstructured data includes text type data, image type data, document type data, audio type data and video type data; the preprocessed unstructured data includes preprocessed text data and preprocessed image data; the preprocessing of multiple first unstructured data to obtain multiple preprocessed unstructured data specifically includes: performing text preprocessing on multiple text type data to obtain multiple preprocessed text data; performing image preprocessing on multiple image type data to obtain multiple preprocessed image data; converting multiple document type data into multiple text type data based on a document parsing tool, and performing text preprocessing on multiple text type data to obtain multiple preprocessed text data; converting multiple audio type data into multiple text type data based on a speech recognition tool, and performing text preprocessing on multiple text type data to obtain multiple preprocessed text data; extracting multiple key frames from multiple video type data, and performing image preprocessing on multiple key frames to obtain multiple preprocessed image data.

[0011] Optionally, the extracting associated features from multiple first features and multiple second features specifically includes: vectorizing multiple first features to obtain multiple first feature vectors, and vectorizing multiple second features to obtain multiple second feature vectors; concatenating multiple first feature vectors and multiple second feature vectors to obtain a first comprehensive feature vector; standardizing and normalizing the first comprehensive feature vector to obtain a second comprehensive feature vector; performing correlation analysis on the second comprehensive feature vector to obtain a feature correlation matrix; performing mutual information analysis on the second comprehensive feature vector to obtain a feature mutual information matrix; and screening the second comprehensive feature vector based on the feature correlation matrix and the feature mutual information matrix to obtain multiple associated features.

[0012] By adopting the above technical solution, multiple first features are vectorized to obtain multiple first feature vectors, and multiple second features are vectorized to obtain multiple second feature vectors, thereby converting the original data into a numerical vector form, retaining the key information in the data, improving the analyzability of the data and the input quality of the model, and laying a foundation for subsequent feature splicing and processing. By splicing multiple first feature vectors and multiple second feature vectors, a first comprehensive feature vector is obtained, thereby integrating feature information from different sources to form a unified feature representation, enhancing the richness and completeness of the feature representation, and improving the expressiveness of the feature vector. By standardizing and normalizing the first comprehensive feature vector, a second comprehensive feature vector is obtained, thereby eliminating the dimensional differences between different features, allowing all features to be compared and analyzed on the same scale, enhancing the comparability and consistency of the data, improving the quality of the feature vector and the stability of the model, and laying a solid foundation for subsequent correlation analysis and mutual information analysis. By performing correlation analysis on the second comprehensive feature vector, a feature correlation matrix is ​​obtained, thereby evaluating the linear relationship between features, identifying the significant correlation between feature pairs, and providing important linear relationship information. By performing mutual information analysis on the second comprehensive feature vector, the feature mutual information matrix is ​​obtained, thereby evaluating the nonlinear relationship between features, capturing the dependency between feature pairs, and providing important nonlinear relationship information, which complements the results of correlation analysis and provides an important basis for comprehensively evaluating the relationship between features. By screening the second comprehensive feature vector based on the feature correlation matrix and the feature mutual information matrix, multiple associated features are obtained, thereby comprehensively utilizing the information of linear and nonlinear relationships, screening out the most representative features, removing redundant features, enhancing the effectiveness and information content of the feature set, and improving the prediction ability and accuracy of the model, laying a solid foundation for realizing accurate automatic classification of public resource transaction data.

[0013] Optionally, the first feature includes a numerical type feature and a non-numerical type feature; the second feature includes a document type feature and a picture type feature; the first feature vector includes a numerical feature vector and a non-numerical feature vector; the second feature vector includes a text feature vector and an image feature vector; the vectorization of multiple first features to obtain multiple first feature vectors, and the vectorization of multiple second features to obtain multiple second feature vectors, specifically includes: normalizing multiple numerical type features to obtain multiple numerical feature vectors; one-hot encoding multiple non-numerical type features to obtain multiple non-numerical feature vectors; vectorizing multiple document type features to obtain multiple text feature vectors; and vectorizing multiple picture type features to obtain multiple image feature vectors.

[0014] Optionally, the second comprehensive feature vector is screened based on the feature correlation matrix and the feature mutual information matrix to obtain multiple correlation features, specifically including: setting a correlation threshold and a mutual information threshold; screening the feature correlation matrix based on the correlation threshold, retaining feature pairs with correlation greater than the correlation threshold, to obtain a first high correlation feature pair set; screening the feature mutual information matrix based on the mutual information threshold, retaining feature pairs with mutual information greater than the mutual information threshold, to obtain a second high mutual information feature pair set; performing an intersection operation on the first high correlation feature pair set and the second high mutual information feature pair set to obtain a comprehensive high correlation feature pair set; extracting features from the comprehensive high correlation feature pair set to obtain multiple correlation features.

[0015] By adopting the above technical solution, by setting the correlation threshold and the mutual information threshold, a clear standard is provided for the feature screening process, ensuring that only feature pairs with significant linear and nonlinear relationships are retained, and the scientificity and rationality of the screening process are enhanced. By screening the feature correlation matrix based on the correlation threshold, retaining the feature pairs with correlation greater than the correlation threshold, and obtaining the first high correlation feature pair set, the feature pairs with significant linear relationships are identified and retained, and the feature pairs with weak linear correlation are removed, thereby improving the quality of the feature set and the representativeness of linear information. By screening the feature mutual information matrix based on the mutual information threshold, retaining the feature pairs with mutual information greater than the mutual information threshold, and obtaining the second high mutual information feature pair set, the feature pairs with significant nonlinear dependencies are identified and retained, and the feature pairs with weak nonlinear dependencies are removed, thereby enhancing the effectiveness of the feature set and the representativeness of nonlinear information. By performing an intersection operation on the first high correlation feature pair set and the second high mutual information feature pair set, a comprehensive high correlation feature pair set is obtained, thereby comprehensively considering the linear and nonlinear relationships of the features, and screening out feature pairs that are outstanding in both aspects, further improving the representativeness and information content of the feature set. By extracting features from a comprehensive set of highly correlated feature pairs, multiple correlated features are obtained, thereby extracting the most informative and representative features, removing redundant and irrelevant features, improving the quality of the feature set and the performance of the model, and providing high-quality input for subsequent classification and analysis.

[0016] Optionally, the classification results include a first classification set, a second classification set and a third classification set; the classification of the multiple third features, the multiple fourth features and the multiple associated features to obtain multiple classification results specifically includes: inputting the multiple third features into a preset classification model and outputting the first classification set; inputting the multiple fourth features into a preset classification model and outputting the second classification set; inputting the multiple associated features into a preset classification model and outputting the third classification set.

[0017] By adopting the above technical solution, by inputting multiple third features into a preset classification model and outputting a first classification set, the structured data features after removing the associated features are used for classification, which effectively improves the performance and accuracy of the model and ensures the reliability of the classification results. By inputting multiple fourth features into a preset classification model and outputting a second classification set, the unstructured data features after removing the associated features are used for classification, which fully utilizes the information of different data sources, improves the diversity and accuracy of classification, and enhances the adaptability and stability of the classification model. By inputting multiple associated features into a preset classification model and outputting a third classification set, the comprehensive high-associated features are used for classification, the most informative features are integrated, the prediction ability and effect of the classification model are significantly improved, and the efficiency and accuracy of the classification results are ensured.

[0018] In a second aspect of the present application, an automatic classification system for public resource transaction data is provided, the system comprising: a data collection module and a processing module; the data collection module is used to collect public resource transaction data from various bidding announcement release channels, the public resource transaction data comprising first structured data and first unstructured data; the processing module is used to preprocess a plurality of the first structured data to obtain a plurality of preprocessed structured data, and to preprocess a plurality of the first unstructured data to obtain a plurality of preprocessed unstructured data; the processing module is also used to extract a first feature from each of the preprocessed structured data, and to extract a second feature from each of the preprocessed unstructured data; the processing module is also used to extract associated features from a plurality of the first features and a plurality of the second features; the processing module is also used to remove the associated features from a plurality of the first features and a plurality of the second features, respectively, to obtain a plurality of third features and a plurality of fourth features, respectively; the processing module is also used to classify a plurality of the third features, a plurality of the fourth features, and a plurality of the associated features to obtain a plurality of classification results.

[0019] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes any method as described in the first aspect of the present application.

[0020] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program that can be loaded by a processor and execute the method as described in any one of the first aspects of the present application.

[0021] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: By collecting public resource transaction data from various bidding announcement release channels, the public resource transaction data includes first structured data and first unstructured data, so as to comprehensively obtain the diversity and richness of bidding information, ensure the integrity and wide coverage of data, and provide high-quality basic data. By preprocessing multiple first structured data, multiple preprocessed structured data are obtained, so as to improve the data quality, eliminate duplicate and redundant information, and provide a reliable basis for subsequent feature extraction and classification. By preprocessing multiple first unstructured data, multiple preprocessed unstructured data are obtained, so as to remove noise and irrelevant information, improve the quality of data and the effectiveness of analysis, and enhance the diversity and integrity of data. By extracting the first feature from each preprocessed structured data and extracting the second feature from each preprocessed unstructured data, the original data is converted into a representative feature vector, the key information in the data is retained, and the analyzability of the data and the input quality of the model are improved. By extracting associated features from multiple first features and multiple second features, the information in structured data and unstructured data is comprehensively utilized to identify the most representative features, enhance the richness and information content of the feature set, and improve the prediction ability and accuracy of the model. By removing the associated features from the multiple first features and the multiple second features, multiple third features and multiple fourth features are obtained respectively, thereby optimizing the feature set, retaining the most informative features, removing redundant and repeated features, improving the quality of the data set and the performance of the model, and providing high-quality input for subsequent classification. By classifying multiple third features, multiple fourth features, and multiple associated features, multiple classification results are obtained, thereby fully utilizing the information in different feature sets, optimizing the input of the classification model, and improving the prediction accuracy of the classification model, thereby ensuring the accuracy of data classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is one of the flow charts of an automatic classification method for public resource transaction data provided in an embodiment of the present application; Figure 2 This is the second flow chart of an automatic classification method for public resource transaction data provided in an embodiment of the present application; Figure 3 It is a structural diagram of an automatic classification system for public resource transaction data provided by an embodiment of the present application; Figure 4 It is a structural schematic diagram of an electronic device disclosed in an embodiment of the present application.

[0023] Explanation of the reference numerals: 1. Data acquisition module; 2. Processing module; 400. Electronic device; 401. Processor; 402. Communication bus; 403. User interface; 404. Network interface; 405. Memory. DETAILED DESCRIPTION

[0024] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0025] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0026] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" can explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0027] This application provides an automatic classification method for public resource transaction data. Figure 1 , which shows one of the flow diagrams of an automatic classification method for public resource transaction data provided in an embodiment of the present application. The method includes steps S1-S7, which are as follows: Step S1: Collect public resource transaction data from various bidding announcement release channels, where the public resource transaction data includes first structured data and first unstructured data.

[0028] In this embodiment, step S1 aims to collect public resource transaction data from various bidding announcement release channels, and the public resource transaction data includes first structured data and first unstructured data. The purpose of collecting these data is to obtain comprehensive and diverse bidding information, so as to perform subsequent data preprocessing and feature extraction to achieve accurate automatic classification.

[0029] First, by establishing connections with various bidding announcement release channels, the system automatically accesses these channels and captures the latest bidding announcements. In order to ensure the comprehensiveness and diversity of data, the system has designed a variety of data collection methods, including API interface calls, web crawler technology, and data crawler tools. API interface calls are suitable for platforms that provide standardized data interfaces to ensure the real-time and accuracy of data acquisition; web crawler technology is suitable for platforms that do not provide API interfaces, and extracts the required bidding announcement information by parsing the web page structure; data crawler tools are used to process data files that need to be manually downloaded and uploaded.

[0030] During the data collection process, the system will preliminarily organize and store the collected data according to predefined formats and standards. The first structured data includes structured fields such as bidding number, project name, and release time. These fields have clear definitions and formats, which are convenient for direct storage and management. The first unstructured data includes text type data, image type data, document type data, audio type data, and video type data, that is, including bidding documents, project descriptions, picture attachments, audio records, and video materials. These data types are diverse and the formats are not fixed, which requires further processing and parsing.

[0031] Step S2: Preprocess the plurality of first structured data to obtain a plurality of preprocessed structured data.

[0032] In a possible implementation, step S2 specifically includes the following steps: Deduplication processing is performed on the multiple first structured data to obtain multiple second structured data.

[0033] Missing value processing is performed on the plurality of second structured data to obtain a plurality of third structured data; the third structured data includes numerical type structured data and non-numerical type structured data.

[0034] A plurality of numerical type structured data are standardized to obtain a plurality of preprocessed numerical type structured data.

[0035] The plurality of non-numeric type structured data are encoded to obtain a plurality of pre-processed non-numeric type structured data.

[0036] In this embodiment, step S2 is to preprocess the plurality of first structured data to obtain a plurality of preprocessed structured data. The purpose of this step is to improve data quality, reduce noise and redundant data, and provide a clean and standardized data set for subsequent feature extraction and classification.

[0037] First, multiple first structured data are deduplicated to obtain multiple second structured data. The purpose of deduplication is to eliminate duplicate records that may exist in the data set, which will have a negative impact on subsequent data analysis and model training. By using a unique identifier (such as a bidding number) as a key field, the system automatically detects and deletes duplicate records to ensure the uniqueness and accuracy of each piece of data.

[0038] Next, multiple second structured data are processed for missing values ​​to obtain multiple third structured data. The purpose of missing value processing is to fill or process incomplete parts of the data set so as not to affect the results of data analysis. For numerical type structured data, mean filling, median filling or interpolation methods can be used for filling; for non-numerical type structured data, common values ​​can be used for filling or records with missing values ​​can be directly deleted. Through these methods, the system can effectively process the missing parts in the data and ensure the integrity and consistency of the data.

[0039] Then, multiple numerical structured data are standardized to obtain multiple preprocessed numerical structured data. The purpose of standardization is to convert numerical data of different dimensions to the same scale, thereby eliminating the impact of dimensional differences on data analysis. Specifically, the Z-score standardization method or the Min-Max normalization method can be used to convert numerical data to a standard normal distribution with a mean of zero and a variance of one, or to scale the data to the range of [0,1]. These processing steps can ensure that the numerical data are comparable and uniform in subsequent analysis.

[0040] Finally, multiple non-numeric structured data are encoded to obtain multiple pre-processed non-numeric structured data. The purpose of encoding is to convert categorical data into numerical data for subsequent training and analysis of machine learning models. Common encoding methods include One-Hot Encoding and Label Encoding. One-Hot Encoding converts each category into a binary vector, which is suitable for unordered categorical data; label encoding assigns a unique numerical identifier to each category, which is suitable for ordered categorical data. Through these encoding methods, the system can convert non-numeric data into a numerical form suitable for processing by machine learning algorithms.

[0041] Through the above preprocessing steps, the system can effectively improve the quality of structured data, eliminate the impact of redundant data and missing values, and ensure the consistency and standardization of data. These preprocessed structured data provide a reliable foundation for subsequent feature extraction and classification, significantly improving the accuracy of data analysis and the performance of the model. Overall, the implementation of step S2 effectively optimizes data quality and lays a solid foundation for efficient and accurate automatic classification of public resource transaction data.

[0042] Step S3: preprocessing the plurality of first unstructured data to obtain a plurality of preprocessed unstructured data.

[0043] In a possible implementation, step S3 specifically includes the following steps: Text preprocessing is performed on multiple text type data to obtain multiple preprocessed text data.

[0044] Image preprocessing is performed on multiple image type data to obtain multiple preprocessed image data.

[0045] Based on the document parsing tool, multiple document type data are converted into multiple text type data, and text preprocessing is performed on the multiple text type data to obtain multiple preprocessed text data.

[0046] Based on a speech recognition tool, multiple audio type data are converted into multiple text type data, and text preprocessing is performed on the multiple text type data to obtain multiple preprocessed text data.

[0047] A plurality of key frames are extracted from a plurality of video type data, and image preprocessing is performed on the plurality of key frames to obtain a plurality of preprocessed image data.

[0048] In this embodiment, step S3 is to preprocess the plurality of first unstructured data to obtain a plurality of preprocessed unstructured data. The purpose of this step is to convert various types of unstructured data into a format that can be analyzed and processed so that the subsequent feature extraction and classification steps can proceed smoothly.

[0049] First, multiple text type data are preprocessed to obtain multiple preprocessed text data. Text type data, such as project descriptions and bidding documents, usually contain rich semantic information, but may also contain noise and irrelevant information. The main steps of text preprocessing include word segmentation, stop word removal, stemming, and standardization. Word segmentation is to divide the text into separate words; stop word removal is to delete common but meaningless words; stemming is to restore words to their basic form; standardization is to convert all words to lowercase. Through these processing steps, noise can be removed, key information in the text can be extracted, and the quality of text data can be improved.

[0050] Next, multiple image type data are preprocessed to obtain multiple preprocessed image data. Image type data, such as project drawings and picture attachments, contain visual information and require processing steps such as format conversion, resizing, and noise removal. First, all images are converted to a unified format, such as JPEG or PNG; second, the images are resized to a fixed size for subsequent analysis and comparison; finally, image filtering technology is used to remove noise from the image and improve the clarity of the image. These preprocessing steps can standardize the image data and enhance the parseability of the image.

[0051] For document type data, such as PDF and Word files, the system uses document parsing tools to convert them into multiple text type data, and performs text preprocessing on these text type data to obtain multiple preprocessed text data. The document parsing tool can extract the text content in the document and convert it into a standard text format for subsequent processing. For the converted text data, the system performs the same preprocessing steps as the text type data to ensure the consistency and quality of the text data.

[0052] For audio data, such as meeting records and voice messages, the system uses speech recognition tools to convert them into multiple text data, and performs text preprocessing on these text data to obtain multiple preprocessed text data. The speech recognition tool can convert speech content into text form and extract key information. The converted text data is also processed by word segmentation, stop word removal, stem extraction and standardization to ensure the quality of the text data.

[0053] Finally, multiple key frames are extracted from multiple video data types, and these key frames are image preprocessed to obtain multiple preprocessed image data. Video data types, such as on-site recordings and project introduction videos, contain a large amount of visual information. By extracting key frames, the video content can be simplified into a series of important image frames, reducing the complexity of data processing. The extracted key frame images are also preprocessed by format conversion, size adjustment, and noise removal to ensure the standardization and clarity of the image data.

[0054] Through the above preprocessing of text, image, document, audio and video data, the system can convert various unstructured data into a unified and standardized format, significantly improving the availability and quality of the data. These preprocessed unstructured data provide a reliable foundation for subsequent feature extraction and classification steps, ensuring the accuracy of data analysis and the performance of the model.

[0055] Step S4: extracting a first feature from each preprocessed structured data, and extracting a second feature from each preprocessed unstructured data.

[0056] In this embodiment, the purpose of step S4 is to extract the first feature from each preprocessed structured data and extract the second feature from each preprocessed unstructured data. Through feature extraction, the original data can be converted into representative feature vectors, which can provide effective input for subsequent associated feature extraction and classification.

[0057] First, the first feature is extracted from the preprocessed structured data. The preprocessed structured data includes numerical structured data and non-numerical structured data. For numerical structured data, the system directly uses it as part of the first feature because these numerical values ​​themselves have important information. For example, the numerical values ​​of fields such as bidding amount and project duration can directly reflect the scale and time requirements of the project. For non-numerical structured data, the system converts it into numerical form through One-Hot Encoding. For example, fields such as the category and region of the bidding project can be converted into binary vectors through One-Hot Encoding, with each category or region corresponding to one dimension. Through these processes, the system can extract a complete first feature vector from the structured data, which contains all important information.

[0058] Secondly, the second feature is extracted from the preprocessed unstructured data. The preprocessed unstructured data includes text data and image data. For text data, the system uses text vectorization methods for feature extraction. Common methods include TF-IDF (term frequency-inverse document frequency) and word embeddings. The TF-IDF method converts text into vector form by calculating the importance of each word in the document; the word embedding method maps each word to a high-dimensional vector space through a pre-trained model (such as Word2Vec, GloVe, etc.) to obtain the semantic representation of the text. Through these methods, the system can convert text data into feature vectors with semantic information as part of the second feature. For image data, the system uses a convolutional neural network (CNN) for feature extraction. CNN extracts local and global features of the image through multi-layer convolution and pooling operations, and converts the image into a high-dimensional feature vector. These feature vectors can effectively represent the content and structure of the image as part of the second feature.

[0059] Through the above feature extraction process, the system can extract representative first and second features from the preprocessed structured data and unstructured data. These feature vectors not only retain the important information in the original data, but also enhance the data's expressiveness, providing a solid foundation for subsequent associated feature extraction and classification.

[0060] Step S5: extracting associated features from the plurality of first features and the plurality of second features.

[0061] In this embodiment, the purpose of step S5 is to extract associated features from the plurality of first features and the plurality of second features. By extracting associated features, information in structured data and unstructured data can be combined to obtain a more comprehensive and effective feature set to provide support for subsequent classification.

[0062] In a possible implementation, step S5 specifically includes steps S51-S56: Step S51: vectorizing the plurality of first features to obtain a plurality of first feature vectors, and vectorizing the plurality of second features to obtain a plurality of second feature vectors.

[0063] In a possible implementation, the first feature includes a numerical type feature and a non-numerical type feature; the second feature includes a document type feature and a picture type feature; the first feature vector includes a numerical feature vector and a non-numerical feature vector; the second feature vector includes a text feature vector and an image feature vector. Step S51 specifically includes the following steps: Multiple numerical type features are normalized to obtain multiple numerical feature vectors.

[0064] Perform one-hot encoding on multiple non-numeric features to obtain multiple non-numeric feature vectors.

[0065] Vectorization is performed on multiple document type features to obtain multiple text feature vectors.

[0066] Vectorization is performed on multiple image type features to obtain multiple image feature vectors.

[0067] In this embodiment, the purpose of step S51 is to vectorize the multiple first features to obtain multiple first feature vectors, and to vectorize the multiple second features to obtain multiple second feature vectors. Through vectorization, different types of features can be converted into a unified numerical vector form, so as to facilitate the subsequent splicing and analysis of comprehensive features.

[0068] First, normalization is performed on the numerical type features in the multiple first features. Numerical type features such as bidding amount, project duration, etc. may have different dimensions and ranges in the original data. Through normalization, these numerical features can be scaled to a fixed range, usually [0,1], to eliminate the impact of dimensional differences on the analysis. Specifically, the minimum-maximum normalization method can be used to map the minimum value of each numerical feature to 0, the maximum value to 1, and the other values ​​​​are scaled according to their relative positions. Through normalization, multiple numerical feature vectors are obtained, which are on the same scale, which is convenient for subsequent splicing and comparison.

[0069] Next, one-hot encoding is performed on the non-numeric features in the multiple first features. Non-numeric features such as project categories, regions, etc. are usually discrete categorical data. One-hot encoding converts category information into numerical form by creating a binary vector for each category. For example, for the project category feature, if there are three categories "A", "B", and "C", category "A" is encoded as [1,0,0], category "B" is encoded as [0,1,0], and category "C" is encoded as [0,0,1]. In this way, non-numeric features can be converted into multiple non-numeric feature vectors, so that discrete category information can participate in numerical calculation and analysis.

[0070] Then, the text type features in the multiple second features are vectorized. Text type features such as project descriptions and bidding documents usually contain rich semantic information and require feature extraction to be converted into numerical vectors. Common methods include TF-IDF and word embeddings. The TF-IDF method converts the text into a sparse vector by calculating the importance of each word in the document. The word embedding method maps each word to a high-dimensional vector space through a pre-trained model (such as Word2Vec, GloVe) to obtain a semantic representation of the text. Through these methods, text type features can be converted into multiple text feature vectors, retaining important information and semantic relationships in the text.

[0071] Finally, the image type features in the multiple second features are vectorized. Image type features, such as project drawings and image attachments, contain rich visual information. Feature extraction through convolutional neural networks (CNNs) can extract local and global features of the image and convert the image into a high-dimensional feature vector. CNN extracts features from the image layer by layer through multi-layer convolution and pooling operations, and finally obtains a feature vector of fixed length. In this way, the image type features can be converted into multiple image feature vectors, retaining important information and structural features in the image.

[0072] Through the above vectorization processing, the system can convert multiple first features and multiple second features into a unified numerical vector form. These vectors not only retain the important information in the original data, but also enhance the expressiveness of the features, providing a solid foundation for subsequent comprehensive feature splicing and analysis.

[0073] Step S52: concatenate the multiple first feature vectors and the multiple second feature vectors to obtain a first comprehensive feature vector.

[0074] In this embodiment, the purpose of step S52 is to concatenate multiple first feature vectors and multiple second feature vectors to obtain a first comprehensive feature vector. Through the concatenation operation, features from different data sources can be integrated together to form a unified feature representation to improve the richness and completeness of the features and provide more information for subsequent analysis and classification.

[0075] First, after the processing of step S51, the system has obtained multiple first feature vectors and multiple second feature vectors. The first feature vector includes a numerical feature vector and a non-numerical feature vector, which represent numerical type features and non-numerical type features in structured data respectively. The second feature vector includes a text feature vector and an image feature vector, which represent text type features and image type features in unstructured data respectively.

[0076] In order to achieve feature concatenation, the system first needs to ensure that the dimensions and formats of all feature vectors are consistent. In actual operations, numerical feature vectors and non-numerical feature vectors are usually already in numerical form, so no further conversion is required. For text feature vectors and image feature vectors, since they may have different dimensions and format differences, the system needs to standardize them using a unified dimension and format. For example, for text feature vectors, a fixed-length word vector representation can be used to pad or truncate all text feature vectors to the same length. For image feature vectors, a fixed-size feature vector representation can be used, and a fixed-length feature vector can be extracted through the last layer output of a convolutional neural network (CNN).

[0077] After ensuring that all feature vectors have consistent dimensions and formats, the system concatenates multiple first feature vectors and multiple second feature vectors. Specifically, the concatenation operation is to connect feature vectors from different sources in the feature dimension to form a new high-dimensional feature vector. For example, if the dimension of the numerical feature vector is 10, the dimension of the non-numerical feature vector is 20, the dimension of the text feature vector is 50, and the dimension of the image feature vector is 100, then the dimension of the concatenated first comprehensive feature vector will be 180 (10+20+50+100).

[0078] The concatenation operation can integrate information from different data sources by connecting different feature vectors in the feature dimension, thereby improving the richness and completeness of feature representation. This operation can retain the information of all original features and combine them together, providing more dimensional information for subsequent feature analysis and classification. In this way, the system can effectively utilize all information in structured and unstructured data and enhance the expressiveness of feature representation.

[0079] Through the above splicing operation, the system finally obtains multiple first comprehensive feature vectors. These first comprehensive feature vectors contain all the information of numerical features, non-numerical features, text features and image features, forming a unified and comprehensive feature representation. The first comprehensive feature vector not only retains the original information of various features, but also enhances the expressiveness and analyzability of the features by integrating different types of information.

[0080] Step S53: Standardize and normalize the first comprehensive feature vector to obtain a second comprehensive feature vector.

[0081] In this embodiment, the purpose of step S53 is to standardize and normalize the first comprehensive feature vector to obtain a second comprehensive feature vector. Through standardization and normalization, the dimensional differences between different features can be eliminated, and the features can be compared and analyzed on the same scale, thereby improving the comparability of the features and the stability of the model.

[0082] First, after the processing of step S52, the system has obtained multiple first comprehensive feature vectors. The first comprehensive feature vector is a high-dimensional feature vector composed of numerical feature vectors, non-numerical feature vectors, text feature vectors and image feature vectors. These feature vectors contain rich information from different data sources, but due to the different value ranges and dimensions of different features, direct analysis and modeling may be affected by dimensional differences.

[0083] In order to eliminate the dimension difference, the first comprehensive eigenvector is first standardized. The purpose of standardization is to convert the feature data into a standard normal distribution with zero mean and unit variance, so that each feature has the same scale. Specifically, for each eigenvalue, the standardized eigenvalue is obtained by subtracting the mean of the feature and dividing it by the standard deviation of the feature.

[0084] Next, the standardized first comprehensive eigenvector is normalized. The purpose of normalization is to scale the eigenvalues ​​to a fixed range, usually [0,1]. Normalization can prevent certain features from having too much influence on the overall eigenvector and keep the eigenvalues ​​on the same scale. Specifically, the minimum-maximum normalization method can be used to subtract the minimum value of each eigenvalue from the minimum value of the feature and then divide it by the value range of the feature (maximum value minus minimum value).

[0085] Through standardization and normalization, all feature vectors are converted to the same scale, which not only eliminates the dimension difference, but also prevents some feature values ​​from having excessive influence on model training. The processed feature vectors are more comparable and stable, which helps to improve the performance and accuracy of the model.

[0086] Step S54: Perform correlation analysis on the second comprehensive feature vector to obtain a feature correlation matrix.

[0087] In this embodiment, the purpose of step S54 is to perform correlation analysis on the second comprehensive feature vector to obtain a feature correlation matrix. Through correlation analysis, the linear relationship between features can be identified and the amount of information contained in each feature in the data set can be evaluated, thereby providing a basis for subsequent feature screening.

[0088] First, after the processing of step S53, the system has obtained a plurality of second comprehensive feature vectors. These feature vectors are standardized and normalized to ensure that all features are compared and analyzed on the same scale. In order to further evaluate the relationship between these features, the system performs correlation analysis on the second comprehensive feature vectors.

[0089] The purpose of correlation analysis is to calculate the correlation coefficient between each pair of features and evaluate the strength of their linear relationship. Specifically, the correlation coefficient can be calculated by the Pearson Correlation Coefficient. The value of the Pearson correlation coefficient is between -1 and 1, indicating the strength and direction of the linear relationship between two features. When the correlation coefficient is 1, it means that there is a completely positive correlation between the two features; when the correlation coefficient is -1, it means that there is a completely negative correlation between the two features; when the correlation coefficient is 0, it means that there is no linear relationship between the two features.

[0090] In order to perform correlation analysis, the system first takes the second comprehensive feature vector as input and calculates the Pearson correlation coefficient between all feature pairs. The specific calculation formula is as follows: Among them, r xy is the Pearson correlation coefficient between feature x and feature y, x iis the value of feature x, y i is the value of feature y, is the mean of feature x, is the mean of feature y.

[0091] By calculating the correlation coefficients between all feature pairs, the system can construct a feature correlation matrix. The feature correlation matrix is ​​a symmetric matrix, and each element in the matrix represents the correlation coefficient between the corresponding feature pairs. Specifically, the element R in the matrix ij Represents the correlation coefficient between the i-th feature and the j-th feature. The feature correlation matrix can intuitively display the linear relationship between all features and help identify which features have strong correlation. By analyzing the feature correlation matrix, the system can identify which features have strong linear relationships. This information is very important for subsequent feature screening, because highly correlated features may contain redundant information and have limited contribution to the model. By identifying and processing these redundant features, the system can improve the effectiveness of the feature set and enhance the performance of the model.

[0092] Step S55: Perform mutual information analysis on the second comprehensive feature vector to obtain a feature mutual information matrix.

[0093] In this embodiment, the purpose of step S55 is to perform mutual information analysis on the second comprehensive feature vector to obtain a feature mutual information matrix. Mutual information analysis can evaluate the nonlinear relationship between features, capture the dependency between features, provide supplementary information for subsequent feature screening, and complement the results of correlation analysis to comprehensively evaluate the relationship between features.

[0094] After processing in step S54, the system has obtained a feature correlation matrix, which reveals the linear relationship between features. However, there may be complex nonlinear relationships between many features, and relying solely on linear correlation analysis may not be able to fully capture these relationships. Therefore, mutual information analysis is used as a complementary method to evaluate the nonlinear dependencies between features. Mutual Information (MI) is a measure of the dependency between two random variables that can capture nonlinear relationships. The larger the value of the mutual information, the higher the degree of information sharing between the two variables and the stronger the dependency.

[0095] In order to perform mutual information analysis, the system first needs to discretize the continuous features. Discretization is done because mutual information is usually used to calculate discrete variables. Continuous features can be converted into discrete variables using equal-width discretization or equal-frequency discretization. The equal-width discretization method divides the feature values ​​into several intervals according to a fixed width, and the equal-frequency discretization method divides the feature values ​​into several intervals according to a fixed frequency. Through discretization, the system converts continuous features into discrete variables, which is convenient for subsequent mutual information calculation.

[0096] Next, the system calculates the joint probability distribution p(x,y) of each pair of features and their respective marginal probability distributions p(x) and p(y). By calculating the joint probability distribution of the features, the joint frequency of occurrence of each pair of features in each discrete interval can be determined, which provides basic data for the calculation of mutual information.

[0097] After obtaining the joint probability distribution, the system calculates the mutual information value between each pair of features according to the mutual information formula. The mutual information formula is: Among them, I(X;Y) represents the mutual information between feature X and feature Y, p(x,y) is the joint probability distribution of feature X and feature Y, p(x) is the marginal probability distribution of feature X, and p(y) is the marginal probability distribution of feature Y. By accumulating the probability values ​​of each discrete interval, the system can calculate the mutual information value between each pair of features.

[0098] Through the above steps, the system constructs a feature mutual information matrix. The feature mutual information matrix is ​​a symmetric matrix, and each element in the matrix represents the mutual information value between the corresponding feature pairs. Specifically, the element I ij Represents the mutual information value between the i-th feature and the j-th feature. The feature mutual information matrix intuitively shows the nonlinear relationship between all features and helps identify which features have strong nonlinear dependencies.

[0099] By analyzing the feature mutual information matrix, the system can identify which features have strong nonlinear relationships. This information is very important for subsequent feature screening, because features with high mutual information values ​​may contain important dependency information and contribute greatly to the model's predictive ability. By identifying and retaining these features with high mutual information values, the system can improve the effectiveness of the feature set and enhance the performance of the model.

[0100] Step S56: Screening the second comprehensive feature vector based on the feature correlation matrix and the feature mutual information matrix to obtain multiple correlation features.

[0101] In this embodiment, the purpose of step S56 is to screen the second comprehensive feature vector based on the feature correlation matrix and the feature mutual information matrix to obtain multiple associated features. By combining the information of linear and nonlinear relationships and screening the features, redundant features can be removed, the most representative features can be retained, and the accuracy and efficiency of the model can be improved.

[0102] In a possible implementation, step S56 specifically includes the following steps: Set the correlation threshold and mutual information threshold.

[0103] Specifically, first, after processing in step S54 and step S55, the system has obtained the feature correlation matrix and the feature mutual information matrix respectively. The feature correlation matrix reflects the linear relationship between features, while the feature mutual information matrix reveals the nonlinear dependency between features. In order to make full use of the information in these matrices, the system needs to set appropriate thresholds to screen out features with significant relationships.

[0104] The purpose of setting the correlation threshold is to filter out feature pairs with strong linear correlation. The selection of the correlation threshold should be based on the distribution of actual data and business needs. Usually, the correlation threshold can be set to 0.7 or 0.8, which means that only feature pairs with a correlation coefficient greater than the threshold will be considered to have a significant linear relationship and retained in subsequent analysis. Selecting such a threshold can ensure that the selected feature pairs are statistically sufficiently correlated and reduce the impact of noise and redundant features.

[0105] At the same time, the purpose of setting the mutual information threshold is to screen out feature pairs with strong nonlinear dependence. The selection of the mutual information threshold should also take into account the actual distribution of the data and the analysis requirements. There is no fixed limit to the range of mutual information, but by observing the distribution of the mutual information value, a moderate threshold value, such as 0.1 or 0.2, can be selected. Such a threshold setting can ensure that the selected feature pairs have a significant dependence in the amount of information and enhance the model's ability to capture complex data patterns.

[0106] In the process of setting the threshold, experiments and data analysis tools can be used to assist in determining the best threshold. The specific steps include calculating the correlation coefficient and mutual information value of all feature pairs, drawing their distribution graphs, observing the distribution characteristics, and selecting the appropriate threshold range. The impact of different thresholds on model performance can also be verified through methods such as cross-validation to determine the optimal threshold.

[0107] The feature correlation matrix is ​​screened based on the correlation threshold, and feature pairs with correlation greater than the correlation threshold are retained to obtain a first high-correlation feature pair set.

[0108] Specifically, after the above steps, the system has set an appropriate correlation threshold, which is used to measure the linear relationship between feature pairs. Only feature pairs with a correlation coefficient greater than the threshold are considered to have significant linear correlation.

[0109] The system first extracts the correlation coefficients of all feature pairs from the feature correlation matrix. The feature correlation matrix is ​​a symmetric matrix in which each element represents the Pearson correlation coefficient between the corresponding feature pairs. By traversing all elements in the matrix, the system can obtain the correlation coefficient between each pair of features.

[0110] Next, the system screens the feature pairs according to the set correlation threshold. Specifically, for each pair of features, if its correlation coefficient is greater than the correlation threshold, the feature pair will be retained and added to the first high correlation feature pair set. This set contains all feature pairs with significant linear relationships. This process ensures that only those feature pairs with statistically significant correlation are retained, and those with lower correlation are removed, reducing the interference of redundant information.

[0111] In order to improve the efficiency of the screening process, the system can use optimization algorithms and data structures. For example, the system can use matrix operation libraries (such as NumPy) for efficient matrix traversal and screening, and use vectorized operations to speed up calculations. Through these technical means, the system can complete the screening of large-scale data in a shorter time.

[0112] Through the above screening process, the system obtains the first set of highly correlated feature pairs. The feature pairs in this set have high linear correlation and can effectively reflect the linear relationship in the data. The purpose of retaining these feature pairs is to ensure that the amount of information in the feature set is maximized in subsequent feature screening and model training, while reducing the impact of redundant features on model performance.

[0113] The feature mutual information matrix is ​​screened based on the mutual information threshold, and feature pairs with mutual information greater than the mutual information threshold are retained to obtain the second highest mutual information feature pair set.

[0114] Specifically, after the above steps, the system has set an appropriate mutual information threshold, which is used to measure the nonlinear dependence between feature pairs. Only feature pairs with mutual information values ​​greater than the threshold are considered to have significant nonlinear dependence.

[0115] The system first extracts the mutual information values ​​of all feature pairs from the feature mutual information matrix. The feature mutual information matrix is ​​a symmetric matrix in which each element represents the mutual information value between the corresponding feature pairs. By traversing all elements in the matrix, the system can obtain the mutual information value between each pair of features.

[0116] Next, the system screens the feature pairs according to the set mutual information threshold. Specifically, for each pair of features, if its mutual information value is greater than the mutual information threshold, the feature pair will be retained and added to the second highest mutual information feature pair set. This set contains all feature pairs with significant nonlinear dependencies. This process ensures that only those feature pairs with statistically significant nonlinear relationships are retained, and those feature pairs with low mutual information values ​​are removed, reducing the interference of redundant information.

[0117] In order to improve the efficiency of the screening process, the system can use optimization algorithms and data structures. For example, the system can use efficient data processing libraries (such as Pandas) to quickly traverse and screen matrices, and use parallel computing to speed up the screening process. Through these technical means, the system can complete the screening of large-scale data in a shorter time.

[0118] Through the above screening process, the system obtains the second highest mutual information feature pair set. The feature pairs in this set have high nonlinear dependencies and can effectively reflect the complex relationships in the data. The purpose of retaining these feature pairs is to ensure that the amount of information in the feature set is maximized in subsequent feature screening and model training, while reducing the impact of redundant features on model performance.

[0119] An intersection operation is performed on the first high correlation feature pair set and the second high mutual information feature pair set to obtain a comprehensive high correlation feature pair set.

[0120] Specifically, after the above steps, the system has obtained the first high correlation feature pair set and the second high mutual information feature pair set. The first high correlation feature pair set includes all feature pairs with strong linear correlation, while the second high mutual information feature pair set includes all feature pairs with strong nonlinear dependence.

[0121] In order to comprehensively consider linear and nonlinear relationships, the system needs to perform an intersection operation on the two sets. The purpose of the intersection operation is to identify those feature pairs that perform well in both correlation and mutual information analysis. These feature pairs contain rich information in the data and have high analytical and predictive value.

[0122] Specifically, the system first traverses each feature pair in the first high correlation feature pair set and checks whether the feature pair also exists in the second high mutual information feature pair set. If a feature pair exists in both sets, it is added to the comprehensive high correlation feature pair set. In this way, the system can effectively screen out those feature pairs that perform well in both independent analyses.

[0123] In order to improve the efficiency of intersection operations, the system can use efficient data structures and algorithms. For example, using a set data structure to store feature pairs and using the intersection operation function of the set can greatly speed up the screening process. In addition, the system can also use parallel processing technology to further improve the efficiency of large-scale data processing.

[0124] Through the above intersection operation, the system finally obtains a set of comprehensive high-correlation feature pairs. The feature pairs in this set not only have strong linear correlation, but also contain significant nonlinear dependencies, which can fully reflect the complex relationships in the data. The purpose of retaining these feature pairs is to ensure that the amount of information in the feature set is maximized in subsequent feature screening and model training, while reducing the impact of redundant features on model performance.

[0125] Features are extracted from a set of comprehensive high-correlation feature pairs to obtain multiple correlation features.

[0126] Specifically, in order to extract features from the set of comprehensive high-correlation feature pairs, the system needs to further analyze each feature pair and identify representative individual features therein. Specifically, the system first traverses each feature pair in the set of comprehensive high-correlation feature pairs and extracts each feature in these feature pairs separately.

[0127] During the extraction process, the system needs to ensure the uniqueness of each feature in the feature set to avoid the existence of duplicate features. To achieve this, the system can use a set data structure to store the extracted features. The set data structure has the function of automatic deduplication, which can ensure that each feature is stored only once.

[0128] By traversing the set of comprehensive high-correlation feature pairs and extracting individual features from each feature pair, the system can gradually build a feature set containing all correlated features. This feature set contains all representative and informative features extracted from the set of comprehensive high-correlation feature pairs.

[0129] In order to improve the efficiency of the feature extraction process, the system can use efficient data processing algorithms and parallel processing techniques. For example, the use of parallel traversal and multi-threaded processing can significantly speed up the processing of large-scale data. In addition, the system can also use optimized data structures, such as hash tables, to speed up feature extraction and deduplication operations.

[0130] Step S6: removing associated features from the plurality of first features and the plurality of second features respectively, to obtain a plurality of third features and a plurality of fourth features respectively.

[0131] Specifically, the system needs to process the first feature and the second feature separately. The first feature includes numerical type features and non-numerical type features, while the second feature includes text type features and image type features. The system first traverses all the first features and checks whether each feature is in the associated feature set. If a feature exists in the associated feature set, it is removed from the first feature, and the remaining part is the third feature. This process ensures that the third feature set does not contain any part that has been identified as an associated feature, avoiding the existence of redundant features.

[0132] Next, the system performs the same process on the second feature. The system traverses all second features and checks whether each feature is in the associated feature set. If a feature exists in the associated feature set, it is removed from the second feature, and the remaining part is the fourth feature. This process also ensures that the fourth feature set does not contain any part that has been identified as an associated feature, ensuring the purity and effectiveness of the feature set.

[0133] In order to improve the efficiency of the removal process, the system can use efficient data structures and algorithms. For example, use the set data structure to store associated features, and use set operations to speed up feature inspection and removal. Through these technical means, the system can complete the processing of large-scale data in a shorter time.

[0134] Step S7: Classify the multiple third features, the multiple fourth features, and the multiple associated features to obtain multiple classification results.

[0135] In a possible implementation, the classification results include a first classification set, a second classification set, and a third classification set. Step S7 specifically includes the following steps: The plurality of third features are input into a preset classification model, and a first classification set is output.

[0136] The plurality of fourth features are input into a preset classification model, and a second classification set is output.

[0137] The plurality of associated features are input into a preset classification model, and a third classification set is output.

[0138] In this embodiment, the purpose of step S7 is to classify the plurality of third features, the plurality of fourth features and the plurality of associated features to obtain a plurality of classification results. Through this process, the data in the feature set can be effectively allocated to different categories, thereby improving the efficiency of data analysis and the prediction accuracy of the model.

[0139] First, after the processing of step S6, the system has obtained the third feature, the fourth feature and the associated feature respectively. The third feature is the remaining feature set after removing the associated feature from the first feature, the fourth feature is the remaining feature set after removing the associated feature from the second feature, and the associated feature is the feature extracted from the comprehensive high-associated feature pair set. These feature sets contain rich information and can provide effective input for the classification model.

[0140] In order to perform classification, the system selects an appropriate classification model. Common classification models include support vector machines (SVM), random forests, neural networks, etc. The selection of a suitable classification model should be based on the characteristics of the data and the specific requirements of the classification task. In this embodiment, a random forest model is selected as a classifier because the random forest model has advantages in processing high-dimensional data and preventing overfitting.

[0141] The system first uses the third feature as input and uses the random forest model for training and classification. To ensure the effectiveness of the model, the system divides the third feature into a training set and a test set, uses the training set to train the random forest model, and adjusts the model parameters to optimize the classification effect. Subsequently, the test set is used to evaluate the classification performance of the model to obtain the first classification set.

[0142] Next, the system performs similar processing on the fourth feature, taking the fourth feature as input and using the same random forest model for training and classification. The system also divides the fourth feature into a training set and a test set, evaluates the model performance through training and testing, and obtains the second classification set.

[0143] Finally, the system classifies the associated features. Since the associated features contain information about the comprehensive high-correlation feature pairs and may be highly complex, special attention should be paid to the adjustment of model parameters during the training process. The system divides the associated features into training sets and test sets, uses the random forest model for training, and evaluates its performance through the test set to obtain the third classification set.

[0144] Through the above classification process, the system finally obtains three classification results, namely the first classification set, the second classification set and the third classification set. These classification results correspond to the classification outputs of the third feature, the fourth feature and the associated feature, respectively, reflecting the classification of different feature sets in the data set.

[0145] Reference Figure 3, which shows a structural schematic diagram of an automatic classification system for public resource transaction data provided by an embodiment of the present application, the system includes: a data collection module 1 and a processing module 2; the data collection module 1 is used to collect public resource transaction data from various bidding announcement release channels, and the public resource transaction data includes first structured data and first unstructured data; the processing module 2 is used to preprocess multiple first structured data to obtain multiple preprocessed structured data, and preprocess multiple first unstructured data to obtain multiple preprocessed unstructured data; the processing module 2 is also used to extract the first feature from each preprocessed structured data, and extract the second feature from each preprocessed unstructured data; the processing module 2 is also used to extract associated features from multiple first features and multiple second features; the processing module 2 is also used to remove associated features from multiple first features and multiple second features respectively, and obtain multiple third features and multiple fourth features respectively; the processing module 2 is also used to classify multiple third features, multiple fourth features and multiple associated features to obtain multiple classification results.

[0146] In a possible implementation, the processing module 2 is also used to perform deduplication processing on multiple first structured data to obtain multiple second structured data; perform missing value processing on multiple second structured data to obtain multiple third structured data; the third structured data includes numerical type structured data and non-numerical type structured data; perform standardization processing on multiple numerical type structured data to obtain multiple preprocessed numerical type structured data; perform encoding processing on multiple non-numerical type structured data to obtain multiple preprocessed non-numerical type structured data.

[0147] In a possible implementation, the processing module 2 is also used to perform text preprocessing on multiple text type data to obtain multiple preprocessed text data; perform image preprocessing on multiple image type data to obtain multiple preprocessed image data; convert multiple document type data into multiple text type data based on a document parsing tool, and perform text preprocessing on the multiple text type data to obtain multiple preprocessed text data; convert multiple audio type data into multiple text type data based on a speech recognition tool, and perform text preprocessing on the multiple text type data to obtain multiple preprocessed text data; extract multiple key frames from multiple video type data, and perform image preprocessing on the multiple key frames to obtain multiple preprocessed image data.

[0148] In a possible implementation, the processing module 2 is also used to vectorize multiple first features to obtain multiple first feature vectors, and to vectorize multiple second features to obtain multiple second feature vectors; to concatenate multiple first feature vectors and multiple second feature vectors to obtain a first comprehensive feature vector; to standardize and normalize the first comprehensive feature vector to obtain a second comprehensive feature vector; to perform correlation analysis on the second comprehensive feature vector to obtain a feature correlation matrix; to perform mutual information analysis on the second comprehensive feature vector to obtain a feature mutual information matrix; and to screen the second comprehensive feature vector based on the feature correlation matrix and the feature mutual information matrix to obtain multiple correlation features.

[0149] In a possible implementation, processing module 2 is also used to normalize multiple numerical type features to obtain multiple numerical feature vectors; perform one-hot encoding on multiple non-numerical type features to obtain multiple non-numerical feature vectors; perform vectorization on multiple document type features to obtain multiple text feature vectors; and perform vectorization on multiple picture type features to obtain multiple image feature vectors.

[0150] In one possible implementation, processing module 2 is also used to set a correlation threshold and a mutual information threshold; based on the correlation threshold, the feature correlation matrix is ​​screened, and feature pairs with correlation greater than the correlation threshold are retained to obtain a first high correlation feature pair set; based on the mutual information threshold, the feature mutual information matrix is ​​screened, and feature pairs with mutual information greater than the mutual information threshold are retained to obtain a second high mutual information feature pair set; the first high correlation feature pair set and the second high mutual information feature pair set are intersected to obtain a comprehensive high correlation feature pair set; features are extracted from the comprehensive high correlation feature pair set to obtain multiple correlation features.

[0151] In a possible implementation, the processing module 2 is also used to input multiple third features into a preset classification model to output a first classification set; input multiple fourth features into a preset classification model to output a second classification set; and input multiple associated features into a preset classification model to output a third classification set.

[0152] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0153] The present application also discloses an electronic device. Figure 4 , Figure 4 The electronic device 400 may include: at least one processor 401 , at least one network interface 404 , a user interface 403 , a memory 405 , and at least one communication bus 402 .

[0154] The communication bus 402 is used to realize the connection and communication between these components.

[0155] The user interface 403 may include a display screen (Display) and a camera (Camera), and the optional user interface 403 may also include a standard wired interface and a wireless interface.

[0156] The network interface 404 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0157] Among them, the processor 401 may include one or more processing cores. The processor 401 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 405, and calling data stored in the memory 405. Optionally, the processor 401 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 401 can integrate one or more combinations of a central processing unit (Central Processing Unit, CPU), a graphics processor (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 401, and it can be implemented by a single chip.

[0158] Among them, the memory 405 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 405 includes a non-transitory computer-readable storage medium. The memory 405 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 405 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 405 may optionally be at least one storage device located away from the aforementioned processor 401. Refer to Figure 4 , the memory 405 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module and an application program.

[0159] exist Figure 4 In the electronic device 400 shown, the user interface 403 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 401 can be used to call an application stored in the memory 405, and when executed by one or more processors 401, the electronic device 400 executes one or more methods in the above-mentioned embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0160] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0161] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0162] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0163] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0164] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0165] The above are only exemplary embodiments of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present disclosure.

[0166] This application is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art not described in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. An automatic classification method for public resource transaction data. It is characterized in that The method comprises: Collecting public resource transaction data from various bidding announcement release channels, wherein the public resource transaction data includes first structured data and first unstructured data; Preprocessing the plurality of first structured data to obtain a plurality of preprocessed structured data; Preprocessing the plurality of first unstructured data to obtain a plurality of preprocessed unstructured data; Extracting a first feature from each of the preprocessed structured data, and extracting a second feature from each of the preprocessed unstructured data; Extracting associated features from the plurality of first features and the plurality of second features; Removing the associated features from the first features and the second features to obtain third features and fourth features; The plurality of third features, the plurality of fourth features, and the plurality of associated features are classified to obtain a plurality of classification results.

2. The method according to claim 1, It is characterized in that The preprocessed structured data includes preprocessed numerical type structured data and preprocessed non-numerical type structured data; The preprocessing of the plurality of first structured data to obtain a plurality of preprocessed structured data specifically includes: Deduplication processing is performed on the plurality of first structured data to obtain a plurality of second structured data; Performing missing value processing on the plurality of the second structured data to obtain a plurality of third structured data; the third structured data includes numerical type structured data and non-numerical type structured data; Performing standardization processing on the plurality of numerical type structured data to obtain the plurality of preprocessed numerical type structured data; The plurality of non-numeric structured data are encoded to obtain a plurality of pre-processed non-numeric structured data.

3. The method according to claim 1, It is characterized in that The first unstructured data includes text type data, image type data, document type data, audio type data and video type data; the preprocessed unstructured data includes preprocessed text data and preprocessed image data; The preprocessing of the plurality of first unstructured data to obtain a plurality of preprocessed unstructured data specifically includes: Performing text preprocessing on the plurality of text type data to obtain a plurality of preprocessed text data; Performing image preprocessing on the plurality of image type data to obtain a plurality of preprocessed image data; Based on a document parsing tool, the plurality of document type data are converted into a plurality of text type data, and text preprocessing is performed on the plurality of text type data to obtain a plurality of preprocessed text data; Converting the plurality of audio data to a plurality of text data based on a speech recognition tool, and performing text preprocessing on the plurality of text data to obtain a plurality of preprocessed text data; A plurality of key frames are extracted from the plurality of video type data, and image preprocessing is performed on the plurality of key frames to obtain a plurality of preprocessed image data.

4. The method according to claim 1, It is characterized in that The extracting associated features from the plurality of the first features and the plurality of the second features specifically includes: Performing vectorization processing on the plurality of the first features to obtain a plurality of first feature vectors, and performing vectorization processing on the plurality of the second features to obtain a plurality of second feature vectors; Concatenate a plurality of the first feature vectors and a plurality of the second feature vectors to obtain a first comprehensive feature vector; Standardizing and normalizing the first comprehensive feature vector to obtain a second comprehensive feature vector; Performing correlation analysis on the second comprehensive feature vector to obtain a feature correlation matrix; Performing mutual information analysis on the second comprehensive feature vector to obtain a feature mutual information matrix; The second comprehensive feature vector is screened based on the feature correlation matrix and the feature mutual information matrix to obtain a plurality of associated features.

5. The method according to claim 4, It is characterized in that The first feature includes a numerical type feature and a non-numerical type feature; the second feature includes a document type feature and a picture type feature; the first feature vector includes a numerical feature vector and a non-numerical feature vector; the second feature vector includes a text feature vector and an image feature vector; the vectorization of the plurality of the first features to obtain a plurality of first feature vectors, and the vectorization of the plurality of the second features to obtain a plurality of second feature vectors specifically include: Normalizing the plurality of numerical type features to obtain a plurality of numerical feature vectors; Performing one-hot encoding on the plurality of non-numeric type features to obtain a plurality of non-numeric feature vectors; Vectorizing the plurality of document type features to obtain a plurality of text feature vectors; Vectorization is performed on the plurality of picture type features to obtain a plurality of image feature vectors.

6. The method according to claim 4, It is characterized in that The screening of the second comprehensive feature vector based on the feature correlation matrix and the feature mutual information matrix to obtain a plurality of associated features specifically includes: Set correlation threshold and mutual information threshold; Screening the feature correlation matrix based on the correlation threshold, retaining feature pairs with correlations greater than the correlation threshold, and obtaining a first high-correlation feature pair set; The feature mutual information matrix is ​​screened based on the mutual information threshold, and feature pairs with mutual information greater than the mutual information threshold are retained to obtain a second high mutual information feature pair set; Performing an intersection operation on the first high correlation feature pair set and the second high mutual information feature pair set to obtain a comprehensive high correlation feature pair set; Features are extracted from the comprehensive high-correlation feature pair set to obtain a plurality of the correlation features.

7. The method according to claim 1, It is characterized in that The classification results include a first classification set, a second classification set, and a third classification set; the classification of the plurality of third features, the plurality of fourth features, and the plurality of associated features to obtain a plurality of classification results specifically includes: Inputting the plurality of third features into a preset classification model, and outputting the first classification set; Inputting the plurality of the fourth features into a preset classification model, and outputting the second classification set; The plurality of associated features are input into a preset classification model, and the third classification set is output.

8. An automatic classification system for public resource transaction data. It is characterized in that The system comprises: a data acquisition module and a processing module; The data collection module is used to collect public resource transaction data from various bidding announcement release channels, and the public resource transaction data includes first structured data and first unstructured data; The processing module is used to preprocess the first structured data to obtain a plurality of preprocessed structured data, and to preprocess the first unstructured data to obtain a plurality of preprocessed unstructured data; The processing module is further used to extract a first feature from each of the preprocessed structured data, and to extract a second feature from each of the preprocessed unstructured data; The processing module is further used to extract associated features from the plurality of the first features and the plurality of the second features; The processing module is further configured to remove the associated features from the plurality of first features and the plurality of second features, respectively, to obtain a plurality of third features and a plurality of fourth features, respectively; The processing module is further used to classify the multiple third features, the multiple fourth features and the multiple associated features to obtain multiple classification results.

9. An electronic device, It is characterized in that The electronic device (400) comprises a processor (401), a memory (405), a user interface (403) and a network interface (404), wherein the memory (405) is used to store instructions, the user interface (403) and the network interface (404) are used to communicate with other devices, and the processor (401) is used to execute the instructions stored in the memory (405) so that the electronic device (400) executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is executed.