Talent data competitiveness matching method for multi-source city data

Abstract

The invention discloses a talent data competitiveness matching method for multi-source city data, and belongs to the technical field of talent data, the talent data competitiveness matching method comprises the steps of collecting city data from multiple sources, performing deep analysis on preprocessed text data by using a natural language processing technology, calculating importance scores of words through a TF-IDF algorithm, and matching the talent data competitiveness of the multi-source city data. According to the technical scheme, the most representative key words are screened out, skill labels appearing in documents are automatically recognized and extracted by applying pattern matching or a deep learning model, a set of multi-layer professional role classification system is designed based on market research results, and layer-by-layer refinement is performed from macroscopic industry categories to specific position levels, so that the technical scheme of the invention is optimized. According to the method, the post release condition on a recruitment website is analyzed, a supervised learning algorithm is used for training an efficient classifier, any given skill combination is mapped to the most suitable professional role, and the model has the self-adaptive characteristic and can be continuously self-updated and improved along with the change of time and market requirements.

Description

Technical Field

[0001] The present invention belongs to the field of talent data technology, and specifically refers to a talent data competitiveness matching method for multi-source city data. Background Art

[0002] With the development of globalization and informatization, competition among cities is becoming increasingly fierce. Talents are the core resources for urban development. The evaluation and reasonable allocation of their competitiveness are crucial to the long-term development of cities. Most existing talent evaluation methods rely on a single data source, which makes it difficult to fully and accurately reflect the true ability and potential of talents.

[0003] However, the existing talent data competitiveness matching methods for multi-source city data still have certain defects. The existing talent data competitiveness matching methods for multi-source city data rely on simple statistical methods to extract keywords. This method is difficult to capture the true semantics of words and is prone to missing important information. The existing occupational role classification system is fixed and cannot reflect market changes and the emergence of new occupations in a timely manner, resulting in classification results lagging behind actual needs. There is a lack of sufficient training data and advanced algorithm support. The accuracy of existing classification models is low, and it is difficult to accurately map skill combinations to the most suitable occupational roles. Existing matching methods usually only consider the direct comparison between skill labels and target job requirements. When the two are not completely consistent, there is a lack of effective approximate matching means, resulting in inaccurate matching results. Therefore, a talent data competitiveness matching method for multi-source city data is proposed. Summary of the invention

[0004] The purpose of the present invention is to provide a talent data competitiveness matching method based on multi-source city data to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a talent data competitiveness matching method of multi-source city data, comprising the following steps:

[0006] S1. Collect urban data from multiple sources;

[0007] S2. Clean and preprocess the collected data to remove duplicate, erroneous and irrelevant information;

[0008] S3. Analyze text content through natural language processing technology to extract keywords and skill tags related to talents;

[0009] S4. Based on market trends, build a classification model that includes various job roles and their required skills;

[0010] S5, matching the parsed skill labels with the occupational roles in the classification model to determine the occupational category corresponding to each data point;

[0011] S6. Establish a scoring mechanism to assign scores to each occupational category based on factors such as scarcity of skills and market demand;

[0012] S7. Generate a talent competitiveness report for each city, which summarizes the scoring results of each occupational category and the overall evaluation of the city’s talent market.

[0013] Among them, S1 collects urban data from multiple sources; according to the target, lists all possible data sources, automatically crawls web page content through web crawlers and big data Internet, and performs preliminary preprocessing on the acquired data.

[0014] Among them, S2 cleans and preprocesses the collected data to remove duplicate, erroneous and irrelevant information; performs data cleaning on the data after preliminary preprocessing. If there are unique identifiers in the data, duplicates are directly removed through these identifiers. For data without unique identifiers, a text similarity algorithm is used to detect and delete records with highly similar content. For data with timestamps, the data is sorted according to the timestamp, and the latest records are retained, and old duplicate records are deleted.

[0015] Among them, S2 performs error detection and correction on the deduplicated data, detects outliers through statistical methods, checks whether the logical relationship between the data is consistent, defines a set of keyword lists to filter out content that is not related to talent competitiveness, applies topic models to classify texts by subject, and only retains topics related to talent. For comments or social media posts, sentiment analysis is used to identify negative emotions or irrelevant discussions, filter out content, standardize, convert formats and verify data, and check whether each record contains all necessary fields.

[0016] In which, S3 parses the text content through natural language processing technology to extract keywords and skill tags related to talents; extracts keywords based on the preprocessed data, and extracts keywords through TF-IDF. Let the word be t, the document be d, and the document set be D. The keyword extraction implementation formula is:

[0017]

[0018] In the formula, TF(t, d) represents the word frequency of word t in document d, IDF(t, D) represents the inverse document frequency of word t, and TF-IDF(t, d, D) represents the TF-IDF value of word t in document d.

[0019] Let K be a skill dictionary containing m skill entries k1, k2, ..., km. The goal of skill tag extraction is to find content from the text that matches the skill entries in the dictionary. Let MM be the matching function, then the implementation formula is:

[0020] L = {k i ∈K|M(w j ,k i )=1}

[0021] In the formula, L is the set of extracted skill labels, M(w j ,k i ) indicates whether the word wj matches the skill entry ki.

[0022] Among them, the S4, according to market trends, builds a classification model that includes multiple professional roles and their required skills; obtains job postings on recruitment websites, extracts frequently appearing job titles, skill requirements, salary levels and other information, preprocesses the obtained confidence and performs feature extraction, builds a multi-layer classification system, divided into primary classification, secondary classification and tertiary classification, the primary classification is divided by industry, the secondary classification is divided by function, and the tertiary classification is divided by job level. According to market changes and the emergence of new occupations, the classification system is regularly updated to build a classification model. The classification model performs classification according to the extracted features. Let the classification model be f(x; θ), the model parameter be θ, the goal is to minimize the loss function on the training set, and the classification model implementation formula is:

[0023]

[0024] In the formula, θ * represents the optimal model parameters, arg min represents finding the parameters that minimize the objective function, represents the sum of all N training samples, L represents the loss function, f(x i ; θ) represents the output of the classification model, x i represents the feature vector of the i-th sample, y i represents the true label of the i-th sample.

[0025] In S4, when new data flows in, the model parameter θ is updated using an incremental learning algorithm, and the implementation formula is:

[0026]

[0027] In the formula, θ t+1 represents the updated model parameters, θ * represents the optimized model parameters, η represents the learning rate, represents the gradient operator and L represents the loss function.

[0028] Among them, S5 matches the parsed skill labels with the occupational roles in the classification model to determine the occupational category corresponding to each data point; according to the parsed skill labels and the classification model results, the skill labels are accurately matched with the skill requirements in the occupational role description. If the skill labels fully meet the occupational role requirements in the classification system, they are directly classified into the occupational role, and the semantic similarity between the skill labels and the occupational role description is calculated for matching.

[0029] Among them, S6 establishes a scoring mechanism to score each occupational category based on factors such as skill scarcity and market demand; establishes a scoring mechanism to evaluate the importance of each scoring dimension, determine the weight of each dimension, and divide it into different scoring segments based on the performance of each dimension. The score of each dimension is multiplied by the corresponding weight, and then summed up to obtain the final total score. The final score is normalized to a fixed range, and personalized career recommendations are provided to users based on the scoring results.

[0030] Among them, S7 generates a talent competitiveness report for each city, which summarizes the scoring results of each occupational category and the overall evaluation of the city's talent market; obtains the scoring results of each occupational category from the previously established scoring mechanism, classifies the scoring results by city, evaluates the talent supply situation of each city, analyzes the talent supply and demand situation of the city in detail, and puts forward improvement suggestions and evaluations.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. The present invention removes duplicates, corrects erroneous records, fills in missing values, converts formats, etc. In particular, for data sets without unique identifiers, a text similarity algorithm is used to identify and merge highly similar entries. In addition, time series data is sorted and the latest version is retained to ensure the timeliness of the data;

[0033] 2. The present invention uses natural language processing technology to deeply analyze the pre-processed text data, calculates the importance score of words through the TF-IDF algorithm, and selects the most representative keywords from them. At the same time, a detailed skill dictionary is established, covering professional terms and technical vocabulary from all walks of life. Then, pattern matching or deep learning models are applied to automatically identify and extract skill tags appearing in the document, which not only helps to understand the individual's ability composition, but also provides a key basis for subsequent professional role matching;

[0034] 3. Based on the market research results, the present invention designs a multi-level occupational role classification system, which is refined layer by layer from macro industry categories to specific job levels. By analyzing the job postings on recruitment websites, we can capture the most popular job types and their corresponding skill requirements in the current market. On this basis, a supervised learning algorithm is used to train an efficient classifier to map any given skill combination to the most suitable occupational role. The model has adaptive characteristics and can continuously update and improve itself as time and market demand change;

[0035] 4. After constructing the classification model, the present invention links each data point with its most matching professional role and directly compares the known skill labels with the requirements of the target position. When the two are not completely consistent, the semantic similarity algorithm is used to find the best approximate solution, which can accurately locate the potential career direction of each candidate, thereby providing them with more targeted career development suggestions. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 The operation flow of the talent data competitiveness matching method of multi-source city data of the present invention is Figure 1 ;

[0037] Figure 2 The operation flow of the talent data competitiveness matching method of multi-source city data of the present invention is Figure 2 ;

[0038] Figure 3 The operation flow of the talent data competitiveness matching method of multi-source city data of the present invention is Figure 3 ;

[0039] Figure 4 The operation flow of the talent data competitiveness matching method of multi-source city data of the present invention is Figure 4 . DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Example

[0042] See also Figure 1-Figure 4 As shown, the present invention provides a technical solution: comprising the following steps:

[0043] S1. Collect urban data from multiple sources;

[0044] S2. Clean and preprocess the collected data to remove duplicate, erroneous and irrelevant information;

[0045] S3. Analyze text content through natural language processing technology to extract keywords and skill tags related to talents;

[0046] S4. Based on market trends, build a classification model that includes various job roles and their required skills;

[0047] S5, matching the parsed skill labels with the occupational roles in the classification model to determine the occupational category corresponding to each data point;

[0048] S6. Establish a scoring mechanism to assign scores to each occupational category based on factors such as scarcity of skills and market demand;

[0049] S7. Generate a talent competitiveness report for each city, which summarizes the scoring results of each occupational category and the overall evaluation of the city’s talent market.

[0050] Among them, S1 collects urban data from multiple sources; according to the target, lists all possible data sources, automatically crawls web page content through web crawlers and big data Internet, and performs preliminary preprocessing on the acquired data.

[0051] Among them, S2 cleans and preprocesses the collected data to remove duplicate, erroneous and irrelevant information; performs data cleaning on the data after preliminary preprocessing. If there are unique identifiers in the data, duplicates are directly removed through these identifiers. For data without unique identifiers, a text similarity algorithm is used to detect and delete records with highly similar content. For data with timestamps, the data is sorted according to the timestamp, and the latest records are retained, and old duplicate records are deleted.

[0052] Among them, S2 performs error detection and correction on the deduplicated data, detects outliers through statistical methods, checks whether the logical relationship between the data is consistent, defines a set of keyword lists to filter out content that is not related to talent competitiveness, applies topic models to classify texts by subject, and only retains topics related to talent. For comments or social media posts, sentiment analysis is used to identify negative emotions or irrelevant discussions, filter out content, standardize, convert formats and verify data, and check whether each record contains all necessary fields.

[0053] In which, S3 parses the text content through natural language processing technology to extract keywords and skill tags related to talents; extracts keywords based on the preprocessed data, and extracts keywords through TF-IDF. Let the word be t, the document be d, and the document set be D. The keyword extraction implementation formula is:

[0054]

[0055] In the formula, TF(t, d) represents the word frequency of word t in document d, IDF(t, D) represents the inverse document frequency of word t, and TF-IDF(t, d, D) represents the TF-IDF value of word t in document d.

[0056] Let K be a skill dictionary containing m skill entries k1, k2, ..., km. The goal of skill tag extraction is to find content from the text that matches the skill entries in the dictionary. Let MM be the matching function, then the implementation formula is:

[0057] L = {k i ∈K|M(w j ,k i )=1}

[0058] In the formula, L is the set of extracted skill labels, M(w j ,k i ) indicates whether the word wj matches the skill entry ki.

[0059] Among them, the S4, according to market trends, builds a classification model that includes multiple professional roles and their required skills; obtains job postings on recruitment websites, extracts frequently appearing job titles, skill requirements, salary levels and other information, preprocesses the obtained confidence and performs feature extraction, builds a multi-layer classification system, divided into primary classification, secondary classification and tertiary classification, the primary classification is divided by industry, the secondary classification is divided by function, and the tertiary classification is divided by job level. According to market changes and the emergence of new occupations, the classification system is regularly updated to build a classification model. The classification model performs classification according to the extracted features. Let the classification model be f(x; θ), the model parameter be θ, the goal is to minimize the loss function on the training set, and the classification model implementation formula is:

[0060]

[0061] In the formula, θ * represents the optimal model parameters, arg min represents finding the parameters that minimize the objective function, represents the sum of all N training samples, L represents the loss function, f(x i ; θ) represents the output of the classification model, xi represents the feature vector of the i-th sample, y i represents the true label of the i-th sample.

[0062] In S4, when new data flows in, the model parameter θ is updated using an incremental learning algorithm, and the implementation formula is:

[0063]

[0064] In the formula, θ t+1 represents the updated model parameters, θ * represents the optimized model parameters, η represents the learning rate, represents the gradient operator and L represents the loss function.

[0065] Among them, S5 matches the parsed skill labels with the occupational roles in the classification model to determine the occupational category corresponding to each data point; according to the parsed skill labels and the classification model results, the skill labels are accurately matched with the skill requirements in the occupational role description. If the skill labels fully meet the occupational role requirements in the classification system, they are directly classified into the occupational role, and the semantic similarity between the skill labels and the occupational role description is calculated for matching.

[0066] Among them, S6 establishes a scoring mechanism to score each occupational category based on factors such as skill scarcity and market demand; establishes a scoring mechanism to evaluate the importance of each scoring dimension, determine the weight of each dimension, and divide it into different scoring segments based on the performance of each dimension. The score of each dimension is multiplied by the corresponding weight, and then summed up to obtain the final total score. The final score is normalized to a fixed range, and personalized career recommendations are provided to users based on the scoring results.

[0067] Among them, S7 generates a talent competitiveness report for each city, which summarizes the scoring results of each occupational category and the overall evaluation of the city's talent market; obtains the scoring results of each occupational category from the previously established scoring mechanism, classifies the scoring results by city, evaluates the talent supply situation of each city, analyzes the talent supply and demand situation of the city in detail, and puts forward improvement suggestions and evaluations.

[0068] Working principle: By collecting urban data from multiple sources, using web crawlers and big data technology to automatically crawl web content, and pre-processing the acquired data to remove duplicate, erroneous and irrelevant information, de-duplicate and sort through unique identifiers, text similarity algorithms and timestamps, perform error detection and correction, including outlier detection, logical relationship consistency check and subject classification, standardize data, convert formats and verify data to ensure that each record contains necessary fields, use natural language processing technology to parse text content, extract talent-related keywords through algorithms such as TF-IDF, use skill dictionary matching algorithms to extract skill tags from text, and extract high-frequency job titles, skill requirements, salary levels and other information based on market trends and job postings on recruitment websites to build a multi-layer classification system. , including primary classification (industry), secondary classification (function) and tertiary classification (position level), use machine learning algorithms to build a classification model, and regularly update the model according to market changes and the emergence of new occupations, match the parsed skill labels with the occupational roles in the classification model, and determine the occupational category corresponding to each data point through methods such as exact matching and semantic similarity calculation, score each occupational category according to factors such as skill scarcity and market demand, evaluate the importance of each scoring dimension, determine the weight, multiply the score of each dimension by the corresponding weight, and sum up to get the final total score. Based on the scoring results, provide users with personalized career recommendations, obtain the scoring results of each occupational category from the scoring mechanism, classify the scoring results by city, evaluate the talent supply situation in each city, analyze the talent supply and demand situation in the city in detail, and put forward improvement suggestions and evaluations.

[0069] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

[0070] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.

Claims

1. A talent data competitiveness matching method for multi-source city data, characterized in that: The following steps are involved: S1. Collect urban data from multiple sources; S2. Clean and preprocess the collected data to remove duplicate, erroneous and irrelevant information; S3. Analyze text content through natural language processing technology to extract keywords and skill tags related to talents; S4. Based on market trends, build a classification model that includes various job roles and their required skills; S5, matching the parsed skill labels with the occupational roles in the classification model to determine the occupational category corresponding to each data point; S6. Establish a scoring mechanism to assign scores to each occupational category based on factors such as scarcity of skills and market demand; S7. Generate a talent competitiveness report for each city, which summarizes the scoring results of each occupational category and the overall evaluation of the city’s talent market.

2. According to claim 1, a talent data competitiveness matching method of multi-source city data is characterized by: The S1 collects urban data from multiple sources; according to the target, lists all possible data sources, automatically crawls web page content through web crawlers and big data Internet, and performs preliminary preprocessing on the acquired data.

3. The talent data competitiveness matching method of multi-source city data according to claim 1 is characterized by: S2, cleaning and preprocessing the collected data to remove duplicate, erroneous and irrelevant information; The data after preliminary preprocessing is cleaned. If there are unique identifiers in the data, deduplication is performed directly through these identifiers. For data without unique identifiers, a text similarity algorithm is used to detect and delete records with highly similar content. For data with timestamps, the data is sorted according to the timestamp, and the latest records are retained and the old duplicate records are deleted.

4. The talent data competitiveness matching method of multi-source city data according to claim 3 is characterized by: The S2 performs error detection and correction on the deduplicated data, detects outliers through statistical methods, checks whether the logical relationship between the data is consistent, defines a set of keyword lists to filter out content that is not related to talent competitiveness, applies topic models to classify texts by subject, and only retains topics related to talent. For comments or social media posts, sentiment analysis is used to identify negative emotions or irrelevant discussions, filter out content, standardize, convert formats, and verify data, and check whether each record contains all necessary fields.

5. The talent data competitiveness matching method of multi-source city data according to claim 1 is characterized by: S3, parses the text content through natural language processing technology, extracts keywords and skill tags related to talents; extracts keywords based on the preprocessed data, and extracts keywords through TF-IDF. Let the word be t, the document be d, and the document set be D. The keyword extraction implementation formula is: In the formula, TF(t, d) represents the word frequency of word t in document d, IDF(t, D) represents the inverse document frequency of word t, and TF-IDF(t, d, D) represents the TF-IDF value of word t in document d. Let K be a skill dictionary containing m skill entries k1, k2, ..., km. The goal of skill tag extraction is to find content from the text that matches the skill entries in the dictionary. Let MM be the matching function, then the implementation formula is: L={k i ∈K|M(w j ,k i )=1}, In the formula, L is the set of extracted skill labels, M(w j ,k i ) indicates whether the word wj matches the skill entry ki.

6. The talent data competitiveness matching method of multi-source city data according to claim 1 is characterized by: The S4, according to market trends, builds a classification model that includes multiple professional roles and their required skills; obtains job postings on recruitment websites, extracts frequently appearing job titles, skill requirements, salary levels and other information, preprocesses the obtained confidence and performs feature extraction, builds a multi-layer classification system, divided into primary classification, secondary classification and tertiary classification, the primary classification is divided by industry, the secondary classification is divided by function, and the tertiary classification is divided by job level. According to market changes and the emergence of new occupations, the classification system is regularly updated to build a classification model. The classification model performs classification according to the extracted features. Let the classification model be f(x; θ), the model parameter be θ, the goal is to minimize the loss function on the training set, and the classification model implementation formula is: In the formula, θ * represents the optimal model parameters, arg min represents finding the parameters that minimize the objective function, represents the sum of all N training samples, L represents the loss function, f(x i ; θ) represents the output of the classification model, x i represents the feature vector of the i-th sample, y i represents the true label of the i-th sample.

7. The talent data competitiveness matching method of multi-source city data according to claim 6 is characterized by: In S4, when new data flows in, the incremental learning algorithm is used to update the model parameters θ, and the implementation formula is: i t+1 =θ * -η▽ θ L(f(x t+1 ;θ t ),y t+1 ), In the formula, θ t+1 represents the updated model parameters, θ * represents the optimized model parameters, η represents the learning rate, ▽ θ represents the gradient operator and L represents the loss function.

8. The talent data competitiveness matching method of multi-source city data according to claim 1 is characterized by: The S5 matches the parsed skill labels with the occupational roles in the classification model to determine the occupational category corresponding to each data point; according to the parsed skill labels and the classification model results, the skill labels are accurately matched with the skill requirements in the occupational role description. If the skill labels fully meet the occupational role requirements in the classification system, they are directly classified into the occupational role, and the semantic similarity between the skill labels and the occupational role description is calculated for matching.

9. The talent data competitiveness matching method of multi-source city data according to claim 1 is characterized by: S6, establishes a scoring mechanism to score each occupational category based on factors such as skill scarcity and market demand; establishes a scoring mechanism to evaluate the importance of each scoring dimension, determine the weight of each dimension, divide each dimension into different scoring segments based on its performance, multiply the score of each dimension by the corresponding weight, and then sum them up to get the final total score, normalize the final score to a fixed range, and provide personalized career recommendations to users based on the scoring results.

10. The talent data competitiveness matching method of multi-source city data according to claim 1 is characterized by: S7 generates a talent competitiveness report for each city, which summarizes the scoring results of each occupational category and the overall evaluation of the city's talent market; obtains the scoring results of each occupational category from the previously established scoring mechanism, classifies the scoring results by city, evaluates the talent supply situation of each city, analyzes the talent supply and demand situation of the city in detail, and puts forward improvement suggestions and evaluations.

Images