A method and device for mining big data of a recruitment post

Abstract

The disclosure provides a recruitment post big data mining method and device, the method comprises: semantic normalization step: the original post name is processed by word segmentation, the word segmentation result is matched with the pre-constructed technical noun library and the function verb library, the core word item is extracted, and the normalized post name is generated according to the preset structured template reconstruction; skill extraction step: extracting skill keywords from the original post description corresponding to the normalized post name, generating a skill keyword set of the post category; feature vectorization step: calculating the term frequency-inverse document frequency of the core word item and the skill keyword set of the post category, generating a sparse feature vector of the post category; adaptive clustering step: based on the sparse feature vector of the post category, the clustering result is obtained.

Description

Technical Field

[0001] This disclosure belongs to the field of big data analysis and mining technology, and in particular relates to a method and apparatus for big data mining of recruitment positions. Background Technology

[0002] With the rapid development of online recruitment platforms, job posting data is becoming increasingly massive and unstructured. Currently, the analysis and mining of recruitment big data mainly faces the following technical challenges: 1. The standardization level of job description documents is low. Current technologies for processing job posting texts primarily employ simple string matching or keyword retrieval methods. Due to the autonomy of companies in publishing job information, the same job title often exhibits numerous heterogeneous expressions. For example, titles such as "Cloud Computing Development Engineer," "Cloud Platform Engineer," and "Cloud Computing Software Engineer" essentially point to the same technical field, but current technologies cannot effectively identify and normalize these heterogeneous expressions. This leads to feature sparsity and reduced word frequency weights in subsequent analysis, severely impacting clustering results and data quality.

[0003] 2. The parameter settings of clustering algorithms lack adaptability. Traditional K-means clustering algorithms require pre-setting the number of clusters K and randomly selecting initial centroids. When dealing with dynamically changing recruitment data, fixed parameter settings are difficult to adapt to different data distribution characteristics, resulting in unstable clustering quality. Although the Canopy algorithm can initially determine the K value through a distance threshold, current technologies typically rely on manual experience to preset the threshold, lacking the ability to adapt to the statistical characteristics of the data itself. This leads to significant arbitrariness in threshold setting and insufficient reliability of clustering results.

[0004] Therefore, there is an urgent need for a big data mining method that can deeply optimize recruitment data based on its characteristics and achieve text normalization and adaptive clustering collaboration. Summary of the Invention

[0005] This disclosure provides a method and apparatus for big data mining of job postings, which can effectively solve the above-mentioned problems.

[0006] This disclosure is implemented as follows: Firstly, this disclosure provides a method for big data mining of job postings, the method comprising: Semantic normalization steps: The original job title is segmented into words, and the segmentation results are matched with a pre-built technical term library and a functional verb library to extract core terms. The normalized job title is then reconstructed according to a preset structured template. The core terms include technical terminology and functional role verbs. Skill extraction steps: Extract skill keywords from the original job description corresponding to the standardized job name to generate a set of skill keywords for the job category, wherein the job category is jointly represented by the standardized job name and the corresponding set of skill keywords; Feature vectorization step: Perform term frequency-inverse document frequency calculation on the core terms and skill keyword set of the job category to generate a sparse feature vector for the job category; Adaptive clustering steps: Based on the sparse feature vectors of the job categories, perform the following operations to obtain the clustering results: The Canopy algorithm is executed to automatically generate initial cluster centers and K values, wherein the distance threshold of the Canopy algorithm is dynamically determined based on the statistical distribution of the distance between sample vectors in the sparse feature vector; Using the initial cluster centers and K values ​​as input, perform K-means iterative clustering until convergence.

[0007] Secondly, this disclosure provides a big data mining device for job postings, the device comprising: The semantic normalization module is used to perform word segmentation on the original job title, match the segmentation results with a pre-built technical term library and functional verb library, extract core terms, and reconstruct and generate a normalized job title according to a preset structured template. The core terms include technical terminology and functional role verbs. The skill extraction module is used to extract skill keywords from the original job description corresponding to the standardized job name, and generate a set of skill keywords for the job category, wherein the job category is jointly represented by the standardized job name and the corresponding set of skill keywords. The feature vectorization module is used to perform word frequency-inverse document frequency calculation on the core terms and skill keyword set of the job category to generate a sparse feature vector for the job category. The adaptive clustering module is used to perform the following operations based on the sparse feature vectors of the job categories to obtain clustering results: The Canopy algorithm is executed to automatically generate initial cluster centers and K values, wherein the distance threshold of the Canopy algorithm is dynamically determined based on the statistical distribution of the distance between sample vectors in the sparse feature vector; Using the initial cluster centers and K values ​​as input, perform K-means iterative clustering until convergence.

[0008] Thirdly, this disclosure provides an electronic device, including: Memory, the memory storing execution instructions; and A processor that executes execution instructions stored in the memory, causing the processor to perform the method described in the first aspect.

[0009] Fourthly, this disclosure provides a readable storage medium storing executable instructions, which, when executed by a processor, are used to implement the method described in the first aspect.

[0010] Compared with the prior art, the beneficial effects of this disclosure are: This disclosure provides a method and apparatus for big data mining of job postings. By performing word segmentation matching and structured reconstruction on the original job titles, it effectively identifies and normalizes heterogeneous expressions, significantly improving the TF of core keywords. IDF weights; simultaneously, the Canopy algorithm threshold is dynamically determined based on the mean and standard deviation of the distance between sample vectors, automatically generating initial cluster centers and K values, realizing adaptive determination of clustering parameters, avoiding the blindness of manual presets, thereby significantly improving the stability and accuracy of clustering results; and combined with the complete extraction of skill keywords, an end-to-end automated big data mining process for recruitment positions is formed, which has good scalability and practical value. Attached Figure Description

[0011] Figure 1 This is a flowchart of the recruitment job big data mining method S100 provided in this embodiment.

[0012] Figure 2 This is an example of a text dataset of job postings provided in this embodiment of the disclosure.

[0013] Figure 3 yes Figure 2 Examples of corresponding normalized text datasets.

[0014] Figure 4 yes Figure 2 Information was extracted by clustering some technical positions.

[0015] Figure 5 yes Figure 4 The weight distribution of key skills for cloud computing positions.

[0016] Figure 6 This is a flowchart of the city-province mapping algorithm provided in this embodiment.

[0017] Figure 7 This is a schematic diagram of the structure of the recruitment job big data mining device 1000 provided in this embodiment. Detailed Implementation

[0018] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0019] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0021] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure. The singular forms “a,” “the,” and “the” used in the embodiments of this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0022] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0023] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0024] The terms "first" and "second" used herein are merely to distinguish similar objects and do not represent a specific ordering of the objects. Understandably, the specific order or sequence of "first" and "second" can be interchanged where permitted. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein.

[0025] Example 1 Please refer to Figure 1 This disclosure provides a method for big data mining of job postings, S100.

[0026] Specifically, method S100 includes: S102, Semantic normalization step: The original job title is segmented into words, the segmentation results are matched with a pre-built technical term library and a functional verb library, core terms are extracted, and a normalized job title is generated by reconstructing according to a preset structured template. The core terms include technical terminology and functional role verbs. S104, Skill extraction step: Extract skill keywords from the original job description corresponding to the standardized job name to generate a set of skill keywords for the job category, wherein the job category is jointly represented by the standardized job name and the corresponding set of skill keywords; S106, Feature vectorization step: Perform word frequency-inverse document frequency calculation on the core terms and skill keyword set of the job category to generate a sparse feature vector for the job category; S108, Adaptive clustering step: Based on the sparse feature vectors of the job categories, perform the following operations to obtain the clustering results: The Canopy algorithm is executed to automatically generate initial cluster centers and K values, wherein the distance threshold of the Canopy algorithm is dynamically determined based on the mean and standard deviation of the distances between sample vectors in the sparse feature vector; Using the initial cluster centers and K values ​​as input, perform K-means iterative clustering until convergence.

[0027] Specifically, this method mines job posting texts, which include the original job title and the original job description.

[0028] Figure 2 The image shows a dataset of job postings with the keyword "cloud computing" collected from a recruitment website on April 15, 2025. Each row represents a job posting, and the fields from left to right include job title, posting date, company name, salary, location, experience, education, job description, and link address. There are a total of 464 data entries.

[0029] The pre-defined structured template in the semantic normalization step is: technical terminology + functional role verb + job title suffix. The technical terminology (N) identifies the core technical direction of the job. The functional role verb (V) identifies the main job functions. The technical terminology comes from a pre-built terminology library, and the functional role verb comes from a pre-built verb library. For each original job title, N and V are obtained through word segmentation and lexicon matching. The job title suffix is ​​selected from a pre-defined set of suffixes, for example, "engineer". The structured template is designed to eliminate non-core modifiers (such as "senior" or "experienced") in job titles, while unifying synonymous and heterogeneous expressions into a standardized format.

[0030] The core of this method relies on a pre-built, multi-level categorized dictionary. This dictionary is specifically built and maintained based on the technical field or business domain of the job title category. For example, for job postings in the "cloud computing" field. Dedicated multi-level categorized dictionaries are constructed to suit the characteristics of each technical field.

[0031] The categorized dictionary database has three preset core lexicons: Technical Field Terminology Database: Used to store technical field terms N, such as "cloud computing," "Java," and "big data." Functional role verb library: Used to store functional role verbs V, such as "development", "operations", and "testing".

[0032] Skill Keyword Library: Used to store skill keywords S, such as "Hadoop", "Spring", "Kubernetes", and "machine learning".

[0033] The categorized dictionary database employs a dynamic addition mechanism to ensure its timeliness. Regular analysis of word frequency statistics from acquired job posting data is performed, and newly emerging high-frequency technical terms (such as "K8s" and "Hongmeng") are dynamically added to the corresponding database to ensure that the database reflects technological development trends in a timely manner.

[0034] Each thesaurus is loaded directly into the memory of the compute node in the form of a configuration file to improve the access speed of thesaurus matching.

[0035] The specific process of the semantic normalization step includes: (1) Token segmentation and parsing: The tokenizer is called to perform streaming processing on the input original job name string, breaking it down into fine-grained tokens. Specifically, Lucene's StandardAnalyzer is used to convert the original job name into a TokenStream, and the text content of each token is extracted using the CharTermAttribute attribute, thus obtaining a term sequence.

[0036] (2) Feature classification and matching: Noun N extraction: Traverse the tokens and match them with the technical terminology database. Verb V extraction: Traverse the tokens and match them with the functional role verb database.

[0037] (3) When multiple nouns N or multiple verbs V are matched from the word segmentation results, the semantic normalization step adopts the following conflict handling mechanisms respectively: (i) First-to-first matching principle: Select the first successfully matched term as the final extraction result according to the order in which each core term appears in the original job title. For example, for the original job title "Senior Java Cloud Computing Development Engineer", the word segmentation sequence is ["Senior", "Java", "Cloud Computing", "Development", "Engineer"]. Among them, "Java" and "Cloud Computing" are terms in the noun database. Select "Java" as the core technology term according to the order of appearance.

[0038] (ii) Core Domain Priority Principle: Selection is based on a predefined verb priority hierarchy. The priority hierarchy can be set according to actual needs, for example: Development > Operations > Testing > Integration > Other. The term with the highest priority is selected as the final extracted verb V. For example, for the original job title "Cloud Computing Development and Operations Engineer", both "Development" and "Operations" are matched. "Development" is selected as the verb V according to priority, and the final standardized job title generated is "Cloud Computing Development Engineer".

[0039] In practice, one of the above principles may be adopted.

[0040] (4) For long-tail data or non-standard descriptions, the algorithm adopts the following strategies to ensure model robustness: (a) Missing verb completion: If the name contains only the noun N (such as "Java engineer"), the default verb V "development" will be automatically filled in.

[0041] (ii) Discarding noisy data: If a certain original job title cannot be matched with any term in the noun database, the system will automatically determine that the corresponding job posting data is noisy data and discard it, thereby ensuring the purity of the clustering data.

[0042] In the skill extraction step, to accurately and efficiently extract skill keywords S that characterize the job skill requirements from the original job description, this step employs a matching and filtering mechanism based on a pre-built skill keyword library. Each term obtained from word segmentation of the original job description is iterated to determine if it exists in the keyword library. If it does, the term is added to the skill keyword set for that job category (e.g., {Hadoop, Spark, Java, Docker}). If a particular original job description cannot match any core term in the keyword library, the system automatically determines that the corresponding job posting data is noise and discards it.

[0043] In some implementations, if the original job description matches fewer than three core terms in the keyword library, the system automatically determines that the corresponding job posting data is noise data and discards it.

[0044] The count is set to the matched skill keywords. The business logic for threshold selection is shown in Table 1.

[0045] Table 1. Business Logic for Threshold Selection

[0046] In some implementations, the skill extraction step includes: The original job description is matched with a pre-built dictionary of compound terms to identify and retain compound technical terms. The remaining text is segmented using a general word segmenter to obtain a word sequence; The composite technical term and the terms in the term sequence are compared with a pre-built keyword library to filter out the terms that exist in the skill point term library, which are then used as skill keywords S.

[0047] General-purpose word segmenters tend to break down complex technical terms into meaningless fragments, resulting in a failure to match core terms. To address this issue, this step employs a two-stage strategy that prioritizes matching complex terms: Phase 1: Compound Word Dictionary Matching A pre-built compound word dictionary (dictionary2) is used to store easily split compound technical terms such as "data warehouse," "load balancing," "machine learning," "C++," "TCP / IP," and "ASP.NET." Before formal word segmentation, the original job description text is scanned to match compound words in the dictionary. Matched compound words are stored as a single term and marked in the text (e.g., by replacing them with special placeholders or recording their positions) to prevent subsequent word segmentation from breaking them down.

[0048] Phase Two: General Word Segmentation and Skill-Based Lexicon Matching After removing compound words, the remaining text is segmented using a general word segmenter to obtain a word sequence. The compound words extracted in stage one are merged with the word segmentation results from stage two and then uniformly matched and filtered against a keyword database. This strategy ensures that "data warehouse" is recognized as a whole, rather than being split into two meaningless non-core words: "data" and "warehouse".

[0049] For example, a snippet from the original job description: "Familiar with data warehouse modeling, proficient in Hadoop and Spark, and knowledgeable about TCP / IP protocol." Compound word dictionary matching: Matches "data warehouse" and "TCP / IP" and retains them as a whole.

[0050] General word segmentation: Remaining word segmentation results: ["familiar", "modeling", "master", "Hadoop", "and", "Spark", "understand", "protocol"].

[0051] Combined terms: ["data warehouse", "TCP / IP", "familiar", "modeling", "master", "Hadoop", "and", "Spark", "understand", "protocol"].

[0052] Keyword matching: Matching results: {"data warehouse", "Hadoop", "Spark", "TCP / IP"}.

[0053] Output skill keyword set: {data warehouse, Hadoop, Spark, TCP / IP}.

[0054] By extracting dual-mode skill features from the original job descriptions, prioritizing matching of job descriptions based on a compound word dictionary to identify compound words, segmenting the remaining text and filtering it based on a skill lexicon to extract core terms and remove common interference words, a set of skill keywords with skill representation significance can be generated. This provides high-quality and high-purity input features for the feature vectorization step, thereby improving the effectiveness and accuracy of cluster analysis.

[0055] Figure 2 After the semantic normalization and skill extraction steps are performed on the data, the normalized text dataset is as follows: Figure 3 As shown in the figure. The first three fields of each row are standardized job titles, and the fourth field and subsequent fields are tab-separated lists of skill keywords. Figure 2 When importing data into the big data platform Hive, a keyword search revealed that before standardization, there were 2 job titles matching "cloud computing development" and 5 matching "cloud computing operations and maintenance." For example... Figure 3As shown, the standardized "cloud computing development engineer" has 41 entries, while the standardized "cloud computing operations and maintenance engineer" has 28 entries. This demonstrates that job title standardization allows for efficient aggregation of word frequencies previously scattered across multiple variant titles, significantly increasing the TF weight of core keywords and enabling subsequent K-means clustering to more accurately identify the skill characteristics of job groups.

[0056] In some implementations, the semantic normalization step and / or the skill extraction step are executed within a parallel computing framework to address the efficiency requirements of processing massive amounts of job posting data. Parallelism includes, but is not limited to, single-machine multi-core processors and distributed clusters.

[0057] In some implementations, the MapReduce parallel computing model is used: Map Phase: Each computing node performs the following operations in parallel on local data shards: semantic parsing and normalization reconstruction of job titles, and dual-modal skill feature extraction of job descriptions. Each processed record outputs a key-value pair, where the key is the normalized job title and the value is the corresponding set of skill keywords.

[0058] Reduce phase: Group all key-value pairs output from the Map phase by key, and aggregate all values ​​corresponding to the same key. Merge and frequency-aggregate the skill keywords in each value to finally generate a complete set of skill keywords for that job category.

[0059] Specifically, the original job posting dataset is divided into multiple data shards, with each shard assigned to a computing node. Each node independently and in parallel performs semantic normalization and skill extraction tasks on its shard, and finally aggregates and outputs the results. Typically, these two steps are completed in the same Mapper, so that normalized names and skill keyword sets are output simultaneously in a single scan, reducing data reading overhead.

[0060] During the Map phase, each compute node performs the following operations on each record in its assigned data shard: Input: Each record contains the job posting text.

[0061] Semantic normalization: The original job titles are segmented, matched with a thesaurus, conflict-handling and reconstructed using structured templates to generate normalized job titles.

[0062] Skill extraction: For the corresponding original job description, strategies such as priority matching of compound words and filtering of skill point thesaurus are used to extract a set of skill keywords.

[0063] Output key-value pairs: Use the normalized job title as the key and the set of skill keywords for that job title as the value.

[0064] Because the processing of different records is independent, the Map phase can achieve a near-linear speedup.

[0065] The Reduce phase includes the following operations: Shuffle: The framework automatically sorts and groups all key-value pairs output by Map by key, bringing together all values ​​with the same key.

[0066] Reduce processing: For each key, merge the sets of multiple skill keywords corresponding to it, and count the total frequency of each skill keyword.

[0067] Output: The final output is a complete set of skill keywords for each job category, which will be used in the feature vectorization step.

[0068] By performing semantic normalization and / or skill extraction steps within a parallel computing framework, this method can fully utilize the computing resources of the cluster, achieving near-linear scaling of preprocessing time and data size. Furthermore, the parallel steps can seamlessly integrate with distributed clustering steps. For example, when using the MapReduce parallel model, its output format is directly compatible with the input format of the adaptive clustering step, eliminating the need for data format conversion, thus forming an end-to-end big data mining pipeline.

[0069] After completing the semantic normalization and skill extraction steps, each job category (i.e., each normalized job title) is associated with the following two types of information: structured job title terms: including extracted technical terminology (N) and functional role verbs (V); and a set of skill keywords: a set of terms extracted from the job description that represent skills. The purpose of the feature vectorization step is to convert these discrete text features into computer-computable numerical vectors for subsequent clustering analysis. Specifically, a vector space model is constructed using the Term Frequency-Inverse Document Frequency (TF-IDF) technique, and sparse feature vectors are generated to save storage and computational resources.

[0070] In some implementations, the feature vectorization step uses the normalized job title as a whole term, and performs term frequency-inverse document frequency calculation together with the set of skill keywords.

[0071] 1. Corpus Construction The standardized job titles and skill keyword sets for all job categories are merged into a document set, with one document corresponding to each job category. Specifically, for each job category, its structured job title terms and skill keyword set are concatenated into a text string, which serves as the original document for that category.

[0072] 2. Word frequency calculation For each document (i.e., each job category), count the occurrences of each term. Specifically, for each term... In the document word frequency in The original frequency count was used.

[0073] 3. Inverse document frequency calculation Inverse document frequency Reflecting a term The prevalence of the term across the entire corpus (all job categories). The calculation formula is: .in, This represents the total number of documents in the corpus (i.e., the total number of job categories). This term is included Number of documents (i.e., terms) (The number of job category feature sets a term appears in). The fewer documents a term appears in, the higher its IDF value, indicating that the term has stronger discriminative power.

[0074] 4. TF-IDF weight calculation Terms In each document The final weights in the formula are: .

[0075] For each job category document, calculate the TF-IDF weights of all its terms to obtain a vector. The dimension of the vector is equal to the total number of different terms in the corpus, and the value of each dimension is the TF-IDF weight of the corresponding term.

[0076] 5. Sparse Eigenvector Representation Since the actual number of terms in a corpus may reach tens of thousands or even hundreds of thousands, but the actual number of terms appearing in each job category document is usually only tens to hundreds, the generated TF-IDF vectors are sparse (the vast majority of dimensions are 0). To save storage space and subsequent computational overhead, this step uses a sparse vector format for storage, for example, only recording a list of key-value pairs with non-zero dimensions and their weights.

[0077] For example, assuming the corpus has 10,000 terms, and the feature vector of a certain job category has only 50 non-zero dimensions, the storage format is: [(term index 1, weight 1), (term index 2, weight 2), …, (term index 50, weight 50)].

[0078] The generated sparse feature vectors are directly used as input to the adaptive clustering step. Since clustering algorithms (such as Canopy and K-means) require calculating the distance between vectors (such as Euclidean distance or cosine similarity), sparse vector representation can efficiently support these operations, requiring only calculations on non-zero dimensions and avoiding traversal of all dimensions.

[0079] This step, through the IDF factor of TF-IDF, can automatically reduce the weight of common high-frequency words (such as "development"), highlight the specific terms corresponding to different technical fields, and achieve effective control of feature dimensions; the use of sparse vector representation greatly reduces storage and computation overhead, which is especially suitable for distributed processing of massive data; in addition, the output format of feature vectorization is directly compatible with the distance calculation function in the adaptive clustering step, without the need for data format conversion, thus achieving seamless connection of the process.

[0080] In some implementations, the TF value of the core terms is enhanced by repeating each normalized job title.

[0081] In the text vectorization process, each document (i.e., each job category) is composed of core terms from the normalized job title (e.g., "cloud computing development engineer") and a set of skill keywords. Term frequency (TF) is a crucial component of the TF-IDF weight. By artificially repeating the normalized job title multiple times, the frequency of occurrence of the core terms can be multiplied, thereby increasing their TF value and thus their TF-IDF weight. Since the IDF value remains constant (IDF only depends on global document frequency), the weight of core terms is significantly amplified relative to other terms. In subsequent clustering distance calculations (e.g., cosine similarity), these high-weight dimensions will have a greater impact on the vector direction, making the clustering results more inclined to be divided according to core technical fields and functional roles. The optimal number of repetitions is 2-3 times.

[0082] A typical implementation method is to write the standardized job title consecutively into multiple fields of the document or to repeatedly concatenate them when constructing the document. For example: Document content: Original: Cloud Computing Development Engineer Hadoop Spark Java; Enhanced: Cloud Computing Development Engineer Cloud Computing Development Engineer Cloud Computing Development Engineer Hadoop Spark Java. The number of times "Cloud Computing Development Engineer" appears in the document has increased from 1 to 3, while other skill keywords remain at their original frequency.

[0083] In some implementations, the feature vectorization step includes: Different weighting coefficients are applied to the standardized job titles and the skill keywords, wherein the weighting coefficient of the core terms is higher than that of the skill keywords.

[0084] This differentiated weighting achieves the same goal as the aforementioned "repeated normalized job titles," enabling clustering algorithms to rely more heavily on the core terms in normalized job titles when calculating vector distances, thereby grouping job categories with similar technical fields and / or similar functional roles together.

[0085] The adaptive clustering step automatically groups the sparse feature vector set generated in the feature vectorization step to discover job categories with similar skill characteristics. This step uses Canopy-Kmeans hybrid clustering and mainly includes two sub-stages: 1. Canopy algorithm (adaptive threshold): Based on the statistical feature of the distance between sample vectors, it automatically generates initial cluster centers and K values. 2. K-means iterative clustering: Using the output of Canopy as initial parameters, it performs standard K-means iterations until convergence, outputting the final clustering results.

[0086] The Canopy algorithm coarsely partitions the data using two distance thresholds (loose threshold T1 and strict threshold T2, T1>T2), each partition being called a Canopy. A data point can belong to multiple Canopies, but the center of each Canopy can serve as the initial centroid of a K-means dataset; the number of Canopies is the K value. Traditional Canopy algorithms require manually setting T1 and T2; this step dynamically determines these thresholds using distance statistics from the sample data. Specifically, a certain number of samples (e.g., 10% of the total or a fixed number, such as 1000) are randomly selected from the sparse feature vector set, and the distance d between every two vectors in the sample is calculated. The distance metric can be Euclidean distance or cosine similarity (preferably cosine similarity). This results in a distance set D = {d1, d2, …, d m}. The so-called "statistical distribution" refers to various statistical measures calculated from sample data that can reflect the central tendency and dispersion of distances, including but not limited to the mean, standard deviation, quantiles, and extreme values of distances. The adaptive threshold determination method based on statistical distribution can eliminate the blindness of manual parameter tuning, make the threshold completely data-driven, automatically adapt to different data distributions, avoid over-coarse or over-fine partitioning caused by fixed thresholds, and at the same time have relatively low computational overhead.

[0087] In some embodiments, the adaptive clustering step includes: The distance thresholds of the Canopy algorithm include a loose threshold T1 and a strict threshold T2, and are calculated according to the following formula: T1 = μ + k1·σ, T2 = μ - k2·σ, where μ is the mean of the distances between the sample vectors, σ is the standard deviation of the distances between the sample vectors, k1 and k2 are preset coefficients and T1 > T2. Exemplarily, k1 takes a value of 1.0 and k2 takes a value of 0.5, which can be adjusted according to the data characteristics.

[0088] For the recruitment position data preprocessed by semantic normalization and skill extraction, the distance distribution between feature vectors is relatively regular and there are few outliers. Therefore, the mean + standard deviation is used to determine the adaptive threshold, which is simple to calculate and supports distributed incremental aggregation. Only the sum and sum of squares are required, without global sorting or approximate algorithms, and is naturally compatible with frameworks such as MapReduce. At the same time, the statistical significance of this method is clear, which is convenient for unified parameter setting, can achieve a good balance between efficiency and robustness, and does not require re-tuning parameters for different data sets.

[0089] The Canopy generation process includes: Input: A set V of sparse feature vectors, thresholds T1, T2.

[0090] Step 1: Mark all vectors in V as "unprocessed". Step 2: Randomly select an unprocessed vector as the center C of the current Canopy. Step 3: Calculate the distance dist(C, v) between C and all other vectors v in V. If dist(C,v) < T1, then include v in the current Canopy. If dist(C, v) < T2, then remove v from the candidate set (i.e., it will no longer be the center of other Canopies). Step 4: Repeat steps 2 - 3 until the candidate set is empty.

[0091] Output: A set of Canopies (each Canopy contains several vectors), the set of centers of the Canopies is used as the initial centroids of K-means, and the number of Canopies is used as the value of K.

[0092] Taking the initial centroid set and the value of K output by Canopy as input, execute the standard K-means algorithm: Assignment steps: For each sparse feature vector, calculate its distance to each centroid and assign it to the cluster represented by the nearest centroid.

[0093] Update steps: For each cluster, calculate the mean (center point) of all vectors within the cluster, and use it as the new centroid.

[0094] Convergence criterion: Repeat the assignment and update steps until the change in centroid is less than a preset threshold (e.g., 1×10). -4 The iterations can continue until the maximum number of iterations (e.g., 50) is reached. The final output is the label of each cluster and the set of vectors within each cluster, which is the clustering result.

[0095] In some implementations, the adaptive clustering step is performed within a distributed computing framework, including: Each computing node calculates distance statistics based on local sparse feature vector partitioning, and the distance statistics include local mean and local standard deviation; The local distance statistics of each node are aggregated to generate global statistics. A global adaptive threshold is determined based on the global statistics. The global adaptive threshold includes a global loose threshold and a global strict threshold. Each computing node performs Canopy partitioning based on the global adaptive threshold to generate a local center vector; The local centroid vectors of each node are aggregated to generate a global initial centroid set.

[0096] Specifically, each computing node reads a sparse feature vector slice allocated to its local area (i.e., a subset of the global sparse feature vectors). The node randomly samples a certain number of sample vectors from this slice, calculates the distance between all pairwise vectors in the sample, and thus obtains the node's local distance statistics, including the local mean (μ). local ) and local standard deviation (σ) local The local statistics (μ) calculated by all nodes (Map task) local σ local Sample size n local The values, as key-value pairs, are aggregated into a single Reduce task by the MapReduce framework. The mean and standard deviation are aggregatable; the global statistic can be reconstructed from the local statistics of each node. This Reduce task uniformly calculates the global mean μ. global and global standard deviation σ global Based on this, the global loose threshold T1 is determined. global and global strict threshold T2 global .

[0097] The calculated global threshold T1 global T2 globalThe data is distributed to all Map tasks. Each Map task uses the exact same global threshold to partition its local sparse feature vectors and executes the standard Canopy algorithm to generate local centroid vectors, outputting them as key-value pairs (e.g., using a fixed key). The local centroid vectors output by all Map tasks are automatically aggregated into a single Reduce task by the MapReduce framework. A simple merging and deduplication process is performed on all collected local centroid vectors to obtain the initial global centroid set. This set can be directly used as input for subsequent K-means iterative clustering without the need for complex secondary merging or re-clustering.

[0098] This method, through the collaboration of adaptive thresholds and distributed computing algorithms and architectures, leverages the aggregatable characteristics of statistics and the natural fit with the aggregation patterns of distributed frameworks. It not only achieves global consistency of thresholds, avoiding the additional overhead of secondary merging or threshold reconciliation required in traditional distributed clustering, but also significantly reduces communication and computing overhead in a distributed environment. Thus, while ensuring the quality of global clustering, it achieves efficient, stable, and scalable big data clustering for job postings. Figure 4 The data mined after job clustering is displayed. Figure 5 Further demonstration Figure 4 The weight distribution of key skills for cloud computing positions is visualized. The higher the weight value of a skill point within a job category, the greater its relevance to the job and the more important it is to that job category.

[0099] In some implementations, the method further includes a big data acquisition step: acquiring job posting text data from recruitment websites and storing it in a data warehouse for use in subsequent steps.

[0100] For example, obtain raw job posting data containing fields such as job title, job description, salary, and location from recruitment websites; store the obtained data in a MySQL database; and use the Sqoop tool to import the data from MySQL into a Hive data warehouse and HDFS distributed file system to form the raw dataset.

[0101] In some embodiments, the method further includes a clustering result skill feature analysis step: for each cluster in the clustering result, the frequency of occurrence of the skill keywords within the cluster is counted, and a number of the most frequently occurring skill keywords are extracted to form a core skill combination; and representative skill features are extracted based on the distribution differences of each skill keyword within and between clusters; and a skill feature fingerprint of the cluster is generated based on the core skill combination and the representative skill features.

[0102] In some implementations, the method further includes visualizing the clustering results to support human verification and decision-making.

[0103] In some embodiments, the method further includes: Multidimensional feature preprocessing steps: Normalize and map the multidimensional discrete features of the job posting data, wherein the multidimensional discrete features include at least one of geographical information and salary information; Data slicing steps: Based on the normalized discrete features, perform semantically adaptive slicing to generate slicing datasets; The semantic normalization step is performed on the fragmented dataset.

[0104] Job posting data typically includes multiple discrete features beyond job titles and descriptions, such as geographic information (city, province), salary information, education requirements, and work experience. While these features don't directly participate in the semantic normalization of job titles, they can be used to pre-group the data, making subsequent processing more domain-consistent. In actual recruitment big data, the skill features of job categories are often coupled with these multi-dimensional discrete features. For example, the distribution of skill keywords differs significantly for the same job category (e.g., cloud computing development engineer) or business domain under different geographic discrete feature values ​​(e.g., first-tier cities versus third-tier cities). If clustering is directly performed on the entire dataset, the strong coupling between skill clusters and discrete feature values ​​leads to clustering results primarily distributed along discrete feature gradients (e.g., clustering by region) rather than reflecting true skill similarity. In this case, the skill purity of the clusters is low, making fair comparison of skill patterns under different discrete feature conditions difficult and resulting in insufficient cross-condition comparability.

[0105] This step involves data slicing to group job data with similar or identical discrete feature values ​​into the same slice, achieving slice isolation. This isolation mechanism makes the skill distribution structure within a slice independent of the influence of other discrete feature values, with the variance of discrete features within a slice approaching zero, and skill differences becoming the dominant direction of variance. Skill feature extraction and adaptive clustering are performed independently on each slice dataset. Distance statistics are dynamically calculated within each slice, and adaptive thresholds are determined to adapt to the local distribution characteristics of the data in that slice. Cluster boundaries are cut along the direction of skill features, generating high-purity skill clusters, significantly improving the identifiability of skill patterns. Simultaneously, the skill clusters generated by independent clustering within each slice are locally comparable, providing a foundation for subsequent cross-slice comparative analysis.

[0106] As shown in Table 2, taking salary information as an example, there are multiple heterogeneous formats: Table 2 Salary Information Format

[0107] If these heterogeneous formats are not normalized and mapped, it will cause significant bias in salary statistics.

[0108] Specifically, geographic information normalization includes: for various formats existing in the work location field (such as "Shenzhen·Nanshan District", "Guangdong Province", "Beijing"), using rule chains (such as symbol segmentation, removal of province suffixes, and preservation of original data) to map them to a unified standard city name; then, through a city-province mapping table, the city is assigned to the province, forming a three-level geographic label (province-city-district). For example, "Shenzhen·Nanshan District" is normalized and mapped to "Guangdong Province-Shenzhen City-Nanshan District". The city-province mapping algorithm flow is as follows: Figure 6 As shown.

[0109] Salary information standardization includes: addressing unit differences (10,000 / month, 1,000 / month, 10,000 / year) and year-end bonus information (13th-month salary, 14th-month salary). Through unit identification, period conversion, and weighted calculation of year-end bonuses, salaries are uniformly converted into standardized monthly salary values ​​(or monthly salary ranges). For example, "15,000-25,000 / month" remains unchanged, "200,000-300,000 / year" is converted to approximately "16,700-25,000 / month", and "10,000-15,000 · 13th-month salary" is converted to a weighted monthly salary.

[0110] The discrete features after the above normalized mapping become comparable and aggregateable labels, providing a basis for data slicing.

[0111] In some implementations, the data slicing step includes: performing semantically adaptive slicing based on business domain characteristics to generate a slicing dataset.

[0112] Specifically, business domain features are pre-grouped based on the original job titles. For example, the business domain feature classification dictionary is shown in Table 3.

[0113] Table 3. Business Domain Feature Classification Dictionary

[0114] By pre-grouping based on business domain dimensions, technical keywords from different domains are isolated into their respective shards, avoiding the problem of intertwined and discrete distribution of technical keywords from different business domains in the full dataset. Only job data related to specific business domains is retained within each shard, making subsequent semantic normalization and skill keyword extraction more focused, thereby significantly improving the purity of skill keyword extraction and the cohesion of subsequent clustering.

[0115] Furthermore, discrete features can be combined to form multidimensional cross-statistical matrices, enabling cross-aggregation and joint analysis under different dimensional combinations. Taking the "cloud computing" business as an example, a cross-statistical matrix can be constructed based on city (Shenzhen, Beijing, Shanghai) × salary level (low, medium, high) × education level (bachelor's, master's). Through the construction and cross-aggregation of this matrix, not only can the discrete features be normalized and partitioned, but the clustering results can also directly reflect the job skill structure under specific geographical, salary, and education conditions, facilitating companies to formulate differentiated recruitment strategies. In addition, cross-aggregation helps to uncover deep skill patterns under multi-dimensional coupling. For example, the "Shenzhen-high salary-bachelor's" combination may emphasize "Go language" skills, while the "Beijing-high salary-bachelor's" combination may emphasize "Java" skills. This significantly enhances the granularity and practicality of recruitment big data analysis.

[0116] The semantic normalization step is performed on the segmented dataset, meaning that semantic normalization is conducted independently within each segment. Since job titles within the same segment have stronger semantic similarities in terms of geography, salary, and education, heterogeneous expressions of job titles may exhibit segment-specific patterns. Therefore, processing within each segment can improve the hit rate and reconstruction accuracy of the normalized dictionary. For example, in the "Guangdong Province" segment, "cloud computing development engineer" might be more commonly referred to as "cloud platform developer," while in the "Beijing" segment, it might be more commonly referred to as "cloud computing R&D engineer." Through segment-specific independent normalization processing, differentiated classification vocabularies can be maintained for different segments, thereby further improving the adaptability of the normalization.

[0117] In some embodiments, the method further includes: Cross-slice analysis steps: Aggregate the clustering results of each slice and analyze the correlation between discrete features and skill clustering patterns.

[0118] This step constructs a three-layer collaborative processing architecture of "sharding isolation - adaptive clustering - cross-shard association," realizing an end-to-end mining process from data preprocessing to knowledge discovery. Cross-shard analysis is built upon sharding isolation and adaptive clustering: sharding isolation ensures the local comparability of skill clusters within each shard, while adaptive clustering guarantees the stability of clustering quality across shards. Their collaboration makes cross-condition comparisons statistically significant. By quantifying the differences in the proportion of the same skill direction across different regions, salary levels, or business areas, the study reveals the regulatory patterns of discrete characteristics on the skill demand structure.

[0119] Taking geographic information zoning as an example: each province is independently clustered within its zoning area to generate the province's skill cluster structure (such as cloud-native development, traditional internet development, enterprise information technology development, etc., and their proportions); comparing the differences in the proportions of the same skill cluster across different provinces can identify the regional technology advancement gradient; combining salary zoning with secondary slicing can further reveal the correlation between salary levels and skill advancement. This analysis provides data support for regional talent development strategies, corporate technology investment directions, and job seekers' skill enhancement paths.

[0120] In some implementations, this method constructs a Hadoop big data platform and utilizes a parallel computing framework to achieve intelligent processing and mining of job posting data.

[0121] The specific steps are as follows: 1. Setting up a distributed environment for a Hadoop big data platform: (1) Cluster architecture design: A Hadoop cluster based on a distributed architecture is adopted, and Ambari is used as the core management tool to achieve automated deployment and monitoring of the cluster.

[0122] Management architecture: It consists of Ambari Server (the core service, responsible for managing components and distributing configurations) and AmbariAgent (deployed on each node, responsible for executing commands and reporting status).

[0123] Software Stack (HDP Stack): Includes HDFS distributed file system, YARN resource manager, MapReduce computing framework, Hive data warehouse, Mahout data mining tool and Sqoop data transfer tool.

[0124] (2) Node configuration and planning The cluster consists of three physical (or virtual) server machines / nodes (which can be dynamically added). The system uniformly uses CentOS 7, and the cluster host planning is shown in Table 4: Table 4 Cluster Host Planning

[0125] (3) Cluster setup: Basic environment configuration: Configure hostname and IP mapping for each node and enable passwordless SSH login to ensure smooth communication between nodes.

[0126] Dependency installation and source optimization: Install necessary system dependency packages and modify the yum source to improve installation speed.

[0127] Ambari service deployment: Execute `yum install ambari-server` to install the core management interface.

[0128] Guided cluster setup: Select the HDP Stack version through the Ambari web interface.

[0129] Specify the slave node and complete the automatic registration of the host.

[0130] Based on research needs (recruitment data mining), select core services such as HDFS, Hive, and Mahout, and the system will automatically optimize and allocate the components.

[0131] Cluster monitoring and management: Use monitoring components (such as Ambari-Metrics) in the Ambari web interface to view data processing progress, cluster health status and alarm information in real time.

[0132] Example 2 This disclosure provides a big data mining device 1000 for job postings.

[0133] The apparatus 1000 may include corresponding modules that execute one or more steps of the flowchart describing the above-described method for big data mining of job postings. Therefore, each or more steps in the flowchart can be executed by a corresponding module, and the apparatus 1000 may include one or more of these modules. A module may be one or more hardware modules specifically configured to execute a corresponding step, or implemented by a processor configured to execute a corresponding step, or stored in a readable storage medium for processor implementation, or implemented through some combination thereof.

[0134] Specifically, such as Figure 7 As shown, the device 1000 includes: The semantic normalization module 1002 is used to perform word segmentation on the original job title, match the word segmentation results with a pre-built technical term library and functional verb library, extract core terms, and reconstruct and generate a normalized job title according to a preset structured template. The core terms include technical terminology and functional role verbs. The skill extraction module 1004 is used to extract skill keywords from the original job description corresponding to the standardized job name and generate a set of skill keywords for the job category, wherein the job category is jointly represented by the standardized job name and the corresponding set of skill keywords. The feature vectorization module 1006 is used to perform word frequency-inverse document frequency calculation on the core terms of the job category and the set of skill keywords to generate a sparse feature vector for the job category. The adaptive clustering module 1008 is used to perform the following operations based on the sparse feature vectors of the job categories to obtain clustering results: The Canopy algorithm is executed to automatically generate initial cluster centers and K values, wherein the distance threshold of the Canopy algorithm is dynamically determined based on the statistical distribution of the distance between sample vectors in the sparse feature vector; Using the initial cluster centers and K values ​​as input, perform K-means iterative clustering until convergence.

[0135] This disclosure also provides an electronic device, including: a memory storing execution instructions; and a processor or other hardware module executing the execution instructions stored in the memory, causing the processor or other hardware module to execute the above-described job posting big data mining method.

[0136] This disclosure also provides a readable storage medium storing executable instructions, which, when executed by a processor, are used to implement the above-described method for big data mining of job postings.

[0137] The hardware architecture of the device 1000, implemented using a processor-based hardware approach, can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0138] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component Architecture (EISA) bus, etc. Bus 1100 can be divided into address bus, data bus, control bus, etc. For ease of representation, only one connection line is used in this diagram, but this does not indicate that there is only one bus or one type of bus.

[0139] Any process or method description in the flowcharts or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain. The processor performs the various methods and processes described above. For example, the method embodiments of this disclosure may be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some embodiments, part or all of the software program may be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform one of the methods described above by any other suitable means (e.g., by means of firmware).

[0140] The logic and / or steps represented in the flowchart or otherwise described herein may be specifically implemented in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-based system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0141] For the purposes of this specification, a "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use in or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM). Furthermore, a readable storage medium can even be paper or other suitable media on which a program can be printed, since a program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in memory.

[0142] It should be understood that various parts of this disclosure can be implemented in hardware, software, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0143] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0144] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a single processing module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. The storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0145] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A method for mining big data of a recruitment position, characterized in that, The method includes: Semantic normalization steps: The original job title is segmented into words, and the segmentation results are matched with a pre-built technical term library and a functional verb library to extract core terms. The normalized job title is then reconstructed according to a preset structured template. The core terms include technical terminology and functional role verbs. Skill extraction steps: Extract skill keywords from the original job description corresponding to the standardized job name to generate a set of skill keywords for the job category, wherein the job category is jointly represented by the standardized job name and the corresponding set of skill keywords; Feature vectorization step: Perform term frequency-inverse document frequency calculation on the core terms and skill keyword set of the job category to generate a sparse feature vector for the job category; Adaptive clustering steps: Based on the sparse feature vectors of the job categories, perform the following operations to obtain the clustering results: The Canopy algorithm is executed to automatically generate initial cluster centers and K values, wherein the distance threshold of the Canopy algorithm is dynamically determined based on the statistical distribution of the distance between sample vectors in the sparse feature vector; Using the initial cluster centers and K values ​​as input, perform K-means iterative clustering until convergence.

2. The method of claim 1, wherein, The semantic normalization step and / or the skill extraction step are executed within a parallel computing framework.

3. The method of claim 1, wherein, The skill extraction steps include: The original job description is matched with a pre-built dictionary of compound words to identify and retain compound technical terms. The remaining text is segmented using a general word segmenter to obtain a word sequence; The composite technical terms and terms in the term sequence are compared with a pre-built skill point term library, and the terms existing in the skill point term library are filtered out as skill keywords.

4. The method as described in claim 1, characterized in that, The feature vectorization step includes: Different weighting coefficients are applied to the standardized job titles and the skill keywords, wherein the weighting coefficient of the core terms is higher than that of the skill keywords.

5. The method as described in claim 1, characterized in that, The adaptive clustering step includes: The distance thresholds of the Canopy algorithm include a loose threshold T1 and a strict threshold T2, and are calculated according to the following formulas: T1=μ+k1·σ, T2=μ-k2·σ, where μ is the mean of the distance between the sample vectors, σ is the standard deviation of the distance between the sample vectors, k1 and k2 are preset coefficients, and T1>T2.

6. The method as described in claim 1, characterized in that, The adaptive clustering step is executed within a distributed computing framework and includes: Each computing node calculates distance statistics based on local sparse feature vector partitioning, and the distance statistics include local mean and local standard deviation; The local distance statistics of each node are aggregated to generate global statistics. A global adaptive threshold is determined based on the global statistics. The global adaptive threshold includes a global loose threshold and a global strict threshold. Each computing node performs Canopy partitioning based on the global adaptive threshold to generate a local center vector; The local centroid vectors of each node are aggregated to generate a global initial centroid set.

7. The method as described in claim 1, characterized in that, The method further includes: Multidimensional feature preprocessing steps: Normalize and map the multidimensional discrete features of the job posting data, wherein the multidimensional discrete features include at least one of geographical information and salary information; Data slicing steps: Based on the normalized discrete features, perform semantically adaptive slicing to generate slicing datasets; The semantic normalization step is performed on the fragmented dataset.

8. A big data mining device for job postings, characterized in that, The device includes: The semantic normalization module is used to perform word segmentation on the original job title, match the segmentation results with a pre-built technical term library and functional verb library, extract core terms, and reconstruct and generate a normalized job title according to a preset structured template. The core terms include technical terminology and functional role verbs. The skill extraction module is used to extract skill keywords from the original job description corresponding to the standardized job name, and generate a set of skill keywords for the job category, wherein the job category is jointly represented by the standardized job name and the corresponding set of skill keywords. The feature vectorization module is used to perform word frequency-inverse document frequency calculation on the core terms and skill keyword set of the job category to generate a sparse feature vector for the job category. The adaptive clustering module is used to perform the following operations based on the sparse feature vectors of the job categories to obtain clustering results: The Canopy algorithm is executed to automatically generate initial cluster centers and K values, wherein the distance threshold of the Canopy algorithm is dynamically determined based on the statistical distribution of the distance between sample vectors in the sparse feature vector; Using the initial cluster centers and K values ​​as input, perform K-means iterative clustering until convergence.

9. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the method according to any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the method described in any one of claims 1-7.

Images