Automated data management system and method for work skills

Abstract

The application relates to an automated data management system and a corresponding method. The system comprises: a data collection module for collecting raw position data from a plurality of public resources; a data preprocessing module for preprocessing the collected raw position data to remove irrelevant information and / or symbols; a data recognition module for recognizing text related to work skill requirements from the preprocessed data and recognizing keywords related to work skills; a semantic analysis module for performing semantic analysis on the recognized keywords related to work skills and forming a skill classification scheme; and a data storage module for establishing a skill database based on the skill classification scheme and storing the skill classification scheme. The system automates the analysis of recruitment information to identify the most in-demand skills in the current market. Through big data analysis, the application accurately assesses the demand for various skills and provides valuable data support for job seekers and enterprises.

Description

Technical Field

[0001] This invention relates to the technical field of data processing and analysis, and more specifically to an automated data management system and method for work skills. Background Technology

[0002] In today's rapidly changing job market, the challenges posed by the mismatch between skills supply and demand are becoming increasingly prominent. Companies face the difficulty of finding that recent graduates and job seekers often lack the key skills required for business operations, creating significant obstacles in the recruitment process. Meanwhile, job seekers often lack clarity regarding the specific skill set required for the jobs they are applying for, leading to inefficient job hunting. With a vast number of job listings available on various recruitment platforms, job seekers need to invest considerable time and effort in collecting and analyzing relevant information to understand current market demands.

[0003] The long-standing problem of the mismatch between talent supply and demand has always been a focus of the job market. While optimizing the allocation and utilization of skills remains an ideal goal, it is often difficult to achieve in practice. Against the backdrop of profound changes in the global economic structure, millions of job seekers are struggling to find jobs that match their skill sets, further exacerbating the existing mismatch.

[0004] Furthermore, businesses are increasingly aware that only a small percentage of graduates or job seekers possess the essential skills necessary to drive business growth and development. Many new graduates require additional training to effectively contribute value. Conversely, a significant number of students lack insight into the skills businesses seek and the skills they need to develop for their own future success. Employers have observed that only a small number of graduates or job seekers possess the basic skills necessary for business operations and expansion, highlighting the importance of supplementary training for enhancing employee effectiveness.

[0005] Therefore, there is an urgent need for a streamlined solution that enables job seekers to understand the specific skills that companies seeking candidates who match their personal qualifications are looking for. Such a tool would significantly simplify the job search process, eliminating the need for individuals to sift through vast amounts of job information and compile skill requirements themselves. This resource not only increases the chances of finding a job that aligns with one's preferences but also helps students gain a comprehensive understanding of the skills required for their ideal positions, thereby promoting strategic planning for their skills development. Summary of the Invention

[0006] This patent application proposes a comprehensive solution to address the widespread inefficiency in the job market by introducing a professional skills analysis system based on Natural Language Processing (NLP). This solution automatically analyzes job postings to identify the key skills required by the current market. The system employs various technologies, including web crawling, to collect data from various recruitment websites and utilizes NLP techniques such as tokenization, word frequency analysis, and semantic analysis to extract key skills information from job descriptions. This method not only saves job seekers time and increases their chances of obtaining their ideal positions but also provides companies with accurate talent matching data, thereby improving recruitment efficiency and reducing human resource costs. Furthermore, the system provides data-driven decision support for the recruitment process, assisting companies in developing targeted recruitment strategies, while also enabling educational institutions to keep pace with market demands, optimize their curriculum, and cultivate talent that meets current market needs.

[0007] According to a first aspect of this application, an automated data management system for job skills is provided. The system includes: a data acquisition module for acquiring raw job data from multiple public resources; a data preprocessing module for preprocessing the acquired raw job data to remove irrelevant information and / or symbols; a data recognition module for identifying text related to job skill requirements from the preprocessed data and identifying keywords related to job skills; a semantic analysis module for performing semantic analysis on the identified keywords related to job skills and forming a skill classification scheme; and a data storage module for establishing a skill database based on the skill classification scheme and storing the skill classification scheme.

[0008] Preferably, the data preprocessing module is configured to perform at least one of the following operations: removing HTML format tags from the collected raw job data, word segmentation, lemmatization, identification and removal of stop words.

[0009] Preferably, the data recognition module stores lists of "skill titles" and "skill words" and is configured to perform the following operations: traverse the preprocessed data from the beginning using a first moving window and a second moving window; evaluate whether any word in the first moving window matches any skill title in the list of skill titles, and calculate the density of skill words in the first and second moving windows, where the density is the ratio of skill words to the total number of words; identify the skill requirement portion from the preprocessed data; and identify keywords related to the job skills from the identified skill requirement portion.

[0010] Preferably, identifying the skill requirement portion from the preprocessed data further includes: determining the start of the skill requirement portion if a skill title is found in the first moving window, or if the density of skill words in the first or second moving window exceeds a first threshold; and determining the end of the skill requirement portion if the density of skill words in two consecutive windows is lower than a second threshold.

[0011] Preferably, the first threshold and the second threshold are between 50% and 90%, and the lengths of the first moving window and the second moving window are fixed or adjustable.

[0012] Preferably, the semantic analysis module is configured to: convert keywords into numerical vectors; cluster the keywords using a statistical clustering method; and align the clusters with relevant skill sets to create a skill classification scheme.

[0013] Preferably, the automated data management system further includes one or more of the following modules: an update module for updating job data and skill requirement information in real time or periodically to update the skill classification scheme; a user interaction module for providing a user interface for communicating with users, displaying query information related to job skills, showing information related to job skill analysis, and / or collecting skill requirement feedback; an intelligent matching module for searching the job skill database using exact string matching or fuzzy algorithms based on user query information to find matching job skills; a user customization module for optimizing the search and matching results of job skills based on user feedback; a statistical analysis module for calculating the demand percentage of specific job skills; a job skill analysis report module for generating customized job skill analysis reports based on the skill classification scheme according to user requests; and a data classification module for classifying the collected raw job data to distinguish different types of recruitment data and feeding the classified information into the data preprocessing module.

[0014] Preferably, the types of recruitment data include at least one of the following: company introduction, job description, benefits, skills requirements, and educational requirements.

[0015] According to a second aspect of this application, an automated data management method for job skills is provided. The method includes the following steps: Step 1, collecting raw job data from multiple public resources; Step 2, preprocessing the collected raw job data to remove irrelevant information and / or symbols; Step 3, identifying text related to job skill requirements from standardized data and identifying keywords related to job skills; Step 4, performing semantic analysis on the identified keywords related to job skills and forming a skill classification scheme; Step 5, establishing a skill database based on the skill classification scheme.

[0016] Preferably, the preprocessing includes at least one of the following operations: removing HTML format tags from the collected raw job data, word segmentation, lemmatization, identification and removal of stop words.

[0017] Preferably, step three further includes: using a first moving window and a second moving window to traverse the preprocessed data from the beginning to find phrases that match the pre-stored lists of "skill titles" and "skill words"; evaluating whether any word in the first moving window matches any skill title in the list of skill titles, and calculating the density of skill words in the first and second moving windows, where the density is the ratio of skill words to the total number of words; identifying the skill requirement section from the preprocessed data; and identifying keywords related to job skills from the identified skill requirement section.

[0018] Preferably, identifying the skill requirement portion from the preprocessed data further includes: determining the start of the skill requirement portion if a skill title is found in the first moving window, or if the density of skill words in the first or second moving window exceeds a first threshold; and determining the end of the skill requirement portion if the density of skill words in two consecutive windows is lower than a second threshold.

[0019] Preferably, the first threshold and the second threshold are between 50% and 90%, and the lengths of the first moving window and the second moving window are fixed or adjustable.

[0020] Preferably, step four further includes: converting keywords into numerical vectors; clustering the keywords using a statistical clustering method; and aligning the clusters with relevant skill sets to create a skill classification scheme.

[0021] Preferably, the automated data management method further includes one or more of the following operations: updating job data and skill requirement information in real time or periodically to update the skill classification scheme; obtaining job skill-related query information from users, displaying information related to job skill analysis to users, and / or collecting user feedback on skill requirements; combining user query information and using exact string matching or fuzzy algorithms to search the job skill database to find job skills that match the user query; obtaining user feedback to optimize the search and matching results for job skills; classifying the collected job posting data to distinguish different types of recruitment data; calculating the demand percentage for specific job skills; generating customized job skill analysis reports based on the skill classification scheme according to user requests; classifying the collected raw job posting data to distinguish different types of recruitment data, and feeding the classified information into the data preprocessing module.

[0022] Preferably, the types of recruitment data include at least one of the following: company introduction, job description, benefits, skills requirements, and educational requirements.

[0023] According to a third aspect of this application, an automated data management system is provided, comprising: a data storage resource configured to store data and instructions; and a processing circuit configured to implement the method defined above.

[0024] According to a fourth aspect of this application, a computer-implemented automated data management method for job skills is provided. This method is executed by one or more processors and includes: acquiring raw job data from public resources at a data acquisition unit; preprocessing the acquired raw job data at a data preprocessing unit to remove irrelevant information and / or symbols; identifying text related to job skill requirements from standardized data and identifying keywords related to job skills at a data recognition unit; performing semantic analysis on the identified keywords related to job skills at a semantic analysis unit and forming a skill classification scheme; and establishing a skill database based on the skill classification scheme at a data storage unit.

[0025] According to a fifth aspect of this application, an automated data management system is provided, comprising: one or more processors; and a computer-readable storage device coupled to the one or more processors and storing instructions thereon, which, when executed by the one or more processors, cause the one or more processors to perform operations for automated data management, including the steps defined above.

[0026] According to a sixth aspect of this application, a non-transitory computer-readable medium is provided that stores a plurality of instructions, which, when executed, control a computer including one or more processors to implement the methods defined above.

[0027] This invention offers several key advantages. First, by utilizing NLP technology for automated job data analysis, it emphasizes automation and efficiency, enabling the system to efficiently process and effectively manage large amounts of job-related information. Second, the system provides real-time updates on market skills requirements, ensuring users have access to the latest data. Furthermore, the implementation of data-driven analysis methods allows for objective analysis, reducing subjective bias and improving the accuracy of skills requirement analysis. Finally, the system's ability to process multiple job postings simultaneously allows it to identify industry skills trends and demand changes, ultimately promoting a better match between job seekers and employers. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate one or more embodiments of the invention and, together with the specification, serve to explain the principles of the invention.

[0030] Figure 1 An automated data management system according to a first embodiment of this application is shown.

[0031] Figure 2 It shows the result of Figure 1 The automated data management system implements automated data management methods.

[0032] Figure 3 An automated data management system for job skills, based on another embodiment of this application, is further illustrated.

[0033] Figures 4A to 4D The results related to the percentage of general skills generated by embodiments of this application are shown.

[0034] Figures 5A to 5C Other results (i.e., percentage of technical skills) generated by embodiments of this application are shown.

[0035] Figure 6 An example of the user interface of a real-time job skills analyzer employing the automated system and method of this application is shown. Detailed Implementation

[0037] The embodiments of this application are now described in detail, and one or more embodiments are illustrated in the accompanying drawings. Reference numerals used repeatedly in this specification and drawings are intended to indicate similar features or elements of this application. It should be noted that the following is provided to help those skilled in the art further understand this application, but does not constitute a limitation on this application in any way. It should be understood that those skilled in the art can make various modifications and changes without departing from the concept of this application. For example, features shown or described as part of one embodiment may be used in conjunction with another embodiment to produce additional implementations. Therefore, this application is intended to cover such variations and changes within the scope of the appended claims and their equivalents.

[0038] Figure 1An automated data management system according to a first embodiment of this application is illustrated. As shown, the system includes a job data acquisition module 110, a job data preprocessing module 120, a job text data recognition module 130, a text data semantic analysis module 140, and a data storage module 150. The job data acquisition module 110 is responsible for collecting relevant data on publicly advertised job positions from public resources 160 using the Internet and database technology, including both text and numerical data. The job data preprocessing module 120 is responsible for standardizing various types of job data to facilitate enhanced analysis by subsequent modules. The job text data recognition module 130 uses natural language processing methods to segment and classify the relevant text data and identify key terms. The text data semantic analysis module 140 performs semantic analysis on the segmented data to extract specific skills required for the job, thereby generating a skill classification scheme. The data storage module 150 establishes a skill database based on this skill classification scheme and stores the skill classification scheme. The skill classification scheme is accessible to users 170.

[0039] Figure 2 It shows the result of Figure 1 The automated data management system implements automated data management methods. For example... Figure 2 As shown, the method includes the following steps: Step 210, collecting raw job data from public resources; Step 220, preprocessing the collected raw job data to remove irrelevant information and / or symbols; Step 230, identifying text related to job skill requirements from standardized data and identifying keywords related to job skills; Step 240, performing semantic analysis on the identified keywords related to job skills and forming a skill classification scheme; Step 250, establishing a skill database based on the skill classification scheme.

[0040] The following details Figure 1 and Figure 2 The automated data management system and method are shown.

[0041] Job data collection module 110

[0042] Step 210 is implemented by the job data collection module 110. This module is designed to collect relevant data, including both textual and numerical information, from publicly available job postings by utilizing the internet and database technologies. This module primarily operates through a data collection module that employs a combination of manual data collection techniques and web crawling methods to gather data.

[0043] The system's data acquisition function facilitates the conversion of information into the required data format through mechanisms such as manual input, web crawling, and human-computer interaction. Users can configure the data acquisition frequency and schedule to ensure timely and convenient data retrieval.

[0044] Web crawling utilizes web crawling languages ​​and computer tools to access publicly available data, significantly improving data update speed and timeliness. Furthermore, self-checking and verification processes enhance data accuracy while reducing data acquisition costs.

[0045] In special cases, such as handwritten job postings, manual data collection is used. For sources that are difficult to locate accurately and require on-site investigation, telephone inquiries, or other forms of human intervention, manual data collection is employed. Throughout the collection process, scanning and image recognition technologies, manual data entry, or storage tools such as Excel are used to acquire and update data. Excel is used to support and guide data entry, thereby enabling manual data updates in the background.

[0046] Job Data Preprocessing Module 120

[0047] Step 220 is implemented by the job data preprocessing module 120. A necessary preliminary stage in preparing to identify skills from job advertisements involves meticulous cleaning and preprocessing of the advertisement text. Typically, this preparation stage involves removing HTML formatting tags, word segmentation, lemmatization, and identifying and subsequently removing stop words. The identification of commonly used English stop words is facilitated by referencing an established list from the NLTK (Natural Language Toolkit).

[0048] The data preprocessing process encompasses a series of activities, including preprocessing tasks, correcting missing data, rectifying formatting errors, resolving logical inconsistencies, deduplicating information, eliminating irrelevant data, and validating relevance. Notably, during the processing of recruitment information, special characters are extracted and archived into arrays to prepare the data for subsequent analysis stages. These rigorous processes ensure that the compiled working data is refined to produce the desired information output.

[0049] Data cleaning involves thoroughly reviewing and validating data to remove duplicates, correct errors, and ensure data consistency. It is implemented as the act of removing data impurities, representing the final stage of identifying and correcting obvious errors in the dataset. Given that databases integrate data from diverse sources (including historical records), erroneous or conflicting data (commonly known as "dirty data") is inevitable. The overall goal is to cleanse this "dirty data" according to predefined criteria, retaining only the necessary information. Data deemed unsuitable due to incomplete, erroneous, or duplicate entries is systematically filtered out, allowing the cleaned-up data to proceed to subsequent processing stages.

[0050] In specific implementation scenarios, the data preprocessing process is achieved through a series of grouping steps aimed at obtaining the target data. This method includes: defining cleaning conditions for each data module group; applying these conditions to clean the data within the corresponding group; and finally extracting the target data module group, thereby providing the final dataset based on the cleaned data.

[0051] In the field of recruiting for job vacancies, potential employers often list detailed skill prerequisites in their online job postings. This article presents three examples of online job advertisements to illustrate how these advertisements are typically divided into different sections.

[0052] Example 1

[0053] Index Stock Portfolio Manager

[0054] full time

[0055] Middle Management

[0056] 7 years of experience

[0057] Banking and Finance

[0058] $15,000 to $19,000 per month

[0059] A total of 24 applications were received. The application period ended on November 24, 2022, with a deadline of December 24, 2022.

[0060] Roles and Responsibilities

[0061] ETF and Index Investing - Portfolio Management

[0062] Your team

[0063] Are you ready to join the world's largest asset management company and lead the index investing trend?

[0064] BlackRock's ETF and Index Investing team (EII) manages over $4 trillion in global index equity assets across developed markets, emerging markets, commodities, and real estate investment trusts. We offer investors one of the industry's most comprehensive index investing options.

[0065] Our investment team comprises professionals with extensive experience in their respective markets, managing assets for both institutional and individual investors. Our clients benefit from our expertise in portfolio management and our corporate culture that prioritizes performance and risk management.

[0066] Our mission is to help more and more people achieve better financial results, and we are passionate about it!

[0067] Your role and influence

[0068] We are looking to recruit a portfolio manager to join our ETF and index investing team. You will be responsible for all aspects of managing our index equity portfolios, including day-to-day portfolio management, risk and performance analysis, and business process restructuring. We are seeking passionate, ambitious individuals with the relevant expertise – both market and technically proficient!

[0069] Your duties

[0070] • Perform daily portfolio management tasks; daily liquidity management, portfolio rebalancing, corporate behavior analysis, and customer activity monitoring; risk and performance monitoring.

[0071] Performance and Risk Management

[0072] • Collaborate with our global trading team to handle rebalancing and other large-scale investment events.

[0073] • Assist in establishing best practices for portfolio management that can be shared globally.

[0074] • Building our next-generation investment platform

[0075] • Identify and drive operational improvements that reduce risk and increase efficiency for global teams.

[0076] • Collaborate with business partners to provide thought leadership and advice to foster high-quality client solutions.

[0077] • Actively contribute to our culture and support diversity, equity and inclusion. You need to have:

[0078] We are looking for talent from diverse backgrounds who possess the following qualifications:

[0079] Passionate about financial markets and technology

[0080] • Familiar with the stock and index ecosystem

[0081] • 7+ years of experience in the investment industry / portfolio management

[0082] • Possesses strong technical skills and is interested in technical solutions related to portfolio management, trading, and data analysis. Prior experience with Python, SQL, or other programming languages ​​is preferred.

[0083] • Strong analytical and problem-solving skills

[0084] • High level of self-motivation and strong work ethic

[0085] • Highly detail-oriented and responsible, with strong organizational and project management skills.

[0086] • Able to work quickly and accurately in fast-paced environments

[0087] • Enjoys teamwork and possesses excellent interpersonal skills.

[0088] Excellent written and oral communication skills

[0089] In the first example, information is organized under different headings, such as “Role and Responsibilities”, “Your Team”, “Your Role and Impact”, “Your Duties”, and “What You Need”, with relevant data on skills analysis typically located in the “What You Need” section.

[0090] The preprocessing results are as follows:

[0091] "Index Stock Portfolio Manager, Full-time, Mid-level Management, Years of Experience, Banking & Finance, Monthly Salary, Apply, Posting, November, December, Role and Responsibilities, ETFs, Index Investing, Portfolio Management, Team, Ready to join the world's largest asset management company, leading the index investing trend, BlackRock ETFs, Index Investing Team, EII, Managing global index stock assets, global developed markets, emerging markets, commodities, real estate investments, trust funds, providing investors with the industry's most comprehensive index investment options, the investment team includes professionals with extensive experience in their respective markets, managing assets, institutional investors, individual investors, etc., clients benefit from professional portfolio management capabilities, culture, performance, risk management, mission, helping people achieve better financial outcomes, passionate, role, impact, sincerely inviting, portfolio manager, join, ETF, index investing team, responsible for, aspects, index stock portfolio management, including daily portfolio management, portfolio, risk, performance analysis, business process reengineering. Looking for, passionate, ambitious, proficient in markets, proficient in technology, responsibilities, executing daily portfolio management tasks, daily..." Liquidity management, portfolio rebalancing, corporate behavior analysis, client activities, risk, performance monitoring, performance, risk management, collaboration, global trading team, handling rebalancing, large-scale investment events, assisting in establishing globally shared portfolio management best practices, building next-generation investment platforms, identifying, driving, and operational improvements, reducing risk, increasing efficiency, global team, collaboration, business partners, providing insightful leadership, advice, promoting high-quality client solutions, contributing to a culture that supports diversity, fairness, and inclusion. Seeking talent from diverse backgrounds, passionate about financial markets and technology, familiar with the stock and index ecosystems, years of experience in the investment industry, portfolio management experience, strong technical skills, interest in portfolio management, trading, and data analysis related technical solutions, Python and SQL programming experience preferred, analytical and problem-solving abilities, high self-motivation, strong work ethic, high attention to detail, responsibility, organizational and project management skills, ability to work quickly and accurately, a fast-paced environment, enjoy teamwork, excellent interpersonal skills, and excellent written and verbal communication skills.

[0092] Example 2

[0093] This position will assist the finance director in performing all financial and accounting functions.

[0094] Duties and Responsibilities

[0095] Internal / External Reporting

[0096] Cash Management

[0097] Regulatory and tax matters

[0098] Assist in system implementation

[0099] Qualification Requirements

[0100] Job Requirements:

[0101] Able to handle multiple tasks simultaneously and work under pressure

[0102] Strong computer skills

[0103] Good English writing and speaking skills

[0104] educate:

[0105] Hold an accounting qualification certificate and have a bachelor's degree in finance or a related field.

[0106] experience:

[0107] At least 5 years of auditing or accounting experience

[0108] Candidates with general insurance accounting and auditing backgrounds will be given preference.

[0109] Interested applicants please click the "Apply Now" button below and send your complete resume via email (indicating your available start date, current / previous job, and expected salary).

[0110] All collected personal data will be kept strictly confidential and used solely for recruitment purposes.

[0111] The second job advertisement organizes its information in two sections under the heading "Job Responsibilities and Duties" and "Qualifications." Required skills are detailed in the "Qualifications" section.

[0112] The preprocessing results are as follows:

[0113] "Position: Assist the Finance Manager in performing financial accounting functions, responsibilities, internal and external reporting, cash management, oversight, tax matters, and assisting with system implementation. Qualifications: Strong computer skills, good written and spoken English, accounting certification, Bachelor's degree in finance or a related field, minimum years of experience required, auditing experience preferred, general insurance accounting experience preferred, interested candidates please send your complete resume via email, including your availability, current job, previous job, and expected salary. Please click the button to apply. All personal data collected will be kept strictly confidential for recruitment purposes."

[0114] Example 3

[0115] Skills & Requirements:

[0116]

[0117] Excellent command of written and spoken English and Chinese with native / near-native proficiency;

[0118] <span style=" "color: rgb (51, 51, 51); background-color: rgb(255, 255, 255); font-size:

[0119] 16px; font-family: Helvetica Neue" ", Helvetica, Calibri, Arial,sans-serif; " ">Ability to translate complicated technical information intoan easily understood format

[0120] <span style=" "color: rgb (51, 51, 51); background-color: rgb(255, 255, 255); font-size:

[0121] 16px; font-family: Helvetica Neue" ", Helvetica, Calibri, Arial,sans-serif; " ">Sound planning and organisational skills

[0122] <span style=" "color: rgb (51, 51, 51); background-color: rgb(255, 255, 255); font-size:

[0123] 16px; font-family: Helvetica Neue" ", Helvetica, Calibri, Arial,sans-serif; " ">Presentation skills

[0124] Entry-level advertising, marketing, copywriting

[0125] Market trends research and reporting

[0126] <span style=" "color: rgb (0, 0, 0); background-color: rgb (255,255, 255); font-size:

[0127] medium; font-family: Roboto, Helvetica, Arial, sans-serif; " ">HKIDHolder

[0128]

[0129]

[0130] Nice to have:

[0131]

[0132] Knowledge and experience in cryptocurrency trading, blockchain / DLT;

[0133] <span style=" "color: rgb (0, 0, 0); background-color: rgb (255,255, 255); font-size:

[0134] medium; font-family: Roboto, Helvetica, Arial, sans-serif; " ">Knowledge in the areas of risk management in financial services industries orfintech environment;

[0135] <span style=" "color: rgb (0, 0, 0); background-color: rgb (255,255, 255); font-size:

[0136] medium; font-family: Roboto, Helvetica, Arial, sans-serif; " ">Broadknowledge of the financial services industries, familiarity with KYC and / orAML / CFT regulatory requirements

[0137]

[0138]

[0139] The third job posting contains job skill requirements text in HTML format. The HTML tags must be cleaned up before identifying any required skills.

[0140] The preprocessing results are as follows:

[0141] "Skills Requirements: Excellent written and spoken English, native Chinese speaker, fluent, ability to translate complex technical information in an easily understandable manner, good planning and organizational skills, presentation skills, entry-level advertising and marketing copywriting, market trend research and reporting, Hong Kong ID card holder, extensive knowledge and experience in cryptocurrency trading, blockchain, DLT knowledge areas, risk management, financial services industry, fintech environment, broad knowledge of the financial services industry, familiarity with KYC, AML, and CFT regulatory requirements."

[0142] Working text data recognition module 130

[0143] Step 230 is implemented by the job text data recognition module 130. The provided example of an online job advertisement highlights a common practice among advertising entities: using text content to describe company background, work environment, job scope, role, and responsibilities. While this text is information-rich, it is considered irrelevant for job skills analysis. Incorporating this irrelevant text into an analysis using statistical models (such as keyword frequency analysis or semantic embedding language models) may introduce bias and inaccuracies into the assessment of the job skills specified in the job posting.

[0144] The key advancement presented in this application lies in recognizing the critical importance of separating the text specifying job skill requirements in online job advertisements from irrelevant textual content. To achieve this, this application designs an algorithm and utilizes Natural Language Processing (NLP) methods to run the algorithm.

[0145] The proposed algorithm relies on two essential textual features in online job postings used to describe skill prerequisites. First, this text typically forms a separate section within the ad body, often indicated by headings such as "Skill Requirements," "Qualification Requirements," or "You Must Possess." Second, these skill requirements are usually not expressed as complete sentences but rather through keywords or phrases. These features help advertising entities clearly articulate the job skill prerequisites and facilitate the screening of potential candidates. The proposed algorithm leverages these features to effectively separate skill requirement text from irrelevant content.

[0146] The algorithm is executed in three consecutive steps. First, the job text data recognition module 130 compiles lists of “skill titles” and “skill terms (including phrases)” typically used to specify skill requirements in online job postings. These lists are constructed by conducting a thorough review of a large number of online job postings to collect commonly used titles and terms representing the required skills.

[0147] Subsequently, the "moving window" traverses the advertisement text starting from the beginning. The job text data recognition module 130 evaluates whether any words in the moving window match the list of skill titles and counts the number of skill words contained within them. The density of skill words within the window (calculated as the ratio of skill words to the total number of words) helps identify the starting position of the skill requirement section. The length of the moving window (fixed or adjustable based on the advertisement text) is crucial to this determination. The algorithm uses a threshold (typically between 50% and 90%) to determine the presence of the skill requirement section based on the skill word density.

[0148] Next, the moving window iterates through the entire advertisement. If the density of skill words in two consecutive windows falls below a predetermined threshold, the algorithm identifies the end of the skill requirement section. This threshold (which is also adjustable within the range of 50% to 90%) can be different from the threshold applied in the previous steps.

[0149] The proposed algorithm demonstrates superior performance in separating skill requirement text from irrelevant content. In a dataset comprising 33,613 online job postings, the algorithm autonomously identified skill requirements in 28,128 of them. In the remaining 5,485 job postings, human review revealed skill requirements in 2,567 instances, confirming that such requirements were absent in the remaining 2,918 postings. This indicates a success rate of 91.6% (=28,128 / (33,613-2,918)). In an independent sample of 8,661 online job postings, the algorithm successfully identified skill requirements in 8,344 postings, achieving a success rate of 96.4% (=8,344 / 8,661).

[0150] Text data semantic analysis module 140

[0151] Step 240 is implemented by the text data semantic analysis module 140. Within this module, keywords are converted into numerical vectors using a semantic embedding language model, enabling them to be represented in mathematical format. Subsequently, statistical clustering methods are applied to classify these keywords into different clusters, and these clusters are then matched with relevant skill sets within the relevant knowledge domain. The result is a skill classification scheme that is user-friendly and conducive to clear and effective communication.

[0152] First, semantic embedding language models are used to convert keywords into numerical vectors, thus encapsulating the semantic essence of these keywords in numerical form. These vectors are used to encapsulate the semantic relationships between keywords, where adjacent vectors represent similar meanings. This process relies on pre-trained language models (such as BERT, Word2Vec, or GloVe) that extract semantic associations from large corpora of text.

[0153] Subsequently, statistical clustering techniques are employed to group keywords into clusters based on the similarity of their numerical representations. This method aggregates related keywords within these clusters, aiding in the identification of semantic groupings.

[0154] Next, the clusters of keywords are mapped to relevant skill sets within the specific domain being analyzed. For example, clusters characterized by keywords such as "Python," "data analysis," and "machine learning" can be linked to the skill set "data science."

[0155] The ultimate outcome of this process is the establishment of a structured skills classification system. This system is designed to be easily understood, thereby facilitating effective communication of the identified skills. By organizing keywords into clusters and matching them with relevant skill sets, this classification scheme provides a clear overview of the skills required within the research domain.

[0156] Essentially, this automated method involves converting keywords into skill sets. It encapsulates keyword meaning through semantic embedding, groups semantically similar keywords using clustering methods, and then associates these groups with specific skill sets, thereby facilitating automatic keyword-to-skill classification. The advantages of this method are its ability to handle large numbers of keywords, automatically identify potential skill relationships, improve efficiency, and reduce the burden of manual annotation.

[0157] One illustrative embodiment of this application uses the terms "general / soft skills" and "technical / hard skills". The term "general / soft skills" encompasses traits that help an individual integrate into the work environment, including aspects related to personality, characteristics, adaptability, motivation, goals, and preferences (Heckman & Kautz, 2012).

[0158] Building on the work of de Freitas and Almendra (2021), this application further refines the classification of general skills, dividing them into social skills and cognitive skills. As de Freitas and Almendra (2021) emphasize, threshold skills play a crucial role in promoting the development of higher-order skills. In contrast, “technical / hard skills” refer to the abilities that enable individuals to excel in specific tasks, such as scientific knowledge, professional skills, and technical competence (Laker & Powell, 2011).

[0159] For clarity, the following two tables present examples of skills classification schemes specifically tailored for the accounting and finance industry. Table 1 lists 26 frequently cited skills, while Table 2 depicts 23 technical skills typically described by companies in the accounting and finance sector for job positions. Each skill in these tables is associated with a corresponding set of keywords. In practice, for each job posting, the text detailing the job skill prerequisites outlined in Step 4 is extracted, and the identified keywords are used to determine the skills mandated by the advertising entity.

[0160] Table 1. The five most frequently used keywords for each general skill

[0161] Table 2. The five most frequently used keywords for each technical skill

[0162] Figure 3 The embodiments depicted further illustrate an automated data management system 300 for job skills based on another embodiment of this application. This embodiment includes... Figure 1 Additional modules besides the module shown. (And) Figure 1 The same modules are identified using corresponding reference numerals in the accompanying drawings, and their functions and operations are the same as those in the drawings. Figure 1 The same as described in [the original text].

[0163] like Figure 3 As shown, the automated data management system 300 includes: an update module 310, a user interaction module 320, an intelligent matching module 330, a user customization module 340, a statistical analysis module 350, a job skills analysis report module 360, and a data classification module 370. It should be noted that the system 300 may include additional modules to meet user needs. Alternatively, any of these modules may be removed to reduce costs.

[0164] The update module 310 is responsible for facilitating real-time or periodic updates of job data and skills requirement information to ensure the accuracy and relevance of skills classification schemes in response to evolving employment market trends.

[0165] The user interaction module 320 serves as the user interface for interaction with the user. It displays query information related to job skills, provides job skills analysis insights, and collects feedback on skill requirements to enhance the user experience and the efficiency of the intelligent matching module system.

[0166] The intelligent matching module 330 utilizes precise string matching or fuzzy algorithms. This module searches a job skills database based on user query information to identify matching job skills, thereby streamlining the job matching process for users.

[0167] The user customization module 340 enables users to optimize job skills search and matching results by integrating user feedback, thereby ensuring personalized and customized job skills recommendations based on individual preferences and requirements.

[0168] The Statistical Analysis Module 350 is specifically designed to calculate the percentage of demand for specific job skills. This module provides valuable insights into job market trends and the popularity of various skill sets, thereby aiding strategic decision-making.

[0169] The Job Skills Analysis Report module generates customized job skills analysis reports based on a skills classification scheme, thereby meeting user requests and providing detailed insights into job skills trends and requirements to enable informed decision-making.

[0170] The data classification module 370 is responsible for classifying the collected raw job data to distinguish different types of recruitment data, thereby ensuring that the data is organized in an orderly and structured manner for further processing and analysis within the system.

[0171] Figures 4A to 4D and Figures 5A to 5CIt shows the result of Figure 3 The system uses the results generated by at least one of the listed modules (such as Statistical Analysis Module 350 and / or Job Skills Analysis Report Module 360).

[0172] Job postings for accounting, banking and finance, e-commerce, and insurance positions were collected from public recruitment resource websites during February, May, August, and November of 2023. The number of job postings is shown in Table 3.

[0173] Table 3

[0174] Automated methods were used to extract required skills, and the keyword-skill mappings shown in Tables 1 and 2 were used to identify the general and technical skills that recruiting companies expect from job candidates. For each general / technical skill, the number of job postings requiring that skill was counted, and the percentage of such postings in all postings (summarizing all four job functions) for the same month was calculated.

[0175] Step 250 is implemented by data storage module 150. Module 150 constructs a skills database using a skills classification scheme. The skills classification scheme is stored in this database and can be accessed as needed.

[0176] Figures 4A to 4D The percentages for all general skills are shown. According to Table 3, general skills are categorized into "Social 0", "Social", "Cognitive 0", and "Cognitive". The "Social 0" category contains threshold-level social skills, the "Social" category includes advanced social skills, the "Cognitive 0" category includes threshold-level cognitive skills, and the "Cognitive" category involves advanced cognitive skills. Figures 4A to 4D In each panel, the horizontal axis index represents the month, where "_8" represents August, "_5" represents May, "_2" represents February, and "11" represents November. Figures 5A to 5C The percentage of all technical skills grouped into "Business", "Technology", and "Other" is presented. Figures 5A to 5C In each panel, the horizontal axis index represents the corresponding month.

[0177] The analysis was successfully applied to nearly 50,000 job postings collected from public recruitment resource websites. Common and technical skills required in each job posting were identified. The demand for each skill was assessed by calculating the percentage of job postings requiring that skill among all job postings. The attached chart shows significant differences in skill demand in the Hong Kong job market, but relatively stable demand was observed throughout the four months of 2023. The analysis and findings documented in the report strongly demonstrate that the Natural Language Processing (NLP)-based job skills analysis system generates valuable job skills information.

[0178] Figure 6 The diagram further illustrates an example of the user interface for a real-time job skills analyzer employing the aforementioned automated systems and methods. As shown in the figure, after entering the desired job title or description, users can easily find the skills they need.

[0179] In summary, this application proposes a professional skills analysis and management system and method based on natural language processing. This system automates the analysis of recruitment information to identify the skills most in demand in the current market. Through big data analysis, this invention accurately assesses the demand for various skills, providing valuable data support for job seekers and enterprises. Job seekers can use the system to understand the skills needed in the market, thereby enhancing their competitiveness and increasing employment opportunities. Enterprises can use the system to quickly identify qualified candidates, thereby improving recruitment efficiency and reducing human resource costs. The system provides data support for the recruitment process, thereby assisting enterprises in developing more precise recruitment strategies. Furthermore, the system enables educational institutions to gain timely insights into market demands, thereby promoting the optimization of their curriculum to cultivate talent that meets market needs.

[0180] Generally, both System 100 and 300 include a processor and memory. The processor may have one or more processing cores, such as a quad-core or octa-core processor. The processor can be implemented using hardware such as a digital signal processor (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor may also include a main processor and coprocessors. The main processor (also known as the central processing unit (CPU)) processes data in the wake-up state, while the coprocessors are low-power processors for backup state data. In some cases, the processor may integrate a graphics processing unit (GPU) to render and draw the content of the display. Furthermore, the processor may have an artificial intelligence (AI) processor for machine learning tasks.

[0181] The memory may be formed of one or more non-transitory computer-readable storage media. It may include high-speed random access memory and non-volatile memory, such as disk storage devices and flash memory devices. In some cases, the memory stores instructions, which a processor executes to implement the methods described in this application.

[0182] Therefore, embodiments of this application provide a system having a processor and a memory for storing instructions, which are executed by the processor to perform... Figure 2 The method shown. This application also includes a computer-readable storage medium storing a program that, when executed by a processor, implements... Figure 2 The method is illustrated. Furthermore, a computer program product containing instructions, when run on a computer, enables the computer to perform... Figure 2 The method shown.

[0183] Those skilled in the art will understand that all or part of the blocks used to implement the embodiments can be implemented by hardware or by a program instructing the relevant hardware to do so. The program can be stored in a computer-readable storage medium (such as a read-only memory, disk, or optical disk). The embodiments described are optional and not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. An automated data management system for job skills, comprising: The data acquisition module is used to collect raw job data from multiple public resources; The data preprocessing module is used to preprocess the collected raw job data to remove irrelevant information and / or symbols; The data recognition module is used to identify text related to job skill requirements from preprocessed data and to identify keywords related to job skills. The semantic analysis module is used to perform semantic analysis on the identified keywords related to job skills and form a skill classification scheme; The data storage module is used to establish a skill database based on the skill classification scheme and store the skill classification scheme.

2. The automated data management system according to claim 1, wherein, The data preprocessing module is configured to perform at least one of the following operations: removing HTML format tags from the collected raw job data, word segmentation, lemmatization, identification, and removal of stop words.

3. The automated data management system according to claim 1, wherein, The data recognition module stores a list of skill titles and skill terms, and is configured to perform the following operations: The preprocessed data is traversed from the beginning using the first and second moving windows; Evaluate whether any word in the first moving window matches any skill title in the list of skill titles, and calculate the density of skill words in the first and second moving windows, wherein the density is the ratio of skill words to the total number of words; Identify the skill requirement portion from the preprocessed data; Identify keywords related to job skills from the identified skill requirements.

4. The automated data management system according to claim 3, wherein, Identifying the skill requirements from the preprocessed data further includes: If a skill title is found in the first moving window, or if the density of skill words in the first or second moving window exceeds a first threshold, then the skill requirement section is determined to begin. If the density of skill words is below the second threshold in two consecutive windows, then the skill requirement section is determined to be terminated.

5. The automated data management system according to claim 4, wherein, The first threshold and the second threshold are between 50% and 90%, and the lengths of the first moving window and the second moving window are fixed or adjustable.

6. The automated data management system according to claim 1, wherein, The semantic analysis module is configured as follows: Convert keywords into numerical vectors; The keywords were clustered using statistical clustering methods. Align the clusters with the relevant skill sets to create the skill classification scheme.

7. The automated data management system according to claim 1 further includes one or more of the following modules: An update module is used to update job data and skill requirement information in real time or periodically in order to update the skill classification scheme; The user interaction module provides a user interface for communicating with users, displaying query information related to job skills, showing information related to job skills analysis, and / or collecting feedback on skill requirements. The intelligent matching module is used to search the job skills database based on user query information using exact string matching or fuzzy algorithms to find matching job skills; The user customization module is used to optimize search and matching results for job skills based on user feedback; The statistical analysis module is used to calculate the percentage of demand for specific job skills; The job skills analysis report module is used to generate customized job skills analysis reports based on the skill classification scheme according to user requests; The data classification module is used to classify the collected raw job data to distinguish different types of recruitment data, and feed the classified information into the data preprocessing module.

8. The automated data management system according to claim 7, wherein, The types of recruitment data include at least one of the following: company introduction, job description, benefits, skills requirements, and educational requirements.

9. An automated data management method for job skills, comprising the following steps: Step 1: Collect raw job data from multiple public resources; Step 2: Preprocess the collected raw job data to remove irrelevant information and / or symbols; Step 3: Identify text related to job skill requirements from standardized data, and identify keywords related to job skills; Step four: Perform semantic analysis on the identified keywords related to job skills and formulate a skill classification scheme; Step 5: Establish a skills database based on the skills classification scheme.

10. The automated data management method according to claim 9, wherein, The preprocessing includes at least one of the following operations: removing HTML format tags from the collected raw job data, word segmentation, lemmatization, identification and removal of stop words.

11. The automated data management method according to claim 9, wherein, Step three also includes: The preprocessed data is traversed from the beginning using the first and second moving windows to find phrases that match the pre-stored lists of "skill titles" and "skill words"; Evaluate whether any word in the first moving window matches any skill title in the list of skill titles, and calculate the density of skill words in the first and second moving windows, wherein the density is the ratio of skill words to the total number of words; Identify the skill requirement portion from the preprocessed data; Identify keywords related to job skills from the identified skill requirements.

12. The automated data management method according to claim 11, wherein, Identifying the skill requirements from the preprocessed data further includes: If a skill title is found in the first moving window, or if the density of skill words in the first or second moving window exceeds a first threshold, then the skill requirement section is determined to begin. If the density of skill words is below the second threshold in two consecutive windows, then the skill requirement section is determined to be terminated.

13. The automated data management method according to claim 12, wherein, The first threshold and the second threshold are between 50% and 90%, and the lengths of the first moving window and the second moving window are fixed or adjustable.

14. The automated data management method according to claim 11, wherein, Step four also includes: Convert keywords into numerical vectors; The keywords were clustered using statistical clustering methods. Align the clusters with the relevant skill sets to create the skill classification scheme.

15. The automated data management method according to claim 9 further includes one or more of the following operations: The job data and skill requirement information are updated in real time or periodically to update the skill classification scheme; Obtain job skills-related query information from users, display job skills analysis-related information to users, and / or collect user feedback on skill requirements; Combine user query information and use exact string matching or fuzzy algorithm to search the job skills database to find job skills that match the user query; Gather user feedback to optimize search and matching results for job skills; The collected job posting data is categorized to distinguish different types of job posting data; Calculate the percentage of demand for a specific job skill; Based on the user's request, a customized job skills analysis report is generated according to the skill classification scheme. The collected raw job data is classified to distinguish different types of recruitment data, and the classified information is fed into the data preprocessing module.

16. The automated data management method according to claim 15, wherein, The types of recruitment data include at least one of the following: company profile, job description, benefits, skills requirements, and educational requirements.

17. An automated data management system, comprising: Data storage resources are configured to store data and instructions; as well as The processing circuit is configured to implement the method according to any one of claims 10-15.

18. A computer-implemented automated data management method for job skills, the method being executed by one or more processors and comprising: Raw job data is collected from multiple public resources at the data collection unit; The raw job data collected is preprocessed at the data preprocessing unit to remove irrelevant information and / or symbols; At the data recognition unit, text related to job skill requirements is identified from standardized data, and keywords related to job skills are also identified. At the semantic analysis unit, semantic analysis is performed on the identified keywords related to job skills, and a skill classification scheme is formed; A skills database is established at the data storage unit based on the skills classification scheme.

19. An automated data management system, comprising: One or more processors; as well as A computer-readable storage device coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for automated data management, the operations including the steps of any one of claims 10-15.

20. A non-transitory computer-readable medium storing a plurality of instructions, which, when executed, control a computer including one or more processors to implement the method according to any one of claims 10-15.

Images