A dynamic resume evaluation method based on a multi-modal large model

Abstract

The present application relates to the technical field of automatic data processing, in particular to a dynamic resume evaluation method based on a multi-modal large model. The method comprises: receiving a file containing the resume information of an applicant and a portfolio containing one or more visual works of the applicant; based on the file containing the resume information of the applicant, analyzing the file using a resume analysis module to generate a resume score; based on the portfolio of the applicant, analyzing the portfolio using a portfolio analysis module to generate a portfolio score; based on the file containing the resume information of the applicant and the portfolio of the applicant, calculating a consistency score using a consistency analysis module; and fusing the resume, portfolio and consistency scores to generate a final matching degree score to determine the matching degree of the applicant and the post. The present application solves the problem that traditional methods cannot evaluate visual materials and verify the authenticity of resumes, and provides a more accurate, objective and comprehensive candidate screening scheme.

Description

Technical Field

[0001] This invention relates to the field of automatic data processing technology, specifically to a dynamic resume evaluation method based on a multimodal large model. Background Technology

[0002] For employers seeking to fill vacancies, resume screening is an essential task. When resumes are screened manually, the human resources department can spend a significant amount of time each time a company fills a position. Computer technology and the internet allow job seekers to submit their resumes with the click of a button, but a single job opening can receive hundreds of resumes. The manual screening process is not only labor-intensive, time-consuming, and costly, but it is also highly susceptible to unintentional human bias, leading to inconsistent evaluations and potentially overlooking qualified candidates.

[0003] Traditional applicant tracking systems developed to address these issues, while automating the screening process, suffer from fundamental limitations. These systems, primarily based on keyword matching and semantic analysis of text, often perform poorly in effectively handling unstructured data. They cannot process or evaluate visual information, such as design portfolios, which are crucial for many creative and design positions. Furthermore, these systems lack any mechanism to verify claims made in resumes, treating all textual content as factual and thus failing to assess its authenticity.

[0004] While some advanced systems have begun to use artificial intelligence to perform tasks such as sentiment analysis or emotion detection in video interviews, these tools are primarily aimed at later stages of the recruitment process (post-screening) and do not address the initial and fundamental issues of comprehensively evaluating applicants’ submissions, particularly assessing the key connections between their resumes and portfolios.

[0005] A drawback of existing technologies is the assumption that a resume is a document of fact. From simple keyword scanners to complex natural language processing models, existing methods aim to interpret claims rather than validate them. This leads to exaggerated skills or project roles being treated as valid data points, resulting in biased candidate ranking. To address this, a cross-modal validation step is needed, computationally simulating recruiters to bridge the credibility gap and introduce new rigor and reliability into the recruitment process. Summary of the Invention

[0006] To overcome the aforementioned deficiencies in the existing technology, this invention provides a dynamic resume evaluation method based on a multimodal large model.

[0007] This invention discloses a dynamic resume evaluation method based on a multimodal large model, comprising the following steps: Receive a file containing the applicant's CV information, and a portfolio of the applicant's work containing one or more visual works; Based on the file containing the applicant's resume information, the resume analysis module is used to analyze the file to generate a resume score; Based on the applicant's portfolio, the portfolio analysis module is used to analyze the portfolio in order to generate a portfolio score; Based on the file containing the applicant's resume information and the applicant's portfolio, a consistency score is calculated using the consistency analysis module; The resume score, portfolio score, and consistency score are combined to generate a final matching score to determine the degree of match between the applicant and the position.

[0008] Preferably, the step of analyzing the file containing the applicant's resume information using a resume analysis module to generate a resume score specifically includes: Text markers and their corresponding two-dimensional position coordinates are extracted from the file using optical character recognition technology. Using a pre-trained document understanding model, text embeddings and layout embeddings are generated based on the text tags and the two-dimensional position coordinates; A semantic score is calculated based on the semantic similarity between the embedded text and the preset job description; Based on the layout embedding, a layout score is calculated using an aesthetic evaluation model; The semantic score and the layout score are combined to generate the resume score.

[0009] Preferably, the step of analyzing the applicant's portfolio using a portfolio analysis module to generate a portfolio score specifically includes: A visual Transformer model is used to generate a feature embedding for each visual work in the portfolio; A content score is calculated based on the content matching degree between the embedded features and the preset job requirements. A style score is calculated by performing cluster analysis on the feature embeddings to evaluate the style attributes among the visual works. The content score and the style score are combined to generate the portfolio score.

[0010] Preferably, the step of calculating a style score by performing cluster analysis on the feature embeddings to evaluate the style attributes among the visual works specifically includes: The features of all visual works in the collection are embedded into a set and input into a clustering algorithm; Based on the output of the clustering algorithm, the intra-cluster distance or variance of one or more clusters is calculated to generate a style consistency index. The style score is determined based on the style consistency index.

[0011] Preferably, the step of calculating a consistency score based on the file containing the applicant's resume information and the applicant's portfolio using a consistency analysis module specifically includes: Use a natural language processing model to identify one or more verifiable textual claims from the text content of the resume; For each identified text claim, a text embedding is generated using a contrastive language-image model text encoder; For each visual work in the portfolio, an image embedding is generated using the image encoder of the contrastive language-image model; A claim-specific consistency score is generated for each text claim by calculating the maximum similarity between the text embedding of each text claim and the image embedding of all visual works. Combine all claim-specific consistency scores to generate the aforementioned consistency score.

[0012] In another aspect, the present invention provides a dynamic resume evaluation system based on a multimodal large model to implement the above method, including a data receiving module, a data storage module, a resume analysis module, a portfolio analysis module, a consistency analysis module, and a fusion module; The data receiving module is used to receive applicant data and job description data, specifically including job description, applicant's resume and portfolio; The data storage module is used to store applicant data and job description data received by the data receiving module; The resume analysis module is used to analyze resumes based on text content and layout features to generate resume scores; The portfolio analysis module is used to analyze portfolios based on visual content and style features to generate portfolio scores; The consistency module is used to calculate a consistency score by comparing textual claims in the resume with visual evidence in the portfolio. The fusion module is used to generate a final matching score by combining the resume score, the portfolio score, and the consistency score.

[0013] Preferably, the step of analyzing resumes based on text content and layout features to generate resume scores specifically includes: Text markers and their corresponding two-dimensional position coordinates are extracted from the file using optical character recognition technology. Using a pre-trained document understanding model, text embeddings and layout embeddings are generated based on the text tags and the two-dimensional position coordinates; A semantic score is calculated based on the semantic similarity between the embedded text and the preset job description; Based on the layout embedding, a layout score is calculated using an aesthetic evaluation model; The semantic score and the layout score are combined to generate the resume score.

[0014] Preferably, the step of analyzing the portfolio based on visual content and style features to generate a portfolio score specifically includes: A visual Transformer model is used to generate a feature embedding for each visual work in the portfolio; A content score is calculated based on the content matching degree between the embedded features and the preset job requirements. A style score is calculated by performing cluster analysis on the feature embeddings to evaluate the style attributes among the visual works. The content score and the style score are combined to generate the portfolio score.

[0015] Preferably, the step of calculating a consistency score by comparing textual claims in the resume with visual evidence in the portfolio specifically includes: Use a natural language processing model to identify one or more verifiable textual claims from the text content of the resume; For each identified text claim, a text embedding is generated using a contrastive language-image model text encoder; For each visual work in the portfolio, an image embedding is generated using the image encoder of the contrastive language-image model; A claim-specific consistency score is generated for each text claim by calculating the maximum similarity between the text embedding of each text claim and the image embedding of all visual works. Combine all claim-specific consistency scores to generate the aforementioned consistency score.

[0016] Preferably, the step of generating a final matching score by combining the resume score, the portfolio score, and the consistency score specifically includes: Assign a preset weight to each of the resume score, the portfolio score, and the consistency score; The weighted sum of the scores is calculated as the final matching score.

[0017] Compared with the prior art, the advantages of this invention are: This invention, by introducing parallel resume analysis and portfolio analysis modules, enables comprehensive processing of both textual and visual modal data. The system can not only understand the textual content of a resume but also "see" and analyze the visual works in a portfolio, thus providing an intuitive and in-depth assessment of the candidate's actual skills and output.

[0018] The resume analysis module of this invention not only extracts text but also analyzes its two-dimensional spatial layout, scoring aesthetic elements such as resume alignment, white space, and font consistency. For potential portfolios, it also uses the portfolio analysis module to quantitatively evaluate the consistency of the candidate's artistic style and form a dominant style profile for comparison. This can greatly improve the efficiency of resume screening and the accuracy of resume matching assessment for visual design positions.

[0019] This invention also utilizes a resume-portfolio consistency module to semantically match and verify textual claims in the resume with visual evidence in the portfolio, enabling it to determine whether the skills claimed by the candidate are reflected in their work. This method significantly improves the reliability and accuracy of the screening results. Attached Figure Description

[0020] Figure 1 This is a flowchart of a dynamic resume evaluation method based on a multimodal large model proposed in this invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0022] refer to Figure 1 This paper describes a dynamic resume evaluation method based on a multimodal large model in this invention.

[0023] First, applicant data is received through the data receiving module. This module can receive data from various sources via an application programming interface (API) or web interface, including job descriptions, applicant resumes (e.g., in PDF format), and portfolios (e.g., a set of image files). The received data is stored in the data storage module for subsequent processing.

[0024] Then, the following analysis process is executed in parallel or sequentially: The resume analysis module performs resume analysis to evaluate the semantic content and visual presentation of the resume. This process includes: Text and layout extraction: Receive resume documents and extract text tags and their corresponding bounding box coordinates using an optical character recognition (OCR) engine.

[0025] Multimodal embedding generation: A text-image multimodal model based on the Transformer architecture (such as LayoutLMv3) is used to process the document image into a series of image patches. These image patches, along with text tags, are then input into an encoder. The specific input embedding is the sum of text embedding, one-dimensional positional embedding, and two-dimensional layout positional embedding. This combination process can be represented by the following formula: ; In the formula, It is the final input embedding. It is the embedding of text content. It is a position embedding of the marker in a one-dimensional sequence. It is a two-dimensional location embedded on the page.

[0026] Semantic and aesthetic scoring: Utilizing the encoder's output context representation, the following is performed: Semantic score evaluation: The text content of the resume and job description is processed by the LayoutLMv3 text encoder to generate document-level embedding vectors that represent the overall semantics of each. Then, the similarity between the two vectors is calculated using cosine similarity. The result (a value between 0 and 1) is the semantic score.

[0027] Aesthetic Score Evaluation: A 2D positional embedding extracted from LayoutLMv3 is input to an auxiliary classification head. This classification head is trained on a resume dataset pre-labeled with aesthetic quality (e.g., scores from 1 to 5) by design experts. The model generates an objective aesthetic score for a new resume by learning to associate specific layout patterns (such as alignment, whitespace usage, and visual balance) with human aesthetic preferences.

[0028] The portfolio analysis module performs portfolio analysis to assess the applicant's artistic and technical skills. This process includes: Feature extraction: Each image in the portfolio is processed by a pre-trained Visual Transformer (ViT) encoder to generate a high-dimensional feature embedding.

[0029] Content analysis: These embeddings are used to categorize the themes of the work to match job requirements, thereby generating a content score. Specifically: A classification head is attached to the ViT model, which has an output node for each predefined content category (e.g., "UI / UX design", "product photography") and generates an independent probability score for each category using a sigmoid activation function. These scores are then compared to the skills required in the job description to calculate the content score.

[0030] Style analysis: The embeddings of all portfolio images are input into a clustering algorithm to evaluate style attributes. Specifically, this includes: Style consistency: Style consistency is quantified by calculating the within-cluster variance of each cluster. ; In the formula For the variance within the cluster, It is the number of works in the cluster. It is the first Feature embedding of each work It is the centroid of the cluster. Lower variance indicates higher consistency, and this value can be normalized to generate a style consistency score.

[0031] Dominant Style Profile: The centroid of the largest cluster is calculated to represent the applicant's dominant style, and then compared with the cosine similarity of the cosine similarity to the "ideal style vector" predefined for a specific job role to generate a style matching score.

[0032] Generating a style score: The style consistency score and style matching score are weighted and combined to generate a style score. The weights can be adjusted according to the job requirements for style consistency.

[0033] The consistency module performs consistency analysis based on a contrastive vision-language model, such as CLIP (Contrastive Language-Image Pre-trained). This process includes: Claim identification: A natural language processing (NLP) component (e.g., a fine-tuned BERT model) parses the resume using named entity recognition (NER) to identify specific, verifiable claims (e.g., “created minimalist logos for enterprise clients”).

[0034] Cross-modal embedding generation: CLIP employs a dual-encoder architecture, comprising an image encoder (such as ViT) and a text encoder (such as Transformer), which project images and text into a shared 512-dimensional embedding space. This module leverages these encoders to generate aligned embeddings for each text claim and each image in the portfolio.

[0035] Similarity calculation: The cosine similarity is calculated between the L2 normalized embedding of each text claim and the L2 normalized embedding of each image in the portfolio. To measure consistency: ; In the formula, The cosine similarity between the L2-normalized embeddings of the text claims and the L2-normalized embeddings of the images. L2-normalized embedding of textual claims This is the L2-normalized embedding for the image.

[0036] For each claim, the maximum similarity score between it and all images is taken as the consistency score of that claim.

[0037] Overall consistency score: Combine the scores of all verifiable claims (e.g., by averaging) to calculate an overall consistency score.

[0038] This step addresses the "credibility gap" in existing technologies by adding a cross-modal fact-checking mechanism. It transforms a resume from a self-statement document into a set of testable hypotheses. Leveraging CLIP's zero-sample capability, this mechanism can validate various claims without requiring specialized training for each possible skill, thus introducing unprecedented trust and accuracy to automated screening.

[0039] Finally, multimodal fusion and scoring are performed. The outputs of the three aforementioned analysis modules (resume semantic score, resume aesthetic score, portfolio content score, portfolio style score, and overall consistency score) are passed to the multimodal fusion and scoring module. This module calculates the final matching score using a fusion strategy. A preferred implementation is post-fusion, combining the scores calculated independently by each module. The final matching score is then calculated. The result is obtained through weighted average calculation: ; In the formula, It is the first The score of each module, These are the weights assigned to that score. These weights can be dynamically adjusted based on the job role.

[0040] The present invention also includes a dynamic resume evaluation system based on a multimodal large model for implementing the above method, comprising a data receiving module, a data storage module, a resume analysis module, a portfolio analysis module, a consistency analysis module, and a fusion module; The data receiving module is used to receive applicant data and job description data, specifically including job description, applicant's resume and portfolio; The data storage module is used to store applicant data and job description data received by the data receiving module; The resume analysis module is used to analyze resumes based on text content and layout features to generate resume scores; The portfolio analysis module is used to analyze portfolios based on visual content and style features to generate portfolio scores; The consistency module is used to calculate a consistency score by comparing textual claims in the resume with visual evidence in the portfolio. The fusion module is used to generate a final matching score by combining the resume score, the portfolio score, and the consistency score.

[0041] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0042] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A dynamic resume evaluation method based on a multimodal large model, characterized in that, Includes the following steps: Receive a file containing the applicant's CV information, and a portfolio of the applicant's work containing one or more visual works; Based on the file containing the applicant's resume information, the resume analysis module is used to analyze the file to generate a resume score; Based on the applicant's portfolio, the portfolio analysis module is used to analyze the portfolio in order to generate a portfolio score; Based on the file containing the applicant's resume information and the applicant's portfolio, a consistency score is calculated using the consistency analysis module. Specifically: Use a natural language processing model to identify one or more verifiable textual claims from the text content of the resume; For each identified text claim, a text embedding is generated using a contrastive language-image model text encoder; For each visual work in the portfolio, an image embedding is generated using the image encoder of the contrastive language-image model; A claim-specific consistency score is generated for each text claim by calculating the maximum similarity between the text embedding of each text claim and the image embedding of all visual works. Combine all claim-specific consistency scores to generate the aforementioned consistency score; The resume score, portfolio score, and consistency score are combined to generate a final matching score to determine the degree of match between the applicant and the position.

2. The dynamic resume evaluation method based on a multimodal large model according to claim 1, characterized in that, The process of analyzing the file containing the applicant's resume information using a resume analysis module to generate a resume score specifically includes: Text markers and their corresponding two-dimensional position coordinates are extracted from the file using optical character recognition technology. Using a pre-trained document understanding model, text embeddings and layout embeddings are generated based on the text tags and the two-dimensional position coordinates; A semantic score is calculated based on the semantic similarity between the embedded text and the preset job description; Based on the layout embedding, a layout score is calculated using an aesthetic evaluation model; The semantic score and the layout score are combined to generate the resume score.

3. The dynamic resume evaluation method based on a multimodal large model according to claim 1, characterized in that, The process of analyzing the applicant's portfolio using a portfolio analysis module to generate a portfolio score specifically includes: A visual Transformer model is used to generate a feature embedding for each visual work in the portfolio; A content score is calculated based on the content matching degree between the embedded features and the preset job requirements. A style score is calculated by performing cluster analysis on the feature embeddings to evaluate the style attributes among the visual works. The content score and the style score are combined to generate the portfolio score.

4. The dynamic resume evaluation method based on a multimodal large model according to claim 3, characterized in that, The step of performing cluster analysis on the feature embeddings to evaluate the style attributes among the visual works, thereby calculating a style score, specifically includes: The features of all visual works in the collection are embedded into a set and input into a clustering algorithm; Based on the output of the clustering algorithm, the intra-cluster distance or variance of one or more clusters is calculated to generate a style consistency index. The style score is determined based on the style consistency index.

5. A dynamic resume evaluation system based on a multimodal large model, used to implement the dynamic resume evaluation method based on a multimodal large model as described in claims 1-4, characterized in that, It includes a data receiving module, a data storage module, a resume analysis module, a portfolio analysis module, a consistency analysis module, and a fusion module; The data receiving module is used to receive applicant data and job description data, specifically including job description, applicant's resume and portfolio; The data storage module is used to store applicant data and job description data received by the data receiving module; The resume analysis module is used to analyze resumes based on text content and layout features to generate resume scores; The portfolio analysis module is used to analyze portfolios based on visual content and style features to generate portfolio scores; The consistency analysis module is used to calculate a consistency score by comparing textual claims in the resume with visual evidence in the portfolio. The fusion module is used to generate a final matching score by combining the resume score, the portfolio score, and the consistency score.

6. The dynamic resume evaluation system based on a multimodal large model according to claim 5, characterized in that, The process of analyzing resumes based on text content and layout features to generate resume scores specifically includes: The text markers and their corresponding two-dimensional location coordinates are extracted from the applicant's resume received by the data receiving module using optical character recognition technology. Using a pre-trained document understanding model, text embeddings and layout embeddings are generated based on the text tags and the two-dimensional position coordinates; A semantic score is calculated based on the semantic similarity between the embedded text and the preset job description; Based on the layout embedding, a layout score is calculated using an aesthetic evaluation model; The semantic score and the layout score are combined to generate the resume score.

7. The dynamic resume evaluation system based on a multimodal large model according to claim 5, characterized in that, The analysis of portfolios based on visual content and style features to generate portfolio scores specifically includes: A visual Transformer model is used to generate a feature embedding for each visual work in the portfolio; A content score is calculated based on the content matching degree between the embedded features and the preset job requirements. A style score is calculated by performing cluster analysis on the feature embeddings to evaluate the style attributes among the visual works. The content score and the style score are combined to generate the portfolio score.

8. A dynamic resume evaluation system based on a multimodal large model according to claim 5, characterized in that, The method of calculating a consistency score by comparing textual claims in the resume with visual evidence in the portfolio specifically includes: Use a natural language processing model to identify one or more verifiable textual claims from the text content of the resume; For each identified text claim, a text embedding is generated using a contrastive language-image model text encoder; For each visual work in the portfolio, an image embedding is generated using the image encoder of the contrastive language-image model; A claim-specific consistency score is generated for each text claim by calculating the maximum similarity between the text embedding of each text claim and the image embedding of all visual works. Combine all claim-specific consistency scores to generate the aforementioned consistency score.

9. A dynamic resume evaluation system based on a multimodal large model according to claim 5, characterized in that, The process of generating a final matching score by combining the resume score, the portfolio score, and the consistency score specifically includes: Assign a preset weight to each of the resume score, the portfolio score, and the consistency score; The weighted sum of the scores is calculated as the final matching score.

Images