A method, apparatus, device, and storage medium for predicting professional competence based on multimodal fusion and cross-scene mapping.

Abstract

This invention provides a method, apparatus, device, and storage medium for predicting vocational abilities based on multimodal fusion and cross-scenario mapping. By constructing a vocational education ability ontology library containing an ability dimension layer, an ability level layer, and a scenario mapping rule layer, it uses knowledge graph embedding technology to map heterogeneous evaluation indicators from multiple scenarios to a unified ability semantic space. On this basis, it uses the semantic definition of the ability dimension to guide a cross-modal attention mechanism to selectively align and fuse multimodal features, and combines weakly supervised multi-instance learning to achieve automatic identification of latent abilities. Finally, it inputs the fused ability features from multiple time points into a temporal encoder to complete the prediction of ability development trends and vocational suitability.

Description

Technical Field

[0001] This invention relates to the field of competency profiling, and in particular to a method, apparatus, device, and storage medium for predicting professional competency based on multimodal fusion and cross-scene mapping. Background Technology

[0002] In vocational education evaluation systems, students' vocational competence performance data is scattered across multiple assessment scenarios, including practical training courses, corporate internships, skills competitions, and online learning. Each scenario employs independent evaluation standards and scoring systems, generating data in various modalities such as text reports, operation videos, work images, and behavior logs. The core technical challenge currently facing the field of intelligent vocational education assessment is how to effectively integrate this dispersed, standardized, and modalized competence performance data to form a unified, dynamic, and predictable assessment of students' vocational competence.

[0003] The essence of this challenge lies in the lack of a common semantic benchmark among evaluation indicators across different scenarios. For example, while the scores for "live-stream script writing quality" in practical training courses and "customer complaint handling satisfaction" in corporate internships reflect students' abilities in content creation and customer service respectively, their scoring dimensions, granularity, and standards are completely different, making it impossible for existing systems to compare and integrate them under the same competency coordinate system. Meanwhile, students' implicit professional qualities (such as teamwork, resilience, and innovative thinking) are often implicitly contained in their behavioral patterns in video operation, the semantic expression of written reports, and the decision-making trajectories in interaction logs. Single-modal data processing methods cannot extract and correlate these implicit signals scattered across different carriers into quantifiable competency indicators. Furthermore, existing assessment systems can only statically score students' current competency levels and cannot model their growth trajectory based on competency data accumulated over multiple time points, thereby predicting their future development direction and career suitability.

[0004] In view of the above, this application is hereby submitted. Summary of the Invention

[0005] This invention discloses a method, device, equipment, and storage medium for predicting vocational abilities based on multimodal fusion and cross-scenario mapping. It aims to solve the technical problem that in the context of diverse assessment scenarios in vocational education, the heterogeneity of evaluation standards leads to the inability to semantically align multimodal ability data, which in turn makes it difficult to uniformly quantify and assess explicit and implicit abilities and predict their development trends.

[0006] The first embodiment of the present invention provides a method for predicting professional competence based on multimodal fusion and cross-scene mapping, comprising:

[0007] Raw performance data of students in multiple vocational education scenarios are collected and preprocessed, and each data point is labeled with a timestamp and scenario tag to generate a structured multimodal dataset with scenario annotations; the raw performance data includes text data, visual data and behavioral data;

[0008] Based on a pre-built vocational education capability ontology, knowledge graph embedding technology is used to semantically map the original evaluation indicators in each scenario of the structured multimodal dataset to generate a unified set of capability semantic vectors; the vocational education capability ontology includes a capability dimension layer, a capability level layer, and a scenario mapping rule layer.

[0009] Feature extraction is performed on each modality data in the structured multimodal dataset to generate a multimodal feature vector group. Based on the capability dimension definition in the unified capability semantic vector set, a capability dimension-guided cross-modal attention mechanism is introduced to align and fuse different modal features expressing the same capability dimension in the multimodal feature vector group in the unified capability semantic space to generate a fused capability feature vector.

[0010] Based on the fusion ability feature vector, implicit abilities are identified from the multimodal feature vector group through weakly supervised multi-instance learning, and implicit ability assessment results are generated. The fusion ability feature vectors of the students at multiple time points are input into the time encoder to predict the ability development trend value and the probability distribution of career suitability, and the ability prediction results are output.

[0011] Preferably, the capability dimension layer defines professional capability categories including data analysis capabilities, customer service capabilities, innovative thinking, and teamwork.

[0012] The capability level layer defines five level standards, from L1 to L5, for each capability dimension;

[0013] The scenario mapping rule layer establishes a mapping relationship from each scenario evaluation index to capability dimensions and corresponding levels.

[0014] Preferably, the knowledge graph embedding technology adopts the TransR model, wherein the semantic mapping process specifically involves: calculating the semantic similarity between the original evaluation indicators in each scenario and the standardized ability nodes in the vocational education ability ontology through the TransR model; mapping each original evaluation indicator to the semantically closest standardized ability node based on the semantic similarity; and assigning corresponding confidence weights to the mapping results according to the semantic similarity.

[0015] Preferably, the feature extraction of each modality data in the structured multimodal dataset specifically involves:

[0016] For text-based data, a domain-adjusted pre-trained language model combined with a prompting and guidance strategy is used to extract semantic feature vectors from the text.

[0017] For artwork images in visual data, the visual Transformer model is used to extract image feature vectors, and a cross-modal semantic association with text semantic feature vectors is achieved through an image-text alignment model; for operation videos in visual data, the SlowFast dual-path network is used to extract video action feature vectors.

[0018] Long Short-Term Memory (LSTM) networks are used to extract feature vectors of behavioral sequences from behavioral data.

[0019] Preferably, the capability dimension-guided cross-modal attention mechanism is as follows: using the semantic representation of each capability dimension in the unified capability semantic vector set as the query vector, attention weighting is applied to the features of each modality in the multimodal feature vector group, selectively enhancing the feature components related to the same capability dimension in each modality and aggregating them; the alignment and fusion process also adopts a contrastive learning strategy, using capability dimension labels as supervision signals to enhance the clustering tightness of cross-modal features belonging to the same capability dimension.

[0020] Preferably, the step of identifying latent capabilities from the multimodal feature vector set through weakly supervised multi-instance learning specifically involves:

[0021] The multimodal feature vectors generated by the same student in a single task are combined into a sample package. The implicit ability label is used as the package-level label. The most relevant feature component to the implicit ability is located from each feature vector in the sample package through the attention pooling mechanism. The implicit ability assessment result is output based on the location result. The implicit ability assessment result is also accompanied by the assessment basis generated by the SHAP interpretability analysis method. The assessment basis includes the specific behavioral indicators that trigger the implicit ability judgment and their corresponding ability levels.

[0022] Preferably, the time encoder adopts a time model based on the Transformer architecture; the prediction time range of the prediction ability development trend value is 3 to 6 months in the future; the career suitability probability distribution represents the matching probability value of the student in multiple candidate career directions.

[0023] The second embodiment of the present invention provides a career ability prediction device based on multimodal fusion and cross-scene mapping, comprising:

[0024] The preprocessing unit is used to collect and preprocess students' raw competency performance data in multiple vocational education scenarios, label each data point with a timestamp and scenario tag, and generate a structured multimodal dataset with scenario annotations; the raw competency performance data includes text data, visual data, and behavioral data;

[0025] A cross-scenario semantic mapping unit is used to perform semantic mapping on the original evaluation indicators of each scenario in the structured multimodal dataset based on a pre-built vocational education capability ontology library and using knowledge graph embedding technology to generate a unified capability semantic vector set; the vocational education capability ontology library includes a capability dimension layer, a capability level layer and a scenario mapping rule layer.

[0026] The multimodal fusion unit is used to extract features from each modality in the structured multimodal dataset to generate a multimodal feature vector group. Based on the capability dimension definition in the unified capability semantic vector set, a capability dimension-guided cross-modal attention mechanism is introduced to align and fuse different modal features expressing the same capability dimension in the multimodal feature vector group in the unified capability semantic space to generate a fused capability feature vector.

[0027] The prediction unit is used to identify latent abilities from the multimodal feature vector group based on the fusion ability feature vector and generate latent ability assessment results through weakly supervised multi-instance learning; input the student's fusion ability feature vector at multiple time points into the time encoder to predict the ability development trend value and career suitability probability distribution, and output the ability prediction result.

[0028] The third embodiment of the present invention provides a career ability prediction device based on multimodal fusion and cross-scene mapping, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to implement a career ability prediction method based on multimodal fusion and cross-scene mapping as described in any of the above embodiments.

[0029] The fourth embodiment of the present invention provides a computer-readable storage medium storing a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, to implement the occupational ability prediction method based on multimodal fusion and cross-scene mapping as described in any of the above embodiments.

[0030] Based on the vocational ability prediction method, device, equipment, and storage medium provided by this invention, which is based on multimodal fusion and cross-scenario mapping, a vocational education ability ontology library is constructed, which includes an ability dimension layer, an ability level layer, and a scenario mapping rule layer. Knowledge graph embedding technology is used to map heterogeneous evaluation indicators of multiple scenarios to a unified ability semantic space. On this basis, the cross-modal attention mechanism guided by the semantic definition of the ability dimension is used to selectively align and fuse multimodal features. Combined with weakly supervised multi-instance learning, the implicit ability is automatically identified. Finally, the fused ability features of multiple time nodes are input into a time-series encoder to complete the prediction of ability development trends and vocational suitability. Attached Figure Description

[0031] Figure 1 This is a flowchart illustrating a method for predicting professional abilities based on multimodal fusion and cross-scene mapping, provided in the first embodiment of the present invention.

[0032] Figure 2 This is a schematic diagram of a professional ability prediction device based on multimodal fusion and cross-scene mapping provided in the second embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0035] This invention discloses a method, device, equipment, and storage medium for predicting vocational abilities based on multimodal fusion and cross-scenario mapping. It aims to solve the technical problem that in the context of diverse assessment scenarios in vocational education, the heterogeneity of evaluation standards leads to the inability to semantically align multimodal ability data, which in turn makes it difficult to uniformly quantify and assess explicit and implicit abilities and predict their development trends.

[0036] The first embodiment of the present invention provides a method for predicting professional competence based on multimodal fusion and cross-scene mapping, which can be executed by a professional competence prediction device based on multimodal fusion and cross-scene mapping (hereinafter referred to as prediction device or system), specifically, by one or more processors within the prediction device, to at least implement the following steps:

[0037] S101, collect and preprocess students' raw competency performance data in multiple vocational education scenarios, label each data point with a timestamp and scenario tag, and generate a structured multimodal dataset with scenario tags; the raw competency performance data includes text data, visual data, and behavioral data;

[0038] In this embodiment, the prediction device can be a desktop computer, laptop computer, server, or other terminal with data processing capabilities. The prediction device can be equipped with a corresponding operating system and application software, and the functions required in this embodiment can be realized through the combination of the operating system and application software.

[0039] The system connects to multiple business subsystems within the vocational education platform to collect raw performance data of students in various vocational education scenarios. Specifically, the system collects structured answer data of students' general and professional abilities from the online assessment platform; it collects data on students' submitted training reports, design drawings, operation videos, 3D models, and code projects from the training and project platform; it obtains internship comments, job assessment results, and behavioral observation records filled out by corporate mentors from the internship management system; and it obtains skills competition scoring sheets, team collaboration records, and innovation achievement documents from the competition and activity system. The above data is divided into three modal types: text data, including assessment answer records, training reports, internship comments, competition instructions, etc.; visual data, including operation videos, project images, and training scene videos, etc.; and behavioral data, including operation logs, response time sequences, interaction event records, error retries, and resource access paths, etc. After data collection, the system preprocesses the raw performance data, including data cleaning, format standardization, and missing value handling. Each data entry is labeled with its timestamp, corresponding scenario tag, and task identifier. The scenario tag indicates the specific scenario from which the data originated, such as online assessments, training courses, corporate internships, or skills competitions. The task identifier links multiple data entries from different modalities generated under the same assessment task. Following this preprocessing, the system organizes all data into a structured multimodal dataset with scenario annotations. Each record in this dataset contains five fields: raw performance data ontology, modality type label, timestamp, scenario tag, and task identifier.

[0040] S102, based on the pre-built vocational education capability ontology, the original evaluation indicators in each scenario of the structured multimodal dataset are semantically mapped using knowledge graph embedding technology to generate a unified capability semantic vector set; the vocational education capability ontology includes a capability dimension layer, a capability level layer, and a scenario mapping rule layer.

[0041] First, a vocational education competency ontology is constructed, which is organized in a three-layer structure. The first layer is the competency dimension layer, which defines multiple vocational competency categories covering the core competencies of vocational education, including but not limited to data analysis ability, customer service ability, content creation ability, innovative thinking, teamwork ability, stress resistance ability, and professional ethics. Each competency dimension is accompanied by a standardized semantic description for subsequent semantic matching with evaluation indicators for various scenarios.

[0042] The second layer is the competency level layer, which defines five levels from L1 to L5 for each of the above competency dimensions. L1 represents the knowledge level, which means that students have a basic understanding of the competency area; L2 represents the comprehension level, which means that they can complete related tasks under guidance; L3 represents the proficiency level, which means that they can complete standardized tasks independently; L4 represents the mastery level, which means that they can flexibly apply the competency in complex situations; and L5 represents the expert level, which means that they can creatively solve unconventional problems in the field.

[0043] The third layer is the scenario mapping rule layer, which establishes a mapping relationship from specific evaluation indicators in various vocational education scenarios to ability dimensions and corresponding levels. For example, the "live broadcast script writing quality" scoring item in the practical training scenario maps to "content creation ability - L3", the "customer complaint handling satisfaction" scoring item in the corporate internship scenario maps to "customer service ability - L4", and the "solution originality score" scoring item in the skills competition scenario maps to "innovative thinking - L5". The above mapping rules are pre-formulated by domain experts based on the evaluation standard documents of each scenario and stored in the knowledge graph in the form of triples. Each triple contains three elements: source node (original evaluation indicator), relation type (mapping to), and target node (standardized ability node and its level).

[0044] After constructing the vocational education competency ontology, the system utilizes knowledge graph embedding technology to automatically semantically map the original evaluation indicators for each scenario in the generated structured multimodal dataset with scene annotations. Specifically, the system employs the TransR model as the knowledge graph embedding implementation. This model embeds entities and relations from the vocational education competency ontology into different vector spaces, and projects entities from the entity space to the corresponding relation space using relation-specific projection matrices. During the training phase, the system uses existing expert-annotated mapping rules in the ontology as positive sample triples and constructs negative sample triples through negative sampling. The optimization objective is to minimize the energy function of the positive sample triples and maximize the energy function of the negative sample triples, thereby training to obtain the vector representations of each original evaluation indicator and standardized competency node in the embedding space. During the mapping phase, for each record in the structured multimodal dataset carrying an original evaluation indicator, the system extracts the embedding vector of that indicator, projects it onto the relation space using the projection matrix of the corresponding relation, and calculates the semantic similarity between the projected vector and each standardized competency node in the ontology within the same relation space. The system selects the standardized capability node with the highest semantic similarity as the mapping target for the original evaluation index, and uses this semantic similarity value as the confidence weight of the mapping result to reflect the reliability of the mapping. For text-based data (such as internship comments and training reports) in the structured multimodal dataset, the system also extracts comment keywords and behavior tags through an entity linking model, mapping these unstructured text information to the corresponding standardized capability nodes in the ontology library. Through the above semantic mapping process, the system uniformly converts the original evaluation indicators from different scenarios and using different evaluation standards in the structured multimodal dataset into standardized capability representations under the ontology library framework. Each mapping result contains three elements: standardized capability node identifier, corresponding capability level, and confidence weight. All mapping results are aggregated to form a unified capability semantic vector set.

[0045] S103, extract features from each modality data in the structured multimodal dataset to generate a multimodal feature vector group; based on the capability dimension definition in the unified capability semantic vector set, introduce a capability dimension-guided cross-modal attention mechanism to align and fuse different modal features expressing the same capability dimension in the multimodal feature vector group in the unified capability semantic space to generate a fused capability feature vector.

[0046] First, features are extracted from each modality of the structured multimodal dataset with scene annotations. For text data, the system uses a BERT pre-trained language model fine-tuned with vocational education corpus. At the input end, a prompting strategy is designed to construct prompt templates oriented towards ability assessment. For example, the training report text is concatenated with prompts such as "The problem analysis ability level reflected in this text is..." before being input into the model. This guides the model to focus on semantic information related to vocational ability during the encoding process, and extracts the hidden state vector of the output layer as the text semantic feature vector of this text data. For images of works in the visual data, the system uses a visual Transformer model to segment the image into a fixed-size sequence of tiles before encoding, and extracts the output vector corresponding to the global classification label as the image feature vector.

[0047] Building upon this, the system establishes cross-modal semantic associations between the image feature vectors and text semantic feature vectors under the same task identifier using the CLIP image-text alignment model. This enables preliminary alignment between visual features in the image and semantic descriptions in the text within a shared embedding space, providing a foundation for fine-grained capability dimension alignment in subsequent cross-modal fusion. For operation videos in visual data, the system employs a SlowFast dual-path network. The slow path captures the spatial semantic features of operation actions at a lower frame rate, while the fast path captures the temporal dynamic changes of operation actions at a higher frame rate. The outputs of the two paths are fused laterally to generate a video action feature vector, which simultaneously represents information such as the type, sequence, and rhythm of the operation actions. For behavioral data, the system arranges operation logs, response time series, and interaction event records into a behavioral event sequence in chronological order. This sequence is then input into a Long Short-Term Memory (LSTM) network for encoding. A gating mechanism is used to capture long-range dependencies in the behavioral sequence, and the hidden state of the final time step is extracted as the behavioral sequence feature vector. This vector reflects the student's decision-making efficiency, operational fluency, error patterns, and problem-solving strategies during task execution. The above four types of feature vectors together form the multimodal feature vector group corresponding to this record.

[0048] After completing multimodal feature extraction, the system, based on the capability dimension definition in the unified capability semantic vector set, introduces a capability dimension-guided cross-modal attention mechanism to align and fuse the multimodal feature vector group. Specifically, the system extracts the semantic representation corresponding to each capability dimension from the unified capability semantic vector set and uses it as the query vector in the attention mechanism. For each capability dimension's query vector, the system uses the text semantic feature vector, image feature vector, video action feature vector, and behavior sequence feature vector in the multimodal feature vector group as the key vector and value vector, respectively. By calculating the attention score between the query vector and each modality's key vector, the system determines the contribution weight of each modality feature to that capability dimension. The attention score is calculated using the scaled dot product attention formula, which involves dividing the inner product of the query vector and the key vector by the square root of the vector dimension and then normalizing it using softmax to obtain the weight distribution. The system performs a weighted summation of the value vectors of each modality according to the weight distribution, selectively enhancing feature components highly correlated with the current capability dimension in different modalities, while suppressing feature components weakly correlated with that capability dimension. Finally, the weighted aggregated vector is used as the cross-modal fusion representation under that capability dimension. The system performs the above attention-weighted aggregation operation on all capability dimensions defined in the unified capability semantic vector set one by one, and after concatenating the fused representations of each capability dimension, it maps them through a fully connected layer to generate the fused capability feature vector of the record.

[0049] In the aforementioned alignment and fusion process, the system further enhances the fusion quality by employing a contrastive learning strategy. Specifically, the system uses the capability dimension labels in the unified capability semantic vector set as supervision signals. During the training phase, it constructs positive and negative sample pairs, where positive sample pairs consist of fused representations from different modalities belonging to the same capability dimension, and negative sample pairs consist of fused representations from different capability dimensions. The system uses a normalized temperature-scaled cross-entropy loss function as the optimization objective of contrastive learning. By minimizing the distance between positive sample pairs and maximizing the distance between negative sample pairs, the system ensures that cross-modal fused features belonging to the same capability dimension are more tightly clustered in the unified capability semantic space, while fused features belonging to different capability dimensions are further apart, thereby improving the ability of the fused capability feature vector to distinguish between different capability dimensions.

[0050] S104, Based on the fusion ability feature vector, latent abilities are identified from the multimodal feature vector group through weakly supervised multi-instance learning and latent ability assessment results are generated; the fusion ability feature vectors of the student at multiple time points are input into the time encoder to predict the ability development trend value and the probability distribution of career suitability, and the ability prediction results are output.

[0051] In terms of implicit ability identification, the system employs a weakly supervised multi-instance learning method to automatically identify implicit professional qualities such as teamwork ability, stress resistance, communication willingness, and innovative thinking from multimodal feature vector sets. Since implicit abilities cannot be directly quantified by a single scoring item like explicit abilities, their manifestations are often scattered across various data carriers, such as behavioral patterns in operation videos, semantic expressions in written reports, and decision-making trajectories in interaction logs. Therefore, the system constructs a sample package from the multimodal feature vector sets generated by the same student in a single task. This sample package contains four instances of the student in that task: textual semantic feature vectors, image feature vectors, video action feature vectors, and behavioral sequence feature vectors. The system uses implicit ability labels as package-level labels, meaning it only labels whether a student demonstrated a certain implicit ability in the task at the sample package level, without requiring precise labeling at the individual instance level. During the model inference phase, the system dynamically assigns attention weights to each instance in the sample bag using an attention pooling mechanism. Instances with higher weights are determined to be most relevant to the performance of the latent ability. Based on this, the system automatically locates the feature components most relevant to the latent ability within the sample bag and outputs the latent ability assessment result based on the location result. This result includes the level judgment of each latent ability dimension. On this basis, the system further uses the SHAP interpretability analysis method to generate assessment basis for the latent ability assessment result. Specifically, the SHAP method calculates the marginal contribution value of each input feature to the model output, quantifies the degree of influence of each feature component on the latent ability judgment result, and extracts the feature components with the highest contribution values. It then transforms the corresponding specific behavioral indicators and their numerical performance into readable assessment basis text, such as "The student's operational error rate in the high-pressure timed task increased by less than 5% and the task completion time did not increase significantly, so the stress resistance ability is judged to be L4," thus making the latent ability assessment result interpretable.

[0052] In terms of predicting temporal ability development, the system collects fusion ability feature vectors generated by the student at multiple time points, arranges them chronologically according to the timestamps of the data records corresponding to each fusion ability feature vector, and constructs the student's ability feature time series. The system inputs this ability feature time series into a temporal encoder based on the Transformer architecture. This encoder captures the long-range dependencies and evolution patterns between ability features at different time points through a multi-head self-attention mechanism, and introduces positional encoding to preserve the sequential information of the time series, thereby modeling the student's ability evolution trajectory. At the output of the temporal encoder, the system connects two parallel prediction heads: the first is a capability trend prediction head, which maps the temporal representation output by the encoder through a fully connected layer to the development trend value of each capability dimension over the next 3 to 6 months. This trend value represents the expected growth or decline of each capability dimension in numerical form. The second prediction head is a career fit prediction head, which maps the temporal representation output by the encoder through a fully connected layer and a softmax normalization layer to the probability distribution of the student's career fit in multiple candidate career directions. Each candidate career direction corresponds to a matching probability value, for example, "the matching probability of e-commerce operation is 78%, and the matching probability of supply chain management is 62%." The outputs of the two prediction heads together constitute the student's capability prediction result.

[0053] The system constructs a dynamic vocational competency profile for the student based on integrated competency feature vectors, implicit competency assessment results, and competency prediction results. This profile includes four dimensions: the first dimension is explicit competency scores, where the system maps the integrated competency feature vectors to the level standards of each competency dimension in the vocational education competency ontology using a classifier, outputting the quantitative scores and corresponding levels for each explicit competency dimension; the second dimension is implicit competency levels, directly referencing the level determination and assessment basis for each implicit competency dimension as determined by the implicit competency assessment results; the third dimension is growth increment data, where the system quantifies the student's stage-wise growth in each competency dimension by calculating the difference between the student's integrated competency feature vector at the current time point and the integrated competency feature vector at historical time points; and the fourth dimension is predicted trend data, directly referencing the development trend values ​​and career suitability probability distributions for each competency dimension output by the competency prediction results. The system re-executes each time it receives new competency performance data, dynamically updating the profile content across the four dimensions to ensure that the dynamic vocational competency profile reflects the student's latest competency status and development trend in real time.

[0054] After the dynamic professional competency profile is generated, the system integrates it with the original scene data in the structured multimodal dataset with scene annotations to automatically generate a visualized dynamic digital profile. This digital profile includes the following visualizations: a competency radar chart based on explicit competency scores, comparing the student's current scores in each explicit competency dimension with the average scores of students in the same group; a growth trend line chart based on incremental growth data, displaying the student's historical trajectory in each competency dimension with time on the horizontal axis and scores in each competency dimension on the vertical axis; a career advancement path map based on predicted trend data and career fit probability distribution, graphically displaying recommended career development directions and the matching probability of each direction, and annotating the competency dimensions and level requirements needed to reach the target career; and a certificate and experience timeline based on the original scene data, integrating the student's participation records, assessment results, and qualification certificates obtained in training courses, corporate internships, skills competitions, and project activities in chronological order. The aforementioned visualized dynamic digital archives support three terminal display formats: web, mini-program, and PDF, to meet the needs of different usage scenarios such as students' self-access, teachers' classroom presentations, and corporate recruitment reviews. At the same time, the system sets up hierarchical access control based on user roles. Students can view their own complete archives, teachers can view the competency profiles and teaching-related analyses of students in their classes, and corporate and university users can only view students' structured competency summary information according to their authorized scope.

[0055] The system also generates differentiated intelligent feedback for different user roles based on dynamic professional competency profiles and visualized dynamic digital archives. For students, the system automatically matches learning resources and practical training tasks from the vocational education platform with the explicit competency scores and implicit competency levels below the target level in the profile, combined with developmental bottlenecks identified in the predicted trend data, generating personalized learning path recommendations and solutions for improving weak competencies. For teachers, the system aggregates dynamic professional competency profile data for all students in their classes, generating a class competency distribution heatmap. This heatmap displays the level distribution of each student across all competency dimensions in a matrix format, identifying overall class competency gaps and teaching weaknesses, and providing teaching improvement suggestions, including recommendations to increase training in certain competency dimensions and adjust teaching methods. For enterprises or universities, the system extracts the four-dimensional information from the dynamic professional competency profile into a structured competency summary, outputting the level, growth trend, and career suitability of each competency dimension in a standardized format, along with a career potential assessment report generated based on the competency prediction results. This supports enterprises in talent recruitment and universities in accurate competency matching and potential assessment for further education recommendations.

[0056] In implementing this invention, the inventors discovered that the capability-dimensional guided cross-modal attention mechanism, when fusing multimodal feature vector groups, assigns attention weights solely based on the semantic similarity between each modal feature and the capability dimension query vector, implicitly assuming that the representation reliability of each modal feature for the same capability dimension is equal. However, in practical applications, the representation reliability of different modalities for the same capability dimension varies significantly with scene type and task characteristics. For example, in practical training scenarios, video action feature vectors have higher representation reliability for the "operational standardization" capability than text semantic feature vectors, while in competition and defense scenarios, text semantic feature vectors have higher representation reliability for the "innovative thinking" capability than video action feature vectors. Because the attention mechanism cannot distinguish between "features semantically related to the capability dimension but with unreliable representation" and "features semantically related to the capability dimension and with reliable representation," modal features with low representation reliability may receive excessively high fusion weights due to high semantic similarity in specific scenarios, introducing noise information and reducing the quality of the fused capability feature vector.

[0057] To address the aforementioned technical challenges, the inventors introduced a scene-aware modal reliability gating mechanism into the cross-modal attention mechanism. Specifically, the system constructs a scene context vector for each data record, based on the scene label and task identifier. This scene context vector is then concatenated with the feature vectors of each modality in the multimodal feature vector group and input into the corresponding gating network for each modality. Each gating network outputs a scalar gating value between zero and one, representing the reliability of the modal feature for capability assessment under the current scene conditions. The system element-wise multiplies the gating values ​​of each modality with the original attention weights calculated by the cross-modal attention mechanism. This ensures that modalities with low reliability in the current scene, even with high semantic similarity, will have their fusion contribution suppressed by the gating value, while the fusion contribution of modalities with high reliability is preserved. The parameters of each gating network are automatically learned through end-to-end training to determine the reliability patterns of each modality under different scene and task conditions, eliminating the need for manual rule pre-setting. After the above-mentioned gating modulation, the cross-modal fusion process can adaptively and dynamically adjust the actual fusion contribution of each modality according to the scene characteristics, so that the generated fusion capability feature vector can preferentially aggregate the modal information with the highest reliability in different scenes, thereby improving the accuracy of capability assessment under diverse scene conditions.

[0058] The second embodiment of the present invention provides a career ability prediction device based on multimodal fusion and cross-scene mapping, comprising:

[0059] The preprocessing unit 201 is used to collect and preprocess the raw ability performance data of students in multiple vocational education scenarios, label each data with a timestamp and scenario label, and generate a structured multimodal dataset with scenario labels; the raw ability performance data includes text data, visual data and behavioral data;

[0060] The cross-scenario semantic mapping unit 202 is used to perform semantic mapping on the original evaluation indicators of each scenario in the structured multimodal dataset based on a pre-built vocational education capability ontology library and using knowledge graph embedding technology to generate a unified capability semantic vector set; the vocational education capability ontology library includes a capability dimension layer, a capability level layer and a scenario mapping rule layer.

[0061] The multimodal fusion unit 203 is used to extract features from each modality data in the structured multimodal dataset to generate a multimodal feature vector group; based on the capability dimension definition in the unified capability semantic vector set, a capability dimension-guided cross-modal attention mechanism is introduced to align and fuse different modal features expressing the same capability dimension in the multimodal feature vector group in the unified capability semantic space to generate a fused capability feature vector.

[0062] The prediction unit 204 is used to identify latent abilities from the multimodal feature vector group based on the fusion ability feature vector and generate latent ability assessment results through weakly supervised multi-instance learning; input the student's fusion ability feature vector at multiple time points into the time encoder to predict the ability development trend value and career suitability probability distribution, and output the ability prediction result.

[0063] The third embodiment of the present invention provides a career ability prediction device based on multimodal fusion and cross-scene mapping, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to implement a career ability prediction method based on multimodal fusion and cross-scene mapping as described in any of the above embodiments.

[0064] The fourth embodiment of the present invention provides a computer-readable storage medium storing a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, to implement the occupational ability prediction method based on multimodal fusion and cross-scene mapping as described in any of the above embodiments.

[0065] Based on the vocational ability prediction method, device, equipment, and storage medium provided by this invention, which is based on multimodal fusion and cross-scenario mapping, a vocational education ability ontology library is constructed, which includes an ability dimension layer, an ability level layer, and a scenario mapping rule layer. Knowledge graph embedding technology is used to map heterogeneous evaluation indicators of multiple scenarios to a unified ability semantic space. On this basis, the cross-modal attention mechanism guided by the semantic definition of the ability dimension is used to selectively align and fuse multimodal features. Combined with weakly supervised multi-instance learning, the implicit ability is automatically identified. Finally, the fused ability features of multiple time nodes are input into a time-series encoder to complete the prediction of ability development trends and vocational suitability.

[0066] Exemplary examples show that the computer program described in the third and fourth embodiments of the present invention can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in implementing a vocational ability prediction device based on multimodal fusion and cross-scene mapping. For example, the apparatus described in the second embodiment of the present invention.

[0067] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the aforementioned vocational ability prediction method based on multimodal fusion and cross-scene mapping, connecting various parts of the method through various interfaces and lines.

[0068] The memory can be used to store the computer program and / or modules. The processor, by running or executing the computer program and / or modules stored in the memory, and by calling the data stored in the memory, implements various functions of a career ability prediction method based on multimodal fusion and cross-scene mapping. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, text conversion function, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, text message data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0069] If the implemented module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0070] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0071] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for predicting professional competence based on multimodal fusion and cross-scene mapping, characterized in that, include: Raw performance data of students in multiple vocational education scenarios are collected and preprocessed, and each data point is labeled with a timestamp and scenario tag to generate a structured multimodal dataset with scenario annotations; the raw performance data includes text data, visual data and behavioral data; Based on a pre-built vocational education capability ontology, knowledge graph embedding technology is used to semantically map the original evaluation indicators in each scenario of the structured multimodal dataset to generate a unified set of capability semantic vectors; the vocational education capability ontology includes a capability dimension layer, a capability level layer, and a scenario mapping rule layer. Feature extraction is performed on each modality data in the structured multimodal dataset to generate a multimodal feature vector group; Based on the capability dimension definition in the unified capability semantic vector set, a capability dimension-guided cross-modal attention mechanism is introduced. The semantic representation of each capability dimension in the unified capability semantic vector set is used as the query vector. Original attention weights are calculated for the feature vectors of each modality in the multimodal feature vector group. A scene context vector is constructed based on the scene label and task identifier carried by each data record. This scene context vector is concatenated with the feature vectors of each modality in the multimodal feature vector group and input into the gating network corresponding to each modality. Each gating network outputs a gating value representing the reliability of the modality feature for capability evaluation under the current scene conditions. The gating values ​​of each modality are multiplied element-wise with the original attention weights to obtain the modulated attention weights. Based on the modulated attention weights, the different modal features expressing the same capability dimension in the multimodal feature vector group are weighted and aggregated, and aligned and fused in a unified capability semantic space to generate a fused capability feature vector. Based on the fusion ability feature vector, implicit abilities are identified from the multimodal feature vector group through weakly supervised multi-instance learning, and implicit ability assessment results are generated. The fusion ability feature vectors of the students at multiple time points are input into the time encoder to predict the ability development trend value and the probability distribution of career suitability, and the ability prediction results are output.

2. The professional ability prediction method based on multimodal fusion and cross-scene mapping according to claim 1, characterized in that, The capability dimension layer is defined to include categories of professional capabilities, including data analysis capabilities, customer service capabilities, innovative thinking, and teamwork. The capability level layer defines five level standards, from L1 to L5, for each capability dimension; The scenario mapping rule layer establishes a mapping relationship from each scenario evaluation index to capability dimensions and corresponding levels.

3. The professional ability prediction method based on multimodal fusion and cross-scene mapping according to claim 1, characterized in that, The knowledge graph embedding technology adopts the TransR model. The semantic mapping process is as follows: the semantic similarity between the original evaluation indicators in each scenario and the standardized ability nodes in the vocational education ability ontology is calculated using the TransR model; based on the semantic similarity, each original evaluation indicator is mapped to the semantically closest standardized ability node; and the corresponding confidence weight is assigned to the mapping result according to the semantic similarity.

4. The professional ability prediction method based on multimodal fusion and cross-scene mapping according to claim 1, characterized in that, The specific steps for extracting features from each modality in the structured multimodal dataset are as follows: For text-based data, a domain-adjusted pre-trained language model combined with a prompting and guidance strategy is used to extract semantic feature vectors from the text. For artwork images in visual data, the visual Transformer model is used to extract image feature vectors, and a cross-modal semantic association with text semantic feature vectors is achieved through an image-text alignment model; for operation videos in visual data, the SlowFast dual-path network is used to extract video action feature vectors. Long Short-Term Memory (LSTM) networks are used to extract feature vectors of behavioral sequences from behavioral data.

5. The professional ability prediction method based on multimodal fusion and cross-scene mapping according to claim 1, characterized in that, The capability dimension-guided cross-modal attention mechanism specifically involves: using the semantic representation of each capability dimension in the unified capability semantic vector set as the query vector, applying attention weights to the features of each modality in the multimodal feature vector group, selectively enhancing and aggregating the feature components related to the same capability dimension in each modality; the alignment and fusion process also employs a contrastive learning strategy, using capability dimension labels as supervision signals to enhance the clustering tightness of cross-modal features belonging to the same capability dimension.

6. The professional ability prediction method based on multimodal fusion and cross-scene mapping according to claim 1, characterized in that, The method of identifying latent capabilities from the multimodal feature vector set through weakly supervised multi-instance learning specifically involves: The multimodal feature vectors generated by the same student in a single task are combined into a sample package. The implicit ability label is used as the package-level label. The most relevant feature component to the implicit ability is located from each feature vector in the sample package through the attention pooling mechanism. The implicit ability assessment result is output based on the location result. The implicit ability assessment result is also accompanied by the assessment basis generated by the SHAP interpretability analysis method. The assessment basis includes the specific behavioral indicators that trigger the implicit ability judgment and their corresponding ability levels.

7. The professional ability prediction method based on multimodal fusion and cross-scene mapping according to claim 1, characterized in that, The time-series encoder adopts a time-series model based on the Transformer architecture; the prediction time range of the prediction ability development trend value is 3 to 6 months in the future; the career suitability probability distribution represents the matching probability value of the student in multiple candidate career directions.

8. A career ability prediction device based on multimodal fusion and cross-scene mapping, characterized in that, include: The preprocessing unit is used to collect and preprocess students' raw competency performance data in multiple vocational education scenarios, label each data point with a timestamp and scenario tag, and generate a structured multimodal dataset with scenario annotations; the raw competency performance data includes text data, visual data, and behavioral data; A cross-scenario semantic mapping unit is used to perform semantic mapping on the original evaluation indicators of each scenario in the structured multimodal dataset based on a pre-built vocational education capability ontology library and using knowledge graph embedding technology to generate a unified capability semantic vector set; the vocational education capability ontology library includes a capability dimension layer, a capability level layer and a scenario mapping rule layer. The multimodal fusion unit is used to extract features from each modality data in the structured multimodal dataset and generate a multimodal feature vector group. Based on the capability dimension definition in the unified capability semantic vector set, a capability dimension-guided cross-modal attention mechanism is introduced. The semantic representation of each capability dimension in the unified capability semantic vector set is used as the query vector. Original attention weights are calculated for the feature vectors of each modality in the multimodal feature vector group. A scene context vector is constructed based on the scene label and task identifier carried by each data record. This scene context vector is concatenated with the feature vectors of each modality in the multimodal feature vector group and input into the gating network corresponding to each modality. Each gating network outputs a gating value representing the reliability of the modality feature for capability evaluation under the current scene conditions. The gating values ​​of each modality are multiplied element-wise with the original attention weights to obtain the modulated attention weights. Based on the modulated attention weights, the different modal features expressing the same capability dimension in the multimodal feature vector group are weighted and aggregated, and aligned and fused in a unified capability semantic space to generate a fused capability feature vector. The prediction unit is used to identify latent abilities from the multimodal feature vector group based on the fusion ability feature vector and generate latent ability assessment results through weakly supervised multi-instance learning; input the student's fusion ability feature vector at multiple time points into the time encoder to predict the ability development trend value and career suitability probability distribution, and output the ability prediction result.

9. A career ability prediction device based on multimodal fusion and cross-scene mapping, characterized in that, The system includes a memory and a processor. The memory stores a computer program that can be executed by the processor to implement a career ability prediction method based on multimodal fusion and cross-scene mapping as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The device contains a computer program that can be executed by a processor of the device in which the computer-readable storage medium is located, to implement the occupational ability prediction method based on multimodal fusion and cross-scene mapping as described in any one of claims 1 to 7.

Images