College entrance examination volunteer consultation system and method based on large language model

Abstract

A method for implementing college entrance examination volunteer consultation based on a large language model is provided, comprising: constructing a multi-modal knowledge graph, wherein part of the entity nodes are linked to multi-modal data sources including videos and audios; receiving a natural language query of a user about college entrance examination volunteer recommendation, and extracting key entity information about the user's intention therefrom; generating a controllable-scale context subgraph from the multi-modal knowledge graph based on the extracted key entity information; processing multi-modal data of the multi-modal data sources linked to the nodes in the context subgraph to convert the multi-modal data into compact text summaries; and generating personalized college entrance examination volunteer recommendation by using a large language model for the natural language query of the user in combination with the structured information of the context subgraph and the text summaries of the multi-modal data.

Description

A College Entrance Examination Volunteer Consultation System and Method Based on a Large Language Model Technical Field

[0001] This invention relates to using artificial intelligence technology to provide college application advice, and more specifically, to a college application advice system and method based on a large language model. Background Technology

[0002] College application is a crucial step in determining a student's future academic and career path. Currently, students and parents face the dilemma of information overload coupled with information asymmetry. Nearly three thousand universities and nearly nine hundred undergraduate majors nationwide, combined with development opportunities in different cities, constitute an extremely vast and complex data network. Traditional application methods rely heavily on manual consultation and consulting scattered materials, which is inefficient and prone to decision-making biases.

[0003] With the development of artificial intelligence technology, Large Language Models (LLMs) have demonstrated powerful natural language understanding and generation capabilities, and are being explored for use in college application guidance. However, directly applying LLMs presents several significant challenges:

[0004] The information dimension is limited: Traditional search-enhanced generation (RAG) primarily processes text data, while the strength of institutions, the characteristics of their majors, and the vitality of cities can be presented more intuitively and richly through short videos on social media, audio sharing from alumni, podcasts, and other formats. Existing technologies struggle to integrate this multimodal information.

[0005] Knowledge limitations and illusions: The internal knowledge of LLM is limited to the date of training and cannot access the latest dynamic data such as admission policies, admission scores, and employment rates, which can easily lead to inaccurate "information illusions".

[0006] Context window limitation: The colleges, majors, regions, and their relationships involved in college application constitute a vast knowledge graph. If all potentially related entity information is attempted to be provided to an LLM, the total amount will easily exceed the context window limitation of current mainstream LLMs (such as GPT-4), resulting in information truncation or the model's inability to process it effectively.

[0007] Lack of structured reasoning ability: Simple text retrieval ignores the deep, structured relationships between entities (such as the strong correlation between a city's advantageous industries and a university's top majors), resulting in an insufficiently rigorous recommendation logic chain in LLM.

[0008] Therefore, in the specific and complex scenario of college application, there is an urgent need for an innovative technical solution that can process multimodal information, effectively deal with large-scale correlated data, and perform deep personalized reasoning. Summary of the Invention

[0009] According to one aspect of the present invention, a method for providing college entrance examination application advice based on a large language model is provided. The method includes: constructing a multimodal knowledge graph, wherein nodes include three types of entities: colleges, majors, and cities / regions, edges represent the relationships between them, and some entity nodes are linked to multimodal data sources including video and audio; receiving a user's natural language query for college entrance examination application recommendations and extracting key entity information about the user's intentions from it, wherein the user's intentions include at least scores, preferred majors, intended cities, and subject combinations; based on the extracted key entity information, adaptively generating a controllable-size context subgraph from the multimodal knowledge graph using a nearest neighbor sampling algorithm; processing the multimodal data from the multimodal data sources linked to the nodes in the context subgraph to convert the multimodal data into a compact text summary; and using the large language model to generate personalized college entrance examination application recommendations for the user's natural language query, combined with the structured information of the context subgraph and the text summary of the multimodal data, wherein a graph neural network is used to generate embedding vectors for the nodes of the context subgraph, and the embedding vectors are converted into attention bias matrices to adjust the attention weights of the large language model.

[0010] Optionally, the steps for generating the context subgraph include: initially selecting a batch of candidate entity nodes as seed nodes in the knowledge graph based on key entity information; calculating a comprehensive relevance score for all seed nodes and their one-hop neighbor nodes based on their matching degree with the user's intention; selecting a first predetermined number of nodes from high to low based on the comprehensive relevance score, and combining these nodes with a second predetermined number of nearest neighbor nodes in the graph embedding space to jointly form the finally generated context subgraph.

[0011] Optionally, generating embedding vectors for nodes in the context subgraph using a graph neural network and converting these embedding vectors into attention bias matrices to adjust the attention weights of the large language model includes: using a graph neural network to aggregate each node in the context subgraph and its neighboring nodes through multiple rounds of information transfer based on the graph structure of the context subgraph. During the aggregation process, the graph neural network combines key entity information encoded as vectors to adaptively determine which neighboring nodes are more important based on the user query, and includes important neighboring nodes related to the user query in the domain structure information of each node, thereby ultimately generating a graph embedding vector including its domain structure information for each node in the context subgraph; converting the text summary of the context subgraph and multimodal data into a structured text input into the large language model, and then using the large language model to perform the language processing. The text is segmented into lexical units, and the graph embedding vectors of nodes in the context subgraph are aligned with the lexical segmentation results of the large language model to obtain a mapping table indicating whether there is a mapping relationship between the lexical segmentation results and the nodes in the context subgraph. Each lexical unit in the lexical segmentation results is traversed, and an attention bias value is assigned to it to obtain a bias matrix. Lexical units with a mapping relationship are assigned a positive bias value, and lexical units without a mapping relationship are assigned a value of 0. The value of the positive bias value is proportional to the comprehensive relevance score of the corresponding node. In the self-attention mechanism of each Transforer layer of the large language model, the bias matrix is ​​added to the original attention score of the softmax function, so that the generated college entrance examination application recommendations pay more attention to entities with higher comprehensive relevance scores in the context subgraph.

[0012] Optionally, converting multimodal data into compact text summaries includes: for video data, locating core frames using a keyframe extraction algorithm and generating descriptions for the key frames using an image-to-text model, combining them into a video summary; for audio data, converting audio data into text using speech recognition technology and generating an audio summary of the core content using a text summarization model.

[0013] Optionally, the large language model and the graph neural network are jointly trained using a joint loss function L_joint. This joint loss function includes a large language model loss component L_lm that evaluates the quality of college application recommendations, and a contrastive learning loss component L_contrastive that brings entities that match the user's preferences closer together and pushes away entities that deviate from them in the embedding vector space. The joint loss function is expressed as: L_joint = λ L_lm + (1 - λ) L_contrastive, where λ is a hyperparameter used to balance the importance of the two loss components. Its value varies between 0 and 1. In the early stage of training, the value of λ is set to be less than 0.5 to focus more on optimizing the embedding quality of the graph neural network; in the later stage of training, the value of λ is set to be greater than 0.5 to focus more on improving the text generation capability of the large language model.

[0014] Optionally, user intent is obtained through the following processes: A large language model guides users to directly express their intent regarding college application choices in natural language queries during the dialogue; and / or collects user terminal usage behaviors related to college application choices within a specific time period, including at least one of the following: text and image content browsing behavior, audio and video content playback behavior, in-app search behavior, web search behavior, voice call behavior, and online chat behavior; collects user biometric information during the dialogue, including at least one of micro-expressions, blood pressure, and heart rate; analyzes user intent based on terminal usage behavior and / or biometric information; and / or generates probe content to mine potential user intent regarding college application choices, outputs the probe content to the user, and collects user intent regarding... Feedback information from the probed content is used to extract user intentions based on user feedback. The probed content includes at least one of the following: short Q&A questions about the user and / or their college application process; immersive experience videos about candidate colleges; and introductions to candidate colleges with emotional background music. The short Q&A questions about the user and / or their college application process are automatically generated by the large language model module during dialogue with the user, combining user information, current dialogue state, and / or contextual information. The module also generates distracting content to suddenly disrupt the user's current dialogue state, outputs the distracting content to the user, and collects user feedback on the distracting content. User intentions are extracted based on this feedback.

[0015] Optionally, if user intentions that are completely opposite to those directly expressed by the user are further extracted based on terminal usage behavior, biometric information, and / or user feedback on the probed content, then interfering content that is completely opposite to those directly expressed by the user is generated, the interfering content is output to the user, and feedback information on whether the user holds a positive or negative attitude towards the interfering content is collected. Then, the user intentions are updated based on the user's feedback information on the interfering content.

[0016] According to another example of the present invention, a system for providing college entrance examination application consultation based on a large language model is provided, comprising: a knowledge graph construction unit for constructing a multimodal knowledge graph, wherein nodes include three types of entities: colleges, majors, and cities / regions, edges represent the relationships between them, and some entity nodes are linked to multimodal data sources including video and audio; a query processing unit for receiving natural language queries from users regarding college entrance examination application recommendations and extracting key entity information about the user's intentions, wherein the user's intentions include at least scores, preferred majors, intended cities, and subject combinations; and a context subgraph generation unit for generating context subgraphs based on the extracted key entities. The system adaptively generates a scalable context subgraph from the multimodal knowledge graph using a nearest neighbor sampling algorithm. A multimodal transformation unit processes the multimodal data from the multimodal data sources linked to the nodes in the context subgraph, converting the multimodal data into a compact text summary. A recommendation generation unit uses a large language model to generate personalized college application recommendations based on the user's natural language query, combined with the structured information from the context subgraph and the text summary of the multimodal data. Specifically, a graph neural network is used to generate embedding vectors for the nodes in the context subgraph, and these embedding vectors are then converted into attention bias matrices to adjust the attention weights of the large language model.

[0017] Optionally, the multimodal conversion unit converts multimodal data into a compact text summary by: for video data, locating the core frame using a keyframe extraction algorithm, and generating descriptions for the key frame using an image-to-text model, combining them into a video summary; for audio data, converting the audio data into text using speech recognition technology, and generating an audio summary of the core content using a text summarization model.

[0018] According to another aspect of the present invention, a computer-readable recording medium having a computer program thereon for performing the above-described method is provided.

[0019] According to an exemplary embodiment of the present invention, in the college entrance examination volunteer consultation system, the limitations of information dimensions are broken through, and multimodal information such as video and audio is integrated into the volunteer recommendation process, making the recommendation results more vivid and intuitive, and providing more comprehensive information to assist decision-making. Through an adaptive subgraph compression algorithm, a small but accurate context is "tailor-made" for LLM from massive data, overcoming the hardware and software processing resource bottlenecks faced by large knowledge graphs in LLM applications. Based on the adaptive context subgraph, combined with user intentions, attention guidance further strengthens the reasoning focus of LLM, ensuring personalized matching between recommended content and user needs.

[0020] Furthermore, this invention fully explores the user's latent desires, especially those not directly expressed in language. Specifically, through a variety of technical exploration methods, this invention delves deeply and effectively into the user's true inner needs while simplifying the processing flow as much as possible. This allows users to not only experience the comfort of being understood by the machine, but also to receive college application recommendations that align with their inner needs, thus improving the accuracy of college application recommendations.

[0021] Furthermore, creatively joint training of large language models and graph neural networks, along with the construction of a phased loss measurement method, enables the recommendation results to be jointly optimized from both the language reasoning level and the context selection level. Attached Figure Description

[0022] Figure 1 shows a flowchart of a method for providing college entrance examination volunteer consultation based on a large language model according to an exemplary embodiment of the present invention;

[0023] Figure 2 shows an example of a session management interface according to an exemplary embodiment of the present invention;

[0024] Figure 3 shows an example of a consultation dialog window according to an exemplary embodiment of the present invention;

[0025] Figure 4 illustrates a flowchart of generating a context subgraph according to an exemplary embodiment of the present invention;

[0026] Figure 5 illustrates a flowchart of adjusting the attention weights of a large language model according to an exemplary embodiment of the present invention; and

[0027] Figure 6 shows a block diagram of a system for providing college entrance examination volunteer consultation based on a large language model according to an exemplary embodiment of the present invention. Detailed Implementation

[0028] The invention will be more readily understood from the following detailed description with reference to the accompanying drawings, wherein like reference numerals designate like elements.

[0029] According to the overall concept of the present invention, a method and system for recommending large language models based on multimodal knowledge graphs and adaptive context compression is provided for college entrance examination application scenarios.

[0030] The described method and system can integrate multimodal data into a knowledge graph, that is, construct a knowledge graph with colleges, majors, and cities as core entities. Unlike traditional knowledge graphs, the graph nodes of this invention not only contain structured attribute information (such as rankings, score lines, and employment rates), but also links (URIs) pointing to unstructured multimodal data, such as official university promotional videos, lecture recordings by renowned professors, podcasts introducing schools and disciplines, and various content on social media, such as vlogs shared by current students and interview videos of current students.

[0031] For knowledge graphs combining multimodal data, this invention employs an adaptive context subgraph compression algorithm to overcome the context window limitation of LLM (Local Level Management). This invention proposes a dynamic, user-intention-based subgraph compression algorithm. When a user inputs a query, the system does not simply traverse the graph. Instead, based on the user's scores and intentions, it initially selects a batch of highly relevant universities, majors, and cities as seeds. Then, a comprehensive scoring model is designed to calculate a "relevance score" for each candidate seed node in the graph, considering multiple dimensions such as score matching, major fit, and regional preference. Based on this, nearest neighbor sampling and compression are further performed. Specifically, based on the relevance score, the entities with the highest scores are selected as cores. Then, to ensure the richness and diversity of information, pre-computed graph embedding vectors are used to find at least one nearest neighbor of the core entities in the vector space. Finally, the cores and nearest neighbors together constitute a context subgraph of controllable size (e.g., no more than 50 entities in total) but with extremely high information value density.

[0032] For multimodal data associated with entities selected into the context subgraph, this invention initiates a lightweight processing flow for real-time summarization. For example, for a 2-minute introductory video of a college, 5 keyframes can be automatically extracted, a graph-to-text model can be used to generate a description, and the data can be integrated into a 100-word text summary.

[0033] The structured text description of the subgraph and the multimodal summary text are input together into the large language model for collaborative reasoning. In this process, the subgraph node embeddings calculated by the graph neural network (GNN) are used to generate attention bias through a small network, which guides the LLM to focus on the entities with the highest "relevance score" and their core features during reasoning, so as to achieve accurate and efficient reasoning.

[0034] Furthermore, considering that existing systems are limited by interaction constraints, making it difficult to fully and effectively capture user intentions, and that users find it difficult to easily express their needs to machines, recommendations often fail to meet users' true requirements. How to enable machines to better guide users, and even "read" their minds, through technological means is one of the technical problems that engineers need to solve in the age of artificial intelligence. Accordingly, this invention focuses on uncovering users' latent needs, not only by directly inquiring about users' application preferences through dialogue, but also by employing other auxiliary methods to further determine users' deeper needs. This allows users to fully reveal their inner world when considering various choices. This invention uses technological means to "read" users' subconscious needs, thereby making more reliable college application decisions.

[0035] Figure 1 shows a flowchart of a method for providing college entrance examination volunteer consultation based on a large language model according to an exemplary embodiment of the present invention.

[0036] In embodiments of the present invention, a session is used to manage the consultation service between the user (e.g., the candidate or their parent) and the AI ​​college application planner. The main interaction within the session is a dialogue interaction that combines college application decision-making schemes. The system interacts with the user through dialogue to exchange information and understand the user's application intentions.

[0037] Specifically, after a user logs into the system for the first time, they can choose to create a session, which will open a consultation dialogue window and collect the user's college entrance examination information (college entrance examination subjects and corresponding scores) and other personal information (name, place of origin, age, gender, height, weight, bonus points, special qualifications, etc.).

[0038] After a session is created, the user can view and manage all currently created sessions through the session management interface the next time they log in to the system. Figure 2 shows an example of a session management interface according to an exemplary embodiment of the present invention, wherein sessions are displayed in the form of a list. In the list, each record corresponds to a session, and the user can easily see the session identifier ID, basic information (such as username, reporting institution level, place of origin, subject and score), creation time, start time of the conversation, and end time of the conversation.

[0039] As can be seen from the exemplary embodiment of the present invention, AI-powered intelligent consultation primarily interacts through dialogue. During this process, the college entrance examination application consultation system can effectively guide users' questions and provide professional answers to their inquiries. Through this dialogue, users can gain a deeper understanding of various university majors, while the system can extract sufficient user preference information as a basis for application decisions.

[0040] To possess sufficient domain knowledge during dialogues with users, the system can pre-construct a multimodal knowledge graph. For example, in step S100, a multimodal knowledge graph is constructed, where nodes include three types of entities: colleges, majors, and cities / regions, edges represent the relationships between them, and some entity nodes are linked to multimodal data sources including video and audio.

[0041] Here, in the pre-built knowledge graph, entity nodes can include structured attribute information and related multimodal data source links. For example, the attributes of the university node "Sun Yat-sen University" include {"Rank": 10, "Doctoral Programs": XX, "Master's Programs": XX, "Introduction": "Sun Yat-sen University is a key university directly under the Ministry of Education of the People's Republic of China..."}, and contain at least one multimodal link, such as video_intro: "http: / / example.com / sysu_intro.mp4". The attributes of the major node "Computer Science and Technology" include {"Employment Salary": 20,000 / month, "Soft Science Ranking": A+}. The edge between the two represents their relationship as Sun Yat-sen University - [Offers] - Computer Science and Technology, and the edge between the university node "Sun Yat-sen University" and the city / region node "Guangzhou" is Sun Yat-sen University - [Located in] - Guangzhou. The edge between the city / region node "Guangzhou" and the major node "Computer Science and Technology" can be indirectly related through the relationship of "advantageous industries". Each entity node can be selectively linked to at least one multimodal data source, many of which come from social media and can fully reflect the real experience of university life, helping candidates make better choices.

[0042] In step S200, a natural language query from a user regarding college application recommendations is received, and key entity information about the user's preferences is extracted from it. The user's preferences include at least scores, preferred majors, intended cities, and subject combinations.

[0043] Figure 3 illustrates an example of a consultation dialogue window according to an exemplary embodiment of the present invention. In the consultation dialogue window shown in Figure 3, the left area can display the dialogue details, while the right side displays a decision-making table for college application. Specifically, the left area can collect the dialogue content expressed directly by the user in natural language. For example, if the user enters dialogue content during the process, such as, "We want to look at schools in Qingdao," then "Qingdao" can be extracted as the user's candidate city of interest (i.e., the city_tier entity).

[0044] Additionally, users can express their intentions and circumstances in a single query, such as, "I am a student from Guangdong, majoring in physics and chemistry, and I estimate my college entrance exam score to be 640. I am very interested in computer science and artificial intelligence and hope to develop my career in a first-tier city. Please recommend some suitable universities and majors." Correspondingly, key entities related to the user's intentions can be extracted from the above query: {province: "Guangdong", subject_combo: ["physics", "chemistry"], score: 640,interest: ["computer science", "artificial intelligence"], city_tier: "first-tier"}. These key entities have possible correspondences with nodes and their attributes in the multimodal knowledge graph, becoming the user intentions that can be prioritized during recommendations.

[0045] Next, in step S300, based on the extracted key entity information, a context subgraph of controllable size is adaptively generated from the multimodal knowledge graph using a nearest neighbor sampling algorithm. Figure 4 shows a flowchart of generating a context subgraph according to an exemplary embodiment of the present invention. As shown in Figure 4, in step S310, a batch of candidate entity nodes are initially selected as seed nodes in the knowledge graph based on the key entity information. Specifically, corresponding seed nodes can be selected from each node and its attribute information in the knowledge graph according to preset matching rules. For example, the admission score of the university seed node is within 10 points above or below the candidate's score, the subject selection is the same as the candidate's subject combination, and the major and region are the same as or similar to the user's intention. Taking the above query as an example, universities with admission scores between 630 and 650, located in first-tier cities, and offering computer science or AI majors can be selected as seeds, such as Sun Yat-sen University, South China University of Technology, and Beijing Institute of Technology.

[0046] In step S320, a comprehensive relevance score is calculated for all seed nodes and their one-hop neighbor nodes based on their matching degree with the user's intention. Here, the comprehensive score can be the sum of multiple weighted factors, which may include at least the matching degree between college admission scores and user scores, the matching degree between major rankings and user preferences, and the relevance between regional advantageous industries and the user's intended major.

[0047] Here, to provide sufficient filtering context, the scoring object can be further extended from the seed node to its one-hop neighbor nodes. For example, a comprehensive relevance score S can be calculated for each candidate university. For example, the score S(SYSU) for Sun Yat-sen University is w1. score_match(640, 642) + w2 major_match("CS / AI") + w3 The function `location_match("first-tier city")` uses coefficients `w1`, `w2`, and `w3` to measure the importance of matching scores, majors, and city regions, respectively. These coefficients can be predetermined based on expert rules or adjusted according to the user's level of attention. Each function returns a normalized score.

[0048] In step S330, based on the comprehensive relevance score, a first predetermined number of nodes are selected from high to low, and these nodes, along with a second predetermined number of nearest neighbor nodes in the graph embedding space, are combined to form the final generated context subgraph. Specifically, after obtaining the comprehensive relevance score of each node in step S320, the most matching nodes are selected in the knowledge graph, and at least one neighboring node is further selected in the graph embedding space.

[0049] For example, in the above example, top-5 universities such as Sun Yat-sen University and South China University of Technology can be selected. Furthermore, a nearest neighbor sampling algorithm can be used to further filter out some similar nodes. Here, at least one nearest neighbor node is found in the graph embedding space to supplement entities that are highly similar to the high-scoring core nodes semantically or functionally, but may not be directly connected in the graph topology, thus enhancing the diversity of recommendations. For example, although "University of Electronic Science and Technology of China" has a slightly lower score and location match, it is semantically very close to "Sun Yat-sen University - Computer Science" in the graph embedding space. It can also be sampled as one of K (K is a predetermined positive integer) nearest neighbor nodes, thus providing test takers with a high-quality option that is "unexpected yet reasonable." As an example, a compact subgraph containing approximately 20 universities, 15 related majors, and 4 first-tier cities can be generated.

[0050] Returning to Figure 1, in step S400, the multimodal data from the multimodal data sources linked to the nodes in the context subgraph are processed to convert the multimodal data into a compact text summary. As an example, for video data, keyframe extraction algorithms are used to locate the core frames, and an image-to-text model is used to generate descriptions for the key frames, combining them into a video summary; for audio data, speech recognition technology (e.g., ASR) is used to convert the audio data into text, and a text summarization model is used to generate an audio summary of the core content.

[0051] In step S500, a large language model is used to generate personalized college application recommendations for users' natural language queries, combined with the structured information of the context subgraph and the text summary of multimodal data. In this process, a graph neural network is used to generate embedding vectors for the nodes of the context subgraph, and the embedding vectors are converted into attention bias matrices to adjust the attention weights of the large language model.

[0052] Specifically, existing large language models, when faced with an entire context subgraph as the basis for reasoning to answer the question "Which university should I apply to?", tend to give similar attention to all candidate nodes, resulting in a generalized or even off-target answer. To address this issue, an exemplary embodiment of this invention first uses a graph neural network (GNN) to combine the user's personal circumstances and preferences (scores, preferences) within the entire context subgraph, providing focused guidance during the LLM's learning context (e.g., emphasizing "Sun Yat-sen University" or "Computer Science"), thus allowing the LLM to give more attention to these aspects and thereby provide recommended university choices.

[0053] Figure 5 shows a flowchart of adjusting the attention weights of a large language model according to an exemplary embodiment of the present invention. As shown in Figure 5, in step S510, a graph neural network is used to aggregate each node in the context subgraph and its neighboring nodes in multiple rounds regarding information transmission based on the graph structure of the context subgraph. During the aggregation process, the graph neural network combines key entity information encoded as vectors and adaptively determines which neighboring nodes are more important based on the user query. Important neighboring nodes related to the user query are included in the domain structure information of each node, thereby generating a graph embedding vector including its domain structure information for each node in the context subgraph.

[0054] Specifically, in order to generate context-aware node embeddings using GNN, the compressed context subgraph (containing nodes such as colleges, majors, and regions) and the user's original query are fed into the processing flow together, and then GNN starts working.

[0055] Here, the context subgraph used as input is a graph structure consisting of dozens of nodes and edges connecting them. Accordingly, a user query embedding is generated, encoding the user's query language into a vector (e.g., "Guangdong examinee, 640 points, wants to study computer science").

[0056] In the information passing process of a Graph Neural Network (GNN), such as the Graph Attention Network (GAT), multiple rounds of "information passing" occur. In each round, each node in the graph (e.g., "Sun Yat-sen University") not only sees its own initial information (ranking, introduction, etc.) but also aggregates information from its neighboring nodes (e.g., "Guangzhou" and "Computer Science"). During aggregation, the GNN utilizes an attention mechanism, combined with the user query embedding, to determine which neighboring information is more important. For example, because the user query contains "computer science," "Sun Yat-sen University" will give higher weight to its neighboring node "computer science" when aggregating information. This process is based on "contextual awareness," because the final generated embedding vector for the same node "Sun Yat-sen University" will be different for students who want to study computer science and those who want to study medicine; that is, it can adaptively obtain different results based on the user's personalized needs.

[0057] Through the above processing, a final embedding vector "V_node" containing neighborhood structure information and highly relevant to the current user query can be generated for each node in the context subgraph (such as "Sun Yat-sen University", "South China University of Technology", "Computer Science", "Artificial Intelligence", etc.).

[0058] Next, in step S520, the text summary of the context subgraph and multimodal data is converted into a structured text input into the large language model. The large language model performs word segmentation on the text and aligns the graph embedding vectors of the nodes in the context subgraph with the word segmentation results of the large language model to obtain a mapping table on whether there is a mapping relationship between the word segmentation results and the nodes of the context subgraph.

[0059] Through the above processing, the node embeddings of the context subgraph can be aligned with the lexical space of the LLM. Specifically, the LLM processes one-dimensional text sequences (tokens), while the GNN outputs vectors of graph nodes. To establish a connection between the two, context serialization is first performed, that is, the context subgraph and the multimodal summary are transformed into a structured text input for the LLM.

[0060] As an example, in addition to the above, the text input for LLM can also include the user's query question, such as, "Context: [Entity 1: Sun Yat-sen University, located in: Guangzhou, flagship major: Computer Science, video summary: beautiful campus, advanced equipment.][Entity 2: South China University of Technology, ...] Question: I am a candidate from Guangdong..."

[0061] Next, token localization will be performed. The LLM tokenizer will segment the above text into tokens so that it can accurately determine which tokens in the text correspond to which nodes in the graph.

[0062] For example, the tokens ["Zhongshan", "University"] in the text sequence correspond to the graph node "Zhongshan University". This localization can be achieved using existing string matching or more complex entity linking techniques. Accordingly, the mapping relationship can be established using the following mapping table: { "Zhongshan University": [token_idx_1, token_idx_2], "Computer Science": [token_idx_15, token_idx_16], ...}.

[0063] Next, in step S530, traverse each token in the token segmentation result, and assign an attention bias value to it to obtain a bias matrix. Specifically, for tokens with a mapping relationship, a positive bias value is assigned, and for tokens without a mapping relationship, a value of 0 is assigned. The value of the positive bias value is proportional to the comprehensive relevance score of the corresponding node.

[0064] Generating the attention bias matrix is an important guiding process in the entire learning process of the LLM. To this end, the output of the GNN and the "comprehensive relevance score" calculated in the subgraph compression stage will be used to create a "cheating matrix" to directly intervene in the attention calculation of the LLM.

[0065] Specifically, first obtain the importance score of the node: Previously, in the subgraph compression step, a comprehensive relevance score "S_node" was calculated for each node (for example, "S(Zhongshan University) = 0.95", "S(South China University of Technology) = 0.92"). This score directly quantifies the importance of each node to the current user's needs.

[0066] Next, calculate the bias value: Traverse each token in the LLM input sequence. If a certain token "token_j" belongs to a certain node in our mapping table, that is, there is a mapping relationship (for example, "university" belongs to "Zhongshan University"), a positive bias value is assigned to it. This value is usually proportional to the importance score "S_node" of the node. For example, "Bias_j = α S_node" (where α is a hyperparameter that can be preset to control the guiding strength). On the contrary, if a token does not belong to any node in the mapping table (for example, the conjunction "is located" in the context), its bias value is 0.

[0067] The above bias values constitute the bias matrix "B". Specifically, in the Transformer self-attention mechanism of the large language model, the model will calculate an attention score matrix "A", where "A_ij" represents the attention degree of the "i"-th token to the "j"-th token. The bias matrix "B" is constructed to have the same scale as "A", and each row of it includes the bias values of all tokens calculated above "[Bias_1, Bias_2,..., Bias_j,...]". This means that no matter which token is "looking" at other tokens currently, those tokens related to important nodes (such as "Zhongshan" and "university") will get an extra "bonus"

[0068] In step S540, in the self-attention mechanism of each Transforer layer of the large language model, the bias matrix is ​​added to the original attention score of the softmax function, so that the generated college entrance examination application recommendations pay more attention to entities with higher comprehensive relevance scores in the context subgraph.

[0069] Specifically, applying a bias matrix in LLM attention calculation is equivalent to highlighting user intent as a key element in the LLM reading context. Within the self-attention module of each Transformer layer in the LLM, the original calculation process is as follows:

[0070] AttentionWeights = Softmax( (Q K^T) / sqrt(d_k) )V

[0071] Where AttentionWeights represents the attention weights, Q, K, and V are the query, key, and value matrices, respectively, K^T is the transpose of K, and Q... K^T calculates the similarity between Q and all K, where d_k represents the dimension of the K vector.

[0072] According to an exemplary embodiment of the present invention, the above process is modified by adding the bias matrix "B" to the original attention score before the Softmax function, i.e.:

[0073] ModifiedScores = (Q K^T) / sqrt(d_k) + B

[0074] AttentionWeights = Softmax(ModifiedScores)V

[0075] Because the Softmax function is an exponential function, it greatly amplifies the differences between input values. A small positive bias, after Softmax, will cause the corresponding lexical unit to receive a disproportionately high attention weight. For example, the original weights might be [0.1, 0.1, 0.1, ...], but after adding the bias and Softmax, the weights might become [0.5, 0.3, 0.01, ...]. Accordingly, when LLM generates sentences about "recommended universities," its internal state will be highly focused on intent-matching lexical units, such as information related to the concepts of "Sun Yat-sen University" and "computer science," while automatically ignoring or weakening information about entities with lower scores.

[0076] Through the above processing, the GNN's understanding of graph structure and user intent (reflected in node embeddings and importance scores) is transformed into precise, quantitative, and adaptive guidance of the attention flow within the LLM. In other words, by applying a positive bias to the lexical units corresponding to high-scoring nodes, the LLM is forced to prioritize and heavily rely on these deemed "most crucial" recommendation criteria when generating answers, thereby significantly improving the accuracy, relevance, and logical consistency of the recommendations.

[0077] Here, different user queries generate different subgraphs and different node importance scores, resulting in completely different attention bias matrices. This makes the entire bootstrapping process highly dynamic and personalized.

[0078] As can be seen, the college entrance examination application consultation system of the exemplary embodiment of the present invention can fully reflect the personalized needs of users. In addition to users expressing their complete wishes in a single query, the professional large language model can also collect the wishes mentioned by users from time to time in multiple rounds of dialogue. For example, it can guide users to express their specific wishes regarding college application by proactively asking them some basic questions. That is, the large language model can guide users to directly express their wishes regarding college entrance examination applications in natural language queries during the dialogue process.

[0079] Furthermore, as a crucial life event, filling out college applications inevitably brings pressure and anxiety for students and parents when faced with numerous choices. A better, more comprehensive understanding of users' true intentions, and even uncovering their unspoken, latent needs, would undoubtedly provide greater assistance to students and parents in making this significant life decision.

[0080] Therefore, exemplary embodiments of the present invention attempt to more comprehensively explore users' potential intentions. For example, the college entrance examination application consultation system according to exemplary embodiments of the present invention can be implemented as an application app installed on a terminal product, or it can be implemented as a website on a PC. In this case, numerous articles or news related to college entrance examination applications can be provided within the app or website, such as detailed introductions to various universities and majors, and prospect analyses of various industries. By collecting users' browsing behavior on related content, such as browsing topics (especially keywords), number of browsing sessions (e.g., whether there are repeated browsing sessions), and browsing duration, users' potential intentions can be analyzed, such as their interest in specific majors or specific universities.

[0081] Specifically, it's also possible to collect user terminal usage behaviors related to college application within a specific time period, including at least one of the following: text and image content browsing behavior, audio and video content playback behavior, in-app search behavior, web search behavior, voice call behavior, and online chat behavior. The keywords for content and searches can be obtained through attribute information. Furthermore, NLP techniques can be used to extract college application-related keywords mentioned in calls or chats. Correspondingly, user intentions can be analyzed based on terminal usage behavior. Here, user behavior can be obtained by reading backend data. For example, user terminal usage behavior within a specific time period (e.g., a few days after creating a college application session and before submission) can be recorded as logs. These logs can include content describing the behavior and characteristic data of the behavior itself, such as the topic of the content, keywords, frequency of occurrence, duration, etc. After obtaining various characteristic data, simple rules can be used to determine user interests. For example, the most frequently viewed topics can be considered as the user's primary intentions, or various indicators such as browsing frequency, duration, and repetition can be statistically weighted to obtain ratings for different topics, with the topic with the highest rating considered as the user's primary intention. Furthermore, machine learning models can be pre-trained to comprehensively predict user intentions based on various user behaviors, thereby enabling more accurate judgments of user interests. Thus, by analyzing this feature data, information related to user intentions can be obtained, guiding specialized large-scale language models to communicate with users on relevant issues during dialogues to further confirm user intentions. This extends the consideration of user intentions when answering queries to include indirect expressions from the user. For example, if a user's reading logs during a session show that the keywords of the articles read all involve the "archaeology" profession, then this is considered the user's intended profession and a bonus is awarded.

[0082] Furthermore, in our practical experience providing offline college application consultations to users, we have noticed that users often show unusual excitement about information regarding certain universities or majors. This excitement often involves novel information that users haven't normally paid attention to. However, this excitement may only last for a moment, and users, for various reasons, may not have the courage to immediately explore this new information further. This can lead to many valuable opportunities being missed. Therefore, exemplary embodiments of this invention aim to focus on how to help users perceive their deeper needs that they are not yet consciously aware of.

[0083] Therefore, according to exemplary embodiments of the present invention, it is possible to monitor the user's biometrics during a conversation using technologies such as cameras and sensors, thereby capturing instantaneous changes in the user's emotions and determining key factors that the user may be interested in based on the preceding and following conversations. For example, if the topic of "archaeology" is mentioned in the conversation and the camera detects that the user's eyes, mouth, or other facial expressions indicate excitement or interest, the conversation can be guided to further explore the user's potential interests. It should be understood that the collection of biometric information is not limited to the user's micro-expressions; it can also be further assessed by detecting features such as blood pressure and heart rate.

[0084] Specifically, biometric information of users during conversations can be collected, including at least one of micro-expressions, blood pressure, and heart rate. Correspondingly, user intentions can be analyzed based on this biometric information. For example, facial images of users during conversations can be continuously monitored (optionally, infrared illumination can be used during monitoring to eliminate ambient light interference). Existing image processing techniques can be utilized, such as detecting faces in videos and then locating and tracking key points (e.g., eyeballs, corners of eyes, corners of mouth, eyebrows, eyebrow tails, eyebrow center points) from the facial images. Based on ROI (Region of Interest) segmentation, sudden emotional states of the user (e.g., sudden excitement micro-expressions, blood pressure, heart rate, etc.) can be captured by analyzing positional changes of these feature points such as eyeballs, corners of mouth, or eyebrows. Specifically, Euler video magnification (EVM) technology can be used to non-linearly magnify micro-expression movements by 20-50 times, and pyramid decomposition can be used to process motion signals of different spatial frequencies. As an example, an adaptive motion magnification algorithm can be introduced to dynamically adjust the magnification coefficient according to the characteristics of facial muscle movement, further improving the signal-to-noise ratio of facial expressions. After extracting various features related to micro-expressions, such as: spatiotemporal features (horizontal / vertical motion components can be extracted using optical flow algorithms to construct optical flow direction histograms); depth features (a modified P3D ResNet network is used to extract spatiotemporal features); and physiological features (heart rate variability (HRV) indicators can be selectively extracted and fused with micro-expression motion features for multimodal analysis), an emotion recognition model based on these features is constructed. The activation intensity of facial action units (AUs) is transformed into spatial coordinates of the emotion dimension, and recognition is performed between the categories of excitement and indifference.

[0085] By correlating the time of video images with the display time of the preceding and following dialogues, it is possible to find dialogue information corresponding to the occurrence of excited expressions, such as entity information mentioned in the dialogue (e.g., profession) or its value (e.g., archaeology). Accordingly, the professional language model communicates with the user on related issues in the dialogue (e.g., proactively outputting prompts such as "Are you interested in learning more about the details of the archaeology profession?"), thus further extending the user's intentions considered when answering queries to the user's physiological expressions in addition to language expression.

[0086] The above two methods can be used individually or in combination; that is, user terminal usage behavior and biometric information can be used separately or in combination to explore user potential intentions. Furthermore, in addition to passively monitoring user terminal usage behavior and / or emotional responses, according to an exemplary embodiment of the present invention, user potential intentions can also be actively explored. For this purpose, probe content for exploring user potential intentions regarding college application choices can be generated, the probe content can be output to the user, and user feedback information can be collected. Accordingly, user intentions can also be extracted based on the user's feedback information regarding the probe content.

[0087] Specifically, since the interaction with users in the college entrance examination application consultation method of this invention mainly relies on a large language model, it is considered to flexibly insert some inquiries about intentions during the dialogue. These inquiries can be pre-generated fixed questions, such as tests about the user's personality, or they can be generated during the dialogue, such as seemingly random questions, to help the system capture the user's intentions. Here, we utilize the generation capabilities of the large language model, guided by the goal of mining the user's potential intentions, to fill in the prompt words with the user's information as auxiliary information, triggering the large model to generate a set of questions in real time to explore the user's potential intentions. The large language model module can automatically generate some short Q&A questions about the user and / or their college entrance examination application by combining user information, the current dialogue state, and / or contextual information during the dialogue with the user. For example, psychological test questions about the user's personality traits, short Q&A questions about the user's mentality when filling out the application, short Q&A questions about candidate universities and / or majors, etc. In addition to this question-and-answer "probe-feedback" mechanism, the user's potential intentions can also be judged by observing the user's feedback on specific content.

[0088] Besides positively guiding users to explore their inner world, according to an exemplary embodiment of the present invention, interference can be created to temporarily induce a state of mental confusion in users. In this state, some inherent cognitions are easily dismantled, allowing users to lower their guard and find their true desires. This aligns well with people's psychological states in daily life. For example, hypnosis techniques in psychology mention that rapid visual stimulation can cause temporary mental confusion, making it difficult to react and think quickly, thus making people more susceptible to suggestion. Therefore, we can utilize this phenomenon to generate distracting content that suddenly disrupts the user's current conversational state (this distracting content may include candidate preferences consistent with the mined user's potential desires after a brief visual stimulus), output the distracting content to the user, and collect user feedback on the distracting content. Furthermore, the user's desires are extracted based on the user's feedback on the distracting content. Specifically, if user intentions that are completely opposite to those directly expressed by the user are further extracted based on terminal usage behavior, biometric information, and / or user feedback on the probed content, then interfering content that is completely opposite to the user's direct intentions is generated, the interfering content is output to the user, and feedback information on whether the user holds a positive or negative attitude towards the interfering content is collected. Then, the user intentions are updated based on the user's feedback information on the interfering content.

[0089] For example, suppose a parent of a student keeps asking about universities in Beijing during the conversation, causing the system to provide several candidate universities based on the "Beijing" region. However, subsequent monitoring reveals that this parent has significantly more views on articles or watched more videos about universities in Shanghai than those in Beijing. Therefore, the system initially judges that the parent may subconsciously have a strong interest in universities in Shanghai. To avoid subconscious resistance and denial from the parent if asked directly, a visual stimulus that confuses the user during the current conversation can be suddenly displayed (e.g., a brief screen flash, black screen, or shaking effect, lasting no more than 2 seconds). After that, the normal dialog box is restored, and the system begins to ask the user about their attitude towards universities in Shanghai in a more focused manner, thereby exploring whether the user's true preference is for Beijing or Shanghai.

[0090] Furthermore, according to an exemplary embodiment of the present invention, the parameters of the graph neural network and the parameters of the large language model can be co-optimized. For example, the retrieval / understanding part in the GNN and the generation part in the LLM can co-evolve. To this end, an exemplary embodiment of the present invention establishes a unified optimization objective, allowing the seemingly independent models GNN and LLM to supervise each other and optimize together in a closed loop.

[0091] Specifically, the Large Language Model (LLM) and Graph Neural Network (GNN) can be jointly trained using a joint loss function, L_joint. This joint loss function includes the L_lm loss component, which evaluates the quality of the generated text, and the contrastive learning loss component, L_contrastive, which brings closer entities matching the user's preferences and pushes away entities deviating from them in the embedding vector space. Here, L_lm is responsible for evaluating the quality of the generated text. If the LLM-generated recommendation report is of poor quality (e.g., recommending obviously unsuitable schools), it indicates that the contextual subgraph (recommendation basis) provided by the GNN itself may be problematic. This evaluation completes the supervision of the GNN by the LLM. L_contrastive, on the other hand, is responsible for the graph embedding space of the GNN. The indirect supervision of the LLM by the GNN is reflected in the GNN providing a structured and focused context, helping the LLM reason better. A good GNN output makes it easier for the LLM to generate high-quality reports.

[0092] Accordingly, the joint loss function can be expressed as:

[0093] L_joint = λ L_lm + (1 - λ) L_contrastive, where λ is a hyperparameter used to balance the importance of the two loss components. Its value varies between 0 and 1. In the early stage of training, the value of λ is set to be less than 0.5 to focus more on optimizing the embedding quality of the graph neural network; in the later stage of training, the value of λ is set to be greater than 0.5 to focus more on improving the text generation capability of the large language model.

[0094] L_lm is the relatively straightforward part of the joint loss, and its goal is to make the recommendation report generated by LLM as close as possible to an expert answer that serves as a "standard answer" or "golden template".

[0095] The type of L_lm can use standard cross-entropy loss, or the standard loss function used during training by other autoregressive language models (such as the GPT series). Accordingly, to compute L_lm, the following data is required:

[0096] The input C of the large language model includes the user's natural language query, structured information from the context subgraph, and text summaries of multimodal data;

[0097] The output P of the large language model represents the recommendation results generated by the large language model word-by-word based on the input C;

[0098] Expert Answer: A high-quality, ideal recommendation provided by a human expert for the same user query. Y = {y_1, y_2, ..., y_T}, where y_t is the t-th term in the report.

[0099] The loss function can be represented as the probability distribution of the next word y_t predicted by the large language model at each time step t, which reflects the difference between the prediction and the actual word y_t.

[0100] L_lm = - Σ{t=1 to T} log P(y_t | y_1,y_2,...,y_[t-1], C; θ_llm)

[0101] P(y_t | y_1,y_2,...,y_[t-1],C; θ_llm) represents the probability that LLM predicts the next word as y_t, given the context and the previously generated words (y_1,y_2,...,y_[t-1]).

[0102] θ_llm are the model parameters of LLM; as an example, they can be the weights of LoRA.

[0103] log P(...) represents the logarithmic probability. If the model predicts a high probability of the correct answer, the value of log P is close to 0, resulting in a small loss; conversely, the loss is large.

[0104] Σ (sigma) represents the sum of the losses of all terms in the recommendation results.

[0105] The gradient of the calculated loss L_lm will be backpropagated to update the parameters θ_llm of the large language model LLM.

[0106] The goal of L_contrastive is to achieve "grouping by type" in the graph embedding space. That is, the embedding vectors of "good" entities that match the user query, i.e., the anchor, i.e., positive samples, should be brought closer; while the embedding vectors of "bad" entities that do not match, i.e., negative samples, should be pushed away.

[0107] L_contrastive can use a triplet loss type, which involves constructing triples (anchor, positive, negative) for learning.

[0108] For each training sample (i.e., for a user query): the user's natural language query is transformed into a query embedding vector "E_query" using a text encoder (such as BERT or LLM's own encoder). Furthermore, one or more "good" entities considered correct answers are extracted from the expert's answer "Y," and their embedding vectors "E_positive" are obtained from the GNN's output. For example, if the expert's volunteer report strongly recommends "Sun Yat-sen University" and "Computer Science," then these two entities are good entities. Conversely, one or more "bad" entities considered incorrect answers can be treated as negative samples, with their corresponding embedding vectors being "E_negative."

[0109] Regarding the selection of negative samples, as an example, some entities can be randomly selected from the entire knowledge graph, as long as they do not appear in the expert answer "Y". To select suitable negative samples more effectively, various mining methods can be used to specifically select entities that the model easily confuses as negative samples. For example, another university with an admission score very close to "Sun Yat-sen University" but a much lower ranking in computer science. This "difficult negative example" forces the GNN to learn more subtle discriminative features.

[0110] For each constructed triple (E_query, E_positive, E_negative), the formula for calculating L_contrastive is as follows:

[0111] L_contrastive= max(0, d(E_query, E_positive) - d(E_query, E_negative)+ margin)

[0112] Where d represents the distance between vectors, which is usually calculated using variants of Euclidean distance or cosine distance.

[0113] d(E_query, E_positive) is the distance between the query and the positive sample, and the goal is to minimize this distance.

[0114] d(E_query, E_negative) is the distance between the query and the negative sample, and the goal is to maximize this distance.

[0115] The margin is a preset hyperparameter (e.g., 0.2), which acts as a "safety boundary." The above formula requires that d(E_query, E_negative) not only be greater than d(E_query, E_positive), but also be at least larger than the margin. If the conditions are met, the result of max(0, ...) is 0, indicating that the loss of this triple is 0, and the model has learned well.

[0116] L_contrastive is ultimately the average of the losses of all triples throughout the entire training batch.

[0117] The gradient of the loss L_contrastive will be backpropagated to update the parameters θ_gnn of the graph neural network GNN and the parameters of the user query encoder.

[0118] In each training iteration: For a received user query, the GNN generates structured information for the context subgraph, and the LLM generates recommendation results. The LLM output is compared with the expert answer to obtain the language model loss. Triples are constructed from the user query, entities (positive samples) in the expert answer, and random negative samples or mined hard negative examples, and the contrastive loss is calculated. The two are weighted and summed to calculate L_joint. After the total gradient of L_joint is calculated, backpropagation is performed, and then, according to the computation graph, different parts of the gradient automatically and simultaneously flow to and update θ_llm and θ_gnn.

[0119] The above process forms a feedback loop to further optimize the recommendation results: if the LLM is poorly generated, L_lm will increase, prompting the LLM to improve; at the same time, the expert answer Y of the LLM defines the "bullseye" of the GNN's learning, and L_contrastive will force the embedding vectors produced by the GNN to better reflect the user's true preferences, thereby providing the LLM with higher quality context, and ultimately helping the LLM generate better recommendation results. For example, "Hello! Based on your situation, we recommend: Sun Yat-sen University, whose admission score is highly matched with your estimated score (640). Its computer science major is A+ level, located in the first-tier city of Guangzhou, with broad development prospects. (Video summary can be inserted here) From the campus introduction, its scientific research equipment is advanced and the learning atmosphere is strong..."

[0120] As can be seen, in the specific and complex scenario of college application, this invention provides a solution that can process multimodal information, effectively deal with large-scale related data, and perform deep personalized reasoning.

[0121] Figure 6 shows a block diagram of a system for college entrance examination volunteer consultation based on a large language model according to an exemplary embodiment of the present invention. The system includes: a knowledge graph construction unit 100, used to construct a multimodal knowledge graph, wherein nodes include three types of entities: colleges, majors, and cities / regions; edges represent the relationships between them; and some entity nodes are linked to multimodal data sources including video and audio; a query processing unit 200, used to receive natural language queries from users regarding college entrance examination volunteer recommendations and extract key entity information about the user's preferences, wherein the user's preferences include at least scores, preferred majors, intended cities, and subject combinations; and a context subgraph generation unit 300, used to adaptively generate a context subgraph based on the extracted key entity information using a nearest neighbor sampling algorithm. A scalable context subgraph is generated from the multimodal knowledge graph; a multimodal transformation unit 400 processes the multimodal data from the multimodal data sources linked to the nodes in the context subgraph to convert the multimodal data into a compact text summary; and a recommendation generation unit 500 uses a large language model to generate personalized college application recommendations based on the user's natural language query, combined with the structured information of the context subgraph and the text summary of the multimodal data. In this process, a graph neural network is used to generate embedding vectors for the nodes of the context subgraph, and the embedding vectors are converted into attention bias matrices to adjust the attention weights of the large language model.

[0122] The various units or modules (shown or not shown) included in the college entrance examination application consultation system shown in Figure 6 can be configured as software, hardware, firmware, or any combination thereof to perform specific functions. For example, these components may correspond to dedicated integrated circuits, pure software code, or units or modules combining software and hardware. Furthermore, one or more functions implemented by these components can also be uniformly executed by components in a physical entity device (e.g., a processor, client, or server). The college entrance examination application consultation system and method according to exemplary embodiments of the present invention have been described above with reference to Figures 1 to 6. It should be understood that the above method can be implemented by a program recorded on a computer-readable medium. On the other hand, the various units shown in Figure 6 can also be implemented by hardware, software, firmware, middleware, microcode, or any combination thereof. When implemented as software, firmware, middleware, or microcode, the program code or code segment used to perform the corresponding operation can be stored in a computer-readable medium such as a storage medium, so that a processor can perform the corresponding operation by reading and running the corresponding program code or code segment.

[0123] The foregoing has described various exemplary embodiments of the present invention. It should be understood that the above description is merely exemplary and not exhaustive, and the present invention is not limited to the disclosed exemplary embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for providing college application guidance based on a large language model, comprising: A multimodal knowledge graph is constructed, where nodes include three types of entities: colleges, majors, and cities / regions. Edges represent the relationships between them, and some entity nodes are linked to multimodal data sources, including video and audio. The system receives natural language queries from users regarding college application recommendations and extracts key entity information about the user's preferences, which include at least scores, preferred majors, intended cities, and subject combinations. Based on the extracted key entity information, a controllable-size context subgraph is adaptively generated from the multimodal knowledge graph using a nearest neighbor sampling algorithm. The multimodal data from the multimodal data sources linked to the nodes in the context subgraph is processed to optimize the multimodal data. The process involves converting text into a compact summary; and using a large language model to generate personalized college application recommendations based on users' natural language queries, combined with structured information from context subgraphs and multimodal data. Specifically, a graph neural network is used to generate embedding vectors for nodes in the context subgraph, and these embedding vectors are then transformed into attention bias matrices to adjust the attention weights of the large language model. This process includes: using a graph neural network to perform multimodal analysis on each node in the context subgraph and its neighboring nodes based on the graph structure of the context subgraph. In the aggregation of information transmission, the graph neural network combines key entity information encoded as vectors to adaptively determine which neighboring nodes are more important based on the user query. It also includes important neighboring nodes related to the user query in the domain structure information of each node, thus generating a graph embedding vector containing its domain structure information for each node in the context subgraph. The text summary of the context subgraph and multimodal data is converted into a structured text input into a large language model. The large language model performs word segmentation on this structured text and aligns the graph embedding vectors of the nodes in the context subgraph with the word segmentation results of the large language model. A mapping table is obtained to determine whether there is a mapping relationship between the word segmentation results and the nodes of the context subgraph. Each word in the word segmentation results is traversed, and an attention bias value is assigned to it to obtain a bias matrix. A positive bias value is assigned to words with a mapping relationship, and a value of 0 is assigned to words without a mapping relationship. The value of the positive bias value is proportional to the comprehensive relevance score of the corresponding node. In the self-attention mechanism of each Transforer layer of the large language model, the bias matrix is ​​added to the original attention score of the softmax function, so that the generated college entrance examination application recommendations pay more attention to entities with higher comprehensive relevance scores in the context subgraph.

2. The method as described in claim 1, wherein, The steps for generating a context subgraph include: initially selecting a batch of candidate entity nodes as seed nodes in the knowledge graph based on key entity information; calculating a comprehensive relevance score for all seed nodes and their one-hop neighbor nodes based on their matching degree with the user's intention; selecting a first predetermined number of nodes from high to low based on the comprehensive relevance score, and combining these nodes with a second predetermined number of nearest neighbor nodes in the graph embedding space to jointly form the finally generated context subgraph.

3. The method as described in claim 1, wherein, Converting multimodal data into compact text summaries involves: for video data, locating key frames using keyframe extraction algorithms and generating descriptions for key frames using an image-to-text model, combining them into a video summary; for audio data, converting audio data into text using speech recognition technology and generating an audio summary of the core content using a text summarization model.

4. The method of claim 1, wherein, The large language model and the graph neural network are jointly trained using a joint loss function L_joint. This joint loss function includes a large language model loss component L_lm that evaluates the quality of college application recommendations, and a contrastive learning loss component L_contrastive that brings entities that match the user's preferences closer together and pushes away entities that deviate from them in the embedding vector space. The joint loss function is expressed as: L_joint = λ L_lm + (1 - λ) L_contrastive, where λ is a hyperparameter used to balance the importance of the two loss components. Its value varies between 0 and 1. In the early stage of training, the value of λ is set to be less than 0.5 to focus more on optimizing the embedding quality of the graph neural network; in the later stage of training, the value of λ is set to be greater than 0.5 to focus more on improving the text generation capability of the large language model.

5. The method of claim 1, wherein, User intent is obtained through the following processes: A large language model guides users to directly express their intent regarding college application choices in natural language queries during the dialogue; and / or collects user terminal usage behaviors related to college application choices within a specific time period, including at least one of the following: text and image content browsing behavior, audio and video content playback behavior, in-app search behavior, web search behavior, voice call behavior, and online chat behavior; collects user biometric information during the dialogue, including at least one of micro-expressions, blood pressure, and heart rate; analyzes user intent based on terminal usage behavior and / or biometric information; and / or generates probe content to uncover potential user intent regarding college application choices, outputs the probe content to the user, and collects user feedback on the probe content. The feedback information of the content is extracted based on the user's feedback on the probed content. The probed content includes at least one of the following: short Q&A questions about the user and / or their college application, immersive experience videos about candidate colleges, and introductions to candidate colleges with emotional background music. The short Q&A questions about the user and / or their college application are automatically generated by the large language model module during the dialogue with the user, combining user information, current dialogue state, and / or contextual information. And / or, distracting content is generated to suddenly disrupt the user's current dialogue state. The distracting content is output to the user, and the user's feedback on the distracting content is collected. The user's intention is extracted based on the user's feedback on the distracting content.

6. The method of claim 5, wherein, If user intentions that are completely opposite to those directly expressed by the user are further extracted based on terminal usage behavior, biometric information, and / or user feedback on the probed content, then interfering content that is completely opposite to those directly expressed by the user is generated, the interfering content is output to the user, and feedback information on whether the user holds a positive or negative attitude towards the interfering content is collected. Then, the user intentions are updated based on the user's feedback information on the interfering content.

7. A system for providing college application guidance based on a large language model, used to execute the method for providing college application guidance based on a large language model as described in claim 1, the system comprising: The system includes the following components: a knowledge graph construction unit for building a multimodal knowledge graph, where nodes include three types of entities: colleges, majors, and cities / regions, edges represent the relationships between them, and some entity nodes are linked to multimodal data sources including video and audio; a query processing unit for receiving natural language queries from users regarding college application recommendations and extracting key entity information about the user's preferences, which include at least scores, preferred majors, intended cities, and subject combinations; a context subgraph generation unit for adaptively generating a controllable-size context subgraph from the multimodal knowledge graph based on the extracted key entity information using a nearest neighbor sampling algorithm; a multimodal transformation unit for processing the multimodal data from the multimodal data sources linked to the nodes in the context subgraph to convert the multimodal data into a compact text summary; and a recommendation generation unit for generating personalized college application recommendations based on the user's natural language query, combined with the structured information of the context subgraph and the text summary of the multimodal data using a large language model. This involves using a graph neural network to generate embedding vectors for the nodes in the context subgraph and converting these embedding vectors into attention bias matrices to adjust the attention weights of the large language model.

8. The system of claim 7, wherein, The multimodal conversion unit converts multimodal data into compact text summaries, including: for video data, locating core frames using a keyframe extraction algorithm and generating descriptions for the key frames using an image-to-text model, combining them into a video summary; for audio data, converting audio data into text using speech recognition technology and generating an audio summary of the core content using a text summarization model.

9. A computer-readable recording medium having a computer program stored thereon for performing the method of claim 1.

Images