Health science popularization resource recommendation method, system and device and storage medium

Abstract

The invention discloses a health science popularization resource recommendation method, system and device and a storage medium, and the method comprises the steps: constructing a multi-modal knowledge graph based on data sets of multi-modal health science popularization resources, user information, a medical field professional dictionary and the like; in combination with a multi-modal knowledge graph, information spreading and aggregation are carried out on the knowledge graph by using a graph attention model GAT, and a structured feature representation vector of each node is obtained; obtaining a deep semantic feature representation vector of each node by using a multi-modal large model oneLLM; the structured features and the semantic features are integrated, a health science popularization resource recommendation model is obtained through LoRA + fine tuning training on the basis of a large language model Llama3, and personalized resource recommendation is carried out. According to the method, the structural modeling capability of the knowledge graph and the semantic understanding capability of the large model are combined, so that comprehensive deep extraction of user features and resource contents is realized, and the accuracy and interpretability of a recommendation result are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, system, device, and storage medium for recommending health science popularization resources. Background Technology

[0002] With the deepening of the Healthy China strategy and the booming development of internet healthcare, health education resources for the public have experienced explosive growth, expanding from traditional text and images to various formats such as video, audio, and live streaming. These resources are of great value in improving public health literacy and popularizing scientific and medical knowledge. However, faced with a massive amount of diverse health education content, how ordinary users can efficiently and accurately access resources that match their own needs has become a pressing issue.

[0003] Currently, technical solutions in the field of health science resource recommendation suffer from limitations. Most existing methods still rely on feature matching based on keywords or preset tags, or on rigid rules formulated by experts. These methods struggle to deeply explore the complex semantic relationships and structured features within resources, and are unable to effectively handle unstructured, information-rich multimodal content such as videos and audio. Consequently, recommendation results are often superficial, lacking accuracy and comprehensiveness. Furthermore, in the highly specialized field of medicine, non-professional users, lacking effective technical assistance, typically rely on limited medical knowledge for manual screening. This is inefficient and highly susceptible to content that is inconsistent with their health status and knowledge level, or even misleading. This not only severely impacts the user experience of acquiring health knowledge but can also cause unnecessary stress and anxiety due to information bias, potentially posing risks to users' physical and mental health. Summary of the Invention

[0004] In view of the above problems, this application provides a method, system, device and storage medium for recommending health science popularization resources.

[0005] The embodiments of this application disclose the following technical solutions:

[0006] The first aspect of this application provides a method for recommending health science popularization resources, including:

[0007] Based on the target dataset, a multimodal knowledge graph including multimodal entities and relationships is constructed. The target dataset includes multimodal health science popularization resources, user information, and professional data in the medical field.

[0008] Based on the multimodal knowledge graph, the graph attention network (GAT) is used to propagate and aggregate information on the nodes in the multimodal knowledge graph to obtain the structured feature representation vector of each node.

[0009] The oneLLM multimodal large model is used to encode health science popularization resources and user information of different modalities to obtain semantic feature representation vectors of each node;

[0010] For each node, structured features and semantic features are fused to obtain a fused feature representation; based on the fused feature representation, a training dataset is constructed through vector retrieval; based on the training dataset, an open-source multimodal pre-trained large language model Llama 3 is used in conjunction with the LoRA+ fine-tuning method to train a health science popularization resource recommendation model;

[0011] Based on the aforementioned health science popularization resource recommendation model, a personalized list of health science popularization resources is generated.

[0012] In one possible implementation, constructing a multimodal knowledge graph comprising multimodal entities and relationships based on the target dataset includes:

[0013] Based on the target dataset, entity types, relation types, and attributes are defined to construct the knowledge graph system architecture.

[0014] The pre-trained model MedBERT is used to extract entities, relationships, and attributes from multimodal data.

[0015] Based on a medical terminology, data from different sources, modalities, or with different names but pointing to the same entity and attribute are merged, and similar or identical relationships from different data sources are integrated.

[0016] The extracted and fused structured triple information is stored in a graph database to obtain a multimodal knowledge graph that includes multimodal entities and relationships.

[0017] In one possible implementation, the extraction of entities, relationships, and attributes from multimodal data using the pre-trained model MedBERT includes:

[0018] For image-based data, corresponding descriptive text is generated using the generative large language model Llama 3;

[0019] For audio data, the Whisper speech recognition model is used to convert it into corresponding descriptive text.

[0020] For video data, the Whisper speech recognition model is used to extract the descriptive text corresponding to the speech, and the Llama 3 generative large language model is used to generate the descriptive text corresponding to the video frames. The final descriptive text of the video is obtained by combining the two descriptive texts.

[0021] Based on the descriptive text converted from the images, audio, and video, as well as the original text data, the MedBERT pre-trained model in the medical field is used to extract entity, relation, and attribute information.

[0022] In one possible implementation, the oneLLM multimodal large model is used to encode health science resources and user information of different modalities to obtain semantic feature representation vectors for each node, including:

[0023] Different modal tokenizers are used to convert text, images, videos, and audio into token sequences respectively;

[0024] The token sequences of each modality are encoded using a unified encoder to obtain the embedding vector of each modality;

[0025] The embedding vectors of each modality are mapped to the same semantic vector space by the projection module, thus obtaining the semantic feature representation vector of each node.

[0026] In one possible implementation, the fusion of structured features and semantic features to obtain the fused feature representation includes:

[0027] Structured feature vectors and semantic feature vectors are mapped to a unified interactive vector space through projection networks;

[0028] The two types of feature vectors after mapping are concatenated to obtain the fused feature representation of each node.

[0029] In one possible implementation, the step of constructing the training dataset based on the fused feature representation via vector retrieval includes:

[0030] Based on the fusion feature vectors of each user, similarity matching retrieval is performed in the space formed by the fusion feature vectors of all health science popularization resources. A preset number of resources are selected according to the similarity ranking results to obtain a candidate resource list.

[0031] The resources in the candidate resource list are filtered, and resources that match the user's needs, health status and interests are retained as positive examples.

[0032] From the remaining health science resources outside the candidate resource list, a preset number of resources are randomly selected as negative sample samples.

[0033] The training dataset is composed of the positive and negative examples. The training dataset contains the correspondence between user information and health science resources, as well as the classification labeling results used to characterize whether a resource is recommended.

[0034] In one possible implementation, the health science popularization resource recommendation model is trained based on the training dataset using the open-source multimodal pre-trained large language model Llama 3 combined with the LoRA+ fine-tuning method, including:

[0035] Based on the health science popularization resource recommendation task, a corresponding prompt template is generated. The prompt template is used to construct an input format that can be understood by a large language model by fusing feature representation and classification labeling results.

[0036] The training dataset is input into the open-source multimodal pre-trained large language model Llama 3, and the model is trained using the LoRA+ fine-tuning method, which adjusts some of the model's weight matrices by introducing low-rank adaptation increments.

[0037] The model parameters are optimized using the gradient descent algorithm until the model reaches the preset performance index on the recommendation task, thus obtaining the trained health science popularization resource recommendation model.

[0038] This application provides a health science popularization resource recommendation system in its second embodiment, including:

[0039] The construction unit is used to construct a multimodal knowledge graph including multimodal entities and relationships based on a target dataset, wherein the target dataset includes multimodal health science popularization resources, user information, and professional data in the medical field;

[0040] The aggregation unit is used to perform information propagation and aggregation on the nodes in the multimodal knowledge graph using the graph attention network (GAT) to obtain the structured feature representation vector of each node.

[0041] The encoding unit is used to encode health science resources and user information of different modalities using the multimodal large model oneLLM to obtain the semantic feature representation vector of each node;

[0042] The training unit is used to fuse structured features and semantic features for each node to obtain a fused feature representation; based on the fused feature representation, a training dataset is constructed through vector retrieval; based on the training dataset, the open-source multimodal pre-trained large language model Llama 3 is used in conjunction with the LoRA+ fine-tuning method to train a health science popularization resource recommendation model.

[0043] The generation unit is used to generate a personalized list of recommended health science resources based on the health science popularization resource recommendation model.

[0044] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the health science popularization resource recommendation method as described in the first aspect above.

[0045] The fourth aspect of this application provides a computer program product, which, when run on a computer, executes the health science popularization resource recommendation method as described in the first aspect above.

[0046] A fifth aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform the health science popularization resource recommendation method as described in the first aspect above.

[0047] Compared with the prior art, this application has the following beneficial effects:

[0048] By constructing a multimodal knowledge graph to systematically organize the complex relationships between users, resources, and medical expertise, the problem of insufficient utilization of resource structure information by traditional tag matching methods is solved. Furthermore, structured features are extracted through the graph attention network (GAT), and semantic features are extracted by combining the multimodal large-scale model oneLLM. This enables deep information mining and unified semantic representation of unstructured resources such as videos, audios, and images, overcoming the limitations of existing technologies in handling multimodal content. On this basis, by fusing the two types of features and training a recommendation model based on the large language model Llama 3, accurate and deep matching of user needs and resource content is achieved, ultimately generating a high-quality personalized recommendation list. This solves the technical problems of low accuracy, poor modal adaptability, and insufficient personalization in health science popularization resource recommendations. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart illustrating a method for recommending health science popularization resources provided in this application embodiment;

[0051] Figure 2 A schematic diagram illustrating the construction process of the multimodal knowledge graph provided in this application embodiment;

[0052] Figure 3This is a schematic diagram of the pattern layer construction provided in an embodiment of this application;

[0053] Figure 4 This is a schematic diagram of the structured feature extraction process provided in an embodiment of this application;

[0054] Figure 5 This is a schematic diagram of the semantic feature extraction process provided in an embodiment of this application;

[0055] Figure 6 This is a schematic diagram illustrating the construction process of the health science popularization resource recommendation model provided in this application embodiment;

[0056] Figure 7 This is a structural diagram of a health science popularization resource recommendation system provided in an embodiment of this application. Detailed Implementation

[0057] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0058] To facilitate understanding of the technical solutions provided in the embodiments of this application, the background technology involved in the embodiments of this application will be described below.

[0059] As mentioned earlier, most health science resource recommendation methods rely on tag matching and manual rules, making it difficult to deeply explore the structural features and deep semantics of the resources. Furthermore, their ability to process multimodal content such as videos, audio, and images is limited, resulting in inaccurate and incomplete recommendations. When faced with a vast amount of resources, ordinary users often have to rely on their limited medical knowledge to manually filter them, which is inefficient and makes them susceptible to encountering mismatched or even misleading content, potentially causing unnecessary anxiety or even health risks.

[0060] To address the aforementioned issues, this application integrates multimodal health science resources, user information, and professional data from the medical field, and constructs a structured multimodal knowledge graph. This not only surpasses traditional text tagging but also incorporates unstructured content such as videos, audio, and images into a unified knowledge system, fundamentally solving the problems of poor adaptability to multimodal resources and the inability to effectively model complex structural relationships between resources in existing methods. Secondly, by introducing a dual-path feature extraction mechanism of Graph Attention Network (GAT) and OneLLM (oneLarge Model), users and resources are deeply represented from both structured association and deep semantic dimensions. This not only fully mines the entity relationships and network topology information contained in the knowledge graph but also utilizes the powerful encoding capabilities of the large model to extract a unified semantic representation of cross-modal content, thereby achieving a comprehensive and accurate characterization of user interests and resource content, overcoming the limitations of traditional methods that rely on shallow feature matching. Furthermore, this application utilizes feature fusion and a large language model based on Llama... Through fine-tuning training of 3, an intelligent recommendation model capable of integrating structural and semantic information was constructed. This model not only achieves more accurate personalized matching but also possesses a certain degree of interpretability and reasoning ability, dynamically adapting to user preferences and resource updates. Ultimately, the recommendation list generated based on this model, while ensuring high accuracy, also improves the diversity, relevance, and user satisfaction of the recommendation results. Thus, it achieves end-to-end optimization from multimodal resource understanding and deep feature extraction to accurate personalized recommendation, effectively solving the problems of low efficiency, poor adaptability, and potential information risks faced by resource recommendation in the field of health science popularization.

[0061] It should be noted that the health science popularization resource recommendation method, system, equipment, and medium provided in this application can be applied to the field of computer technology. The above are merely examples and do not limit the application field of the health science popularization resource recommendation method, system, equipment, and medium provided in this application. Furthermore, the embodiments of this application may not limit the executing entity of the health science popularization resource recommendation. For example, the health science popularization resource recommendation method of this application embodiment can be applied to data processing devices such as terminal devices or servers. The terminal device can be an electronic device such as a computer or a personal digital assistant (PDA). The server can be a standalone server, a cloud server, or a cluster server composed of multiple servers.

[0062] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0063] The following embodiment illustrates a method for recommending health science popularization resources provided in this application. See also... Figure 1 ,Should Figure 1 A flowchart illustrating a method for recommending health science popularization resources provided in this application embodiment, the method comprising:

[0064] S101. Based on the target dataset, construct a multimodal knowledge graph that includes multimodal entities and relationships.

[0065] The target dataset includes multimodal health education resources, user information, and professional data from the medical field. Specifically, it involves constructing a multimodal knowledge graph based on diverse data sources, including multimodal health education resources (such as videos, audio, images, and text) and user information (including basic user information and behavioral data). The construction process begins with building a multimodal health education knowledge system based on prior professional knowledge in the medical field. Then, the MedBERT pre-trained model from the medical field is used to extract entity information, entity attributes, and inter-entity relationships from the multimodal data. These extracted triples are used to construct a structured multimodal knowledge graph, which is then stored in the graph database Neo4j for unified management and efficient access.

[0066] For example, see Figure 2 , Figure 2 This is a schematic diagram illustrating the construction process of a multimodal knowledge graph provided in an embodiment of this application. The construction of the multimodal knowledge graph includes the following steps:

[0067] First, data collection. This involves collecting basic user information (such as occupation, age, and gender) and behavioral data (such as historical records of searches, browsing, sharing, and saving). Simultaneously, it collects various health education resources, covering multimodal data including educational texts, images, and videos, and extracts relevant metadata such as resource name, author, type, description, and publication time. Furthermore, it integrates authoritative professional data from the Medical Subject Headings (MeSH), DrugBank, and the International Classification of Diseases (ICD-10), providing rich medical background knowledge support for the construction of the knowledge graph.

[0068] Second, schema layer construction. See also Figure 3 , Figure 3 This diagram illustrates the construction of the schema layer in an embodiment of this application. Based on collected information, a knowledge graph architecture is designed, a knowledge ontology framework is constructed, and various entity types, relationship types, and related attributes are clearly defined to form a structured schema layer. This provides a standardized semantic framework for subsequent knowledge extraction and fusion. Entity types include user, resource, resource type, health domain, population, disease, symptom, drug, image, video, and audio, while relationship types include browsing, collection, sharing, classification, suitability, mention, contraindication, having symptoms, having images, having videos, and having audio.

[0069] Third, knowledge extraction. For data of different modalities, the first step is to transform them into a unified text description format. Specifically, for image data, the generative large language model Llama3 is used to generate corresponding descriptive text; for audio data, the speech recognition model Whisper is used to transcribe it into text; for video data, Whisper is used to extract speech text, and Llama3 is used to generate descriptive text from the extracted video frame images. These two methods are then merged to form the final video text description. Based on this, the pre-trained model MedBERT is used uniformly to extract entity, relation, and attribute information from each text.

[0070] Fourth, knowledge fusion. By leveraging medical terminology, entity alignment and merging are performed on information from different sources, different modalities, or with different names but pointing to the same entity. At the same time, the same or similar relationships in different data sources are merged and unified to eliminate ambiguity and redundancy, achieving structured alignment and semantic consistency integration of cross-modal knowledge.

[0071] For example, the medical subject thesaurus includes MeSH (Medical Subject Headings), UMLS (Unified Medical Language System), and the Chinese Medical Subject Headings; the entity types include diseases, symptoms, drugs, examination items, healthy lifestyles, and popular science resource types; the relationship types include "disease-symptom", "disease-drug", "disease-examination item", "user-disease of concern", "popular science resource-disease involved", and "popular science resource-applicable population"; the attributes include the entity's name, definition, characteristics, timeliness, and credibility, wherein the credibility attribute is calculated based on the qualifications of the publishing entity of the popular science resource and the level of medical evidence cited, and the value range is 0-1.

[0072] Fifth, knowledge storage and management. Finally, the extracted and fused structured triple information is stored in the graph database Neo4j, realizing efficient storage, querying and management of multimodal knowledge graphs, and providing reliable knowledge support for subsequent feature extraction and recommendation tasks.

[0073] S102. Based on the multimodal knowledge graph, the graph attention network (GAT) is used to propagate and aggregate information on the nodes in the multimodal knowledge graph to obtain the structured feature representation vector of each node.

[0074] After obtaining the multimodal knowledge graph, this application employs a Graph Attention Network (GAT) to extract structured features from the nodes in the knowledge graph. By using GAT to propagate and aggregate information on the knowledge graph, complex dependencies between nodes can be captured, thereby generating high-quality structured feature representation vectors for each node and achieving deep structured modeling of health science resources and user characteristics.

[0075] For example, see Figure 4 , Figure 4 This is a schematic diagram of the structured feature extraction process provided in an embodiment of this application. The specific implementation process may include the following steps:

[0076] First, initial vector learning is performed. The DistMult knowledge graph representation learning model is used to vectorize the nodes in the graph. DistMult effectively models entities and relationships in the knowledge graph. By semantically encoding nodes and their adjacent relationships, it generates initial vector representations of the nodes, providing a preliminary feature foundation for the subsequent introduction of graph attention mechanisms.

[0077] Second, subgraph construction. For each target node, a weighted neighbor sampling method is used to construct its local subgraph. During sampling, different sampling weights are assigned based on the node's degree in the network (i.e., the number of connecting edges). Nodes with higher degrees are considered more important in the network and therefore have a higher selection probability during sampling. The paths obtained after sampling are reconstructed into a subgraph structure centered on the target node. This step facilitates efficient feature propagation and aggregation within the local area while reducing computational complexity.

[0078] Third, based on the completion of subgraph construction, this application adopts Graph Attention Network (GAT) for structured feature representation learning to fully integrate relational information in the knowledge graph and enhance the feature expression ability of nodes.

[0079] First, the initial vectors of the nodes are projected into a new feature space through a linear transformation layer. This step aims to map and align node features from different sources with a unified dimension, providing an adapted representation basis for subsequent attention calculations. Next, an attention mechanism is used to calculate the attention coefficients between the target node and each of its neighboring nodes. These coefficients quantify the importance of neighboring nodes in the target node's feature representation. Specifically, the attention coefficients combine the similarity between node features with their correlation strength within the network structure, enabling the model to adaptively focus on neighboring nodes that have a greater influence on the target node.

[0080] To facilitate subsequent weighted aggregation, the attention coefficients of all neighboring nodes of the target node are normalized to convert them into standard attention weights that sum to 1, ensuring that the model maintains numerical stability when aggregating neighbor features.

[0081] Subsequently, the features of all neighboring nodes are weighted and summed according to the normalized attention weights to obtain the updated feature representation of the target node. This aggregation process enables the target node to effectively integrate the structural and semantic information of its local neighborhood.

[0082] To further enhance the model's ability to capture multifaceted features of nodes, a multi-head attention mechanism is introduced. This involves executing multiple sets of attention calculations and aggregation processes in parallel, with each set focusing on different aspects or subspaces of the node's features. Finally, the outputs of the multiple attention heads are concatenated or averaged to obtain the final structured feature representation vector of the node. This mechanism not only improves the richness and robustness of feature representation but also enables the model to more comprehensively understand the structural and semantic roles of nodes in the knowledge graph.

[0083] Through the above process, graph attention networks achieve efficient extraction of node structural features in knowledge graphs, providing highly expressive structured feature vectors for subsequent feature fusion and recommendation tasks.

[0084] S103. Use the multimodal large model oneLLM to encode health science popularization resources and user information of different modalities to obtain semantic feature representation vectors of each node.

[0085] To effectively capture the deep semantic information contained in health science resources and user information, this application employs a multimodal large model (oneLLM) to uniformly encode data from various modalities, thereby obtaining cross-modal aligned semantic feature representation vectors. This method achieves consistent semantic expression by uniformly modeling data from multiple modalities such as text, images, audio, and video, mapping them to the same semantic vector space.

[0086] For example, see Figure 5 , Figure 5This is a schematic diagram of the semantic feature extraction process provided in an embodiment of this application. The specific implementation process may include the following steps:

[0087] First, data preprocessing. This involves preprocessing the input data of different modalities using appropriate tokenizers. For text data (such as popular science articles, user basic information, and descriptions of historical behavior), a text tokenizer is used to convert it into a token sequence. For image data, a visual tokenizer is used to segment it into a visual token sequence. For audio data, an audio tokenizer is used to extract token representations of its acoustic features. For video data, both visual and audio tokenizers are combined to extract token sequences for visual frames and audio segments, respectively. This step unifies the heterogeneous multimodal data into serialized token input, providing a structured representation for subsequent encoding.

[0088] Second, feature encoding. This involves inputting the token sequences from each modality into the oneLLM's universal encoder for deep encoding. This encoder, based on the Transformer architecture, possesses powerful cross-modal feature extraction and fusion capabilities, enabling it to capture the semantic structure within each modality's data and the relationships between them. After encoding, the input of each modality is transformed into an embedding vector with rich semantic information, containing both intra-modal features and implicit cross-modal semantic correspondences.

[0089] Third, vector mapping and alignment. To further achieve cross-modal semantic alignment, a unified projection module is used to map the embedding vectors obtained from each modality to the same low-dimensional semantic space. This projection module is typically composed of fully connected layers or lightweight transform networks. Through optimization of the alignment loss (such as contrastive learning loss), semantically similar content in different modalities is placed close together in the vector space, while semantically dissimilar content is placed far apart. Ultimately, each node (whether a resource node or a user node) can obtain a unified semantic feature representation vector that can reflect its deep semantic content across modalities.

[0090] The projection module, composed of several fully connected layers (or lightweight transform networks), possesses non-linear activation capabilities and can map high-dimensional embedding vectors from different modalities such as text, images, and audio to the same low-dimensional semantic space. By optimizing alignment loss functions such as contrastive learning loss, the module brings semantically similar content from different modalities closer together in the mapped vector space, while widening the distance between content with significant semantic differences. This mechanism ensures consistent alignment across modal semantics, ultimately generating a unified feature vector for each node with cross-modal semantic representation capabilities, supporting subsequent multimodal fusion and recommendation tasks.

[0091] Through the above process, oneLLM not only achieves unified semantic encoding of multimodal data, but also provides an interactive and comparable cross-modal semantic representation foundation for subsequent feature fusion and recommendation modeling through semantic alignment of vector space, which strongly supports the deep semantic understanding of user needs and resource content.

[0092] S104. For each node, structured features and semantic features are fused to obtain a fused feature representation; based on the fused feature representation, a training dataset is constructed through vector retrieval; based on the training dataset, the health science popularization resource recommendation model is trained by using the open-source multimodal pre-trained large language model Llama 3 and combined with the LoRA+ fine-tuning method.

[0093] After obtaining structured and semantic features, this application further integrates these two types of features to construct a unified feature representation, and trains a health science popularization resource recommendation model based on this representation. This model achieves more accurate and personalized resource recommendations by comprehensively utilizing the structured association information of the knowledge graph and the deep semantic understanding capabilities of a large model.

[0094] For example, see Figure 6 , Figure 6 The diagram illustrates the construction process of the health science popularization resource recommendation model provided in this application embodiment. The specific implementation process may include the following steps:

[0095] First, feature projection and fusion. First, a projection network is used to map structured feature vectors and semantic feature vectors into the same vector space. This projection network is typically a multilayer perceptron or a lightweight neural network, achieving spatial alignment and interaction enhancement of features from different sources through nonlinear transformations. Then, the two types of projected feature vectors are concatenated to obtain a fused feature representation for each user and resource. This representation simultaneously preserves the interpretability of structural relationships and the richness of semantic content, providing a comprehensive feature foundation for the recommender system.

[0096] Second, a training dataset is constructed. Based on the aforementioned fusion features, a vector retrieval technique is used to construct the training dataset. Specifically, firstly, similarity retrieval (e.g., using cosine similarity or inner product retrieval) is performed in the resource vector space based on user feature vectors to obtain a preliminary list of candidate recommended resources. To ensure recommendation quality, this candidate list is manually screened and labeled as positive samples. Simultaneously, a portion of the resources that have never been retrieved is randomly selected as negative samples. The positive and negative samples together constitute a labeled training dataset, with the label indicating whether the resource should be recommended to the corresponding user.

[0097] Third, model training. The open-source multimodal pre-trained large language model Llama 3 was used as the basic architecture, and supervised fine-tuning was performed in conjunction with a health science popularization resource recommendation task. Specifically, targeted prompt templates were designed, taking user features and resource features as input, and outputting a binary classification result indicating whether a resource is recommended. To efficiently adapt to the recommendation task and avoid large-scale parameter adjustments, the LoRA+ (Low-Rank Adaptation Plus) method was used to fine-tune the model. This method introduces low-rank adaptation increments into some weight matrices, significantly improving task adaptability and training efficiency while maintaining the generality of the pre-trained model. After fine-tuning, the model can understand the complex matching relationship between users and resources, thereby achieving accurate recommendation decisions.

[0098] The LoRA+ fine-tuning method for training the model includes: the rank of the low-rank adaptation increment is in the range of 8-64; for the Transformer layer of the Llama 3 model, the low-rank adaptation increment is introduced only into the Q, K, and V matrices of the attention layer; during fine-tuning, the original weight parameters of the Llama 3 model are fixed, and only the parameters of the low-rank adaptation increment and the parameters of the projection module are updated; a learning rate decay strategy is adopted, with the initial learning rate set to 1e-4 to 5e-4, and the learning rate decayed to 0.9 to 0.95 times the original value after a preset number of iterations; the preset performance indicators include accuracy, recall, F1 score, and normalized depreciation cumulative gain (NDCG); training stops when the NDCG value of the model on the validation set does not improve for a preset number of consecutive iterations.

[0099] Fourth, model optimization. After the model goes live, it continuously collects user feedback on recommended content, such as implicit feedback like clicks, favorites, and shares, as well as explicit feedback like ratings and comments. This feedback data is used to build an online learning or incremental learning mechanism to dynamically update model parameters, enabling the recommendation system to adapt to changes in user interests and updates to resource content, maintaining the timeliness and accuracy of recommendations.

[0100] Through the above process, this method constructs a health science popularization resource recommendation model that can deeply integrate structured and semantic features, is trainable and optimizable, and ultimately achieves personalized, precise and dynamic health knowledge services.

[0101] S105. Based on the health science popularization resource recommendation model, generate a personalized health science popularization resource recommendation list.

[0102] After completing the training and optimization of the health science popularization resource recommendation model, this application enters the resource recommendation stage, aiming to generate a personalized and dynamically updated health science popularization resource recommendation list for users based on the model output, so as to achieve accurate response and continuous service to user needs.

[0103] In one possible implementation, generating a personalized health science popularization resource recommendation list based on the health science popularization resource recommendation model includes: sorting the health science popularization resources according to the recommendation probability or recommendation strength output by the health science popularization resource recommendation model; dynamically adjusting the sorting results according to the user's current fusion feature representation and real-time behavioral data; selecting the top-ranked health science popularization resources that meet the preset recommendation quantity, generating a personalized health science popularization resource recommendation list, and pushing it to the corresponding user in a preset display format.

[0104] In one possible implementation, after generating a personalized list of recommended health science resources, the following is also included:

[0105] The system collects user interaction data on health science resources in the recommended list, including at least one of clicks, favorites, shares, dwell time, and evaluation feedback; based on the interaction data, the health science resource recommendation model is continuously optimized and iteratively updated.

[0106] The step of continuously optimizing and iteratively updating the health science popularization resource recommendation model based on the interaction behavior data includes: using the interaction behavior data as incremental training samples to update the training dataset; and using the incremental training samples to fine-tune the trained health science popularization resource recommendation model online to adapt to dynamic changes in user interests and updates to resource content.

[0107] For example, the specific implementation process may include the following steps:

[0108] First, personalized recommendation list generation. For a specific user, the corresponding fusion feature representation vector is extracted and input into a pre-trained health science resource recommendation model. The model performs deep inference and preference prediction based on the matching degree between user features and all resource features, outputting a quantitative result of whether each resource is recommended and the strength of the recommendation. Based on this result, candidate resources are ranked, and the resources with the highest recommendation strength are selected to form a personalized recommendation list for that user.

[0109] Second, dynamic recommendations. To adapt to the timeliness requirements of health education content and the dynamic changes in user interests, the recommendation system not only bases its recommendations on current user characteristics but also integrates real-time behavioral feedback (such as recent browsing, favorites, and search history) for real-time reordering and updates. The system can recalculate recommendation results periodically (e.g., daily or weekly) through online inference or incremental prediction mechanisms to ensure that recommended content remains synchronized with users' current interests and the latest health education resources. For newly published resources or newly registered users, the system has cold start processing capabilities, providing initial recommendations through strategies such as resource content similarity or user clustering, and gradually optimizing in subsequent interactions.

[0110] Third, multi-dimensional recommendation presentation. The generated recommendation list can be further structured and visualized based on resource type (such as video, text, and audio), health theme (such as disease prevention, nutrition, and sports rehabilitation), and recommendation reasons (such as matching users' historical interests, high-authority sources, and high interaction popularity), thereby enhancing the transparency of the recommendation results and user acceptance.

[0111] Through the aforementioned recommendation phase, this application not only achieves precise and personalized distribution of health science popularization resources, but also constructs a sustainable and iterative recommendation service mechanism, helping to improve users' efficiency in acquiring health knowledge and the effectiveness of health literacy cultivation.

[0112] One possible implementation also includes a dynamic update step for the multimodal knowledge graph:

[0113] Regularly collect newly added multimodal health science popularization resources, user update information, and the latest professional data in the medical field; use the entity, relation, and attribute extraction method described in the embodiments of this application to perform structured processing on the new data to obtain incremental triple information; based on the medical professional thesaurus and the existing knowledge graph system, perform conflict detection and fusion on the entities, relations, and attributes in the incremental triples; if the new entity has a synonym or subordinate relationship with an existing entity, update the attributes and associations of the existing entity; write the fused incremental triples into the graph database to realize the incremental update of the multimodal knowledge graph; at the same time, delete entity, relation, and attribute information in the graph database that exceeds the preset timeliness threshold or is marked as invalid.

[0114] Therefore, the beneficial effects of this application include:

[0115] First, by constructing a multimodal knowledge graph that integrates multimodal health science resources and user information, the model's knowledge understanding and reasoning capabilities are significantly enhanced. The knowledge graph not only structurally represents the complex relationships between professional concepts, resource content, and user behavior in the health field, but also provides users with more intuitive and traceable recommendation criteria through the visualization of entities and relationships. This enhances the interpretability and credibility of recommendation results, helps users understand "why recommendations are made," and increases user trust and acceptance of the recommendation system.

[0116] Second, this application focuses on dual modeling of structured and semantic features, achieving a more comprehensive and in-depth representation of user interests and resource content. Structured features capture the association patterns and topological structures between users and resources using knowledge graphs and graph attention networks, while semantic features extract deep semantics of resources and implicit user preferences through multimodal large models. The organic integration of the two enables the recommender system to not only understand explicit behavioral associations but also mine potential semantic matches, thereby significantly improving recommendation efficiency, achieving more accurate and personalized recommendation results, and ultimately significantly enhancing user satisfaction and system stickiness.

[0117] Third, the recommendation model is built based on the open-source large language model Llama 3 and combined with the LoRA+ fine-tuning method, giving the system stronger deep reasoning and content understanding capabilities. The large model already possesses rich knowledge reserves and a solid foundation in language understanding during the pre-training stage, enabling complex semantic associations and inferences between user historical behavior and resource semantics. Furthermore, the large model supports exploratory recommendations, breaking through the limitations of traditional collaborative filtering or content matching to recommend novel and relevant high-quality content to users. This ensures recommendation accuracy while improving resource discovery efficiency and enriching the user experience.

[0118] In summary, this application, through the construction of a multimodal knowledge graph, dual-feature fusion modeling, and a large-model-driven recommendation strategy, not only achieves efficient, accurate, and interpretable resource recommendation, but also provides strong technical support for the personalized distribution of health science popularization content and the improvement of users' health literacy, which has important practical value and social significance.

[0119] The above are some specific implementations of the health science popularization resource recommendation method provided in the embodiments of this application. Based on this, this application also provides a corresponding health science popularization resource recommendation system. The system provided in the embodiments of this application will be described below from the perspective of functional modularization. Figure 7 This is a structural diagram of a health science popularization resource recommendation system provided in an embodiment of this application.

[0120] The system includes:

[0121] The construction unit 110 is used to construct a multimodal knowledge graph including multimodal entities and relationships based on the target dataset, wherein the target dataset includes multimodal health science popularization resources, user information and professional data in the medical field;

[0122] Aggregation unit 111 is used to perform information propagation and aggregation on nodes in the multimodal knowledge graph using graph attention network GAT based on the multimodal knowledge graph, so as to obtain the structured feature representation vector of each node;

[0123] Encoding unit 112 is used to encode health science popularization resources and user information of different modalities using the multimodal large model oneLLM to obtain semantic feature representation vectors of each node;

[0124] Training unit 113 is used to fuse structured features and semantic features for each node to obtain a fused feature representation; based on the fused feature representation, a training dataset is constructed through vector retrieval; based on the training dataset, a health science popularization resource recommendation model is trained by using the open-source multimodal pre-trained large language model Llama 3 and combining it with the LoRA+ fine-tuning method.

[0125] The generation unit 114 is used to generate a personalized health science popularization resource recommendation list based on the health science popularization resource recommendation model.

[0126] This application also provides corresponding devices and computer storage media for implementing the health science popularization resource recommendation scheme provided in this application.

[0127] The device includes a memory and a processor. The memory is used to store instructions or code, and the processor is used to execute the instructions or code so that the device performs the health science popularization resource recommendation method according to any embodiment of this application.

[0128] The computer storage medium stores code, and when the code is executed, the device running the code implements the health science popularization resource recommendation method described in any embodiment of this application.

[0129] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0130] It should be understood that in this application, "at least one" refers to one or more items, and "more" refers to two or more items. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one" or similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, "at least one" of a, b, or c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0131] It should be understood that the terms center, longitudinal, transverse, up, down, front, back, left, right, vertical, horizontal, top, bottom, inside, outside, etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0132] It should be noted that, unless otherwise explicitly specified and limited, the terms installation, connection, and linking should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0133] It should also be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the statement "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0134] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0135] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for recommending health science popularization resources, characterized in that, include: Based on the target dataset, a multimodal knowledge graph including multimodal entities and relationships is constructed. The target dataset includes multimodal health science popularization resources, user information, and professional data in the medical field. Based on the multimodal knowledge graph, the graph attention network (GAT) is used to propagate and aggregate information on the nodes in the multimodal knowledge graph to obtain the structured feature representation vector of each node. The oneLLM multimodal large model is used to encode health science popularization resources and user information of different modalities to obtain semantic feature representation vectors of each node; For each node, structured features and semantic features are fused to obtain a fused feature representation; based on the fused feature representation, a training dataset is constructed through vector retrieval; based on the training dataset, an open-source multimodal pre-trained large language model Llama 3 is used in conjunction with the LoRA+ fine-tuning method to train a health science popularization resource recommendation model; Based on the aforementioned health science popularization resource recommendation model, a personalized list of health science popularization resources is generated.

2. The method for recommending health science popularization resources according to claim 1, characterized in that, The construction of a multimodal knowledge graph, including multimodal entities and relationships, based on the target dataset includes: Based on the target dataset, entity types, relation types, and attributes are defined to construct the knowledge graph system architecture. The pre-trained model MedBERT is used to extract entities, relationships, and attributes from multimodal data. Based on a medical terminology, data from different sources, modalities, or with different names but pointing to the same entity and attribute are merged, and similar or identical relationships from different data sources are integrated. The extracted and fused structured triple information is stored in a graph database to obtain a multimodal knowledge graph that includes multimodal entities and relationships.

3. The method for recommending health science popularization resources according to claim 2, characterized in that, The extraction of entities, relationships, and attributes from multimodal data using the pre-trained model MedBERT includes: For image-based data, corresponding descriptive text is generated using the generative large language model Llama 3; For audio data, the Whisper speech recognition model is used to convert it into corresponding descriptive text. For video data, the Whisper speech recognition model is used to extract the descriptive text corresponding to the speech, and the Llama 3 generative large language model is used to generate the descriptive text corresponding to the video frames. The final descriptive text of the video is obtained by combining the two descriptive texts. Based on the descriptive text converted from the images, audio, and video, as well as the original text data, the MedBERT pre-trained model in the medical field is used to extract entity, relation, and attribute information.

4. The method for recommending health science popularization resources according to claim 1, characterized in that, The method utilizes a multimodal large model (oneLLM) to encode health science resources and user information of different modalities, obtaining semantic feature representation vectors for each node, including: Different modal tokenizers are used to convert text, images, videos, and audio into token sequences respectively; The token sequences of each modality are encoded using a unified encoder to obtain the embedding vector of each modality; The embedding vectors of each modality are mapped to the same semantic vector space by the projection module, thus obtaining the semantic feature representation vector of each node.

5. The method for recommending health science popularization resources according to claim 1, characterized in that, The fusion of structured features and semantic features yields a fused feature representation including: Structured feature vectors and semantic feature vectors are mapped to a unified interactive vector space through projection networks; The two types of feature vectors after mapping are concatenated to obtain the fused feature representation of each node.

6. The method for recommending health science popularization resources according to claim 1, characterized in that, The training dataset, constructed based on the fused feature representation and through vector retrieval, includes: Based on the fusion feature vectors of each user, similarity matching retrieval is performed in the space formed by the fusion feature vectors of all health science popularization resources. A preset number of resources are selected according to the similarity ranking results to obtain a candidate resource list. The resources in the candidate resource list are filtered, and resources that match the user's needs, health status and interests are retained as positive examples. From the remaining health science resources outside the candidate resource list, a preset number of resources are randomly selected as negative sample samples. The training dataset is composed of the positive and negative examples. The training dataset contains the correspondence between user information and health science resources, as well as the classification labeling results used to characterize whether a resource is recommended.

7. The method for recommending health science popularization resources according to claim 1, characterized in that, Based on the training dataset, a health science popularization resource recommendation model is trained using the open-source multimodal pre-trained large language model Llama 3 combined with the LoRA+ fine-tuning method, including: Based on the health science popularization resource recommendation task, a corresponding prompt template is generated. The prompt template is used to construct an input format that can be understood by a large language model by fusing feature representation and classification labeling results. The training dataset is input into the open-source multimodal pre-trained large language model Llama 3, and the model is trained using the LoRA+ fine-tuning method, which adjusts some of the model's weight matrices by introducing low-rank adaptation increments. The model parameters are optimized using the gradient descent algorithm until the model reaches the preset performance index on the recommendation task, thus obtaining the trained health science popularization resource recommendation model.

8. A health science popularization resource recommendation system, characterized in that, include: The construction unit is used to construct a multimodal knowledge graph including multimodal entities and relationships based on a target dataset, wherein the target dataset includes multimodal health science popularization resources, user information, and professional data in the medical field; The aggregation unit is used to perform information propagation and aggregation on the nodes in the multimodal knowledge graph using the graph attention network (GAT) to obtain the structured feature representation vector of each node. The encoding unit is used to encode health science resources and user information of different modalities using the multimodal large model oneLLM to obtain the semantic feature representation vector of each node; The training unit is used to fuse structured features and semantic features for each node to obtain a fused feature representation; based on the fused feature representation, a training dataset is constructed through vector retrieval; based on the training dataset, the open-source multimodal pre-trained large language model Llama 3 is used in conjunction with the LoRA+ fine-tuning method to train a health science popularization resource recommendation model. The generation unit is used to generate a personalized list of recommended health science resources based on the health science popularization resource recommendation model.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the health science popularization resource recommendation method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the health science popularization resource recommendation method as described in any one of claims 1-7.

Images