Rescue information knowledge graph construction method and system based on large language model

Abstract

The invention belongs to the technical field of knowledge graph construction, and relates to a rescue information knowledge graph construction method based on a large language model. Firstly, multi-modal data are collected and preprocessed; then, newly-added knowledge confidence degree judgment is carried out; taking the preprocessed multi-modal data as basic data to obtain a plurality of single-modal feature sequences; performing cross-modal fusion on the plurality of single-modal feature sequences to obtain a fused structured text; taking the fused structured text as input, and extracting a relationship among a plurality of entities; relation extraction between entities is converted into generation tasks of entity pairs and relation types; and designing a rescue information knowledge graph ontology framework, converting the generated task into a triple, storing the graph by adopting a graph database, and outputting the graph by adopting a preset format. The invention further provides a rescue information knowledge graph construction system based on the large language model and a terminal. Compared with a traditional manual annotation method, the knowledge extraction efficiency and accuracy are improved.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph construction technology, and in particular to a method and system for constructing a rescue information knowledge graph based on a large language model. Background Technology

[0002] Rescue work during historical periods is an important part of Chinese history, containing a wealth of historical information, including numerous organizations, figures, and events. This historical information is scattered across various historical materials, such as official archives, organizational documents, historical photographs, historical videos, and personal accounts, lacking systematic integration. This makes it difficult for people to fully and clearly understand the overall picture of rescue work during historical periods and the network for the dissemination of rescue knowledge.

[0003] Knowledge graphs, as a structured representation of knowledge, can intuitively present complex knowledge systems through the association of entities and relationships, providing strong support for historical research. However, traditional knowledge graph construction methods mostly rely on manual annotation, which is not only inefficient but also limited by the knowledge scope and effort of human personnel, making it difficult to cover a large amount of scattered historical materials, resulting in insufficient completeness and accuracy of the constructed knowledge graphs.

[0004] With the development of multimodal large language models, they have demonstrated powerful capabilities in natural language processing tasks, such as entity recognition and relation extraction. However, there are currently no technologies to apply multimodal large language models to the construction of knowledge graphs for historical rescue work, which prevents the full utilization of the advantages of large language models to solve the problems existing in traditional methods. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for constructing a knowledge graph of emergency medical care information based on a large language model. This method utilizes large language model technology to address the difficulties in integrating relevant knowledge of emergency medical care work from different historical periods, the disorganized distribution of historical materials, and the low efficiency, incompleteness, and inaccuracy of traditional knowledge graph construction methods. By combining multimodal data such as historical photographs and audio materials, it develops knowledge reasoning capabilities, enabling deeper exploration and application of emergency medical care work from different historical periods.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for constructing a knowledge graph of ambulance information based on a large language model, comprising the following steps: S10. Multimodal data collection and preprocessing; multimodal data includes text data, image data, audio data, and video data; preprocessing includes data cleaning, format standardization, word segmentation, and part-of-speech tagging; S20. New Knowledge Confidence Determination: Based on the knowledge database established by prior knowledge, evaluate the confidence of new knowledge. New knowledge with a confidence level higher than the threshold is included in the knowledge graph for further processing, while new knowledge with a confidence level lower than the threshold is identified as unreliable and discarded. The confidence level is obtained by weighted summation of data source, data recording time, cross-validation results, data integrity, and expert scores. S30. Knowledge extraction and fusion; specifically including: defining multiple types of entities, using preprocessed text data, image data, audio data, and video data as the base data, extracting entity information for each to obtain multiple single-modal feature sequences; performing cross-modal fusion of multiple single-modal feature sequences to obtain fused structured text; S40. Relationship extraction; specifically including: taking the fused structured text as input, extracting relationships between multiple entities; and transforming the extracted relationships into entity pairs and relationship types. S50. Design an ontology framework for a rescue information knowledge graph, transform the generation task into a triple, store the graph in a graph database, and output the graph in a preset format.

[0007] As one possible implementation method, entity information extraction specifically includes: For text data, the BERT-WWM-Chinese model is used to extract text entity information; For image and video data, a Transformer-based model is used to identify behavioral entity information of people and / or objects in the images and / or videos, and / or event entity information. When there are subtitles or narration in the video data, optical character recognition methods are used to extract the subtitle text, and then the BERT-WWM-Chinese model is used to extract the text entity information. For audio data, speech recognition methods are used to extract audio text, and the visual and auditory content of preset characters and events within a preset event range is correlated with text semantics through cross-modal association.

[0008] As one possible implementation, cross-modal fusion of multiple unimodal feature sequences is performed to obtain fused structured text, specifically including: Pre-trained cross-modal information fusion model based on Transformer architecture; The application of a cross-modal information fusion model based on the Transformer architecture binds text feature sequences, image feature sequences, audio feature sequences, and video feature sequences to a common expression space, and uses images as a hub modality to achieve retrieval and generation between any modalities.

[0009] As one possible implementation, when pre-training a cross-modal information fusion model based on the Transformer architecture, initial training samples are configured and the confidence of the initial training samples is set to 1. Entity information obtained after knowledge extraction is continuously added to the initial training samples to continuously update them. When updating the initial training samples, the confidence of newly added knowledge is first calculated, and entity information with a confidence of newly added knowledge greater than or equal to a preset threshold is updated to the initial training samples.

[0010] As one possible implementation, the confidence score of the newly added knowledge is denoted as Y, and is calculated as follows: Y=[X,Z,L,M,N]W=X×0.25+Z×0.15+L×0.35+M×0.10+N×0.15 Where X represents the data source, Z represents the data recording time, L represents one-time cross-validation, M represents data integrity, N represents expert scores, and W represents weight values.

[0011] As one possible implementation, cross-modal fusion includes self-attention within the same modality and cross-attention between different modalities to support the fusion of image and text data, and extends to the cross-modal fusion of audio and video data. For cross-modal fusion of text, audio and video data, each modality data is first encoded into a feature sequence, and then the cross-attention mechanism of Transformer is reused to achieve fusion.

[0012] As one possible implementation, for text data, BERT is used to encode the text sequence, outputting a text feature sequence T=[t0,t1,...,t...]. n For image data, Faster R-CNN is used to extract features of target regions in the image, outputting a visual feature sequence U=[u0,u1,...,u...]. m For audio data, Mel-frequency spectrum and CNN or AudioBERT are used to generate audio feature sequences A=[a0,a1,...,a2]. k ]; Based on this, cross-modal fusion specifically involves adding audio / video streams to the ViLBERT / LXMERT framework and achieving multimodal interaction through multi-stream attention. The general formula for attention calculation is as follows:

[0013] Where d is the hidden layer dimension, Q is the query, K is the key, and V is the value, which comes from different modalities or the same modality to achieve cross-modal attention; Cross-modal layer update formula: Out = LayerNorm(Attention(Q,K,V) + Input).

[0014] As one possible implementation, the T5 model is used to transform the extraction of relationships between entities into the task of generating entity pairs and relationship types.

[0015] Secondly, this invention provides a rescue information knowledge graph construction system based on a large language model, comprising: The data processing module is used for multimodal data collection and preprocessing; multimodal data includes text data, image data, audio data, and video data; preprocessing includes data cleaning, format standardization, word segmentation, and part-of-speech tagging; The knowledge extraction module defines multiple types of entities and extracts entity information from preprocessed text data, image data, audio data, and video data to obtain multiple single-modal feature sequences. The knowledge fusion module performs cross-modal fusion on multiple single-modal feature sequences to obtain fused structured text. The knowledge modeling module takes the fused structured text as input and extracts the relationships between multiple entities; it then transforms the extracted relationships into entity pairs and relationship types. The knowledge storage module designs an ontology framework for a rescue information knowledge graph, transforms the generated tasks into triples, stores the graph in a graph database, and outputs the graph in a preset format.

[0016] Thirdly, the present invention provides a terminal, comprising: The memory is used to store the program for constructing a knowledge graph of ambulance information based on a large language model; The processor is used to implement the steps of the rescue information knowledge graph construction method based on the large language model provided in the first aspect when executing the rescue information knowledge graph construction program based on the large language model.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention utilizes a large language model to extract knowledge about rescue work in historical periods, which improves the efficiency and accuracy of knowledge extraction. Compared with traditional manual annotation methods, it can process more historical materials and cover a wider range of knowledge. 2. The knowledge graph constructed in this invention integrates scattered historical materials on rescue work from different periods. Through the association of entities and relationships, it intuitively presents the overall picture of rescue work during historical periods, providing systematic knowledge support for historical research. By combining multimodal data such as historical photographs and audio materials, it develops knowledge reasoning functions to achieve deeper exploration and application of rescue work during historical periods. 3. The knowledge graph constructed by this invention is characterized by visualization and reasoning capabilities, which can help researchers discover historical patterns, fill in missing historical materials, and provide new tools and methods for the study of historical rescue history. The methods and systems of this invention can provide a reference for the construction of knowledge graphs for other historical periods and have certain promotional value. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 and Figure 2 The flowchart of the method for constructing a rescue information knowledge graph based on a large language model provided by the present invention is shown. Detailed Implementation

[0019] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, the first threshold and the second threshold are merely used to distinguish different thresholds and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0020] It should be noted that in this invention, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0021] In this invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one" or similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, "at least one of a, b, or c" can represent: a, b, c, a combination of a and b, a combination of a and c, a combination of b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0022] This invention aims to provide a method and system for constructing a knowledge graph of emergency medical care information based on a large language model. It utilizes large model technology to address the difficulties in integrating relevant knowledge of emergency medical care work from different historical periods, the disorganized distribution of historical materials, and the low efficiency, incompleteness, and inaccuracy of traditional knowledge graph construction methods. By combining multimodal data such as historical photographs and audio materials, it develops knowledge reasoning capabilities, enabling deeper exploration and application of emergency medical care work from different historical periods.

[0023] Firstly, embodiments of the present invention provide a method for constructing a knowledge graph of ambulance information based on a large language model, see [link to relevant documentation]. Figure 1 and Figure 2 It includes the following steps: S10. Multimodal data collection and preprocessing; multimodal data includes text data, image data, audio data, and video data; preprocessing includes data cleaning, format standardization, word segmentation, and part-of-speech tagging; As an example, historical materials related to rescue work during historical periods were collected, including official archives, organizational documents, historical photos, historical videos, and personal accounts. The collected materials underwent text cleaning, image cleaning, and video cleaning to remove noise from scanned documents. The format was standardized using the large language model GPT-3.5, and the date format was unified as "YYYY-MM-DD". Word segmentation tools were used in conjunction with a historical domain dictionary for word segmentation, and the BERT-base model was used to complete part-of-speech tagging.

[0024] S20. New Knowledge Confidence Determination: Based on the knowledge database established by prior knowledge, evaluate the confidence of new knowledge. New knowledge with a confidence level higher than the threshold is included in the knowledge graph for further processing, while new knowledge with a confidence level lower than the threshold is identified as unreliable and discarded. The confidence level is obtained by weighted summation of data source, data recording time, cross-validation results, data integrity, and expert scores. S30. Knowledge extraction and fusion; specifically including: defining multiple types of entities, using preprocessed text data, image data, audio data, and video data as the base data, extracting entity information for each to obtain multiple single-modal feature sequences; performing cross-modal fusion of multiple single-modal feature sequences to obtain fused structured text; As one possible implementation method, entity information extraction specifically includes: For text data, the BERT-WWM-Chinese model is used to extract text entity information; For image and video data, a Transformer-based model is used to identify behavioral entity information of people and / or objects in the images and / or videos, and / or event entity information. When there are subtitles or narration in the video data, optical character recognition methods are used to extract the subtitle text, and then the BERT-WWM-Chinese model is used to extract the text entity information. For audio data, speech recognition methods are used to extract audio text, and the visual and auditory content of preset characters and events within a preset event range is correlated with text semantics through cross-modal association.

[0025] As an example, we define m core entities, namely people, organizations, locations, events, medical resources, and time. Among them, organization entities include ambulance headquarters and training centers for medical personnel in various regions; location entities include hospitals in various regions; event entities include joint training of medical personnel at a certain time; medical resource entities include sulfonamide drugs, mobile operating vehicles, ambulances, portable X-ray machines, serum vaccines, and first aid training materials; and time entities include specific years and months.

[0026] The BERT-WWM-Chinese model was used for text entity recognition. 5000 sample sentences were extracted from historical materials, and entity boundaries and types were manually labeled to construct a training set. The training and validation sets were divided in an 8:2 ratio. Learning rate and iteration count were set, and adversarial training (Fast Gradient Sign Method, FGSM) was incorporated to enhance model robustness. Low-Rank Adaptation (LoRA) was used for fine-tuning to optimize the general-purpose model's knowledge extraction capabilities in the medical care field. A Transformer-based model was used to recognize actions of people / objects in images and videos (such as "bandaging," "stopping bleeding," "debridement," "injection," "administering medication," "waving," "opening doors," "lifting," "falling," etc.), or complex events (such as "rescue," "surgery," "medical accident," "patient transfer," "ambulance escort," "patient rehabilitation training," etc.). If the video had subtitles or narration, OCR (Optical Character Recognition) was used to extract the subtitle text. By combining speech recognition (ASR) to extract audio text, the visual and auditory content of a person or event within a certain time interval is matched with the text semantics through cross-modal association (such as matching the scene of "medical staff bandaging a patient" in the video with the interview audio of "organizing a group of medical staff to bandage and stop the bleeding of the patient").

[0027] As one possible implementation, cross-modal fusion of multiple unimodal feature sequences is performed to obtain fused structured text, specifically including: Pre-trained cross-modal information fusion model based on Transformer architecture; As one possible implementation, when pre-training a cross-modal information fusion model based on the Transformer architecture, initial training samples are configured and the confidence of the initial training samples is set to 1. Entity information obtained after knowledge extraction is continuously added to the initial training samples to continuously update them. When updating the initial training samples, the confidence of newly added knowledge is first calculated, and entity information with a confidence of newly added knowledge greater than or equal to a preset threshold is updated to the initial training samples.

[0028] As an example, knowledge confidence refers to the degree of credibility of extracted historical knowledge. The initial training samples of the large model database are labeled initial historical knowledge (official archives, official images, official video materials, etc.) with a confidence level of 1. After new knowledge is extracted, it needs to be compared with the existing historical knowledge in the training samples to calculate the confidence level of the new knowledge. New knowledge with a confidence level exceeding 80% can be included in the database for training as training samples of the large model. Knowledge with a confidence level below 80% is not included in the database and is promptly discarded without further relation extraction.

[0029] The confidence level (Y) for newly added knowledge needs to be calculated based on five factors: data source (X), data recording time (Z), one-time cross-validation (L), data completeness (M), and expert rating (N), resulting in five sub-confidence levels. Data source (X): The confidence level of primary historical knowledge (such as official archives, manuscripts, and eyewitness diaries) is generally higher than that of secondary historical knowledge (such as later recounts and historical works), and even higher than that of unofficial historical knowledge (unofficial histories, novels, etc.). Generally, the confidence level of primary historical knowledge is set to 1, the confidence level of secondary historical knowledge is set to 80%, and the confidence level of unofficial historical knowledge is set to 50%. Data Recording Time (Z): Immediate records at the time of the event have higher confidence than subsequent recollections (such as memoirs written many years later). When calculating the knowledge recording time-distributed confidence score, five gradients are used: immediate records are considered 1 point, records within six months are considered 0.9 points, records within one year are considered 0.8 points, records within three years are considered 0.7 points, records within ten years are considered 0.6 points, records within twenty years are considered 0.5 points, and all others are considered 0.3 points. One-Time Cross-Validation (L): This assesses the degree of agreement with other validated historical materials and whether they conform to the logic of the historical context (e.g., whether the systems, technologies, and social customs are reasonable). A fuzzy neural network algorithm is used for comparison and training. The validation results are fuzzily matched to five gradients: 1, 0.8, 0.6, 0.4, and 0.2, and the gradient values ​​are used as the one-time cross-validation-distributed confidence score results. Data integrity (M): Whether the original historical knowledge is complete and incomplete, including seven aspects: time, place, people, events, organizations, medical resources, and physical integrity. People, events, and medical resources each account for 20%, and the remaining four each account for 10%. If any condition is not met, subtract from 1 in turn. The result is the knowledge integrity-sub-confidence score. Expert rating (N): Five experts in the field rate the authenticity of a historical knowledge (0-1). The average of the results is taken as the expert rating-sub-confidence score. Sub-confidence weight allocation: The weights of each dimension are set as follows: data source (25%), data recording time (15%), one-time cross-validation (35%), data integrity (10%), and expert rating (15%). The weight vector is W=[0.25,0.15,0.35,0.10,0.15].

[0030] As one possible implementation, the confidence score of the newly added knowledge is denoted as Y, and is calculated as follows: Y=[X,Z,L,M,N]W=X×0.25+Z×0.15+L×0.35+M×0.10+N×0.15 Where X represents the data source, Z represents the data recording time, L represents one-time cross-validation, M represents data integrity, N represents expert scores, and W represents weight values.

[0031] The application of a cross-modal information fusion model based on the Transformer architecture binds text feature sequences, image feature sequences, audio feature sequences, and video feature sequences to a common expression space, and uses images as a hub modality to achieve retrieval and generation between any modalities.

[0032] As one possible implementation, cross-modal fusion includes self-attention within the same modality and cross-attention between different modalities to support the fusion of image and text data, and extends to the cross-modal fusion of audio and video data. For cross-modal fusion of text, audio and video data, each modality data is first encoded into a feature sequence, and then the cross-attention mechanism of Transformer is reused to achieve fusion.

[0033] As an example, semantic recognition relies on the fusion of cross-modal information (visual, audio, and text). It deeply mines information and cross-links semantics across different modalities through attention mechanisms and feature concatenation. Leveraging the powerful self-attention and cross-attention mechanisms of the Transformer, visual, audio, and text features interact and enhance each other in multi-layered networks, forming a highly fused joint representation. Models such as ViLBERT and LXMERT under the Transformer framework effectively handle cross-modal information fusion tasks, achieving good results in tasks such as visual question answering and image-text retrieval. By binding multiple modalities such as image, text, audio, and depth to a common representation space, and using the image as a "hub" modality, retrieval and generation between any modalities are achieved. This pre-trained large model can capture a wider range of cross-modal semantic information and has strong generalization ability, quickly adapting to various downstream tasks with limited labeled data.

[0034] This invention employs deeper cross-modal interaction, including self-attention (within the same modality) and cross-attention (cross-modal), supporting visual-text fusion and extending to cross-modal fusion of audio and video. For text, audio (such as speech), and video (temporal images), the core is to first encode them into feature sequences and then reuse the Transformer's cross-attention mechanism.

[0035] As one possible implementation, for text data, BERT is used to encode the text sequence, outputting a text feature sequence T=[t0,t1,...,t...]. n For image data, Faster R-CNN is used to extract features of target regions in the image, outputting a visual feature sequence U=[u0,u1,...,u...]. m For audio data, Mel-frequency spectrum and CNN or AudioBERT are used to generate audio feature sequences A=[a0,a1,...,a2]. k]; Based on this, cross-modal fusion specifically involves adding audio / video streams to the ViLBERT / LXMERT framework and achieving multimodal interaction through multi-stream attention. The general formula for attention calculation is as follows:

[0036] Where d is the hidden layer dimension, Q is the query, K is the key, and V is the value, which comes from different modalities or the same modality to achieve cross-modal attention; Cross-modal layer update formula: Out = LayerNorm(Attention(Q,K,V) + Input).

[0037] For example: MultiModalAttention(T,U,A) = Attention(T,[U;A],[U;A]), which concatenates visual and audio features as keys and values, and uses text as the query.

[0038] S40. Relationship extraction; specifically including: taking the fused structured text as input, extracting relationships between multiple entities; and transforming the extracted relationships into entity pairs and relationship types. As one possible implementation, the T5 model is used to transform the extraction of relationships between entities into the task of generating entity pairs and relationship types.

[0039] As an example, we can identify n core relationships between entities, including the "leading" and "participating" relationship between organization and person, the "initiating" and "collaborating" relationship between organization and event, the "leading" and "passed away" relationship between person and event, and the "occurring" relationship between location and event.

[0040] The T5 model was used for relation extraction, which was transformed into a task of generating "entity pairs + relation types". 3000 sets of triples (entity 1-relation-entity 2) were manually labeled as training data.

[0041] S50. Design an ontology framework for a rescue information knowledge graph, transform the generation task into a triple, store the graph in a graph database, and output the graph in a preset format.

[0042] As an example, an ontology framework for emergency medical services is designed based on extracted entities and relationships. The top-level concept is "Emergency Medical Element," which includes "Subject" (person, organization), "Object" (medical resources), "Scene" (location, event), and "Time Dimension." Attributes are added to entities, such as "Place of Origin" and "Emergency Medical Expertise" for "Person," and "Establishment Time" and "Location" for "Organization." The extracted entities and relationships are transformed into triples (subject-verb-object). The Neo4j graph database is used to store the graph, containing several entities (including organizational and person entities) and several triples. The graph is visualized using the Neo4j Bloom tool, allowing analysis of organizational collaboration networks, person movement trajectories, and event timelines. The graph reasoning function is used to fill in missing historical data, and the authenticity of the reasoning results is verified by consulting relevant literature.

[0043] The knowledge graph constructed in this invention integrates scattered historical materials on rescue work from different periods. Through the association of entities and relationships, it intuitively presents the overall picture of rescue work during historical periods, providing systematic knowledge support for historical research. By combining multimodal data such as historical photographs and audio materials, it develops knowledge reasoning functions to achieve deeper exploration and application of rescue work during historical periods.

[0044] Secondly, this invention provides a rescue information knowledge graph construction system based on a large language model, comprising: The data processing module is used for multimodal data collection and preprocessing; multimodal data includes text data, image data, audio data, and video data; preprocessing includes data cleaning, format standardization, word segmentation, and part-of-speech tagging; The knowledge extraction module defines multiple types of entities and extracts entity information from preprocessed text data, image data, audio data, and video data to obtain multiple single-modal feature sequences. The knowledge fusion module performs cross-modal fusion on multiple single-modal feature sequences to obtain fused structured text. The knowledge modeling module takes the fused structured text as input and extracts the relationships between multiple entities; it then transforms the extracted relationships into entity pairs and relationship types. The knowledge storage module designs an ontology framework for a rescue information knowledge graph, transforms the generated tasks into triples, stores the graph in a graph database, and outputs the graph in a preset format.

[0045] Thirdly, the present invention provides a terminal, comprising: The memory is used to store the program for constructing a knowledge graph of ambulance information based on a large language model; The processor is used to implement the steps of the rescue information knowledge graph construction method based on the large language model provided in the first aspect when executing the rescue information knowledge graph construction program based on the large language model.

[0046] The knowledge graph constructed in this invention is characterized by its visualization and reasoning capabilities, enabling researchers to discover historical patterns, fill in missing historical materials, and provide new tools and methods for the study of rescue history. The methods and systems of this invention can serve as a reference for the construction of knowledge graphs for other historical periods and have certain promotional value.

[0047] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and other materials. In this specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several functions listed in the specification. While certain measures are described in different embodiments, this does not mean that these measures cannot be combined to produce good results.

[0048] Although the invention has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made therein without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely illustrative of the invention and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if such modifications and modifications fall within the scope of the invention and its equivalents, the invention is also intended to include such modifications and modifications.

Claims

1. A method for constructing an ambulance information knowledge graph based on a large language model, characterized in that, Comprising the following steps: S10. Multimodal data collection, preprocessing; the multimodal data comprises text data, image data, audio data and video data; the preprocessing comprises data cleaning, format standardization, word segmentation and part-of-speech tagging; S20. New knowledge confidence discrimination: based on the knowledge database established by prior knowledge, the confidence of the new knowledge is evaluated, the new knowledge higher than the threshold value is listed in the knowledge graph for subsequent processing, and the new knowledge lower than the threshold value is identified as untrusted and discarded, the confidence evaluation is obtained by weighted summation of data source, data recording time, cross-validation result, data integrity and expert score; S30. Knowledge extraction and fusion; specifically comprising: defining multiple entities, taking the preprocessed text data, image data, audio data and video data as basic data, respectively performing entity information extraction to obtain multiple single-modal feature sequences; performing cross-modal fusion on the multiple single-modal feature sequences to obtain fused structured text; S40. Relationship extraction; specifically comprising: taking the fused structured text as input, extracting the relationship between multiple entities; converting the entity relationship extraction into an entity pair and relationship type generation task; S50. Designing a rescue information knowledge graph ontology framework, converting the generation task into a triple, storing the graph in a graph database, and outputting the graph in a preset format.

2. The method of claim 1, wherein the method is performed by a computer system. The entity information extraction specifically comprises: For text data, a BERT-WWM-Chinese model is used for text entity information extraction; For image data and video data, a Transformer-based model is used to identify the behavior entity information and / or event entity information of the characters and / or objects in the image and / or video; in the case of subtitles or voiceovers in video data, the subtitle text is extracted using an optical character recognition method, and then a BERT-WWM-Chinese model is used for text entity information extraction; For audio data, a speech recognition method is used to extract audio text, and the visual and auditory content of the preset characters and preset events in the preset event interval is associated with the text semantics through cross-modal association. 3.The method of claim 1, wherein, The multiple single-modal feature sequences are fused to obtain fused structured text, specifically comprising: Pre-training a cross-modal information fusion model based on the Transformer architecture; Applying the cross-modal information fusion model based on the Transformer architecture to bind the text feature sequence, image feature sequence, audio feature sequence and video feature sequence to a common expression space, and using the image as the pivot modality to realize retrieval and generation between any modalities.

4. The method of claim 3, wherein the method is performed by a computer system. When pre-training the cross-modal information fusion model based on the Transformer architecture, configure the initial training samples, and set the confidence of the initial training samples to 1, and continuously supplement the entity information obtained after knowledge extraction to the initial training samples to continuously update the initial training samples; When updating the initial training samples, first calculate the new knowledge confidence, and update the entity information with a new knowledge confidence greater than or equal to a preset threshold to the initial training samples.

5. The method of claim 4, wherein the method further comprises: The new knowledge confidence is denoted as Y, and is calculated as follows: Y=[X,Z,L,M,N] W=Xx0.25+Zx0.15+Lx0.35+Mx0.10+Nx0.15 Wherein, X is data source, Z is data recording time, L is one-time cross-validation, M is data integrity, N is expert score, and W is weight value.

6. The method of claim 3, wherein the method further comprises: The cross-modal fusion includes self-attention within the same modal and cross-attention between different modes to support the fusion of image data and text data, and is extended to the cross-modal fusion of audio data and video data; for the cross-modal fusion of text data, audio data and video data, each modal data is first encoded into a feature sequence, and then the cross-attention mechanism of the Transformer is reused to realize the fusion.

7. The method of claim 6, wherein the method further comprises: For text data, the BERT is used to encode the text sequence, and the output text feature sequence T = [t0, t1,..., t n ] is obtained; for image data, the Faster R-CNN is used to extract the target region features in the image, and the output visual feature sequence U = [u0, u1,..., u m ] is obtained; for audio data, the mel spectrum and CNN or AudioBERT are used to generate the audio feature sequence A = [a0, a1,..., a k ] Based on this, the cross-modal fusion is specifically: adding audio / video stream in the ViLBERT / LXMERT framework, realizing multi-modal interaction through multi-stream attention, and the general formula for attention calculation is: Wherein d is the hidden layer dimension, Q is the query, K is the key, V is the value, which comes from different modes or the same mode, and the cross-modal attention is realized; The cross-modal layer update formula is: Out=LayerNorm(Attention(Q,K,V)+Input). 8.The method of claim 1, wherein, The T5 model is adopted to convert the entity relationship extraction into the generation task of entity pair and relationship type. 9.A system for constructing an ambulance information knowledge graph based on a large language model, characterized in that, It includes: A data processing module for collecting and preprocessing multi-modal data; the multi-modal data includes text data, image data, audio data and video data; The preprocessing includes data cleaning, format standardization, word segmentation and part-of-speech tagging; A knowledge extraction module defines multiple entities, and takes the preprocessed text data, image data, audio data and video data as basic data to extract entity information respectively to obtain multiple single-modal feature sequences; A knowledge fusion module performs cross-modal fusion on the multiple single-modal feature sequences to obtain fused structured text; A knowledge modeling module takes the fused structured text as input to extract multiple inter-entity relationships; converts the entity relationship extraction into the generation task of entity pair and relationship type; A knowledge storage module designs a rescue information knowledge graph ontology framework, converts the generation task into a triple, stores the graph in a graph database, and outputs the graph in a preset format.

10. A terminal, characterized by comprising: It includes: A memory for storing a large language model-based rescue information knowledge graph construction program; A processor for executing the large language model-based rescue information knowledge graph construction program to realize the steps of the large language model-based rescue information knowledge graph construction method according to any one of claims 1-8.

Images