Multimodal knowledge graph application system and computer readable medium

Abstract

Embodiments of the present disclosure disclose a multi-modal knowledge graph application system and a computer readable medium. A specific embodiment of the system comprises: a processor configured to perform the following steps: generating individual thyroid triple data; performing data conflict detection processing on the individual thyroid triple data to obtain individual conflict detection results; determining data modification method information; performing data update processing on the thyroid knowledge graph; determining thyroid execution information; obtaining task execution results; and a data transmitter configured to send the task execution results to a user terminal. This embodiment can reduce the waste of sending resources of the multi-modal knowledge graph application system.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and more specifically to multimodal knowledge graph application systems and computer-readable media. Background Technology

[0002] Multimodal knowledge graph applications provide a method for performing corresponding application tasks using pre-built multimodal knowledge graphs. Currently, the typical approach for completing application tasks using multimodal knowledge graph application systems is as follows: First, features are extracted from the user-input data. Then, the obtained feature vectors are matched with entities in the multimodal knowledge graph. Finally, the entities in the multimodal knowledge graph corresponding to the user-input data are sent to the user terminal.

[0003] However, when using the aforementioned multimodal knowledge graph application system to perform application tasks, the following technical problems often arise:

[0004] Waste of system resources. Data in multimodal knowledge graphs is time-sensitive. Directly matching multimodal knowledge graphs with user input data can easily lead to low reliability of the entities obtained after matching, resulting in wasted transmission resources when the system sends these less reliable entities to the user terminal. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure propose a multimodal knowledge graph application system and a computer-readable medium to address one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a multimodal knowledge graph application system, the system comprising: a processor configured to perform the following steps: generating thyroid triplet data based on thyroid unstructured data and thyroid image data received from a data collector; performing data conflict detection processing on the thyroid triplet data based on a thyroid knowledge graph to obtain conflict detection results; for each conflict detection result, in response to determining that the conflict detection result satisfies preset conflict detection conditions, determining data modification method information based on the thyroid triplet data corresponding to the conflict detection result; performing data update processing on the thyroid knowledge graph based on the determined data modification method information and the thyroid triplet data; determining thyroid execution information based on thyroid task information sent by a user terminal; executing a knowledge graph application task corresponding to the thyroid execution information based on the updated thyroid knowledge graph and the thyroid task information to obtain a task execution result; and a data transmitter configured to send the task execution result to the user terminal.

[0008] Secondly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the system described in any of the implementations of the first aspect above.

[0009] The above embodiments of this disclosure have the following beneficial effects: The multimodal knowledge graph application system of some embodiments of this disclosure can reduce the waste of transmission resources. Specifically, the reason for the waste of transmission resources is the waste of system resources. Data in the multimodal knowledge graph has time sensitivity; directly matching the multimodal knowledge graph with user-input data easily leads to low reliability of the entities obtained after matching, resulting in wasted transmission resources when the system sends entities with low reliability to the user terminal. Based on this, the multimodal knowledge graph application system of some embodiments of this disclosure first configures the processor to perform the following steps: First, based on the unstructured thyroid data and thyroid image data received from the data collector, generate thyroid triplet data. This yields new thyroid triplet data. Second, based on the thyroid knowledge graph, perform data conflict detection processing on the aforementioned thyroid triplet data to obtain conflict detection results. This determines whether the newly generated thyroid triplet data conflicts with the original thyroid knowledge graph. Then, for each conflict detection result, in response to determining that the conflict detection result meets the preset conflict detection conditions, data modification method information is determined based on the thyroid triplet data corresponding to the conflict detection result. Thus, the method requiring modification of the thyroid knowledge graph can be obtained. Then, based on the determined data modification method information and the aforementioned thyroid triplet data, the thyroid knowledge graph is updated. Thus, the updated thyroid knowledge graph is obtained. Then, based on the thyroid task information sent by the user terminal, thyroid execution information is determined. Thus, the thyroid execution information is obtained. Then, based on the updated thyroid knowledge graph and the aforementioned thyroid task information, the knowledge graph application task corresponding to the aforementioned thyroid execution information is executed, and the task execution result is obtained. Thus, the knowledge graph application task can be executed, and the task execution result is obtained. Finally, the data transmitter is configured to send the aforementioned task execution result to the aforementioned user terminal. Because before using the thyroid knowledge graph to perform corresponding knowledge graph application tasks, new thyroid triple data can be generated based on the updated data in the data source, and conflict detection can be performed on the thyroid knowledge graph using the new thyroid triple data, thereby determining the accuracy of the data in the thyroid knowledge graph and updating the thyroid knowledge graph. This can improve the reliability of each entity obtained after matching the user input data, thereby reducing the probability of the system sending less reliable entities to the user terminal and reducing the waste of sending resources. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0011] Figure 1 This is a schematic diagram of the structure of some embodiments of the multimodal knowledge graph application system according to the present disclosure. Detailed Implementation

[0012] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0013] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0014] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0015] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0016] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0017] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] Figure 1 A schematic diagram of the structure of some embodiments of a multimodal knowledge graph application system according to the present disclosure is shown. Figure 1 It includes a processor 1 and a data transmitter 2. The processor 1 can be a device capable of processing data. The processor 1 can be, but is not limited to, a central processing unit (CPU). The data transmitter 2 can be a device capable of transmitting data. For example, the data transmitter 2 can be, but is not limited to, a network interface card.

[0019] In some embodiments, the processor 1 described above can be configured to perform the following steps:

[0020] First, based on the unstructured thyroid data and image data of each thyroid gland received from the data collector, each thyroid triplet data is generated.

[0021] In some embodiments, the processor 1 can generate thyroid triplet data based on the thyroid unstructured data and thyroid image data received from the data acquisition device. The data acquisition device can be a device capable of acquiring data. For example, the data acquisition device can be, but is not limited to, a data acquisition card.

[0022] Each of the aforementioned unstructured thyroid data points can be unstructured text data used to characterize the effects of various substances on the thyroid gland and thyroid diseases. For example, unstructured thyroid data could be "Iodine-131 damages thyroid tissue and reduces the production of thyroid hormones." The aforementioned thyroid diseases can be diseases that affect the structure or function of the thyroid gland. For example, the aforementioned thyroid diseases can be, but are not limited to: thyroiditis, thyroid nodules, and thyroid cancer. Each of the aforementioned thyroid image data points can be an X-ray of the thyroid gland of a target user. The aforementioned target user can be a user suffering from a thyroid disease.

[0023] Each of the thyroid triplets in the above-mentioned thyroid triplet data can be a triplet generated from the above-mentioned unstructured thyroid data and the above-mentioned thyroid image data.

[0024] In the process of adopting technical solutions to address the aforementioned technical problems, the following issues often arise:

[0025] The system wastes computational resources. Directly matching user-input data with knowledge graphs can easily lead to low reliability of the entities obtained after matching. This forces the system to consume computational resources to perform multiple matches on the user-input data to improve the reliability of the obtained entities, resulting in a waste of computational resources during multiple matches.

[0026] In response to the aforementioned technical problems, the following solution was adopted:

[0027] In some optional implementations of some embodiments, the processor 1 described above can be further configured to:

[0028] The first step, for each thyroid unstructured data point in the above-mentioned thyroid unstructured data, is to perform the following steps:

[0029] The first sub-step involves preprocessing the aforementioned unstructured thyroid data to obtain preprocessed unstructured data. This preprocessed unstructured data can be the already preprocessed unstructured thyroid data. In practice, the processor 1 can remove non-text symbols from the unstructured thyroid data using a data removal function, obtaining the removed unstructured thyroid data as the preprocessed unstructured data. This data removal function can be any function capable of removing non-text symbols from text data. For example, the data removal function could be the `re.sub()` function in regular expressions. The non-text symbols can be symbols unrelated to the text data. These non-text symbols can include, but are not limited to, ":". "、Well mark".

[0030] The second sub-step involves segmenting the preprocessed unstructured data into sentences to obtain a thyroid sentence data sequence. Each thyroid sentence in this sequence can be a sentence from the preprocessed unstructured data. In practice, the processor 1 can segment the preprocessed unstructured data into sentences using a segmentation function based on preset characters to obtain individual thyroid sentence data. This segmentation function can be any function capable of segmenting text data into sentences based on specified characters. For example, the segmentation function could be the `split()` function. The preset characters can be characters pre-specified by the technician. For example, the preset character could be a period. Then, the individual thyroid sentence data can be arranged into a thyroid sentence data sequence according to their order within the preprocessed unstructured data.

[0031] The third sub-step involves performing entity extraction processing on each thyroid segment data in the above thyroid segment data sequence to obtain individual thyroid entity data.

[0032] Each thyroid entity data point in the aforementioned thyroid entity data set can be an entity extracted from the aforementioned thyroid clause data. Each thyroid entity data point corresponds to an entity category label, an entity start position, and an entity end position. The aforementioned entities can be things or concepts with explicit names and specific attributes. The aforementioned entities can include, but are not limited to: personal names, place names, organization names, and proper nouns.

[0033] In practice, firstly, the processor 1 can input the thyroid segment data into a pre-trained entity recognition model to obtain the thyroid entity recognition data corresponding to the thyroid segment data.

[0034] Among them, the above entity recognition model can be a neural network model that takes thyroid clause data as input and outputs the corresponding thyroid entity recognition data for each thyroid clause data. For example, the above entity recognition model can be a pre-trained BERT model (Bidirectional Encoder Representations from Transformers). The above pre-training can be a process of fine-tuning the BERT model using the labeled thyroid clause dataset and the cross-entropy loss function. Each thyroid entity recognition data in the above various thyroid entity recognition data can be the data obtained after recognizing the entities in the above thyroid clause data. Each thyroid entity recognition data in the above various thyroid entity recognition data can include an entity name, an entity start position, an entity end position, and an entity category label. The above entity name can be the name of the entity. For example, the above entity name can be "thyroid". The above entity category label can be a label used to represent the category to which the corresponding entity name belongs. The above entity category label can be, but is not limited to: organ, disease name, drug name. The above entity start position can be the index of the first character in the entity name in the thyroid clause data. The above entity end position can be the index of the last character in the entity name in the thyroid clause data. For example, when the thyroid clause data is "Zhang San has hyperthyroidism" and the corresponding entity name is "hyperthyroidism", since the first character "Jia" in the entity name is the 5th character in the thyroid clause data, the entity start position is 5. Since the last character "Kang" in the entity name is the 6th character in the thyroid clause data, the entity end position is 6.

[0035] Secondly, the various entity names included in the above various thyroid entity recognition data can be determined as various thyroid entity data.

[0036] Fourth sub-step, for each thyroid clause data in the above thyroid clause data sequence, in response to determining that the above thyroid clause data meets the preset entity data condition, perform the following steps:

[0037] Sub-step one, determine the various thyroid entity data corresponding to the above thyroid clause data as various to-be-processed entity data. Among them, the above entity data condition can be that the entity quantity data corresponding to the thyroid clause data is greater than the preset entity quantity. The above entity quantity data can be the quantity of the various thyroid entity data corresponding to the thyroid clause data. The above entity quantity can be a preset value greater than 1. For example, the above entity quantity can be 2.

[0038] Sub-step two: For every two entity data to be processed in the above-mentioned entity data to be processed, based on the above-mentioned thyroid clause data, generate entity relationship data corresponding to the above two entity data to be processed.

[0039] The aforementioned entity relationship data can be labels used to characterize the relationship between two entity data to be processed. For example, the aforementioned entity relationship data can be "cause", "treat", or "alleviate".

[0040] In practice, for each pair of entity data to be processed in the aforementioned entity data sets, firstly, the processor 1 can identify the text data located between the two entity data to be processed in the thyroid segment data as the text to be processed. For example, when the thyroid segment data is "Aspirin can treat myocardial infarction," and the corresponding two entity data to be processed are "Aspirin" and "myocardial infarction," the corresponding text to be processed is "can treat." Secondly, in response to determining that the text to be processed meets the preset text conditions, the text to be processed can be input into a pre-trained relation extraction model to obtain entity relation data. The text conditions can be that the text to be processed does not contain any entity data to be processed. The relation extraction model can be a neural network model that takes the text to be processed as input and the entity relation data corresponding to the text to be processed as output. For example, the relation extraction model can be a pre-trained BERT model. The pre-training can be a process of fine-tuning the BERT model using an annotated set of text to be processed and a cross-entropy loss function.

[0041] Sub-step three involves generating entity triplet data sets based on the aforementioned entity data to be processed and the generated entity relation data. Each entity triplet in the aforementioned entity triplet data set can be a triplet consisting of two entity data sets to be processed and one entity relation data set.

[0042] In practice, for each entity relationship data in the aforementioned entity relationship data, firstly, the processor 1 can determine the two entity data to be processed corresponding to the entity relationship data as two target entity data to be processed. Secondly, the target entity data with a smaller starting position among the two target entity data to be processed can be determined as the first entity data. Then, the target entity data with a larger starting position among the two target entity data to be processed can be determined as the second entity data. Then, the first entity data, the entity relationship data, and the second entity data can be combined into a triple as entity triple data according to a preset order. The preset order can be the order in which the first entity data, entity relationship data, and second entity data are combined. For example, the preset order can be {first entity data, entity relationship data, second entity data}.

[0043] The fifth sub-step involves defining the generated entity triplet data sets as entity triplet data sets.

[0044] The second step is to generate thyroid triplet data based on the determined sets of entity triplet data and the aforementioned thyroid image data.

[0045] The above technical solution and its related content, combined with the content executed by the processor 1 and the data transmitter 2, serve as an inventive point of this disclosure, solving the problem of "waste of system computing resources." Factors leading to wasted system computing resources often include: Directly using knowledge graphs to match user-input data can easily result in low reliability of the entities obtained after matching, causing the system to consume computing resources to perform multiple matches on the user-input data to improve the reliability of the obtained entities, thus wasting computing resources during multiple matches. Solving these factors can reduce the waste of system computing resources. To achieve this effect, this disclosure first performs the following steps for each of the unstructured thyroid data: First, preprocessing the unstructured thyroid data to obtain preprocessed unstructured data. This yields preprocessed unstructured data. Second, segmenting the preprocessed unstructured data into sentences to obtain a thyroid sentence data sequence. This yields a thyroid sentence data sequence. Then, for each thyroid clause in the above thyroid clause data sequence, entity extraction processing is performed to obtain individual thyroid entity data. Thus, individual thyroid entity data can be obtained. Then, for each thyroid clause in the above thyroid clause data sequence, in response to determining that the thyroid clause data meets preset entity data conditions, the following steps are performed: Then, each thyroid entity data corresponding to the above thyroid clause data is determined as individual entity data to be processed. Thus, individual entity data to be processed can be obtained. Then, for every two individual entity data to be processed, entity relation data corresponding to the two individual entity data to be processed is generated based on the above thyroid clause data. Thus, entity relation data can be obtained. Then, based on the above individual entity data to be processed and the generated entity relation data, entity triplet data sets are generated. Thus, entity triplet data sets are obtained. Then, the generated entity triplet data sets are determined as entity triplet data set sets. Finally, based on the determined entity triplet data set sets and the above thyroid image data, individual thyroid triplet data are generated. Therefore, the data for each thyroid triplet can be obtained. Because data can be collected from the data source to generate each thyroid triplet before executing the corresponding knowledge graph application task using the multimodal knowledge graph application system, and then the thyroid knowledge graph can be updated using the generated thyroid triplet data, instead of directly using the knowledge graph to execute the corresponding knowledge graph application task, the reliability of the entities obtained after matching the user input data can be improved. This reduces the probability of the system needing to consume computational resources to perform multiple matches on the user input data, thus reducing the waste of computational resources.

[0046] In the process of adopting technical solutions to address the aforementioned technical problems, the following issues often arise:

[0047] Waste of system computing resources. Image data in knowledge graphs is prone to loss, which can easily lead to blank or garbled image data when directly matching user input data using the knowledge graph, resulting in wasted computing resources when the system calls upon computing resources for matching.

[0048] In response to the aforementioned technical problems, the following solution was adopted:

[0049] In some optional implementations of some embodiments, the processor 1 described above is further configured to:

[0050] The first step, for each thyroid imaging data point, is to perform the following steps:

[0051] The first sub-step involves performing image recognition processing on the aforementioned thyroid image data to obtain the bounding box information for each image. Each bounding box information can be an array consisting of the coordinates of its top-left and bottom-right corners. For example, when the coordinates of the top-left corner of an image bounding box are (1,4) and the coordinates of its bottom-right corner are (3,2), the corresponding bounding box information could be (1,4,3,2).

[0052] The aforementioned image bounding box can be the bounding box corresponding to the region of interest (ROI) of the thyroid gland. The aforementioned ROI of the thyroid gland can be a region of interest identified from the aforementioned thyroid image data. For example, the aforementioned ROI of the thyroid gland can be the image region containing the organ or the image region containing the nodule.

[0053] In practice, the processor 1 can input the thyroid image data into the target detection network of a pre-trained image entity recognition model to obtain the bounding box information of each image. The image entity recognition model can be a neural network model that takes the thyroid image data as input and outputs the corresponding image entity data and the feature information of each image region. The image entity recognition model can include three layers.

[0054] The first layer can be an object detection network. This object detection network can be a neural network that takes thyroid image data as input and outputs the bounding box information corresponding to each image in the thyroid image data. For example, the object detection network could be Faster R-CNN (Faster Region-based Convolutional Neural Networks).

[0055] The second layer can be an image processing layer. This image processing layer can include an image processing network and an image entity recognition network.

[0056] The aforementioned image processing network can be a neural network model that takes thyroid image data and bounding box information as input, and outputs image description information and image region feature information corresponding to each bounding box. For example, the aforementioned image processing network can be a large-scale visual language model. The image description information can be text data used to describe the image region corresponding to the bounding box information. For example, the image description information could be "There is a round nodule in the upper left of the thyroid gland." The image region feature information can be the feature vector of the image region corresponding to the bounding box information.

[0057] The aforementioned image entity recognition network can be a neural network model that takes the image description information output by the aforementioned image processing network as input and outputs the image entity data corresponding to each image description information. For example, the aforementioned image entity recognition network can be the BioBERT model (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining). The aforementioned image entity data can be entities identified from the image description information. The aforementioned image entity data corresponds to entity label information. The aforementioned entity label information can be a label used to characterize the category to which the corresponding image entity data belongs. The aforementioned entity label information can be, but is not limited to: lesion, anatomical location. Each image entity data in each image entity data corresponding to each of the aforementioned image description information corresponds to a region of interest for the thyroid gland.

[0058] The second sub-step involves generating image entity data and image region feature information for each image bounding box based on the aforementioned image bounding box information and the aforementioned thyroid image data.

[0059] In practice, for each image bounding box, firstly, the processor 1 can input the image bounding box information and the thyroid image data into the image processing network of the image processing layer in the image entity recognition model to obtain the image description information and image region feature information corresponding to the image bounding box information. Secondly, the image description information can be input into the image entity recognition network of the image processing layer in the image entity recognition model to obtain image entity data.

[0060] The third sub-step involves determining the image entity data group and the image region feature information group based on the generated image entity data and the generated image region feature information.

[0061] In practice, the processor 1 can determine the aforementioned image entity data into an image entity data group, and can determine the aforementioned image region feature information into an image region feature information group.

[0062] Fourth, for each entity data to be processed in the above-mentioned entity triplet data set, perform the following steps:

[0063] The first sub-step involves performing feature extraction on the aforementioned entity data to obtain entity feature information. This entity feature information can be the feature vector corresponding to the entity data to be processed.

[0064] In practice, the processor 1 described above can perform feature extraction processing on the entity data to be processed using a pre-trained feature extraction model to obtain entity feature information. The feature extraction model can be a neural network model that takes the entity data to be processed as input and outputs the entity feature information corresponding to the entity data. For example, the feature extraction model can be a pre-trained convolutional neural network. Pre-training can be a process of fine-tuning the convolutional neural network using a feature training dataset and a cross-entropy loss function. Each feature training data in the feature training dataset can be data used for pre-training the convolutional neural network. Each feature training data in the feature training dataset can include entity data to be processed and entity image data to be processed. The entity image data to be processed can be an image corresponding to the entity data to be processed. For example, when the entity data to be processed is "apple," the corresponding entity image data to be processed can be an image of an apple. Through pre-training, the cosine similarity between the feature vector of the entity data to be processed generated by the feature extraction model and the feature vector of the corresponding entity image data to be processed can be greater than a preset threshold. The preset threshold can be a pre-set value. Here, the specific setting of the preset threshold is not limited.

[0065] The second sub-step involves identifying the image entity data corresponding to the entity data to be processed within each determined image entity data group as the image entity data to be processed. In practice, the processor 1 can identify the image entity data in each of the image entity data groups that is identical to the entity data to be processed as the image entity data to be processed.

[0066] The third sub-step involves determining the image region feature information corresponding to the aforementioned image entity data in each determined image region feature information group as the target image feature information.

[0067] The fourth sub-step involves comparing the similarity between the entity feature information and the target image feature information to obtain a similarity comparison result. This similarity comparison result can be the cosine similarity between the entity feature information and the target image feature information. In practice, the processor 1 can determine the cosine similarity between the entity feature information and the target image feature information as the similarity comparison result.

[0068] The fifth sub-step, in response to determining that the similarity comparison result meets a preset similarity comparison condition, generates multimodal triplet data based on the entity data to be processed, the thyroid image data corresponding to the entity data to be processed, and the image bounding box information corresponding to the entity data to be processed. The similarity comparison condition can be that the similarity comparison result is greater than a preset similarity threshold. The similarity threshold can be a pre-set value. Here, the specific setting of the similarity threshold is not limited. The multimodal triplet data can be a triplet used to characterize the relationship between the entity data to be processed and the thyroid image data corresponding to the entity data to be processed.

[0069] In practice, firstly, the processor 1 can determine the thyroid image data and image bounding box information corresponding to the image entity data to be processed as the thyroid image data to be processed and the bounding box information to be processed, respectively. Secondly, the thyroid image data to be processed and the bounding box information to be processed can be input into an image cropping function to crop the image region corresponding to the bounding box information to be processed in the thyroid image data to be processed, obtaining the image region corresponding to the bounding box information to be processed as the thyroid region image data. Finally, the entity data to be processed, the preset connection relationship data, and the thyroid region image data can be combined into a triple as multimodal triple data according to a preset combination order.

[0070] The preset combination order can be the order in which the entity data to be processed, the connection relationship data, and the thyroid region image data are combined. For example, the preset combination order can be {entity data to be processed, connection relationship data, thyroid region image data}. The connection relationship data can be a label used to characterize the connection relationship between the entity data to be processed and the thyroid region image data. For example, the connection relationship data can be "correspondence".

[0071] Fifth, based on the aforementioned entity triplet datasets and the generated multimodal triplet datasets, generate individual thyroid triplet datasets. Each thyroid triplet dataset can be either an entity triplet or a multimodal triplet.

[0072] In practice, the processor 1 described above can identify each entity triplet in the aforementioned entity triplet dataset as a thyroid triplet. It can also identify each multimodal triplet in the aforementioned multimodal triplet dataset as a thyroid triplet. Thus, various thyroid triplet datasets can be generated.

[0073] The above technical solution and its related content, combined with the content executed by the processor 1 and the data transmitter 2, serve as an inventive point of this disclosure, solving the problem of "waste of system computing resources." Factors leading to wasted system computing resources often include: Image data in knowledge graphs is prone to loss, etc. When directly using the knowledge graph to match user-input data, the matched image data is easily blank or garbled, resulting in wasted computing resources when the system calls computing resources for matching. Solving these factors can reduce the waste of system computing resources. To achieve this effect, this disclosure first performs the following steps for each thyroid image data: Second, image recognition processing is performed on the thyroid image data to obtain bounding box information for each image. Thus, bounding box information for each image can be obtained. Then, for each bounding box information, image entity data and image region feature information are generated based on the bounding box information and the thyroid image data. Thus, image entity data and image region feature information are obtained. Then, based on the generated image entity data and the generated image region feature information, image entity data groups and image region feature information groups are determined. Therefore, image entity data sets and image region feature information sets can be obtained. Then, for each entity data set to be processed within each entity data set, the following steps are performed: Feature extraction is then performed on the entity data to be processed to obtain entity feature information. Then, the image entity data corresponding to the entity data to be processed in each determined image entity data set is identified as the image entity data to be processed. Then, the image region feature information corresponding to the image entity data to be processed in each determined image region feature information set is identified as the target image feature information. Then, a similarity comparison is performed between the entity feature information and the target image feature information to obtain a similarity comparison result. Then, in response to the determination that the similarity comparison result meets the preset similarity comparison conditions, multimodal triple data is generated based on the entity data to be processed, the thyroid image data corresponding to the image entity data to be processed, and the image bounding box information corresponding to the image entity data to be processed. Thus, multimodal triple data is obtained. Finally, based on the aforementioned entity triplet datasets and the generated multimodal triplet datasets, individual thyroid triplet datasets are generated. Thus, the individual thyroid triplet datasets can be obtained.Because image data can be collected from the data source to generate various multimodal triples before the multimodal knowledge graph application system is used to execute the corresponding knowledge graph application tasks, and then the image data stored in the thyroid knowledge graph is updated, the probability of the matched image data being blank or garbled can be reduced when using the knowledge graph to match the user input data, thereby reducing the waste of computing resources when the system calls computing resources for matching.

[0074] Second, based on the thyroid knowledge graph, data conflict detection processing was performed on the above-mentioned thyroid triplet data to obtain the conflict detection results.

[0075] In some embodiments, the processor 1 can perform data conflict detection processing on the thyroid triplet data based on the thyroid knowledge graph to obtain conflict detection results. The thyroid knowledge graph can be a knowledge graph pre-constructed using historical unstructured thyroid data and historical thyroid image data. The thyroid knowledge graph can include graph nodes and edges. Each graph node can be a node in the knowledge graph. Each graph node corresponds to node information. The node information can be the entity or image represented by the graph node. For example, the node information can be entities such as "hyperthyroidism" or "thyroid hormones," or previously generated thyroid region image data. Each edge corresponds to connection information. The connection information can be a label representing the relationship between two corresponding graph nodes. For example, the connection information can be "cause," "treat," "relieve," or "correspond."

[0076] Each of the aforementioned historical unstructured thyroid data sets can be previously acquired unstructured thyroid data. Similarly, each of the aforementioned historical thyroid image data sets can be previously acquired thyroid image data.

[0077] Each of the conflict detection results described above can be a label used to characterize whether there is a conflict between the thyroid triplet data and the thyroid knowledge graph. For example, the conflict detection results could be "conflict exists" or "no conflict exists".

[0078] In some optional implementations of some embodiments, the processor 1 described above is further configured to:

[0079] For each of the thyroid triplet data points above, perform the following steps:

[0080] The first step is to combine the first entity data and the second entity data included in the above thyroid triplet data into the data to be queried.

[0081] The second step is to identify the entity relationship data included in the above thyroid triplet data as the connection data to be compared.

[0082] The third step is to query the aforementioned data from the thyroid knowledge graph to obtain the data query results. These results can include all connection information that matches the content of the data being queried.

[0083] In practice, the processor 1 described above can use data query instructions to search for edges in the thyroid knowledge graph that have the same information as the first entity data and the second entity data included in the data to be queried, and use these edges as target edges. The data query instructions can be any instructions that can be queried within the knowledge graph. For example, the data query instructions can be SPARQL instructions. Furthermore, the connection information corresponding to each of the target edges can be determined as the data query results.

[0084] Fourth, in response to determining that the above data query results satisfy the preset data query conditions, based on the above data query results and the thyroid knowledge graph, determine each target connection data. The above data query conditions can be that the above data query results are not empty. Each target connection data can be the connection information corresponding to an edge in the thyroid knowledge graph. In practice, the processor 1 can determine each connection information included in the above data query results as each target connection data.

[0085] The fifth step involves comparing the target connectivity data and the connectivity data to be compared to obtain the comparison results. Each comparison result can be a label indicating whether there is a conflict between the target connectivity data and the connectivity data to be compared. For example, the comparison result could be "conflict exists" or "no conflict exists".

[0086] In practice, for each of the target connection data mentioned above, the processor 1 can identify two connection pieces that are identical to both the target connection data and the connection data to be compared from the connection information included in the preset data conflict correspondence table as two target connection pieces. Secondly, the conflict information corresponding to the two target connection pieces can be determined as the data comparison result. The data conflict correspondence table can be a table used to characterize whether there is a conflict between any two connection pieces in each connection information. Each pair of connection pieces corresponds to one conflict piece. The conflict information can be a label used to characterize whether the corresponding two connection pieces conflict. For example, the conflict information can be "conflict exists" or "no conflict exists".

[0087] Step 6: Based on the comparison results of each data set, the aforementioned thyroid triplet data, and the thyroid knowledge graph, determine the conflict detection results corresponding to the aforementioned thyroid triplet data.

[0088] In practice, in response to determining that the above data comparison results meet the preset data comparison conditions, "no conflict" can be determined as the conflict detection result. The above data comparison conditions can be that all the above data comparison results are "no conflict". In response to determining that the above data comparison results do not meet the above data comparison conditions, "conflict exists" can be determined as the conflict detection result.

[0089] Third, for each conflict detection result, in response to determining that the above conflict detection result meets the preset conflict detection conditions, the data modification method information is determined based on the thyroid triplet data corresponding to the above conflict detection result.

[0090] In some embodiments, for each conflict detection result, in response to determining that the conflict detection result meets a preset conflict detection condition, the processor 1 can determine data modification method information based on the thyroid triplet data corresponding to the conflict detection result. The data modification method information can be information characterizing the method required to modify the thyroid triplet data. The data modification method information can be "delete," "retain," or "replace." The conflict detection condition can be that the conflict detection result is "conflict exists."

[0091] In practice, for each conflict detection result, in response to determining that the conflict detection result meets the aforementioned conflict detection conditions, the thyroid triplet data corresponding to the conflict detection result can be sent to the target terminal. The target terminal can be the terminal corresponding to a technician. Secondly, in response to receiving modification method information sent by the target client, the modification method information can be identified as data modification method information. The modification method information can be information used to characterize the method required to modify the thyroid triplet data. The data modification method information can be "delete," "keep," or "replace."

[0092] Fourth, based on the determined data modification methods and the aforementioned thyroid triplet data, the thyroid knowledge graph is updated.

[0093] In some embodiments, the processor 1 can perform data update processing on the thyroid knowledge graph based on the determined data modification method information and the thyroid triplet data.

[0094] In practice, for each of the above data modification methods, the thyroid triplet data corresponding to the above data modification method information can first be identified as the triplet data to be modified.

[0095] Secondly, in response to the determination that the data modification method information is "delete", the triplet data to be modified can be deleted using a delete command. This delete command can be any command capable of deleting data. For example, the delete command can be a DELETE statement.

[0096] Then, in response to the determination that the above data modification method information is "retain", the above triplet data to be modified can be added to the thyroid knowledge graph using a knowledge graph addition statement to update the thyroid knowledge graph. The above knowledge graph addition statement can be an instruction that can add new nodes to the knowledge graph. For example, the above knowledge graph addition statement can be a MERGE statement.

[0097] Next, in response to the determination that the data modification method information is "replace," firstly, the triplet data to be modified can be added to the thyroid knowledge graph using the aforementioned knowledge graph add statement to update the thyroid knowledge graph. Secondly, the edges in the updated thyroid knowledge graph that correspond to the aforementioned data modification method information can be identified as edges to be modified. Then, the edges to be modified can be deleted using the aforementioned delete command to update the thyroid knowledge graph.

[0098] Fifth, based on the user terminal, send thyroid task information to determine the thyroid execution information.

[0099] In some embodiments, the processor 1 can determine thyroid task information by sending thyroid task information based on a user terminal. The user terminal can be a terminal corresponding to a user. The thyroid task information can be information sent by the user to prompt the multimodal knowledge graph application system to perform a corresponding knowledge graph application task. The thyroid task information corresponds to task type information. The task type information can be a tag representing the task type corresponding to the thyroid task information. For example, the task type information can be "question-answering task" or "recommendation information generation task".

[0100] When the task type information corresponding to the thyroid task information is "question-answering task," the aforementioned thyroid task information can be text data corresponding to questions related to the thyroid. For example, the aforementioned thyroid task information could be "Will iodine-131 damage thyroid tissue?".

[0101] When the task type information corresponding to the thyroid task information is "Recommendation Information Generation Task," the aforementioned thyroid task information can be data used to generate thyroid recommendation information. In this case, the aforementioned thyroid task information may include available value data, equipment information sets, technical information sets, and maximum cycle data. The aforementioned available value data can be the maximum value allowed to be spent (e.g., research budget) when researching thyroid recommendation information. Each piece of equipment information in the aforementioned equipment information set can be the name of the equipment required to research thyroid recommendation information. For example, the equipment information could be "centrifuge." Each piece of technical information in the aforementioned technical information set can be the technology required to research thyroid recommendation information. For example, the technical information could be "cell culture." The aforementioned maximum cycle data can be the longest allowed cycle when researching thyroid recommendation information. For example, the aforementioned maximum cycle data could be 60 days. The aforementioned thyroid recommendation information can be the research content when researching thyroid-related content. For example, the thyroid recommendation information could be "Can tyrosine stimulate thyroid hormones?"

[0102] The aforementioned knowledge graph application tasks can be tasks for the multimodal knowledge graph application system to complete corresponding functions based on the knowledge graph. For example, these knowledge graph application tasks may include, but are not limited to, question-answering tasks and recommendation information generation tasks. The question-answering task can be a task for the multimodal knowledge graph application system to answer information sent by a user terminal based on the knowledge graph. The recommendation information generation task can be a task for the multimodal knowledge graph application system to generate thyroid recommendation information based on the thyroid knowledge graph and information sent by the user terminal.

[0103] The aforementioned thyroid execution information can be information used to characterize the type of knowledge graph application task that the user needs to perform. For example, the aforementioned thyroid execution information could be a "question-answering task" or a "recommendation information generation task".

[0104] In some optional implementations of some embodiments, the processor 1 described above is further configured to:

[0105] The first step involves determining that the aforementioned thyroid task information meets a preset first execution condition, and then defining the preset first execution information as the thyroid task information. The first execution condition can be that the task type information corresponding to the thyroid task information is "question-answering task". The first execution information can be a label used to characterize that the thyroid task information belongs to a question-answering task. For example, the first execution information can be "question-answering task".

[0106] The second step involves determining that the aforementioned thyroid task information meets a preset second execution condition, and then identifying the preset second execution information as the thyroid task information. The second execution condition can be that the task type information corresponding to the thyroid task information is "recommendation information generation task". The second execution information can be a label used to characterize that the thyroid task information belongs to a recommendation information generation task. For example, the second execution information can be "recommendation information generation task".

[0107] Sixth, based on the updated thyroid knowledge graph and the aforementioned thyroid task information, execute the knowledge graph application task corresponding to the aforementioned thyroid execution information to obtain the task execution result.

[0108] In some embodiments, the processor 1 can execute a knowledge graph application task corresponding to the thyroid task information based on the updated thyroid knowledge graph and the thyroid task information, and obtain a task execution result. The task execution result can be data obtained after executing the corresponding knowledge graph application task.

[0109] In some optional implementations of some embodiments, the processor 1 described above is further configured to:

[0110] In response to the determination that the above thyroid execution information meets the preset question-and-answer task conditions, the following question-and-answer steps are performed:

[0111] The first step involves keyword recognition processing of the aforementioned thyroid task information to obtain individual thyroid keyword data. The question-and-answer task condition can be that the thyroid task is a "question-and-answer task." Each thyroid keyword data point can be an entity identified from the aforementioned thyroid task information. In practice, the processor 1 can first input the aforementioned thyroid task information into a pre-trained task information entity recognition model to obtain the individual thyroid keyword data corresponding to the aforementioned thyroid task information. The task information entity recognition model can be a neural network model that takes the thyroid task information as input and outputs the individual thyroid keyword data corresponding to the thyroid task information. For example, the task information entity recognition model can be a pre-trained BERT model. The pre-training process can be a fine-tuning of the BERT model using an annotated thyroid task information set and a cross-entropy loss function.

[0112] The second step involves generating graph entity data based on the updated thyroid knowledge graph and the aforementioned thyroid keyword data. Each graph entity data point can be the node information corresponding to a graph node in the thyroid knowledge graph that meets preset node conditions. These node conditions can be that the graph node belongs to either a first-type or a second-type graph node. First-type graph nodes can be graph nodes whose corresponding node information is identical to the thyroid keyword data. Second-type graph nodes can be graph nodes in the thyroid knowledge graph that are directly connected to first-type graph nodes.

[0113] In practice, the processor 1 can identify each graph node in the thyroid knowledge graph that meets the above-mentioned node conditions as a target graph node. Then, the node information corresponding to each target graph node can be identified as graph entity data.

[0114] The third step involves generating thyroid-related answer information based on the aforementioned graph entity data and pre-defined prompts. This thyroid-related answer information can be used to answer the aforementioned thyroid task information. The aforementioned prompts can be used to provide feedback on the output of the large language model. For example, the prompt could be, "Please refer to the aforementioned graph entity data to answer the aforementioned thyroid task information." The aforementioned large language model can be DeepSeek.

[0115] In practice, the processor 1 can input the graph entity data and preset prompts into the large language model to obtain the data output by the large language model as the thyroid answer information.

[0116] The fourth step is to determine the above thyroid-related answer information as the task execution result.

[0117] In some optional implementations of some embodiments, the processor 1 described above is further configured to:

[0118] In response to determining that the above thyroid performance information meets the preset recommended task conditions, the following recommended information generation steps are performed:

[0119] The first step is to generate various thyroid-related recommendation messages based on the updated thyroid knowledge graph. The recommendation task condition can be that the thyroid-related execution information is a "recommendation message generation task." Each thyroid-related recommendation message can be research content related to the thyroid, generated from the thyroid knowledge graph. For example, a thyroid-related recommendation message could be "Can tyrosine stimulate thyroid hormones?"

[0120] In practice, firstly, the processor 1 can send the aforementioned thyroid knowledge graph to the target terminal. Then, in response to receiving the various thyroid information items sent by the target terminal, each of the thyroid information items can be identified as a specific thyroid recommendation. Each of the aforementioned thyroid information items can be thyroid-related research content generated based on the thyroid knowledge graph.

[0121] The second step is to perform resource modeling on each thyroid recommendation information to obtain thyroid resource data.

[0122] The aforementioned thyroid resource data can be used for research on thyroid recommendation information. This thyroid resource data may include value data, information on various devices, technical information, and cycle data.

[0123] The aforementioned value data can represent the value (e.g., research cost) used in researching thyroid recommendations. Each piece of equipment information above can be the name of the equipment required for researching thyroid recommendations. For example, the equipment information could be "centrifuge." Each piece of technical information above can be the technique required for researching thyroid recommendations. For example, the technique information could be "cell culture." The aforementioned cycle data can represent the time required for researching thyroid recommendations. For example, the cycle data above could be 30 days.

[0124] In practice, the processor 1 can input the thyroid recommendation information into a pre-trained resource generation model to obtain thyroid resource data corresponding to the thyroid recommendation information.

[0125] The resource generation model described above can be used to characterize the relationship between thyroid recommendation information and thyroid resource data. As an example, the resource generation model may include a feature extraction part and a correspondence table. The feature extraction part can be used to extract feature vectors from the thyroid recommendation information. For example, the feature extraction part can be a Long Short-Term Memory (LSTM) network. The correspondence table can be a pre-defined table by technicians based on statistics of a large number of feature vectors and thyroid resource data, storing the correspondence between multiple feature vectors and thyroid resource data. Thus, for thyroid recommendation information, the resource generation model can first use the feature extraction part to extract features from the thyroid recommendation information, obtaining the feature vector of the thyroid recommendation information as the target feature vector. Then, the target feature vector is compared sequentially with multiple feature vectors in the correspondence table. If a feature vector in the correspondence table is the same as or similar to the target feature vector, then the thyroid resource data corresponding to that feature vector in the correspondence table is used as the thyroid resource data of the thyroid recommendation information.

[0126] The third step, for each piece of thyroid resource data obtained, is to perform the following steps:

[0127] The first sub-step involves generating technical indicator data, processing matching data, equipment matching data, technical matching data, and cycle matching data based on the aforementioned thyroid task information and thyroid resource data.

[0128] The aforementioned technical indicator data can be numerical values ​​used to characterize the difficulty of thyroid task information. The aforementioned processing matching degree data can be numerical values ​​used to characterize the degree of matching between the available value data included in the thyroid task information and the value data included in the thyroid resource data. The aforementioned equipment matching degree data can be numerical values ​​used to characterize the degree of matching between the equipment information set included in the thyroid task information and the individual equipment information included in the thyroid resource data. The aforementioned technology matching degree data can be numerical values ​​used to characterize the degree of matching between the technology information set included in the thyroid task information and the individual technology information included in the thyroid resource data. The aforementioned cycle matching degree data can be numerical values ​​used to characterize the degree of matching between the maximum cycle data included in the thyroid task information and the cycle data included in the thyroid resource data.

[0129] In practice, firstly, in response to determining that the value data included in the aforementioned thyroid resource data is greater than the usable value data included in the aforementioned thyroid task information, a preset first matching data can be determined as the processing matching degree data. Secondly, in response to determining that the value data included in the aforementioned thyroid resource data is less than or equal to the usable value data included in the aforementioned thyroid task information, a preset second matching data can be determined as the processing matching degree data. The aforementioned first matching data can be a pre-set value. Here, the specific setting of the aforementioned first matching data is not limited. For example, the aforementioned first matching data can be 0.7. The aforementioned second matching data can be a pre-set value. Here, the specific setting of the aforementioned second matching data is not limited. For example, the aforementioned second matching data can be 0.3.

[0130] Secondly, in response to determining that all the device information included in the aforementioned thyroid resource data belongs to the device information set included in the aforementioned thyroid task information, the aforementioned first matching data can be determined as device matching degree data. In response to determining that any one of the device information included in the aforementioned thyroid resource data does not belong to the device information set included in the aforementioned thyroid task information, the aforementioned second matching data can be determined as device matching degree data.

[0131] Then, in response to determining that all the technical information included in the aforementioned thyroid resource data belongs to the set of technical information included in the aforementioned thyroid task information, the aforementioned first matching data can be determined as technical matching degree data. In response to determining that any one of the technical information included in the aforementioned thyroid resource data does not belong to the set of technical information included in the aforementioned thyroid task information, the aforementioned second matching data can be determined as technical matching degree data.

[0132] Then, in response to determining that the periodic data included in the aforementioned thyroid resource data is less than or equal to the maximum periodic data included in the aforementioned thyroid task information, the aforementioned first matching data can be determined as period matching degree data. In response to determining that the periodic data included in the aforementioned thyroid resource data is greater than the maximum periodic data included in the aforementioned thyroid task information, the aforementioned second matching data can be determined as period matching degree data.

[0133] Finally, the sum of the above-mentioned processing matching data, the above-mentioned equipment matching data, the above-mentioned technology matching data, and the above-mentioned periodic matching data can be determined as the technical indicator data.

[0134] The second sub-step involves weighting the aforementioned technical matching degree data and technical indicator data to obtain first feasibility data. This first feasibility data can be a numerical value derived from the aforementioned technical matching degree data and technical indicator data, used to characterize the feasibility of the corresponding thyroid recommendation information.

[0135] In practice, firstly, the processor 1 can determine the technical weight data by multiplying the preset first coefficient data and the technical matching degree data. Secondly, it can determine the technical indicator weight data by multiplying the preset second coefficient data and the technical indicator data. Finally, it can determine the first feasibility data by summing the technical weight data and the technical indicator weight data. The first coefficient data can be a positive number less than a preset value. For example, the first coefficient data can be 0.6. The second coefficient data can be a positive number less than the preset value. For example, the second coefficient data can be 0.4. The sum of the first coefficient data and the second coefficient data can be the preset value. The preset value can be 1.

[0136] The third sub-step involves weighting the above-mentioned matching data and device matching data to obtain the second feasibility data.

[0137] The second feasibility data mentioned above can be a numerical value used to characterize the feasibility of the corresponding thyroid recommendation information, obtained from the processing matching data and the device matching data mentioned above.

[0138] In practice, firstly, the processor 1 can determine the processing weight data by multiplying the preset third coefficient data and the processing matching degree data. Secondly, it can determine the device weight data by multiplying the preset fourth coefficient data and the device matching degree data. Finally, it can determine the second feasibility data by summing the processing weight data and the device weight data. The third coefficient data can be a positive number less than the preset value. For example, the third coefficient data can be 0.7. The fourth coefficient data can be a positive number less than the preset value. For example, the fourth coefficient data can be 0.3. The sum of the third coefficient data and the fourth coefficient data can be the preset value.

[0139] The fourth sub-step involves weighting the aforementioned first feasibility data, second feasibility data, and cycle matching degree data to obtain a feasibility score. This feasibility score can be a numerical value used to characterize the feasibility of the thyroid recommendation information.

[0140] The fourth step involves determining the target thyroid gland recommendation based on the obtained feasibility scores and the aforementioned thyroid gland recommendations. The target thyroid gland recommendation is the one with the highest corresponding feasibility score.

[0141] In practice, firstly, the processor 1 can determine the feasibility score data with the largest value among the aforementioned feasibility score data as the target feasibility score data. Secondly, it can determine the thyroid recommendation information corresponding to the target feasibility score data among the aforementioned thyroid recommendation information as the target thyroid recommendation information.

[0142] The fifth step is to determine the above-mentioned target thyroid recommendation information as the task execution result.

[0143] In some optional implementations of some embodiments, the processor 1 described above is further configured to:

[0144] The first step is to determine the first weighted data by multiplying the preset first parameter data by the aforementioned first feasibility data. The aforementioned first parameter data can be a pre-defined decimal.

[0145] The second step is to determine the second weighted data by multiplying the preset second parameter data by the aforementioned second feasibility data. The aforementioned second parameter data can be a pre-defined decimal.

[0146] The third step involves determining the third weighted data by multiplying the preset third parameter data by the aforementioned periodic matching degree data. The third parameter data can be a pre-defined decimal. The sum of the first, second, and third parameter data can be a preset total value. This preset total value can be a pre-defined value; for example, it can be 1.

[0147] The fourth step is to determine the feasibility score data by summing the first weighted data, the second weighted data, and the third weighted data.

[0148] In some embodiments, the data transmitter 2 can be configured to send the task execution result to the user terminal.

[0149] The above embodiments of this disclosure have the following beneficial effects: The multimodal knowledge graph application system of some embodiments of this disclosure can reduce the waste of transmission resources. Specifically, the reason for the waste of transmission resources is the waste of system resources. Data in the multimodal knowledge graph has time sensitivity; directly matching the multimodal knowledge graph with user-input data easily leads to low reliability of the entities obtained after matching, resulting in wasted transmission resources when the system sends entities with low reliability to the user terminal. Based on this, the multimodal knowledge graph application system of some embodiments of this disclosure first configures the processor to perform the following steps: First, based on the unstructured thyroid data and thyroid image data received from the data collector, generate thyroid triplet data. This yields new thyroid triplet data. Second, based on the thyroid knowledge graph, perform data conflict detection processing on the aforementioned thyroid triplet data to obtain conflict detection results. This determines whether the newly generated thyroid triplet data conflicts with the original thyroid knowledge graph. Then, for each conflict detection result, in response to determining that the conflict detection result meets the preset conflict detection conditions, data modification method information is determined based on the thyroid triplet data corresponding to the conflict detection result. Thus, the method requiring modification of the thyroid knowledge graph can be obtained. Then, based on the determined data modification method information and the aforementioned thyroid triplet data, the thyroid knowledge graph is updated. Thus, the updated thyroid knowledge graph is obtained. Then, based on the thyroid task information sent by the user terminal, thyroid execution information is determined. Thus, the thyroid execution information is obtained. Then, based on the updated thyroid knowledge graph and the aforementioned thyroid task information, the knowledge graph application task corresponding to the aforementioned thyroid execution information is executed, and the task execution result is obtained. Thus, the knowledge graph application task can be executed, and the task execution result is obtained. Finally, the data transmitter is configured to send the aforementioned task execution result to the aforementioned user terminal. Because before using the thyroid knowledge graph to perform corresponding knowledge graph application tasks, new thyroid triple data can be generated based on the updated data in the data source, and conflict detection can be performed on the thyroid knowledge graph using the new thyroid triple data, thereby determining the accuracy of the data in the thyroid knowledge graph and updating the thyroid knowledge graph. This can improve the reliability of each entity obtained after matching the user input data, thereby reducing the probability of the system sending less reliable entities to the user terminal and reducing the waste of sending resources.

[0150] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0151] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0152] The aforementioned computer-readable medium may be included in the aforementioned multimodal knowledge graph application system; or it may exist independently and not be assembled into the multimodal knowledge graph application system. The aforementioned computer-readable medium carries one or more programs. When the one or more programs are executed by the multimodal knowledge graph application system, the multimodal knowledge graph application system, specifically the processor, is configured to perform the following steps: generating thyroid triplet data based on various unstructured thyroid data and thyroid image data received from a data collector; performing data conflict detection processing on the thyroid triplet data based on the thyroid knowledge graph to obtain various conflict detection results; for each conflict detection result, in response to determining that the conflict detection result satisfies preset conflict detection conditions, determining data modification method information based on the thyroid triplet data corresponding to the conflict detection result; performing data update processing on the thyroid knowledge graph based on the determined data modification method information and the thyroid triplet data; determining thyroid execution information based on thyroid task information sent by a user terminal; executing a knowledge graph application task corresponding to the thyroid execution information based on the updated thyroid knowledge graph and the thyroid task information to obtain a task execution result; and a data transmitter is configured to send the task execution result to the user terminal.

[0153] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0155] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0156] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A multimodal knowledge graph application system, comprising: The processor is configured to perform the following steps: Based on the individual unstructured thyroid data and individual thyroid image data received from the data collector, individual thyroid triplet data are generated; Based on the thyroid knowledge graph, data conflict detection processing is performed on the data of each thyroid triplet to obtain the conflict detection results. For each conflict detection result, in response to determining that the conflict detection result meets the preset conflict detection conditions, based on the thyroid triplet data corresponding to the conflict detection result, data modification method information is determined; Based on the determined data modification methods and the thyroid triplet data, the thyroid knowledge graph is updated. Based on the thyroid task information sent by the user terminal, determine the thyroid execution information; Based on the updated thyroid knowledge graph and the thyroid task information, a knowledge graph application task corresponding to the thyroid task information is executed to obtain a task execution result. The processor is further configured to: In response to determining that the thyroid execution information meets the preset recommended task conditions, the following recommended information generation steps are performed: Based on the updated thyroid knowledge graph, generate recommendation information for each thyroid gland; For each thyroid recommendation, resource modeling is performed on the thyroid recommendation to obtain thyroid resource data; For each thyroid resource data point obtained, perform the following steps: Based on the thyroid task information and the thyroid resource data, technical indicator data, processing matching degree data, equipment matching degree data, technical matching degree data and cycle matching degree data are generated; The technology matching degree data and the technology indicator data are weighted to obtain the first feasibility data; The processing matching data and the device matching data are weighted to obtain the second feasibility data; The first feasibility data, the second feasibility data, and the periodic matching degree data are weighted to obtain feasibility score data; Based on the obtained feasibility scores and the thyroid recommendation information, the target thyroid recommendation information is determined. The target thyroid recommendation information is determined as the task execution result; A data transmitter is configured to send the task execution result to the user terminal.

2. The multimodal knowledge graph application system according to claim 1, wherein, The processor is further configured to: In response to determining that the thyroid task information meets a preset first execution condition, the preset first execution information is determined as the thyroid execution information; In response to determining that the thyroid task information meets a preset second execution condition, the preset second execution information is determined as the thyroid execution information.

3. The multimodal knowledge graph application system according to claim 1, wherein, The processor is further configured to: In response to determining that the thyroid execution information meets the preset question-and-answer task conditions, the following question-and-answer steps are performed: The thyroid task information is processed by keyword recognition to obtain thyroid keyword data. Based on the updated thyroid knowledge graph and the various thyroid keyword data, various graph entity data are generated; Based on the data of each graph entity and the preset prompts, thyroid answer information is generated; The thyroid answer information is determined as the task execution result.

4. The multimodal knowledge graph application system according to claim 1, wherein, Each thyroid triplet data set includes first entity data, entity relation data, and second entity data; and the processor is further configured to: For each thyroid triplet in the aforementioned thyroid triplet data, perform the following steps: The first entity data and the second entity data included in the thyroid triplet data are combined into the data to be queried; The entity relationship data included in the thyroid triplet data is identified as the connection data to be compared; The query results are obtained by retrieving the data to be queried from the thyroid knowledge graph. In response to determining that the data query results meet preset data query conditions, based on the data query results and the thyroid knowledge graph, each target connection data is determined; The target connection data and the connection data to be compared are compared to obtain the comparison results. Based on the comparison results of each data set, the thyroid triplet data, and the thyroid knowledge graph, the conflict detection results corresponding to the thyroid triplet data are determined.

5. The multimodal knowledge graph application system according to claim 1, wherein, The processor is further configured to: The product of the preset first parameter data and the first feasibility data is determined as the first weighted data; The product of the preset second parameter data and the second feasibility data is determined as the second weighted data; The product of the preset third parameter data and the period matching degree data is determined as the third weighted data; The sum of the first weighted data, the second weighted data, and the third weighted data is determined as the feasibility score data.

6. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method executed by the processor as described in any one of claims 1 to 5.

Images