A method and device for predicting state reference information of artificial intelligence under multimodality

Through multimodal data analysis, the similarity comparison of electrocardiogram, cardiac sound chart and pulse wave chart is improved, the efficiency of heart disease type judgment is solved, the problem of low judgment efficiency in the existing technology is achieved, and high-precision state reference is achieved Information prediction.

CN118280573BActive Publication Date: 2024-11-15无锡珞迦云脉医疗科技有限公司 +3
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410505650.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-25
Publication Date
2024-11-15
Estimated Expiration
2044-04-25

AI Technical Summary

Technical Problem

When the prior art screening and monitoring heart disease in early screening, it relies on doctors and high-end medical equipment, resulting in low index specificity and sensitivity, and the inability to consider high sensitivity in many aspects, resulting in low efficiency in judging disease types.

Method used

By using the method of artificial intelligence predicting state reference information under multimodal, the similarity between image state characteristics and image reference characteristics is calculated by obtaining the image state characteristics of state images such as electrocardiogram, cardiac sound chart, pulse wave chart, etc., based on the similarity ratio The status reference information prediction results of the object to be predicted are determined for the results, and the status reference information with high precision is realized for multimodal data analysis and judgment.

Benefits of technology

Improve the efficiency of disease type judgment and solve the problem of incomplete data. With the enrichment of image reference feature database data, the accuracy of analytical status reference information has also been improved, providing intermediate reference information in medical judgment to help ensure Status reference information accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118280573B_ABST
    Figure CN118280573B_ABST
Patent Text Reader

Abstract

The present application relates to a method and device for predicting state reference information by artificial intelligence under multimodality. The method comprises: obtaining the image state feature of the state image corresponding to the object to be predicted and the first fusion similarity corresponding to the first type of image reference feature and the second fusion similarity corresponding to the second type of image reference feature; calculating the similarity between the image reference feature and the image reference feature of each reference image matching the state image in the image reference feature library, taking the maximum similarity between the image state feature and the first type of image reference feature as the first type similarity, taking the maximum similarity between the image state feature and the second type of image reference feature as the second type similarity; determining the prediction result of the state reference information of the object to be predicted based on the comparison results of the first type similarity, the second type similarity, the first fusion similarity and the second similarity. The use of this method can improve the efficiency of judging the type of disease.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for predicting state reference information of artificial intelligence under multimodality. Background Art

[0002] With the development of computer technology, computer technology has gradually been applied to the medical field. At present, the early screening and monitoring of heart disease mainly relies on doctors and high-end medical equipment, which makes the specificity and sensitivity of certain indicators low in some complex situations, and it is impossible to consider multiple aspects with high sensitivity, resulting in low efficiency in judging the type of disease. Summary of the invention

[0003] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for multimodal artificial intelligence prediction status reference information to address the above-mentioned technical problems, which can improve the efficiency of determining disease types.

[0004] In a first aspect, the present application provides a method for predicting state reference information of artificial intelligence under multimodality, which is applied to a computer device, comprising:

[0005] Acquire image state features of a state image corresponding to an object to be predicted, and acquire a first fusion similarity corresponding to an image reference feature of a first type and a second fusion similarity corresponding to an image reference feature of a second type in an image reference feature library; the first fusion similarity is used to characterize the fusion of the maximum similarity between image reference features of the first type, and the second fusion similarity is used to characterize the fusion of the maximum similarity between image reference features of the second type; the state image includes at least one of an electrocardiogram, a phonocardiogram, and a pulse wave graph;

[0006] For each image reference feature of the reference image matching the state image in the image reference feature library, calculate the similarity between the image state feature and the image reference feature, take the maximum similarity between the image state feature and the image reference features belonging to the first type as the first type similarity, and take the maximum similarity between the image state feature and the image reference features belonging to the second type as the second type similarity;

[0007] Based on the comparison results of the first type of similarity, the second type of similarity, the first fusion similarity and the second similarity, a state reference information prediction result corresponding to the object to be predicted is determined.

[0008] In one of the embodiments, based on the comparison results of the first type of similarity, the second type of similarity, the first fusion similarity and the second similarity, determining the prediction result of the state reference information corresponding to the object to be predicted includes:

[0009] If the first type of similarity is greater than the fusion value between the first preset value and the first fusion similarity, and the second type of similarity is greater than the fusion value between the second preset value and the second fusion similarity, then based on the comparison result of the first type of similarity and the second type of similarity, determine the state reference information prediction result corresponding to the object to be predicted;

[0010] If the first type similarity is greater than the fusion value between the first preset value and the first fusion similarity, and the second type similarity is less than the fusion value between the second preset value and the second fusion similarity, then determining that the state reference information prediction result is that the state reference information of the object to be predicted is of the first type;

[0011] If the first type similarity is less than the fusion value between the first preset value and the first fusion similarity, and the second type similarity is greater than the fusion value between the second preset value and the second fusion similarity, then the state reference information prediction result is determined to be that the state reference information of the object to be predicted is of the second type.

[0012] In one of the embodiments, based on the comparison result of the first type of similarity and the second type of similarity, determining the prediction result of the state reference information corresponding to the object to be predicted includes:

[0013] If the first type of similarity is greater than the second type of similarity, determining that the state reference information prediction result is that the state reference information of the object to be predicted is of the first type;

[0014] If the first type similarity is less than or equal to the second type similarity, it is determined that the state reference information prediction result is that the state reference information of the object to be predicted is of the second type.

[0015] In one of the embodiments, the state reference information prediction method further includes:

[0016] Acquire a first type of fusion feature and a second type of fusion feature, wherein the first type of fusion feature is used to characterize the fusion of image reference features belonging to the first type in the image reference feature library, and the second type of fusion feature is used to characterize the fusion of image reference features belonging to the second type in the image reference feature library;

[0017] Calculating a third similarity between the image state feature and the first type of fused feature, and calculating a fourth similarity between the image state feature and the second type of fused feature;

[0018] If the first type similarity is less than the fusion value between the first preset value and the first fusion similarity, and the second type similarity is less than the fusion value between the second preset value and the second fusion similarity, then based on the comparison result of the third similarity and the fourth similarity, the state reference information prediction result corresponding to the object to be predicted is determined.

[0019] In one of the embodiments, determining the prediction result of the state reference information corresponding to the object to be predicted based on the comparison result of the third similarity and the fourth similarity includes:

[0020] If the third similarity is greater than the fourth similarity, determining that the state reference information prediction result is that the state reference information of the object to be predicted is of the first type;

[0021] If the third similarity is less than or equal to the fourth similarity, it is determined that the state reference information prediction result is that the state reference information of the object to be predicted is of the second type.

[0022] In a second aspect, the present application also provides a device for predicting state reference information of artificial intelligence under multimodality, including:

[0023] An acquisition module is used to acquire image state features of a state image corresponding to the object to be predicted, and acquire a first fusion similarity corresponding to an image reference feature of a first type and a second fusion similarity corresponding to an image reference feature of a second type in an image reference feature library; the first fusion similarity is used to characterize the fusion of the maximum similarity between the image reference features of the first type, and the second fusion similarity is used to characterize the fusion of the maximum similarity between the image reference features of the second type; the state image includes at least one of an electrocardiogram, a phonocardiogram, and a pulse wave graph;

[0024] a calculation module, configured to calculate the similarity between the image state feature and the image reference feature for each image reference feature of the reference image matching the state image in the image reference feature library, taking the maximum similarity between the image state feature and the image reference features belonging to the first type as the first type similarity, and taking the maximum similarity between the image state feature and the image reference features belonging to the second type as the second type similarity;

[0025] The comparison module is used to determine the state reference information prediction result corresponding to the object to be predicted based on the comparison results of the first type similarity, the second type similarity, the first fusion similarity and the second similarity.

[0026] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above method when executing the computer program.

[0027] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps in the above method when executed by a processor.

[0028] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps in the above method when executed by a processor.

[0029] The above-mentioned method, apparatus, computer device, computer-readable storage medium and computer program product for predicting state reference information by artificial intelligence under multimodality obtain the image state features of the state image corresponding to the object to be predicted, including at least one of the electrocardiogram, phonocardiogram and pulse wave graph, obtain the first fusion similarity used to characterize the fusion of the maximum similarity between the image reference features belonging to the first type in the image reference feature library, and obtain the second fusion similarity used to characterize the fusion of the maximum similarity between the image reference features belonging to the second type in the image reference feature library; for each image reference feature of the reference image matching the state image in the image reference feature library, calculate the similarity between the image state feature and the image reference feature, take the maximum similarity between the image state feature and the image reference features belonging to the first type as the first type similarity, and take the image state feature as the image reference feature. The maximum similarity with the image reference features belonging to the second type is used as the second type similarity; based on the comparison results of the first type similarity, the second type similarity, the first fusion similarity and the second similarity, the state reference information prediction result corresponding to the object to be predicted is determined, and the multimodal data analysis of the electrocardiogram, the phonocardiogram and the pulse wave graph is used to judge the high-precision state reference information, so that even in the case of missing feature dimensions, the existing dimensions can be used for similarity matching, which solves the problem of incomplete data. With the enrichment of the image reference feature library data, the accuracy of the corresponding analysis state reference information can also be improved, and the state reference information can be applied to the intermediate reference information of medical judgment of disease types, providing a consideration direction for the judgment that it may be a certain type of disease, which is conducive to improving the efficiency of judging the disease type on the basis of ensuring the accuracy of the state reference information. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0031] FIG1 is an application environment diagram of a method for predicting state reference information using artificial intelligence under multimodality in one embodiment;

[0032] Figure 2 A schematic diagram of a flow chart of a method for predicting state reference information using artificial intelligence under multimodality in one embodiment;

[0033] Figure 3 is a schematic diagram of various state images in one embodiment;

[0034] Figure 4A structural block diagram of a device for predicting state reference information by artificial intelligence under multimodality in one embodiment;

[0035] FIG5 is a diagram showing the internal structure of a computer device in one embodiment;

[0036] FIG. 6 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0038] A method for predicting state reference information of artificial intelligence under multimodality provided in an embodiment of the present application can be applied in an application environment as shown in FIG1. ​​Among them, the terminal 102 communicates with the server 104 through a network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0039] Both the terminal and the server can be used separately to execute a method for predicting state reference information of artificial intelligence under multimodality provided in an embodiment of the present application.

[0040] For example, the terminal obtains the image state feature of the state image corresponding to the object to be predicted, obtains the first fusion similarity corresponding to the image reference feature belonging to the first type in the image reference feature library, and the second fusion similarity corresponding to the image reference feature belonging to the second type; the first fusion similarity is used to characterize the fusion of the maximum similarity between the image reference features of the first type, and the second fusion similarity is used to characterize the fusion of the maximum similarity between the image reference features of the second type; the state image includes at least one of an electrocardiogram, a phonocardiogram, and a pulse wave graph; for each image reference feature of the reference image matching the state image in the image reference feature library, calculate the similarity between the image state feature and the image reference feature, take the maximum similarity between the image state feature and the image reference feature belonging to the first type as the first type similarity, and take the maximum similarity between the image state feature and the image reference feature belonging to the second type as the second type similarity; based on the comparison results of the first type similarity, the second type similarity, the first fusion similarity and the second similarity, determine the state reference information prediction result corresponding to the object to be predicted.

[0041] For example, the server obtains the image state feature of the state image corresponding to the object to be predicted, obtains the first fusion similarity corresponding to the image reference feature belonging to the first type in the image reference feature library, and the second fusion similarity corresponding to the image reference feature belonging to the second type; the first fusion similarity is used to characterize the fusion of the maximum similarity between the image reference features of the first type, and the second fusion similarity is used to characterize the fusion of the maximum similarity between the image reference features of the second type; the state image includes at least one of an electrocardiogram, a phonocardiogram, and a pulse wave graph; for each image reference feature of the reference image matching the state image in the image reference feature library, calculate the similarity between the image state feature and the image reference feature, take the maximum similarity between the image state feature and the image reference feature belonging to the first type as the first type similarity, and take the maximum similarity between the image state feature and the image reference feature belonging to the second type as the second type similarity; based on the comparison results of the first type similarity, the second type similarity, the first fusion similarity and the second similarity, determine the state reference information prediction result corresponding to the object to be predicted.

[0042] In an exemplary embodiment, Figure 2 As shown, a method for predicting state reference information of artificial intelligence under multi-modality is provided, and the method is applied to the server in FIG. 1 as an example for explanation, including the following steps S202 to S204. Among them:

[0043] Step S200, obtaining image state features of a state image corresponding to an object to be predicted, obtaining a first fusion similarity corresponding to an image reference feature of a first type in an image reference feature library, and a second fusion similarity corresponding to an image reference feature of a second type; the first fusion similarity is used to characterize the fusion of maximum similarity between image reference features of the first type, and the second fusion similarity is used to characterize the fusion of maximum similarity between image reference features of the second type; the state image includes at least one of an electrocardiogram, a phonocardiogram, and a pulse wave graph.

[0044] The object to be predicted refers to an object whose state reference information is to be analyzed; it may be a patient whose disease type reference information is to be analyzed. The state image refers to an image related to the analyzed state reference information; it includes at least one of an electrocardiogram, a phonocardiogram, and a pulse wave graph. Image state features refer to image features of a state image, which may be represented in the form of a feature vector; different state graphs correspond to different features for analyzing state reference information. If the state image is an electrocardiogram, the features corresponding to the electrocardiogram for analyzing state reference information include but are not limited to the second lead, heart rate (HR), heart rate variability (HRV), ventricular late potential (VLP), atrial late potential (ALP), ECG spectrum, Q wave amplitude, R wave width, R wave amplitude, S wave width, S wave amplitude, ST segment elevation and ST segment depression, T prolongation. If the state image is a phonocardiogram, the features corresponding to the phonocardiogram for analyzing state reference information include but are not limited to the first heart sound (S1) amplitude, the second heart sound (S2) amplitude, S1 / S2, the third heart sound (S3) amplitude, and the fourth heart sound (S4) amplitude. If the state image is a pulse wave graph, the features corresponding to the pulse wave graph for analyzing state reference information include but are not limited to systolic blood pressure (SBP), diastolic blood pressure (DBP), and pulse pressure (PP).

[0045] The image reference feature library refers to an image reference feature library of reference images corresponding to reference objects belonging to different types of states; the state may be positive indicating heart disease or negative indicating non-heart disease; the reference images include at least one of the electrocardiogram, phonocardiogram and pulse wave graph corresponding to each positive reference patient with heart disease, and at least one of the electrocardiogram, phonocardiogram and pulse wave graph corresponding to each negative reference patient without heart disease; the reference images corresponding to different reference objects in the image reference feature library are all real data obtained and permitted for use in the medical field, and the combinations of reference images corresponding to different reference objects may be different, for example, the reference images of some reference objects include electrocardiograms, the reference images of some reference objects include state graphs and phonocardiograms, and the reference images of some reference objects include state graphs, phonocardiograms and pulse wave graphs.

[0046] Image reference features refer to image state features of reference images corresponding to each reference object in the image reference feature library; they can be represented in the form of feature vectors; reference images also include at least one of electrocardiogram, phonocardiogram, and pulse wave graph; the features of reference images belonging to electrocardiogram also include but are not limited to the second lead, heart rate (HR), heart rate variability (HRV), ventricular late potential (VLP), atrial late potential (ALP), ECG spectrum, Q wave amplitude, R wave width, R wave amplitude, S wave width, S wave amplitude, ST segment elevation and ST segment depression, T prolongation; the features of reference images belonging to phonocardiogram also include but are not limited to the first heart sound (S1) amplitude, the second heart sound (S2) amplitude, S1 / S2, the third heart sound (S3) amplitude, and the fourth heart sound (S4) amplitude; the features of reference images belonging to pulse wave graph also include but are not limited to systolic blood pressure (SBP), diastolic blood pressure (DBP), and pulse pressure (PP). The first type refers to a type of state, that is, a positive type, which can be a positive representation of heart disease. The second type refers to another type of state, ie, a negative type, which may be a negative type indicating that the condition does not involve a heart disease.

[0047] The first fusion similarity is used to characterize the fusion of the maximum similarity between the image reference features of the first type; its calculation method may be: for each image reference feature of the first type in the image reference feature library, the maximum similarity between the image reference feature and other image references of the first type is used as the similarity to be fused of the corresponding image reference feature, the similarities to be fused corresponding to each image reference feature of the first type in the image reference feature library are added and averaged, and the calculated average similarity is used as the first fusion similarity. The second fusion similarity is used to characterize the fusion of the maximum similarity between the image reference features of the second type; its calculation method may be: for each image reference feature of the second type in the image reference feature library, the maximum similarity between the image reference feature and other image reference features of the second type is used as the similarity to be fused of the corresponding image reference feature, the similarities to be fused corresponding to each image reference feature of the second type in the image reference feature library are added and averaged, and the calculated average similarity is used as the second fusion similarity.

[0048] Exemplarily, multiple reference objects corresponding to different reference image combinations can be screened from a database of the same type of disease in the medical field, and image reference features corresponding to the reference images of the screened reference objects can be stored in an image reference feature library as reference data, thereby constructing an image reference feature library to which reference data can be added at any time, and the image reference feature library can be enriched with the diversity of data in the medical field that allows the use of this type of disease. In addition, in order to determine the state reference information of the object to be predicted, the state image of the object to be predicted can be obtained, and the image state features corresponding to the state image can be extracted, and the similarity between the image state features and the reference image features corresponding to each reference image matching the state image in the image reference feature library is calculated, and then the fusion value of the maximum similarity between each image reference feature belonging to the first type in the image reference feature library is calculated, and the fusion value is used as the first fusion similarity, and the fusion value of the maximum similarity between each image reference feature belonging to the second type in the image reference feature library is calculated, and the fusion value is used as the second fusion similarity. The first fusion similarity and the second fusion similarity can reflect the average similarity between the reference features of the same type in the image reference features, and can reflect the comprehensive situation of the same type of state. Using the first fusion similarity and the second fusion similarity as a comparison basis can ensure the accuracy of the analysis of the state reference information.

[0049] Step S202, for each image reference feature of the reference image that matches the state image in the image reference feature library, calculate the similarity between the image state feature and the image reference feature, take the maximum similarity between the image state feature and the image reference features belonging to the first type as the first type similarity, and take the maximum similarity between the image state feature and the image reference features belonging to the second type as the second type similarity.

[0050] The first type of similarity refers to the maximum similarity among the similarities between the image reference features corresponding to each reference image matching the state image in the image reference feature library and the image state features. The second type of similarity refers to the maximum similarity among the similarities between the image reference features corresponding to each reference image matching the state image in the image reference feature library and the image state features.

[0051] Exemplarily, in order to analyze whether the object to be predicted belongs to the first type in the image reference feature library, the similarity between the image reference feature of each reference image that matches the state image in the image reference feature library and the image state feature of the object to be predicted can be calculated, and the maximum similarity between the image state feature and the image reference features belonging to the first type can be taken as the first type similarity, and the maximum similarity between the image state feature and the image reference features belonging to the second type can be taken as the second type similarity. The first type similarity and the second type similarity can reflect the degree of closeness between the object to be predicted and different types of states in the image reference feature library.

[0052] Step S204: determining a prediction result of the state reference information corresponding to the object to be predicted based on the comparison results of the first type similarity, the second type similarity, the first fusion similarity and the second similarity.

[0053] Among them, the state reference information prediction result refers to the result of predicting the state type of the object to be predicted; it can be predicted to belong to heart disease (positive) or not to heart disease (negative); the state reference information prediction result can provide medical experts with reference information for disease judgment. It is not a direct result of judging whether the object to be tested belongs to a certain disease type, but only an intermediate reference information. The final result of the specific disease type is judged by the corresponding medical expert.

[0054] Exemplarily, the first fused similarity can reflect the comprehensive situation of the similarity between the image reference features belonging to the first type in the image reference feature library. Using the first fused similarity as one of the judgment thresholds of the first type can improve the accuracy of judging the first type. The second fused similarity can reflect the comprehensive situation of the similarity between the image reference features belonging to the second type in the image reference feature library. Using the second fused similarity as one of the judgment thresholds of the second type can also improve the accuracy of judging the second type to a certain extent. The first type similarity is compared with the first fused similarity to determine the closeness of the state reference information of the object to be predicted to the first type. The second type similarity is compared with the second fused similarity to determine the closeness of the state reference information of the object to be predicted to the second type. The two comparison results are combined to determine whether the state reference information of the object to be predicted belongs to the first type or the second type.

[0055] It should be noted that the similarities mentioned above (including but not limited to the first type of similarity, the second type of similarity, the first fusion similarity, and the second fusion similarity) are all that the greater the similarity, the higher the degree of similarity, and the above similarity can be but is not limited to cosine similarity. In one embodiment, other values ​​that characterize the degree of similarity between feature vectors can also be used, such as Euclidean distance, but the smaller the Euclidean distance, the higher the degree of similarity. If the value used to judge the degree of similarity is inversely proportional to the similarity, the judgment logic should also be adjusted according to the situation where the value of the degree of similarity is inversely proportional to the similarity during the corresponding comparison.

[0056] The above-mentioned method for predicting state reference information by artificial intelligence under multimodality obtains image state features of a state image corresponding to an object to be predicted, including at least one of an electrocardiogram, a phonocardiogram, and a pulse wave graph, obtains a first fusion similarity for representing the fusion of the maximum similarity between image reference features belonging to a first type in an image reference feature library, and obtains a second fusion similarity for representing the fusion of the maximum similarity between image reference features belonging to a second type in an image reference feature library; for each image reference feature of a reference image matching the state image in the image reference feature library, calculates the similarity between the image state feature and the image reference feature, takes the maximum similarity between the image state feature and the image reference feature belonging to the first type as the first type similarity, and takes the maximum similarity between the image state feature and the image reference feature belonging to the second type as the first type similarity. The largest similarity among the features is taken as the second type similarity; based on the comparison results of the first type similarity, the second type similarity, the first fusion similarity and the second similarity, the state reference information prediction result corresponding to the object to be predicted is determined, and the multimodal data analysis of electrocardiogram, phonocardiogram and pulse wave graph is used to judge the high-precision state reference information, so that even in the case of missing feature dimensions, similarity matching can be performed using existing dimensions, which solves the problem of incomplete data. With the enrichment of image reference feature library data, the accuracy of corresponding analysis state reference information can also be improved, and the state reference information can be applied to the intermediate reference information of medical judgment of disease type, providing a consideration direction for judgment that may be a certain type of disease, which is conducive to improving the efficiency of judging disease type on the basis of ensuring the accuracy of state reference information.

[0057] In some embodiments, step S204 includes:

[0058] Step S300: If the first type similarity is greater than the fusion value between the first preset value and the first fusion similarity, and the second type similarity is greater than the fusion value between the second preset value and the second fusion similarity, then based on the comparison result of the first type similarity and the second type similarity, determine the state reference information prediction result corresponding to the object to be predicted.

[0059] Step S302: if the first type similarity is greater than the fusion value between the first preset value and the first fusion similarity, and the second type similarity is less than the fusion value between the second preset value and the second fusion similarity, then it is determined that the state reference information prediction result is that the state reference information of the object to be predicted is of the first type.

[0060] Step S304: if the first type similarity is less than the fusion value between the first preset value and the first fusion similarity, and the second type similarity is greater than the fusion value between the second preset value and the second fusion similarity, then determining that the state reference information prediction result is that the state reference information of the object to be predicted is of the second type.

[0061] The first preset value refers to a coefficient when the first type of similarity is compared with the first fusion similarity, and the size can be set according to actual needs, which can reflect the influence of the first fusion similarity on the state reference information in different situations. The second preset value refers to a coefficient when the second type of similarity is compared with the second fusion similarity, and the size can also be set according to actual needs, which can reflect the influence of the first fusion similarity on the state reference information in different situations.

[0062] Exemplarily, if the first type of similarity is greater than the fusion value between the first preset value and the first fusion similarity, and the second type of similarity is greater than the fusion value between the second preset value and the second fusion similarity, that is, the similarities are both greater than the different types of comparison thresholds, it indicates that the image state features corresponding to the state image of the object to be predicted, and the image reference features of the first type and the image reference features of the second type are similar to each other in a comprehensive manner, and have reached the average level of the image reference feature library. To improve the accuracy of the analysis and judgment, the first type of similarity and the second type of similarity may be further compared, and the state reference information may be judged according to the magnitude of the first type of similarity and the second type of similarity. If the first type of similarity is greater than the fusion value between the first preset value and the first fusion similarity, and the second type of similarity is less than the fusion value between the second preset value and the second fusion similarity, it indicates that the image state features corresponding to the state image of the object to be predicted are more similar to the image reference features of the first type, and less similar to the image reference features of the second type. It may be determined that the state reference information prediction result is that the state reference information of the object to be predicted is of the first type.

[0063] Further, if the first type of similarity is smaller than the fusion value between the first preset value and the first fusion similarity, and the second type of similarity is greater than the fusion value between the second preset value and the second fusion similarity, it indicates that the image state features corresponding to the state image of the object to be predicted are not very similar to the image reference features of the first type, but are more similar to the image reference features of the second type, and it can be determined that the state reference information prediction result is that the state reference information of the object to be predicted is of the second type.

[0064] In the above embodiment, by comparing the first type of similarity with the first fused similarity, and comparing the second type of similarity with the second fused similarity, it is analyzed whether the image state characteristics of the object to be predicted are closer to the first type of image reference characteristics or the second type of image reference characteristics, thereby ensuring better the accuracy of the state reference information prediction results and improving the efficiency of analyzing the state reference information prediction results based on the effectiveness and accuracy of the first fused similarity and the second fused similarity as comparison objects.

[0065] In some embodiments, determining the prediction result of the state reference information corresponding to the object to be detected based on the comparison result of the first type of similarity and the second type of similarity in step S300 includes:

[0066] Step S400: If the first type similarity is greater than the second type similarity, it is determined that the state reference information prediction result is that the state reference information of the object to be predicted is of the first type.

[0067] Step S402: If the first type similarity is less than or equal to the second type similarity, determining that the state reference information prediction result is that the state reference information of the object to be predicted is of the second type.

[0068] Exemplarily, if the first type of similarity is greater than the first preset value and the first fusion similarity, and the second type of similarity is also greater than the second preset value and the second fusion similarity, the state reference information of the object to be predicted cannot be accurately determined at this time, and the first type of similarity and the second type of similarity can be further judged. If the first type of similarity is greater than the second type of similarity, it is determined that the state reference information prediction result is that the state reference information of the object to be predicted is of the first type; if the first type of similarity is less than or equal to the second type of similarity, it is determined that the state reference information prediction result is that the state information of the object to be predicted is of the second type.

[0069] In the above embodiment, when the first type of similarity is greater than the first preset value and the first fusion similarity, and the second type of similarity is also greater than the second preset value and the second fusion similarity, in order to more accurately analyze the state reference information of the object to be predicted, the size of the first type of similarity and the second type of similarity can be further compared, thereby further improving the accuracy of judging the state reference information.

[0070] In some embodiments, the method for predicting state reference information of artificial intelligence under multimodality further includes:

[0071] Step S500, obtaining a first type of fusion feature and a second type of fusion feature, wherein the first type of fusion feature is used to characterize the fusion of image reference features belonging to the first type in the image reference feature library, and the second type of fusion feature is used to characterize the fusion of image reference features belonging to the second type in the image reference feature library.

[0072] Step S502: calculating a third similarity between the image state feature and the first type fusion feature, and calculating a fourth similarity between the image state feature and the second type fusion feature.

[0073] Step S504: If the first type similarity is less than the fusion value between the first preset value and the first fusion similarity, and the second type similarity is less than the fusion value between the second preset value and the second fusion similarity, then based on the comparison result of the third similarity and the fourth similarity, determine the state reference information prediction result corresponding to the object to be predicted.

[0074] Among them, the first type of fusion feature is used to characterize the fusion of image reference features belonging to the first type in the image reference feature library, and can be a feature vector obtained by adding and averaging the image reference features belonging to the first type in the image reference feature library. The second type of fusion feature is used to characterize the fusion of image reference features belonging to the second type in the image reference feature library, and can be a feature vector obtained by adding and averaging the image reference features belonging to the second type in the image reference feature library. The third similarity refers to the similarity between the image state feature corresponding to the object to be predicted and the first type of fusion feature. The fourth similarity refers to the similarity between the image state feature corresponding to the object to be predicted and the second type of fusion feature.

[0075] Exemplarily, in order to further improve the accuracy of analyzing the state reference information, the similarity between the state image of the object to be predicted and the comprehensive features of the image reference features of the reference image matching the state image in the image reference feature library can be further analyzed, and the fusion features used to characterize the image reference features belonging to the first type in the image reference feature library, that is, the first type fusion features, can be obtained, and the fusion features used to characterize the image reference features belonging to the second type in the image reference feature library, that is, the second type fusion features, can be obtained, and the similarity between the image state features corresponding to the object to be predicted and the first type fusion features (that is, the third similarity) is calculated, and the similarity between the image state features corresponding to the object to be predicted and the second type fusion features (that is, the fourth similarity) is calculated, so as to clarify the proximity between the image state features and the first type fusion features and the second type fusion features. If the first type similarity is less than the fusion value between the first preset value and the first fusion similarity, and the second type similarity is less than the fusion value between the second preset value and the second fusion similarity, then the state reference information of the object to be predicted cannot be accurately determined at this time, and the third similarity and the fourth similarity can be further compared, so as to further and more accurately determine the state reference information of the object to be predicted according to the size of the third similarity and the fourth similarity.

[0076] In the above embodiment, by obtaining the first type of fusion feature used to characterize the fusion of image reference features belonging to the first type in the image reference feature library, and obtaining the second type of fusion feature used to characterize the fusion of image reference features belonging to the second type in the image reference feature library, and respectively calculating the similarity between the image state feature and the first type of fusion feature and the second type of fusion feature, the corresponding third similarity and fourth similarity are obtained. When the first type similarity is less than the fusion value between the first preset value and the first fusion similarity, and the second type similarity is less than the fusion value between the second preset value and the second fusion similarity, the third similarity and the fourth similarity are further compared, so as to determine the state reference information of the object to be predicted according to the size of the third similarity and the fourth similarity, thereby improving the accuracy of judging the state reference information of the object to be predicted.

[0077] In some embodiments, determining the prediction result of the state reference information corresponding to the object to be detected based on the comparison result of the third similarity and the fourth similarity in step S504 includes:

[0078] Step S600: If the third similarity is greater than the fourth similarity, it is determined that the state reference information prediction result is that the state reference information of the object to be predicted is of the first type.

[0079] Step S602: If the third similarity is less than or equal to the fourth similarity, determining that the state reference information prediction result is that the state reference information of the object to be predicted is of the second type.

[0080] Exemplarily, if the first type of similarity is less than the first preset value and the first fusion similarity, and the second type of similarity is also less than the second preset value and the second fusion similarity, it means that the similarity between the image state feature and the image reference feature of the same type in the image reference feature library at this time has not reached the comprehensive situation of the maximum similarity between the image reference features of the same type in the image reference feature library. In order to further and more accurately determine the state reference information of the object to be predicted, the third similarity and the fourth similarity can be further judged. When the third similarity is greater than the fourth similarity, it means that the image state feature is more similar to the image state feature of the first type, and it can be determined that the state reference information prediction result is that the state reference information of the object to be predicted is the first type. When the third similarity is less than or equal to the fourth similarity, it means that the image state feature is more similar to the image reference feature of the second type, and it can be determined that the state reference information prediction result is that the state reference information of the object to be predicted is the second type.

[0081] In the above embodiment, when the first type of similarity is less than the first preset value and the first fusion similarity, and the second type of similarity is also less than the second preset value and the second fusion similarity, in order to more accurately analyze the state reference information of the object to be predicted, the third similarity and the fourth similarity can be further compared, thereby further improving the accuracy of judging the state reference information.

[0082] It should be noted that the similarities mentioned above (including but not limited to the first type of similarity, the second type of similarity, the first fusion similarity, the second fusion similarity, the third similarity, and the fourth similarity) are all the greater the similarity, the higher the degree of similarity. In one embodiment, other values ​​that characterize the degree of similarity between feature vectors can also be used, such as but not limited to the Euclidean distance, but the smaller the Euclidean distance, the higher the degree of similarity. If the value used to judge the degree of similarity is inversely proportional to the similarity, the judgment logic should also be adjusted according to the situation where the value of the similarity is inversely proportional to the similarity during the corresponding comparison; for example, if the value of the similarity between feature vectors is represented by the Euclidean distance, it can be specifically as follows:

[0083] Step S700, obtaining an image state feature of a state image corresponding to the object to be predicted, obtaining a first fused Euclidean distance corresponding to an image reference feature of a first type in an image reference feature library, and a second fused Euclidean distance corresponding to an image reference feature of a second type; wherein the first fused Euclidean distance is used to characterize the fusion of the minimum Euclidean distance between image reference features of the first type, and the second fused Euclidean distance is used to characterize the fusion of the minimum Euclidean distance between image reference features of the second type; the state image includes at least one of an electrocardiogram, a phonocardiogram, and a pulse wave graph;

[0084] Specifically, the calculation method of the first fused Euclidean distance may be: for each image reference feature of the first type in the image reference feature library, the minimum Euclidean distance between the image reference feature and other image references belonging to the first type is used as the Euclidean distance to be fused of the corresponding image reference feature, the Euclidean distances to be fused corresponding to each image reference feature of the first type in the image reference feature library are added and averaged, and the calculated average value of the Euclidean distances is used as the first fused Euclidean distance; the calculation method of the second fused Euclidean distance may be, for each image reference feature of the second type in the image reference feature library, the minimum Euclidean distance between the image reference feature and other image reference features belonging to the second type is used as the Euclidean distance to be fused of the corresponding image reference feature, the Euclidean distances to be fused corresponding to each image reference feature of the second type in the image reference feature library are added and averaged, and the calculated average value of the Euclidean distances is used as the second fused Euclidean distance;

[0085] Step S702, for each image reference feature of the reference image matching the state image in the image reference feature library, calculate the Euclidean distance between the image state feature and the image reference feature, take the smallest Euclidean distance between the image state feature and the image reference features belonging to the first type as the first type Euclidean distance, and take the smallest Euclidean distance between the image state feature and the image reference features belonging to the second type as the second type Euclidean distance;

[0086] Step S704, obtaining a first type fusion feature and a second type fusion feature; wherein the first type fusion feature is used to characterize the fusion of image reference features belonging to the first type in the image reference feature library, and the second type fusion feature is used to characterize the fusion of image reference features belonging to the second type in the image reference feature library; calculating a third Euclidean distance between the image state feature and the first type fusion feature, and calculating a fourth Euclidean distance between the image state feature and the second type fusion feature;

[0087] Step S706, if the first type of Euclidean distance is less than the fusion value between the first preset value and the first fused Euclidean distance, and the second type of Euclidean distance is less than the fusion value between the second preset value and the second fused Euclidean distance, then based on the comparison result of the first type of Euclidean distance and the second type of Euclidean distance, determine the state reference information prediction result corresponding to the object to be detected; wherein, based on the comparison result of the first type of Euclidean distance and the second type of Euclidean distance, determine the state reference information prediction result corresponding to the object to be detected, including: if the first type of Euclidean distance is less than the second type of Euclidean distance, determine the state reference information prediction result is that the state reference information of the object to be predicted is of the first type; if the first type of Euclidean distance is greater than or equal to the second type of Euclidean distance, determine the state reference information prediction result is that the state reference information of the object to be predicted is of the second type;

[0088] Step S708, if the first type Euclidean distance is smaller than the fusion value between the first preset value and the first fused Euclidean distance, and the second type Euclidean distance is larger than the fusion value between the second preset value and the second fused Euclidean distance, then determining that the state reference information prediction result is that the state reference information of the object to be predicted is of the first type;

[0089] Step S710, if the first type Euclidean distance is greater than the fusion value between the first preset value and the first fused Euclidean distance, and the second type Euclidean distance is less than the fusion value between the second preset value and the second fused Euclidean distance, then determining that the state reference information prediction result is that the state reference information of the object to be predicted is of the second type;

[0090] Step S712, if the first type of Euclidean distance is greater than the fusion value between the first preset value and the first fused Euclidean distance, and the second type of Euclidean distance is greater than the fusion value between the second preset value and the second fused Euclidean distance, then based on the comparison result of the third Euclidean distance and the fourth Euclidean distance, determine the state reference information prediction result corresponding to the object to be detected; wherein, based on the comparison result of the third Euclidean distance and the fourth Euclidean distance, determine the state reference information prediction result corresponding to the object to be detected, including: if the third Euclidean distance is less than the fourth Euclidean distance, determine that the state reference information prediction result is that the state reference information of the object to be predicted is of the first type; if the third Euclidean distance is greater than or equal to the fourth Euclidean distance, determine that the state reference information prediction result is that the state reference information of the object to be predicted is of the second type.

[0091] In one embodiment, if the state information of a reference object in the image reference feature library needs to be adjusted, the state information of the reference object can be changed directly, for example, the reference object needs to be changed from not having heart disease (negative) to having heart disease (positive). The change of specific data is based on the diagnosis results of medical experts in the corresponding medical field, and is not changed arbitrarily. The change of data is based on the diagnosis of the reference object. In addition, the image reference features of the reference image corresponding to the new reference object can also be added to the image reference feature library, thereby enriching the image reference feature library to improve the accuracy of judging the state reference information of the object to be predicted. And the image reference feature library does not need to re-optimize and train the model like model prediction. On the basis of ensuring the accuracy of judging the state reference information of the object to be predicted, the computing power is reduced, and the analysis efficiency of the state reference information is greatly improved.

[0092] In some embodiments, taking the analysis of a patient's heart disease prediction reference result as an example for description, data related to the detection of heart disease of multiple subjects diagnosed with heart disease (coronary heart disease and heart failure patients) through clinical gold standards and data related to the detection of heart disease of a certain number of subjects randomly selected from a group of normal subjects can be screened from a medical database with permission for use, and added to an image reference feature library; wherein, data related to the detection of heart disease of subjects diagnosed with heart disease through clinical gold standards are characterized as belonging to a first type that is positive, and data related to the detection of normal subjects without heart disease are characterized as a second type that is negative, and data related to the detection of heart disease include at least one of an electrocardiogram, a phonocardiogram, and a pulse wave graph, and the electrocardiogram, the phonocardiogram, and the pulse wave graph can be specifically as follows: Figure 3 As shown, the detection-related data can be used as reference images; for different objects, the corresponding reference data image combination types when detecting heart diseases may be different, and the data image combination types include but are not limited to electrocardiogram + phonocardiogram + pulse wave graph, electrocardiogram + phonocardiogram, electrocardiogram + pulse wave graph, phonocardiogram + pulse wave graph, single electrocardiogram, single phonocardiogram, and single pulse wave graph. According to the preset feature extraction model, the reference image corresponding to each object in the image reference feature library is extracted to obtain the image reference features of each reference image corresponding to each object, wherein the feature extraction model includes but is not limited to convolutional neural network (CNN), recurrent neural network (RNN), etc. In one embodiment, 12 leads and a sampling frequency of 1000Hz can be used when collecting electrocardiograms; electronic stethoscopes are used for phonocardiograms, with a frequency response range of 20-1000Hz; and pressure sensors are used for pulse wave graphs; in addition, the collected electrocardiograms, phonocardiograms, and pulse wave graphs can be amplified, filtered, and AD-converted by computer equipment before being processed by the computer equipment.

[0093] In detail, since the amount of multimodal data (i.e., electrocardiogram, phonocardiogram, and pulse wave graph) about heart diseases calibrated in the medical field is generally tens of thousands of people, the total amount of features selected from the features extracted from the reference images in the image reference feature library is much lower than the 1000 dimensions of general vector retrieval, and 150 to 200 features can be selected. In addition, the features of the electrocardiogram include but are not limited to the second lead heart rate (HR), heart rate variability (HRV), ventricular late potential (VLP), atrial late potential (ALP), ECG spectrum, Q wave amplitude, R wave width, R wave amplitude, S wave width, S wave amplitude, ST segment elevation and ST segment depression, ST prolongation; the features of the phonocardiogram include but are not limited to the first heart sound (S1) amplitude, the second heart sound (S2) amplitude, S1 / S2, the third heart sound (S3) amplitude, and the fourth heart sound (S4) amplitude; the features of the pulse wave graph include but are not limited to systolic blood pressure (SBP), diastolic blood pressure (DBP), and pulse pressure (PP).

[0094] Furthermore, at least one of the electrocardiogram, phonocardiogram, and pulse wave graph obtained by detecting the object to be predicted is obtained, and the electrocardiogram, phonocardiogram, and pulse wave graph obtained by detecting the object to be predicted are used as state images, and image state features corresponding to the state images are extracted according to the feature extraction model, and a first fusion similarity for representing the fusion of the maximum similarity between image reference features belonging to the first type in the image reference feature library, and a second fusion similarity for representing the fusion of the maximum similarity between image reference features belonging to the second type in the image reference feature library are obtained, and a first type of fusion feature for representing the fusion of image reference features belonging to the first type in the image reference feature library, and a second type of fusion feature for representing the fusion of image reference features belonging to the second type in the image reference feature library are obtained. a second type of fusion feature that is a fusion of the first type of image reference features; for each image reference feature of the reference image that matches the state image in the image reference feature library, calculating the similarity between the image state feature and the image reference feature, and taking the maximum similarity between the image state feature and the image reference features belonging to the first type as the first type of similarity, and taking the maximum similarity between the image state feature and the image reference features belonging to the second type as the second type of similarity; calculating the third similarity between the image state feature and the first type of fusion feature, and calculating the fourth similarity between the image state feature and the second type of fusion feature; wherein, the greater the similarity, the higher the degree of similarity, and the similarity can be but is not limited to cosine similarity.

[0095] Further, if the first type of similarity is greater than the fusion value between the first preset value and the first fusion similarity, and the second type of similarity is greater than the fusion value between the second preset value and the second fusion similarity, then when the first type of similarity is greater than the second type of similarity, it is determined that the state reference information prediction result corresponding to the object to be predicted is that the state reference information is of the first type, that is, the heart disease prediction reference result of the object to be predicted is that it belongs to heart disease; when the first type of similarity is less than or equal to the second type of similarity, it is determined that the state reference information prediction result of the object to be predicted is of the second type of state reference information, that is, the heart disease prediction reference result of the object to be predicted is that it does not belong to heart disease; If the first type of similarity is greater than the fusion value between the first preset value and the first fusion similarity, and the second type of similarity is less than the fusion value between the second preset value and the second fusion similarity, then it is determined that the state reference information prediction result corresponding to the object to be predicted is that the state reference information is of the first type, that is, the heart disease prediction reference result of the object to be predicted is that it belongs to heart disease; if the first type of similarity is less than the fusion value between the first preset value and the first fusion similarity, and the second type of similarity is greater than the fusion value between the second preset value and the second fusion similarity, then it is determined that the state reference information prediction result of the object to be predicted is the second type of state reference information, that is, the heart disease of the object to be predicted The prediction reference result is not heart disease; if the first type similarity is less than the fusion value between the first preset value and the first fusion similarity, and the second type similarity is less than the fusion value between the second preset value and the second fusion similarity, then when the third similarity is greater than the fourth similarity, it is determined that the state reference information prediction result corresponding to the object to be predicted is that the state reference information is of the first type, that is, the heart disease prediction reference result of the object to be predicted is that it belongs to heart disease, and when the third similarity is less than or equal to the fourth similarity, it is determined that the state reference information prediction result of the object to be predicted is the second type of state reference information, that is, the heart disease prediction reference result of the object to be predicted is that it does not belong to heart disease Disease, thereby realizing the reference information of whether it is possible to analyze and judge whether it is a heart disease with high accuracy based on the multimodal data of electrocardiogram, phonocardiogram and pulse wave graph. The situation of missing vector dimensions such as ECG leads is taken into account, and matching is performed based on the feature dimensions that can be extracted from the existing data, which is conducive to solving the problem of incomplete data in the medical clinical field. The reference information can be used by medical experts in the medical field as an intermediate reference information to judge whether it is a heart disease in the end, and it is not used as the actual disease diagnosis result. The analysis process of the reference information is also obtained through computer equipment processing and analysis, which is conducive to shortening the time to judge whether it is a heart disease and improving the efficiency of judging the type of disease.

[0096] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0097] Based on the same inventive concept, the embodiment of the present application also provides a device for implementing the method for artificial intelligence prediction state reference information under multimodality involved in the above-mentioned method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiments of one or more devices for artificial intelligence prediction state reference information under multimodality provided below can be referred to the limitations of the method for artificial intelligence prediction state reference information under multimodality above, and will not be repeated here.

[0098] In an exemplary embodiment, Figure 4 As shown, a device for predicting state reference information of artificial intelligence under multi-modality is provided, including: an acquisition module 400, a calculation module 402 and a comparison module 404, wherein:

[0099] The acquisition module 400 is used to obtain the image state features of the state image corresponding to the object to be predicted, and obtain the first fusion similarity corresponding to the image reference features belonging to the first type in the image reference feature library, and the second fusion similarity corresponding to the image reference features belonging to the second type; the first fusion similarity is used to characterize the fusion of the maximum similarity between the image reference features of the first type, and the second fusion similarity is used to characterize the fusion of the maximum similarity between the image reference features of the second type; the state image includes at least one of an electrocardiogram, a phonocardiogram, and a pulse wave graph.

[0100] The calculation module 402 is used to calculate the similarity between the image state feature and the image reference feature for each image reference feature of the reference image matching the state image in the image reference feature library, and take the maximum similarity between the image state feature and the image reference features belonging to the first type as the first type similarity, and take the maximum similarity between the image state feature and the image reference features belonging to the second type as the second type similarity.

[0101] The comparison module 404 is used to determine the state reference information prediction result corresponding to the object to be predicted based on the comparison results of the first type similarity, the second type similarity, the first fusion similarity and the second similarity.

[0102] In some embodiments, the comparison module 404 is also used to determine the state reference information prediction result corresponding to the object to be predicted based on the comparison result of the first type similarity and the second type similarity if the first type similarity is greater than the fusion value between the first preset value and the first fusion similarity, and the second type similarity is greater than the fusion value between the second preset value and the second fusion similarity; if the first type similarity is greater than the fusion value between the first preset value and the first fusion similarity, and the second type similarity is less than the fusion value between the second preset value and the second fusion similarity, then determine that the state reference information prediction result is that the state reference information of the object to be predicted is of the first type; if the first type similarity is less than the fusion value between the first preset value and the first fusion similarity, and the second type similarity is greater than the fusion value between the second preset value and the second fusion similarity, then determine that the state reference information prediction result is that the state reference information of the object to be predicted is of the second type.

[0103] In some embodiments, the comparison module 404 is also used to determine that the state reference information prediction result is that the state reference information of the object to be predicted is of the first type if the first type similarity is greater than the second type similarity; if the first type similarity is less than or equal to the second type similarity, then determine that the state reference information prediction result is that the state reference information of the object to be predicted is of the second type.

[0104] In some embodiments, a device for predicting state reference information by artificial intelligence in multimodality also includes other comparison modules 406, which are used to obtain first-type fusion features and second-type fusion features, the first-type fusion features are used to characterize the fusion of image reference features belonging to the first type in the image reference feature library, and the second-type fusion features are used to characterize the fusion of image reference features belonging to the second type in the image reference feature library; calculate the third similarity between the image state feature and the first-type fusion feature, and calculate the fourth similarity between the image state feature and the second-type fusion feature; if the first-type similarity is less than the fusion value between the first preset value and the first fusion similarity, and the second-type similarity is less than the fusion value between the second preset value and the second fusion similarity, then based on the comparison result of the third similarity and the fourth similarity, determine the state reference information prediction result corresponding to the object to be predicted.

[0105] In some embodiments, other comparison modules 406 are also used to determine that if the third similarity is greater than the fourth similarity, then the state reference information prediction result is that the state reference information of the object to be predicted is of the first type; if the third similarity is less than or equal to the fourth similarity, then the state reference information prediction result is determined to be that the state reference information of the object to be predicted is of the second type.

[0106] Each module in the device for predicting state reference information of artificial intelligence under multimodality can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0107] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG5. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to the reference information of the artificial intelligence prediction state under multimodality. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for predicting the reference information of the artificial intelligence prediction state under multimodality is implemented.

[0108] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG6. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for predicting state reference information of artificial intelligence under multimodality is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0109] Those skilled in the art will appreciate that the structure shown in FIG. 5 or FIG. 6 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0110] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0111] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0112] In one embodiment, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0113] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0114] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0115] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0116] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for predicting state reference information under multimodal artificial intelligence, applied to computer equipment, characterized in that: The method comprises: The image state features of the state image corresponding to the object to be predicted are obtained, and a first fusion similarity corresponding to the image reference features of the first type in the image reference feature library and a second fusion similarity corresponding to the image reference features of the second type are obtained; the first fusion similarity is used to characterize the fusion of the maximum similarity between the image reference features of the first type, and the first fusion similarity is obtained by taking each image reference feature of the first type in the image reference feature library and the maximum similarity among other image reference features of the first type as the corresponding image reference feature to be fused similarity, and taking the image reference features of the first type in the image reference feature library and the image reference features to be fused corresponding to the image reference features of the first type in the image reference feature library as the corresponding image reference feature to be fused similarity. The second fusion similarity is used to characterize the fusion of the maximum similarity between the image reference features of the second type, and the second fusion similarity is obtained by adding and averaging the similarities to be fused corresponding to the image reference features of the second type in the image reference feature library, and calculating the average value according to the calculated similarity average value; the state image includes at least one of an electrocardiogram, a phonocardiogram, and a pulse wave graph; For each image reference feature of the reference image matching the state image in the image reference feature library, calculate the similarity between the image state feature and the image reference feature, take the maximum similarity between the image state feature and the image reference features belonging to the first type as the first type similarity, and take the maximum similarity between the image state feature and the image reference features belonging to the second type as the second type similarity; If the first type similarity is greater than a fusion value between a first preset value and the first fusion similarity, and the second type similarity is greater than a fusion value between a second preset value and the second fusion similarity, then based on a comparison result of the first type similarity and the second type similarity, a prediction result of the state reference information corresponding to the object to be predicted is determined; If the first type similarity is greater than the fusion value between the first preset value and the first fusion similarity, and the second type similarity is less than the fusion value between the second preset value and the second fusion similarity, determining that the state reference information prediction result is that the state reference information of the object to be predicted is of the first type; If the first type similarity is smaller than the fusion value between the first preset value and the first fusion similarity, and the second type similarity is larger than the fusion value between the second preset value and the second fusion similarity, then it is determined that the state reference information prediction result is that the state reference information of the object to be predicted is of the second type.

2. The method according to claim 1, characterized in that The method further comprises: Data related to heart disease detection in multiple subjects diagnosed with heart disease are screened from medical data, and data related to heart disease detection in a preset number of subjects are randomly selected from a group of normal subjects, and the relevant data are added to the image reference feature library; the subjects diagnosed with heart disease correspond to the first type of representation, and the normal subjects in the group of normal subjects correspond to the second type of representation.

3. The method according to claim 1, characterized in that The determining, based on the comparison result of the first type of similarity and the second type of similarity, a prediction result of the state reference information corresponding to the object to be predicted includes: If the first type similarity is greater than the second type similarity, determining that the state reference information prediction result is that the state reference information of the object to be predicted is of the first type; If the first type similarity is less than or equal to the second type similarity, it is determined that the state reference information prediction result is that the state reference information of the object to be predicted is of the second type.

4. The method according to claim 1, characterized in that: The method further comprises: Acquire a first type of fusion feature and a second type of fusion feature, wherein the first type of fusion feature is used to characterize the fusion of image reference features belonging to the first type in the image reference feature library, and the second type of fusion feature is used to characterize the fusion of image reference features belonging to the second type in the image reference feature library; Calculating a third similarity between the image state feature and the first type of fused feature, and calculating a fourth similarity between the image state feature and the second type of fused feature; If the first type of similarity is smaller than the fusion value between the first preset value and the first fusion similarity, and the second type of similarity is smaller than the fusion value between the second preset value and the second fusion similarity, then based on the comparison result of the third similarity and the fourth similarity, the state reference information prediction result corresponding to the object to be predicted is determined.

5. The method according to claim 4, characterized in that The determining, based on the comparison result of the third similarity and the fourth similarity, a prediction result of the state reference information corresponding to the object to be predicted includes: If the third similarity is greater than the fourth similarity, determining that the state reference information prediction result is that the state reference information of the object to be predicted is of the first type; If the third similarity is less than or equal to the fourth similarity, it is determined that the state reference information prediction result is that the state reference information of the object to be predicted is of the second type.

6. A device for predicting state reference information of artificial intelligence under multi-modality, characterized in that: The device comprises: The acquisition module is used to acquire the image state features of the state image corresponding to the object to be predicted, and acquire the first fusion similarity corresponding to the image reference features of the first type in the image reference feature library, and the second fusion similarity corresponding to the image reference features of the second type; the first fusion similarity is used to characterize the fusion of the maximum similarity between the image reference features of the first type, and the first fusion similarity is obtained by combining each image reference feature of the first type in the image reference feature library with the maximum similarity among other image reference features of the first type as the corresponding image reference feature to be fused, and combining each image reference feature of the first type in the image reference feature library with the maximum similarity among other image reference features of the first type. The corresponding similarities to be fused are added and averaged, and the result is obtained according to the calculated average similarity value; the second fused similarity is used to characterize the fusion of the maximum similarity between the image reference features of the second type, and the second fused similarity is obtained by taking each image reference feature of the second type in the image reference feature library and the maximum similarity among other image reference features of the second type as the similarity to be fused of the corresponding image reference feature, and adding and averaging the similarities to be fused corresponding to each image reference feature of the second type in the image reference feature library, and the result is obtained according to the calculated average similarity value; the state image includes at least one of an electrocardiogram, a phonocardiogram, and a pulse wave graph; a calculation module, configured to calculate, for each image reference feature of the reference image matching the state image in the image reference feature library, a similarity between the image state feature and the image reference feature, taking the maximum similarity between the image state feature and the image reference features belonging to the first type as the first type similarity, and taking the maximum similarity between the image state feature and the image reference features belonging to the second type as the second type similarity; A comparison module, used for determining, based on the comparison result of the first type similarity and the second type similarity, a state reference information prediction result corresponding to the object to be predicted if the first type similarity is greater than the fusion value between the first preset value and the first fusion similarity, and the second type similarity is greater than the fusion value between the second preset value and the second fusion similarity; if the first type similarity is greater than the fusion value between the first preset value and the first fusion similarity, and the second type similarity is less than the fusion value between the second preset value and the second fusion similarity, determining that the state reference information prediction result is that the state reference information of the object to be predicted is of the first type; if the first type similarity is less than the fusion value between the first preset value and the first fusion similarity, and the second type similarity is greater than the fusion value between the second preset value and the second fusion similarity, determining that the state reference information prediction result is that the state reference information of the object to be predicted is of the second type.

7. The device according to claim 6, characterized in that The comparison module is further configured to determine, if the first type similarity is greater than the second type similarity, that the state reference information prediction result is that the state reference information of the object to be predicted is of the first type; If the first type similarity is less than or equal to the second type similarity, it is determined that the state reference information prediction result is that the state reference information of the object to be predicted is of the second type.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.