Multi-mode intelligent health consultation interaction system and method based on AI intelligence

Abstract

The invention belongs to the technical field of online health consultation, and particularly relates to a multi-mode intelligent health consultation interaction system and method based on AI intelligence. According to the method, the accuracy of health state assessment is improved by fusing the multi-modal health data, the health state of the user can be more comprehensively understood specifically through comprehensive analysis of image and text data, so that more accurate health consultation is provided, in addition, when the health state of the user is assessed to be abnormal, an alarm mechanism can be triggered in time, and the user experience is improved. According to the method, the health data of the user can be recorded and displayed and the future health trend can be predicted in a normal state, so that the health consultation is more personalized and intelligent, and the user is helped to better manage the health of himself / herself.

Description

Technical Field

[0001] The present invention belongs to the technical field of online health consultation, and specifically relates to a multimodal intelligent health consultation interaction system and method based on AI intelligence. Background Art

[0002] With the development of science and technology, people's demand for health management and disease prevention is growing. Traditional health consultation methods often rely on the doctor's personal experience and knowledge, and cannot provide comprehensive and personalized health assessments. At the same time, the efficiency of doctors' diagnosis is also limited by time and energy and cannot meet the needs of all patients. Therefore, it is particularly important to develop an intelligent system that can automatically collect, analyze and respond to user health data.

[0003] The health consultation system in the existing technology may not be able to monitor the user's health status in real time, or lack sufficient accuracy and personalization in data processing and analysis. It can only realize statistical patient case data but cannot provide timely health risk warnings, resulting in users being unable to receive timely intervention and treatment when health problems occur. Based on this, this solution provides a multimodal intelligent health consultation interactive system based on AI intelligence to solve the above problems. Summary of the invention

[0004] The purpose of the present invention is to provide a multimodal intelligent health consultation interactive system and method based on AI intelligence, which can automatically collect a variety of health data of users and monitor and evaluate the health status of users in real time through corresponding data analysis.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A multimodal intelligent health consultation interaction method based on AI intelligence, comprising:

[0007] Acquire health data of the user, wherein the health data includes image data and text data;

[0008] Performing feature extraction on the image data and the text data to obtain first feature data and second feature data;

[0009] Screening the first feature data and the second feature data to remove invalid items in the first feature data and the second feature data;

[0010] Performing a fusion analysis based on the first feature data and the second feature data to obtain the user's health status, wherein the health status includes a normal state and an abnormal state;

[0011] In the normal state, the user's health data is recorded, and the predicted health data is generated synchronously, and then pushed to the interactive interface for display;

[0012] In the abnormal state, an alarm mechanism is triggered, and the user's health data deviation is collected and synchronously pushed to the interactive interface for display.

[0013] In a preferred solution, after the user's health data is output, data preprocessing is performed synchronously, and the preprocessing step includes:

[0014] Performing denoising processing on the user's health data to remove noise and interference in the health data;

[0015] Perform image enhancement and natural language processing on the denoised health data to enhance the accuracy and readability of the health data and obtain transitional data;

[0016] The transition data is standardized to unify the format of the image data and the semantic structure of the text data.

[0017] In a preferred embodiment, the step of extracting features from the image data and text data to obtain first feature data and second feature data includes:

[0018] Obtain user's image data and text data;

[0019] Obtain image requirement features and text requirement features, and compare them with image data and text data respectively;

[0020] Extracting a region in the image data that is consistent with the required image feature and recording it as first feature data;

[0021] The content consistent with the text requirement feature in the text data is extracted and recorded as second feature data.

[0022] In a preferred solution, the step of screening the first feature data and the second feature data to remove invalid items in the first feature data and the second feature data includes:

[0023] Acquire the first characteristic data and the second characteristic data, and arrange the first characteristic data and the second characteristic data in a time sequence of occurrence;

[0024] Grouping the first characteristic data and the second characteristic data in pairs according to the arrangement order of the first characteristic data and the second characteristic data to obtain a plurality of data groups to be evaluated;

[0025] Comparing the first characteristic data and the second characteristic data in each of the data groups to be evaluated to obtain a first characteristic deviation and a second characteristic deviation corresponding to the first characteristic data;

[0026] Acquire a first evaluation threshold corresponding to the first feature data, and a second evaluation threshold corresponding to the second feature data;

[0027] When the first feature deviation is greater than a first evaluation threshold, it indicates that the first feature data corresponding to the first feature deviation is an invalid item; otherwise, the first feature data corresponding to the first feature deviation is a valid item;

[0028] When the second feature deviation is greater than the second evaluation threshold, it indicates that the second feature data corresponding to the second feature deviation is an invalid item. Otherwise, the second feature data corresponding to the second feature deviation is a valid item.

[0029] In a preferred solution, the step of performing fusion analysis based on the first feature data and the second feature data to obtain the user's health status includes:

[0030] Acquire first feature data and second feature data, and perform vectorization processing to obtain a first feature vector corresponding to the first feature data and a second feature vector corresponding to the second feature data;

[0031] Obtaining a comprehensive evaluation function, and inputting the first feature vector and the second feature vector into the comprehensive evaluation function together, and recording an output result of the comprehensive evaluation function as a comprehensive evaluation score;

[0032] Obtaining a classification threshold, and comparing the comprehensive evaluation score with the classification threshold;

[0033] When the comprehensive evaluation score is greater than the classification threshold, it indicates that the user's health status is abnormal;

[0034] When the comprehensive evaluation score is less than or equal to the classification threshold, it indicates that the user's health status is normal.

[0035] In a preferred solution, in the normal state, the first characteristic data and the second characteristic data are collected and arranged in a time sequence of occurrence;

[0036] Get demand forecast duration;

[0037] A prediction function is obtained, and the demand prediction duration, the first characteristic data, and the second characteristic data are input into the prediction function to obtain first predicted health data corresponding to the first characteristic data, and second predicted health data corresponding to the second characteristic data.

[0038] In a preferred embodiment, after the first characteristic data and the second characteristic data in the normal state are arranged, they are input into a preset trend evaluation function to obtain a first trend parameter corresponding to the first characteristic data and a second trend parameter corresponding to the second characteristic data;

[0039] Randomly select multiple groups of adjacent first characteristic data and second characteristic data, and record them as a first verification data group and a second verification data group respectively;

[0040] Obtaining a calibration function, and performing calculations in combination with the first calibration data group and the second calibration data group to obtain a first calibration offset corresponding to the first calibration data and a second calibration offset corresponding to the second calibration data;

[0041] Obtaining a first verification threshold corresponding to the first verification data and a second verification threshold corresponding to the second verification data, and comparing the first verification offset with the first verification threshold, and comparing the second verification offset with the second verification threshold;

[0042] When the first verification deviation is greater than or equal to the first verification threshold, it indicates that the amount of data of the first verification data group involved in calculating the first verification deviation is too much, and the amount of data of the first verification data group is reduced, and the first trend parameter is recalculated until the first verification deviation is less than the first verification threshold;

[0043] When the second verification bias is greater than the second verification threshold, it indicates that the amount of data in the second verification data group involved in calculating the second verification bias is too much, and the amount of data in the second verification data group is reduced, and the second trend parameter is recalculated until the second verification bias is less than the second verification threshold.

[0044] In a preferred solution, in the abnormal state, all health data deviations are recorded, wherein the health data deviations include a first data deviation corresponding to the first characteristic data, and a second data deviation corresponding to the second characteristic data;

[0045] When the first data offset and the second data offset are pushed to the interactive interface, both are highlighted.

[0046] The present invention also provides a multimodal intelligent health consultation interaction system based on AI intelligence, using the above-mentioned multimodal intelligent health consultation interaction method based on AI intelligence, including:

[0047] A data acquisition module, the data acquisition module is used to acquire the user's health data, wherein the health data includes image data and text data;

[0048] A feature extraction module, the feature extraction module is used to extract features from the image data and the text data to obtain first feature data and second feature data;

[0049] A data screening module, the data screening module is used to screen the first characteristic data and the second characteristic data to remove invalid items in the first characteristic data and the second characteristic data;

[0050] A comprehensive analysis module, the comprehensive analysis module is used to perform a fusion analysis based on the first feature data and the second feature data to obtain the user's health status, wherein the health status includes a normal state and an abnormal state;

[0051] A first push module, the first push module is used to record the user's health data in the normal state, and simultaneously generate predicted health data, and then synchronously push it to the interactive interface for display;

[0052] The second push module is used to trigger an alarm mechanism in the abnormal state, collect the user's health data deviation, and synchronously push it to the interactive interface for display.

[0053] And, an electronic device, the electronic device comprising:

[0054] at least one processor;

[0055] and a memory communicatively coupled to the at least one processor;

[0056] Among them, the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the above-mentioned AI-based multimodal intelligent health consultation interaction method.

[0057] The technical effects achieved by the present invention are:

[0058] The present invention improves the accuracy of health status assessment by fusing multimodal health data. Specifically, through comprehensive analysis of image and text data, it can more comprehensively understand the user's health status, thereby providing more accurate health consultation. In addition, when the user's health status is assessed as abnormal, it can promptly trigger an alarm mechanism and highlight the abnormal part to ensure that the user can receive timely attention and processing. Under normal conditions, it can not only record and display the user's health data, but also predict future health trends, making health consultation more personalized and intelligent, and helping users better manage their own health. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a schematic flow chart of the method of the present invention;

[0060] Figure 2 It is a schematic diagram of the system module of the present invention;

[0061] Figure 3It is a schematic diagram of the structure of an electronic device of the present invention. DETAILED DESCRIPTION

[0062] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0063] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0064] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present invention. The phrase "in a preferred embodiment" that appears in different places in this specification does not refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0065] See also Figure 1 As shown, the present invention provides a multi-modal intelligent health consultation interaction method based on AI intelligence, comprising:

[0066] S1. Obtaining health data of the user, wherein the health data includes image data and text data;

[0067] In step S1, during the process of smart health consultation, the user's health data needs to be collected first. The health data is multimodal, including image data and text data. The image data can be medical images obtained by various imaging devices, such as X-rays, CT scan images, MRI images, etc. The text data may include the user's medical records, physical examination reports, doctor's diagnosis opinions, etc. After the user's health data is output, data preprocessing is performed synchronously. The preprocessing steps include:

[0068] De-noising the user's health data to remove noise and interference in the health data;

[0069] Perform image enhancement and natural language processing on the denoised health data to enhance the accuracy and readability of the health data and obtain transitional data;

[0070] Standardize the transition data, unify the format of image data, and the semantic structure of text data;

[0071] Specifically, after the user's health data is output, a series of data preprocessing will be performed to ensure the quality and availability of the data. First, the user's health data will be denoised to remove possible noise and interference factors in the data and ensure the purity of the data. Secondly, the denoised health data will be further subjected to image enhancement and natural language processing. Image enhancement processing aims to improve the clarity and contrast of image data, while natural language processing is used to optimize the expression and semantic understanding of text data, thereby improving the accuracy and readability of the data and obtaining transition data. Finally, the transition data will be standardized in order to unify the format of image data and ensure the consistency and compatibility of images. At the same time, the semantic structure of text data will be normalized to facilitate subsequent data analysis and processing.

[0072] S2, extracting features from the image data and the text data to obtain first feature data and second feature data;

[0073] In step S2, the acquired image data and text data are subjected to in-depth feature extraction to obtain first feature data and second feature data. Feature extraction is to identify and extract key information in the data through a series of algorithms, which is crucial for subsequent health status analysis. The steps of extracting features from the image data and text data to obtain the first feature data and the second feature data include:

[0074] Obtain user's image data and text data;

[0075] Obtain image requirement features and text requirement features, and compare them with image data and text data respectively;

[0076] Extracting an area in the image data that is consistent with the required image features and recording it as first feature data;

[0077] Extracting the content consistent with the text requirement feature from the text data and recording it as the second feature data;

[0078] Specifically, when processing image data and text data, it is first necessary to perform feature extraction to obtain the first feature data and the second feature data. First, it is necessary to obtain the user's image data and text data, and it is also necessary to clarify the image requirement features and the text requirement features. Then, the requirement features are compared with the image data and the text data respectively to determine which parts meet the requirements. For image data, the area consistent with the image requirement features will be extracted, and the extracted area will be recorded as the first feature data. Similarly, for text data, the content consistent with the text requirement features will be extracted, and the extracted content will be recorded as the second feature data for subsequent analysis and calling.

[0079] S3, screening the first feature data and the second feature data to remove invalid items in the first feature data and the second feature data;

[0080] In step S3, after feature extraction, the first feature data and the second feature data are screened to remove invalid items therein to ensure the accuracy and efficiency of the analysis process. The invalid items may include unclear images and irrelevant text information. The first feature data and the second feature data are screened to remove invalid items therein, including:

[0081] Acquire the first characteristic data and the second characteristic data, and arrange the first characteristic data and the second characteristic data in a time sequence of occurrence;

[0082] Grouping the first characteristic data and the second characteristic data in pairs according to the arrangement order of the first characteristic data and the second characteristic data to obtain a plurality of data groups to be evaluated;

[0083] Comparing the first characteristic data and the second characteristic data in each of the data groups to be evaluated to obtain a first characteristic deviation and a second characteristic deviation corresponding to the first characteristic data;

[0084] Acquire a first evaluation threshold corresponding to the first feature data, and a second evaluation threshold corresponding to the second feature data;

[0085] When the first feature deviation is greater than the first evaluation threshold, it indicates that the first feature data corresponding to the first feature deviation is an invalid item, otherwise, the first feature data corresponding to the first feature deviation is a valid item;

[0086] When the second feature deviation is greater than the second evaluation threshold, it indicates that the second feature data corresponding to the second feature deviation is an invalid item, otherwise, the second feature data corresponding to the second feature deviation is a valid item;

[0087] Specifically, when screening the first feature data and the second feature data, it is first necessary to screen the first feature data and the second feature data to remove invalid items therein. First, it is necessary to obtain the first feature data and the second feature data and arrange them in the order of their occurrence. Then, the first feature data and the second feature data are grouped in pairs according to the arrangement order of the first feature data and the second feature data, so as to obtain multiple data groups to be evaluated. Then, it is necessary to perform corresponding comparison and analysis on the first feature data and the second feature data in each data group to be evaluated, so as to obtain a first feature deviation corresponding to the first feature data and a second feature deviation corresponding to the second feature data. In order to further evaluate the validity of the first feature data and the second feature data, it is necessary to obtain a preset first evaluation threshold corresponding to the first feature data and a second evaluation threshold corresponding to the second feature data. Then, the validity of the first feature data can be judged by comparing the size of the first feature deviation and the first evaluation threshold. Specifically, If the first feature deviation is greater than the first evaluation threshold, then the first feature data corresponding to the first feature deviation is considered invalid. Conversely, if the first feature deviation is less than or equal to the first evaluation threshold, then the first feature data corresponding to the first feature deviation is considered valid. The same method is also applicable to the second feature data. By comparing the second feature deviation with the second evaluation threshold, the validity of the second feature data can be judged. If the second feature deviation is greater than the second evaluation threshold, then the second feature data corresponding to the second feature deviation is considered invalid. If the second feature deviation is less than or equal to the second evaluation threshold, then the second feature data corresponding to the second feature deviation is considered valid. Among them, the first evaluation threshold and the second evaluation threshold can be obtained based on statistical analysis of historical data, or can be set according to actual needs in a specific application scenario. Through this screening process, the accuracy and efficiency of data processing can be effectively improved, ensuring the accuracy and reliability of subsequent health consultation interactions.

[0088] S4, performing a fusion analysis based on the first feature data and the second feature data to obtain the user's health status, wherein the health status includes a normal state and an abnormal state;

[0089] In step S4, after the first feature data and the second feature data are screened, a corresponding fusion analysis is performed based on the first feature data and the second feature data, and the health status of the user is reflected according to the result of the fusion analysis. In this embodiment, the evaluation result of the health status is divided into a normal state and an abnormal state, providing the user with an intuitive overview of the health status. The step of performing a fusion analysis based on the first feature data and the second feature data to obtain the health status of the user includes:

[0090] Acquire the first feature data and the second feature data, and perform vectorization processing to obtain a first feature vector corresponding to the first feature data and a second feature vector corresponding to the second feature data;

[0091] Obtaining a comprehensive evaluation function, and inputting the first eigenvector and the second eigenvector into the comprehensive evaluation function together, and recording the output result of the comprehensive evaluation function as a comprehensive evaluation score;

[0092] Obtaining a classification threshold, and comparing the comprehensive evaluation score with the classification threshold;

[0093] When the comprehensive evaluation score is greater than the classification threshold, it indicates that the user's health status is abnormal;

[0094] When the comprehensive evaluation score is less than or equal to the classification threshold, it indicates that the user's health status is normal;

[0095] Specifically, in order to accurately evaluate the health status of the user, it is first necessary to collect the first characteristic data and the second characteristic data of the user, and then the first characteristic data and the second characteristic data will be converted into numerical form for subsequent calculation processing. Specifically, the first characteristic data and the second characteristic data will be vectorized to obtain the first characteristic vector and the second characteristic vector, which correspond to the first characteristic data and the second characteristic data, respectively. Then, a pre-set comprehensive evaluation function is introduced. The comprehensive evaluation function can comprehensively consider the information of the first characteristic vector and the second characteristic vector, and analyze and process them. The expression of the comprehensive evaluation function is: In the formula, S z represents the comprehensive evaluation score, α and β represent the weight coefficients of the first eigenvector and the second eigenvector, n and m represent the number of feature points in the first eigenvector and the second eigenvector, respectively. i and b i Respectively represent the first eigenvector and the second eigenvector, pass the first eigenvector and the second eigenvector as input parameters to the comprehensive evaluation function, and record the output result of the comprehensive evaluation function as a comprehensive evaluation score. In order to judge the health status of the user, it is also necessary to introduce a preset classification threshold as a reference standard, and then compare the comprehensive evaluation score with the preset classification threshold to determine the health status of the user. If the comprehensive evaluation score exceeds the classification threshold, it indicates that the health status of the user is abnormal and requires further examination or attention. On the contrary, if the comprehensive evaluation score is less than or equal to the classification threshold, it can be considered that the health status of the user is normal and no special medical intervention is currently required;

[0096] S5. Under normal conditions, the user's health data is recorded, and the predicted health data is generated synchronously, and then pushed to the interactive interface for display;

[0097] In step S5, when the user's health status is evaluated as normal, the user's health data will be recorded, and the predicted health data will be calculated and generated synchronously. The predicted health data is calculated based on the user's current health status and historical data to predict the user's health trend in the future. The predicted health data and the user's current health data will then be pushed to the interactive interface for display, and the user can intuitively see his current health status and future trends. In the normal state, the first feature data and the second feature data are collected and arranged in chronological order.

[0098] Get demand forecast duration;

[0099] Obtain a prediction function, and input the demand prediction duration, the first characteristic data, and the second characteristic data into the prediction function to obtain first predicted health data corresponding to the first characteristic data, and second predicted health data corresponding to the second characteristic data;

[0100] Specifically, under normal working conditions, it is necessary to collect the first characteristic data and the second characteristic data, and ensure that the first characteristic data and the second characteristic data are arranged in the order of occurrence. Then, it is necessary to obtain the duration of demand forecasting, which will be used as one of the parameters for subsequent forecasting. Then, it is necessary to introduce a pre-set prediction function, which will be used to process the first characteristic data and the second characteristic data that have been collected. The expression of the prediction function is: HL = F m +QT, where HL represents the first predicted health data or the second predicted health data (specifically, it needs to be selected according to the parameters input into the prediction function. For example, if the input into the prediction function is the first feature data, then the output result of the prediction function is the first predicted health data. Similarly, the processing method and output result of the second feature data are the same). m represents the first characteristic data or the second characteristic data at the end of the arrangement order, Q represents the first trend parameter corresponding to the first characteristic data or the second trend parameter corresponding to the second characteristic data, T represents the demand forecast duration, through the calculation of the prediction function, the first predicted health data corresponding to the first characteristic data and the second predicted health data corresponding to the second characteristic data can be obtained, and then the user's future health status can be evaluated according to the first predicted health data and the second predicted health data. In addition, after the first characteristic data and the second characteristic data in the normal state are arranged, they are input into the preset trend evaluation function to obtain the first trend parameter corresponding to the first characteristic data and the second trend parameter corresponding to the second characteristic data;

[0101] Randomly select multiple groups of adjacent first characteristic data and second characteristic data, and record them as a first verification data group and a second verification data group respectively;

[0102] Obtain a calibration function, and perform calculations in combination with the first calibration data group and the second calibration data group to obtain a first calibration offset corresponding to the first calibration data and a second calibration offset corresponding to the second calibration data;

[0103] Obtaining a first verification threshold corresponding to the first verification data and a second verification threshold corresponding to the second verification data, and comparing the first verification offset with the first verification threshold, and comparing the second verification offset with the second verification threshold;

[0104] When the first verification deviation is greater than or equal to the first verification threshold, it indicates that the amount of data of the first verification data group involved in calculating the first verification deviation is too much, and the amount of data of the first verification data group is reduced, and the first trend parameter is recalculated until the first verification deviation is less than the first verification threshold;

[0105] When the second verification deviation is greater than the second verification threshold, it indicates that the amount of data in the second verification data group involved in calculating the second verification deviation is too much, and the amount of data in the second verification data group is reduced, and the second trend parameter is recalculated until the second verification deviation is less than the second verification threshold;

[0106] In the above, after the first characteristic data and the second characteristic data are arranged, the first characteristic data and the second characteristic data are input into a pre-set trend evaluation function. Through the calculation of the trend evaluation function, the first trend parameter corresponding to the first characteristic data can be obtained, and the second trend parameter corresponding to the second characteristic data can also be obtained. The expression of the trend evaluation function is: Wherein, t represents the span length of the first feature data or the second feature data, r represents the number of the first feature data or the second feature data, and U k and U k-1 Represents adjacent first feature data or adjacent second feature data, and then it is necessary to randomly select multiple groups of adjacent first feature data and second feature data and record them as the first verification data group and the second verification data group respectively. In order to further verify the accuracy of the first feature data and the second feature data, it is necessary to introduce a pre-set verification function. The verification function will be combined with the first verification data group and the second verification data group for calculation, so as to obtain the first verification deviation corresponding to the first verification data and the second verification deviation corresponding to the second verification data. The expression of the verification function is: U p =U h -(U q +QT j ), where U p Indicates the first calibration offset, U h represents the first characteristic data with a lower ranking in the first verification data group, or the second characteristic data with a lower ranking in the second verification data group, Uq represents the first feature data with a higher ranking in the first verification data group, or the second feature data with a higher ranking in the second verification data group, T j It represents the time interval between adjacent first characteristic data in the first verification data group, or the time interval between adjacent second characteristic data in the first verification data group. Then, it is necessary to obtain the first verification threshold corresponding to the first verification data and the second verification threshold corresponding to the second verification data, and then compare the first verification deviation with the first verification threshold, and at the same time compare the second verification deviation with the second verification threshold, so as to judge whether the first trend parameter or the second trend parameter is within an acceptable error range. If it is found that the first verification deviation is greater than or equal to the first verification threshold, it means that the amount of data in the first verification data group involved in calculating the first verification deviation is too much. In this case, it is necessary to reduce the first verification data group to reduce the amount of data, and then recalculate. The first trend parameter is calculated until the first check bias is less than the first check threshold. Only when the first check bias is less than the first check threshold will the adjustment and recalculation be stopped. Similarly, if it is found that the second check bias is greater than the second check threshold, it means that the amount of data in the second check data group involved in calculating the second check bias is too much. In this case, it is necessary to reduce the second check data group to reduce the amount of data, and then recalculate the second trend parameter until the second check bias is less than the second check threshold. Only when the second check bias is less than the second check threshold will the adjustment and recalculation be stopped, thereby ensuring that the final first trend parameter and the second trend parameter have high accuracy and reliability, providing a solid foundation for subsequent decision-making and analysis.

[0107] S6. In an abnormal state, the alarm mechanism is triggered, and the user's health data deviation is collected and pushed to the interactive interface for display;

[0108] In step S6, if the user's health status is evaluated as abnormal, an alarm mechanism will be triggered synchronously. The purpose of the alarm mechanism is to promptly notify the user or medical personnel so that further inspection or treatment measures can be taken. At the same time, the user's health data deviation will be collected, that is, the difference in health data compared with the normal state. The health data deviation will also be synchronously pushed to the interactive interface for display so that the user and medical personnel can quickly understand the specific details of the abnormal situation. In the abnormal state, all health data deviations are recorded, wherein the health data deviation includes a first data deviation corresponding to the first feature data, and a second data deviation corresponding to the second feature data. In order to attract the user's attention, the first data deviation and the second data deviation are both highlighted when they are pushed to the interactive interface.

[0109] See also Figure 2, a multimodal intelligent health consultation interactive system based on AI intelligence, using the above-mentioned multimodal intelligent health consultation interactive method based on AI intelligence, including:

[0110] A data acquisition module, which is used to acquire the user's health data, wherein the health data includes image data and text data;

[0111] A feature extraction module, the feature extraction module is used to extract features from the image data and the text data to obtain first feature data and second feature data;

[0112] A data screening module, the data screening module is used to screen the first characteristic data and the second characteristic data to remove invalid items in the first characteristic data and the second characteristic data;

[0113] A comprehensive analysis module, which is used to perform a fusion analysis based on the first feature data and the second feature data to obtain the user's health status, wherein the health status includes a normal state and an abnormal state;

[0114] The first push module is used to record the user's health data under normal conditions, and simultaneously generate predicted health data, and then simultaneously push it to the interactive interface for display;

[0115] The second push module is used to trigger the alarm mechanism under abnormal conditions, collect the user's health data deviation, and simultaneously push it to the interactive interface for display.

[0116] In the above, the system includes a data acquisition module, a feature extraction module, a data screening module, a comprehensive analysis module, a first push module and a second push module. The data acquisition module is responsible for collecting the user's health data. The health data covers various types, including image data and text data. The image data can be the user's physical examination photos, X-rays, MRI images, etc., while the text data may include the user's medical records, physical examination reports, daily health logs, etc. The feature extraction module will perform in-depth feature extraction processing on the acquired image data and text data, extract key visual features from the image data to form the first feature data, and at the same time, extract key semantic features from the text data to form the second feature data to determine the basis for subsequent analysis and decision-making. The data screening module will perform detailed screening processing on the first feature data and the second feature data to ensure the accuracy and reliability of subsequent analysis by removing invalid items. The comprehensive analysis module comprehensively considers the first feature data. The system collects the user's health data and the second feature data, and performs fusion analysis to accurately assess the user's health status. The health status can be divided into two categories: normal status and abnormal status. For the normal status, the system will record the user's health data and generate predicted health data based on these data. Under normal conditions, the first push module will synchronously record and generate the user's health data and predicted health data, and then push this information to the interactive interface for display. The user can intuitively understand his or her health status through the interactive interface. Under abnormal conditions, the second push module will immediately trigger the alarm mechanism. The system will collect the user's health data deviation, that is, abnormal data changes compared with the normal state, and synchronously push this information to the interactive interface for display, so that the user can understand his or her health status in time and take necessary measures. In this way, the system can provide users with comprehensive, accurate and timely health consulting services, greatly improving the efficiency and quality of health management.

[0117] See also Figure 3 , an electronic device, the electronic device comprising:

[0118] at least one processor;

[0119] and a memory communicatively coupled to the at least one processor;

[0120] Among them, the memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the above-mentioned AI-based multimodal intelligent health consultation interaction method.

[0121] The processor of the above-mentioned electronic device can be any type of central processing unit (CPU), such as a processor in a personal computer, server or mobile device. The memory can be a random access memory (RAM), a read-only memory (ROM), a flash memory or other forms of non-volatile memory. The computer program can be encoded as a part of a software application, firmware or an operating system. The electronic device can also include an arithmetic unit, an input device and an output device. The arithmetic unit can perform various computing tasks. The input device allows the user to interact with the electronic device, such as inputting instructions through a keyboard, a mouse or a touch screen. The output device is responsible for presenting the processing results to the user, such as a display, a printer or a speaker. The memory of the electronic device also contains an operating system, which is responsible for managing the hardware resources and software programs of the electronic device to ensure that the computer program can be executed smoothly. In addition, the electronic device can also communicate with other devices or servers through a network interface to achieve remote transmission and reception of data.

[0122] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0123] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention shall be implemented according to the conventional means in the art unless otherwise specified and limited.

Claims

1. A multimodal intelligent health consultation interactive method based on AI intelligence, characterized by: include: Acquire health data of the user, wherein the health data includes image data and text data; Performing feature extraction on the image data and the text data to obtain first feature data and second feature data; Screening the first feature data and the second feature data to remove invalid items in the first feature data and the second feature data; Performing a fusion analysis based on the first feature data and the second feature data to obtain the user's health status, wherein the health status includes a normal state and an abnormal state; In the normal state, the user's health data is recorded, and the predicted health data is generated synchronously, and then pushed to the interactive interface for display; In the abnormal state, an alarm mechanism is triggered, and the user's health data deviation is collected and synchronously pushed to the interactive interface for display.

2. According to claim 1, a multi-modal intelligent health consultation interaction method based on AI intelligence is characterized by: After the user's health data is output, data preprocessing is performed synchronously, and the preprocessing steps include: Performing denoising processing on the user's health data to remove noise and interference in the health data; Perform image enhancement and natural language processing on the denoised health data to enhance the accuracy and readability of the health data and obtain transitional data; The transition data is standardized to unify the format of the image data and the semantic structure of the text data.

3. According to claim 1, a multi-modal intelligent health consultation interaction method based on AI intelligence is characterized by: The step of extracting features from the image data and the text data to obtain first feature data and second feature data comprises: Obtain user's image data and text data; Obtain image requirement features and text requirement features, and compare them with image data and text data respectively; Extracting a region in the image data that is consistent with the required image feature and recording it as first feature data; The content consistent with the text requirement feature in the text data is extracted and recorded as second feature data.

4. The AI-based multi-modal intelligent health consultation interaction method according to claim 1, characterized in that: The step of screening the first feature data and the second feature data to remove invalid items in the first feature data and the second feature data includes: Acquire the first characteristic data and the second characteristic data, and arrange the first characteristic data and the second characteristic data in a time sequence of occurrence; Grouping the first characteristic data and the second characteristic data in pairs according to the arrangement order of the first characteristic data and the second characteristic data to obtain a plurality of data groups to be evaluated; Comparing the first characteristic data and the second characteristic data in each of the data groups to be evaluated to obtain a first characteristic deviation and a second characteristic deviation corresponding to the first characteristic data; Acquire a first evaluation threshold corresponding to the first feature data, and a second evaluation threshold corresponding to the second feature data; When the first feature deviation is greater than a first evaluation threshold, it indicates that the first feature data corresponding to the first feature deviation is an invalid item; otherwise, the first feature data corresponding to the first feature deviation is a valid item; When the second feature deviation is greater than the second evaluation threshold, it indicates that the second feature data corresponding to the second feature deviation is an invalid item. Otherwise, the second feature data corresponding to the second feature deviation is a valid item.

5. The AI-based multi-modal intelligent health consultation interaction method according to claim 1, characterized in that: The step of performing fusion analysis based on the first feature data and the second feature data to obtain the health status of the user includes: Acquire first feature data and second feature data, and perform vectorization processing to obtain a first feature vector corresponding to the first feature data and a second feature vector corresponding to the second feature data; Obtaining a comprehensive evaluation function, and inputting the first feature vector and the second feature vector into the comprehensive evaluation function together, and recording an output result of the comprehensive evaluation function as a comprehensive evaluation score; Obtaining a classification threshold, and comparing the comprehensive evaluation score with the classification threshold; When the comprehensive evaluation score is greater than the classification threshold, it indicates that the user's health status is abnormal; When the comprehensive evaluation score is less than or equal to the classification threshold, it indicates that the user's health status is normal.

6. The AI-based multimodal intelligent health consultation interaction method according to claim 1, characterized in that: In the normal state, the first characteristic data and the second characteristic data are collected and arranged in a time sequence of occurrence; Get demand forecast duration; A prediction function is obtained, and the demand prediction duration, the first characteristic data, and the second characteristic data are input into the prediction function to obtain first predicted health data corresponding to the first characteristic data, and second predicted health data corresponding to the second characteristic data.

7. The AI-based multimodal intelligent health consultation interaction method according to claim 1, characterized in that: After the first characteristic data and the second characteristic data in the normal state are arranged, they are input into a preset trend evaluation function to obtain a first trend parameter corresponding to the first characteristic data and a second trend parameter corresponding to the second characteristic data; Randomly select multiple groups of adjacent first characteristic data and second characteristic data, and record them as a first verification data group and a second verification data group respectively; Obtaining a calibration function, and performing calculations in combination with the first calibration data group and the second calibration data group to obtain a first calibration offset corresponding to the first calibration data and a second calibration offset corresponding to the second calibration data; Obtaining a first verification threshold corresponding to the first verification data and a second verification threshold corresponding to the second verification data, and comparing the first verification offset with the first verification threshold, and comparing the second verification offset with the second verification threshold; When the first verification deviation is greater than or equal to the first verification threshold, it indicates that the amount of data of the first verification data group involved in calculating the first verification deviation is too much, and the amount of data of the first verification data group is reduced, and the first trend parameter is recalculated until the first verification deviation is less than the first verification threshold; When the second verification bias is greater than the second verification threshold, it indicates that the amount of data in the second verification data group involved in calculating the second verification bias is too much, and the amount of data in the second verification data group is reduced, and the second trend parameter is recalculated until the second verification bias is less than the second verification threshold.

8. The AI-based multi-modal intelligent health consultation interaction method according to claim 1, characterized in that: In the abnormal state, all health data deviations are recorded, wherein the health data deviations include a first data deviation corresponding to the first characteristic data and a second data deviation corresponding to the second characteristic data; When the first data offset and the second data offset are pushed to the interactive interface, both are highlighted.

9. A multi-modal intelligent health consultation interactive system based on AI intelligence, characterized by: The AI-based multimodal intelligent health consultation interaction method according to any one of claims 1 to 8 comprises: A data acquisition module, the data acquisition module is used to acquire the user's health data, wherein the health data includes image data and text data; A feature extraction module, the feature extraction module is used to extract features from the image data and the text data to obtain first feature data and second feature data; A data screening module, the data screening module is used to screen the first characteristic data and the second characteristic data to remove invalid items in the first characteristic data and the second characteristic data; A comprehensive analysis module, the comprehensive analysis module is used to perform a fusion analysis based on the first feature data and the second feature data to obtain the user's health status, wherein the health status includes a normal state and an abnormal state; A first push module, the first push module is used to record the user's health data in the normal state, and simultaneously generate predicted health data, and then synchronously push it to the interactive interface for display; The second push module is used to trigger an alarm mechanism in the abnormal state, collect the user's health data deviation, and synchronously push it to the interactive interface for display.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein, the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the AI ​​intelligence-based multimodal intelligent health consultation interaction method described in any one of claims 1 to 8.

Images