Smart cloud outpatient service system and method

By combining time flight technology and self-attention mechanism to optimize the tongue state recognition model, combined with the shooting environment brightness and facial features in self-photo image information, the problem of inaccurate judgment of tongue and mental state in the existing technology is solved, and the diagnosis is improved accuracy and communication efficiency.

CN119069111APending Publication Date: 2024-12-03ZHUHAI HAORUI DIGITAL TECH CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202411082169.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art lacks the consideration of shooting distance and facial features when identifying the patient's tongue and mental state, resulting in inaccurate judgment and reduced the accuracy and rationality of the diagnosis.

Method used

The tongue state recognition model is optimized by time flight technology and self-attention mechanism. Combined with the shooting environment brightness, eyelid opening degree and mouth dryness in self-timer image information, the tongue and mental state are identified through the pre-trained model, and it is recommended to be appropriate. doctor information.

Benefits of technology

It improves the accuracy of recognition of tongue and mental state, reduces the workload of doctors, improves the communication efficiency of online consultation and the rationality of doctor recommendations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119069111A_ABST
    Figure CN119069111A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of online health diagnosis, and discloses a smart cloud outpatient service system and method. The method comprises the steps of collecting basic information of a patient; various sensors are used for collecting body characteristics of a patient; selfie image information and tongue image information sent by the patient are received, the tongue image information of the patient is input into the trained tongue state recognition model, the tongue state of the patient is obtained, and the tongue state comprises the tongue fur fat and thin degree, the tongue surface color and the tongue edge smoothness; inputting the selfie image information into a pre-trained patient mental state recognition model to obtain a mental state category; the accuracy of identifying the tongue image information of the patient is effectively improved, the reasonability of doctor recommendation is improved, and the doctor can further judge and predict the illness condition of the patient.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of online health diagnosis technology, and more specifically, to a smart cloud outpatient system and method. Background Art

[0002] The prior art is as follows:

[0003] Publication number CN104490361A, IPC classification number A61B5 / 00, Chinese patent discloses a remote screening method for skin diseases based on a network hospital, including: an information collection device collects the patient's skin information; the information transmission device sends the patient's skin information to the information screening device; the information screening device obtains the skin disease data pre-stored in the skin disease database, performs skin disease screening on the patient's skin information, and returns the screening results to the information collection device through the information transmission device. The invention's remote screening system for skin diseases based on a network hospital allows patients to choose whether to go to the hospital for medical treatment according to their own conditions, so that they will not miss the opportunity to treat the disease due to delays, nor will they go to the hospital in vain due to worries about gains and losses, thereby avoiding unnecessary expenses and cumbersome processes for medical treatment;

[0004] Publication number CN106557746A, IPC classification number G06K9 / 00, Chinese patent discloses a tongue image detection method and system based on image recognition technology, including S1, collecting first tongue image data; S2, operating and processing the collected first tongue image data; S3, comparing the third tongue image data with the tongue image data in the preset tongue image database one by one, and respectively calculating the similarity between the third tongue image data and the tongue image data in the preset tongue image database; S4, obtaining the fourth tongue image data in the tongue image database with the highest similarity to the third tongue image data, and obtaining the tongue image detection report corresponding to the fourth tongue image data. The time data, the illumination value of the surrounding environment, and the tongue length in the tongue image data are used as the preliminary qualified judgment criteria for the tongue image data, and then the second operation processing of matting and color correction is carried out to improve the accuracy of subsequent image analysis and comparison;

[0005] Announcement No. TWM568691U, IPC Classification No. A61B5 / 00, Taiwan Patent discloses a smart health device, which is connected to a remote server and at least one clinic at the same time. The remote server provides multiple big data required by the smart health device; the smart health device includes a graphics processing unit, a feedback disease reinforcement learning module, a pediatric health data module, multiple data modules, a family doctor pediatric clinic interface and a cold patient treatment terminal. The graphics processing unit (GPU) can execute artificial intelligence learning modules to process a large amount of related health data, health information and different health conditions of different medical disciplines. Through the created smart health device, auxiliary information outside the professional field of family doctors can be effectively provided, and through the medical record data fed back by each clinic, learning reinforcement is performed in the multiple health data modules, so that family doctors in the original professional field can initially solve the patient problems in their area of ​​responsibility.

[0006] The existing technology still has the following problems:

[0007] 1. When identifying the tongue status in the patient's tongue image, the impact of the shooting distance on the tongue status is not considered, resulting in inaccurate judgment of the patient's health status;

[0008] 2. The diagnosis is mostly based on the patient's physical characteristics data, such as hospital examination reports, without considering the patient's mental state, which reduces the accuracy of the judgment of the patient's mental state; secondly, it also reduces the rationality of the doctor's recommendation.

[0009] In view of this, the present invention proposes a smart cloud outpatient system and method to solve the above problems. Summary of the invention

[0010] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solutions: a smart cloud outpatient method, comprising:

[0011] Collect basic information of patients;

[0012] Using various sensors to collect the patient's physical characteristics;

[0013] Receive the selfie image information and tongue image information sent by the patient,

[0014] Input the patient's tongue image information into the trained tongue state recognition model to obtain the patient's tongue state, wherein the tongue state includes the thickness of the tongue coating, the color of the tongue surface, and the smoothness of the tongue edge;

[0015] Input the selfie image information into the pre-trained patient mental state recognition model to obtain the mental state category; the mental state category includes fatigue, anxiety and normal;

[0016] According to the symptom description, the disease classification is identified; the patient's age, gender, mental state category and disease classification are input into the pre-trained doctor allocation model to obtain doctor recommendation information, and the doctor recommendation information is sent to the corresponding patient end; the doctor recommendation information includes the doctor's name, doctor's professional field, doctor's working age, doctor's consultation time and doctor's evaluation satisfaction;

[0017] And receive the doctor selected by the patient, determine the corresponding doctor end according to the doctor selected by the patient, establish communication link information between the patient end and the doctor end, and send the communication link information to the patient end and the doctor end; and synchronously send the patient's basic information, physical characteristics and tongue status category to the doctor end.

[0018] Furthermore, the self-portrait image information is obtained from a self-portrait image of the patient in the early morning of the day of the consultation, and the self-portrait image information includes the self-portrait photo, the brightness of the first shooting environment, the brightness of the patient's eyes, the degree of eyelid opening, and the degree of dryness of the mouth;

[0019] The tongue image information includes the tongue photo, shooting distance, brightness of the second shooting environment, shooting angle and pixels of the tongue photo; the selfie photo provides the selfie image itself to be analyzed;

[0020] The shooting distance is the shooting distance between the camera and the patient's tongue when the patient uses the camera to take a tongue photo; the tongue photo is a selfie of the patient sticking out his tongue in the early morning of the consultation day;

[0021] The physical characteristics include blood pressure, body temperature, heart rate, BMI and blood oxygen, etc.

[0022] Furthermore, the method for obtaining the patient's eye brightness includes:

[0023] Locate the face area through pre-trained face detection algorithm;

[0024] Locate the eyes of the face through key point detection algorithm;

[0025] The eye area image is cropped from the original selfie image according to the eye position of the face, and only the pixel information of the eye is retained. The pixel information includes pixel value, pixel density and pixel color space;

[0026] Convert the extracted eye area image from RGB color space to grayscale color space as follows:

[0027] Gray=0.299×R+0.587×G+0.114×B.

[0028] Among them, R, G, and B are the values ​​of the red, green, and blue channels of the pixel respectively; 0.299, 0.587, and 0.114 are the channel weights for red, green, and blue respectively; Gray is the corresponding grayscale value;

[0029] The color value in the original RGB image is replaced by the calculated grayscale value to obtain a grayscale image, in which each pixel in the eye area image corresponds to a grayscale value, expressed as brightness;

[0030] The brightness of the patient's eyes is obtained by calculating the average grayscale value of all pixels in the eye area image;

[0031] The patient's eyelid opening degree method includes:

[0032] The eyelid edge and the lip edge are detected by using the Canny edge detection algorithm to obtain the eyelid edge information and the lip edge information;

[0033] By selecting the median points of the upper and lower eyelid edges and the eyelid edge information, the distance between the upper and lower eyelids is detected using Hough transform, which is the degree of eyelid opening.

[0034] The method for obtaining the degree of mouth dryness comprises:

[0035] The color features of the lip area are collected by counting the color mean and variance of the lip area;

[0036] According to the color characteristics of the lip area, the degree of dryness of the mouth is classified; the judgment thresholds of the color mean and color variance are set. When the color mean is not higher than the preset color threshold and the variance is not lower than the threshold variance threshold, the lip area is dry, otherwise it is moist.

[0037] Furthermore, the method for obtaining the shooting distance includes:

[0038] The time-of-flight technique is used to obtain the distance between the patient's tongue and the camera when the patient's tongue image is taken:

[0039] When the patient takes a tongue image, the camera sends a short pulse of light to the patient's tongue, and the camera records the time when the light pulse is emitted T 1 ;

[0040] The light signal reflected from the patient's tongue surface is received by the camera; the camera records the time when the light signal is received T 2 ;

[0041] By calculating the time difference between the emission and reception of the short pulse of light, the time it takes for the light signal to travel from the camera to the patient's tongue and then back is determined and recorded as the flight time FT, FT = |T 2 -T 1 |;

[0042] By formula Calculate the distance between the patient's tongue and the camera;

[0043] Where DS is the distance between the patient's tongue and the camera; c is the speed of light;

[0044] The method for acquiring the shooting angle includes:

[0045] Use the gyroscope sensor to obtain the shooting angle when taking a picture. The steps are as follows:

[0046] Read the data of the gyroscope sensor, which includes the rotation speed ω(t) of the camera;

[0047] By integrating the formula Get the cumulative rotation angle of the camera;

[0048] Where θ(t 2 ) is the camera in the time period [t 1 , t 2 ]; θ(t 1 ) is the camera at t 1 The initial angle at the moment; ω(t) is the angle of the camera at [t 1 , t 2 ]The rotation speed of the time period.

[0049] Furthermore, the training method of the tongue state recognition model includes:

[0050] Collect a historical tongue data set, the historical tongue data set includes n groups of historical tongue image information, and historical tongue states corresponding to the historical tongue image information, where n is an integer greater than 1; divide the historical tongue data set into a training set and a test set, construct a classifier, use the historical tongue image information in the training set as input data, use the historical tongue states in the training set as output data, train the classifier to obtain an initial classifier, test the initial classifier using the test set, and output a classifier that meets a preset accuracy as a tongue state recognition model; the tongue state recognition model is a network CSPDarknet53 model;

[0051] The collection method of historical tongue image information includes:

[0052] Collect tongue photos of patients under different shooting distances, shooting angles and tongue photo pixels;

[0053] By randomly adjusting the color channels of tongue photos under different shooting distances, shooting angles and tongue photo pixels, changing the brightness and contrast, and simulating the shooting effects under different lighting conditions, tongue photos with different shooting distances, shooting angles, tongue photo pixels and shooting environment brightness were obtained.

[0054] Furthermore, the tongue state recognition model is optimized using the self-attention mechanism. The optimization method includes:

[0055] The historical tongue image information is optimized using the self-attention mechanism; the self-attention module is inserted into the shallow position of the CSPDarknet53 model;

[0056] The methods for building the self-attention mechanism include:

[0057] Define the input dimension n of the self-attention mechanism input and output dimension n output , where n input and n output is an integer greater than or equal to 1;

[0058] n input =n output =5;

[0059] Define input feature tensor X = (A, B, C, D, E); where A is the feature vector of the tongue photo; B is the feature vector of the shooting distance; C is the feature vector of the brightness of the second shooting environment; D is the feature vector of the shooting angle; E is the feature vector of the tongue photo pixel;

[0060] Define the weight matrix W(n) for linear transformation input , n output ), and initialize the weight matrix with He, and initialize the weights of the weight matrix to be uniformly distributed within the range of

[0061] Construct query vector and key vector: Linearly transform and combine the output data of the input layer through the fully connected layer to obtain the query tensor Q, key tensor K and value tensor V;

[0062] Among them, the linear transformation combinations are: Q = X·Wq+b, K = X·Wk+b, V = X·W v +b, where X represents the input feature tensor, W q , W k , W v represents the respective weight matrices, and b represents the bias vector;

[0063] The attention score S(Q, K) is calculated by the dot product between the query tensor Q and the key tensor K. Where d k is the dimension of the key tensor and η is a shared bias vector among the entire attention scores.

[0064] Furthermore, the attention scores are normalized by applying a softmax function to the attention scores to obtain attention weights; the attention score normalization steps include:

[0065]

[0066] Where S(Q, K) i is the i-th score in the vector, S(Q, K) j is the jth score in the vector; L is the length of the attention score vector; i∈L, j∈L.

[0067] The value tensor V is weighted and summed by the attention weight to obtain a weighted value tensor VW, and the elements in the weighted value tensor VW are summed, including A, B, C, D, and E, to obtain a final output tensor OP; the output tensor reflects the degree of attention of the self-attention mechanism to the value tensor given the query tensor and the key tensor;

[0068] The calculation steps of the weighted value tensor VW include:

[0069] VW = softmax(S(Q, K)) i ×V i ;

[0070] Where V i is the i-th element of the value tensor;

[0071]

[0072] Where N is the length of the value tensor V; (VW)k is the kth element of the weighted value tensor;

[0073] The weighted summed value tensor VW is used as the output of the self-attention mechanism and returned to the upper network for use as the data set for the next group of training.

[0074] Furthermore, the training method of the patient mental state recognition model includes:

[0075] Collecting a historical selfie image dataset, where the historical selfie image dataset includes selfie image information and mental state categories corresponding to the selfie image information;

[0076] For self-portraits, a convolutional neural network is used to extract facial features of the face, and the facial features of the face, the brightness of the first shooting environment, the brightness of the patient's eyes, the degree of eyelid opening, the degree of mouth dryness, and the annotated mental state category label are used as feature sequences and used as inputs of the RNN model;

[0077] The RNN model uses a set of mental state category labels corresponding to each set of selfie image information as output, and uses the mental state category labels actually corresponding to each set of selfie image information as the prediction target, and defines the cross entropy loss function Stop training when the RNN model loss function value is less than or equal to the target loss value;

[0078] Among them, y n represents the actual mental state category label corresponding to the nth group of selfie image information, p n Represents the predicted mental state category label probability corresponding to the nth group of selfie image information.

[0079] Further, based on the symptom description, methods for identifying disease classification include:

[0080] Use text processing techniques to identify keywords in the text;

[0081] Establish a medical vocabulary, which includes various disease names, symptom names and their corresponding disease classifications; match the identified keywords with the entries in the medical vocabulary;

[0082] According to the matched disease name, match the corresponding disease classification;

[0083] According to the matching results, the text is divided into corresponding disease classifications and the disease classifications are output.

[0084] Furthermore, the training method of the doctor allocation model includes:

[0085] Get r groups of data, where r is a positive integer greater than 1. The data include the patient's age group, gender, mental state category, disease classification, and doctor recommendation information. Take the patient's age, gender, mental state category, and disease classification as a sample set. Divide the sample set into a training set and a test set. Construct a classifier. Take the patient's age group, gender, mental state category, and disease classification in the training set as input data. Take the doctor recommendation information in the training set as output data. Train the classifier to obtain an initial classifier. Use the test set to test the initial classifier. Output a classifier that meets the preset accuracy as the doctor assignment model. The classifier is a BERT model.

[0086] A smart cloud outpatient system, implementing the smart cloud outpatient method, comprising:

[0087] The patient basic information collection module is used to collect the patient's basic information, including name, age, gender, contact information, historical diagnosis data, family medical history and symptom description;

[0088] The patient's physical characteristics collection module uses various sensors to collect the patient's physical characteristics;

[0089] The image information acquisition module is used to receive the self-portrait image information and tongue image information sent by the patient.

[0090] The tongue state recognition module inputs the patient's tongue image information into the trained tongue state recognition model to obtain the patient's tongue state, which includes the thickness of the tongue coating, the color of the tongue surface, and the smoothness of the tongue edge;

[0091] The patient facial state judgment module inputs the selfie image information into the pre-trained patient mental state recognition model to obtain the mental state category; the mental state category includes fatigue, anxiety and normal;

[0092] The doctor assignment module identifies the disease classification based on the symptom description; inputs the patient's age, gender, mental state category and disease classification into the pre-trained doctor assignment model to obtain doctor recommendation information, and sends the doctor recommendation information to the corresponding patient end; the doctor recommendation information includes the doctor's name, doctor's professional field, doctor's working age, doctor's consultation time and doctor's evaluation satisfaction;

[0093] The remote consultation module receives the doctor selected by the patient, determines the corresponding doctor end according to the doctor selected by the patient, establishes communication link information between the patient end and the doctor end, and sends the communication link information to the patient end and the doctor end; and synchronously sends the patient's basic information, physical characteristics and tongue status category to the doctor end.

[0094] Technical effects and advantages of a smart cloud outpatient system and method of the present invention:

[0095] 1. When identifying the tongue state, the shooting distance, the second shooting environment brightness, the shooting angle and the pixels of the tongue photo are taken into consideration, and their effects on the tongue state are considered. Compared with the existing technology, the patient's tongue state can be identified more accurately. When judging the patient's mental state, the first shooting environment brightness, the patient's eye brightness, the degree of eyelid opening and the degree of mouth dryness are taken into consideration, and their effects on the mental state are considered more comprehensively than the existing technology. The patient's mental state can be identified from more details and facial features, which is more accurate. This enables the subsequent recommendation of doctor recommendation information that is more suitable for the patient based on the patient's mental state and basic information.

[0096] 2. Secondly, when the patient establishes a communication channel with the doctor, the system directly sends the recognized tongue status to the corresponding doctor, reducing the doctor's workload and enabling the doctor to further understand the patient's condition. It also improves the communication efficiency between the patient and the doctor during online consultations.

[0097] 3. When identifying the patient's tongue state, the randomness of each patient's shooting is taken into account. In order to reduce the impact of this random shooting on subsequent identification, the shooting distance is considered and the time-of-flight technology is combined with the patient's tongue state recognition for the first time, which effectively eliminates the impact of random shooting.

[0098] 4. At the same time, the self-attention mechanism was introduced for the first time in the tongue state recognition model for optimization, so that the tongue state recognition model pays more attention to the areas related to the tongue state in the tongue image compared with the existing technology, effectively improving the accuracy of recognizing the patient's tongue image information.

[0099] 5. By adding the patient tongue status recognition module and the patient facial status judgment module, the present invention improves the rationality of the doctor's recommendation on the basis of the original remote consultation, which is beneficial for the doctor to further judge and predict the patient's condition; by adding the doctor allocation module and the remote consultation module, the consultation efficiency is greatly improved. Patients can learn the details according to the provided doctor allocation information, and independently choose the most suitable doctor to see a doctor in time without delaying the condition. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] Figure 1 This is a schematic diagram of data transmission of a smart cloud outpatient system according to Embodiment 1 of the present invention;

[0101] Figure 2 This is a flow chart of a smart cloud outpatient method according to Embodiment 3 of the present invention;

[0102] Figure 3 This is a schematic diagram of an electronic device according to Embodiment 4 of the present invention;

[0103] Figure 4 This is a schematic diagram of a storage medium according to Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0104] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0105] Embodiment 1:

[0106] See also Figure 1 As shown, the smart cloud outpatient system described in this embodiment includes a patient basic information collection module, a patient physical feature collection module, a picture information collection module, a patient tongue status recognition module, a patient facial status judgment module, a doctor allocation module and a remote consultation module, and each module is connected by wire and / or wirelessly.

[0107] The patient basic information collection module is used to collect the patient's basic information, including name, age, gender, contact information, historical diagnosis data, family medical history and symptom description;

[0108] It should be noted that the patient's basic information is collected with the patient's consent and does not involve infringement of personal privacy; that is, the information is collected after signing an informed consent form and a privacy protection agreement. The two agreements jointly ensure that the information collection process is legal and compliant and respects the patient's privacy rights.

[0109] The patient's physical characteristics collection module uses various sensors to collect the patient's physical characteristics, including detailed data on blood pressure, body temperature, heart rate, BMI, and blood oxygen;

[0110] The image information acquisition module is used to receive the self-portrait image information and tongue image information sent by the patient. The self-portrait image information is obtained from the self-portrait image of the patient in the early morning of the consultation day. The self-portrait image information includes the self-portrait photo, the first shooting environment brightness, the patient's eye brightness, the degree of eyelid opening and the degree of mouth dryness; the tongue image information includes the tongue photo, shooting distance, the second shooting environment brightness, the shooting angle and the tongue photo pixels; the self-portrait photo provides the self-portrait image itself to be analyzed; the first shooting environment brightness affects the contrast and clarity of the self-portrait image, thereby affecting the patient's mental state. The brightness of the patient's eyes, the degree of eyelid opening and the dryness of the mouth are all facial features of the patient, which are conducive to further judging the patient's mental state from a medical perspective; the tongue photo provides the tongue image itself to be analyzed; the pixels of the tongue photo determine the resolution of the tongue image, which affects the judgment of the tongue state; the shooting distance affects the size and proportion of the tongue image, thereby affecting the judgment of the fatness and thinness of the tongue coating; the second shooting environment brightness affects the contrast and clarity of the tongue image, thereby affecting the judgment of the tongue surface color; the shooting angle affects the angle of observing the tongue, thereby affecting the judgment of the smoothness of the tongue edge;

[0111] The method for obtaining the patient's eye brightness comprises:

[0112] Locate the face area through pre-trained face detection algorithm;

[0113] Locate the eyes of the face through key point detection algorithm;

[0114] The eye area image is cropped from the original selfie image according to the eye position of the face, and only the pixel information of the eye is retained. The pixel information includes pixel value, pixel density and pixel color space;

[0115] Convert the extracted eye area image from RGB color space to grayscale color space as follows:

[0116] Gray=0.299×R+0.587×G+0.114×B.

[0117] Among them, R, G and B are the values ​​of the red, green and blue channels of the pixel respectively; 0.299, 0.587 and 0.114 are the channel weights of red, green and blue respectively; Gray is the corresponding grayscale value;

[0118] The color value in the original RGB image is replaced by the calculated grayscale value to obtain a grayscale image, in which each pixel in the eye area image corresponds to a grayscale value, expressed as brightness;

[0119] The brightness of the patient's eyes is obtained by calculating the average grayscale value of all pixels in the eye area image;

[0120] The method for obtaining the eyelid opening degree of the patient comprises:

[0121] Locate the face area through pre-trained face detection algorithm;

[0122] Locate the eyes of the face through key point detection algorithm;

[0123] The eyelid edge and the lip edge are detected by using the Canny edge detection algorithm to obtain the eyelid edge information and the lip edge information;

[0124] By selecting the median points of the upper and lower eyelid edges and the eyelid edge information, the distance between the upper and lower eyelids is detected using Hough transform, which is the degree of eyelid opening.

[0125] Methods for obtaining the degree of mouth dryness include:

[0126] The color features of the lip area are collected by counting the color mean and variance of the lip area;

[0127] According to the color characteristics of the lip area, the dryness of the mouth is classified; the judgment thresholds of the color mean and color variance are set. When the color mean is not higher than the preset color threshold and the variance is not lower than the threshold variance threshold, the lip area is dry, otherwise it is wet; for example, when the color mean is not higher than 100 and the color variance is not lower than 50, the lip area is judged to be dry, otherwise it is judged to be wet;

[0128] The shooting distance is the shooting distance between the camera and the patient's tongue when the patient uses the camera to take a tongue photo; the tongue photo is a selfie of the patient sticking out his tongue in the early morning of the consultation day; the method for obtaining the shooting distance includes:

[0129] Time of flight technology (ToF) is used to timely obtain the distance between the patient's tongue and the camera when the patient's tongue image is taken. The distance acquisition steps are as follows:

[0130] When the patient takes a tongue image, the camera sends a short pulse of light to the patient's tongue, and the camera records the time when the light pulse is emitted T 1 ;

[0131] The light signal reflected from the patient's tongue surface is received by the camera; the camera records the time when the light signal is received T 2 ;

[0132] By calculating the time difference between the emission and reception of the short pulse of light, the time it takes for the light signal to travel from the camera to the patient's tongue and then back is determined and recorded as the flight time FT, FT = |T 2 -T 1 |;

[0133] By formula Calculate the distance between the patient's tongue and the camera;

[0134] Where DS is the distance between the patient's tongue and the camera; c is the speed of light;

[0135] ToF technology can provide high-precision distance measurement in tongue distance measurement, usually within a few millimeters, and ToF technology can obtain distance information in real time;

[0136] The first shooting environment brightness and the second shooting environment brightness are both obtained through an ambient light sensor equipped by the camera device;

[0137] The method for acquiring the shooting angle includes:

[0138] Use the gyroscope sensor to obtain the shooting angle when taking a picture. The steps are as follows:

[0139] Read the data of the gyroscope sensor, which includes the rotation speed ω(t) of the camera;

[0140] By integrating the formula Get the cumulative rotation angle of the camera;

[0141] Where θ(t 2 ) is the camera in the time period [t 1 , t 2 ]; θ(t 1 ) is the camera at t 1 The initial angle at the moment; ω(t) is the angle of the camera at [t 1 , t 2 ]The rotation speed of the time period;

[0142] The tongue photo pixels are obtained using Adobe Photoshop software.

[0143] The tongue state recognition module inputs the patient's tongue image information into the trained tongue state recognition model to obtain the patient's tongue state, which includes the thickness of the tongue coating, the color of the tongue surface, and the smoothness of the tongue edge;

[0144] The training method of the tongue state recognition model includes:

[0145] Collect a historical tongue data set, the historical tongue data set includes n groups of historical tongue image information, and historical tongue states corresponding to the historical tongue image information, where n is an integer greater than 1; divide the historical tongue data set into a training set and a test set, build a classifier, use the historical tongue image information in the training set as input data, and use the historical tongue states in the training set as output data, train the classifier to obtain an initial classifier, test the initial classifier using the test set, and output a classifier that meets a preset accuracy as a tongue state recognition model; the tongue state recognition model is a network CSPDarknet53 or other suitable models.

[0146] The collection method of historical tongue image information includes:

[0147] Collect tongue photos of patients under different shooting distances, shooting angles and tongue photo pixels;

[0148] By randomly adjusting the color channels of tongue photos under different shooting distances, shooting angles and tongue photo pixels, changing the brightness and contrast, simulating the shooting effects under different lighting conditions, and obtaining tongue photos under different shooting distances, shooting angles, tongue photo pixels and shooting environment brightness; compared with the prior art that collects tongue image information for each situation, here, the inventors can effectively reduce the difficulty of collecting historical tongue image information by randomly adjusting the color channels of tongue photos;

[0149] By horizontally flipping tongue photos at different shooting distances, shooting angles, tongue photo pixels and second shooting environment brightness, the robustness of the tongue state recognition model can be trained to improve; random noise is added to the tongue photos to simulate the noise interference in the actual shooting process and optimize the application of the model in the actual environment.

[0150] Furthermore, the tongue state recognition model is optimized using the self-attention mechanism. The specific optimization method is as follows:

[0151] The historical tongue image information is optimized using the self-attention mechanism; the tongue state recognition model is the CSPDarknet53 model; the self-attention module is inserted into the shallow position of the CSPDarknet53 model to improve the CSPDarknet53 model's attention to the details of the input tongue image information, so that the CSPDarknet53 model pays more attention to the areas in the tongue image related to the tongue state, thereby improving the accuracy of the CSPDarknet53 model in recognizing the patient's tongue image information;

[0152] The methods for building the self-attention mechanism include:

[0153] Define the input dimension n of the self-attention mechanism input and output dimension n output , where n input and n output is an integer greater than or equal to 1;

[0154] For example, in this embodiment, n input =n output =5;

[0155] Define input feature tensor X = (A, B, C, D, E), where A is the feature vector of the tongue photo; B is the feature vector of the shooting distance; C is the feature vector of the brightness of the second shooting environment; D is the feature vector of the shooting angle; and B is the feature vector of the tongue photo pixel;

[0156] Define the weight matrix W(n) for linear transformation input , n output ), and initialize the weight matrix with He, and initialize the weights of the weight matrix to be uniformly distributed within the range of

[0157] Construct query vector and key vector: Linearly transform and combine the output data of the input layer through the fully connected layer to obtain the query tensor Q, key tensor K and value tensor V;

[0158] Among them, the linear transformation combinations are: Q = X·W q +b,K=X·W k +b,V=X·W v +b, where X represents the input feature tensor, W q , W k , W v represents the respective weight matrices, and b represents the bias vector;

[0159] The attention score S(Q, K) is calculated by the dot product between the query tensor Q and the key tensor K. Where d k is the dimension of the key tensor, η is a shared bias vector in the entire attention score;

[0160] The attention scores are normalized by applying the softmax function to the attention scores to obtain the attention weights. The attention score normalization steps include:

[0161]

[0162] Where S(Q, K) i is the i-th score in the vector, S(Q, K) jis the jth score in the vector; L is the length of the attention score vector; i∈L, j∈L.

[0163] The value tensor V is weighted and summed by the attention weight to obtain the weighted value tensor VW. The elements in the weighted value tensor VW are summed, including A, B, C, D, and E, to obtain the final output tensor OP; the output tensor reflects the degree of attention of the self-attention mechanism to the value tensor given the query tensor and the key tensor.

[0164] The calculation steps of the weighted value tensor VW include:

[0165] VW = softmax(S(Q, K)) i ×V i ;

[0166] Where V i is the i-th element of the value tensor;

[0167]

[0168] Where N is the length of the value tensor V; (VW) k is the kth element of the weight value tensor;

[0169] The weighted summed value tensor VW is used as the output of the self-attention mechanism and returned to the upper network for use as the data set for the next group of training;

[0170] By introducing the feature extraction network CSPDarknet53 model with self-attention mechanism and using supervised learning method, the input of CSPDarknet53 model is the patient's historical tongue image information, and the output is the predicted value of tongue state recognition;

[0171] Use supervised learning and self-supervised learning methods, use labeled image data and unlabeled image data for comprehensive training to further optimize model performance;

[0172] The CSPDarknet53 model can accurately predict the tongue state by optimizing the model parameters through the back-propagation algorithm;

[0173] The validation set was used to evaluate the performance of the model. Different videos containing the tongue were selected for real-time detection and judgment, and the model was tuned based on the evaluation results.

[0174] The patient facial state judgment module inputs the selfie image information into the pre-trained patient mental state recognition model to obtain the mental state category; the mental state category includes fatigue, anxiety and normal;

[0175] The training method of the patient mental state recognition model includes:

[0176] Collecting a historical selfie image dataset, where the historical selfie image dataset includes selfie image information and mental state categories corresponding to the selfie image information;

[0177] For self-portraits, a convolutional neural network (CNN) is used to extract facial features of the face, and the facial features of the face, the brightness of the first shooting environment, the brightness of the patient's eyes, the degree of eyelid opening, the degree of mouth dryness, and the annotated mental state category label are used as feature sequences and used as inputs of the RNN model;

[0178] The RNN model uses a set of mental state category labels corresponding to each set of selfie image information as output, and uses the mental state category labels actually corresponding to each set of selfie image information as the prediction target, and defines the cross entropy loss function Stop training when the RNN model loss function value is less than or equal to the target loss value;

[0179] Among them, y n represents the actual mental state category label corresponding to the nth group of selfie image information, p n Represents the predicted mental state category label probability corresponding to the nth group of selfie image information.

[0180] The doctor assignment module identifies the disease classification based on the symptom description; inputs the patient's age, gender, mental state category and disease classification into the pre-trained doctor assignment model to obtain doctor recommendation information, and sends the doctor recommendation information to the corresponding patient end; the doctor recommendation information includes the doctor's name, doctor's professional field, doctor's working age, doctor's consultation time and doctor's evaluation satisfaction;

[0181] Based on the symptom description, methods for identifying the disease classification include:

[0182] Use text processing techniques to identify keywords in the text;

[0183] Establish a medical vocabulary or medical knowledge graph that contains a variety of disease names, symptom names and their corresponding disease classifications; match the identified keywords with the entries in the medical vocabulary;

[0184] According to the matched disease name, match the corresponding disease classification;

[0185] According to the matching results, the text is divided into corresponding disease classifications and the disease classifications are output;

[0186] The training method of the doctor assignment model includes:

[0187] Obtain r groups of data, where r is a positive integer greater than 1, and the data includes the patient's age group, gender, mental state category, disease classification, and doctor recommendation information. The patient's age, gender, mental state category, and disease classification are used as a sample set, and the sample set is divided into a training set and a test set. A classifier is constructed, and the patient's age group, gender, mental state category, and disease classification in the training set are used as input data, and the doctor recommendation information in the training set is used as output data. The classifier is trained to obtain an initial classifier, and the initial classifier is tested using the test set, and a classifier that meets a preset accuracy is output as a doctor allocation model, and the classifier is a BERT model or other models;

[0188] The remote consultation module receives the doctor selected by the patient, determines the corresponding doctor according to the doctor selected by the patient, establishes communication link information between the patient and the doctor, and sends the communication link information to the patient and the doctor; and synchronously sends the patient's basic information, physical characteristics and tongue status category to the doctor; the communication link information includes the communication link and communication time; it is convenient for the patient and the corresponding doctor to adjust their personal time for communication; whether further confirmation is needed in the hospital for further treatment; the communication link includes video calls and text exchanges.

[0189] When identifying the tongue state, the effects of shooting distance, second shooting environment brightness, shooting angle and tongue photo pixels on the tongue state are taken into account, and the tongue state of the patient can be identified more accurately than the existing technology; when judging the patient's mental state, the effects of the first shooting environment brightness, the patient's eye brightness, eyelid opening degree and mouth dryness on the mental state are taken into account, and the consideration is more comprehensive than the existing technology, and the patient's mental state can be identified from more details and facial features, which is more accurate; so that more suitable doctor recommendation information can be recommended to the patient based on the patient's mental state and basic information;

[0190] Secondly, when the patient establishes a communication channel with the doctor, the system directly sends the recognized tongue status to the corresponding doctor, reducing the doctor's workload and enabling the doctor to further understand the patient's condition. It also improves the communication efficiency between the patient and the doctor during online consultations.

[0191] When identifying the patient's tongue state, the randomness of each patient's shooting is taken into account. In order to reduce the impact of this random shooting on subsequent identification, the shooting distance is considered and the time-of-flight technology is combined with the patient's tongue state recognition for the first time, effectively eliminating the impact of random shooting.

[0192] At the same time, the self-attention mechanism is introduced for the first time in the tongue state recognition model for optimization, so that the tongue state recognition model pays more attention to the area related to the tongue state in the tongue image compared with the existing technology, effectively improving the accuracy of the recognition of the patient's tongue image information;

[0193] By adding the patient tongue status recognition module and the patient facial status judgment module, the present invention further judges and predicts the patient's condition on the basis of the original remote consultation; by adding the doctor allocation module and the remote consultation module, the consultation efficiency is greatly improved. Patients can learn the details based on the provided doctor allocation information, and independently choose the most suitable doctor to see a doctor in time without delaying the condition.

[0194] The smart cloud outpatient system provided in this embodiment can be used in hospitals or health centers and can be connected to the hospital system. If the patient has examination data in the hospital, the patient's examination data in the hospital can also be sent to the doctor's end.

[0195] Embodiment 2:

[0196] The method for obtaining the shooting distance also includes:

[0197]

[0198] Wherein, JL is the depth value corresponding to the pixel point in the tongue photo, which is the distance between the camera and the patient's face, in millimeters; JX is the baseline distance between the projector in the camera and the camera head, which is a known value, in millimeters, and the projector is a light source device in the camera specifically used to project the coded pattern; SC is the parallax of the pixel point in the tongue photo, that is, the displacement of the coded pattern, in millimeters; JA is the focal length of the camera when taking the tongue photo, in millimeters;

[0199] SC is calculated by comparing the displacement between the projected coding pattern and the received coding pattern. The specific steps to obtain SC are:

[0200] Model the projected coding pattern (usually a regular dot pattern) as a function Q(x, y);

[0201] Detect the coded pattern reflected by the tongue in the tongue photo and extract the corresponding function P(x, y);

[0202] Calculate the position difference between Q(x, y) and P(x, y) to get the disparity d(x, y);

[0203] Substituting d(x, y) into the formula, we can get the depth JL corresponding to each pixel point (x, y).

[0204] This embodiment estimates the patient shooting distance in the tongue image information based on the structured light ranging principle, thereby ensuring the accuracy of the feature sequence input to the training model of the patient tongue recognition module.

[0205] Embodiment 3:

[0206] See also Figure 2 As shown, a smart cloud outpatient method described in this embodiment includes:

[0207] Collect the patient's basic information, including name, age, gender, contact information, historical diagnosis data, family medical history and symptom description;

[0208] Using various sensors to collect the patient's physical characteristics;

[0209] Receive the selfie image information and tongue image information sent by the patient,

[0210] Input the patient's tongue image information into the trained tongue state recognition model to obtain the patient's tongue state, wherein the tongue state includes the thickness of the tongue coating, the color of the tongue surface, and the smoothness of the tongue edge;

[0211] Input the selfie image information into the pre-trained patient mental state recognition model to obtain the mental state category; the mental state category includes fatigue, anxiety and normal;

[0212] According to the symptom description, the disease classification is identified; the patient's age, gender, mental state category and disease classification are input into the pre-trained doctor allocation model to obtain doctor recommendation information, and the doctor recommendation information is sent to the corresponding patient end; the doctor recommendation information includes the doctor's name, doctor's professional field, doctor's working age, doctor's consultation time and doctor's evaluation satisfaction;

[0213] And receive the doctor selected by the patient, determine the corresponding doctor end according to the doctor selected by the patient, establish communication link information between the patient end and the doctor end, and send the communication link information to the patient end and the doctor end; and synchronously send the patient's basic information, physical characteristics and tongue status category to the doctor end.

[0214] Embodiment 4:

[0215] See also Figure 3 As shown, according to another aspect of the present application, an electronic device 500 is also provided. The electronic device 500 may include one or more processors and one or more memories. The memories store computer readable codes, and when the computer readable codes are run by one or more processors, a smart cloud outpatient method as described above may be executed.

[0216] The method or system according to the embodiment of the present application can also be used by Figure 3 The electronic device architecture shown in FIG. Figure 3As shown, the electronic device 500 may include a bus 501, one or more CPUs 502, ROM 503, RAM 504, a communication port 505 connected to a network, an input / output 506, a hard disk 507, etc. The storage device in the electronic device 500, such as ROM 503 or hard disk 507, may store a smart cloud outpatient method provided by the present application. Further, the electronic device 500 may also include a user interface 508. Of course, Figure 3 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 3 One or more components of an electronic device are shown.

[0217] Embodiment 5:

[0218] See also Figure 4 As shown, a computer-readable storage medium 600 according to an embodiment of the present application is shown. Computer-readable instructions are stored on the computer-readable storage medium 600. When the computer-readable instructions are executed by the processor, a smart cloud outpatient method according to an embodiment of the present application described with reference to the above figures can be executed. The storage medium 600 includes, but is not limited to, for example, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0219] In addition, according to the embodiments of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the present application provides a non-transitory machine-readable storage medium, which stores machine-readable instructions, and the machine-readable instructions can be executed by a processor to execute instructions corresponding to the method steps provided in the present application, for example: a smart cloud outpatient method. When the computer program is executed by a central processing unit (CPU), the above functions defined in the method of the present application are executed.

[0220] If the disclosed technical solution involves personal information, the product using the disclosed technical solution has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the disclosed technical solution involves sensitive personal information, the product using the disclosed technical solution has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0221] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0222] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A smart cloud outpatient method, characterized in that: include: Collect basic information of patients; Use various sensors and medical devices to collect patients’ physical characteristics; Receive the selfie image information and tongue image information sent by the patient, Input the patient's tongue image information into the trained tongue state recognition model to obtain the patient's tongue state, wherein the tongue state includes the thickness of the tongue coating, the color of the tongue surface, and the smoothness of the tongue edge; Input the selfie image information into the pre-trained patient mental state recognition model to obtain the mental state category; the mental state category includes fatigue, anxiety and normal; According to the symptom description, the disease classification is identified; the patient's age, gender, mental state category and disease classification are input into the pre-trained doctor assignment model to obtain doctor recommendation information, and the doctor recommendation information is sent to the corresponding patient end; The doctor recommendation information includes the doctor's name, doctor's professional field, doctor's working age, doctor's consultation time and doctor's evaluation satisfaction; And receive the doctor selected by the patient, determine the corresponding doctor end according to the doctor selected by the patient, establish communication link information between the patient end and the doctor end, and send the communication link information to the patient end and the doctor end; and synchronously send the patient's basic information, physical characteristics and tongue status category to the doctor end.

2. A smart cloud outpatient method according to claim 1, characterized in that: The self-portrait image information is obtained from a self-portrait image of the patient in the early morning of the day of the consultation, and the self-portrait image information includes the self-portrait photo, the brightness of the first shooting environment, the brightness of the patient's eyes, the degree of eyelid opening, and the degree of dryness of the mouth; The tongue image information includes the tongue photo, shooting distance, brightness of the second shooting environment, shooting angle and pixels of the tongue photo; the selfie photo provides the selfie image itself to be analyzed; The shooting distance is the shooting distance between the camera and the patient's tongue when the patient uses the camera to take a tongue photo; the tongue photo is a selfie of the patient sticking out his tongue in the early morning of the consultation day; The physical characteristics include blood pressure, body temperature, heart rate, BMI and blood oxygen.

3. A smart cloud outpatient method according to claim 2, characterized in that: The method for obtaining the patient's eye brightness comprises: Locate the face area through pre-trained face detection algorithm; Locate the eyes of the face through key point detection algorithm; The eye area image is cropped from the original selfie image according to the eye position of the face, and only the pixel information of the eye is retained. The pixel information includes pixel value, pixel density and pixel color space; Convert the extracted eye area image from RGB color space to grayscale color space as follows: Gray=0.299×R+0.587×G+0.114×B. Among them, R, G and B are the values ​​of the red, green and blue channels of the pixel respectively; 0.299, 0.587 and 0.114 are the channel weights of red, green and blue respectively; Gray is the corresponding grayscale value; The color value in the original RGB image is replaced by the calculated grayscale value to obtain a grayscale image, in which each pixel in the eye area image corresponds to a grayscale value, expressed as brightness; The brightness of the patient's eyes is obtained by calculating the average grayscale value of all pixels in the eye area image; The patient's eyelid opening degree method includes: The eyelid edge and the lip edge are detected by using the Canny edge detection algorithm to obtain the eyelid edge information and the lip edge information; By selecting the median points of the upper and lower eyelid edges and the eyelid edge information, the distance between the upper and lower eyelids is detected using Hough transform, which is the degree of eyelid opening. The method for obtaining the degree of mouth dryness comprises: The color features of the lip area are collected by counting the color mean and variance of the lip area; According to the color characteristics of the lip area, the degree of dryness of the mouth is classified; the judgment thresholds of the color mean and color variance are set. When the color mean is not higher than the preset color threshold and the variance is not lower than the threshold variance threshold, the lip area is dry, otherwise it is moist.

4. A smart cloud outpatient method according to claim 2, characterized in that: The method for obtaining the shooting distance includes: The time-of-flight technique is used to obtain the distance between the patient's tongue and the camera when the patient's tongue image is taken: When the patient takes a tongue image, the camera sends a short pulse of light to the patient's tongue, and the camera records the light pulse emission time T1; The light signal reflected from the surface of the patient's tongue is received by the camera; the moment T2 when the light signal is received is recorded by the camera; By calculating the time difference between the emission and reception of the short pulse of light, the time it takes for the light signal to travel from the camera to the patient's tongue and then back is determined and recorded as the flight time FT, FT = |T2-T1|; by the formula Calculate the distance between the patient's tongue and the camera; Where DS is the distance between the patient's tongue and the camera; c is the speed of light; The method for acquiring the shooting angle includes: Use the gyroscope sensor to obtain the shooting angle when taking a picture. The steps are as follows: Read the data of the gyroscope sensor, which includes the rotation speed ω(t) of the camera; By integrating the formula Get the cumulative rotation angle of the camera: Where θ(t2) is the camera’s position in the time period [t1, t2] 的累计旋转角度 ;θ(t1) 为相机在 The initial angle at time t1; ω(t) 为相机在 [t1, t2] 时间段的旋转速度 .

5. The smart cloud outpatient method according to claim 1, characterized in that: The training method of the tongue state recognition model includes: Collect a historical tongue data set, the historical tongue data set includes n groups of historical tongue image information, and historical tongue states corresponding to the historical tongue image information, where n is an integer greater than 1; divide the historical tongue data set into a training set and a test set, construct a classifier, use the historical tongue image information in the training set as input data, use the historical tongue states in the training set as output data, train the classifier to obtain an initial classifier, test the initial classifier using the test set, and output a classifier that meets a preset accuracy as a tongue state recognition model; the tongue state recognition model is a network CSPDarknet53 model; The collection method of historical tongue image information includes: Collect tongue photos of patients under different shooting distances, shooting angles and tongue photo pixels; By randomly adjusting the color channels of tongue photos under different shooting distances, shooting angles and tongue photo pixels, changing the brightness and contrast, and simulating the shooting effects under different lighting conditions, tongue photos with different shooting distances, shooting angles, tongue photo pixels and shooting environment brightness were obtained.

6. A smart cloud outpatient method according to claim 5, characterized in that: The tongue state recognition model is optimized using the self-attention mechanism. The optimization methods include: The historical tongue image information is optimized using the self-attention mechanism; the self-attention module is inserted into the shallow position of the CSPDarknet53 model; The methods for building the self-attention mechanism include: Define the input dimension n of the self-attention mechanism input and output dimension n output , where n input and n output is an integer greater than or equal to 1; Define input feature tensor X = (A, B, C, D, E); where A is the feature vector of the tongue photo; B is the feature vector of the shooting distance; C is the feature vector of the brightness of the second shooting environment; D is the feature vector of the shooting angle; E is the feature vector of the tongue photo pixel; Define the weight matrix W(n) for linear transformation input , n output ), and initialize the weight matrix with He, and initialize the weights of the weight matrix to be uniformly distributed within the range of Construct query vector and key vector: Linearly transform and combine the output data of the input layer through the fully connected layer to obtain the query tensor Q, key tensor K and value tensor V; Among them, the linear transformation combinations are: Q = X·W q +b,K=X·W k +b,V=X·W v +b, where X represents the input feature tensor, W q , W k , W v represents the respective weight matrices, and b represents the bias vector; The attention score S(Q, K) is calculated by the dot product between the query tensor Q and the key tensor K. Where d k is the dimension of the key tensor and η is a shared bias vector among the entire attention scores.

7. A smart cloud outpatient method according to claim 6, characterized in that: The attention weights are obtained by normalizing the attention scores by applying the softmax function to them. The attention score normalization steps include: Where S(Q, K) i is the i-th score in the vector, S(Q, K) j is the jth score in the vector; L is the length of the attention score vector; i∈L, j∈L; The value tensor V is weighted and summed by the attention weight to obtain a weighted value tensor VW, and the elements in the weighted value tensor VW are summed, including A, B, C, D, B, to obtain a final output tensor OP; the output tensor reflects the degree of attention of the self-attention mechanism to the value tensor given the query tensor and the key tensor; The calculation steps of the weighted value tensor VW include: VW=softmax(S(Q,K)) i ×V i ; Where V i is the i-th element of the value tensor; Where N is the length of the value tensor V; (VW)k is the kth element of the weighted value tensor; The weighted summed value tensor VW is used as the output of the self-attention mechanism and returned to the upper network for use as the data set for the next group of training.

8. The smart cloud outpatient method according to claim 1, characterized in that: Based on the symptom description, methods for identifying the disease classification include: Use text processing techniques to identify keywords in the text; Establish a medical vocabulary, which includes various disease names, symptom names and their corresponding disease classifications; match the identified keywords with the entries in the medical vocabulary; According to the matched disease name, match the corresponding disease classification; According to the matching results, the text is divided into corresponding disease classifications and the disease classifications are output.

9. A smart cloud outpatient method according to claim 8, characterized in that: The training method of the doctor allocation model includes: Get r groups of data, where r is a positive integer greater than 1. The data include the patient's age group, gender, mental state category, disease classification, and doctor recommendation information. Take the patient's age, gender, mental state category, and disease classification as a sample set. Divide the sample set into a training set and a test set. Construct a classifier. Take the patient's age group, gender, mental state category, and disease classification in the training set as input data. Take the doctor recommendation information in the training set as output data. Train the classifier to obtain an initial classifier. Use the test set to test the initial classifier. Output a classifier that meets the preset accuracy as the doctor assignment model. The classifier is a BERT model.

10. A smart cloud outpatient system, implementing a smart cloud outpatient method as described in any one of claims 1 to 9, characterized in that: include: Patient basic information collection module, used to collect basic information of patients; The patient's physical characteristics collection module uses various sensors to collect the patient's physical characteristics; The image information acquisition module is used to receive the self-portrait image information and tongue image information sent by the patient. The tongue state recognition module inputs the patient's tongue image information into the trained tongue state recognition model to obtain the patient's tongue state, which includes the thickness of the tongue coating, the color of the tongue surface, and the smoothness of the tongue edge; The patient facial state judgment module inputs the selfie image information into the pre-trained patient mental state recognition model to obtain the mental state category; the mental state category includes fatigue, anxiety and normal; The doctor assignment module identifies the disease classification based on the symptom description; inputs the patient's age, gender, mental state category and disease classification into the pre-trained doctor assignment model to obtain doctor recommendation information, and sends the doctor recommendation information to the corresponding patient end; the doctor recommendation information includes the doctor's name, doctor's professional field, doctor's working age, doctor's consultation time and doctor's evaluation satisfaction; The remote consultation module receives the doctor selected by the patient, determines the corresponding doctor end according to the doctor selected by the patient, establishes communication link information between the patient end and the doctor end, and sends the communication link information to the patient end and the doctor end; and synchronously sends the patient's basic information, physical characteristics and tongue status category to the doctor end.

Citation Information

Patent Citations

  • Mental state assessment method and system based on big data

    CN113192603A

  • Tongue photo processing method and device

    CN113362334A

  • Tiny-YOLO v4 natural environment-based tongue image detection positioning method

    CN114998220A

  • Tongue diagnosis image processing method and device based on key point detection and medium

    CN115661142A

  • Mental state analysis system, electronic equipment and storage medium

    CN117617975A