Traditional Chinese medicine diagnosis equipment and system based on image recognition and medium

Abstract

The invention relates to the field of image recognition technology and medical health, and discloses an electronic device which is used for achieving traditional Chinese medicine diagnosis based on image recognition and comprises the steps that basic information, pulse data and blood oxygen data of a user are obtained, and a face image and a lingual surface image of the user are obtained through a camera; generating a face diagnosis result according to the face image by using a pre-trained image diagnosis model, extracting lingual surface features of the lingual surface image, generating a lingual diagnosis result according to the lingual surface features by using the image diagnosis model, obtaining audio data of reading a preset text by a user, carrying out flood degree analysis on the audio data, and obtaining a lingual diagnosis result. And obtaining a sound flood value of the user, and generating a comprehensive diagnosis report in combination with the basic information, the pulse data, the blood oxygen data, the face diagnosis result, the tongue diagnosis result and the sound flood value. The invention further provides a traditional Chinese medicine diagnosis system based on image recognition and a storage medium. According to the invention, the efficiency of health diagnosis can be improved.

Description

Technical Field

[0001] The present invention relates to the fields of image recognition and medical health technology, and in particular to a traditional Chinese medicine diagnosis device, system and medium based on image recognition. Background Art

[0002] With the rapid development of modern science and technology, traditional Chinese medicine diagnosis is being combined with new technologies to improve the accuracy and convenience of diagnosis.

[0003] At present, the most common attempts and explorations on the market are TCM diagnostic and treatment instruments, such as smart mirrors, smart screens, TCM robots, etc. These TCM smart devices can detect and identify tongue features through intelligent inspection and questioning, and conveniently identify the user's physical health status, but users need to purchase relevant collection equipment or go to designated institutions to collect the user's health data, which is costly and inefficient. Summary of the invention

[0004] The present invention provides a traditional Chinese medicine diagnosis device, system and medium based on image recognition, the main purpose of which is to solve the problems of low efficiency and high user cost of existing health data collection methods.

[0005] To achieve the above object, the present invention provides an electronic device for implementing TCM diagnosis based on image recognition, characterized in that the electronic device comprises:

[0006] at least one processor;

[0007] and, a memory communicatively coupled to the at least one processor;

[0008] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor performs the following steps:

[0009] Obtain the user's basic information, and use the preset collection device to obtain the user's pulse data and blood oxygen data;

[0010] Using a preset camera to determine whether the user is in a preset shooting position, if the user is not in the preset shooting position, a prompt message is sent to the user according to the current position, if the user is in the preset shooting position, the camera is used to obtain the user's facial image, and a pre-trained image diagnosis model is used to generate a facial diagnosis result according to the facial image;

[0011] Acquire a tongue surface image of the user, extract tongue surface features of the tongue surface image, and generate a tongue diagnosis result according to the tongue surface features using the image diagnosis model;

[0012] Acquire audio data of a user reading a preset text, perform loudness analysis on the audio data, and obtain a loudness value of the user's voice;

[0013] A comprehensive diagnosis report is generated by combining the basic information, the pulse data, the blood oxygen data, the facial diagnosis result, the tongue diagnosis result and the voice loudness value.

[0014] In order to solve the above problems, a TCM diagnosis system based on image recognition is proposed, characterized in that the system includes a data acquisition module, a user face diagnosis module, a user tongue diagnosis module, an audio analysis module, and a comprehensive diagnosis module, wherein:

[0015] The data acquisition module is used to acquire basic information of the user and acquire the pulse data and blood oxygen data of the user using a preset acquisition device;

[0016] The user face diagnosis module is used to use a preset camera to determine whether the user is in a preset shooting position. If the user is not in the preset shooting position, a prompt message is sent to the user according to the current position. If the user is in the preset shooting position, the camera is used to obtain the user's facial image, and a pre-trained image diagnosis model is used to generate a face diagnosis result according to the facial image;

[0017] The user tongue diagnosis module is used to obtain a tongue surface image of the user, extract tongue surface features of the tongue surface image, and generate a tongue diagnosis result according to the tongue surface features using the image diagnosis model;

[0018] The audio analysis module is used to obtain audio data of a user reading a preset text, and perform loudness analysis on the audio data to obtain a loudness value of the user's voice;

[0019] The comprehensive diagnosis module is used to generate a comprehensive diagnosis report by combining the basic information, the pulse data, the blood oxygen data, the facial diagnosis results, the tongue diagnosis results and the voice loudness value.

[0020] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned image recognition-based Chinese medicine diagnosis.

[0021] The embodiment of the present invention obtains the basic information of the user. The pulse data and blood oxygen data of the user are obtained by using a preset acquisition device, and the preset camera is used to determine whether the user is in a preset shooting position. If the user is not in the preset shooting position, a prompt message is sent to the user according to the current position. If the user is in the preset shooting position, the facial image of the user is obtained by using the camera, and a face diagnosis result is generated according to the facial image using a pre-trained image diagnosis model, and the tongue image of the user is obtained. The tongue surface features of the tongue surface image are extracted, and the tongue diagnosis result is generated according to the tongue surface features using the image diagnosis model, and the audio data of the user reading the preset text is obtained. The audio data is analyzed for loudness to obtain the loudness value of the user's voice, and a comprehensive diagnosis report is generated by combining the basic information, the pulse data, the blood oxygen data, the face diagnosis result, the tongue diagnosis result and the voice loudness value. The cost of health data collection is reduced, and the efficiency of health data collection is improved. Users can flexibly select detection items according to their needs, meet the different detection needs of users, and improve the efficiency of user health diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a schematic diagram of an application environment of a traditional Chinese medicine diagnostic device based on image recognition in one embodiment of the present invention;

[0023] Figure 2 A functional module diagram of a traditional Chinese medicine diagnosis system based on image recognition provided by an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of the structure of an electronic device for implementing the traditional Chinese medicine diagnostic device based on image recognition provided by one embodiment of the present invention.

[0025] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0026] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0027] The embodiment of the present application provides a Chinese medicine diagnosis system based on image recognition. The execution subject of the Chinese medicine diagnosis system based on image recognition includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided in the embodiment of the present application. In other words, the Chinese medicine diagnosis system based on image recognition can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0028] The TCM diagnosis system based on image recognition provided by the embodiment of the present invention can be applied in Figure 1 In the application environment, the client communicates with the server through the network. The server can obtain the basic information of the user through the client, obtain the pulse data and blood oxygen data of the user by using the preset acquisition device, and use the preset camera to determine whether the user is in the preset shooting position. If the user is not in the preset shooting position, a prompt message is sent to the user according to the current position. If the user is in the preset shooting position, the facial image of the user is obtained by using the camera, and the face diagnosis result is generated according to the facial image by using the pre-trained image diagnosis model, and the tongue image of the user is obtained. The tongue surface features of the tongue surface image are extracted, and the tongue diagnosis result is generated according to the tongue surface features by using the image diagnosis model, and the audio data of the user reading the preset text is obtained, and the loudness of the audio data is analyzed to obtain the loudness value of the user's voice, and a comprehensive diagnosis report is generated by combining the basic information, the pulse data, the blood oxygen data, the face diagnosis result, the tongue diagnosis result and the voice loudness value. Among them, the client can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0029] An electronic device for implementing traditional Chinese medicine diagnosis based on image recognition, characterized in that the electronic device comprises:

[0030] at least one processor;

[0031] and, a memory communicatively coupled to the at least one processor;

[0032] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor performs the following steps:

[0033] S1. Used to obtain basic information of users.

[0034] In the embodiment of the present invention, the basic information of the user includes the user's gender, age, height and weight.

[0035] In the embodiment of the present invention, the basic information of the user is obtained by displaying a visual interface to the user, wherein the visual interface is provided with an input box and prompt information prompting the user to input basic information, and each input box corresponds to the basic information of one user.

[0036] In detail, the user's basic information can be obtained after the user fills in the basic information in the input box and clicks the confirmation button.

[0037] In detail, the basic information of the user can be obtained by using a mobile device (mobile phone).

[0038] In the embodiment of the present invention, the accuracy of subsequent diagnosis is improved by obtaining the basic information of the user.

[0039] S2, used to obtain the user's pulse data and blood oxygen data using a preset collection device.

[0040] In the embodiment of the present invention, the collection device may be a pulse oximeter, or an infrared sensor in a mobile device.

[0041] In detail, the pulse oximeter is a non-invasive medical device that can optically measure and display two main physiological parameters: blood oxygen saturation (SpO2) and pulse rate (PR). Blood oxygen saturation refers to the percentage of hemoglobin capacity in the blood that binds oxygen to the total hemoglobin capacity, and is an important indicator for evaluating human respiratory function and oxygen carrying capacity. Pulse rate refers to the number of heartbeats per minute.

[0042] In detail, the working principle of the pulse oximeter is based on the change in the amount of light absorbed during arterial pulsation. The instrument transmits light of different wavelengths (usually red light and infrared light) through body tissues (such as fingertips or earlobes) and measures the absorption rate of these lights. Since oxygenated hemoglobin and reduced hemoglobin have different absorption rates of light of these wavelengths, the instrument can calculate the oxygen saturation value. The pulse oximeter usually contains a microprocessor, memory, LED lights, photodiodes and other components, and analyzes the changes in light absorption rate through algorithms to obtain the values ​​of blood oxygen saturation and pulse rate.

[0043] In the embodiment of the present invention, the use of a preset acquisition device to obtain the user's pulse data and blood oxygen data is specifically used for:

[0044] Acquire the user's PPG signal using the acquisition device;

[0045] The user's pulse data and blood oxygen data are calculated according to the PPG signal.

[0046] In detail, the PPG signal refers to a photoplethysmography signal, which is a non-invasive detection method for monitoring blood volume changes in living tissues by photoelectric means. The PPG sensor is mainly composed of one or more light-emitting diodes (LEDs) and photodetectors, which irradiate the skin surface with a light beam of a specific wavelength and then transmit it to the photoelectric receiver by transmission or reflection. Due to the absorption and attenuation of skin muscles and blood, the light intensity detected by the detector will change accordingly. Among them, the absorption of light by skin muscle tissue, etc. remains constant throughout the blood circulation, while the blood volume in the skin changes in a pulsating manner under the action of the heart, causing the received light intensity to change in a pulsating manner.

[0047] In detail, the detection principle of the PPG signal is based on the fact that the absorption of light by blood changes with the contraction and relaxation of the heart. When the heart contracts, the blood flow increases, the light absorption increases, and the detected light intensity is the minimum; when the heart relaxes, the blood flow decreases, the light absorption decreases, and the detected light intensity is the maximum. By converting this light intensity change signal, the change of volume pulse blood flow can be obtained, thereby obtaining a PPG waveform containing important physiological information of the cardiovascular system.

[0048] In the embodiment of the present invention, by using a preset collection device to obtain the user's pulse data and blood oxygen data, the efficiency of subsequently generating a comprehensive diagnosis report is improved.

[0049] S3, used to determine whether the user is in a preset shooting position using a preset camera.

[0050] In the embodiment of the present invention, the use of a preset camera to determine whether the user is in a preset shooting position is specifically used for:

[0051] Using the camera to obtain the latest user face image;

[0052] Identify the coordinates of the center point of the user in the latest user face image;

[0053] Determine whether the center point coordinates are within a preset center point coordinate range;

[0054] If the center point coordinates are not within the center point coordinate range, it is determined that the user is not at the preset shooting position, and prompt information is generated according to the positional relationship between the center point coordinates and the center point coordinate range;

[0055] If the center point coordinates are within the center point coordinate range, acquiring the user's face angle according to the latest user face image;

[0056] Determining whether the face angle is within a preset angle range;

[0057] If the face angle exceeds the angle range, it is determined that the user is not in the preset shooting position, and a prompt message is generated according to the numerical relationship between the face angle and the angle range;

[0058] If the face angle is within the angle range, obtaining the face distance between the user and the camera;

[0059] Determining whether the face distance is within a preset distance range;

[0060] If the face distance exceeds the distance range, it is determined that the user is not in the preset shooting position. Prompt information is generated according to the size relationship between the face distance and the distance range;

[0061] If the face distance is within the distance range, it is determined that the user is at a preset shooting position.

[0062] In detail, the method of obtaining the user's facial angle based on the latest user facial image is carried out by identifying the positions of the user's facial features in the latest user facial image, identifying the offset distance between the user's facial features and the midline based on the positions of the user's facial features, and calculating the user's facial angle based on the offset distance.

[0063] If the user is not at the preset shooting position, S4 is executed to send a prompt message to the user according to the current position.

[0064] In the embodiment of the present invention, the prompt information is sent to the user according to the current position, and the prompt information is generated according to the relationship between the position data related to the user's face position and the preset standard range. For example, if the coordinates of the center point of the user's face are to the left of the preset center point coordinate range, prompt information is generated to prompt the user to move the face to the right; if the angle of the user's face is offset to the left compared to the preset angle range, prompt information is generated to prompt the user to rotate the face to the right; if the distance of the user's face is too close, prompt information is generated to prompt the user to move away from the camera.

[0065] In the embodiment of the present invention, the prompt information is sent to the user according to the current position, and a visualization screen that displays the user's face image in real time is used to display the prompt information to the user.

[0066] In the embodiment of the present invention, when the user is not in a preset shooting position, prompt information is sent to the user according to the current position, thereby improving the efficiency of acquiring the user's facial image.

[0067] If the user is in a preset shooting position, S5 is executed to obtain the user's facial image using the camera, and generate a facial diagnosis result based on the facial image using a pre-trained image diagnosis model.

[0068] In an embodiment of the present invention, the image diagnosis model is a convolutional neural network model, which can perform health diagnosis on the user based on the user's basic information combined with the corresponding physiological images.

[0069] In detail, in the process of training the image diagnosis model, a large amount of basic information of patients and corresponding facial images and tongue images are obtained as training data, and the image diagnosis model is used to perform health diagnosis based on the training data, and the difference between the diagnosis result and the standard result is obtained. The hyperparameters of the image diagnosis model are continuously optimized according to the back propagation algorithm based on the difference value until the performance of the model meets expectations or reaches a preset number of iterations, and the model training is confirmed to be completed.

[0070] In detail, the hyperparameters include weight parameters, bias terms and learning rates.

[0071] In detail, the weight parameters represent the strength of the connections between neurons in the model. In a neural network, each neuron is connected to neurons in the previous layer, and the weights determine the strength of these connections. The values ​​of the weights are usually randomly initialized at the beginning of model training, and then continuously updated during the training process through optimization algorithms (such as gradient descent) to minimize the loss function. The size and positive or negative of the weights directly affect the degree of response of the model to the input data, thereby affecting the prediction results of the model.

[0072] In detail, the bias term is used to add a constant offset to the output of the model, which acts like an intercept in linear regression. The presence of the bias term enables the model to not only fit the pattern of the data, but also adjust the overall position of the data. If the bias term is not used, the model may only be able to capture the data relationship through the origin, which is unrealistic in most cases. Therefore, the bias term increases the flexibility of the model, enabling it to better adapt to and predict the data.

[0073] In detail, the learning rate is an important hyperparameter that controls the parameter update step size of the model during the training process. It determines the magnitude of the adjustment of the weights and biases according to the gradient of the loss function at each iteration. If the learning rate is set too large, the model may "jump" near the optimal solution or even diverge; if the learning rate is set too small, the training process of the model will be very slow and may even fall into a local minimum. Therefore, choosing an appropriate learning rate is crucial for the rapid convergence and performance of the model.

[0074] In detail, the backpropagation algorithm is a key algorithm for training models in deep learning and neural networks. It updates the network parameters (weights and biases) by calculating the gradient of the loss function with respect to these parameters to minimize the loss function. The backpropagation algorithm is often used in conjunction with gradient descent or its variants such as stochastic gradient descent.

[0075] In the embodiment of the present invention, when the user face diagnosis module 102 generates a face diagnosis result according to the facial image using a pre-trained image diagnosis model, it is specifically used to:

[0076] Extracting features from the facial image to obtain facial features;

[0077] Performing one-hot encoding on the gender data contained in the basic information to obtain gender coding information, and normalizing the weight data and age data contained in the basic information to obtain normalized weight and normalized age;

[0078] Converting the gender coding information, the normalized weight and the normalized age into a vector form to obtain a basic information feature vector;

[0079] Concatenating the facial features with the basic information feature vector to obtain facial fusion features;

[0080] Using the image diagnosis model to perform convolution processing on the facial fusion features to obtain facial convolution features;

[0081] Using the image diagnosis model to perform pooling processing on the facial convolution features to obtain facial pooling features;

[0082] Activating the facial pooling features using the image diagnosis model to obtain facial activation features;

[0083] The image diagnosis model is used to perform full connection processing on the facial activation features to obtain a facial diagnosis result.

[0084] In detail, the facial fusion feature is convolved using the image diagnosis model. The convolution kernel is slid on the facial fusion feature according to a preset step size, and the dot product between the convolution kernel and the covered part after each sliding is calculated to obtain the convolution feature, wherein the convolution kernel is a matrix with a scale much smaller than the facial fusion feature.

[0085] In detail, the one-hot encoding of the gender data included in the basic information may refer to converting "male" into [0, 1] and converting "female" into [1, 0].

[0086] In detail, the facial convolution features are pooled using the image diagnosis model by sliding a rectangular window of a preset size on the facial convolution features, obtaining the average value or maximum value of the area covered by the rectangular window after each sliding to replace the original value of the area to obtain the pooled features.

[0087] In detail, the activation process of the facial pooling feature using the image diagnosis model is to add a nonlinear factor to the facial pooling feature using a preset activation function to enhance the feature recognition capability of the model. Common activation functions include Sigmoid function, Tanh function and ReLU function.

[0088] In the embodiment of the present invention, the fully connected processing of the facial activation features using the image diagnosis model is to calculate the output of each neuron of the model according to the weight parameters and bias items of the model to obtain the final facial diagnosis result.

[0089] In the embodiment of the present invention, when the user is in a preset shooting position, the user's facial image is acquired by using the camera, and a pre-trained image diagnosis model is used to generate a facial diagnosis result based on the facial image, thereby improving the efficiency of subsequent generation of a comprehensive diagnosis report.

[0090] S6, used to obtain the user's tongue surface image, extract the tongue surface features of the tongue surface image, and use the image diagnosis model to generate a tongue diagnosis result according to the tongue surface features.

[0091] In an embodiment of the present invention, the tongue surface features of the tongue surface image are extracted by performing texture feature extraction on the tongue surface image to obtain an initial tongue surface image feature, and then the brightness and smoothness of the user's tongue surface are identified based on the tongue surface image, and then the brightness and smoothness are integrated into the initial tongue surface features to obtain the final tongue surface features.

[0092] In the embodiment of the present invention, the extraction of the tongue surface features of the tongue surface image is specifically used for:

[0093] Extract features from the tongue surface image to obtain initial tongue surface features

[0094] Identifying the brightness of the user's tongue surface according to the tongue surface image;

[0095] Calculate the smoothness of the user's tongue surface according to the tongue surface image:

[0096] Calculating the absolute value of the difference between the brightness and a preset standard brightness to obtain a brightness difference value;

[0097] Calculating the absolute value of the difference between the smoothness and a preset standard smoothness to obtain a smoothness difference value;

[0098] Adding the brightness difference value and the smoothness difference value to obtain a health judgment value;

[0099] If the health discrimination value is greater than or equal to a preset health threshold, a preset first weight coefficient is used to perform feature amplification processing on the initial tongue surface feature to obtain a tongue surface feature;

[0100] If the health judgment value is less than the health threshold, the brightness and the smoothness are normalized and then merged into a vector form to obtain a tongue surface feature vector. The tongue surface feature vector and the tongue surface initial feature are weightedly fused using a preset second weight coefficient to obtain a tongue surface feature.

[0101] In detail, the preset standard brightness and the preset standard smoothness refer to the smoothness and brightness of a healthy tongue surface under normal circumstances.

[0102] In detail, when the health judgment value is greater than or equal to the preset health threshold, it means that the patient may have major health problems, so the initial tongue surface features need to be amplified to emphasize the importance of tongue surface related features in diagnosis.

[0103] In detail, the feature amplification processing of the initial tongue surface feature by using a preset first weight coefficient refers to amplifying the feature value of the initial tongue surface feature to enhance the feature strength.

[0104] In the embodiment of the present invention, calculating the smoothness of the user's tongue surface according to the tongue surface image includes:

[0105] Calculate the absolute value of the difference between the pixel values ​​of each pixel and the adjacent pixel points respectively;

[0106] The difference product between each pixel point and an adjacent pixel point is calculated according to the calculated absolute value of the difference, and the sum of the calculated difference products is used as the smoothness of the user's tongue surface.

[0107] In one embodiment of the present invention, the smoothness of the user's tongue surface is calculated using the following formula:

[0108] D=∑ y ∑ x (|f(x,y)-f(x+1,y)|*|f(x,y)-f(x,y+1)|)

[0109] Among them, D is the smoothness, f(x,y) represents the pixel value of the pixel point with coordinates (x,y) in the coordinate system established by the tongue surface image, f(x+1,y) represents the pixel value of the pixel point with coordinates (x+1,y) in the coordinate system established by the tongue surface image, f(x,y+1) represents the pixel value of the pixel point with coordinates (x,y+1) in the coordinate system established by the tongue surface image, and (x+1,y) and (x,y+1) are adjacent pixel points of the pixel point (x,y).

[0110] In detail, the brightness of the user's tongue surface is identified based on the tongue surface image, which is to use a preset image recognition algorithm to identify the color features of the user's tongue surface based on the tongue surface image, and then compare the color features with the standard tongue surface color features to obtain a comparison result, and map the comparison result into a numerical form to obtain the brightness;

[0111] In detail, the brightness is a numerical value for reflecting the color of the tongue surface.

[0112] In detail, the brightness can reflect the health status of the user. A low brightness may indicate that the user may be deficient in Qi and blood, while a high brightness may indicate that the user has body heat.

[0113] In detail, the smoothness is a value used to reflect the roughness of the user's tongue surface, and further reflects the user's health condition.

[0114] In the embodiment of the present invention, the step of using the image diagnosis model to generate a tongue diagnosis result based on the tongue surface features is the same as the step of using a pre-trained image diagnosis model to generate a facial diagnosis result based on the facial image, which will not be repeated here.

[0115] In an embodiment of the present invention, by acquiring the user's tongue surface image and extracting the tongue surface features of the tongue surface image, the efficiency of generating tongue diagnosis results is improved. The image diagnosis model is used to generate tongue diagnosis results based on the tongue surface features, thereby improving the efficiency of subsequently generating a comprehensive diagnosis report.

[0116] S7, used to obtain audio data of the user reading a preset text, and perform loudness analysis on the audio data to obtain the user's voice loudness value.

[0117] In the embodiment of the present invention, the audio data of the user reading the preset text is obtained by displaying the preset text to the user through a visual screen, prompting the user to read the preset text, and obtaining the user's audio data by using a recording device. The audio data is greater than 2 seconds and less than 30 seconds.

[0118] In the embodiment of the present invention, before obtaining the audio data of the user reading the preset text, the environmental noise detection is performed first, and the user audio data is obtained after the environmental noise meets the requirements.

[0119] In the embodiment of the present invention, the loudness value is a numerical value reflecting the loudness of the user's voice, and can further reflect the user's health condition after being combined with the user's personal basic data.

[0120] In detail, the loudness value, as a numerical value with a specific meaning, is mainly used to reflect the loudness of the user's voice. This loudness value does not exist in isolation. When it is combined with the user's personal basic data, it can play a greater role. Through this combination, the user's health status can be further reflected in depth. For example, the user's personal basic data such as age, gender, and physical condition are interrelated with the loudness value, which can provide a more comprehensive and accurate basis for assessing the user's health status. Users of different age groups may have different ranges of loudness values, and gender differences may also affect the relationship between loudness value and health status to a certain extent. At the same time, users in good physical condition may show higher loudness values, while users with certain physical problems may have different loudness values. In short, the combination of loudness value and user's personal basic data provides us with a new perspective and method to gain a deeper understanding of the user's health status.

[0121] In the embodiment of the present invention, the loudness analysis of the audio data to obtain the loudness value of the user's voice is specifically used for:

[0122] Sampling the audio data at a preset interval to obtain sampled audio;

[0123] The sound loudness value is calculated according to the sampled audio using the following formula:

[0124]

[0125] Wherein, L is the loudness value of the sound, k is the preset weight coefficient, N is the number of sampling points of the sampled audio, and x is i represents the amplitude value of the i-th sampling point in the sampled audio, and P is the preset reference sound pressure.

[0126] In detail, the preset reference sound pressure may be 20 microPascals (20 μPa).

[0127] In an embodiment of the present invention, by acquiring audio data of a user reading a preset text, the efficiency of subsequently obtaining the user's voice loudness value is improved. By performing loudness analysis on the audio data, the user's voice loudness value is obtained, thereby improving the efficiency of generating a comprehensive diagnostic report.

[0128] S8, used to generate a comprehensive diagnosis report by combining the basic information, the pulse data, the blood oxygen data, the facial diagnosis result, the tongue diagnosis result and the voice loudness value.

[0129] In the embodiment of the present invention, since the corresponding standard health values ​​of users of different ages, genders, heights and weights are different, it is necessary to provide a comprehensive diagnosis report in combination with the basic information of the users.

[0130] In an embodiment of the present invention, when the basic information, the pulse data, the blood oxygen data, the facial diagnosis result, the tongue diagnosis result and the voice loudness value are combined to generate a comprehensive diagnosis report, it is specifically used for:

[0131] Generate an individual vital sign label of the user according to the basic information;

[0132] Acquire a diagnostic data set from a pre-established diagnostic database according to the individual physical sign label;

[0133] The comprehensive diagnosis report is obtained by matching the pulse data, the blood oxygen data, the facial diagnosis results, the tongue diagnosis results and the voice loudness value with the diagnosis data set.

[0134] In the embodiment of the present invention, the individual physical sign label is a label representing the user's physical condition. Since different users have different individual physical sign labels and different individual physical sign labels have different health standards, data retrieval needs to be performed based on the individual physical sign labels.

[0135] In the embodiment of the present invention, the comprehensive diagnosis report is generated by combining the basic information, the pulse data, the blood oxygen data, the facial diagnosis results, the tongue diagnosis results and the voice loudness value. It is based on the user's basic information to judge whether the user's four diagnostic results are within the standard range, and give relevant diagnostic suggestions based on the user's four diagnostic results.

[0136] In detail, when the user's four diagnostic results are normal, the user is judged to have no health risks and only some health suggestions are provided to the user.

[0137] In detail, if a certain diagnosis result of the user is abnormal, the comprehensive diagnosis report is displayed to the user, and a physical examination appointment portal is provided to the user.

[0138] In detail, if the user's symptoms have worsened compared to past diagnostic data, the comprehensive diagnostic report will be displayed to the user, and an online physician consultation portal will be provided.

[0139] In the embodiment of the present invention, the collection of user health data can be completed on a mobile device, which greatly reduces the cost of health data collection and improves the efficiency of health data collection.

[0140] In an embodiment of the present invention, a comprehensive diagnosis report is generated by combining the basic information, the pulse data, the blood oxygen data, the facial diagnosis results, the tongue diagnosis results and the voice loudness value, thereby improving the effect of user health diagnosis.

[0141] It can be seen that in the embodiment of the present invention, the user's personal basic data and four health data are obtained by using a mobile device, four health diagnoses are performed based on the four health data, four diagnostic results are obtained, and a comprehensive diagnostic report is given in combination with the personal basic data and the four diagnostic results. The cost of health data collection is reduced, the efficiency of health data collection is improved, and the user can flexibly select the test items according to the needs, which meets the different testing needs of the user and improves the efficiency of the user's health diagnosis. Figure 3 1 is a functional module diagram of a traditional Chinese medicine diagnosis system based on image recognition provided by one embodiment of the present invention.

[0142] Reference Figure 2 , which is a functional module diagram of a traditional Chinese medicine diagnosis system based on image recognition provided by one embodiment of the present invention.

[0143] The image recognition-based TCM diagnosis system 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the image recognition-based TCM diagnosis system 100 can include a data acquisition module 101, a user face diagnosis module 102, a user tongue diagnosis module 103, an audio analysis module 104, and a comprehensive diagnosis module 105. The module of the present invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0144] The following is a description of each component and specific workflow of the TCM diagnosis system based on image recognition in conjunction with a specific embodiment:

[0145] The data acquisition module 101 is used to acquire basic information of the user, and acquire the pulse data and blood oxygen data of the user by using a preset acquisition device;

[0146] The user face diagnosis module 102 is used to use a preset camera to determine whether the user is in a preset shooting position. If the user is not in the preset shooting position, a prompt message is sent to the user according to the current position. If the user is in the preset shooting position, the camera is used to obtain the user's facial image, and a pre-trained image diagnosis model is used to generate a face diagnosis result according to the facial image;

[0147] The user tongue diagnosis module 103 is used to obtain a tongue surface image of the user, extract tongue surface features of the tongue surface image, and generate a tongue diagnosis result according to the tongue surface features using the image diagnosis model;

[0148] The audio analysis module 104 is used to obtain audio data of a user reading a preset text, and perform loudness analysis on the audio data to obtain a loudness value of the user's voice;

[0149] The comprehensive diagnosis module 105 is used to generate a comprehensive diagnosis report by combining the basic information, the pulse data, the blood oxygen data, the facial diagnosis result, the tongue diagnosis result and the voice loudness value.

[0150] The image recognition-based TCM diagnosis system 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the image recognition-based TCM diagnosis system 100 can include a data acquisition module 101, a user face diagnosis module 102, a user tongue diagnosis module 103, an audio analysis module 104, and a comprehensive diagnosis module 105. The module of the present invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0151] like Figure 3 , which is a schematic diagram of the structure of an electronic device of a traditional Chinese medicine diagnosis system based on image recognition provided by an embodiment of the present invention.

[0152] The electronic device 1 may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a traditional Chinese medicine diagnosis program based on image recognition.

[0153] In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core (ControlUnit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules (such as traditional Chinese medicine diagnosis programs based on image recognition, etc.) stored in the memory 11, and calls data stored in the memory 11 to execute various functions of the electronic device and process data.

[0154] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 can also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 can also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of a traditional Chinese medicine diagnosis program based on image recognition, but also can be used to temporarily store data that has been output or is to be output.

[0155] The communication bus 12 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0156] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in this embodiment, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0157] The figure only shows an electronic device with components. Those skilled in the art will understand that the structure shown in the figure does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0158] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0159] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0160] The TCM diagnosis program based on image recognition stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve:

[0161] Get basic information of the user;

[0162] Using a preset collection device to obtain the user's pulse data and blood oxygen data;

[0163] Using a preset camera to determine whether the user is in a preset shooting position;

[0164] If the user is not at the preset shooting position, a prompt message is sent to the user according to the current position;

[0165] If the user is in a preset shooting position, the camera is used to obtain the user's facial image, and a pre-trained image diagnosis model is used to generate a facial diagnosis result based on the facial image;

[0166] Acquire a tongue surface image of the user, extract tongue surface features of the tongue surface image, and generate a tongue diagnosis result according to the tongue surface features using the image diagnosis model;

[0167] Acquire audio data of a user reading a preset text, perform loudness analysis on the audio data, and obtain a loudness value of the user's voice;

[0168] A comprehensive diagnosis report is generated by combining the basic information, the pulse data, the blood oxygen data, the facial diagnosis result, the tongue diagnosis result and the voice loudness value.

[0169] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, which will not be repeated here.

[0170] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0171] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can implement:

[0172] Get basic information of the user;

[0173] Using a preset collection device to obtain the user's pulse data and blood oxygen data;

[0174] Using a preset camera to determine whether the user is in a preset shooting position;

[0175] If the user is not at the preset shooting position, a prompt message is sent to the user according to the current position;

[0176] If the user is in a preset shooting position, the camera is used to obtain the user's facial image, and a pre-trained image diagnosis model is used to generate a facial diagnosis result based on the facial image;

[0177] Acquire a tongue surface image of the user, extract tongue surface features of the tongue surface image, and generate a tongue diagnosis result according to the tongue surface features using the image diagnosis model;

[0178] Acquire audio data of a user reading a preset text, perform loudness analysis on the audio data, and obtain a loudness value of the user's voice;

[0179] A comprehensive diagnosis report is generated by combining the basic information, the pulse data, the blood oxygen data, the facial diagnosis result, the tongue diagnosis result and the voice loudness value.

[0180] In the embodiments provided by the present invention, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0181] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0183] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0184] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0185] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0186] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0187] Finally, it should be noted that if software tools or components other than those of the Company appear in the application embodiments, they are only used for illustration and do not represent actual use. The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A Chinese medicine diagnostic device based on image recognition, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor performs the following steps: Obtain the user's basic information, and use the preset collection device to obtain the user's pulse data and blood oxygen data; Using a preset camera to determine whether the user is in a preset shooting position, if the user is not in the preset shooting position, a prompt message is sent to the user according to the current position, if the user is in the preset shooting position, the camera is used to obtain the user's facial image, and a pre-trained image diagnosis model is used to generate a facial diagnosis result according to the facial image; Acquire a tongue image of the user, extract tongue features of the tongue image, and generate a tongue diagnosis result according to the tongue features using the image diagnosis model; Acquire audio data of a user reading a preset text, perform loudness analysis on the audio data, and obtain a loudness value of the user's voice; A comprehensive diagnosis report is generated by combining the basic information, the pulse data, the blood oxygen data, the facial diagnosis result, the tongue diagnosis result and the voice loudness value.

2. The Chinese medicine diagnostic device based on image recognition as claimed in claim 1, characterized in that: The method of obtaining the user's pulse data and blood oxygen data by using a preset collection device includes: Acquire the user's PPG signal using the acquisition device; The user's pulse data and blood oxygen data are calculated according to the PPG signal.

3. The TCM diagnostic device based on image recognition as claimed in claim 1, characterized in that: The method of using a preset camera to determine whether the user is in a preset shooting position includes: Using the camera to obtain the latest user face image; Identify the coordinates of the center point of the user in the latest user face image; Determine whether the center point coordinates are within a preset center point coordinate range; If the center point coordinates are not within the center point coordinate range, it is determined that the user is not at the preset shooting position, and the prompt information is generated according to the positional relationship between the center point coordinates and the center point coordinate range; If the center point coordinates are within the center point coordinate range, acquiring the user's face angle according to the latest user face image; Determining whether the face angle is within a preset angle range; If the face angle exceeds the angle range, it is determined that the user is not in the preset shooting position, and the prompt information is generated according to the numerical relationship between the face angle and the angle range; If the face angle is within the angle range, obtaining the face distance between the user and the camera; Determining whether the face distance is within a preset distance range; If the face distance exceeds the distance range, it is determined that the user is not in the preset shooting position; and the prompt information is generated according to the size relationship between the face distance and the distance range; If the face distance is within the distance range, it is determined that the user is at a preset shooting position.

4. The TCM diagnostic device based on image recognition as claimed in claim 1, characterized in that: The method of using a pre-trained image diagnosis model to generate a facial diagnosis result according to the facial image includes: Extracting features from the facial image to obtain facial features; Performing one-hot encoding on the gender data contained in the basic information to obtain gender coding information, and normalizing the weight data and age data contained in the basic information to obtain normalized weight and normalized age; Converting the gender coding information, the normalized weight and the normalized age into a vector form to obtain a basic information feature vector; Concatenating the facial features with the basic information feature vector to obtain facial fusion features; Using the image diagnosis model to perform convolution processing on the facial fusion features to obtain facial convolution features; Using the image diagnosis model to perform pooling processing on the facial convolution features to obtain facial pooling features; Activating the facial pooling features using the image diagnosis model to obtain facial activation features; The image diagnosis model is used to perform full connection processing on the facial activation features to obtain a facial diagnosis result.

5. The TCM diagnostic device based on image recognition as claimed in claim 1, characterized in that: The step of extracting the tongue surface features of the tongue surface image comprises: Extract features from the tongue surface image to obtain initial tongue surface features Identifying the brightness of the user's tongue surface according to the tongue surface image; Calculating the smoothness of the user's tongue surface according to the tongue surface image; Calculating the absolute value of the difference between the brightness and a preset standard brightness to obtain a brightness difference value; Calculating the absolute value of the difference between the smoothness and a preset standard smoothness to obtain a smoothness difference value; Adding the brightness difference value and the smoothness difference value to obtain a health judgment value; If the health discrimination value is greater than or equal to a preset health threshold, a preset first weight coefficient is used to perform feature amplification processing on the initial tongue surface feature to obtain a tongue surface feature; If the health judgment value is less than the health threshold, the brightness and the smoothness are normalized and then merged into a vector form to obtain a tongue surface feature vector. The tongue surface feature vector and the tongue surface initial feature are weightedly fused using a preset second weight coefficient to obtain a tongue surface feature.

6. The TCM diagnostic device based on image recognition as claimed in claim 5, characterized in that: The calculating the smoothness of the user's tongue surface according to the tongue surface image comprises: Calculate the absolute value of the difference between the pixel values ​​of each pixel and the adjacent pixel points respectively; The difference product between each pixel point and an adjacent pixel point is calculated according to the calculated absolute value of the difference, and the sum of the calculated difference products is used as the smoothness of the user's tongue surface.

7. The TCM diagnostic device based on image recognition as claimed in claim 1, characterized in that: The performing loudness analysis on the audio data to obtain the loudness value of the user's voice includes: Sampling the audio data at a preset interval to obtain sampled audio; The sound loudness value is calculated according to the sampled audio using a predetermined formula: The parameters of the predetermined formula include a preset weight coefficient, the number of sampling points of the sampled audio, the amplitude value of the i-th sampling point in the sampled audio, and a preset reference sound pressure.

8. The TCM diagnostic device based on image recognition as claimed in claim 1, characterized in that: The generating of a comprehensive diagnosis report by combining the basic information, the pulse data, the blood oxygen data, the facial diagnosis result, the tongue diagnosis result and the voice loudness value includes: Generate an individual vital sign label of the user according to the basic information; Acquire a diagnostic data set from a pre-established diagnostic database according to the individual physical sign label; The comprehensive diagnosis report is obtained by matching the pulse data, the blood oxygen data, the facial diagnosis results, the tongue diagnosis results and the voice loudness value with the diagnosis data set.

9. A Chinese medicine diagnosis system based on image recognition, characterized in that: The system includes a data acquisition module, a user face diagnosis module, a user tongue diagnosis module, an audio analysis module, and a comprehensive diagnosis module, wherein: The data acquisition module is used to acquire basic information of the user and acquire the pulse data and blood oxygen data of the user using a preset acquisition device; The user face diagnosis module is used to use a preset camera to determine whether the user is in a preset shooting position. If the user is not in the preset shooting position, a prompt message is sent to the user according to the current position. If the user is in the preset shooting position, the camera is used to obtain the user's facial image, and a pre-trained image diagnosis model is used to generate a face diagnosis result according to the facial image; The user tongue diagnosis module is used to obtain a tongue surface image of the user, extract tongue surface features of the tongue surface image, and generate a tongue diagnosis result according to the tongue surface features using the image diagnosis model; The audio analysis module is used to obtain audio data of a user reading a preset text, and perform loudness analysis on the audio data to obtain a loudness value of the user's voice; The comprehensive diagnosis module is used to generate a comprehensive diagnosis report by combining the basic information, the pulse data, the blood oxygen data, the facial diagnosis results, the tongue diagnosis results and the voice loudness value.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the following steps are performed: Obtain the user's basic information, and use the preset collection device to obtain the user's pulse data and blood oxygen data; Using a preset camera to determine whether the user is in a preset shooting position, if the user is not in the preset shooting position, a prompt message is sent to the user according to the current position, if the user is in the preset shooting position, the camera is used to obtain the user's facial image, and a pre-trained image diagnosis model is used to generate a facial diagnosis result according to the facial image; Acquire a tongue image of the user, extract tongue features of the tongue image, and generate a tongue diagnosis result according to the tongue features using the image diagnosis model; Acquire audio data of a user reading a preset text, perform loudness analysis on the audio data, and obtain a loudness value of the user's voice; A comprehensive diagnosis report is generated by combining the basic information, the pulse data, the blood oxygen data, the facial diagnosis result, the tongue diagnosis result and the voice loudness value.

Images