Traditional Chinese medicine diagnosis device, system and medium based on image recognition

Abstract

The present application relates to image recognition technology and the medical health field, disclose an electronic device for realizing the traditional Chinese medicine diagnosis based on image recognition, including: obtaining the basic information of user, pulse data and blood oxygen data, using the camera to obtain the face image and the tongue surface image of user, using the image diagnosis model trained in advance to generate the facial diagnosis result according to the face image, the tongue surface feature of the tongue surface image is extracted, the tongue diagnosis result is generated according to the tongue surface feature using the image diagnosis model, the audio data of user reading preset text is obtained, the audio data is carried out the loudness analysis, the voice loudness value of user is obtained, the basic information, the pulse data, the blood oxygen data, the facial diagnosis result, the tongue diagnosis result and the voice loudness value are combined to generate comprehensive diagnosis report.The present application also proposes a traditional Chinese medicine diagnosis system based on image recognition and storage medium.The present application can improve the efficiency of health diagnosis.

Description

Technical Field

[0001] This invention relates to the fields of image recognition and medical and health technology, and in particular to a TCM diagnostic device, system and medium based on image recognition. Background Technology

[0002] With the rapid development of modern technology, traditional Chinese medicine diagnosis is being combined with new technologies to improve the accuracy and convenience of diagnosis.

[0003] Currently, the most common attempts and explorations in the market involve TCM diagnostic devices, such as smart mirrors, smart screens, and TCM robots. These TCM intelligent devices can detect and identify tongue features through intelligent observation and questioning, conveniently identifying the user's health status. However, users need to purchase related data collection equipment or go to designated institutions to collect their health data, which is costly and inefficient. Summary of the Invention

[0004] This invention provides a TCM diagnostic device, system, and medium based on image recognition, with the main purpose of solving the problems of low efficiency and high user cost in existing health data collection methods.

[0005] To achieve the above objectives, the present invention provides an electronic device for realizing traditional Chinese medicine diagnosis based on image recognition, characterized in that the electronic device comprises:

[0006] At least one processor;

[0007] And, a memory communicatively connected to the at least one processor;

[0008] The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the at least one processor to perform the following steps:

[0009] Obtain the user's basic information and use a pre-set data acquisition device to acquire the user's pulse and blood oxygen data;

[0010] The system uses a preset camera to determine whether the user is in a preset shooting position. If the user is not in the preset shooting position, a prompt message is sent to the user based on the current position. If the user is in the preset shooting position, the system uses the camera to acquire the user's facial image and uses a pre-trained image diagnosis model to generate a facial diagnosis result based on the facial image.

[0011] Obtain a user's tongue image, extract the tongue features from the tongue image, and use the image diagnosis model to generate a tongue diagnosis result based on the tongue features;

[0012] Acquire audio data of the user reading a preset text, perform loudness analysis on the audio data, and obtain the loudness value of the user's voice;

[0013] A comprehensive diagnostic report is generated by combining the basic information, pulse data, blood oxygen data, facial diagnosis results, tongue diagnosis results, and voice volume value.

[0014] To address the aforementioned problems, a traditional Chinese medicine diagnostic system based on image recognition is proposed, characterized in that the system comprises a data acquisition module, a user face diagnosis module, a user tongue diagnosis module, an audio analysis module, and a comprehensive diagnosis module, wherein:

[0015] The data acquisition module is used to acquire the user's basic information and to acquire the user's pulse data and blood oxygen data using a preset acquisition device.

[0016] The user face diagnosis module is used to determine whether the user is in a preset shooting position using a preset camera. If the user is not in the preset shooting position, a prompt message is sent to the user based on the current position. If the user is in the preset shooting position, the camera is used to acquire the user's facial image, and a pre-trained image diagnosis model is used to generate a face diagnosis result based on the facial image.

[0017] The user tongue diagnosis module is used to acquire the user's tongue image, extract the tongue features from the tongue image, and generate a tongue diagnosis result based on the tongue features using the image diagnosis model.

[0018] The audio analysis module is used to acquire audio data of the user reading a preset text, perform loudness analysis on the audio data, and obtain the loudness value of the user's voice.

[0019] The comprehensive diagnostic module is used to generate a comprehensive diagnostic report by combining the basic information, pulse data, blood oxygen data, facial diagnosis results, tongue diagnosis results, and voice loudness value.

[0020] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the image recognition-based TCM diagnosis described above.

[0021] This invention acquires basic user information. It uses a pre-set acquisition device to obtain the user's pulse and blood oxygen data. A pre-set camera determines if the user is in a pre-set shooting location. If the user is not in the pre-set shooting location, a prompt is sent to the user based on the current location. If the user is in the pre-set shooting location, the camera acquires the user's facial image, and a pre-trained image diagnostic model generates a facial diagnosis result based on the facial image. The invention also acquires the user's tongue image, extracts tongue features from the tongue image, and uses the image diagnostic model to generate a tongue diagnosis result based on the tongue features. Finally, it acquires audio data of the user reading a pre-set text, performs loudness analysis on the audio data to obtain the user's voice loudness value, and combines the basic information, pulse data, blood oxygen data, facial diagnosis result, tongue diagnosis result, and voice loudness value to generate a comprehensive diagnostic report. This reduces the cost of health data collection, improves the efficiency of health data collection, allows users to flexibly select testing items according to their needs, meets different user testing requirements, and improves the efficiency of user health diagnosis. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of an application environment for a TCM diagnostic device based on image recognition, according to one embodiment of the present invention.

[0023] Figure 2 This is a functional block diagram of a traditional Chinese medicine diagnostic system based on image recognition provided in an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of the structure of an electronic device for implementing the image recognition-based TCM diagnostic device according to an embodiment of the present invention.

[0025] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0026] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0027] This application provides a traditional Chinese medicine (TCM) diagnostic system based on image recognition. The executing entity of the image recognition-based TCM diagnostic system includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the image recognition-based TCM diagnostic system can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0028] The image recognition-based TCM diagnostic system provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain the user's basic information through the client, acquire the user's pulse and blood oxygen data using a preset acquisition device, determine whether the user is in a preset shooting position using a preset camera, and if the user is not in the preset shooting position, issue a prompt message to the user based on the current position; if the user is in the preset shooting position, acquire the user's facial image using the camera, generate a facial diagnosis result based on the facial image using a pre-trained image diagnosis model, acquire the user's tongue image, extract the tongue features from the tongue image, generate a tongue diagnosis result based on the tongue features using the image diagnosis model, acquire the audio data of the user reading preset text, perform loudness analysis on the audio data to obtain the user's voice loudness value, and generate a comprehensive diagnostic report by combining the basic information, pulse data, blood oxygen data, facial diagnosis result, tongue diagnosis result, and voice loudness value. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0029] An electronic device for realizing traditional Chinese medicine diagnosis based on image recognition, characterized in that the electronic device comprises:

[0030] At least one processor;

[0031] And, a memory communicatively connected to the at least one processor;

[0032] The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the at least one processor to perform the following steps:

[0033] S1 is used to obtain basic user information.

[0034] In this embodiment of the invention, the user's basic information includes the user's gender, age, height, and weight.

[0035] In this embodiment of the invention, obtaining the user's basic information is achieved by displaying a visual interface to the user. The visual interface includes input boxes and prompts for the user to input basic information, with each input box corresponding to a user's basic information.

[0036] In detail, the user's basic information can be obtained after the user fills in the basic information in the input box and clicks the confirmation button.

[0037] In detail, basic user information can be obtained using mobile devices (phones).

[0038] In this embodiment of the invention, obtaining the user's basic information improves the accuracy of subsequent diagnoses.

[0039] S2 is used to acquire the user's pulse data and blood oxygen data using a preset acquisition device.

[0040] In this embodiment of the invention, the data acquisition device may be a pulse oximeter or an infrared sensor in a mobile device.

[0041] In detail, the pulse oximeter is a non-invasive medical device that optically measures and displays two main physiological parameters: oxygen saturation (SpO2) and pulse rate (PR). Oxygen saturation refers to the percentage of oxygen-binding hemoglobin in the blood relative to the total hemoglobin volume, and is an important indicator for assessing respiratory function and oxygen-carrying capacity. Pulse rate refers to the number of heartbeats per minute.

[0042] In detail, the working principle of the pulse oximeter is based on the change in light absorption during arterial pulsation. The instrument emits light of different wavelengths (usually red and infrared light) through body tissues (such as fingertips or earlobes) and measures the absorption rate of these light wavelengths. Because oxyhemoglobin and deoxyhemoglobin absorb these wavelengths of light differently, the instrument can calculate the oxygen saturation value. Pulse oximeters typically contain components such as a microprocessor, memory, LEDs, and photodiodes, which analyze changes in light absorption rate through algorithms to obtain blood oxygen saturation and pulse rate values.

[0043] In this embodiment of the invention, when acquiring the user's pulse data and blood oxygen data using a preset acquisition device, it is specifically used for:

[0044] The user's PPG signal is acquired using the acquisition device.

[0045] The user's pulse and blood oxygen data are calculated based on the PPG signal.

[0046] In detail, the PPG signal refers to photoplethysmography (PPG), a non-invasive detection method that monitors changes in blood volume in living tissue using photoelectric means. The PPG sensor mainly consists of one or more light-emitting diodes (LEDs) and a photodetector. A beam of light of a specific wavelength is irradiated onto the skin surface and then transmitted to a photodetector via transmission or reflection. Due to absorption and attenuation by the skin, muscles, and blood, the light intensity detected by the detector changes accordingly. While the absorption of light by skin and muscle tissue remains constant throughout the blood circulation, the blood volume within the skin undergoes pulsatile changes under the action of the heart, resulting in pulsatile changes in the received light intensity.

[0047] In detail, the detection principle of PPG signals is based on the fact that blood's absorption of light changes with the contraction and relaxation of the heart. During heart contraction, blood flow increases, light absorption increases, and the detected light intensity is minimal; during heart relaxation, blood flow decreases, light absorption decreases, and the detected light intensity is maximum. By converting this light intensity change signal, the change in volumetric pulse flow can be obtained, thus yielding a PPG waveform containing important physiological information of the cardiovascular system.

[0048] In this embodiment of the invention, by using a preset acquisition device to acquire the user's pulse data and blood oxygen data, the efficiency of generating a comprehensive diagnostic report is improved.

[0049] S3 is used to determine whether the user is in a preset shooting position using a preset camera.

[0050] In this embodiment of the invention, the step of using a preset camera to determine whether the user is in a preset shooting position is specifically used for:

[0051] The latest user facial image is acquired using the camera;

[0052] Identify the center point coordinates of the user in the latest user face image;

[0053] Determine whether the coordinates of the center point are within a preset center point coordinate range;

[0054] If the center point coordinates are not within the range of the center point coordinates, it is determined that the user is not in the preset shooting position, and a prompt message is generated based on the positional relationship between the center point coordinates and the range of the center point coordinates.

[0055] If the coordinates of the center point are within the range of the center point coordinates, then the user's face angle is obtained based on the latest user face image;

[0056] Determine whether the angle of the face is within a preset angle range;

[0057] If the face angle exceeds the angle range, it is determined that the user is not in the preset shooting position, and a prompt message is generated according to the numerical relationship between the face angle and the angle range.

[0058] If the face angle is within the specified angle range, obtain the distance between the user's face and the camera's face;

[0059] Determine whether the distance to the face is within a preset distance range;

[0060] If the distance to the face exceeds the specified distance range, it is determined that the user is not in the preset shooting position. A prompt message is generated based on the relationship between the distance to the face and the specified distance range.

[0061] If the distance to the face is within the specified distance range, the user is determined to be in the preset shooting position.

[0062] In detail, obtaining the user's facial angle based on the latest user facial image involves identifying the positions of the user's facial features in the latest user facial image, identifying the offset distance between the user's facial features and the midline based on the positions of the user's facial features, and calculating the user's facial angle based on the offset distance.

[0063] If the user is not in the preset shooting position, then execute S4, which is used to send a prompt message to the user based on the current position.

[0064] In this embodiment of the invention, the step of issuing a prompt to the user based on the current location is to generate the prompt based on the relationship between the relevant location data of the user's face and a preset standard range. For example, if the coordinates of the user's face center point are to the left of the preset center point coordinate range, a prompt is generated prompting the user to move their face to the right; if the angle of the user's face is shifted to the left compared to the preset angle range, a prompt is generated prompting the user to rotate their face to the right; if the user's face is too close to the camera, a prompt is generated prompting the user to move away from the camera.

[0065] In this embodiment of the invention, the step of issuing a prompt to the user based on the current location is to display the prompt to the user using a visualization screen that displays the user's facial image in real time.

[0066] In this embodiment of the invention, when the user is not in the preset shooting position, a prompt message is sent to the user based on the current position, thereby improving the efficiency of acquiring the user's facial image.

[0067] If the user is in a preset shooting position, then execute S5, which uses the camera to acquire the user's facial image and uses a pre-trained image diagnosis model to generate a facial diagnosis result based on the facial image.

[0068] In this embodiment of the invention, the image diagnosis model is a convolutional neural network model that can perform health diagnosis on the user based on the user's basic information and corresponding physiological images.

[0069] In detail, during the training of the image diagnostic model, a large amount of basic information of patients, as well as corresponding facial images and tongue images, are acquired as training data. The image diagnostic model is used to perform health diagnoses based on the training data, and the difference between the diagnostic results and the standard results is obtained. Based on the difference values, the hyperparameters of the image diagnostic model are continuously optimized using the backpropagation algorithm until the model's performance meets expectations or reaches the preset number of iterations, at which point the model training is confirmed to be complete.

[0070] Specifically, the hyperparameters include weight parameters, bias terms, and learning rate.

[0071] In detail, the weight parameters represent the connection strength between neurons in the model. In a neural network, each neuron is connected to neurons in the previous layer, and the weights determine the strength of these connections. The values ​​of the weights are typically randomly initialized at the start of model training and then continuously updated during training using optimization algorithms (such as gradient descent) to minimize the loss function. The magnitude and sign of the weights directly affect how well the model responds to input data, thus influencing the model's prediction results.

[0072] In detail, the bias term adds a constant offset to the model's output, acting similarly to the intercept in linear regression. The presence of the bias term allows the model not only to fit patterns in the data but also to adjust the overall position of the data. Without the bias term, the model might only capture data relationships passing through the origin, which is impractical in most cases. Therefore, the bias term increases the model's flexibility, enabling it to better adapt to and predict data.

[0073] In detail, the learning rate is a crucial hyperparameter controlling the step size of parameter updates during model training. It determines the magnitude by which the model adjusts the weights and biases based on the gradient of the loss function in each iteration. If the learning rate is set too large, the model may "jump" around the optimal solution or even diverge; if the learning rate is set too small, the model training process will be slow, and it may even get stuck in a local minimum. Therefore, choosing an appropriate learning rate is crucial for the model's fast convergence and performance.

[0074] In detail, the backpropagation algorithm is a key algorithm used in deep learning and neural networks for training models. It minimizes the loss function by calculating the gradient of the loss function with respect to the network parameters (weights and biases) and updating these parameters. The backpropagation algorithm is often used in conjunction with gradient descent or its variants, such as stochastic gradient descent.

[0075] In this embodiment of the invention, when the user facial diagnosis module 102 generates a facial diagnosis result based on the facial image using a pre-trained image diagnosis model, it is specifically used for:

[0076] Feature extraction is performed on the facial image to obtain facial features;

[0077] The gender data contained in the basic information is one-hot encoded to obtain gender encoding information. The weight data and age data contained in the basic information are normalized to obtain normalized weight and normalized age.

[0078] The combination of the gender encoding information, the normalized weight, and the normalized age is converted into a vector form to obtain the basic information feature vector;

[0079] The facial features are concatenated with the basic information feature vector to obtain facial fusion features;

[0080] The facial fusion features are processed by convolution using the image diagnostic model to obtain facial convolution features;

[0081] The facial convolutional features are pooled using the image diagnostic model to obtain facial pooled features;

[0082] The facial pooling features are activated using the image diagnostic model to obtain facial activation features;

[0083] The facial activation features are processed using the image diagnostic model to obtain the facial diagnosis results.

[0084] In detail, the process of performing convolution processing on the facial fusion features using the image diagnostic model involves sliding a convolution kernel across the facial fusion features with a preset stride, calculating the dot product between the convolution kernel and the covered portion after each slide, and obtaining the convolution feature. The convolution kernel is a matrix with a scale much smaller than the facial fusion features.

[0085] In detail, the one-hot encoding of the gender data contained in the basic information can refer to converting "male" to [0, 1] and "female" to [1, 0].

[0086] In detail, the process of pooling the facial convolutional features using the image diagnostic model involves sliding a rectangular window of a preset size over the facial convolutional features, obtaining the average or maximum value of the area covered by the rectangular window after each slide, and replacing the original value of that area with the pooled feature.

[0087] In detail, the activation processing of the facial pooling features using the image diagnostic model involves adding a non-linear factor to the facial pooling features using a preset activation function, thereby improving the model's feature recognition capability. Common activation functions include the Sigmoid function, the Tanh function, and the ReLU function.

[0088] In this embodiment of the invention, the process of performing fully connected processing on the facial activation features using the image diagnostic model involves calculating the output of each neuron in the model based on the model's weight parameters and bias terms to obtain the final facial diagnosis result.

[0089] In this embodiment of the invention, when the user is in a preset shooting position, the camera acquires the user's facial image, and a pre-trained image diagnostic model generates a facial diagnosis result based on the facial image. This improves the efficiency of subsequently generating a comprehensive diagnostic report.

[0090] S6. Used to acquire a user's tongue image, extract tongue features from the tongue image, and generate a tongue diagnosis result based on the tongue features using the image diagnosis model.

[0091] In this embodiment of the invention, the extraction of tongue surface features from the tongue surface image is achieved by extracting texture features from the tongue surface image to obtain an initial tongue surface image feature, then identifying the brightness and smoothness of the user's tongue surface based on the tongue surface image, and finally fusing the brightness and smoothness into the initial tongue surface feature to obtain the final tongue surface feature.

[0092] In this embodiment of the invention, the extraction of tongue surface features from the tongue surface image is specifically performed as follows:

[0093] Feature extraction is performed on the tongue surface image to obtain initial tongue surface features.

[0094] The brightness of the user's tongue surface is identified based on the tongue surface image;

[0095] The smoothness of the user's tongue surface is calculated based on the tongue surface image:

[0096] Calculate the absolute value of the difference between the brightness and the preset standard brightness to obtain the brightness difference value;

[0097] Calculate the absolute value of the difference between the smoothness and the preset standard smoothness to obtain the smoothness difference value;

[0098] The brightness difference value is added to the smoothness difference value to obtain the health discrimination value;

[0099] If the health discrimination value is greater than or equal to the preset health threshold, the initial tongue surface features are amplified using the preset first weighting coefficient to obtain the tongue surface features.

[0100] If the health discrimination value is less than the health threshold, the brightness and smoothness are normalized and then merged into a vector form to obtain the tongue surface feature vector. The tongue surface feature vector is then weighted and fused with the initial tongue surface features using a preset second weight coefficient to obtain the tongue surface features.

[0101] In detail, the preset standard brightness and the preset standard smoothness refer to the smoothness and brightness of a healthy tongue surface under normal circumstances.

[0102] Specifically, when the health discrimination value is greater than or equal to the preset health threshold, it indicates that the patient may have a significant health problem. Therefore, it is necessary to amplify the initial tongue surface features to emphasize the importance of tongue surface-related features in diagnosis.

[0103] In detail, the process of amplifying the initial tongue surface features using a preset first weighting coefficient refers to amplifying the feature values ​​in the initial tongue surface features and enhancing the feature intensity.

[0104] In this embodiment of the invention, calculating the smoothness of the user's tongue surface based on the tongue surface image includes:

[0105] Calculate the absolute value of the difference between the pixel value of each pixel and its neighboring pixels;

[0106] The product of the differences between each pixel and its neighboring pixels is calculated based on the absolute value of the calculated differences, and the sum of the calculated product of differences is used as the smoothness of the user's tongue surface.

[0107] In one embodiment of the present invention, the smoothness of the user's tongue surface is calculated using the following formula:

[0108] D = ∑ y ∑ x (|f(x,y)-f(x+1,y)|*|f(x,y)-f(x,y+1)|)

[0109] Wherein, D represents the smoothness, f(x,y) represents the pixel value of the pixel with coordinates (x,y) in the coordinate system established by the tongue image, f(x+1,y) represents the pixel value of the pixel with coordinates (x+1,y) in the coordinate system established by the tongue image, f(x,y+1) represents the pixel value of the pixel with coordinates (x,y+1) in the coordinate system established by the tongue image, and (x+1,y) and (x,y+1) are adjacent pixels of pixel (x,y).

[0110] In detail, the step of identifying the brightness of the user's tongue surface based on the tongue surface image involves using a preset image recognition algorithm to identify the color features of the user's tongue surface based on the tongue surface image, comparing the color features with standard tongue surface color features to obtain a comparison result, and mapping the comparison result into a numerical form to obtain the brightness.

[0111] In detail, the brightness is a numerical value used to reflect the color of the tongue surface.

[0112] In detail, the brightness can reflect the user's health status. Low brightness may indicate that the user has insufficient blood and qi, while high brightness may indicate that the user has body heat.

[0113] In detail, the smoothness is a numerical value used to reflect the roughness of a user's tongue surface, and thus reflect the user's health condition.

[0114] In this embodiment of the invention, the steps of generating tongue diagnosis results based on the tongue surface features using the image diagnosis model are the same as generating facial diagnosis results based on the facial image using a pre-trained image diagnosis model, and will not be repeated here.

[0115] In this embodiment of the invention, by acquiring a user's tongue image and extracting the tongue features from the tongue image, the efficiency of generating tongue diagnosis results is improved. The image diagnosis model is then used to generate tongue diagnosis results based on the tongue features, which improves the efficiency of generating a comprehensive diagnostic report.

[0116] S7. Used to acquire audio data of the user reading a preset text, perform loudness analysis on the audio data, and obtain the loudness value of the user's voice.

[0117] In this embodiment of the invention, the audio data of the user reading a preset text is obtained by displaying the preset text to the user on a visual screen, prompting the user to read the preset text, and then using a recording device to acquire the user's audio data. The audio data is longer than 2 seconds and less than 30 seconds.

[0118] In this embodiment of the invention, before acquiring the audio data of the user reading the preset text, an environmental noise detection is performed, and the user's audio data is acquired only after the environmental noise meets the requirements.

[0119] In this embodiment of the invention, the loudness value is a numerical value that reflects the loudness of the user's voice. When combined with the user's basic personal data, it can further reflect the user's health status.

[0120] In detail, the loudness score, as a numerical value with specific significance, primarily reflects the loudness of a user's voice. This loudness score does not exist in isolation; it plays a greater role when combined with the user's basic personal data. Through this combination, a deeper understanding of the user's health status can be gained. For example, the user's age, gender, physical condition, and other basic personal data are correlated with the loudness score, providing a more comprehensive and accurate basis for assessing the user's health status. Users of different age groups may have different ranges of loudness scores, and gender differences may also influence the relationship between loudness score and health status to some extent. Furthermore, users in good health may exhibit higher loudness scores, while users with certain health issues may have varying loudness scores. In short, the combination of loudness score and the user's basic personal data provides a new perspective and method for gaining a deeper understanding of the user's health status.

[0121] In this embodiment of the invention, when performing loudness analysis on the audio data to obtain the user's voice loudness value, it is specifically used for:

[0122] The audio data is sampled at preset intervals to obtain sampled audio.

[0123] The loudness value of the sound is calculated based on the sampled audio using the following formula:

[0124]

[0125] Where L is the loudness value of the sound, k is a preset weighting coefficient, N is the number of sampling points of the sampled audio, and x i This represents the amplitude value of the i-th sampling point in the sampled audio, where P is a preset reference sound pressure level.

[0126] Specifically, the preset reference sound pressure can be 20 micropascals (20 μPa).

[0127] In this embodiment of the invention, by acquiring audio data of a user reading a preset text, the efficiency of obtaining the user's voice loudness value is improved. By performing loudness analysis on the audio data, the user's voice loudness value is obtained, thereby improving the efficiency of generating a comprehensive diagnostic report.

[0128] S8. Used to combine the basic information, pulse data, blood oxygen data, facial diagnosis results, tongue diagnosis results, and voice loudness value to generate a comprehensive diagnostic report.

[0129] In this embodiment of the invention, since users of different ages, genders, heights and weights have different standard health values, it is necessary to provide a comprehensive diagnostic report based on the user's basic information.

[0130] In this embodiment of the invention, when generating a comprehensive diagnostic report by combining the basic information, the pulse data, the blood oxygen data, the facial diagnosis results, the tongue diagnosis results, and the voice loudness value, it is specifically used for:

[0131] Generate the user's individual vital sign tags based on the aforementioned basic information;

[0132] Based on the individual vital sign labels, a set of diagnostic data is obtained from a pre-established diagnostic database;

[0133] The comprehensive diagnostic report is obtained by matching the pulse data, blood oxygen data, facial diagnosis results, tongue diagnosis results, and voice loudness value with the diagnostic data set.

[0134] In this embodiment of the invention, the individual vital sign tag is a tag representing the user's physical condition. Since different users have different individual vital sign tags, and different individual vital sign tags have different health standards, it is necessary to retrieve data based on the individual vital sign tags.

[0135] In this embodiment of the invention, the step of generating a comprehensive diagnostic report by combining the basic information, pulse data, blood oxygen data, facial diagnosis results, tongue diagnosis results, and voice loudness value is based on the user's basic information, determining whether the user's four diagnostic results are within the standard range, and providing relevant diagnostic suggestions based on the user's four diagnostic results.

[0136] In detail, if all four diagnostic results of a user are normal, the user is deemed to have no health risks, and only some health advice is provided to the user.

[0137] Specifically, if a user's diagnostic result is abnormal, the comprehensive diagnostic report will be displayed to the user, and an access point for making a physical examination appointment will be provided to the user.

[0138] Specifically, if a user's symptoms have worsened compared to previous diagnostic data, the comprehensive diagnostic report will be displayed to the user, and an online doctor consultation portal will be provided.

[0139] In this embodiment of the invention, the collection of user health data can be completed on a mobile device, which greatly reduces the cost of health data collection and improves the efficiency of health data collection.

[0140] In this embodiment of the invention, a comprehensive diagnostic report is generated by combining the basic information, pulse data, blood oxygen data, facial diagnosis results, tongue diagnosis results, and voice loudness value, thereby improving the effectiveness of user health diagnosis.

[0141] As can be seen, in this embodiment of the invention, by utilizing a mobile device to acquire the user's basic personal data and four health data items, four health diagnoses are performed based on the four health data items, resulting in four diagnostic results. A comprehensive diagnostic report is then provided by combining the basic personal data and the four diagnostic results. This reduces the cost of health data collection, improves the efficiency of health data collection, allows users to flexibly select testing items as needed, meets different user testing requirements, and improves the efficiency of user health diagnosis. Figure 3 The diagram shown is a functional block diagram of a TCM diagnostic system based on image recognition provided in an embodiment of the present invention.

[0142] Reference Figure 2 The diagram shown is a functional block diagram of a TCM diagnostic system based on image recognition provided in an embodiment of the present invention.

[0143] The image recognition-based TCM diagnostic system 100 of this invention can be installed in an electronic device. Depending on the functions implemented, the image recognition-based TCM diagnostic system 100 may include a data acquisition module 101, a user facial diagnosis module 102, a user tongue diagnosis module 103, an audio analysis module 104, and a comprehensive diagnosis module 105. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, stored in the memory of the electronic device.

[0144] The following describes the components and workflow of an image recognition-based TCM diagnostic system using specific embodiments:

[0145] The data acquisition module 101 is used to acquire the user's basic information and to acquire the user's pulse data and blood oxygen data using a preset acquisition device.

[0146] The user face diagnosis module 102 is used to determine whether the user is in a preset shooting position using a preset camera. If the user is not in the preset shooting position, a prompt message is sent to the user based on the current position. If the user is in the preset shooting position, the user's facial image is acquired using the camera, and a face diagnosis result is generated based on the facial image using a pre-trained image diagnosis model.

[0147] The user tongue diagnosis module 103 is used to acquire the user's tongue image, extract the tongue features of the tongue image, and generate a tongue diagnosis result based on the tongue features using the image diagnosis model;

[0148] The audio analysis module 104 is used to acquire audio data of the user reading a preset text, perform loudness analysis on the audio data, and obtain the loudness value of the user's voice.

[0149] The comprehensive diagnostic module 105 is used to generate a comprehensive diagnostic report by combining the basic information, the pulse data, the blood oxygen data, the facial diagnosis results, the tongue diagnosis results, and the voice loudness value.

[0150] The image recognition-based TCM diagnostic system 100 of this invention can be installed in an electronic device. Depending on the functions implemented, the image recognition-based TCM diagnostic system 100 may include a data acquisition module 101, a user facial diagnosis module 102, a user tongue diagnosis module 103, an audio analysis module 104, and a comprehensive diagnosis module 105. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, stored in the memory of the electronic device.

[0151] like Figure 3 The diagram shown is a schematic diagram of the electronic device of the image recognition-based TCM diagnostic system provided in an embodiment of the present invention.

[0152] The electronic device 1 may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13. It may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a traditional Chinese medicine diagnostic program based on image recognition.

[0153] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., image recognition-based traditional Chinese medicine diagnostic programs) and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0154] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as the code of a traditional Chinese medicine diagnostic program based on image recognition, but also to temporarily store data that has been output or will be output.

[0155] The communication bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0156] The communication interface 13 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), or, optionally, a standard wired or wireless interface. Optionally, in this embodiment, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0157] The figure only shows an electronic device with components. Those skilled in the art will understand that the structure shown in the figure does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0158] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0159] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0160] The image recognition-based TCM diagnostic program stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When run in the processor 10, it can achieve the following:

[0161] Obtain basic user information;

[0162] The system uses a pre-set data acquisition device to obtain the user's pulse and blood oxygen data.

[0163] Use a preset camera to determine if the user is in a preset shooting position;

[0164] If the user is not in the preset shooting position, a prompt message will be sent to the user based on the current position;

[0165] If the user is in a preset shooting position, the camera is used to acquire the user's facial image, and a pre-trained image diagnosis model is used to generate a facial diagnosis result based on the facial image.

[0166] Obtain a user's tongue image, extract the tongue features from the tongue image, and use the image diagnosis model to generate a tongue diagnosis result based on the tongue features;

[0167] Acquire audio data of the user reading a preset text, perform loudness analysis on the audio data, and obtain the loudness value of the user's voice;

[0168] A comprehensive diagnostic report is generated by combining the basic information, pulse data, blood oxygen data, facial diagnosis results, tongue diagnosis results, and voice volume value.

[0169] Specifically, the specific implementation method of the processor 10 for the above instructions can be referred to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, and will not be repeated here.

[0170] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0171] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:

[0172] Obtain basic user information;

[0173] The system uses a pre-set data acquisition device to obtain the user's pulse and blood oxygen data.

[0174] Use a preset camera to determine if the user is in a preset shooting position;

[0175] If the user is not in the preset shooting position, a prompt message will be sent to the user based on the current position;

[0176] If the user is in a preset shooting position, the camera is used to acquire the user's facial image, and a pre-trained image diagnosis model is used to generate a facial diagnosis result based on the facial image.

[0177] Obtain a user's tongue image, extract the tongue features from the tongue image, and use the image diagnosis model to generate a tongue diagnosis result based on the tongue features;

[0178] Acquire audio data of the user reading a preset text, perform loudness analysis on the audio data, and obtain the loudness value of the user's voice;

[0179] A comprehensive diagnostic report is generated by combining the basic information, pulse data, blood oxygen data, facial diagnosis results, tongue diagnosis results, and voice volume value.

[0180] In the embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0181] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0182] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0183] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0184] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0185] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0186] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0187] Finally, it should be noted that if any software tools or components not belonging to this company appear in the embodiments of the application, they are merely illustrative examples and do not represent actual use. The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A TCM diagnostic device based on image recognition, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the at least one processor to perform the following steps: Obtain the user's basic information and use a pre-set data acquisition device to acquire the user's pulse and blood oxygen data; The system uses a preset camera to acquire the latest user face image, identifies the position of the user's facial features in the latest user face image, determines the user's offset distance based on the position of the user's facial features, and combines the coordinates of the user's center point in the latest user face image, the offset distance, and the distance between the user's face and the camera to determine whether the user is in the preset shooting position. If the user is not in the preset shooting position, a prompt message is issued to the user based on the user's current position. If the user is in the preset shooting position, the system uses the camera to acquire the user's facial image, and uses a pre-trained image diagnosis model to generate a facial diagnosis result based on the facial image. The process involves acquiring a user's tongue image, extracting features from the image to obtain initial tongue features, identifying the brightness of the tongue based on the image, calculating the smoothness of the tongue based on the image, calculating the absolute value of the difference between the brightness and a preset standard brightness to obtain a brightness difference value, calculating the absolute value of the difference between the smoothness and a preset standard smoothness to obtain a smoothness difference value, and adding the brightness difference value and the smoothness difference value to obtain a health assessment value. Here, the preset standard brightness refers to the brightness of a healthy tongue surface. Smoothness refers to the smoothness of a healthy tongue surface. If the health discrimination value is greater than or equal to a preset health threshold, the initial tongue surface features are amplified using a preset first weighting coefficient to obtain tongue surface features. If the health discrimination value is less than the health threshold, the brightness and smoothness are normalized and then merged into a vector form to obtain a tongue surface feature vector. The tongue surface feature vector is then weighted and fused with the initial tongue surface features using a preset second weighting coefficient to obtain tongue surface features. The image diagnostic model generates a tongue diagnosis result based on the tongue surface features. Acquire audio data of the user reading a preset text, perform loudness analysis on the audio data, and obtain the loudness value of the user's voice; A comprehensive diagnostic report is generated by combining the basic information, pulse data, blood oxygen data, facial diagnosis results, tongue diagnosis results, and voice volume value.

2. The image recognition-based TCM diagnostic device as described in claim 1, characterized in that, The process of acquiring the user's pulse and blood oxygen data using a pre-set acquisition device includes: The user's PPG signal is acquired using the acquisition device. The user's pulse and blood oxygen data are calculated based on the PPG signal.

3. The image recognition-based TCM diagnostic device as described in claim 1, characterized in that, The step of determining whether the user is in the preset shooting position by combining the user's center point coordinates in the latest user face image, the offset distance, and the distance between the user's face and the camera includes: Determine whether the coordinates of the center point are within a preset center point coordinate range; If the center point coordinates are not within the range of the center point coordinates, it is determined that the user is not in the preset shooting position, and the prompt information is generated according to the positional relationship between the center point coordinates and the range of the center point coordinates; If the coordinates of the center point are within the range of the center point coordinates, then the user's face angle is calculated based on the offset distance; Determine whether the angle of the face is within a preset angle range; If the face angle exceeds the angle range, it is determined that the user is not in the preset shooting position, and the prompt information is generated according to the numerical relationship between the face angle and the angle range. If the face angle is within the specified angle range, determine whether the face distance is within a preset distance range; If the distance to the face exceeds the distance range, it is determined that the user is not in the preset shooting position; the prompt message is generated based on the relationship between the distance to the face and the distance range. If the distance to the face is within the specified distance range, the user is determined to be in the preset shooting position.

4. The image recognition-based TCM diagnostic device as described in claim 1, characterized in that, The process of generating a facial diagnosis result based on the facial image using a pre-trained image diagnosis model includes: Feature extraction is performed on the facial image to obtain facial features; The gender data contained in the basic information is one-hot encoded to obtain gender encoding information. The weight data and age data contained in the basic information are normalized to obtain normalized weight and normalized age. The combination of the gender encoding information, the normalized weight, and the normalized age is converted into a vector form to obtain the basic information feature vector; The facial features are concatenated with the basic information feature vector to obtain facial fusion features; The facial fusion features are processed by convolution using the image diagnostic model to obtain facial convolution features; The facial convolutional features are pooled using the image diagnostic model to obtain facial pooled features; The facial pooling features are activated using the image diagnostic model to obtain facial activation features; The facial activation features are processed using the image diagnostic model to obtain the facial diagnosis results.

5. The image recognition-based TCM diagnostic device as described in claim 1, characterized in that, The step of calculating the smoothness of the user's tongue surface based on the tongue surface image includes: Calculate the absolute value of the difference between the pixel value of each pixel and its neighboring pixels; The product of the differences between each pixel and its neighboring pixels is calculated based on the absolute value of the calculated differences, and the sum of the calculated product of differences is used as the smoothness of the user's tongue surface.

6. The image recognition-based TCM diagnostic device as described in claim 1, characterized in that, The step of performing loudness analysis on the audio data to obtain the user's voice loudness value includes: The audio data is sampled at preset intervals to obtain sampled audio. The loudness value of the sound is calculated based on the sampled audio and using a predetermined formula: The parameters of the predetermined formula include preset weighting coefficients, the number of sampling points in the sampled audio, and the number of sampling points in the sampled audio. The amplitude value of each sampling point, and the preset reference sound pressure.

7. The image recognition-based TCM diagnostic device as described in claim 1, characterized in that, The process of generating a comprehensive diagnostic report by combining the basic information, pulse data, blood oxygen data, facial diagnosis results, tongue diagnosis results, and voice loudness value includes: Generate the user's individual vital sign tags based on the aforementioned basic information; Based on the individual vital sign labels, a set of diagnostic data is obtained from a pre-established diagnostic database; The comprehensive diagnostic report is obtained by matching the pulse data, blood oxygen data, facial diagnosis results, tongue diagnosis results, and voice loudness value with the diagnostic data set.

8. A traditional Chinese medicine diagnostic system based on image recognition, characterized in that, The system includes a data acquisition module, a user face diagnosis module, a user tongue diagnosis module, an audio analysis module, and a comprehensive diagnosis module, wherein: The data acquisition module is used to acquire the user's basic information and to acquire the user's pulse data and blood oxygen data using a preset acquisition device. The user face diagnosis module is used to acquire the latest user face image using a preset camera, identify the position of the user's facial features in the latest user face image, determine the user's offset distance based on the position of the user's facial features, and determine whether the user is in a preset shooting position by combining the coordinates of the user's center point in the latest user face image, the offset distance, and the distance between the user's face and the camera. If the user is not in the preset shooting position, a prompt message is issued to the user based on the user's current position. If the user is in the preset shooting position, the camera is used to acquire the user's facial image, and a pre-trained image diagnosis model is used to generate a face diagnosis result based on the facial image. The user tongue diagnosis module is used to acquire a user's tongue image, extract features from the tongue image to obtain initial tongue features, identify the brightness of the user's tongue based on the tongue image, calculate the smoothness of the user's tongue based on the tongue image, calculate the absolute value of the difference between the brightness and a preset standard brightness to obtain a brightness difference value, calculate the absolute value of the difference between the smoothness and a preset standard smoothness to obtain a smoothness difference value, and add the brightness difference value and the smoothness difference value to obtain a health judgment value. The preset standard brightness refers to the brightness of a healthy tongue surface. The standard smoothness is defined as the smoothness of a healthy tongue surface. If the health discrimination value is greater than or equal to a preset health threshold, the initial tongue surface features are amplified using a preset first weighting coefficient to obtain tongue surface features. If the health discrimination value is less than the health threshold, the brightness and smoothness are normalized and then merged into a vector form to obtain a tongue surface feature vector. The tongue surface feature vector is then weighted and fused with the initial tongue surface features using a preset second weighting coefficient to obtain tongue surface features. The image diagnostic model generates a tongue diagnosis result based on the tongue surface features. The audio analysis module is used to acquire audio data of the user reading a preset text, perform loudness analysis on the audio data, and obtain the loudness value of the user's voice. The comprehensive diagnostic module is used to generate a comprehensive diagnostic report by combining the basic information, pulse data, blood oxygen data, facial diagnosis results, tongue diagnosis results, and voice loudness value.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it performs the following steps: Obtain the user's basic information and use a pre-set data acquisition device to acquire the user's pulse and blood oxygen data; The system uses a preset camera to acquire the latest user face image, identifies the position of the user's facial features in the latest user face image, determines the user's offset distance based on the position of the user's facial features, and combines the coordinates of the user's center point in the latest user face image, the offset distance, and the distance between the user's face and the camera to determine whether the user is in the preset shooting position. If the user is not in the preset shooting position, a prompt message is issued to the user based on the user's current position. If the user is in the preset shooting position, the system uses the camera to acquire the user's facial image, and uses a pre-trained image diagnosis model to generate a facial diagnosis result based on the facial image. The process involves acquiring a user's tongue image, extracting features from the image to obtain initial tongue features, identifying the brightness of the tongue based on the image, calculating the smoothness of the tongue based on the image, calculating the absolute value of the difference between the brightness and a preset standard brightness to obtain a brightness difference value, calculating the absolute value of the difference between the smoothness and a preset standard smoothness to obtain a smoothness difference value, and adding the brightness difference value and the smoothness difference value to obtain a health assessment value. Here, the preset standard brightness refers to the brightness of a healthy tongue surface. Smoothness refers to the smoothness of a healthy tongue surface. If the health discrimination value is greater than or equal to a preset health threshold, the initial tongue surface features are amplified using a preset first weighting coefficient to obtain tongue surface features. If the health discrimination value is less than the health threshold, the brightness and smoothness are normalized and then merged into a vector form to obtain a tongue surface feature vector. The tongue surface feature vector is then weighted and fused with the initial tongue surface features using a preset second weighting coefficient to obtain tongue surface features. The image diagnostic model generates a tongue diagnosis result based on the tongue surface features. Acquire audio data of the user reading a preset text, perform loudness analysis on the audio data, and obtain the loudness value of the user's voice; A comprehensive diagnostic report is generated by combining the basic information, pulse data, blood oxygen data, facial diagnosis results, tongue diagnosis results, and voice volume value.

Images