Tongue diagnosis auxiliary method and system

Abstract

This invention discloses a tongue diagnosis assistance method and system. The method includes: acquiring subject information and acquisition requirements; acquiring the subject's facial positioning information via a camera on a robotic arm, the robotic arm being able to move and rotate in the X, Y, and Z directions in space; adjusting the robotic arm to a first precise position based on the facial positioning information to center the subject's face; acquiring the subject's tongue image positioning information via the camera based on the first precise position; adjusting the robotic arm to a second precise position based on the tongue image coordinates; acquiring the subject's tongue image via the camera based on the second precise position according to the acquisition requirements; and uploading the subject information and the tongue image. This method can be used to quickly and conveniently acquire tongue images, solving problems such as the complexity of current tongue diagnosis assistance operations, inability to adapt to different usage environments, and low acquisition efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of traditional Chinese medicine diagnosis and treatment equipment, and in particular to a tongue diagnosis auxiliary method and system. Background Art

[0002] Tongue diagnosis is an important part of traditional Chinese medicine diagnosis. It judges a person's health status by observing the color, shape, and tongue coating of the tongue. However, traditional tongue image collection mainly relies on direct observation by doctors, which is not only time-consuming and labor-intensive, but also cannot be guaranteed in terms of accuracy and repeatability due to subjective judgment. With the development of technology, although there have been some attempts to automatically collect tongue images, most of them still have problems such as complex operation, inability to adapt to different usage environments, and low collection efficiency. In view of the current problems of tongue image collection and other tongue diagnosis auxiliary operations being complex, inability to adapt to different usage environments, and low collection efficiency, it is necessary to conduct further research on tongue diagnosis auxiliary technology. Summary of the invention

[0003] The embodiments of the present invention provide a tongue diagnosis auxiliary method, device and storage medium to solve the problems of current tongue diagnosis auxiliary operations such as tongue image collection being complex, unable to adapt to different usage environments, and having low collection efficiency.

[0004] In one aspect, the present invention provides a tongue diagnosis auxiliary method, comprising:

[0005] Obtain information about the person being collected and collection requirements;

[0006] The camera on the robotic arm obtains the facial positioning information of the person being collected. The robotic arm can move and rotate in three directions: X, Y, and Z.

[0007] Adjusting the robotic arm to a first precise position according to the face positioning information so as to center the face of the person being collected;

[0008] Based on the first precise position, the tongue image positioning information of the subject is obtained through the camera;

[0009] Adjust the robotic arm to a second precise position according to the tongue image coordinate information;

[0010] According to the collection requirements, the tongue image of the subject is collected through the camera based on the second precise position;

[0011] Upload the information of the person being collected and the tongue image.

[0012] Compared with the prior art, the present invention has the following advantages:

[0013] By using a robotic arm that can move and rotate in the X, Y, and Z directions in space, the camera can be automatically adjusted to the second precise position based on precise face positioning information and tongue positioning information. Tongue images can be quickly and conveniently acquired, thus solving the current problems of tongue image acquisition and other tongue diagnosis auxiliary operations being complex, unable to adapt to different usage environments, and having low acquisition efficiency.

[0014] In another aspect, the present invention provides a tongue diagnosis auxiliary system, comprising:

[0015] The first acquisition module is used to obtain the information of the person being collected and the collection requirements;

[0016] The second acquisition module is used to obtain the face positioning information of the subject through the camera on the robotic arm. The robotic arm can move and rotate in the three directions of X, Y and Z in space;

[0017] A first adjustment module is used to adjust the robot arm to a first precise position according to the face positioning information so as to center the face of the person being collected;

[0018] A third acquisition module is used to acquire tongue image location information of the subject through a camera based on the first precise position;

[0019] A second adjustment module, used for adjusting the robot arm to a second precise position according to the tongue image coordinate information;

[0020] A collection module, used for collecting the tongue image of the subject through a camera based on the second precise position according to the collection requirements;

[0021] The upload module is used to upload the information of the person being collected and the tongue image.

[0022] In another aspect, the present invention provides a tongue diagnosis auxiliary system, comprising a mechanical arm and a computing center; the computing center is used for:

[0023] Obtain information about the person being collected and collection requirements;

[0024] Obtain a face image captured by the robotic arm, analyze it to obtain face positioning information, and send the face positioning information to the robotic arm;

[0025] Obtaining the tongue image collected by the robotic arm, analyzing to obtain tongue image positioning information, and sending the tongue image positioning information to the robotic arm;

[0026] After receiving the tongue image collected by the robotic arm, upload the information of the person being collected and the tongue image;

[0027] The robot arm can move and rotate in three directions, X, Y, and Z, in space. The robot arm includes a camera and is used to:

[0028] Obtain information about the person being collected and collection requirements;

[0029] Collect facial images based on the camera and send them to the computing center;

[0030] Obtain face positioning information sent by the computing center;

[0031] Adjusting the robotic arm to a first precise position according to the face positioning information so as to center the face of the person being collected;

[0032] Based on the first precise position, a tongue image is collected via a camera and sent to a computing center;

[0033] Obtain tongue image positioning information sent by the computing center;

[0034] Adjust the robotic arm to a second precise position according to the tongue image coordinate information;

[0035] According to the collection requirements, the tongue image of the subject is collected through the camera based on the second precise position;

[0036] Upload tongue images to the computing center.

[0037] Furthermore, the robotic arm and the computing center are connected via a remote or local area network. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0039] Figure 1 It is a flow chart of a tongue diagnosis auxiliary method according to an embodiment of the present invention.

[0040] Figure 2 This is a schematic diagram of a tongue image positioning model data set and a model output effect diagram according to an embodiment of the present invention.

[0041] Figure 3 Schematic diagram of a tongue diagnosis auxiliary device according to another embodiment of the present invention.

[0042] Figure 4 Schematic diagram of a tongue diagnosis auxiliary device according to another embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] See also Figure 1 A tongue diagnosis auxiliary method according to an embodiment of the present invention comprises the following steps:

[0045] Step 11, obtain the information of the person being collected and the collection requirements.

[0046] In the embodiment of the present invention, the system can be activated by clicking the start button. The system first runs a Python script to confirm the start command and check the set number of samples collected. In addition, since the system startup method is based on network methods such as http, after the subsequent adjustment of the user interface, the user can connect to the hotspot of the robotic arm on the mobile phone, log in to the control interface, and control the collection of the robotic arm through the mobile phone, which greatly facilitates the use of the robotic arm.

[0047] Step 12, the face positioning information of the person being collected is obtained through the camera on the robotic arm, and the robotic arm can move and rotate in three directions of X, Y, and Z in space.

[0048] In this step, once the system is started, another Python script activates data transmission between the robotic arm and the computing hub. The computing hub processes the images from the robotic arm’s camera in real time, determines the position of the face through face tracking technology, and sends movement instructions to the robotic arm to adjust its position to ensure that the face is always in the center of the picture.

[0049] As one implementation method, step 12 may include:

[0050] Based on the face positioning model, the face positioning information is obtained through the camera; the training steps of the face positioning model include:

[0051] Apply the yolov8n model, and use the open source training set coco2017 and faces4coco pre-training to obtain a pre-trained model;

[0052] Re-label 500 photos and about 2,000 face data, and use the pre-trained model for subsequent training and verification operations to obtain a face positioning model.

[0053] In an embodiment of the present invention, the computer for model training and the computer of the computing center are the same computer. The only difference is that the system used for training is WSL-Ubuntu-22.04, and when used as a computing center, it is windows11--version23H2; the python environment used is created using Anaconda3, and the cuda version is 12.2; the detailed configuration of the computer is: CPU-AMD Ryzen9 7950x, memory-DDR5 6000Mhz 32G, GPU-NVIDIA GeForce RTX4090.

[0054] Specifically: the initial training of the model used the yolov8n model for training, and the training sets used were the open source training sets coco2017 and faces4coco. After 50 epochs of training, the accuracy rate was stable at 60% for a long time; after searching the data set, it was found that in addition to the accuracy rate, there were also problems with the recognition accuracy, and the faces in the pictures would not be recognized. Then, after visualizing the training set, it was found that the problem was with the faces4coco training set; therefore, 500 photos, about 2,000 face data, were re-labeled in the end, and the model with 60% accuracy was used as the pre-training model for subsequent training and verification operations; in the later stage of training, the training of the yolov8m model was also added. Under the training of poch, the yolo8n model achieved an accuracy of 0.8875, and the yolo8m model reached 0.8617, and it was often found in subsequent training that the problem that the accuracy of yolo8m could not be increased appeared in the insufficient number of data sets; in the recall parameter, the optimal performance of yolov8m was 0.9027, and the optimal performance of yolov8n was 0.8844: in terms of processing speed, the two models were inferred 10 times, and the average of yolo8n was 50.03ms, and the average of yolo8m was 60.69ms; therefore, under comprehensive consideration, the selection of the yolov8n model will further improve the overall response of the system, and can also meet the use requirements of the present invention. Please refer to Table 1, Yuan 1 is a comparison of the inference speed of the two models of the face recognition model of an embodiment of the present invention under 10 inferences, and its unit is milliseconds (ms).

[0055] 1 2 3 4 5 6 7 8 9 10 ave yolov8n 50.2 48.7 49.1 52 49.6 49.7 56.7 50.2 45.3 48.8 50.03 yolov8m 64.8 62.9 63.4 61.3 56.6 56.5 62.4 61.4 57.9 59.7 60.69

[0056] Table 1 Comparison of inference speed of two models of face recognition model under 10 inferences

[0057] Step 13, adjust the robotic arm to a first precise position according to the face positioning information so as to center the face of the person being collected.

[0058] Step 14: Based on the first precise position, the tongue image positioning information of the subject is obtained through a camera.

[0059] As one implementation method, step 14 may include:

[0060] Based on the first precise position of the tongue image positioning model, the tongue image positioning information is obtained through the camera; the training steps of the tongue image positioning model include:

[0061] The U-net model or the yolo model was applied, and the learning rate of 1e-8 was adopted for 100 epochs of training, so that the model loss dropped to 0.2202 at 4344 steps, and the tongue image localization model was obtained.

[0062] Specifically: the training set (1012 photos) of this model comes from Guangzhou University of Chinese Medicine. Since the data set only contains a small number of faces in the mouth and near the mouth, the U-net model is used in this example implementation, while the yolo model should be used in the conception of the present invention to improve the positioning speed. The following learning rates (learning rate, e represents the power of 10) are used in the training of this model: 1e-3, 1e-5, 1e-6, 1e-7, 1e-8. After actual training, it was found that due to the complexity of the data, when the learning rate was 1e-3, the gradient disappeared when the model training reached the 300th step: among the remaining four learning rates, each was tested for 10 epochs. In the case of trial training, the model loss decreased the fastest at the learning rate of 1e-5 and the final loss was the lowest at 0.2844. However, after actual observation, the loss curves of the first three of the remaining four learning rates were very steep and often rebounded to a higher loss. It can be judged that the loss often encountered during the gradient descent process due to the high learning rate did not converge and the gradient descent did not work properly. Therefore, the example finally selected a learning rate of 1e-8 for 100 epochs of training, which reduced the model loss to 0.2202 at the 4344th step (92nd epoch). At this time, the model can output the tongue mask more accurately. Please refer to Figure 2 , Figure 2 This is a schematic diagram of a tongue image positioning model data set and a model output effect diagram according to an embodiment of the present invention.

[0063] Step 15, adjusting the robotic arm to a second precise position according to the tongue image coordinate information.

[0064] Step 16: According to the acquisition requirement, the tongue image of the subject is acquired through a camera based on the second precise position.

[0065] In an embodiment of the present invention, as one of the implementation methods, the system can enter the tongue image collection mode after the face of the person is stable in the center of the screen for more than 2 seconds. The robotic arm adjusts its position close to the user's tongue according to the instructions sent back by the computing center. The computing center analyzes the collected tongue image and saves the image to a designated folder according to the naming rules set by the user when the position is correct.

[0066] Step 17: Upload the information of the person being collected and the tongue image.

[0067] As one implementation manner, after step 17, the method further includes:

[0068] Analyze the tongue image based on the classification model to obtain classification information of the tongue image;

[0069] According to the classified information, upload the information of the person being collected and the tongue image.

[0070] In an embodiment of the present invention, the classification model includes a crack classification model, and the crack classification model selects SwinTransformer to process image data.

[0071] As one implementation method, the training steps of the crack classification model include:

[0072] Apply the swin_base_patch4_window7_224 model of swin-transformer, use the python script to read all the images and labels in the crack folder and the crack-free folder, and split the training and validation data sets in a ratio of 7:3. Create folders 0 and 1 in the train and val folders to represent the two labels and put the images in them.

[0073] With an initial training rate of 0.01, 200 epochs of training were performed. When the loss in the training set reached 0.2944, the loss in the validation set reached 0.4855, and the Acc@1 in the validation set was 81.188%, the crack classification model was obtained.

[0074] Specifically: The training set of this model (1012 photos) comes from Guangzhou University of Chinese Medicine. The classification model used in this example is the swin_base_patch4_window7224 model of swin-transformer: The training set was originally divided into folders only by train and test, and the labels (cracks and no cracks) were saved in an excel table. Therefore, before training, a python script was first used to read all the pictures in the two folders, and then the labels corresponding to the pictures were found from the excel table, and the number of training (train) and verification (val) data sets were divided according to a ratio of 7:3. Finally, new folders 0 and 1 were created in the train and val folders to represent the two labels and put the pictures in; at an initial training rate of 0.01, 200 epochs of training were performed, and the best result after the overall training was

[0075] At 178 epochs, the loss in the training set reached 0.2944, the loss in the val set reached 0.4855, and the Acc@1 (the probability that the item with the highest probability in the output result is the correct label) in the validation set was 81.188%, which basically meets the requirements of the present invention.

[0076] As one implementation mode, the tongue diagnosis auxiliary method further includes:

[0077] When the robot moves abnormally, a restart prompt is issued. Specifically: To ensure the stable operation of the system and the safety of users, the robot arm of this system is designed with an automatic reset function, which will be executed every time the system is turned on to ensure the accuracy of the motion trajectory. If abnormal movement is encountered during use, the system will immediately issue a prompt to guide the user to turn off the robot arm according to the safe operation instructions and manually adjust it to the initial state. After completion, the user can restart the robot arm, and the system will execute the reset procedure again to ensure that the acquisition activity can proceed smoothly.

[0078] When the camera cannot recognize a face, it will issue a check prompt. After obtaining the transmission image of the test gesture, it will obtain the face positioning information. Specifically: When the face cannot be recognized, the system will first prompt the user to check whether there are any obstructions to the camera or whether the camera angle is correct. The user can test whether the camera image transmission is normal by performing simple actions, such as waving. If the problem persists, the system will further guide the user to contact technical support so that professionals can provide deeper technical assistance to ensure that the user can smoothly use the system to collect tongue images.

[0079] When the tongue image is not clear, a troubleshooting step prompt is issued and the tongue image is recaptured. Specifically: For the problem of unclear tongue images, the system will guide the user to perform simple self-service troubleshooting steps. First, the user is prompted to wipe the camera to remove possible stains or water mist. Then, it is recommended that the user briefly leave the camera's field of view and then re-enter so that the system can readjust the focus and light to optimize the image acquisition effect. Through these steps, the system aims to help users obtain clearer tongue images for subsequent analysis and evaluation.

[0080] In the field of artificial intelligence, especially the progress of machine learning and image processing technology, the emergence of the Transformer architecture marks a new milestone. This architecture can process sequence data through its self-attention mechanism, which improves the model's ability to process complex data, especially in the fields of natural language processing (NLP) and computer vision (CV). For the automated tongue image acquisition system proposed in this patent, these technological breakthroughs mean that face tracking and tongue image recognition and analysis can be performed more accurately. Using advanced architectures such as Transformer, the system can learn deeper features from complex image data, thereby achieving more accurate target recognition and positioning, and further improving the efficiency and accuracy of tongue image acquisition.

[0081] The embodiment of the present invention realizes high-precision automatic collection of tongue images by integrating advanced face tracking technology and precise mechanical arm operation, which is particularly suitable for the fields of medical diagnosis and health examination. First, the user's start-up instruction is received and the collection parameters, such as the number of samples collected, are set; after receiving the start-up instruction, it is activated and real-time data exchange is performed with the computing center, which is responsible for executing face tracking and tongue image positioning; the computing center waits for the data transmitted by the mechanical arm to perform face tracking or tongue image positioning, and returns the corresponding data to the mechanical arm: then the face positioning is performed, and the coordinates of the face in the received data are returned; then the tongue image positioning is performed, and the coordinates of the tongue image in the received data are returned. Finally, the tongue image is simply analyzed and the corresponding results are returned to the user to assist in medical judgment. Face tracking technology ensures that the target face can be accurately identified and tracked in a dynamic environment, and the mechanical arm adjusts its position according to the tracking results so that the tongue image is always at the best collection angle and distance. When the system detects that the face remains in the center of the screen for a certain period of time, it automatically switches to the tongue image acquisition mode. At this time, the computing center analyzes the tongue image position according to the preset algorithm and guides the mechanical arm to make fine adjustments to capture high-quality tongue images. After the acquisition is completed, the system automatically saves the image to a specific location according to the naming rules specified by the user, which is convenient for subsequent processing and analysis.

[0082] The tongue diagnosis auxiliary method proposed in the embodiment of the present invention can quickly and conveniently collect tongue images by automatically adjusting the camera to the second precise position according to the precise face positioning information and tongue image positioning information through a mechanical arm that can move and rotate in the three directions of X, Y, and Z in space, thereby solving the problems of complex tongue diagnosis auxiliary operations such as tongue image collection, inability to adapt to different usage environments, and low collection efficiency. By introducing the latest artificial intelligence technology and some existing stabilization technologies, the embodiment of the present invention can improve the accuracy of face and tongue image collection, thereby optimizing the overall performance of medical diagnosis and related application fields.

[0083] Finally, considering that the problem of crack judgment may be related to the fact that the patient collects tongue images at different times, because everyone's tongue image is different, in order to better judge the cracks of the tongue image or add more judgments on the tongue image in the future, the crack classification model in the computing center uses swin-transformer. When processing image data, Swin Transformer can more effectively capture the global dependencies and detail features of the image through its self-attention mechanism, which is crucial for the accurate classification of tongue images. Compared with traditional CV methods, Swin Transformer provides better feature representation capabilities, which can improve the accuracy and robustness of classification tasks, especially in small sample learning scenarios. In addition, the hierarchical structure of Swin Transformer makes it more efficient when processing large-scale image data, thereby optimizing the use of computing resources while maintaining high accuracy.

[0084] In summary, the embodiments of the present invention combine cutting-edge artificial intelligence technology with advanced mechanical automation to create an automated tongue image collection system that is both efficient and accurate. This system is not only user-friendly, but also has achieved certain results in improving the quality and efficiency of tongue image collection for medical diagnosis and health examinations. Through sophisticated face tracking and tongue image positioning, as well as auxiliary analysis of intelligent algorithms, this system can ensure that the collected tongue images have extremely high accuracy, thus providing medical professionals with a powerful tool to support more accurate health status assessment and disease diagnosis.

[0085] See also Figure 3 A tongue diagnosis auxiliary system 20 according to another embodiment of the present invention can be used to implement the corresponding steps of the above method embodiment, including:

[0086] The first acquisition module 21 is used to acquire the information of the person being collected and the collection requirements;

[0087] The second acquisition module 22 is used to acquire the face location information of the subject through the camera on the robotic arm, and the robotic arm can move and rotate in three directions of X, Y and Z in space;

[0088] A first adjustment module 23, used for adjusting the robot arm to a first precise position according to the face positioning information, so as to center the face of the person being collected;

[0089] A third acquisition module 24 is used to acquire tongue image location information of the subject through a camera based on the first precise position;

[0090] The second adjustment module 25 is used to adjust the robot arm to a second precise position according to the tongue image coordinate information;

[0091] The acquisition module 26 is used to acquire the tongue image of the subject through the camera based on the second precise position according to the acquisition requirement;

[0092] The uploading module 27 is used to upload the information of the person being collected and the tongue image.

[0093] See also Figure 4 A tongue diagnosis auxiliary device 30 according to another embodiment of the present invention can be used to implement the corresponding steps of the above method embodiment, including a mechanical arm 31 and a computing center 32; the computing center 32 is used to:

[0094] Obtain information about the person being collected and collection requirements;

[0095] Obtain a face image captured by the robotic arm, analyze it to obtain face positioning information, and send the face positioning information to the robotic arm;

[0096] Obtaining the tongue image collected by the robotic arm, analyzing to obtain tongue image positioning information, and sending the tongue image positioning information to the robotic arm;

[0097] After receiving the tongue image collected by the robotic arm, upload the information of the person being collected and the tongue image.

[0098] In an embodiment of the present invention, the deployment of each module on the computing center is based on the flask package of python; first, the overall deployment of flask is performed. Since the computing center is a computing center and does not require a web page, no web page template is deployed; secondly, a YOLO inference function based on YOLOv8 of ultralytics is encapsulated, and then the application is deployed in flask with a route of " / yoloface". This route first detects whether the request sent by the robot contains a picture, and then temporarily caches the picture, and then calls the YOLO inference function for face detection, and returns the resulting face box (boxes) and center point (midpoint) to the robot; the application is deployed in flask with a route of " / unetmaskcenter" again, and this route also The request sent by the robot arm will be checked to see if it contains the content of the picture, and then the inference function will be temporarily cached and called to locate the tongue image and return the center point of the corresponding tongue image to the robot arm; finally, it will not only receive the picture transmitted by the robot arm, but also receive the information of the person being collected transmitted by the robot arm, and use the information of the person being collected to detect whether there is a folder of the person being collected under the specified folder. If not, it will be created. If there is, a new folder will be created in it according to the time information to save the picture of the person being collected, and then the photo of the person being collected will be input into the trained swin-tranformer model for inference, and the inference result will be sent to the medical big language model through the set medical big language model API interface (this example is the interface of OpenAI, and the model used is GPT-4Turbo) for the cause analysis and life advice result generation. Among them, the information of the person being collected and the collection requirements can also be directly collected through the computing center. In addition, the computing center is equipped with face recognition and tongue image positioning functions for accurately analyzing the characteristics of the tongue image and performing simple classification, such as judging whether there is a crack.

[0099] The robot arm 31 can move and rotate in three directions of X, Y, and Z in space. The robot arm 31 includes a camera. The robot arm 32 is used for:

[0100] Obtain information about the person being collected and collection requirements;

[0101] Collect facial images based on the camera and send them to the computing center;

[0102] Obtain face positioning information sent by the computing center;

[0103] Adjusting the robotic arm to a first precise position according to the face positioning information so as to center the face of the person being collected;

[0104] Based on the first precise position, a tongue image is collected via a camera and sent to a computing center;

[0105] Obtain tongue image positioning information sent by the computing center;

[0106] Adjust the robotic arm to a second precise position according to the tongue image coordinate information;

[0107] According to the collection requirements, the tongue image of the subject is collected through the camera based on the second precise position;

[0108] Upload tongue images to the computing center.

[0109] In an embodiment of the present invention, the robotic arm is a six-axis small robotic arm, which is equipped with a camera for capturing images and is equipped with an embedded system. The system can run complex Python scripts and support ROS (Robot Operating System) environment to implement precise target tracking and positioning operations. In addition, the robotic arm is also integrated with a Wi-Fi module to support wireless data transmission, thereby achieving efficient communication with the computing center. A user-friendly startup control module can also be included, which is integrated into the embedded system of the robotic arm and provides an intuitive user interface. The user can input the information of the person being collected and the required number of collected samples through this interface to start or terminate the tongue image collection process. At the same time, the module can also display the camera screen in real time, allowing the user to intuitively monitor the collection process. The user-friendly startup control module includes: a user interface that allows the user to set collection parameters, such as the number of collected samples, the information of the person being collected, view the real-time camera screen, etc., and start or stop the collection process; a feedback mechanism that provides the user with real-time feedback on the current collection status, including but not limited to the number of samples that have been collected and the current position status of the robotic arm; and an analysis result that provides the user with a simple analysis result of the current tongue image.

[0110] In an embodiment of the present invention, the robotic arm uses the flask package in Python for rapid deployment. On the one hand, a basic user interactive web page front end is built. The front end and the Python back end use the HTTP protocol for data transmission, and a real-time display of camera content is set up on the back end to detect whether the content input by the user meets the requirements. Finally, an area is reserved for displaying the collected photos, so that users can check whether the collection is normal. On the other hand, in order to facilitate the overall data transmission and control, this example encapsulates the objects related to the robotic arm control and the camera control into an object called mar_client. This object also encapsulates functions related to face positioning, tongue positioning, tongue image collection and reception analysis. The overall logic is to coordinate with the computing center for three face positioning according to the input tongue image collection number, and then coordinate with the computing center for three tongue image positioning, and finally perform tongue image collection according to the input number and send the picture to the computing center. In addition, the partial deployment of the robotic arm is to transmit the current camera image and the information of the person being collected to the computing center, receive the information returned by the computing center and display it in the result column of the user interface, and then display the collected photos at the bottom of the user interface for user observation. Finally, the python script of the robot arm flask application is set to start when the onboard system of the robot arm is started (in this example, the onboard system is a Raspberry Pi with Ubuntu-20.04 installed), and the IP address of the robot arm is set to a static address.

[0111] In an embodiment of the present invention, the data transmission uses Wi-Fi or wired connection to ensure stable communication between the small robotic arm and the computing center. Users can connect to the robotic arm through the network or wired means to control it, and technicians can perform remote or on-site maintenance on the robotic arm through the network or offline means.

[0112] Furthermore, the robotic arm and the computing center are connected via a remote or local area network.

[0113] As one of the implementation methods, the robotic arm and the computing center use a method of connecting to the same wireless network, that is, the devices are in the same local area network.

[0114] After the tongue diagnosis auxiliary system of the embodiment of the present invention is deployed, the execution process is as follows:

[0115] 1. Turn on the power of the robot arm and wait for the robot arm to start; at this time, the robot arm will automatically initialize the position function, first return to the mechanical origin of the robot arm, and then move to the program origin set by the program; then the robot arm enters the waiting instruction state and will not connect to the computing center to reduce the load of the computing center;

[0116] 2. Connect to the same local area network as the robot arm through devices such as personal computers, tablets and mobile phones; according to the static IP address set by the robot arm, use port 5000 (i.e. xxx.xxx.xxx.xxx:5000) to enter the user interface; the real-time image of the camera can be seen in the user interface;

[0117] 3. Enter the information of the person being collected and the number of tongue images to be collected in the user interface, and then click the Start button to start tongue image collection.

[0118] 4. The flask application on the robotic arm receives the request sent by the front end, and the program starts to obtain the current image of the camera and convert the current image into a byte stream in JPEG format; then, the program detects whether the face counter is less than 3. If it is less than 3, the current image is sent to the " / yoloface" route of the computing center using the POST method.

[0119] 5. After the computing center receives the current request through the " / yoloface" route, it will first detect whether the request contains image information. If not, it will return an error message to the robot arm. If the request contains an image, the computing center will then check whether the uploaded file is of an allowed file type (such as png, jpg, jpeg, bmp) to ensure that a valid and safe file type is processed. Once the file type is verified to be correct, the computing center will create a new cache folder and temporarily save the image to the server. After successful saving, the computing center uses the YOLO model to perform facial recognition processing on the image. Here, a pre-trained YOLO model is used, and then the face in the image is identified and the coordinate information of the face is obtained. Finally, the computing center returns the processing results in JSON format, including the coordinates of the center point of the face and the coordinates of the face frame (if a face is detected). If no face is detected, special coordinates indicating that no face is detected will be returned.

[0120] 6. After receiving the result returned by the computing center in JSON format, the robotic arm extracts the center coordinates of the face. If no face is detected, it returns to the third step. If a face is detected, the robotic arm control function (called "successful positioning") is called to detect whether the face is too far from the center of the screen. If the face is near the center of the screen (this example is set to 30 pixels, which is also the absolute value of 30 on the y coordinate of the robotic arm), the robotic arm does not move. If the face deviates too much from the center of the screen (more than 30 pixels), the difference between the center coordinates of the face and the coordinates of the center of the current screen is used as the moving distance for movement in the corresponding direction.

[0121] 7. After successful positioning in step 6, the program will repeat steps 4 to 6 twice and then proceed to step 8.

[0122] 8. Next, the program on the robotic arm will obtain the current image from the camera and convert it into a byte stream in JPEG format; then, the program will detect whether the final result of the sixth step is that no face is detected. If not, it will return to the third step; if a face is detected, it will start to detect whether the face count is 3 and whether the tongue detection is less than 3.

[0123] 9. If the last condition in step 8 is not met, go back to step 3; if the condition is met, send the current screen to the " / unetmaskcenter" route of the computing center using the POST method.

[0124] 10. After the computing center " / unetmaskcenter" route receives the request, it will first perform the same inspection steps as step 5; after the inspection passes and is saved, the U-net model is called to identify the tongue image; finally, the computing center returns the processing results in JSON format, including the center and contour coordinates of the tongue image. If the tongue image is not detected, special coordinates indicating that the tongue image is not detected will be returned.

[0125] 11. After receiving the result returned by the computing center in JSON format, the robotic arm extracts the center coordinates of the tongue. If no tongue is detected, it returns to step 3. If a tongue is detected, the robotic arm control function is called to detect whether the tongue is too far from the center of the screen. If the tongue is near the center of the screen (this example is set to 30 pixels, which is also the absolute value of 30 on the y coordinate of the robotic arm), the robotic arm does not move. If the tongue deviates too much from the center of the screen (more than 30 pixels), the difference between the center coordinates of the tongue and the coordinates of the center of the current screen is used as the moving distance for movement in the corresponding direction.

[0126] 12. After successful positioning in step 11, the program will repeat steps 8 to 11 twice, and then proceed to step 13.

[0127] 13. Next, the program on the robotic arm enters the tongue image collection state. The robotic arm no longer moves, but executes steps similar to face positioning and tongue image positioning. The difference is that the request is sent to the " / martcpredict" route, and the information of the person being collected is also sent together.

[0128] 14. After the computing center " / martcpredict" route receives the collected person's information and collected pictures, it will first look for a folder that matches the collected person's information in the path specifically used to save the collected pictures in the computing center. If not, a new folder will be created based on the collected person's information; if so, no new folder will be created; then the program will create a new folder named with the current date under the folder named after the collected person's information to save the tongue image of the collected person.

[0129] 15. Steps 13 to 14 are repeated a specified number of times according to the number of collections entered in the user interface, and then the computing center " / martcpredict" route will call the pre-trained swin-transformer model to classify the tongue image of the person being collected, and then call the openai api with the result label (crack or no crack) with a set prompt (similar to a fixed sentence) to use gpt-4-turbo to generate possible causes and life suggestions, and then return the results to the robotic arm and save them in the same folder as the current tongue image photo.

[0130] 16. After receiving the results, the robotic arm displays the results on the user interface, and then the main acquisition program pauses and waits for the next acquisition; at this time, the robotic arm enters the state of step 3, waiting for the user to start the acquisition process.

[0131] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0132] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0133] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A tongue diagnosis auxiliary method, characterized in that: include: Obtain information about the person being collected and collection requirements; The face positioning information of the subject is obtained through the camera on the robotic arm, and the robotic arm can move and rotate in three directions of X, Y, and Z in space; Adjust the robotic arm to a first precise position according to the face positioning information so as to center the face of the subject; Based on the first precise position, obtaining the tongue image positioning information of the subject through the camera; adjusting the robotic arm to a second precise position according to the tongue image positioning information; According to the acquisition requirement, based on the second precise position, the tongue image of the subject is acquired through the camera; Uploading the collected person information and the tongue image; Analyzing the tongue image based on a classification model to obtain classification information of the tongue image; Uploading the collected person information and the tongue image according to the classification information; When the robot arm moves abnormally, a restart prompt is issued; When the camera cannot recognize the face, a check prompt is issued, and after the transmission picture of the test gesture is obtained, the face positioning information is obtained; When the tongue image is unclear, a troubleshooting step prompt is issued and the tongue image is re-collected.

2. The tongue diagnosis auxiliary method according to claim 1, characterized in that: The classification model includes a crack classification model, and the crack classification model uses Swin Transformer to process image data.

3. The tongue diagnosis auxiliary method according to claim 2, characterized in that: The training steps of the crack classification model include: Apply the swin_base_patch4_window7_224 model of swin-transformer, use the python script to read all the images and labels in the crack folder and the crack-free folder, and split the training and validation data sets in a ratio of 7:

3. Create folders 0 and 1 in the train and val folders to represent the two labels and put the images in them. Under the initial training rate of 0.01, 200 epochs of training were performed. When the loss in the training set reached a loss of 0.2944, the loss in the validation set reached a loss of 0.4855, and the Acc@1 in the validation set was 81.188%, the crack classification model was obtained.

4. The tongue diagnosis auxiliary method according to claim 1, characterized in that: The step of obtaining the face positioning information of the subject through the camera on the robotic arm includes: Based on the face positioning model, the face positioning information is obtained through the camera; the training step of the face positioning model includes: Apply the yolov8n model, and use the open source training set coco2017 and faces4coco pre-training to obtain a pre-trained model; 500 photos and about 2,000 face data were re-labeled, and subsequent training and verification operations were performed using the pre-trained model to obtain the face positioning model.

5. The tongue diagnosis auxiliary method according to claim 4, characterized in that: The step of obtaining the tongue image location information of the subject through the camera based on the first precise position includes: Based on the first precise position determined by the tongue image positioning model, the tongue image positioning information is acquired through the camera; the training step of the tongue image positioning model includes: The U-net model or the yolo model was applied, and a learning rate of 1e-8 was adopted for 100 epochs of training, so that the model loss dropped to 0.2202 at 4344 steps, thereby obtaining the tongue image localization model.

6. A tongue diagnosis auxiliary system, characterized in that: include: The first acquisition module is used to obtain the information of the person being collected and the collection requirements; The second acquisition module is used to acquire the face location information of the subject through a camera on a robotic arm, and the robotic arm can move and rotate in three directions of X, Y, and Z in space; A first adjustment module, used for adjusting the mechanical arm to a first precise position according to the face positioning information, so as to center the face of the subject; A third acquisition module is used to acquire the tongue image location information of the subject through the camera based on the first precise position; A second adjustment module, used for adjusting the robotic arm to a second precise position according to the tongue image positioning information; The acquisition module is used to acquire the tongue image of the subject through the camera based on the second precise position according to the acquisition requirements; the acquisition module is also used to: issue a restart prompt when the robot arm moves abnormally; issue a check prompt when the camera cannot recognize the face, and acquire the face positioning information after acquiring the transmission picture of the test gesture; issue a troubleshooting step prompt and re-acquire the tongue image when the tongue image is unclear; The uploading module is used to upload the collected person information and the tongue image; it is also used to analyze the tongue image based on the classification model to obtain the classification information of the tongue image; it is also used to upload the collected person information and the tongue image according to the classification information.

7. A tongue diagnosis auxiliary system, characterized in that: The invention comprises a mechanical arm and a computing center; the computing center is used for: Obtain information about the person being collected and collection requirements; Acquire a face image captured by the robotic arm, analyze to obtain face positioning information, and send the face positioning information to the robotic arm; Acquire the tongue image collected by the robotic arm, analyze and obtain tongue image positioning information, and send the tongue image positioning information to the robotic arm; After receiving the tongue image collected by the robot arm, uploading the information of the person being collected and the tongue image; The mechanical arm can move and rotate in three directions of X, Y and Z in space, and the mechanical arm includes a camera, and the mechanical arm is used for: Obtaining the information of the person being collected and the collection requirements; Collecting a face image based on a camera and sending it to the computing center; Obtaining the face positioning information sent by the computing center; Adjust the robotic arm to a first precise position according to the face positioning information so as to center the face of the subject; Based on the first precise position, a tongue image is collected by the camera and sent to the computing center; Acquire the tongue image positioning information sent by the computing center; adjusting the robotic arm to a second precise position according to the tongue image positioning information; According to the acquisition requirement, based on the second precise position, the tongue image of the subject is acquired through the camera; Uploading the tongue image to the computing center; Analyzing the tongue image based on a classification model to obtain classification information of the tongue image; Uploading the collected person information and the tongue image according to the classification information; When the robot arm moves abnormally, a restart prompt is issued; When the camera cannot recognize the face, a check prompt is issued, and after the transmission picture of the test gesture is obtained, the face positioning information is obtained; When the tongue image is unclear, a troubleshooting step prompt is issued and the tongue image is re-collected.

8. The tongue diagnosis auxiliary system according to claim 7, characterized in that: The robotic arm and the computing center are connected via a remote or local area network.

Images