Intelligent tongue image data collection system and method

Abstract

The present application discloses an intelligent tongue image data acquisition system and method, which includes: an image sensor is provided in the intelligent terminal, which is used to obtain the user's face image and send it to the server; the server verifies the current user according to the face image and the user's basic information, and sends a tongue image acquisition instruction to the intelligent terminal to determine the tongue position in the face image, calculate the histogram features according to the pixel information corresponding to the tongue position, generate the user feature vector, and determine the color correction matrix according to the Euclidean vector distance between the feature vector and the sample feature vector; the intelligent terminal collects the user's tongue image according to the tongue image acquisition instruction, and sends the tongue image to the server; the server performs color correction on the tongue image according to the color correction matrix, and stores the corrected tongue image. Through the technical solution in the present application, the problem that the existing tongue diagnosis instrument is expensive and not convenient and intelligent enough is solved.

Description

Technical Field

[0001] The present application relates to the technical field of tongue diagnosis instruments, and more specifically, to an intelligent tongue image data acquisition system and an intelligent tongue image data acquisition method. Background Art

[0002] With the continuous development of modern urban life, people's work and life rhythms are gradually accelerating, and stress and the health problems it causes are also coming along. At the same time, society is also facing the problem of health management of empty-nest elderly people at home due to the aging population. Therefore, it is particularly important to track health data, manage and predict diseases in daily life. The "observation" diagnostic method in traditional Chinese medicine can well achieve daily family health management.

[0003] However, the existing home health management related equipment is expensive and is still in the laboratory research stage, and has not been put into practical use. For example, Professor Xu Jiatuo of Shanghai University of Traditional Chinese Medicine conducted an experimental study on the acquisition conditions of TCM visual diagnosis images, and concluded that the images involved in visual diagnosis can be improved by adjusting the light source intervention, but the TDA-1 tongue diagnosis instrument independently developed by him is mostly used in university research and has not yet been put on the market.

[0004] Another example: The DS01-B tongue diagnostic instrument developed and produced by Shanghai Daosheng Medical Technology Co., Ltd. uses a 1500M pixel image sensor and OSRAM full-spectrum L18 / 72965BIOLUX standard light source, and can use deep learning methods to analyze more than 50 tongue facial features. However, this tongue diagnostic instrument is not only expensive, but also has strict requirements on the lighting conditions in the external environment, and is not suitable for popularization by the whole people. Summary of the invention

[0005] The purpose of this application is to solve the problems of existing tongue diagnosis instruments being expensive, inconvenient and not intelligent enough, and to use artificial intelligence and cloud computing technologies to achieve convenient personal data division and tongue image detection based on deep learning, and to establish a low-cost tongue image information database suitable for daily family health management.

[0006] The technical solution of the first aspect of the present application is: an intelligent tongue image data acquisition system is provided, the system includes: an intelligent terminal and a server; an image sensor is provided in the intelligent terminal, the intelligent terminal is used to obtain the user's face image, and send the face image to the server; the server is used to verify the current user according to the face image and the stored user basic information, and when the verification is passed, a tongue image acquisition instruction is sent to the intelligent terminal, and the tongue position of the face image is determined, and the histogram features are calculated according to the pixel information corresponding to the tongue position, and the histogram features are reduced in dimension to generate a user feature vector, and a color correction matrix is ​​determined according to the Euclidean vector distance between the user feature vector and the sample feature vector; the intelligent terminal is also used to collect the user's tongue image according to the tongue image acquisition instruction, and send the tongue image to the server; the server is also used to perform color correction on the tongue image according to the color correction matrix, and store the corrected tongue image.

[0007] In any of the above technical solutions, further, the process of the server generating the user feature vector specifically includes:

[0008] Step B, calculating the brightness information of the tongue image according to the pixel information in the tongue image;

[0009] Step C, calculating the channel tensor of the channel component on the chromaticity measurement according to the channel component of the tongue image on the RGB channel;

[0010] Step D, calculating the histogram features of the tongue image according to the brightness information and the channel tensor;

[0011] Step E: using a linear conversion function to extract the user feature vector of the tongue image from the histogram feature, the corresponding formula is:

[0012] v(h(I) (u，v，C) )=ω T (vec(h(I) (u，v，C) )-β)

[0013] Where vec(·) is the linear conversion function, v(h(I) (u，v，C) ) is the user feature vector, ω is the principal component coefficient matrix, β is the average histogram vector, h(I) (u，v，C) is the histogram feature.

[0014] In any of the above technical solutions, further, a color correction matrix is ​​determined based on the Euclidean vector distance between the user feature vector and the sample feature vector, specifically including: when it is determined that all Euclidean vector distances are greater than a distance threshold, a preset correction matrix is ​​used as the color correction matrix.

[0015] Any of the above technical solutions may further include:

[0016] When there is a calculated Euclidean vector distance less than the distance threshold, select the sample feature vector whose Euclidean vector distance is less than the distance threshold The corresponding color conversion matrix M (i) , forming a color correction matrix set And the color correction matrix M2 is calculated by the weighted linear combination method, and the corresponding calculation formula is:

[0017]

[0018]

[0019] Among them, α j is the matrix weight, j∈[1, 2, ..., k], σ is the radial attenuation factor, and d is the vector of Euclidean distances.

[0020] In any of the above technical solutions, the device further includes: an ambient light intensity sensor and a fill light; the ambient light intensity sensor is used to collect the ambient light illumination and send the illumination to the smart terminal; the smart terminal is used to convert the received light illumination into a numerical value and compare the numerical value with a preset threshold value, and when it is determined that the numerical value is less than or equal to the preset threshold value, a fill light instruction is sent to the fill light; the fill light is used to switch from an off state to an on state according to the fill light instruction.

[0021] In any of the above technical solutions, further, the current user is verified based on the facial image and the stored user basic information, specifically including: verifying whether the facial image is a human face, when the verification is passed, extracting the first face in the facial image, and mapping the first face to the Euclidean space, and recording the mapping result as a first vector; determining the unique device number id of the smart terminal corresponding to the facial image, searching the stored user basic information according to the unique device number id, mapping the second face in the found user basic information to the Euclidean space, and recording the mapping result as a second vector; calculating the Euclidean distance between the first vector and the second vector, and when it is determined that the Euclidean distance is less than the verification threshold, it is determined that the current user has passed the verification.

[0022] In any of the above technical solutions, further, before uploading the facial image to the server, the smart terminal sets the naming format of the facial image to a combination of the timestamp serial number and the unique device number id of the smart terminal, and the user basic information is bound one by one to the unique device number id.

[0023] In any of the above technical solutions, further, the basic information of the user includes at least the user's name, the user's height, the user's weight, the user's age, and the user's bare-headed facial image.

[0024] In any of the above technical solutions, further, the sample feature vector is determined by a color error image in a training data set.

[0025] The technical solution of the second aspect of the present application is: to provide an intelligent tongue image data collection method, the method comprising: step 10, obtaining a face image of a user, and verifying the current user according to the face image and stored basic information of the user;

[0026] Step 20, when the verification is passed, determine the tongue position of the face image, and calculate the histogram features according to the pixel information corresponding to the tongue position, and perform dimensionality reduction processing on the histogram features to generate a user feature vector;

[0027] Step 30, determining a color correction matrix according to the Euclidean vector distance between the user feature vector and the sample feature vector;

[0028] Step 40: perform color correction on the collected tongue image of the user according to the color correction matrix, and store the corrected tongue image.

[0029] The beneficial effects of this application are:

[0030] In the technical solution of this application, we have used the existing low-cost development board, cloud computing and artificial intelligence technology to realize the convenient and fast collection and correction equipment of tongue diagnosis data and the back-end collection based on face recognition and color correction technology. Through a large number of experiments on the influence of light sources on tongue color, we believe that the color temperature of the light source of 6500K has the best effect on restoring tongue color, so we use a 6500K color temperature light source with adjustable brightness to weaken the influence of ambient light, and obtain good results.

[0031] During the current user's identity authentication process, we use a face recognition algorithm based on which the user can enter the collection step without noticing. The server automatically obtains the current user's matching information based on the unique device number id of the smart terminal to authenticate the user, which shortens the time of identity authentication and improves the accuracy of verification. It helps to achieve convenient and fast tongue image collection, and stores and archives the user information and the tongue image collected by the user in combination with the unique device number id of the smart terminal. Compared with tongue diagnostic instruments on the market, it has excellent user data classification and user data tracking features.

[0032] Secondly, because the open acquisition method of supplementary lighting cannot completely avoid the color authenticity deviation caused by different environmental conditions. Therefore, we designed a color correction step. Through the training data set collected in the early stage, the color error image in the training set is mapped with the collected tongue image to obtain the feature vector, so as to search for the color conversion matrix corresponding to the color error image, and then obtain the color correction matrix to correct the tongue image data with color deviation. Through experimental verification, it is proved that the color correction process can achieve the ideal effect and ensure the accuracy of tongue image acquisition.

[0033] In particular, in the color correction part, we use the ratio of the average chromaticity D and the chromaticity center distance M in the CIE Lab color space as the color cast factor K to measure the degree of color cast of the image. Generally, we believe that images with a ratio less than 2 usually do not have color cast, and the more severe the color cast, the greater the color cast factor. We collected 20 tongue images in different environments and calculated their color cast factors to evaluate our final results. The average color cast factor K was 1.8504, which achieved the purpose of color cast correction and fully met our project requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The advantages of the above and / or additional aspects of the present application will become apparent and easily understood in the description of the embodiments in conjunction with the following drawings, in which:

[0035] Figure 1 is a schematic block diagram of an intelligent tongue image data acquisition system according to an embodiment of the present application;

[0036] Figure 2 is a schematic diagram of a face detection model according to an embodiment of the present application;

[0037] Figure 3 is a schematic diagram of a detected face image and an extracted tongue image according to an embodiment of the present application;

[0038] Figure 4 It is the data statistical information collected according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0040] In the following description, many specific details are elaborated to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited to the specific embodiments disclosed below.

[0041] Embodiment 1:

[0042] like Figure 1 As shown, this embodiment provides an intelligent tongue image data acquisition system, which includes: an intelligent terminal and a server; a processor and an image sensor are provided in the intelligent terminal, and the intelligent terminal is used to obtain a user's facial image and send the facial image to the server; the server is used to verify the current user based on the facial image and the stored basic information of the user, and when the verification is passed, a tongue image acquisition instruction is sent to the intelligent terminal, and the tongue position of the facial image is determined, and the histogram features are calculated based on the pixel information corresponding to the tongue position, and the histogram features are reduced in dimension to generate a user feature vector, and a color correction matrix is ​​determined based on the Euclidean vector distance between the user feature vector and the sample feature vector; the intelligent terminal is also used to collect the user's tongue image according to the tongue image acquisition instruction, and send the tongue image to the server; the server is also used to perform color correction on the tongue image according to the color correction matrix, and store the corrected tongue image.

[0043] In this embodiment, the smart terminal is implemented by Raspberry (microcomputer), and each smart terminal corresponds to a unique device number id. The basic information of the user can be collected through the smart terminal, such as the user's name, user height, user weight, user age, and the user's bare-headed facial image. When uploading the basic information of the user, the smart terminal binds the basic information of the user with the unique device number id of the smart terminal, so that based on this binding, the server can extract all users of the smart device by querying the unique device number id of the smart device for face recognition and find out the user currently using the smart device.

[0044] For example, during the collection process, a form can be embedded in the smart terminal to submit the user's basic information data to the Django server. During the submission process, the data in the form is converted into json format and written into the database.

[0045] Furthermore, the device also includes: an ambient light intensity sensor and a fill light; the ambient light intensity sensor is used to collect the ambient light illumination and send the illumination to the smart terminal; the smart terminal is used to convert the received light illumination into a numerical value and compare the numerical value with a preset threshold value, and when it is determined that the numerical value is less than or equal to the preset threshold value, a fill light instruction is sent to the fill light; the fill light is used to switch from an off state to an on state according to the fill light instruction.

[0046] The light source device in this embodiment includes a MAX44009 ambient light intensity sensor and a fill light, wherein the fill light is an aperture with a diameter of 26 cm and a color temperature of 6500K.

[0047] Before each data collection, the MAX44009 ambient light intensity sensor is used to obtain the current ambient light intensity, which is connected and transmitted to the smart terminal through the CSI interface. The smart terminal converts the received light intensity into a value in lux, and compares the value with a preset threshold value, which can be 2000 lux. If the value is less than or equal to 2000 lux, the smart terminal sends a fill light instruction to the fill light, and the fill light performs fill light, thereby ensuring that the image quality is as close to natural light as possible.

[0048] The smart terminal is provided with a processor and an image sensor, and is used to obtain the user's face image. The naming format of the face image is "timestamp serial number + unique device number id", and the server uses this to obtain the unique device number id. During the face recognition process, the server will load the user information of the registered user under the current unique device number id, perform face matching, and find out the user currently using the smart device;

[0049] The image sensor in this embodiment is an IMX219 800W pixel image sensor, which is electrically connected to the smart terminal through the Picamera module to collect the user's face data, and the maximum resolution of the image is set to 3280 × 2464. After receiving the image collected by the image sensor, the smart terminal compresses the image using the jpeg format and then uploads it to a server based on the Django Web framework.

[0050] Preferably, before uploading the facial image to the server, the smart terminal sets the naming format of the facial image to a combination of the timestamp serial number and the unique device number id of the smart terminal, and the user's basic information is bound one by one to the unique device number id, with the timestamp serial number as the primary key.

[0051] After the server obtains the face image uploaded by the smart terminal, it modifies the size of the face image to 224×224×3, and uses a deep learning-based method to detect the face from the face image. First, VGG16 is used to extract the features of the original face image to obtain a feature map, which is then input into the Region Proposal Network (RPN network) to generate the corresponding recommended region with the face. After the feature map and the recommended region are comprehensively extracted, the recommended feature map is finally determined.

[0052] This embodiment shows a process of determining / verifying whether a face image is a human face, which specifically includes:

[0053] Step 1: Based on the VGG16 model and the set convolution layer parameters, a convolution layer is constructed, and the initial feature maps in the face image uploaded by the smart terminal are extracted. The convolution layer parameters include 13 convolution layers, 3 convolution kernels, 1 step size, ReLU activation function between layers, and the pixel padding of the convolution layer is 1, that is, a layer of 0 is filled around the edge of the image, so that the convolution layer does not change the size of the image.

[0054] In this embodiment, the calculation formula corresponding to the ReLU activation function is:

[0055]

[0056] Where x is the input value in the neural network propagation process. Preferably, the convolution layer parameters also include a maximum pooling layer kernel of 2 and a step size of 2. By setting the kernel and step size of the maximum pooling layer, the length and width of each image passing through the pooling layer will become 1 / 2 of the original. After a total of 4 maximum poolings, if the input image is N×M, it will be converted to a fixed size of N / 16×M / 16 after feature extraction by the above convolution layer.

[0057] Step 2: Based on the proposed candidate region network, traverse the initial feature maps and select the optimal anchor to determine the candidate region (G′) of the face image. x , G′ y , G′ w , G′ h ).

[0058] Specifically, the Region Proposal Network (RPN) is used as the network for recommending candidate regions. The initial feature maps output by the convolutional layer are traversed in a traversal manner to generate 9 anchors for each pixel. Each anchor is represented by the coordinates (x1, y1, x2, y2), which are the coordinates of the upper left corner and the lower right corner of a rectangle.

[0059] In the process of determining the candidate region, the initial feature maps output by the convolution layer are first convolved with a 1×1 convolution, and the number of convolution kernels is set to 18 to correspond to the positive and negative samples of the 9 anchors. Through a binary classification task of the softmax function, the negative samples are removed, leaving only the anchors of the positive samples. After obtaining the anchors of the positive samples, the window is fine-tuned. The window generally uses a four-dimensional vector (x, y, w, h) to represent the coordinates of the center point of the window as well as the width and height.

[0060] In the process of determining the optimal anchor, assume that the given anchor A and the ground truth GT are:

[0061] A=(A x , A y , A w , A h )

[0062] GT=[G x , G y , G w , G h ]

[0063] In the formula, (A x , A y , A w , A h ) and [G x , G y , G w , G h ] are the coordinates of the upper left corner and lower right corner of the given anchor A and the ground truth GT respectively.

[0064] The following mapping function F is used to map the given anchor A to determine the candidate region (G′ x , G′ y , G′ w , G′ h ), the corresponding formula is:

[0065] (G′ x , G′ y , G′ h , G′ h )=F(A x , A y , A w , A h )

[0066] st(G′ x , G′ y , G′ w , G′ h )≈(G x , G y , G w , G h )

[0067] Among them, the transformation corresponding to the mapping function F is as follows:

[0068] G′ x =A w ·d x (A)+A x

[0069] G′ y =A h ·d y (A)+A y

[0070] G′ w =A w ·exp(d w (A)

[0071] G′ h =A h ·exp(d h (A)

[0072] Among them, d x (A), d y (A), d w (A), d h (A) is the transformation function obtained by linear regression. x (A), d y (A), d w (A), d h (A) Fine-tune the given anchor A, sort the positive samples from small to large according to the softmax value, extract the first 6000 given anchors A, then remove the given anchors A that exceed the image boundary, and then perform non-maximum suppression on the remaining given anchors A to select the optimal anchor and determine the candidate region of the face image.

[0073] Step 3: Based on the ROI pooling layer, according to the initial feature maps and the candidate region (G′ x , G′ y , G′ w , G′ h ), determine a facial feature map, and calculate the probability that the facial feature map is a human face, so as to verify whether the facial image is a human face.

[0074] Specifically, based on the ROI pooling layer, the initial feature maps output by the convolution layer are converted to a fixed size, that is, a size transformation is performed. The optimal anchor corresponding to the candidate region is mapped back to the scale of the feature map corresponding to (N / 16)×(M / 16), and then the region of the feature map corresponding to each optimal anchor is horizontally divided into w×h grids, and each grid is processed by the max pooling layer, and finally a 7×7 feature map is obtained, which is used as the face feature map.

[0075] Afterwards, the face feature map (δ1 δ2 δ3) is calculated through the fully connected network and the softmax function to obtain the probability of whether the face image uploaded by the smart terminal is a face. The corresponding calculation formula is:

[0076]

[0077]

[0078] B=(b1 b2)

[0079] In the formula, matrix W is the weight matrix, and matrix B is the bias matrix. They are parameters obtained through training and are used to discriminate and classify facial feature maps. The obtained y1 is the probability of being a face, and y2 is the probability of not being a face. The sum of the two is 1.

[0080] In this embodiment, when it is determined that the facial image of the current user collected by the smart terminal through the image sensor is a human face, the server maps the image to Euclidean space through convolutional neural network learning, and determines whether it is the same person by calculating the Euclidean distance between the face in the database and the face of the current user.

[0081] The server verifies whether the face image is a human face through the above process. When the verification is passed, the server extracts the first face in the face image and maps the first face to the Euclidean space, and records the mapping result as the first vector.

[0082] Determine the unique device number ID of the smart terminal corresponding to the face image, search the stored user basic information according to the unique device number ID, map the second face in the found user basic information to the Euclidean space, and record the mapping result as the second vector;

[0083] The Euclidean distance between the first vector and the second vector is calculated, and when it is determined that the Euclidean distance is less than the verification threshold, it is determined that the current user has passed the verification.

[0084] like Figure 2 As shown, the face verification in step 3 is first performed according to the candidate area of ​​the face image calculated in step 2. After the verification is passed, the corresponding face area is extracted from the uploaded face image according to the determined candidate area of ​​the face image, and then the extracted face is mapped to the Euclidean space through the trained ZF network in deep learning to obtain a 128-dimensional first vector.

[0085] Similarly, the server first analyzes the name of the facial image to determine the unique device number id of the corresponding smart terminal. Then, it searches its database according to the unique device number id to obtain all the basic information of users under the corresponding device corresponding to the unique device number id, extracts the user's bare-headed facial image from all the basic information of the users found, and calculates to obtain a 128-dimensional second vector.

[0086] Finally, the Euclidean distances between the first vector and the multiple second vectors are calculated respectively, and the calculated Euclidean distances are compared with the set threshold, where the set threshold is 0.4. If the Euclidean distance is less than 0.4, it is determined that the current user and the user corresponding to the Euclidean distance and the user basic information found are the same person, and the verification of the current user is completed.

[0087] Specifically, in the face verification process of the current user, the convolution layer is set to use 12 convolution operations, 4 maximum pooling operations, and two batch normalization operations, among which 5 convolution operations are 1×1 convolution kernels, and the remaining 6 convolution operations are 3×3 convolution kernels, and the final output is a 1×1×128 vector.

[0088] This completes the mapping of face images to Euclidean space, in which the spatial distance between images is directly related to image similarity: different images of the same person have a small spatial distance, but images of different people have a large distance in space. Based on this theory, we can easily calculate and match faces and verify the user's identity information based on the current user's face image.

[0089] Set the current user's face image X (first vector) to: X = [x0, ..., x 127 ], the face image Y (second vector) in the user basic information stored in the database is: Y = [y0, ..., y 127 ], then the corresponding Euclidean distance is:

[0090]

[0091] When the verification is passed, the server sends a tongue image acquisition instruction to the smart terminal, and the smart terminal acquires the user's tongue image through the image sensor and sends the tongue image to the server.

[0092] The server first determines the tongue position of the facial image, and calculates the histogram features based on the pixel information corresponding to the tongue position, and performs dimensionality reduction processing on the histogram features to generate a user feature vector, and determines the color correction matrix based on the Euclidean distance between the user feature vector and the sample feature vector in the training data set, and then performs color correction on the tongue image according to the color correction matrix, and stores the corrected tongue image.

[0093] Specifically, Figure 3As shown, when the face detected by the smart terminal passes the verification, that is, the current user has registered (logged in) on the current device (smart terminal), the backend (server) returns a message of verification success, and the smart terminal collects tongue images of the current user, including images of the tongue surface and tongue bottom.

[0094] After the server verifies the face of the current user, it sends a tongue image acquisition instruction to the smart terminal. After receiving the instruction, the smart terminal sends a voice prompt to the current user and starts to collect tongue extension (tongue surface) and tongue curling (tongue bottom) action images. The resolution of the collected images is also 3280×2464, and they are compressed using the jpeg format. They are then uploaded to the server based on the Django Web framework, and the image file is named with the timestamp of the current acquisition process, that is, the id of the acquisition process and the image sequence number as the suffix, and the image location is written into the database.

[0095] In this embodiment, the server also calculates the tongue position based on the face area in the face image according to the relative position experience, and the corresponding relationship is:

[0096] t=(top-(top-bottom)×2 / 3)

[0097] b=bottom×1.1

[0098] l = (left + (right - left) × 1 / 4)

[0099] r = (right - (right - left) × 1 / 4)

[0100] In the formula, the face area (top, bottom, left, right) represents the relative position of the face in the face image, (top, left) represents the coordinates of the upper left vertex, and (bottom, right) represents the coordinates of the lower right vertex. The tongue position (t, b, l, r) can be calculated through the above calculation.

[0101] According to the calculated tongue position, the currently collected tongue image is extracted and cropped to determine the pixel information corresponding to the tongue position, and a color correction matrix is ​​selected to perform color correction on the tongue image. The process specifically includes:

[0102] Step A, based on color error images in the training dataset and color correct images Calculate the color conversion matrix M (i) , the corresponding calculation formula is:

[0103]

[0104] In the formula, argmin||·|| F is the distance minimum function. Through the above calculation, the color error image can be Color Correct Image And the color conversion matrix M (i) One-to-one correspondence is performed, where the color conversion matrix M (i) Color error image Color Correct Image The mapping relationship between them, i is the number of the image in the training data set, and the color error image For color correct images The corresponding image.

[0105] It should be noted that the color conversion matrix M calculated above (i) It will be stored in the database of the server so as to subsequently search for the color correction matrix corresponding to the tongue image currently collected during the color correction process.

[0106] Step B, calculating the brightness information of the tongue image according to the pixel information in the tongue image, the corresponding calculation formula is:

[0107]

[0108] Where I is the tensor value of the image, R, G, B represent the color channels in the tensor value I, where i = {1, ..., N, I y(i) is the brightness information, I R(i) ,I G(i) ,I B(i) It is the component of the image on the R, G, and B channels.

[0109] Step C, define two chromaticity metrics, denoted as u and v, respectively, according to the channel component I of the tongue image on the RGB channel R(i) ,I G(i) ,I B(i) , calculate the channel tensor I of the channel component on the two chromaticity metrics u and v u1(i) ,I u2(i) ,I u3(i) ,I v1(i) ,I v2(i) ,I v3(i) .

[0110] I u1(i) =log(I R(i) )-log(I G(i) )

[0111] I v1(i) =log(I R(i) )-log(I B(i) )

[0112] I u2(i) =-I u1(i) , I v2(i) =-I u1(i) +I v1(i)

[0113] I u3(i) =-I v1(i) , I v3(i) =-I v1(i) +I u1(i)

[0114] It should be noted that the definition process of the two chromaticity metrics u and v is not limited, and a conventional definition method can be used.

[0115] Step D, based on the calculated brightness information I of the tongue image y(i) The channel tensors of the channel components on the two chromaticity metrics are used to calculate the histogram features of the tongue image. The corresponding calculation formula is:

[0116]

[0117]

[0118] In the formula, C∈{1, 2, 3} represents each color channel in the histogram, ε is the width of the histogram, and finally h(I) is obtained (u，v，C) is the uv histogram information, ∑∑H(I) (u，v，C) To measure the intermediate parameter H(I) on the two chromaticity metrics (u，v，C) The sum of the accumulated values.

[0119] Step E: In order to perform efficient retrieval, a linear conversion function is used to extract a user feature vector with a lower dimension of the tongue image from the histogram feature. The corresponding formula is:

[0120] v(h(I) (u，v，C) )=ω T (vec(h(I) (u，v，C) )-β)

[0121] Where vec(·) is the linear conversion function,

[0122] is the user feature vector of principal component analysis, is the principal component coefficient matrix and is the average histogram vector, h(I) (u，v，C) is the histogram feature.

[0123] Similarly, calculate the color error images in the training dataset The histogram information is extracted using a linear transformation function to obtain the color error image. The low-dimensional sample feature vector

[0124] Step F, calculate the user feature vector v(h(I)) after dimension reduction of each tongue image collected (u，v，C) ) and the sample feature vector The Euclidean vector distance between them is calculated, and the color correction matrix of the tongue image is determined according to the relationship between the calculated Euclidean vector distance and the threshold value. The tongue image is color corrected according to the color correction matrix.

[0125] When the calculated Euclidean vector distances are all greater than the given distance threshold, that is, there is no corresponding image in the training data set collected earlier, a preset correction matrix is ​​given for such tongue image, and based on the given preset correction matrix, the currently collected tongue image is corrected. The corresponding calculation formula is:

[0126] I=M1*(I) (u，v，C)

[0127]

[0128] Where M1 is the preset correction matrix, and I is the corrected tongue image.

[0129] Through test experiments, this matrix has a relative advantage in restoring the visual effect of tongue image data, and its color restoration performance is better in most scenes;

[0130] When there is a calculated Euclidean vector distance less than the distance threshold, the color conversion matrix M calculated in step A is (i) In the example, select the sample feature vector whose Euclidean vector distance is less than the distance threshold The corresponding color conversion matrix M (i) , forming a color correction matrix set And the color correction matrix M2 is calculated by the weighted linear combination method, and the corresponding calculation formula is:

[0131]

[0132]

[0133] Among them, α jis the matrix weight, j∈[1, 2, ..., k], σ is the radial attenuation factor, d is the Euclidean distance vector, which represents a given input feature, that is, the Euclidean distance vector between the color histogram features obtained by reducing the dimension of the image used for correction through steps B and C and the color histogram features of the image in the kth training data set after dimensionality reduction, and j and k′ are both intermediate labels.

[0134] Finally, the obtained color correction matrix M2 is used to perform color conversion (correction) on the tongue image:

[0135] I=M2*(I) (u，v，C)

[0136] Step G, storing the corrected tongue image, when storing, also uses the timestamp serial number of the current acquisition process and the unique device number id of the intelligent terminal as the naming elements of the tongue image, while retaining the original tongue image, and writing the absolute path of the location of the image into the database.

[0137] Preferably, after collecting the tongue image, the intelligent terminal also uses voice prompts to collect videos of tongue extension and tongue curling movements, uploads them to the back end, converts them from h264 to mp4 format using MP4Box, and writes the storage address into the database.

[0138] like Figure 4 As shown, in this embodiment, a Web page is designed to display the collected data, and a simple statistical analysis is performed on the collected data to calculate the amount of data collected every day within 7 days, the total amount of data in the database, the number of users, and the number of devices.

[0139] Specifically, three data tables can be designed, namely user table, device table, and collection data table. The user table is used to store the user's age, height, weight, and device ID information collected from the WeChat Town program terminal;

[0140] The equipment table stores the production time and equipment ID of the equipment;

[0141] The collected data table is used to store the timestamp of the collected data, the address of the face data, the address of the tongue data, the user to which it belongs, and the device ID used.

[0142] Embodiment 2:

[0143] Based on the above embodiment, this embodiment further provides an intelligent tongue image data collection method, which is applicable to the collection system in the above embodiment. The method includes:

[0144] Step 10, obtaining the user's facial image, and verifying the current user based on the facial image and the stored user basic information;

[0145] Step 20, when the verification is passed, determine the tongue position of the face image, and calculate the histogram features according to the pixel information corresponding to the tongue position, and perform dimensionality reduction processing on the histogram features to generate a user feature vector;

[0146] Step 30, determining a color correction matrix according to the Euclidean vector distance between the user feature vector and the sample feature vector;

[0147] Step 40: perform color correction on the collected tongue image of the user according to the color correction matrix, and store the corrected tongue image.

[0148] The above detailed description of the technical solution of the present application is combined with the accompanying drawings. The present application proposes an intelligent tongue image data acquisition system and method, which includes: an intelligent terminal and a server; an image sensor is provided in the intelligent terminal, and the intelligent terminal is used to obtain the user's face image and send the face image to the server; the server is used to verify the current user according to the face image and the stored basic information of the user, and when the verification is passed, the tongue image acquisition instruction is sent to the intelligent terminal, and the tongue position of the face image is determined, and the histogram features are calculated according to the pixel information corresponding to the tongue position, and the histogram features are reduced in dimension to generate a user feature vector, and the color correction matrix is ​​determined according to the Euclidean vector distance between the user feature vector and the sample feature vector; the intelligent terminal is also used to collect the user's tongue image according to the tongue image acquisition instruction, and send the tongue image to the server; the server is also used to perform color correction on the tongue image according to the color correction matrix, and store the corrected tongue image. Through the technical solution in the present application, the problem that the existing tongue diagnosis instrument is expensive and not convenient and intelligent enough is solved.

[0149] The steps in this application can be adjusted in order, combined, and deleted according to actual needs.

[0150] The units in the device of the present application can be combined, divided and deleted according to actual needs.

[0151] Although the present application is disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and are not intended to limit the application of the present application. The scope of protection of the present application is defined by the appended claims and may include various modifications, alterations and equivalents made to the invention without departing from the scope and spirit of the present application.

Claims

1. An intelligent tongue image data collection system, characterized in that: The system comprises: an intelligent terminal and a server; The smart terminal is provided with an image sensor, and the smart terminal is used to obtain a facial image of a user and send the facial image to the server; The server is used to verify the current user based on the facial image and the stored basic user information, When the verification is passed, a tongue image acquisition instruction is sent to the intelligent terminal, and the tongue position of the face image is determined. According to the pixel information corresponding to the tongue position, the histogram features are calculated, and the histogram features are subjected to dimensionality reduction processing to generate a user feature vector. Determining a color correction matrix according to the Euclidean vector distance between the user feature vector and the sample feature vector; When it is determined that all the Euclidean vector distances are greater than the distance threshold, using a preset correction matrix as the color correction matrix; When there is a calculated Euclidean vector distance less than the distance threshold, the sample feature vector whose Euclidean vector distance is less than the distance threshold is selected. The corresponding color conversion matrix M (i) , forming a color correction matrix set The color correction matrix M2 is calculated by a weighted linear combination method, and the corresponding calculation formula is: Among them, α j is the matrix weight, j∈[1,2,…,k], σ is the radial attenuation factor, d is the vector of Euclidean distance, j, k ′ All are middle numbers; The intelligent terminal is also used to collect the tongue image of the user according to the tongue image collection instruction, and send the tongue image to the server; The server is also used to perform color correction on the tongue image according to the color correction matrix, and store the corrected tongue image.

2. The intelligent tongue image data collection system according to claim 1, characterized in that: The process of the server generating the user feature vector specifically includes: Step B, calculating brightness information of the tongue image according to pixel information in the tongue image; Step C, calculating the channel tensor of the channel component on the chromaticity measurement according to the channel component of the tongue image on the RGB channel; Step D, calculating the histogram features of the tongue image according to the brightness information and the channel tensor; Step E, using a linear conversion function, extracting the user feature vector of the tongue image from the histogram feature, the corresponding formula is: v(h(I) (u,v,C) )=ω T (vec(h(I) (u,v,C) )-b) Where vec(·) is the linear conversion function, v(h(I) (u,v,C) ) is the user feature vector, ω is the principal component coefficient matrix, h(I) (u,v,C) is the histogram feature, is the mean histogram vector.

3. The intelligent tongue image data collection system according to claim 1, characterized in that: The system also includes: an ambient light intensity sensor and a fill light; The ambient light intensity sensor is used to collect ambient light illumination and send the illumination to the smart terminal; The intelligent terminal is used to convert the received light illumination into a numerical value, and compare the numerical value with a preset threshold value, and when it is determined that the numerical value is less than or equal to the preset threshold value, send a fill light instruction to the fill light; The fill light is used to switch from an off state to an on state according to the fill light instruction.

4. The intelligent tongue image data collection system according to claim 1, characterized in that: Verify the current user based on the facial image and the stored basic user information, specifically including: Verifying whether the face image is a human face, and if the verification is successful, extracting a first face from the face image, and mapping the first face to a Euclidean space, recording the mapping result as a first vector; Determine a unique device number ID of the smart terminal corresponding to the face image, search the stored user basic information according to the unique device number ID, map a second face in the found user basic information to the Euclidean space, and record the mapping result as a second vector; The Euclidean distance between the first vector and the second vector is calculated, and when it is determined that the Euclidean distance is less than a verification threshold, it is determined that the current user has passed the verification.

5. The intelligent tongue image data acquisition system according to any one of claims 1 to 4, characterized in that: Before uploading the face image to the server, the smart terminal sets the naming format of the face image to a combination of a timestamp serial number and a unique device ID of the smart terminal. The user basic information is bound one by one with the unique device number id.

6. The intelligent tongue image data collection system according to claim 5, characterized in that: The user's basic information includes at least the user's name, user's height, user's weight, user's age, and the user's bare-headed facial image.

7. The intelligent tongue image data collection system according to claim 1, characterized in that: The sample feature vector is determined by color error images in a training data set.

8. An intelligent tongue image data collection method, characterized in that: The method comprises: Step 10, obtaining a facial image of the user, and verifying the current user based on the facial image and the stored basic information of the user; Step 20, when the verification is passed, determine the tongue position of the face image, and calculate the histogram features according to the pixel information corresponding to the tongue position, and perform dimensionality reduction processing on the histogram features to generate a user feature vector; Step 30, determining a color correction matrix according to the Euclidean vector distance between the user feature vector and the sample feature vector; Step 40: perform color correction on the collected tongue image of the user according to the color correction matrix, and store the corrected tongue image.

Images