Tongue tracking method and device, terminal equipment and storage medium

Abstract

The invention relates to the field of combination of traditional Chinese medicine and artificial intelligence, and discloses a tongue tracking method and device, terminal equipment and a storage medium, and the method comprises the steps: carrying out the tongue tracking of a current frame image through a basic tracker in response to the input current frame image, and outputting a first tracking result; under the condition that the first tongue body tracking result is verified to be unreliable, performing region suggestion generation and target image matching on the current frame image by utilizing a global search module so as to output the region suggestion with the highest matching degree as a target tracking result of the current frame; iteratively updating the tongue tracking model in the basic tracker by using the output tracking result; and continuously performing tongue tracking on the input next frame of image by using the basic tracker until the last frame of image is tracked. The method can realize continuous accurate tracking of the tongue body.

Description

Technical Field

[0001] The present application relates to the field of integration of traditional Chinese medicine and artificial intelligence, and in particular to a tongue tracking method, apparatus, terminal device and storage medium. Background Art

[0002] In Traditional Chinese Medicine (TCM), with the rapid development of artificial intelligence (AI), a growing number of scenarios are incorporating various technologies, such as visual tracking, image recognition, and deep learning, to aid diagnosis. For example, visual tracking often requires observation of the tongue when conducting multi-feature studies of tongue imagery. However, due to factors such as body position, camera imaging, and tongue condition, anomalies such as missing labels and frequent sudden changes in the target's appearance can often occur during target tracking. Missing labels typically occur when the target is obscured or out of view, interrupting its trajectory. Frequent sudden changes in target appearance refer to changes in the target's appearance, such as changes in lighting, viewing angle, or deformation. Current target tracking algorithms cannot effectively meet the requirements for continuous tracking of targets like the tongue. Summary of the Invention

[0003] In view of this, the present application provides a tongue tracking method, apparatus, terminal device and storage medium, which can achieve reliable tracking of the tongue and facilitate further tongue image identification and analysis.

[0004] In a first aspect, the present application provides a tongue tracking method, comprising: In response to an input current frame image, a basic tracker is used to track the tongue in the current frame image and output a first tracking result; the basic tracker includes a feature extraction network and a tongue tracking model for generating the first tracking result; When the first tracking result is verified to be unreliable, using a global search module to generate region proposals and perform target image matching on the current frame image, so as to output the region proposal with the highest matching degree as the tongue tracking result of the current frame; Iteratively updating the tongue tracking model using the output tongue tracking result; The basic tracker is used to continue tracking the tongue on the next input frame image until the last frame image is tracked.

[0005] In a second aspect, the present application provides a tongue tracking device, comprising: a tracking module configured to, in response to an input current frame image, track the tongue in the current frame image using a basic tracker and output a first tracking result; the basic tracker comprising a feature extraction network and a tongue tracking model for generating the first tracking result; A verification module, configured to verify whether the first tracking result is reliable; a global search module, configured to, when verifying that the first tracking result is unreliable, generate a region proposal and perform target image matching on the current frame image using the global search module, so as to output the region proposal with the highest matching degree as the tongue tracking result of the current frame; a network updating module, configured to iteratively update the tongue tracking model using the output tongue tracking result; The tracking module is further configured to continue tracking the tongue on the next input frame image using the basic tracker until the last frame image is tracked.

[0006] In a third aspect, the present application provides a terminal device, which includes a processor and a memory, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the tongue tracking method.

[0007] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program. When the computer program is executed, the tongue tracking method according to any one of the aforementioned embodiments is implemented.

[0008] The embodiments of the present application have the following advantages: The tongue tracking method proposed in the embodiments of the present application utilizes a basic tracker to track the tongue in the current frame image. If the tongue tracking result is unreliable, a global search module is used to generate region proposals and perform target image matching on the current frame image to obtain the tracking result for the current frame. The output tracking result is then used as a template sample to iteratively update the tongue tracking model in the basic tracker. Tongue tracking is then continued for the next input frame image until tracking is complete. By combining the basic tracker with a region proposal strategy based on global search, this method can achieve stable and reliable tracking of the tongue. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A flow chart of a tongue tracking method according to an embodiment of the present application is shown; Figure 2 Another flow chart of the tongue tracking method according to an embodiment of the present application is shown; Figure 3 A flowchart of region suggestion generation according to an embodiment of the present application is shown; Figure 4 A flowchart of an area location update based on three search phases according to an embodiment of the present application is shown; Figure 5 A schematic diagram illustrating a region suggestion guided by a swarm intelligence algorithm according to an embodiment of the present application is shown; Figure 6 A flowchart of iteratively updating the tongue tracking model according to an embodiment of the present application is shown; Figure 7 A schematic diagram illustrating an application of the tongue tracking method according to an embodiment of the present application is shown; Figure 8 A schematic structural diagram of a tongue tracking device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0011] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0012] The components of the embodiments of the present application generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but rather merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0013] Hereinafter, the terms "including", "having" and their cognates, which may be used in various embodiments of the present application, are intended only to indicate specific features, numbers, steps, operations, elements, components or combinations of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing items or the possibility of adding one or more features, numbers, steps, operations, elements, components or combinations of the foregoing items.

[0014] Furthermore, the terms “first,” “second,” “third,” etc., are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.

[0015] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meaning as in the context of the relevant technical field and will not be interpreted as having an idealized meaning or an overly formal meaning unless clearly defined in the various embodiments of the present application.

[0016] This application proposes a tongue tracking method based on screening out "high-quality" region proposals related to the target and an adaptive short-term memory network. The method first uses a basic tracker to track the target, and if tracking fails, switches to a global search of the current frame image to screen specific region proposals to solve the missing label problem; at the same time, reliable template samples are collected to establish a robust tracking model, which can effectively solve the problem of frequent appearance changes. By combining the two to form a target tracking algorithm for tongue images, the tracking effect can be improved and continuous and reliable tracking of the target can be achieved.

[0017] The tongue tracking method of the embodiment of the present application is described below.

[0018] Figure 1 A flow chart of a tongue tracking method according to an embodiment of the present application is shown. Exemplarily, the tongue tracking method includes the following steps: S100 , in response to an input current frame image, using a basic tracker to track the tongue in the current frame image, and outputting a first tracking result; wherein the basic tracker includes a feature extraction network and a tongue tracking model.

[0019] Among them, the basic tracker is used to perform preliminary tracking of the target in the input image. For example, in one embodiment, the basic tracker mainly includes two parts, namely a feature extraction network and a tongue tracking model. Among them, the feature extraction network is used to extract tongue-related features from the input image. For example, it can be constructed using a ResNet network, which is not specifically limited here. The tongue tracking model is used to identify and locate the tongue target based on the extracted features. For example, the tongue tracking model can be constructed using a Faster R-CNN network, a YOLO series network, etc., which is not specifically limited here.

[0020] It is understood that in this application, the tracking target refers to the entire tongue or a designated area of ​​the tongue that can reflect the characteristics of the tongue image. This is not limited here and can be set according to actual needs. The tongue image is an image composed of the color and shape of the tongue and the tongue coating. The input image should usually be a global or local image of the face including the tongue. Due to the shaking of the human body or the change of the tongue state (such as rolling the tongue, flipping the tongue, etc.), it is easy to fail to capture the tongue image of the tongue or a local area of ​​the tongue, thereby affecting the subsequent tongue image analysis or accuracy.

[0021] It's worth noting that the first input image frame is typically captured focusing on the tongue area. Therefore, the tongue area can be annotated and used as the first template sample for training the tongue tracking model. As more images are subsequently input, more template samples can be collected for iterative training of the tongue tracking model.

[0022] Exemplarily, after receiving one of the input image frames, the basic tracker first uses the feature extraction network to extract feature information from the current frame image. Then, the tongue tracking model is used to perform fixed-body tongue tracking based on the extracted feature information from the current frame to obtain a first tracking result. It will be appreciated that to facilitate the distinction between the tongue tracking results output by the basic tracker and those output by the global search module, the term "first" will be used to describe the tongue tracking result output by the basic tracker, while the term "second" will be used to describe the tongue tracking result output by the global search module.

[0023] After the feature information is extracted, the depth feature information of the current frame can be convolved with the current tongue tracking model to obtain the confidence scores of the center points of multiple predicted targets, and the point with the largest confidence score is output as the prediction result of the current frame. For example, in one embodiment, the confidence score for: ; Where, represents the feature information of the t-th frame, represents the tongue tracking model, Represents convolution.

[0024] S120: Verify whether the first tracking result is reliable. If not, proceed to step S130. Alternatively, if reliable, proceed to step S130.

[0025] For example, Figure 2 As shown, the verifier can be used to verify the reliability of the first tracking result of the current frame to further confirm whether it can be used as a training sample for the tongue tracking model. If it is unreliable, the global search module can be switched to perform tongue target region proposal; if it is reliable, the output of the basic tracker can be directly used as the tracking result.

[0026] For example, the MDNet (Multi-Domain Network) target tracking algorithm can be used to evaluate the reliability of tongue tracking results. For example, when the overlap (IoU) between the predicted target box and the sample's ground-truth box is greater than the threshold parameter in the verification metric, it is judged as reliable; otherwise, it is judged as unreliable. It can be understood that using MDNet for reliability assessment can prompt this application to switch between the basic tracker and the global search module, thereby adjusting the tongue tracking strategy in a timely manner.

[0027] S130 : When the first tracking result is verified to be reliable, the first tracking result is directly output as the tongue tracking result of the current frame.

[0028] S140: When the first tracking result is verified to be unreliable, a global search module is used to generate a region proposal and perform target image matching on the current frame image, so as to output the region proposal with the highest matching degree as the tongue tracking result of the current frame.

[0029] In order to achieve continuous target tracking, when the basic tracker fails to track, this application will switch to a global search mechanism, that is, perform a global search in the current frame image and filter out target-specific suggestions through fitness as the second tracking result to solve the missing label problem.

[0030] In one embodiment, for the global search module, for example, a swarm intelligence algorithm (such as the MPA marine predator algorithm) is used as a search strategy to filter out target-specific suggestions, such as Figure 3 Shown, including: S210: Generate a preset number of tongue candidate regions with known initial positions on the current frame image.

[0031] For example, suppose that Np initial candidate region proposals (x, y, s) about the tongue are given on the current frame image P, where (x, y) represents the pixel coordinate position of the predicted target center and s is the scale parameter. Then, the position of the i-th region proposal can be described by the following expression: ; Where, and Indicates the lower and upper bounds of the width and height of the current frame image P, and rand is a random number between 0 and 1.

[0032] S230, iteratively update the positions of the generated tongue candidate regions through three search stages and perform fitness evaluation on all candidate regions generated each time to determine several specific candidate regions that match the target image in each iteration, until the iteration stop condition is met, and determine the tongue region suggestion with the highest matching degree from the several specific candidate regions obtained most recently.

[0033] As an alternative implementation, for different search stages, different wave speed ratios will be used to update the positions of these tongue body candidate regions. Demonstratively, the above three search stages can be divided according to the range interval where the iteration number is located. For example, assuming that the maximum iteration number is T, if the current iteration number t satisfies t < T t < T / 3, it means that it is in the first search stage, that is, the high wave speed ratio stage; if the current iteration number t satisfies T / 3 < t < 2T / 3, it means that it is in the second search stage, that is, the unit wave speed ratio stage; if the current iteration number t satisfies t > 2T / 3, it means that it is in the third search stage, that is, the low wave speed ratio stage.

[0034] After obtaining the initial position of the tongue body region suggestion, if the tongue body region suggestion is generated and updated according to the three search stages divided above, as Figure 4 shown, including: S31,0 At the high wave speed ratio stage, update the positions of each tongue body candidate region based on the first Brownian motion strategy.

[0035] Among them, the high wave speed ratio stage is the exploration stage of global search. The positions of the region suggestions in this stage will be updated using the Brownian motion strategy. If described by an expression, there is: , ; In the formula, represents the i th region suggestion, is a uniform random vector between 0 and 1; represents the elite matrix with the best fitness among the i th region suggestions; is a random vector generated based on Brownian motion, which follows a normal distribution; Np is the number of initial candidate regions.

[0036] S330, At the unit wave speed ratio stage, update the positions of the first half of the tongue body candidate regions in the preset quantity based on the first Levy flight strategy, and update the positions of the second half of the tongue body candidate regions in the preset quantity based on the second Brownian motion strategy.

[0037] Among them, entering the unit wave speed ratio stage marks the transition of the tongue body candidate region suggestion from the global exploration stage to the local development stage, aiming to quickly identify the potential regions of the target.

[0038] Demonstratively, in the second search stage, the position update of the preset quantity of tongue body candidate regions is divided into two parts. For example, for the first half of the tongue body region suggestions, the Levy flight strategy can be used for update. If described by an expression, there is: ; Where, Represents a random vector based on Levy flight.

[0039] For example, the Brownian motion strategy can be used to update the tongue region suggestion in the back half. If described by an expression, we have: ; Where, the adaptive parameter of the moving step is , used to control the moving distance of the area; is the current iteration number, is the maximum number of iterations.

[0040] S350: In the low wave speed ratio stage, the positions of the candidate tongue regions are updated based on the second Levy flight strategy.

[0041] The low wave velocity ratio stage is used to perform a more detailed search on some potential areas searched in the unit wave velocity ratio stage. If described by an expression, we have: .

[0042] It is understandable that there are some differences in the expressions of the Levy flight strategy and the Brownian motion strategy adopted in different stages. For better description, "first" and "second" are used above to distinguish them.

[0043] This embodiment initializes a preset number of tongue candidate regions and updates the positions of the tongue candidate regions in three search stages. After obtaining the positions of these candidate regions each time, Figure 5 As shown, by extracting the HOG (Histogram of Oriented Gradients) features of these candidate regions and evaluating the fitness of these candidate regions, for example, in one embodiment, the fitness of each candidate region of the tongue can be evaluated according to the correlation between the current candidate region of the tongue and the target image, so as to screen out several specific candidate regions with higher fitness after each iteration. It can be understood that the HOG feature characterizes the shape and appearance characteristics of the target object by statistically analyzing the gradient direction distribution of the local area of ​​the image. Finally, when the stopping condition is met (such as reaching the maximum number of iterations, etc.), the region proposal with the greatest correlation (i.e., the highest matching degree) is selected from the latest several specific candidate regions recorded and output as the tongue tracking result of the current frame.

[0044] Among them, the fitness is calculated based on the correlation between the candidate region and the target image. At this time, the expression of the fitness evaluation is: ; Where X is the HOG feature of the current candidate region, Y is the target image, represents the covariance, represents variance; Represents the correlation. The target image is determined based on the ground-truth sample value of the first frame tongue image.

[0045] S160: Iteratively update the tongue tracking model using the output tracking results.

[0046] In order to train a robust tongue tracking model, this application adopts an adaptive short-term memory network to collect reliable target tracking results output each time; and then uses these sample features to perform feature fusion to construct and train a tongue tracking model.

[0047] In one embodiment, Figure 6 As shown, the iterative update of the tongue tracking model includes: S410: Using an adaptive short-term memory network, the tongue tracking results output each time are collected as template samples to generate a template set.

[0048] The template set may include several template samples. Typically, during each iteration of the tongue tracking model, the template samples in the template set can be divided into different batches and input into the model for training. In addition, to prevent error accumulation, this embodiment also uses an adaptive short-term memory network to update the collected template samples. It can be understood that the adaptive short-term memory network is a short-term memory network that is dynamically adjusted based on the confidence score. It can capture long-term dependencies in the input data and achieve more flexible information retention.

[0049] For example, in a feasible implementation, when using an adaptive short-term memory network to collect tongue tracking results outputted each time as template samples to generate a template set, the following steps are also included: In response to a configured memory interval of the adaptive short-term memory network, a template sample with the highest confidence score is retained from a first number of template samples for each interval. For example, assuming the memory interval is k, the template sample with the highest confidence score from the k frames is retained for every k frames to ensure that the template sample used for network update has a higher confidence.

[0050] Further optionally, considering the target state change during continuous tracking, the method further includes: When the number of acquired template samples (i.e., the memory size) exceeds the second number, the earliest template sample input into the network is discarded according to the first-in, first-out principle. Typically, the first number is smaller than the second number. The specific value can be set based on actual needs and is not limited here. For example, assuming the memory size is n, when the n+1th sample is detected, the first oldest sample collected is discarded. This ensures that the number of collected samples does not exceed n and that template samples can be updated in a timely manner to ensure sample timeliness.

[0051] S430: After extracting sample features of each template sample in the template set based on the feature extraction network, a convolution operation is performed based on the current sample feature and the tongue tracking model obtained in the previous frame to obtain a weight value of the current sample feature.

[0052] The weight value reflects the quality level of the corresponding sample feature. For example, after obtaining the tongue features in the first frame image and the template samples collected from the basic tracker and global search module, the basic tracker first extracts the feature information of each template sample (referred to as sample features) through the feature extraction network. Next, the weight value of each sample feature is calculated. For example, for the i-th sample feature, its weight value is determined by combining the sample feature with the previously acquired tongue tracking model. For example, convolution processing can be used to calculate the weight value of the sample feature.

[0053] If we use expressions to describe it, we have: , Where, represents the feature extraction operation, where , C 、 H 、 W Represent the channel, height and width of the input convolution feature respectively; is the number of template samples.

[0054] S450, normalize the weight value of each sample feature, and perform feature fusion on all sample features and the corresponding normalized weight values, and combine the sample true value of the first frame tongue image as the residual term to obtain the updated tongue tracking model.

[0055] Next, the expression for the normalization of the weight value of each sample feature is as follows: Where, is the normalized weight value of the i-th sample feature.

[0056] Furthermore, the obtained weight values ​​are fused with the corresponding sample features, and residual learning is used to introduce the sample ground-truth of the first frame tongue image as the residual term to further enhance the target representation ability and reduce the cumulative error of the template sample.

[0057] Exemplarily, the updated tongue tracking model can be expressed as: Where, is the updated tongue tracking model, ( ) means that each channel of each template sample is normalized separately. is the residual term corresponding to the first frame.

[0058] It can be understood that by dynamically collecting template samples through the adaptive short-term memory network designed above for iterative updating of the tongue tracking model, a more robust tracking model can be established.

[0059] S180: Continue tracking the tongue on the next input frame image using the basic tracker until the last frame image is tracked.

[0060] Exemplarily, after obtaining the tongue tracking result of the current frame, the process will return to step S100 to continue tongue tracking for the next input frame image until it is determined to be the last frame image and the target tracking is completed according to steps S100-S180, and then the tracking is ended.

[0061] In order to verify the effectiveness of the tongue tracking method of this application, Figure 7 As shown, the following operations are performed by using the adaptive short-term memory network based on this application to collect samples to continuously update the tongue tracking model, and the global search module based on the MPA algorithm to generate region suggestions: Step 1: Initialize the target state parameters, verifier, adaptive short-term memory network and global search module parameters. For example, it includes: reading the target information of the first frame containing the tongue image and recording it as the initial state parameter , where [x, y] is the coordinate value of the center pixel of the tongue region in the target image, w is the width of the tongue region in the target image, and h is the height of the tongue region in the target image; and, the threshold value in the MDNet algorithm used for verification is configured to be 0; and, the memory size n=20 and the memory interval k=5 in the adaptive short-term memory network are configured; and, the number of global region proposals in the global search module is configured Np=50 , the maximum number of iterations T = 100, the number of target specific region suggestions N = 10, that is, the top 10 candidate regions with the largest fitness are selected as target specific regions. It should be understood that the above configuration values ​​are only a feasible example and are not intended to be the only limitation.

[0062] Step 2: Use the basic tracker to predict the tongue tracking result of the input image. For example, ResNet-50 is used to extract the deep feature information of the input image and convolve it with the currently stored tongue tracking model to obtain the confidence score of the tongue tracking result of the frame image, which is , where Indicates the t The depth feature information of the frame, It can be understood that the calculation method of the confidence score of the tongue tracking result of each frame image in this embodiment is the same as the calculation method of the weight value of the sample feature.

[0063] Step 3: Use MDNet to evaluate the reliability of the tongue tracking results. MDNet can crop the input image into image blocks of size 107×107×3. When the threshold parameter is set to 0, when the overlap rate (IoU) between the predicted target box and the sample ground truth box is greater than 0, it is judged to be reliable, otherwise it is judged to be unreliable.

[0064] Step 4: When the result is reliable, the tongue tracking result is output as the tracking result, and the tracking result is input into the adaptive short-term memory network as a valid template sample, and then used to build and train a robust tongue tracking model; otherwise, the swarm intelligence algorithm MPA is used to generate region proposals and obtain the target specific area matching the target image based on fitness evaluation as the output tongue tracking result; similarly, for the tongue tracking result output by the global search module, it is also input into the adaptive short-term memory network as a template sample for iterative training of the tongue tracking model.

[0065] Step 5: Determine whether the current frame is the last frame. If so, the tracking ends; otherwise, return to step 2 to perform target tracking for the next frame.

[0066] The tongue tracking method of the embodiment of the present application generates global region suggestions through a swarm intelligence algorithm, and screens target-specific suggestions through fitness evaluation to improve tracking accuracy, which can solve the recovery performance of basic tracker failures; then, a robust tongue tracking model is constructed through an adaptive short-term memory network, which can further improve tracking performance and achieve sustainable tracking of tongue targets, which is of great significance for the subsequent high-precision identification, understanding and analysis of targets in traditional Chinese medicine.

[0067] Figure 8 FIG. 1 is a schematic diagram showing a structure of a tongue tracking device 100 according to an embodiment of the present application. Exemplarily, the tongue tracking device 100 includes: The basic tracker module 110 is configured to, in response to an input current frame image, track the tongue in the current frame image using a basic tracker and output a first tracking result; the basic tracker includes a feature extraction network and a tongue tracking model for generating the first tracking result; A verification module 120 is used to verify whether the first tracking result is reliable; A global search module 130 is configured to generate a region proposal and perform target image matching on the current frame image using the global search module when the first tongue tracking result is verified to be unreliable, so as to output the region proposal with the highest matching degree as the tongue tracking result of the current frame; A model updating module 140 is configured to iteratively update the tongue tracking model using the output tongue tracking results; The basic tracker module 110 is further configured to continue tracking the tongue on the next input frame image using the basic tracker until the last frame image is tracked.

[0068] Optionally, the basic tracker module 110 is further configured to output the first tracking result directly as the tongue tracking result of the current frame when the first tracking result is verified to be reliable.

[0069] In an optional embodiment, the global search module 130 performs region proposal generation and target image matching on the current frame image, specifically including: A candidate region generating unit, configured to generate a preset number of tongue candidate regions with known initial positions on the current frame image; The candidate region iteration unit is used to iteratively update the position of the tongue candidate region generated through three search stages and perform fitness evaluation on all tongue candidate regions generated each time to determine several specific candidate regions that match the target image in each iteration until the iteration stop condition is met, and then determine the tongue region suggestion with the highest matching degree from the several specific candidate regions newly obtained.

[0070] In an optional embodiment, the candidate region iteration unit is configured to iteratively update the position of the generated tongue candidate region through three search phases, including: In the high wave velocity ratio stage, the positions of the candidate tongue regions are updated based on the first Brownian motion strategy; In the unit wave speed ratio stage, the positions of the tongue candidate regions in the first half of the preset number are updated based on the first Levy flight strategy, and the positions of the tongue candidate regions in the second half of the preset number are updated based on the second Brownian motion strategy; In the low wave speed ratio stage, the positions of the candidate tongue regions are updated based on the second Levy flight strategy.

[0071] In an optional embodiment, the fitness is calculated based on the correlation between the tongue candidate region and the target image; wherein the target image is determined based on the true value of the sample of the first frame of the tongue image.

[0072] In an optional embodiment, the model updating module 140 is configured to iteratively update the tongue tracking model using the output tongue tracking result, specifically including: a sample collection unit, configured to use an adaptive short-term memory network to collect the tongue tracking results output each time as template samples to generate a template set; a weight updating unit, configured to, after extracting sample features of each template sample in the template set based on the feature extraction network, perform a convolution operation based on the current sample feature and the tongue tracking model acquired last time, so as to obtain a weight value of the current sample feature; The feature fusion unit is used to normalize the weight value of each sample feature, and perform feature fusion on all the sample features and the corresponding normalized weight values, and combine the sample true value of the first frame tongue image as the residual term to obtain the updated tongue tracking model.

[0073] Further optionally, the sample collection unit is configured to use the adaptive short-term memory network to collect the tongue tracking results output each time as template samples to generate a template set, and further includes: In response to the configured memory interval of the adaptive short-term memory network, a template sample with the largest confidence score is retained in each interval of a first number of the plurality of template samples.

[0074] Further optionally, when the number of the collected template samples exceeds a second number, the earliest input template sample is discarded according to a first-in-first-out principle to make the number of template samples equal to the second number; the first number is less than the second number.

[0075] It can be understood that the device of this embodiment corresponds to the tongue tracking method of the above embodiment, and the options in the above embodiment are also applicable to this embodiment, so they will not be described again here.

[0076] In addition, the present application also provides a terminal device, which can be, for example, a computer, a notebook, a mobile terminal, etc., and its form is not limited. Exemplarily, the terminal device includes a processor and a memory, and the memory stores a computer program. The processor runs the computer program, thereby causing the terminal device to execute the functions of each module in the tongue tracking method or tongue tracking device of the embodiments of the present application, thereby achieving continuous, stable and accurate tracking of the tongue target.

[0077] The processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including at least one of a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, etc., and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0078] The memory may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM). The memory is used to store computer programs, and the processor can execute the computer programs accordingly after receiving an execution instruction.

[0079] This application also provides a computer-readable storage medium for storing the computer program used in the terminal device. For example, the computer-readable storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or flowchart, and the combination of boxes in the structure diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0081] In addition, the functional modules or units in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0082] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a smart phone, personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0083] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A tongue tracking method, characterized in that: include: In response to an input current frame image, using a basic tracker to track the tongue in the current frame image, and outputting a first tracking result; The basic tracker includes a feature extraction network and a tongue tracking model for generating the first tracking result; When the first tracking result is verified to be unreliable, using a global search module to generate region proposals and perform target image matching on the current frame image, so as to output the region proposal with the highest matching degree as the tongue tracking result of the current frame; Iteratively updating the tongue tracking model using the output tongue tracking result; The basic tracker is used to continue tracking the tongue on the next input frame image until the last frame image is tracked.

2. The tongue tracking method according to claim 1, characterized in that: The outputting of the first tracking result further includes: When the first tracking result is verified to be reliable, the first tracking result is directly output as the tongue tracking result of the current frame.

3. The tongue tracking method according to claim 1, wherein: The utilizing a global search module to generate a region proposal and perform target image matching on the current frame image includes: Generating a preset number of tongue candidate regions with known initial positions on the current frame image; The positions of the generated tongue candidate regions are iteratively updated through three search stages and the fitness of all tongue candidate regions generated each time is evaluated to determine several specific candidate regions that match the target image in each iteration. When the iteration stop condition is met, the tongue region proposal with the highest matching degree is determined from the several specific candidate regions most recently obtained.

4. The tongue tracking method according to claim 3, characterized in that: The iterative updating of the position of the tongue candidate region generated through three search stages includes: In the high wave velocity ratio stage, the positions of the candidate tongue regions are updated based on the first Brownian motion strategy; In the unit wave speed ratio stage, the positions of the tongue candidate regions in the first half of the preset number are updated based on the first Levy flight strategy, and the positions of the tongue candidate regions in the second half of the preset number are updated based on the second Brownian motion strategy; In the low wave speed ratio stage, the positions of the candidate tongue regions are updated based on the second Levy flight strategy.

5. The tongue tracking method according to claim 3, characterized in that: The fitness is calculated based on the correlation between the tongue candidate region and the target image; wherein the target image is determined based on the true value of the sample of the first frame of the tongue image.

6. The tongue tracking method according to claim 1 or 2, characterized in that: The step of iteratively updating the tongue tracking model by utilizing the output tongue tracking result includes: Using an adaptive short-term memory network to collect the tongue tracking results output each time as template samples to generate a template set; After extracting the sample features of each template sample in the template set based on the feature extraction network, performing a convolution operation based on the current sample feature and the tongue tracking model obtained last time to obtain a weight value of the current sample feature; The weight value of each sample feature is normalized, and all the sample features and the corresponding normalized weight values ​​are fused, and the true value of the sample of the first frame tongue image is combined as the residual term to obtain the updated tongue tracking model.

7. The tongue tracking method according to claim 6, characterized in that: When the method uses the adaptive short-term memory network to collect the tongue tracking results output each time as template samples to generate a template set, it also includes: In response to the configured memory interval of the adaptive short-term memory network, retaining a template sample with the largest confidence score among a first number of the plurality of template samples at each interval; And, when the number of the collected template samples exceeds a second number, the earliest input template sample is discarded according to the first-in-first-out principle to make the number of template samples equal to the second number; the first number is less than the second number.

8. A tongue tracking device, characterized in that: include: a tracking module, configured to, in response to an input current frame image, track the tongue body on the current frame image using a basic tracker and output a first tracking result; The basic tracker includes a feature extraction network and a tongue tracking model for generating the first tracking result; A verification module, configured to verify whether the first tracking result is reliable; a global search module, configured to, when verifying that the first tracking result is unreliable, generate a region proposal and perform target image matching on the current frame image using the global search module, so as to output the region proposal with the highest matching degree as the tongue tracking result of the current frame; a network updating module, configured to iteratively update the tongue tracking model using the output tongue tracking result; The tracking module is further configured to continue tracking the tongue on the next input frame image using the basic tracker until the last frame image is tracked.

9. A terminal device, characterized in that: The terminal device includes a processor and a memory, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the tongue tracking method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The device stores a computer program, which, when executed, implements the tongue tracking method according to any one of claims 1 to 7.

Images