Traditional Chinese medicine syndrome differentiation auxiliary analysis system based on multi-modal image acquisition

Abstract

The application discloses a traditional Chinese medicine syndrome differentiation auxiliary analysis system based on multi-modal image acquisition and relates to the technical field of traditional Chinese medicine syndrome differentiation auxiliary analysis.The application comprises a camera preview frame acquisition module, a applicability evaluation module, a focus optimization distribution module and a multi-modal fusion enhancement module.The preview frame acquisition module is responsible for capturing the camera preview frame and carrying out multi-modal image conversion, detecting the face and extracting the mouth key points, calculating the facial structure coverage and the tongue activity;further based on the above indexes, a structure strength graph is generated and the focus candidate area is extracted;the focus optimization distribution module combines the device pixel processing capacity and the acquisition time length, carries out importance evaluation on the candidate area, selects and outputs the main focus area and the resource configuration vector;finally, a soft boundary mask is constructed according to the above, background completion and Laplacian pyramid fusion are executed, and a tongue image and face image enhanced image pair is generated, so that the multi-modal feature reservation quality is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of traditional Chinese medicine syndrome differentiation auxiliary analysis, and more specifically, the present application relates to a traditional Chinese medicine syndrome differentiation auxiliary analysis system based on multi-modal image acquisition. BACKGROUND

[0002] The traditional Chinese medicine syndrome differentiation auxiliary analysis system fuses multi-modal features by collecting tongue images and face images to provide objective basis for constitution identification and syndrome classification. However, in the mobile terminal real-time acquisition scene, due to the constraints of short acquisition window and terminal computing power, the existing system has the following key defects: the tongue image acquisition needs the oral cavity to be fully opened, and the face image needs the complete face structure, and the two have acquisition conflicts in the dynamic scene. The existing system adopts a fixed area acquisition strategy and does not dynamically allocate resources according to the actual tongue / facial region quality in the frame, resulting in overexposure of the face image when the tongue area is blurred or the tongue image is out of focus when the face is blocked.

[0003] The traditional scheme presets fixed pixel resource quotas for the tongue and face, without considering whether the tongue region in the current frame has key pathological features such as discoloration and cracking, and whether the face region is blocked by hair, gestures, etc. When the light suddenly changes or the user moves, the key modal information loss rate is high.

[0004] When the system detects that the quality of a certain mode is low, it only prompts the user to retake or extend the acquisition time, which violates the "rapid diagnosis" principle of traditional Chinese medicine diagnosis, and cannot meet the real-time needs of mobile scenes such as subways and outdoors.

[0005] The existing technology lacks a dynamic resource optimization mechanism and a quality-driven focus allocation strategy, and in the short acquisition window, it faces the fundamental contradiction that the quality of the tongue image and the face image cannot be compatible.

[0006] In view of the above problems, the present application provides a solution. SUMMARY

[0007] The present application aims to provide a traditional Chinese medicine syndrome differentiation auxiliary analysis system based on multi-modal image acquisition, which solves the tongue image and face image quality imbalance problem of the existing traditional Chinese medicine syndrome differentiation auxiliary analysis in the short acquisition window.

[0008] The purpose of the present application can be achieved by the following technical solutions:

[0009] The TCM syndrome differentiation auxiliary analysis system based on multi-modal image acquisition comprises the following modules: a camera preview frame acquisition module is used to capture the camera preview frame, convert it into an HSV color space graph and a grayscale graph, and locate the face area and the tongue candidate window on the image; the face structure coverage index of the face area and the activity index of the tongue candidate window are calculated respectively, and after normalization, the two form a structure response vector; based on the structure response vector, the absolute difference value is calculated, and if the absolute difference value exceeds the preset threshold, the inapplicability identifier is output and the applicability evaluation module is started;

[0010] The applicability evaluation module generates an original structure intensity graph based on the structure response vector, and extracts a connected candidate region set and a structure description vector according to the coordinate mapping relationship of each normalized edge response in the face area and the tongue candidate window;

[0011] The focus optimization distribution module is used to obtain the acquisition window length and the device pixel processing capability, and calculate the pixel resource budget; the single connected region pixel number exceeding the pixel resource budget is removed for the connected candidate region, and an initial focus selectable region set is obtained; for each focus selectable region in the initial focus selectable region set, the structure response mean value and the region pixel number are normalized respectively, and the tongue and face modal labels are linearly weighted to generate a ranking score; the initial focus selectable region set is arranged in descending order of the ranking score, and is sequentially added to the pixel resource budget to form a selected focus region set; the focus region with the highest score in the selected focus region set is selected as the main focus region, and the center coordinates and size parameters of the region and the pixel resource budget are output as the focus region configuration parameters.

[0012] As a further scheme of the application: the face area positioning adopts a lightweight face detection algorithm, and records the confidence score corresponding to all face key points in the face area; a mouth contour detection method is further used in the face area to extract key points and determine the position and size of the tongue candidate window.

[0013] As a further scheme of the application: the acquisition logic of the face structure coverage index is as follows: the face area is calculated according to the width and height of the face area, and the mouth polygon area surrounded by the mouth key point set obtained by the mouth contour detection method is calculated; the mouth ratio is defined as the ratio of the calculated area of the mouth polygon area to the face area; the face key point average confidence is obtained by arithmetically averaging all face key point confidence scores, and the face structure coverage index is obtained by weighted summation of the mouth ratio and the face key point average confidence.

[0014] As a further scheme of the present application: the acquisition logic of the activity index of the tongue candidate window region is: taking the center of the mass center coordinates of the mouth key point set as the center, defining a fixed size tongue candidate window region in the face region, and performing conversion from the RGB color space to the HSV color space on the pixels in the tongue candidate window region; screening out the pixels with the hue value in the preset tongue color range, and obtaining the color saliency by accumulating the saturation values of the pixels and then dividing by the total number of the screened pixels; at the same time, using the Sobel operator to calculate the gradient amplitude of all pixels in the window and obtaining the average gradient amplitude; and weighting and summing the color saliency index and the average gradient amplitude with a preset weight coefficient to generate the activity index and mark it as the tongue activity index.

[0015] As a further scheme of the present application: the calculation of the gradient amplitude of all pixels in the window using the Sobel operator includes: calculating the horizontal direction gradient component and the vertical direction gradient component of each pixel in the tongue candidate window through the Sobel convolution kernel; performing square and square root operation on the horizontal and vertical gradient components of each pixel to obtain the edge intensity amplitude of the pixel; and accumulating the edge intensity amplitudes of all pixels in the tongue candidate window and then dividing by the total number of pixels to output the average gradient amplitude.

[0016] As a further scheme of the present application: the specific logic of outputting the inapplicability identifier is: performing normalization processing on the face structure coverage index and the tongue activity index obtained by analysis respectively to obtain a two-channel structure response vector; and performing difference value on the face structure coverage index and the tongue activity index in the two-channel structure response vector; comparing the difference value with a preset threshold τ diff ; if the difference value is greater than or equal to the preset threshold τ diff , setting the inapplicability identifier; otherwise, setting the applicability identifier, and then keeping the default dual-mode acquisition.

[0017] As a further scheme of the present application: based on the structure response vector, the normalized edge responses in the face region and the tongue candidate window are fused to generate an original structure intensity map according to the coordinate mapping relationship, including: cropping the face sub-image and the tongue candidate window sub-image from the original frame image, and recording the one-to-one mapping relationship between the pixel coordinates of each sub-image and the coordinates of the original frame image; applying Sobel edge detection to the face sub-image and the tongue candidate window sub-image respectively to obtain horizontal edge response and vertical edge response in sequence;

[0018] In each sub-image, the horizontal edge response and the vertical edge response are combined into the corresponding edge intensity response graph, and the pixel intensity of the edge intensity response graph is normalized; and the processed edge intensity response graph is taken as the normalized edge graph of the face region and the tongue candidate window.

[0019] Based on the double-channel structure response vector A = [Mface, Mst], the face region edge intensity response map And the tongue candidate window edge intensity response map Respectively map fill to the original image coordinates (x, y), and fuse to obtain the original structure intensity map S (x, y) according to the following rules: if the original image coordinates If Then If the two overlap, then Wherein, st represents the tongue candidate window region, Iface represents the face region; the rest of the region is 0, to obtain the complete original structure intensity map S.

[0020] As a further scheme of the application: the rule of fusing to generate the original structure intensity map according to the coordinate mapping relationship is: first performing row Gaussian smoothing on the obtained original structure intensity map, then performing binaryzation processing with a preset threshold Ts, and finally performing morphological opening and closing operation for denoising and hole filling to obtain the purified binary mask;

[0021] By performing connected region analysis on the purified binary mask, by scanning the pixel neighborhood relationship, the mutually connected foreground pixels are divided into independent initial connected region sets; the boundary box coordinates, the number of connected pixels and the average structure intensity of each initial connected region are calculated, and the modal label is determined according to the overlap rate of the boundary box coordinates of each initial connected region and the tongue candidate window region; if the overlap rate exceeds the preset threshold, the modal label is determined as the modal label; otherwise, the modal label is determined as the face modal label; and the tiny regions with the number of connected pixels less than the preset lower limit are removed, and the significant candidates are retained; and the initial connected regions are sorted in descending order of the average structure intensity, and the first J items with the maximum average structure intensity are taken as the final connected regions; for each connected region after screening, a structure description vector is generated.

[0022] As a further scheme of the application: the acquisition logic of the sorting score is: performing multi-scale neighbor division on all initial focus selectable regions, regarding each initial focus selectable region as a spatial information granule, adopting a coarse-grained method based on neighbor criterion, based on the neighbor criterion, dividing the upper approximation set and the lower approximation set corresponding to each initial focus candidate region and its description vector as the center under a plurality of neighbor numbers and neighborhood radii; under each neighbor scale, the average structure response intensity and the vector similarity of each initial focus selectable region are counted, and the multi-scale fusion response and the fusion similarity are obtained by fusing the statistical results of each scale according to the preset weight; the description vector includes the boundary box center corresponding to the initial focus selectable region, the width and height of the boundary box center, the average structure response corresponding to the initial focus selectable region and the modal label;

[0023] ​Introducing the self-information integrated focus metric to evaluate the importance of each initial focus selectable region to obtain a self-information focus metric value: In the formula, Indicates the local self-information amount of the initial focus selectable region in the upper approximate set and the lower approximate set of the neighborhood; And Respectively measure the significant confidence and non-redundancy of the region, and p is the probability mass function, and epsilon is the self-information fusion coefficient, Is a multi-scale fusion response, Is a fusion similarity; the self-information focus metric value, the pixel area normalization value of the initial focus selectable region and the modal priority are linearly integrated, and multiplied by a preset capacity perception penalty coefficient to obtain a ranking score of each initial focus selectable region;

[0024] As a further scheme of the application: after executing the focus optimization distribution module, a multi-modal fusion enhancement module is further included for constructing an image acquisition region template on the original preview frame according to the center coordinates and size parameters of the main focus region, to intercept the focus region image of the corresponding modality; the intercepted focus region image and the remaining non-focus region image content are spatially completed and image fused according to the modal labels of the regions, and the enhanced image pair is taken as the feature input for subsequent traditional Chinese medicine syndrome differentiation auxiliary analysis.

[0025] The application has the following beneficial effects:

[0026] (1) The application, through the traditional Chinese medicine syndrome differentiation auxiliary analysis system based on multi-modal image acquisition, effectively solves the multi-modal image fusion problem under the limitation of mobile terminal computing power and bandwidth through lightweight real-time analysis and dynamic focus optimization distribution of the tongue surface double-modal preview frame. The system completes the fast calculation of the face key points and the tongue activity index in a single frame, and realizes multi-dimensional importance evaluation and gradient response attenuation of the initial focus candidate region based on the near neighbor particle division, self-information focus metric and capacity perception penalty, effectively eliminates insignificant or redundant regions, ensures that the optimal main focus region configuration vector is always output under the extremely short acquisition window, and improves the accuracy of tongue feature acquisition and the diagnosis reliability;

[0027] (2) The application, through the multi-modal fusion enhancement module, realizes seamless completion and multi-layer fusion of the focus area and the background based on the Laplacian pyramid and feathering mask technology, not only retains the high-frequency details of the tongue surface and the face, but also considers the integrity of the surrounding context information, outputs high-quality tongue image and face image enhancement pairs, and provides clearer and more diagnostic input data for the subsequent traditional Chinese medicine syndrome differentiation model. The closed-loop acquisition-fusion-feedback mechanism can dynamically adjust the focus parameters of the next frame according to the analysis results, continuously optimize the acquisition effect, and significantly improve the real-time performance and stability of the mobile terminal traditional Chinese medicine syndrome differentiation auxiliary diagnosis. Attached Figure Description

[0028] The invention will now be further described with reference to the accompanying drawings.

[0029] Figure 1 This is a schematic diagram of the system framework of the TCM syndrome differentiation auxiliary analysis system based on multimodal image acquisition according to the present invention;

[0030] Figure 2 This is a schematic diagram illustrating the implementation logic of the camera preview frame acquisition module in this invention;

[0031] Figure 3 This is a schematic diagram illustrating the implementation logic of the applicability assessment module in this invention;

[0032] Figure 4 This is a schematic diagram illustrating the implementation logic of the focus optimization allocation module in this invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Example 1

[0035] Please see Figure 1 As shown, the present invention is a TCM syndrome differentiation auxiliary analysis system based on multimodal image acquisition, including the following modules: camera preview frame acquisition module, applicability assessment module, focus optimization allocation module, and multimodal fusion enhancement module;

[0036] In this embodiment, lightweight pre-evaluation is performed on bimodal images, including tongue and face images, with extremely short acquisition windows in mobile scenarios to determine whether to enter the dynamic focus configuration process, thereby ensuring the rapid acquisition of subsequent high-value images.

[0037] Please see Figure 2 As shown, the camera preview frame acquisition module is used to capture the real-time preview frame of the terminal camera and convert the frame into an HSV color space image and a grayscale image to obtain the current frame color image I, which is used for subsequent color and gradient analysis; and based on a lightweight face detection algorithm, it locates and crops the face region Iface on the current frame color image I. This region is defined by the upper left corner coordinates (x... f ,y f ) and width and height (w f ,h f) identification; for subsequent face region Iface within the face region Iface all face key point positioning algorithm such as 68 point detector for key point detection, and record each face key point output confidence score, all face key point confidence score arithmetic average get face key point average confidence;

[0038] Further use mouth contour detection method to extract mouth key point set Lmouth, used for subsequent tongue area positioning, get face region Iface;

[0039] It should be noted that the lightweight face detection algorithm is a mature known technology, such as Haar feature based cascade classifier (Viola-Jones algorithm), deep neural network based SSD-MobileNet, RetinaFace, or feature pyramid network based lightweight model. These methods have been widely used in mobile and embedded scenarios, and can quickly output face bounding box under limited computing power and real-time constraints; In this embodiment, no more elaboration;

[0040] In face region Iface, the calculation of face structure coverage index Mface is as follows: according to the width and height of face region Iface, the face area Aface = w f ×h f , Then calculate the area of the mouth polygon region surrounded by the mouth key point set Lmouth Amouth = PolyArea(Lmouth); Define the mouth ratio Rmouth as the ratio of the mouth polygon area calculation area Amouth and the face area Aface;

[0041] And the weighted sum of mouth ratio Rmouth and face key point average confidence Cface is obtained as face structure coverage index Mface;

[0042] It should be noted that the key point positioning algorithm can refer to Dlib68-point face marking, or face key point detection technology based on lightweight convolutional network such as MobileNetV2-HRNet. These schemes can predict the position coordinates of face key structure such as binocular, nose tip, mouth corner and chin in real time, which is used for subsequent mouth contour or tongue candidate region construction;

[0043] Further, the centroid coordinates (x c ,x c) as the center, a fixed-size tongue candidate window region st is defined within the face region Iface, RGB→HSV conversion is performed on the pixels within the tongue candidate window region st, and the pixel saturation is screened for hue H in the preset tongue color range [H1, H2]; and the color saliency is obtained by mean calculation In the formula, δ is a hue range indication function, bh(x, y) is the pixel saturation, and N is the total number of pixels in the tongue candidate window region st; the average gradient amplitude Gst within the tongue candidate window is obtained by using the gradient amplitude within the region of the Sobel operator and averaging, and the tongue candidate window region activity index is obtained by combining the preset weights β and (1-β), which is marked as the tongue activity index Mst;

[0044] The use of the gradient amplitude within the region of the Sobel operator includes calculating the horizontal gradient component and the vertical gradient component of each pixel within the tongue candidate window through the Sobel convolution kernel; the horizontal and vertical gradient components of each pixel are squared and square-rooted to obtain the edge intensity amplitude of the pixel; and the average gradient amplitude is obtained by accumulating the edge intensity amplitudes of all pixels within the tongue candidate window and dividing by the total number of pixels

[0045] The face structure coverage index Mface and the tongue activity index Mst obtained by analysis are normalized to obtain a two-channel structure response vector A=[Mface, Mst]; and the difference between the two-channel values in the two-channel structure response vector A is calculated to obtain ΔA;

[0046] The difference ΔA is compared with a preset threshold τ diff If ΔA≥τ diff , it is considered that there is a significant difference between the face and tongue structures in the current image, and the inapplicability flag flag=1 is set; otherwise, flag=0 is set; wherein the flag is used to control the subsequent focus dynamic configuration process: if flag=1, the subsequent focus candidate region extraction is entered; if flag=0, the default dual-mode acquisition is maintained or a user prompt for re-shooting and other compensation mechanisms are triggered;

[0047] The index calculation and comparison processes in this step are all completed within a single frame, and the processing time is not more than a single frame time threshold, so as to ensure real-time response in a very short acquisition window. All modules use lightweight algorithms or accelerated implementation to adapt to the computing power and energy consumption limitations of mobile terminals.

[0048] Please refer to Figure 3As shown, the applicability assessment module is used to calculate the distribution of local salient regions of the image based on the face region image corresponding to the applicability label and by using its coordinate position intensity projection, to obtain the original structural intensity map representing the key regions of the face and tongue; based on the generated original structural intensity map, the connected regions with response values ​​exceeding the preset threshold are extracted as a set of focal candidate regions, and the bounding box coordinates, mean response value and modality attribution label tongue / face of each candidate region are recorded, and finally the list of focal candidate regions and its structural description vector are output;

[0049] The structural intensity map is an edge response distribution map corresponding to significant regions in the image, such as the oral cavity opening and the center of the face.

[0050] Based on the face region image corresponding to the applicability label, and by using its position intensity projection to calculate the distribution of local salient regions in the image, the original structural intensity map representing the key regions of the face and tongue is obtained; based on the generated original structural intensity map, the connected regions with response values ​​exceeding a preset threshold are extracted as a set of focal candidate regions, and the bounding box coordinates, mean response value and modality attribution label (including tongue / face) of each candidate region are recorded, and finally the list of focal candidate regions and its structural description vector are output;

[0051] The original structure intensity map is the edge density distribution map corresponding to significant regions in the image, such as the oral cavity opening and the center of the face;

[0052] Extracting the original frame image I with verified inapplicability flags and its face region Iface = (x) from the camera preview frame acquisition module f ,y f ,w f ,h f ) and tongue candidate window region st = (x t ,y t ,w t ,h t Facial sub-images I are cropped from the original image I. f =I[y f :y f +h f ,x f :x f +w f ] and tongue sub-diagram I t =I[y t :y t +h t ,x t :x t +w t It records the mapping relationship between the pixel coordinates (u,v) of the sub-image and the coordinates (x,y) of the original image, that is, the pixel (u,v) of the sub-image corresponds to the pixel (x,y) of the original image. f +u,y f+v) or (x t +s,y t +t) to ensure that subsequent processing results can accurately reflect back the original image;

[0053] Furthermore, for facial sub-image I f and tongue sub-diagram I t The horizontal edge response G was calculated using the Sobel operator. x and vertical edge response G y In each sub-image, the horizontal edge response and the vertical edge response are combined to form the corresponding edge intensity response map. and Subsequently, based on the dual-channel structural response vector A = [Mface, Mst], the edge intensity response map of the face region is... And tongue candidate window edge intensity response map The original image coordinates (x, y) are mapped and filled back to respectively, and then fused according to the following rules to obtain the original structure intensity map S(x, y): If the original image coordinates but like but If the two overlap, then Set the remaining areas to 0 to obtain the complete original structural strength diagram S;

[0054] To remove salt-and-pepper noise and isolated low-intensity points, the system applies a Gaussian filter to S with a kernel size of k×k and a standard deviation of σ, thereby obtaining a smooth original structural intensity map Ssmooth; and performs binarization processing with a preset threshold Ts, and finally generates a mask B(x,y)=1 (Ssmooth≥T_s) through morphological opening and closing operations for noise reduction and hole filling.

[0055] Performing connected component analysis on B(x,y) essentially involves scanning pixel neighborhood relationships to divide interconnected foreground pixels (i.e., pixels with a value of 1) into independent initial connected component sets {C}. i}; where i represents the index of each connected region; for each connected region C i Calculate its bounding box BBox i =[x min ,y min ,x max ,y max ]; Traverse each connected region C i All pixels get the number of connected pixels Area i ; and based on the original structural strength map S in the connected region C i The average response on the surface yields the average structural strength. For each initial connected region C i According to its bounding box BBox iThe overlap rate determination mode label Label i , if the overlap rate exceeds a preset threshold , the determination mode label Label i is tongue; otherwise, the determination mode label Label i is face; this mechanism effectively distinguishes the source of the bimodal focus area in the same frame; small areas with a number of connected pixels Area i less than a preset lower limit A min are removed, and significant candidates are retained; and the first J items in descending order of Mean i are sorted, and the first J items, such as J = 3, are taken as the final connected area C ' i ;

[0056] For each connected area C ' i after screening, a structure description vector D i = [x ci , y ci , w i , h i , Mean i , Label i ] is generated, where (x ci , y ci ) is the center of the bounding box BBox i , (w i , h i ) is the width and height thereof, Mean i is the average structure intensity, and Label i is the label of the mode to which it belongs;

[0057] All D i are encapsulated in a list form to output, so as to provide complete and lightweight input data for subsequent modules; the list contains area coordinates, average structure intensity, and mode information, which helps the subsequent frame rate allocation and depth analysis modules;

[0058] This step uses classical image processing operators and simple geometric logic, without deep learning model inference, and can be completed in real time in a mobile terminal resource-limited environment, ensuring accurate extraction of the tongue surface bimodal focus area in a very short acquisition window.

[0059] Please refer to Figure 4As shown, the focus optimization distribution module is used to determine the pixel resource budget according to the connected region and the structure description vector, in combination with the device pixel processing capability and the acquisition time length; based on the response intensity, the area size and the modal label of the candidate region, the structure score of each candidate region is integrated with the spatial resource constraint to form a region priority ranking list; and the optimal focus region, i.e. the main modal main region, is screened out according to the region priority ranking list, and the center coordinates and size parameters of the region are output;

[0060] The current acquisition window time Tw of the device and the device pixel processing capability P are acquired; the acquisition window time Tw and the device pixel processing capability P are multiplied to obtain the pixel resource budget Budp;

[0061] Based on the pixel resource budget Budp, the connected regions and their structure description vectors output by the preceding module are integrated and analyzed, i.e. multiplied, to eliminate any single connected region exceeding the pixel resource budget Budp, and an initial focus selectable region set is obtained; ' i The connected pixel number Area i of each connected region C l is integrated and analyzed, i.e. multiplied, with the data volume estimation, to eliminate any single connected region exceeding the pixel resource budget Budp, and an initial focus selectable region set is obtained;

[0062] All the initial focus selectable regions are divided into multiple scales of near neighbors, each initial focus selectable region is regarded as a spatial information granule, and multi-dimensional information such as spatial coordinates and structure features is comprehensively considered. Specifically, a coarse-grained method based on a near neighbor criterion is adopted, each initial focus selectable region is taken as a spatial center, and its corresponding structure description vector is combined to divide the upper and lower approximation sets at a plurality of near neighbor numbers k l and neighborhood radii λ l , the average structure intensity and the vector similarity at the corresponding scale are counted; the statistical results of each scale are fused according to a preset weight w l to obtain a multi-scale fusion response and a fusion similarity

[0063] It should be noted that the coarse-grained method based on the near neighbor criterion can capture the local coherence and global representativeness of the initial focus selectable region set at different spatial and feature granularities, thereby avoiding the interference of isolated small spots or large patches on the final focusing result;

[0064] Subsequently, a self-information integrated focus metric is introduced to evaluate the importance of each initial focus selectable region: wherein represents the local self-information amount of the initial focus selectable region in the near neighbor upper and lower approximation set; and measure the significance confidence and non-redundancy of the region respectively, and p is a probability mass function; ε is a self-information fusion coefficient;

[0065] The aforementioned self-information focusing metric is further combined with the area of ​​the initial selectable focus region and modal priority using capacity-aware penalty and linear integration. The logic for obtaining the preset capacity-aware penalty coefficient includes log-normalizing the number of connected pixels in each initial selectable focus region to obtain a normalized value for the number of connected pixels; dividing the normalized value of the number of connected pixels by the pixel resource budget Budp, and then multiplying it by the preset penalty coefficient η to obtain the penalty CF. i This means that larger areas will be subject to more severe penalties, thus automatically lowering their priority in the overall evaluation.

[0066] Finally, the self-information focusing metric calculated earlier, the normalized number of connected pixels obtained above, and their corresponding modal labels are weighted and summed, and then the penalty term CF is subtracted. i The final sorting score G for each initial focus selectable region is obtained. i Specifically: first use w z As a weight for the self-information focusing metric, this self-information focusing... Divide by the maximum value S in all initially selectable focus areas max Perform normalization; use w A The weights used as the normalized value of the number of connected pixels; w L As a weight of the tongue / face mapping factor, where the mapping factor labels the tongue region with a label. st The facial area is marked with a Label face After weighted summing of these three parts, subtract the penalty term CF. i ;

[0067] Furthermore, the initial focus selectable regions are sorted according to the final sort score G. i Sort the regions from highest to lowest score, and sequentially extract each initial selectable focus region without replacement. For the current region, check if its connected pixel count, after adding it to the existing connected pixel count, still does not exceed the pixel resource budget Budp, and the number of selected regions has not yet reached the preset maximum value R. Then, include the initial selectable focus region in the final set of selected focus regions and add its connected pixel count to the existing connected pixel count. Otherwise, skip the initial selectable focus region and continue to the next one. If, after the above process, the number of selected regions is still less than the preset minimum focus number R, then supplement from the remaining unselected regions in the same scoring order until the number of selected regions reaches R. The existing connected pixel count is the cumulative number of pixels occupied by the selected regions.

[0068] Furthermore, in the final list of selected focus areas, the existence of a tongue region is first checked; if it exists, the tongue region with the highest score is selected as the final main focus area C.main If all are facial regions, the one with the highest score is selected in the facial region; when there are multiple items with similar scores in the same category, i.e., tongue or face, the distance of these regions from the center point of the image is further compared, and the one with the smallest distance is selected;

[0069] Finally, the system reads the center coordinates (x main ,y main ), size (w, h) of the main focus region C main , the main focus region modal label Label main , and combines the pixel resource budget Budp to output the spatial focus resource configuration vector F main =[x main ,y main ,w,h,Label main ,Budp];

[0070] For real-time driving of camera ROI setting and subsequent diagnostic module, efficient multi-modal focus acquisition.

[0071] For constructing an image acquisition region template on the original preview frame according to the center coordinates and size parameters of the main focus region, to intercept the focus region image of the corresponding modality; according to the modal label of each region, the contents of the intercepted focus region image and the remaining non-focus region image are spatially completed and fused, and the enhanced image pair is taken as the feature input for subsequent TCM syndrome differentiation auxiliary analysis;

[0072] Through the fusion image pair and the focus configuration metadata, the image information of the tongue image and the face image under the spatial priority order is integrated and analyzed, to realize multi-modal image structure information preservation and focus-driven enhanced acquisition closed loop under extremely short acquisition window and limited acquisition resources, specifically: first, the center coordinates and size parameters of the main focus region in the output spatial focus resource configuration vector F main are used to establish a focus region template on the original frame image; in this stage, the following aspects are realized:

[0073] According to the given center coordinates and width and height information, the top-left and bottom-right vertex positions of the template on the original image are determined, and boundary detection is performed to ensure that the template does not exceed the image range;

[0074] In order to take into account the context information around the region, the embodiment appropriately enlarges the region boundary based on the template size according to a preset expansion ratio;

[0075] The expanded rectangular region is converted into a binary mask, in which the pixel values inside the template are valid and the pixel values outside are invalid;

[0076] Smooth processing such as feathering or Gaussian filtering is applied to the edge of the mask to obtain a soft boundary ROI mask, to reduce the harshness in subsequent fusion.

[0077] It should be noted that by moderate boundary expansion and mask feathering, part of the surrounding context can be integrated while preserving the key ROI information, improving the naturalness and diagnostic integrity of subsequent image fusion;

[0078] After obtaining the soft boundary ROI mask, the system divides the original image into focus and non-focus regions according to the mask value:

[0079] Multiply the part with valid mask value with the original image pixel to obtain the focus region containing only the focal point content;

[0080] Multiply the part with invalid mask value with the original image pixel to obtain the background subgraph, i.e. the non-focus region, which contains only the background content;

[0081] For the separated non-focus region, only existing image completion techniques such as PatchMatch, bilateral filtering or lightweight deep completion network are needed to quickly fill the texture and smooth the edge of the region, so as to restore its visual coherence and retain sufficient background structure information;

[0082] For tongue and face images of two modalities, the system uses calibration parameters or affine transformation based on feature matching to align the soft boundary ROI mask and background image of the two modalities to the same coordinate system, preparing for pixel-level fusion;

[0083] For the aligned focus subgraph and completed background graph, a multi-layer Laplacian pyramid is constructed, and a weighted synthesis is performed according to the soft boundary mask at each level, which not only preserves the focus area details, but also considers the background global structure. Finally, the single-modality fusion image is reconstructed;

[0084] According to the modality label, different priority weights are assigned to the single-modality fusion images of the two modalities, which are combined at the channel or pixel level to generate the final multi-modality fusion image pair;

[0085] After generating the multi-modality fusion image pair, the system packages it together with the focal point configuration vector and feeds it back to the camera or acquisition control module: the fusion image pair is provided to the back-end analysis or local diagnosis model for feature extraction and classification; according to the analysis result, the size or position of the focus area of the next frame is dynamically adjusted to realize the closed loop of acquisition-fusion-analysis-feedback; when the device is running continuously, the closed loop can converge to the optimal acquisition configuration at any time, ensuring the acquisition of stable and high-quality multi-modality image sequences in a very short acquisition window.

[0086] In the embodiment, through the traditional Chinese medicine syndrome differentiation auxiliary analysis system based on multi-modal image acquisition, through the lightweight real-time analysis of the tongue surface double-mode preview frame and the dynamic focus optimization allocation, the multi-modal image fusion problem under the limitation of mobile terminal computing power and bandwidth is effectively solved. The system completes the fast calculation of the face key point and the tongue activity index in a single frame, and based on the near neighbor particle division, the self-information focusing measurement and the capacity perception punishment, realizes the multi-dimensional importance evaluation and gradient response attenuation of the initial focus candidate area, effectively eliminates the insignificant or redundant area, ensures that the optimal main focus area configuration vector is always output under the extremely short acquisition window, improves the accuracy of tongue feature acquisition and the diagnosis reliability; through the multi-modal fusion enhancement module, based on the Laplacian pyramid and feathering mask technology, the seamless completion and multi-layer fusion of the focus area and the background are realized, not only the high frequency details of the tongue surface and the face are reserved, but also the integrity of the surrounding context information is considered, a high quality tongue image and face image enhancement pair are output, which provides clearer and more diagnostic value input data for the subsequent traditional Chinese medicine syndrome differentiation model. The closed-loop acquisition-fusion-feedback mechanism can dynamically adjust the focus parameters of the next frame according to the analysis results, continuously optimize the acquisition effect, and significantly improve the real-time performance and stability of the mobile terminal traditional Chinese medicine syndrome differentiation auxiliary diagnosis.

[0087] The above formulas are dimensionless numerical calculations, and the formulas are obtained by software simulation of a large amount of data to obtain a formula of the nearest real situation, and the preset parameters in the formula are set by a person skilled in the art according to the actual situation.

[0088] The above embodiments can be realized wholly or partially by software, hardware, firmware or any combination thereof. When realized by software, the above embodiments can be realized wholly or partially in the form of a computer program product.

[0089] Those skilled in the art can realize that the modules and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application of the technical solution and the constraints of the invention. Professional technicians can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0090] In addition, the functional modules in each embodiment of the present application can be integrated in one processing module, or each module can exist physically alone, or two or more modules can be integrated in one module.

[0091] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, and all of them should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0092] Finally, the above merely provides the preferred embodiments of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A traditional Chinese medicine syndrome differentiation auxiliary analysis system based on multi-modal image acquisition, characterized in that, The method comprises the following modules: a camera preview frame acquisition module is used to capture a camera preview frame, convert it into an HSV color space graph and a grayscale graph, and locate a face region and a tongue candidate window on the image; a face structure coverage index of the face region and an activity index of the tongue candidate window are calculated respectively, and after normalization, the two indexes form a structure response vector; An absolute difference value is calculated based on the structure response vector, and if the absolute difference value exceeds a preset threshold, an inapplicability identifier is output and an applicability evaluation module is started; The applicability evaluation module generates an original structure intensity graph based on the structure response vector, fuses each normalized edge response in the face region and the tongue candidate window according to a coordinate mapping relationship, and extracts a connected candidate region set and a structure description vector thereof; A focus point optimization distribution module is used to acquire a collection window time length and a device pixel processing capability, calculate a pixel resource budget, and remove a single connected region pixel number that exceeds the pixel resource budget from the connected candidate region to obtain an initial focus point selectable region set. For each focus point selectable region in the initial focus point selectable region set, the structure response mean value and the region pixel number thereof are normalized respectively, and a ranking score is generated by linear weighting in combination with a tongue and face modal label; the initial focus point selectable region set is arranged in descending order of the ranking score, and is sequentially added to the pixel resource budget to form a selected focus point region set; the focus point region with the highest score in the selected focus point region set is preferentially selected as a main focus region, and the center coordinates and size parameters of the region and the pixel resource budget are output as focus region configuration parameters.

2. The traditional Chinese medicine syndrome differentiation auxiliary analysis system based on multi-modal image acquisition according to claim 1, characterized in that, The face region positioning adopts a lightweight face detection algorithm, and records a confidence score corresponding to each face key point in the face region; a mouth contour detection method is further used in the face region to extract key points and determine the position and size of the tongue candidate window.

3. The traditional Chinese medicine syndrome differentiation auxiliary analysis system based on multi-modal image acquisition according to claim 2, characterized in that, The face structure coverage index is obtained by calculating the face area according to the width and height of the face region, and then calculating the area of a mouth polygon region surrounded by a mouth key point set obtained by the mouth contour detection method; A mouth ratio is defined as the ratio of the calculated area of the mouth polygon region to the face area; The face key point average confidence is obtained by performing an arithmetic average on the confidence scores of all face key points, and the face structure coverage index is obtained by performing a weighted summation on the mouth ratio and the face key point average confidence. 4.The traditional Chinese medicine syndrome differentiation auxiliary analysis system based on multi-modal image acquisition of claim 1, characterized in that, The tongue candidate window region activity index is obtained by defining a fixed size tongue candidate window region with the centroid coordinates of the mouth key point set as the center in the face region, performing conversion from the RGB color space to the HSV color space on the pixels in the tongue candidate window region, screening out the pixels with a hue value in a preset tongue color range, dividing the accumulated saturation values of the pixels by the total number of the screened pixels to obtain color saliency, simultaneously calculating the gradient amplitude of all pixels in the window using a Sobel operator and obtaining the average gradient amplitude, and performing a weighted summation on the color saliency index and the average gradient amplitude with a preset weight coefficient to generate the activity index and mark it as the tongue activity index.

5. The traditional Chinese medicine syndrome differentiation auxiliary analysis system based on multi-modal image acquisition according to claim 4, characterized in that, The Sobel operator is used to calculate the gradient amplitude of all pixels in the window, including: calculating the horizontal gradient component and the vertical gradient component of each pixel in the tongue candidate window by the Sobel convolution kernel; square and square root operations are performed on the horizontal and vertical gradient components of each pixel to obtain the edge intensity amplitude of the pixel; the average gradient amplitude is output by accumulating the edge intensity amplitudes of all pixels in the tongue candidate window and dividing by the total number of pixels. 6.The traditional Chinese medicine syndrome differentiation auxiliary analysis system based on multi-modal image acquisition of claim 1, characterized in that, The specific logic of outputting the inapplicability identifier is: the face structure coverage index and the tongue activity index obtained by analysis are respectively normalized to obtain a double-channel structure response vector; and the difference between the face structure coverage index and the tongue activity index in the double-channel structure response vector is calculated; The difference value is compared with a preset threshold τ diff If the difference value is greater than or equal to the preset threshold τ diff , an inapplicability identifier is set; otherwise, an applicability identifier is set, and the default dual-mode acquisition is maintained. 7.The traditional Chinese medicine syndrome differentiation auxiliary analysis system based on multi-modal image acquisition of claim 1, characterized in that, Based on the structure response vector, the normalized edge responses in the face region and the tongue candidate window are fused to generate an original structure intensity map according to the coordinate mapping relationship, including: cropping the face sub-image and the tongue candidate window sub-image from the original frame image, and recording the one-to-one mapping relationship between the pixel coordinates of each sub-image and the coordinates of the original frame image; applying Sobel edge detection to the face sub-image and the tongue candidate window sub-image respectively to obtain horizontal edge response and vertical edge response in sequence; In each sub-image, the horizontal edge response and the vertical edge response are combined into the corresponding edge intensity response graph, and the pixel intensity of the edge intensity response graph is normalized; the processed edge intensity response graph is taken as the normalized edge graph of the face region and the tongue candidate window; Based on the dual-channel structure response vector A = [Mface, Mst], the face region edge intensity response map and the tongue candidate window edge intensity response map are mapped to the original image coordinates (x, y) respectively, and are fused according to the following rules to obtain the original structure intensity map S(x, y): if the original image coordinates then if then if both overlap, then where st represents the tongue candidate window region, Iface represents the face region; the rest of the region is 0, to obtain the complete original structure intensity map S.

8. The traditional Chinese medicine syndrome differentiation auxiliary analysis system based on multi-modal image acquisition according to claim 7, characterized in that, The obtained original structure intensity map is first subjected to row Gaussian smoothing, then subjected to row binarization processing with a preset threshold Ts, and finally subjected to morphological opening and closing operation for denoising and hole filling to obtain a purified binary mask; By performing connected region analysis on the purified binary mask, the mutually connected foreground pixels are divided into independent initial connected region sets by scanning the pixel neighborhood relationship; the bounding box coordinates, the number of connected pixels and the average structure intensity of each initial connected region are calculated, and the modality label is determined according to the overlap rate of the bounding box coordinates of each initial connected region and the tongue candidate window region; if the overlap rate exceeds a preset threshold, the modality label is determined as the modality label; otherwise, the modality label is determined as the face modality label; Small regions with a number of connected pixels less than a preset lower limit are removed, and significant candidates are retained; and the structure intensity average values are sorted in descending order, and the first J items with the largest structure intensity average values are taken as the final connected regions; for each connected region after screening, a structure description vector is generated. 9.The traditional Chinese medicine syndrome differentiation auxiliary analysis system based on multi-modal image acquisition of claim 1, characterized in that, The sorting score acquisition logic is: performing multi-scale neighbor division on all initial focus selectable region sets, regarding each initial focus selectable region as a spatial information granule, adopting a coarse-grained manner based on a neighbor criterion, based on the neighbor criterion, under a plurality of neighbor numbers and neighborhood radii, taking each initial focus selectable region and its description vector as the center, respectively dividing out corresponding upper and lower approximation sets; under each neighbor scale, statistics the average structure response intensity and vector similarity of each initial focus selectable region, and fuses the scale statistics results according to a preset weight, to obtain a multi-scale fusion response and a fusion similarity; The description vector includes a bounding box center corresponding to the initial focus selectable region, a width and height of the bounding box center, an average structure response corresponding to the initial focus selectable region, and a modal label; Importing self-information integrated focus metric to evaluate the importance of each initial focus selectable region to obtain a self-information focus metric value: wherein, represents the local self-information of the initial focus selectable region in the upper and lower approximation sets of the proximate neighborhood; and respectively measure the significant confidence and non-redundancy of the region, and ρ is a probability mass function; ε is a self-information fusion coefficient, is a multi-scale fusion response, is a fusion similarity; the self-information focus metric value, the pixel area normalized value of the initial focus selectable region, and the modality priority are linearly integrated, and multiplied by a preset capacity perception penalty coefficient to obtain a ranking score of each initial focus selectable region. 10.The traditional Chinese medicine syndrome differentiation auxiliary analysis system based on multi-modal image acquisition of claim 1, characterized in that, After executing the focus optimization distribution module, a multi-modal fusion enhancement module is further included for constructing an image acquisition region template on the original preview frame according to the center coordinates and size parameters of the main focus region, to intercept a focus region image of the corresponding modal; The intercepted focus region image and the remaining non-focus region image content are spatially completed and image fused according to the modal labels of the regions, and the enhanced image pair is taken as a feature input for subsequent traditional Chinese medicine syndrome differentiation auxiliary analysis.

Images