Tongue diagnosis and skin diagnosis method and device based on artificial intelligence non-scanning spectral imaging

Abstract

The present application proposes a tongue diagnosis and skin diagnosis method and device based on artificial intelligence non-scanning spectral imaging, including the following steps: using a compressed observation module to collect a subject's compressed observation image to be tested; inputting the compressed observation image and the compressed transfer matrix into an image reconstruction network to obtain hyperspectral image data; inputting the hyperspectral image data into a trained image segmentation network to obtain hyperspectral segmentation image data; inputting the hyperspectral segmentation image data into a trained joint analysis network to obtain symptom information, wherein the symptom information includes symptom areas and symptom description information of each symptom area. Under the premise of ensuring low cost and low power consumption, this solution fully utilizes the advantages of hyperspectral image data, jointly analyzes the spatial distribution characteristics and fine-grained spectral dimensional characteristics of hyperspectral data, and improves the symptom recognition rate and accuracy.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a tongue diagnosis and skin diagnosis method and device based on artificial intelligence non-scanning spectral imaging. Background Art

[0002] Traditional tongue and skin diagnosis instruments rely on the doctor's subjective qualitative judgment, have poor repeatability, and are easily affected by external lighting conditions. Tongue diagnosis analysis based on machine vision helps to objectify diagnostic results and improve diagnostic accuracy. In the existing technology, tongue and skin diagnosis imaging solutions based on machine vision mainly include RGB imaging and hyperspectral imaging. Hyperspectral imaging can be subdivided into scanning and non-scanning solutions according to the different imaging principles of spectral cameras.

[0003] Since the number of image spectral bands collected by the RGB imaging scheme is limited to three broadband bands, it is difficult to reflect some pathological characteristics of human tissues. At the same time, if multiple pathological characteristics are to be inverted from low-dimensional image data, the inversion algorithm will become more complicated. Therefore, when the RGB imaging scheme is applied to tongue diagnosis and skin diagnosis, it will have all the disadvantages of the RGB imaging scheme, because its essence is only to analyze the data of three broadband bands, the spectral resolution is extremely poor, and it basically has no ability to extract symptoms from specific spectral characteristic peaks. For example, the machine learning classification scheme based on the bimodal feature map of publication number CN 117197139 A can only rely on a few color spaces such as RGB images or RGB's nonlinear transformation color space HSV for feature analysis, and its calculation of color temperature features and image moments will blur the spatial distribution features while splitting the image spectrum; the tongue diagnosis detection scheme based on RGB images and convolutional neural networks of publication number CN 117541574 A also only realizes the semantic segmentation of the tongue surface area, lacks other auxiliary diagnosis information of diseases; application number CN 117611581 A's multimodal tongue diagnosis solution only extracts image feature maps through CNN. The feature maps only have spatial distribution information of the image and lack spectral dimensions.

[0004] Traditional scanning spectral cameras need to take thousands of images to obtain complete three-dimensional data, and the imaging speed is slow, and video-level imaging cannot be achieved. At the same time, scanning spectral cameras require large-volume high-precision rotating mechanisms and have high requirements for incident light intensity. The complex mechanical structure and lighting system further increase the system cost and power consumption. Traditional non-scanning spectral cameras often use a multi-lens plus filter solution, which will reduce the spatial resolution while limiting the increase in the number of spectral channels. For medical applications, higher spatial resolution and more spectral channels also mean more objective and accurate diagnosis results. For example, the tongue color classification solution of publication number CN 116849612A directly uses CNN to lose context modeling capabilities to a certain extent, that is, the ability to associate spatial distribution and spectral distribution. At the same time, because it only uses CNN to achieve tongue color classification with a dimension far smaller than that of the hyperspectral dimension for auxiliary diagnosis, this is equivalent to directly performing dimensionality reduction classification processing on the hyperspectral image data, which will not only lose the spectral resolution advantage of the hyperspectral image data over the multispectral image data and even the RGB image, but also cannot further assist in determining the area where the disease is located and the specific type of disease.

[0005] In summary, there is a need for a relatively low-cost, low-power, small-volume, high-efficiency, high-spectral-dimension, high-spatial-resolution hyperspectral camera, while at the same time, a method that can use hyperspectral data in combination with spatial and spectral distribution to perform tongue and skin diagnosis to assist in the diagnosis of diseases. Summary of the invention

[0006] The present application provides a tongue diagnosis and skin diagnosis method and device based on artificial intelligence non-scanning spectral imaging. By jointly analyzing the spatial characteristics and spectral characteristics of hyperspectral images, the advantages of hyperspectral image data are fully utilized to improve the recognition rate and accuracy of tongue and skin diagnosis while ensuring low cost and low power consumption.

[0007] In a first aspect, an embodiment of the present application provides a tongue diagnosis and skin diagnosis method based on artificial intelligence non-scanning spectral imaging, the method comprising:

[0008] The compressed observation image to be tested of the subject is collected by using a compressed observation module, wherein the compressed observation module sequentially performs spatial modulation and dispersion modulation on the image to be tested corresponding to the area to be tested of the subject to obtain a compressed observation image to be tested composed of encoded images to be tested of different wavelengths;

[0009] The compressed observation image and the compressed transfer matrix are input into an image reconstruction network to obtain hyperspectral image data, wherein the compressed transfer matrix is ​​a transfer matrix of a compressed observation module, and the image reconstruction network is obtained by training the image reconstruction network architecture using a spectral image data set composed of known compressed observation images and a compressed transfer matrix;

[0010] Inputting the hyperspectral image data into a trained image segmentation network to obtain hyperspectral segmentation image data, wherein the hyperspectral segmentation image data only includes a diagnosis area; inputting the hyperspectral segmentation image data into a trained joint analysis network to obtain symptom information, wherein the symptom information includes a symptom area and symptom description information of each symptom area;

[0011] The joint analysis network has the same neural network architecture as the image segmentation network, and the neural network architecture includes a first encoding module, a second encoding module, a grid point generation module and a decoding module. The first encoding module encodes the input feature to obtain a first encoding feature, the second encoding module encodes the first encoding feature and the input feature to obtain a second encoding feature, the grid point generation module generates grid points from the input feature to obtain grid point features, and the decoding module decodes the grid point features and the second encoding features; the image segmentation network is obtained by training the neural network architecture using hyperspectral segmentation image data marked with diagnostic areas, wherein the hyperspectral image data is processed into RGB images as input features of the image segmentation network, the joint analysis network is obtained by training the neural network architecture using hyperspectral segmentation image data marked with disease information, and specified symptom text features are added in the grid point generation module of the joint analysis network.

[0012] In a second aspect, the embodiment of the present application provides a tongue diagnosis and skin diagnosis device based on artificial intelligence non-scanning spectral imaging, including:

[0013] An acquisition module collects a compressed observation image to be tested of a subject using a compressed observation module, wherein the compressed observation module sequentially performs spatial modulation and dispersion modulation on the image to be tested corresponding to the area to be tested of the subject to obtain a compressed observation image to be tested composed of encoded images to be tested of different wavelengths;

[0014] A reconstruction module, used for inputting the compressed observation image and the compressed transfer matrix into an image reconstruction network to obtain hyperspectral image data, wherein the compressed transfer matrix is ​​a transfer matrix of the compressed observation module, and the image reconstruction network is obtained by training the image reconstruction network architecture with a spectral image data set composed of known compressed observation images and a compressed transfer matrix;

[0015] The segmentation diagnosis module is used to input the hyperspectral image data into the trained image segmentation network to obtain hyperspectral segmentation image data, wherein the hyperspectral segmentation image data only includes the diagnosis area; and input the hyperspectral segmentation image data into the trained joint analysis network to obtain disease information, wherein the disease information includes the disease area and the disease description information of each disease area.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform a tongue diagnosis and skin diagnosis method based on artificial intelligence non-scanning spectral imaging.

[0017] In a fourth aspect, an embodiment of the present application provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute a process, and the process includes a tongue diagnosis and skin diagnosis method based on artificial intelligence non-scanning spectral imaging.

[0018] The main contributions and innovations of the present invention are as follows:

[0019] The compression observation module in this scheme obtains the compressed observation image by spatially modulating and dispersion modulating the image to be measured and then using the detector array to obtain the dispersion image by single exposure, which requires low cost and low power consumption;

[0020] This solution uses an image reconstruction network to quickly reconstruct images based on compressed observation images. By pre-training the image reconstruction network in advance, the speed of reconstructing compressed observation images can be achieved in milliseconds during the use phase, and the size of the reconstructed image is controlled by compressing the transmission matrix, which is convenient for subsequent segmentation and analysis.

[0021] This scheme further improves the efficiency of disease judgment by constructing a segmented network to first segment the hyperspectral image data to obtain hyperspectral segmentation image data, and then analyzes the obtained hyperspectral segmentation image data. In the joint analysis network, multi-head self-attention is assigned to each spectral band to identify the disease area and the disease description information of each disease area, making full use of the advantages of hyperspectral image data, jointly analyzing the spatial distribution characteristics and fine-grained spectral dimension characteristics of hyperspectral data, and improving the disease recognition rate and accuracy.

[0022] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0024] Figure 1 It is a flow chart of a tongue diagnosis and skin diagnosis method based on artificial intelligence non-scanning spectral imaging according to an embodiment of the present application;

[0025] Figure 2It is a structural diagram of the compression observation module of this scheme;

[0026] Figure 3 This is a schematic diagram of the compression principle of the compression observation module of this scheme;

[0027] Figure 4 This is the training flow chart of the image reconstruction network of this scheme;

[0028] Figure 5 This is the flow chart of the image reconstruction network of this scheme when reconstructing the image;

[0029] Figure 6 This is the overall flow chart of the scheme for segmenting the hyperspectral image data and analyzing symptoms;

[0030] Figure 7 It is a structural block diagram of a tongue diagnosis and skin diagnosis device based on artificial intelligence non-scanning spectral imaging according to an embodiment of the present application;

[0031] Figure 8 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0033] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0034] Embodiment 1

[0035] The embodiment of the present application provides a tongue diagnosis and skin diagnosis method based on artificial intelligence non-scanning spectral imaging, which can make full use of the advantages of hyperspectral image data, jointly analyze the spatial distribution characteristics and fine-grained spectral dimension characteristics of hyperspectral data, and improve the disease recognition rate and accuracy while ensuring low cost, low power consumption and small size. Specifically, reference Figure 1 , the method comprising:

[0036] The compressed observation image to be tested of the subject is collected by using a compressed observation module, wherein the compressed observation module sequentially performs spatial modulation and dispersion modulation on the image to be tested corresponding to the area to be tested of the subject to obtain a compressed observation image to be tested composed of encoded images to be tested of different wavelengths;

[0037] The compressed observation image and the compressed transfer matrix are input into an image reconstruction network to obtain hyperspectral image data, wherein the compressed transfer matrix is ​​a transfer matrix of a compressed observation module, and the image reconstruction network is obtained by training the image reconstruction network architecture using a spectral image data set composed of known compressed observation images and a compressed transfer matrix;

[0038] Inputting the hyperspectral image data into a trained image segmentation network to obtain hyperspectral segmentation image data, wherein the hyperspectral segmentation image data only includes a diagnosis area; inputting the hyperspectral segmentation image data into a trained joint analysis network to obtain symptom information, wherein the symptom information includes a symptom area and symptom description information of each symptom area;

[0039] The joint analysis network has the same neural network architecture as the image segmentation network, and the neural network architecture includes a first encoding module, a second encoding module, a grid point generation module and a decoding module. The first encoding module encodes the input feature to obtain a first encoding feature, the second encoding module encodes the first encoding feature and the input feature to obtain a second encoding feature, the grid point generation module generates grid points from the input feature to obtain grid point features, and the decoding module decodes the grid point features and the second encoding features; the image segmentation network is obtained by training the neural network architecture using hyperspectral segmentation image data marked with diagnostic areas, wherein the hyperspectral image data is processed into RGB images as input features of the image segmentation network, the joint analysis network is obtained by training the neural network architecture using hyperspectral segmentation image data marked with disease information, and specified symptom text features are added in the grid point generation module of the joint analysis network.

[0040] In this solution, the structure of the compression observation module is as follows: Figure 2 As shown, the image to be tested is an image containing the tongue or skin, and the compression observation module includes a first lens, a random mask, a second lens, a dispersion element, a third lens and a detector array arranged along the optical path, wherein the first lens images the image to be tested on the random mask surface to obtain a random mask surface image, the second lens images the random mask surface image on the dispersion element to obtain a dispersion image, and the third lens images the dispersion image on different areas of the detector array to obtain a compression observation image to be tested.

[0041] Specifically, the method for acquiring the image to be tested is: acquiring the area to be tested, using the lighting system to illuminate the area to be tested, and then imaging the tongue or skin to obtain the image to be tested, wherein the lighting system should uniformly illuminate the area to be tested. Preferably, only the light source of the lighting system should illuminate the area to be tested to exclude environmental stray light and avoid introducing additional noise.

[0042] Specifically, the random mask is used to spatially modulate the image to be measured to form spatial coding, the first lens is used to optically align the object to be measured and image it onto the random mask surface, the second lens is used to optically align the image on the random mask surface and image it onto the dispersion element with parallel light, the third lens is used to optically align the dispersion image and image it onto different areas of the detector array, the dispersion element is used to split the wide-band light beam of the random mask surface image and disperse-modulate the light beams of different wavelengths respectively so that the encoded images of the object to be measured of different wavelengths can be imaged on different lateral areas of the detector array to achieve compressed observation, and the detector array is used to collect compressed observation images.

[0043] Furthermore, all optoelectronic devices in the compression observation module have no restrictions on the wavelength range of use, and a matching working wavelength should be selected according to the application scenario.

[0044] In some embodiments, the compression principle diagram of the compression observation module is as follows: Figure 3 As shown, the spectral range of the image to be measured is expressed as [λ min ,λ max ], the three-dimensional light field of the image to be tested can be expressed as s(x, y, λ), s(x, y, λ) represents the light field intensity at wavelength λ and spatial position (x, y), the transfer function of the random mask is h(x, y), then the three-dimensional light field s of the image to be tested is modulated by the random mask with the transfer function h. f (x, y, λ) is represented by the following formula:

[0045] S f (x,y,λ)=h(x,y)s(x,y,λ)

[0046] The random mask is composed of randomly distributed 0 and 1 pixels, 0 represents that light is completely transmitted, and 1 represents that light is not transmitted. The random mask modulates different wavelengths and different spaces of the image to be tested to ensure that spectral images of different wavelengths are independent of each other and are used for subsequent reconstruction.

[0047] The dispersion element will perform dispersion modulation on the random mask surface image, so that light fields of different wavelengths will have different offsets on a specific axis. Assuming that the axis where the offset occurs is the y-axis, the dispersion image after dispersion on the detector array is expressed by the following formula:

[0048] S shift (x,y,λ)=S f (x,y+d(λ),λ)

[0049] Among them, S shift (x, y, λ) represents the dispersion image of wavelength λ and spatial position (x, y), and d(λ) represents the dispersion offset produced by the dispersion element in the detector array plane.

[0050] A pixel space coordinate system (u, v) is introduced into the detector array, where u represents the row of pixels and v represents the column of pixels. On the plane of the detector array, the photocurrent intensity that can be collected by each pixel can be expressed as follows:

[0051]

[0052] Where R(x, y, λ) represents the responsivity of different regions of the detector array to different wavelengths, and the number of pixels in the spatial dimension x and y is defined as N. x , N y , spectral range [λ min ,λ max ] contains N λ channels, and the three-dimensional spectrum of the object to be tested is represented as a vector Then Y(u, v) can be expressed as a vector The compressed observation image collected by the compression observation module can be expressed using the following formula:

[0053] y=Hs

[0054] in, is the compressed transfer matrix, which is obtained by the calibration process in actual use, and y is the compressed observation image. In practical applications, due to the existence of some system noise Therefore, the compressed observation image actually obtained degenerates into:

[0055] y=Hs+g

[0056] That is to say, this scheme controls the size of the compressed observation image through the compression transfer matrix, and the complete compressed observation image can be obtained by multiplying the image processed by the compression observation module by the compression transfer matrix and adding the system noise.

[0057] In summary, the compressed observation module in this scheme can collect compressed observation images through a single exposure without the need for a scanning mechanism, so it can be collected at the video frame rate. Moreover, since the natural spectrum image signal has sparse characteristics, according to the compressed sensing theory, in N λ>>1, even if there is system noise, the signal can still be recovered from the compressed observation image, so the compressed observation image will be sent to the image reconstruction network for spectral reconstruction.

[0058] Before the step of "inputting the compressed observation image and the compressed transfer matrix into an image reconstruction network to obtain hyperspectral image data", the compressed observation image is subjected to denoising, grayscale equalization and cropping in sequence.

[0059] Specifically, the purpose of performing preprocessing such as denoising, grayscale equalization and cropping on the compressed observation image in sequence is to match the input data requirements of the image reconstruction network.

[0060] In the step of "the image reconstruction network is obtained by training the image reconstruction network architecture using a spectral image data set composed of known compressed observation images and a compressed transfer matrix", the training flow chart of the image reconstruction network is as follows: Figure 4 As shown, the known compressed observation images in the spectral image data set are matched with the system band, and each compressed spectral image is combined with the compressed transfer matrix and input into the image reconstruction network architecture to obtain a training reconstructed spectral image, and the loss value of each training reconstructed spectral image and the corresponding spectral image is calculated. The image reconstruction network architecture is back-propagated according to the loss value and the parameters of the image reconstruction network architecture are updated. When the loss value is less than the set threshold, the training is stopped to obtain the image reconstruction network.

[0061] In this solution, the image reconstruction network is responsible for reconstructing the image to be measured from the compressed observation image using algorithms and computing equipment to complete the hyperspectral medical image acquisition.

[0062] Specifically, when training the image reconstruction network architecture, a training data set consisting of compressed spectral images matching the system channel is prepared. Where n represents the batch of training data, using The training data in is combined with the compressed transmission matrix H to obtain the compressed training data y n , and then H and y n Input into the image reconstruction network architecture for forward propagation operation to obtain the training reconstructed spectral image s n , and then construct the loss function calculation With s n The loss value is obtained, and the back propagation operation is performed on the image reconstruction network architecture according to the loss value and the parameters of the image reconstruction network architecture are updated. After the parameters of the image reconstruction network architecture are updated, the next batch of calculations are performed until the loss value is less than the set threshold. The training is stopped, and the parameters of the current image reconstruction network architecture are saved to obtain the image reconstruction network.

[0063] Specifically, the flowchart of the image reconstruction network when reconstructing an image is as follows: Figure 5 shown by Figure 5 It can be seen that in actual application, the image reconstruction network in this scheme only needs to input the compressed observation image and the compressed transfer matrix into the image reconstruction network to obtain the hyperspectral image data, without the need to perform loss calculation and other operations, so the time consumed in the actual application stage is usually in milliseconds.

[0064] In this solution, the diagnosis area is the tongue or skin area, that is, the image segmentation network is used to segment the tongue or skin area in the entire image, while excluding invalid information such as background in the image, so as to facilitate subsequent targeted analysis.

[0065] In this solution, the overall flow chart of segmenting and analyzing the hyperspectral image data is as follows: Figure 6 As shown, the image segmentation network and the joint analysis network adopt the same neural network architecture, and the image segmentation network and the joint analysis network are explained below.

[0066] The neural network architecture is constructed based on the pre-trained Open-Vocabulary SAM model. Since the hyperspectral image data may contain information about objects other than the tongue / skin, an image segmentation network is required to filter out parts that are not relevant to the diagnosis.

[0067] In this scheme, the first encoding module is composed of a SAM encoder and a SAM2CLIP encoder connected in series in sequence, and the second encoding module is composed of a CLIP encoder and a CLIP2SAM encoder connected in series in sequence, the SAM encoder encodes the input features to obtain a SAM encoding feature map, the CLIP encoder encodes the input features to obtain a CLIP encoding feature sub-map, the SAM2CLIP encoder converts the SAM encoding feature map into a first encoding feature in the form of a feature pyramid of the same dimension as the CLIP encoding feature sub-map and inputs it into the CLIP encoder, the CLIP encoder learns the parameters of the SAM encoder by knowledge distillation based on the first encoding feature, and inputs the first encoding feature and the CLIP encoding feature sub-map into the CLIP2SAM encoder, and the CLIP2SAM encoder aligns the first encoding feature with the CLIP encoding feature sub-map to obtain a second encoding feature.

[0068] Specifically, the region of interest in this solution is the tongue or skin area.

[0069] Specifically, the SAM encoder is trained using a large dataset with mask labels, making it highly adaptable to a wide range of downstream tasks through interactive prompts. The CLIP encoder obtains unprecedented capabilities in zero-shot visual recognition by training billions of text-image pairs. The SAM2CLIP encoder transfers the knowledge of the SAM encoder to the CLIP encoder through distillation and a learnable Transformer Adapter. The CLIP2SAM encoder transfers the knowledge of the CLIP encoder to the SAM encoder, thereby enhancing its recognition capabilities.

[0070] Specifically, the input feature of the image segmentation network is to transform the hyperspectral image data s n Perform RGB synthesis and reduce dimension The resulting RGB image.

[0071] Specifically, the SAM encoder inputs feature s RGB Encoded as the first encoding feature and input to the SAM2CLIP encoder, where N d is the number of channels of the first encoding feature, the first encoding feature is in the form of a feature map, and the CLIP encoder converts s RGB Encoded as CLIP encoding feature E I .

[0072] Specifically, the SAM2CLIP encoder converts the first encoding feature F SAM Convert to CLIP encoding feature sub I Feature pyramid of the same dimension And input to the CLIP encoder, which is based on the feature pyramid The parameters of the SAM encoder are learned by knowledge distillation, and the formula of its distillation loss is as follows:

[0073]

[0074] Among them, MSE represents the mean square error calculated pixel by pixel, and Fusion represents the feature image pyramid fusion process. During the training process of the image segmentation network, the distillation error will be optimized to the minimum to minimize the feature conversion error.

[0075] Specifically, the CLIP2SAM encoder is built based on the multi-scale feature extraction network of R-CNN. The CLIP2SAM encoder aligns the first encoded feature with the CLIP encoded feature sub-point of interest and outputs the classification result Q of the interest area. label And the corresponding image features to the decoding module.

[0076] In this scheme, the grid point generation module includes a grid point generation layer and a grid point sparse layer. The grid point generation layer rasterizes the input feature according to the size of the input feature to obtain the focus point grid feature, and the grid point sparse layer rasterizes the focus point grid feature to obtain the grid point feature.

[0077] Specifically, the grid generation layer will be based on the input feature s RGB The size and spacing are rasterized to obtain the grid features of the points of interest Where N G is the grid point spacing, the grid sparse layer is a PromptDecoder structure, and the grid sparse layer converts G ROI Encoded as grid point feature Q sp .

[0078] In this solution, the decoding module is a SAM decoder, which decodes and generates a segmentation mask according to the grid point feature and the second coding feature, performs non-maximum suppression on the segmentation mask and then performs feature multiplication with the hyperspectral image data to obtain hyperspectral segmentation image data, wherein the hyperspectral segmentation image data only contains the tongue or skin area. Furthermore, the segmentation mask is subjected to non-maximum suppression, which has the advantage of filtering out noise in the segmentation mask and containing less information in the area.

[0079] In this scheme, an attention head is assigned to each spectral band in the hyperspectral segmentation image data in the SAM encoder of the first encoding module of the joint analysis network, and the attention value of each attention head is obtained by combining Q, K, and V by dimensionality reduction of the hyperspectral segmentation image data, and the attention results of each attention head are aggregated and the spatial position encoding information output is fused in a linear mapping manner.

[0080] In this scheme, an attention head is assigned to each spectral band in the hyperspectral segmentation image data in the CLIP encoder of the second encoding module of the joint analysis network, and the attention value of each attention head is obtained by combining Q, K, and V by dimensionality reduction of the hyperspectral segmentation image data, and the attention results of each attention head are aggregated and the spatial position encoding information output is fused in a linear mapping manner.

[0081] Take the example of assigning an attention head to each spectral band in the hyperspectral segmentation image data in the SAM encoder of the first encoding module of the joint analysis network:

[0082] Specifically, the hyperspectral segmentation image data is reduced in dimension by downsampling in the SAM encoder, and combined with the query vector of the SAM encoder key vector and the value vector Get the attention value of each attention head. The formula is as follows:

[0083]

[0084] in, are all learnable parameters, downsample(·) represents downsampling and dimensionality reduction. Dimensionality reduction Then O′K′V′ is divided into To assign attention values ​​to each attention head, where N c is the number of attention heads, so the dimension of each attention head can be expressed as Then the attention value formula of each attention head is expressed as:

[0085]

[0086] Where T represents the matrix transpose, σ j ∈R 1 In order to learn parameters to distinguish the importance of spectral information in different bands, the attention value of each band is finally aligned with the feature to output the multi-head attention result. The multi-head attention result is linearly mapped and fused with the spatial position encoding information to obtain the analysis encoding feature map. The formula is as follows:

[0087]

[0088] Among them, concate is aggregation, represents the learnable parameters, f p (·) represents spatial position encoding information, which usually includes a convolution layer, an activation layer, and a reshape operation.

[0089] The self-attention calculation method of assigning an attention head to each spectral band in the hyperspectral segmentation image data in the CLIP encoder of the second encoding module of the joint analysis network is the same as that of the SAM encoder and will not be repeated.

[0090] In this scheme, the doctor's preliminary rough judgment symptom words for different tongues or skins are obtained as prompts and input into the grid sparse layer of the joint analysis network grid point generation module to obtain the specified symptom text features, and the specified symptom text features are input together with the grid point features into the decoding module of the joint analysis network.

[0091] In this scheme, a joint analysis network predicts the disease based on the input tongue or skin image.

[0092] In this scheme, the output of the decoding module of the joint analysis network is the disease area mask and the disease description information of the disease area. The disease area mask is feature multiplied by the hyperspectral segmentation image number to obtain the disease area, and then the disease area and the disease description information are aligned to obtain the disease information.

[0093] In this solution, the algorithm flow of the joint analysis network and the image segmentation network is the same. It is worth mentioning that when training the image segmentation network and the joint analysis network, since the two focus on different contents, the annotation of the training data and the design of the loss function also need to be changed according to the goal. For example, the image segmentation network is more concerned with whether the tongue or skin can be distinguished from the surrounding environment, background and other interfering images, and the joint analysis network is more concerned with whether specific symptoms can be extracted from the tongue or skin area. In theory, since the framework of the image segmentation network and the joint analysis network is similar, the two can be integrated into the same set of algorithm flows. Here, the functions and flows of each algorithm are explained in more detail. However, it should be noted that if the two algorithms are combined, it is necessary to appropriately improve the dimensions of the feature map and the feature sub-dimensions and the annotation accuracy of the training data, and the loss function is designed in stages to ensure that there will be no data underfitting when the segmentation function and the symptom extraction function are realized. In summary, the artificial intelligence-based auxiliary diagnosis and analysis algorithm proposed in the present invention can not only analyze the symptom characteristics in the spatial dimension by utilizing ViT, but also make full use of the spectral dimension information to further improve the diagnostic accuracy.

[0094] Embodiment 2

[0095] Based on the same idea, refer to Figure 7 The present application also proposes a tongue diagnosis and skin diagnosis device based on artificial intelligence non-scanning spectral imaging, comprising:

[0096] An acquisition module collects a compressed observation image to be tested of a subject using a compressed observation module, wherein the compressed observation module sequentially performs spatial modulation and dispersion modulation on the image to be tested corresponding to the area to be tested of the subject to obtain a compressed observation image to be tested composed of encoded images to be tested of different wavelengths;

[0097] A reconstruction module, used for inputting the compressed observation image and the compressed transfer matrix into an image reconstruction network to obtain hyperspectral image data, wherein the compressed transfer matrix is ​​a transfer matrix of the compressed observation module, and the image reconstruction network is obtained by training the image reconstruction network architecture with a spectral image data set composed of known compressed observation images and a compressed transfer matrix;

[0098] The segmentation diagnosis module is used to input the hyperspectral image data into a trained image segmentation network to obtain hyperspectral segmentation image data, wherein the hyperspectral segmentation image data only includes a diagnosis area; and input the hyperspectral segmentation image data into a trained joint analysis network to obtain symptom information, wherein the symptom information includes a symptom area and symptom description information of each symptom area.

[0099] Embodiment 3

[0100] This embodiment also provides an electronic device, referring to Figure 8 , comprises a memory 404 and a processor 402, wherein the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to execute the steps in any of the above method embodiments.

[0101] Specifically, the processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0102] Among them, the memory 404 may include a large capacity memory 404 for data or instructions. By way of example and not limitation, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 404 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 404 may be inside or outside the data processing device. In a specific embodiment, the memory 404 is a non-volatile memory. In a specific embodiment, the memory 404 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable Programmable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (Static Random-Access Memory, abbreviated as SRAM) or a dynamic random access memory (Dynamic Random Access Memory, abbreviated as DRAM), wherein the DRAM can be a fast page mode dynamic random access memory 404 (Fast Page Mode Dynamic Random Access Memory, abbreviated as FPMDRAM), an extended data output dynamic random access memory (Extended Date Out Dynamic Random Access Memory, abbreviated as EDODRAM), a synchronous dynamic random access memory (Synchronous Dynamic Random-Access Memory, abbreviated as SDRAM), etc.

[0103] The memory 404 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402 .

[0104] The processor 402 reads and executes the computer program instructions stored in the memory 404 to implement any one of the tongue diagnosis and skin diagnosis methods based on artificial intelligence non-scanning spectral imaging in the above embodiments.

[0105] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408 , wherein the transmission device 406 is connected to the processor 402 , and the input / output device 408 is connected to the processor 402 .

[0106] The transmission device 406 can be used to receive or send data via a network. The specific examples of the above-mentioned network may include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0107] The input / output device 408 is used to input or output information. In this embodiment, the input information may be an image to be tested, etc., and the output information may be a symptom region and symptom description information of each symptom region, etc.

[0108] Optionally, in this embodiment, the processor 402 may be configured to perform the following steps through a computer program:

[0109] S101, using a compression observation module to collect a compression observation image to be tested of a subject, wherein the compression observation module sequentially performs spatial modulation and dispersion modulation on the image to be tested corresponding to the area to be tested of the subject to obtain a compression observation image to be tested composed of encoded images to be tested of different wavelengths;

[0110] S102, inputting the compressed observation image and the compressed transfer matrix into an image reconstruction network to obtain hyperspectral image data, wherein the compressed transfer matrix is ​​a transfer matrix of a compressed observation module, and the image reconstruction network is obtained by training the image reconstruction network architecture using a spectral image data set composed of known compressed observation images and a compressed transfer matrix;

[0111] S103, inputting the hyperspectral image data into the trained image segmentation network to obtain hyperspectral segmentation image data, wherein the hyperspectral segmentation image data only includes the diagnosis area; inputting the hyperspectral segmentation image data into the trained joint analysis network to obtain symptom information, wherein the symptom information includes the symptom area and symptom description information of each symptom area;

[0112] S104. The joint analysis network has the same neural network architecture as the image segmentation network, and the neural network architecture includes a first encoding module, a second encoding module, a grid point generation module and a decoding module. The first encoding module encodes the input feature to obtain a first encoding feature, the second encoding module encodes the first encoding feature and the input feature to obtain a second encoding feature, the grid point generation module generates grid points from the input feature to obtain grid point features, and the decoding module decodes the grid point features and the second encoding features; the neural network architecture is trained using the hyperspectral segmentation image data marked with the diagnosis area to obtain the image segmentation network, wherein the hyperspectral image data is processed into an RGB image as the input feature of the image segmentation network, the neural network architecture is trained using the hyperspectral segmentation image data marked with disease information to obtain the joint analysis network, and the specified symptom text features are added in the grid point generation module of the joint analysis network.

[0113] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0114] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0115] Embodiments of the present invention may be implemented by computer software that is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros may be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer executable components configured to perform an embodiment when the program is run. One or more computer executable components may be at least one software code or a portion thereof. In addition, at this point, it should be noted that, for example, Figure 8 Any block of the logic flow in the program may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media such as memory chips or storage blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs, etc. Physical media are non-transitory media.

[0116] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0117] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A tongue diagnosis and skin diagnosis method based on artificial intelligence non-scanning spectral imaging, characterized in that: The following steps are involved: The compressed observation image to be tested of the subject is collected by using a compressed observation module, wherein the compressed observation module sequentially performs spatial modulation and dispersion modulation on the image to be tested corresponding to the area to be tested of the subject to obtain a compressed observation image to be tested composed of encoded images to be tested of different wavelengths; The compressed observation image to be tested and the compressed transmission matrix are input into the image reconstruction network to obtain hyperspectral image data, wherein the compressed transmission matrix is ​​the transmission matrix of the compressed observation module, and the image reconstruction network is obtained by training the image reconstruction network architecture with a spectral image data set composed of the known compressed observation image to be tested and the compressed transmission matrix; Inputting the hyperspectral image data into a trained image segmentation network to obtain hyperspectral segmentation image data, wherein the hyperspectral segmentation image data only includes a diagnosis area; inputting the hyperspectral segmentation image data into a trained joint analysis network to obtain symptom information, wherein the symptom information includes a symptom area and symptom description information of each symptom area; The joint analysis network has the same neural network architecture as the image segmentation network, and the neural network architecture includes a first encoding module, a second encoding module, a grid point generation module and a decoding module. The first encoding module encodes the input feature to obtain a first encoding feature, the second encoding module encodes the first encoding feature and the input feature to obtain a second encoding feature, the grid point generation module generates grid points from the input feature to obtain grid point features, and the decoding module decodes the grid point features and the second encoding features; the image segmentation network is obtained by training the neural network architecture using hyperspectral segmentation image data marked with diagnostic areas, wherein the hyperspectral image data is processed into RGB images as input features of the image segmentation network, the joint analysis network is obtained by training the neural network architecture using hyperspectral segmentation image data marked with disease information, and specified symptom text features are added in the grid point generation module of the joint analysis network.

2. The tongue diagnosis and skin diagnosis method based on artificial intelligence non-scanning spectral imaging according to claim 1 is characterized in that: The compression observation module includes a first lens, a random mask, a second lens, a dispersion element, a third lens and a detector array arranged along the optical path, wherein the first lens images the image to be measured on the random mask surface to obtain a random mask surface image, the second lens images the random mask surface image on the dispersion element to obtain a dispersion image, and the third lens images the dispersion image on different areas of the detector array to obtain a compression observation image to be measured.

3. The tongue diagnosis and skin diagnosis method based on artificial intelligence non-scanning spectral imaging according to claim 1 is characterized in that: In the step of "the image reconstruction network is obtained by training the image reconstruction network architecture using a spectral image data set consisting of known compressed observation images to be tested and a compression transfer matrix", the known compressed observation images to be tested in the spectral image data set are matched with the system band, each compressed spectral image is input into the image reconstruction network architecture in combination with the compression transfer matrix to obtain a training reconstructed spectral image, a loss value between each training reconstructed spectral image and the corresponding spectral image is calculated, a back propagation operation is performed on the image reconstruction network architecture based on the loss value and the parameters of the image reconstruction network architecture are updated, and the training is stopped when the loss value is less than a set threshold to obtain the image reconstruction network.

4. The tongue diagnosis and skin diagnosis method based on artificial intelligence non-scanning spectral imaging according to claim 1 is characterized in that: The first encoding module is composed of a SAM encoder and a SAM2CLIP encoder connected in series in sequence, and the second encoding module is composed of a CLIP encoder and a CLIP2SAM encoder connected in series in sequence. The SAM encoder encodes the input feature to obtain a SAM encoding feature map, and the CLIP encoder encodes the input feature to obtain a CLIP encoding feature sub-map. The SAM2CLIP encoder converts the SAM encoding feature map into a first encoding feature in the form of a feature pyramid with the same dimension as the CLIP encoding feature sub-map, and inputs the first encoding feature into the CLIP encoder. The CLIP encoder learns the parameters of the SAM encoder in a knowledge distillation manner based on the first encoding feature, and inputs the first encoding feature and the CLIP encoding feature sub-map into the CLIP2SAM encoder. The CLIP2SAM encoder aligns the first encoding feature with the CLIP encoding feature sub-map for interest points to obtain a second encoding feature. The grid point generation module includes a grid point generation layer and a grid point sparse layer. The grid point generation layer performs gridization according to the size of the input feature to obtain a focus point grid feature, and the grid point sparse layer performs grid sparseization on the focus point grid feature to obtain a grid point feature.

5. The tongue diagnosis and skin diagnosis method based on artificial intelligence non-scanning spectral imaging according to claim 1 is characterized in that: The decoding module is a SAM decoder, which decodes and generates a segmentation mask according to the grid point features and the second encoding features, performs non-maximum suppression on the segmentation mask, and then performs feature multiplication with the hyperspectral image data to obtain hyperspectral segmentation image data, wherein the hyperspectral segmentation image data only contains the tongue or skin area.

6. The tongue diagnosis and skin diagnosis method based on artificial intelligence non-scanning spectral imaging according to claim 1 is characterized in that: In the SAM encoder of the first encoding module of the joint analysis network, an attention head is assigned to each spectral band in the hyperspectral segmentation image data, and the attention value of each attention head is obtained by combining Q, K, and V by performing dimensionality reduction on the hyperspectral segmentation image data, the attention results of each attention head are aggregated, and the spatial position coding information output is fused in a linear mapping manner. In the CLIP encoder of the second encoding module of the joint analysis network, an attention head is assigned to each spectral band in the hyperspectral segmentation image data, and the attention value of each attention head is obtained by performing dimensionality reduction on the hyperspectral segmentation image data, combining Q, K, and V, the attention results of each attention head are aggregated, and the spatial position coding information output is fused in a linear mapping manner, wherein Q is the query vector in the attention mechanism, K is the key vector in the attention mechanism, and V is the value vector in the attention mechanism.

7. The tongue diagnosis and skin diagnosis method based on artificial intelligence non-scanning spectral imaging according to claim 1 is characterized in that: The doctor's preliminary rough judgment symptom words for different tongues or skins are obtained as prompts and input into the grid sparse layer of the joint analysis network grid point generation module to obtain the specified symptom text features, and the specified symptom text features and the grid point features are input into the decoding module of the joint analysis network together. The output of the decoding module of the joint analysis network is the disease area mask and the disease description information of the disease area. The disease area mask is feature multiplied by the hyperspectral segmentation image number to obtain the disease area, and then the disease area and the disease description information are aligned to obtain the disease information.

8. A tongue diagnosis and skin diagnosis device based on artificial intelligence non-scanning spectral imaging, characterized in that: include: An acquisition module collects a compressed observation image to be tested of a subject using a compressed observation module, wherein the compressed observation module sequentially performs spatial modulation and dispersion modulation on the image to be tested corresponding to the area to be tested of the subject to obtain a compressed observation image to be tested composed of encoded images to be tested of different wavelengths; A reconstruction module, used for inputting the compressed observation image to be tested and the compression transfer matrix into an image reconstruction network to obtain hyperspectral image data, wherein the compression transfer matrix is ​​a transfer matrix of the compression observation module, and the image reconstruction network is obtained by training the image reconstruction network architecture with a spectral image data set composed of the known compressed observation image to be tested and the compression transfer matrix; A segmentation diagnosis module is used to input the hyperspectral image data into a trained image segmentation network to obtain hyperspectral segmentation image data, wherein the hyperspectral segmentation image data only includes a diagnosis area; input the hyperspectral segmentation image data into a trained joint analysis network to obtain symptom information, wherein the symptom information includes a symptom area and symptom description information of each symptom area; the joint analysis network has the same neural network architecture as the image segmentation network, and the neural network architecture includes a first encoding module, a second encoding module, a grid point generation module and a decoding module; the first encoding module encodes the input feature to obtain a first encoding feature, the second encoding module encodes the first encoding feature and the input feature to obtain a second encoding feature, the grid point generation module generates grid points for the input feature to obtain grid point features, and the decoding module decodes the grid point features and the second encoding features; the image segmentation network is obtained by training the neural network architecture using the hyperspectral segmentation image data marked with a diagnosis area, wherein the hyperspectral image data is processed into an RGB image as an input feature of the image segmentation network, the hyperspectral segmentation image data marked with symptom information is used to train the neural network architecture to obtain a joint analysis network, and the specified symptom text features are added to the grid point generation module of the joint analysis network.

9. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to run the computer program to execute the tongue diagnosis and skin diagnosis method based on artificial intelligence non-scanning spectral imaging as described in any one of claims 1-7.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes a program code for controlling a process to execute a process, and the process includes a tongue diagnosis and skin diagnosis method based on artificial intelligence non-scanning spectral imaging according to any one of claims 1-7.

Images