Postprandial blood sugar trend prediction method and device based on multi-modal fusion

Abstract

The invention discloses a postprandial blood sugar trend prediction method and device based on multi-modal fusion, and relates to the field of blood sugar monitoring based on artificial intelligence, and the method comprises the steps: obtaining a preprandial blood sugar curve and a preprandial and postprandial dining picture of a user; the trained image encoder is utilized to extract image features of dining pictures before and after meals; according to the image features of the pre-meal and post-meal dining pictures, selecting a blood glucose curve most related to the pre-meal and post-meal dining pictures from a blood glucose curve database to obtain a reference blood glucose curve; respectively inputting the preprandial blood glucose curve and the reference blood glucose curve into the trained time sequence encoder to obtain preprandial blood glucose curve characteristics and reference blood glucose curve characteristics; and inputting the preprandial blood glucose curve characteristics and the reference blood glucose curve characteristics into the trained time sequence decoder to obtain a postprandial blood glucose curve prediction result. According to the method, the two modes of the pre-meal blood glucose curve and the pre-meal and post-meal image of the user are fused, and the post-meal blood glucose curve trend of the user is accurately predicted.

Description

Technical Field

[0001] The present application relates to the field of blood glucose monitoring based on artificial intelligence, and particularly to a method and device for predicting the postprandial blood glucose trend based on multimodal fusion. Background Art

[0002] Globally, the incidence of diabetes has been continuously rising and has become a major public health problem seriously threatening human health. Fasting blood glucose plays an important warning role in the early diagnosis of diabetes. In the stage of latent disease development, many patients have no typical symptoms, and the increase in fasting blood glucose may be the only clue. For diabetic patients, relying solely on fasting blood glucose monitoring far from comprehensively reflects the true situation of their blood glucose control. Postprandial blood glucose, as an important part of the blood glucose spectrum, plays an extremely important role in the occurrence and development of diabetes. Therefore, in-depth exploration of blood glucose monitoring, especially the optimization of postprandial blood glucose monitoring techniques and strategies, has extremely important scientific significance and social value. It can provide key data support for clinicians to formulate personalized treatment plans, help patients achieve self-blood glucose management, and ultimately achieve the goals of improving the blood glucose control level of diabetic patients, reducing the risk of complications, and reducing the medical burden related to diabetes, and promoting the further development of the field of diabetes prevention and treatment. Summary of the Invention

[0003] The purpose of the present application is to provide a method and device for predicting the postprandial blood glucose trend based on multimodal fusion, which fuse two modalities of the user's pre-meal blood glucose curve and pre- and post-meal images to accurately predict the trend of the user's post-meal blood glucose curve.

[0004] To achieve the above purpose, the present application provides the following solutions:

[0005] In a first aspect, the present application provides a method for predicting the postprandial blood glucose trend based on multimodal fusion, including:

[0006] Obtain the user's pre-meal blood glucose curve and pre- and post-meal dining pictures;

[0007] Extract the image features of the pre- and post-meal dining pictures by using a trained image encoder;

[0008] According to the image features of the pre- and post-meal dining pictures, select the blood glucose curve most relevant to the pre- and post-meal dining pictures from the blood glucose curve database to obtain a reference blood glucose curve;

[0009] Input the pre-meal blood glucose curve into a trained time series encoder to obtain pre-meal blood glucose curve features;

[0010] Input the reference blood glucose curve into a trained time series encoder to obtain reference blood glucose curve features;

[0011] Input the pre-meal blood glucose curve features and the reference blood glucose curve features into the trained time series decoder to obtain the predicted result of the post-meal blood glucose curve.

[0012] In a second aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the above-mentioned post-meal blood glucose trend prediction method based on multi-modal fusion.

[0013] In a third aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the above-mentioned post-meal blood glucose trend prediction method based on multi-modal fusion.

[0014] In a fourth aspect, the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the above-mentioned post-meal blood glucose trend prediction method based on multi-modal fusion.

[0015] According to the specific embodiments provided by the present application, the following technical effects are disclosed:

[0016] The present application provides a post-meal blood glucose trend prediction method and device based on multi-modal fusion. First, an image encoder is used to extract the image features of the user's pre-meal and post-meal dining pictures. Then, the image features of the user's pre-meal and post-meal dining pictures are used to match the most relevant blood glucose curve from the blood glucose curve database as a reference. Next, a time series encoder is used to extract the features of the reference blood glucose curve and the features of the user's pre-meal blood glucose curve respectively. Finally, the features of the reference blood glucose curve and the features of the user's pre-meal blood glucose curve are input into the time series decoder to obtain a more accurate predicted trend of the post-meal blood glucose curve. The present application fuses two modalities, namely the user's pre-meal blood glucose curve and the pre-meal and post-meal images, evaluates the user's food intake situation through the pre-meal and post-meal images (evaluated by the blood glucose curve), and combines the pre-meal blood glucose curve to achieve an accurate and rapid prediction of the post-meal blood glucose change. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 It is an application environment diagram of a post-meal blood glucose trend prediction method based on multi-modal fusion in an embodiment of the present application;

[0019] Figure 2Schematic flowchart of a postprandial blood glucose trend prediction method provided by an embodiment of the present application;

[0020] Figure 3 Schematic diagram of the overall network structure composed of a modality-aligned image-time series encoder and a time series decoder provided by an embodiment of the present application;

[0021] Figure 4 Schematic diagram of the structure of an image encoder module provided by an embodiment of the present application;

[0022] Figure 5 Schematic diagram of the structure of a Transformer layer provided by an embodiment of the present application;

[0023] Figure 6 Schematic diagram of the structure of a single image self-attention mechanism module provided by an embodiment of the present application;

[0024] Figure 7 Schematic diagram of the structure of a time series encoder provided by an embodiment of the present application;

[0025] Figure 8 Schematic diagram of the structure of a temporal self-attention mechanism module provided by an embodiment of the present application;

[0026] Figure 9 Schematic diagram of the structure of a time series decoder model provided by an embodiment of the present application;

[0027] Figure 10 Schematic diagram of the structure of a single cross-attention mechanism module provided by an embodiment of the present application;

[0028] Figure 11 Schematic diagram of the structure of a mapping module provided by an embodiment of the present application;

[0029] Figure 12 Schematic flowchart of the training process of a modality-aligned image-time series encoder network provided by an embodiment of the present application;

[0030] Figure 13 Schematic diagram of the structure of an image-blood glucose sequence network provided by an embodiment of the present application;

[0031] Figure 14 Schematic flowchart of the main training process of a time series decoder provided by an embodiment of the present application;

[0032] Figure 15 Schematic flowchart of the specific training process of a time series decoder provided by an embodiment of the present application;

[0033] Figure 16 Schematic diagram of the structure of a computer device provided by an embodiment of the present application. Detailed implementation manners

[0034] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0035] The method for predicting postprandial blood glucose trend based on multimodal fusion provided by the embodiments of the present application can be applied to an application environment as Figure 1 shown. Among them, the terminal communicates with the server through the network. The data storage system can store the data that the server needs to process. The data storage system can be set up separately, integrated on the server, or placed on the cloud or other servers. The terminal can send the pre-meal blood glucose curve and pre- and post-meal dining pictures of the user to the server. After receiving the pre-meal blood glucose curve and pre- and post-meal dining pictures of the user, the server uses the trained image encoder to extract the image features of the pre- and post-meal dining pictures; according to the image features of the pre- and post-meal dining pictures, select the blood glucose curve most relevant to the pre- and post-meal dining pictures from the blood glucose curve database to obtain a reference blood glucose curve; input the pre-meal blood glucose curve into the trained time series encoder to obtain pre-meal blood glucose curve features; input the reference blood glucose curve into the trained time series encoder to obtain reference blood glucose curve features; input the pre-meal blood glucose curve features and the reference blood glucose curve features into the trained time series decoder to obtain the predicted result of the post-meal blood glucose curve. The server can feedback the obtained predicted result of the post-meal blood glucose curve to the terminal. In addition, in some embodiments, the method for predicting postprandial blood glucose trend based on multimodal fusion can also be implemented by the server or the terminal alone. For example, the terminal can directly perform the prediction of the postprandial blood glucose trend based on multimodal fusion for the pre-meal blood glucose curve and pre- and post-meal dining pictures of the user, or the server can obtain the pre-meal blood glucose curve and pre- and post-meal dining pictures of the user from the data storage system and perform the prediction of the postprandial blood glucose trend based on multimodal fusion.

[0036] Among them, the terminal can be, but is not limited to, various desktop computers, laptop computers, smartphones, tablet computers, Internet of Things devices, and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers, and can also be a cloud server.

[0037] In an exemplary embodiment, as Figure 2 and Figure 3As shown, a method for predicting postprandial blood glucose trends based on multimodal fusion is provided. This method is executed by a computer device, which can be specifically executed by a computer device such as a terminal or a server alone, or jointly executed by a terminal and a server. In the embodiments of the present application, taking the application of this method to Figure 1 the server in

[0038] Step 101: Obtain the preprandial blood glucose curve and pre- and postprandial meal pictures of the user.

[0039] Since the purpose of this application is to provide a technology for predicting the postprandial blood glucose curve of a user based on the user's preprandial blood glucose curve and pre- and postprandial food pictures, the method proposed in this application needs to input the preprandial blood glucose information of the current user and the pre- and postprandial food pictures. Among them, the preprandial blood glucose curve of the current user is represented as S t ∈R 1×L×1 , where 1×L×1 indicates that the length of the blood glucose sequence of the current user is L. The pair of pre- and postprandial food pictures of the current user is represented as I img ∈R 1×H×W×C , where the "1" in the first dimension is the same as the "1" in the first dimension of the blood glucose curve, representing the current user, and H, W, and C respectively represent the height, width, and number of channels of the input picture. The pre- and postprandial meal pictures are two different pictures. In this application, the two pictures are spliced in the channel dimension and then sent to the image encoder together. That is, for each user, the height, width, and number of channels of the pre- and postprandial meal pictures are actually 1×H×W×(C / 2). Finally, the blood glucose curve S t and the pre- and postprandial meal pictures I img of the user are mapped to a tensor as the input of the image encoder, and

[0040] Step 102: Use the trained image encoder to extract the image features of the pre- and postprandial meal pictures.

[0041] Step 103: According to the image features of the pre- and postprandial meal pictures, select the blood glucose curve most relevant to the pre- and postprandial meal pictures from the blood glucose curve database to obtain the reference blood glucose curve (corresponding to Figure 2 the selected curve in

[0042] Encode each blood glucose curve in the blood glucose curve database through a trained time series encoder to obtain blood glucose curve features At this time, use the cosine similarity to match the image features with each blood glucose curve feature in the blood glucose curve database and find the blood glucose curve with the largest cosine similarity As a reference blood glucose curve. N represents the number of blood glucose curves in the blood glucose curve database.

[0043] Step 104: Input the pre-meal blood glucose curve into the trained time series encoder to obtain the pre-meal blood glucose curve features.

[0044] Step 105: Input the reference blood glucose curve into the trained time series encoder to obtain the reference blood glucose curve features. The reference blood glucose curve features correspond to Figure 2 the key value feature sequence of the selected curve in.

[0045] Step 106: Input the pre-meal blood glucose curve features and the reference blood glucose curve features into the trained time series decoder to obtain the predicted result of the post-meal blood glucose curve.

[0046] The network structure of this application mainly includes three parts: 1) A time series encoder responsible for extracting blood glucose curve features, which is mainly used to extract the time dependence of blood glucose curve features; 2) A time series decoder responsible for predicting the post-meal blood glucose curve of the user, which is mainly used to fuse the above image features and blood glucose curve time series features to achieve post-meal blood glucose prediction; 3) An image encoder responsible for extracting image features. By fusing two modalities of the user's pre-meal blood glucose curve and the dining image, the trend of the user's post-meal blood glucose curve is predicted. The present invention takes time series prediction as the core and uses image recognition technology to capture food intake in the dining image to achieve the prediction of post-meal blood glucose trend. This application proposes a deep network framework for post-meal blood glucose trend prediction based on multi-modal fusion, which takes two-modal data of pre-meal blood glucose time series and dining image as input to predict the post-meal blood glucose trend. Compared with strictly restricting the types and weights of ingredients ingested by patients, this application innovatively uses pre- and post-meal image modalities to estimate dietary information. The first advantage of this method is that it reduces many restrictions on the test users, is closer to the daily life scenarios of users, greatly reduces the complexity of participating in the test, and reduces the difficulty of collecting data. Due to reducing the quantitative restrictions on dining, this application studies the impact of more generalized dietary information on post-meal blood glucose.

[0047] In another exemplary embodiment of this application, as Figure 4 shown, in step 102, the image encoder includes a convolutional layer, a first linear layer, a first normalization layer, a Transformer layer unit, and a second normalization layer connected in series; the Transformer layer unit includes a plurality of Transformer layers connected in series; the input of the convolutional layer is the pre- and post-meal dining pictures; the output of the second normalization layer is the image features of the pre- and post-meal dining pictures.

[0048] The specific processing flow of the image encoder is divided into the following five steps:

[0049] (a1) First, the pictures of eating before and after meals After data preprocessing, the picture data is obtained The Through the convolutional layer (denoted as the first convolutional layer), a feature map with a height and width of 14×14 is obtained, and the feature dimension C is increased from 6 to 1536. Finally, the convolutional layer outputs a new feature map with a feature scale of N×14×14×1536, and this feature map will be used as the input of the first linear layer. Among them, as an example, the convolutional kernel of the first convolutional layer is 16×16, the stride is 16, and the padding is 15.

[0050] The image encoder uses the first convolutional layer to extract the features of the user's pictures of eating before and after meals, so as to obtain richer semantic information.

[0051] (a2) Through the first linear layer, the features obtained in the previous step (a1) are straightened from the two-dimensional space, and the number of feature channels becomes 512. The feature scale output by the first linear layer will change from N×14×14×1536 to N×196×512.

[0052] The first linear layer is used to straighten the two-dimensional image features, that is, the two-dimensional features are changed into one-dimensional features in space through the first linear layer, which is convenient for the subsequent Transformer layer to further extract the image features.

[0053] (a3) Through the first normalization layer, the features obtained in the previous step (a2) are normalized, and the obtained features are denoted as And

[0054] The first normalization layer is used to normalize the image features obtained by the first linear layer, reduce the internal covariate shift of the features, and maintain the consistency of the data distribution.

[0055] (a4) The features after passing through the first normalization layer Will pass through multiple cascaded Transformer layers. As an example, assume that there are 11 Transformer layers. Through the cascaded Transformer layers, new features will be obtained And

[0056] The Transformer layer is used to model the whole picture, and the complex features in the image are further captured through the global receptive field, so as to obtain a better representation of the user's pictures of food before and after meals.

[0057] (a5) Use the second normalization layer to normalize the features obtained through multiple cascaded Transformer layers To obtain the output feature vector F of the image encoderimg , and

[0058] The second normalization layer is used to normalize the features obtained by the Transformer layer to prevent the network from having vanishing or exploding gradients.

[0059] In another exemplary embodiment of the present application, multiple cascaded Transformer layers in the image encoder are mainly used for global modeling of the user's pre-meal and post-meal dining pictures. Here, a single Transformer layer is described in detail. The structure of the Transformer layer is as Figure 5 shown. The Transformer layer includes a third normalization layer, an image self-attention mechanism layer module, a first splicing layer, a second linear layer, a first addition layer, a fourth normalization layer, a multi-layer perceptron, and a second addition layer connected in series; the input of the first addition layer is also connected to the input of the third normalization layer, and the input of the second addition layer is also connected to the output of the first addition layer.

[0060] The specific processing flow of the Transformer layer is divided into the following eight steps:

[0061] (b1) First, the features of the first normalization layer are copied as the input of the third normalization layer, and high-dimensional features are obtained through the third normalization layer and

[0062] The third normalization layer is used to normalize the input features to reduce the internal covariate shift of the features and maintain the consistency of the data distribution.

[0063] (b2) The high-dimensional features extract important information in the image through the image self-attention mechanism module. Here, the image self-attention mechanism is multi-head attention, and the number of heads is h. Then, h high-dimensional features will be obtained through the image self-attention mechanism module, and the features are mapped to a tensor denoted as A img , and

[0064] The image self-attention mechanism module is stacked by multiple self-attention mechanism modules. Through the self-attention mechanism, the network focuses on the key information in the image and further ignores the irrelevant information, thereby improving the learning ability of the network.

[0065] (b3) The first splicing layer is used to splice the features extracted by the image self-attention mechanism module in the patch dimension to obtain and

[0066] The features output by the image self-attention mechanism module are concatenated using the first concatenation layer. Since the image self-attention mechanism module in the Transformer layer is composed of multiple self-attention mechanism modules stacked together, it is necessary to integrate the features extracted by all self-attention mechanism modules.

[0067] (b4) Features obtained by the first concatenation layer Are dimensionally reduced through the second linear layer, such that the features output by the second linear layer Have the same scale as the input features of the Transformer layer For facilitating the subsequent residual fusion.

[0068] The high-dimensional image features are dimensionally reduced using the second linear layer, such that the scale of the output features is the same as that of the input features of the Transformer layer, thereby retaining important features while removing redundant information.

[0069] (b5) The initial input features of the Transformer layer are fused through residual cross-line With the features obtained by the second linear layer The residual fusion features are denoted as And The residual fusion here corresponds to Figure 5 The first addition operation in, denoted as the first addition layer.

[0070] (b6) The fusion features are copied And the copied Is used as the input of the fourth normalization layer, and the fourth normalization layer is used to normalize the fusion features To obtain high-dimensional features Which are used as the input of the multi-layer perceptron.

[0071] The fourth normalization layer is used to normalize its input features again, because the input high-dimensional features at this time are fused from the initial input features of the Transformer layer and the high-dimensional features obtained by the second linear layer through a residual structure.

[0072] (b7) The multi-layer perceptron receives the output of the fourth normalization layer To further extract the high-dimensional image features. The multi-layer perceptron is composed of a fifth linear layer, a first activation layer, and a sixth linear layer connected in series. Through multi-layer stacking, high-dimensional features

[0073] (b8) Through a residual structure (corresponding to Figure 5 The second addition operation in, denoted as the second addition layer) to fuse the Output by the multi-layer perceptron and the fusion features output by the first addition layer To obtain the final features.

[0074] In another exemplary embodiment of the present application, the image self-attention mechanism module is composed of a stack of multiple self-attention mechanism modules. Here, a single self-attention mechanism module in the image self-attention mechanism module is described in detail, and its structure is as Figure 6 shown. A single self-attention mechanism module mainly consists of a linear layer IQ, a linear layer IK, a linear layer IV, and operations such as matrix multiplication, scaling, and the SoftMax(·) function. Here, the processing flow of a single self-attention mechanism module is described in detail.

[0075] (c1) Duplicate the input feature three times and use them as the inputs of the linear layer IQ, the linear layer IK, and the linear layer IV respectively. Through the above three linear layers, three high-dimensional features IQ, IK, and IV will be obtained in sequence.

[0076] (c2) Transpose the high-dimensional feature IQ and perform matrix multiplication with the high-dimensional feature IK, and scale the feature obtained after multiplication through division calculation.

[0077] (c3) Normalize the scaled feature through SoftMax(·) and perform matrix multiplication with the output feature IV of the linear layer IV to finally obtain the high-dimensional feature and represents the high-dimensional feature output by the i-th self-attention mechanism module.

[0078] In another exemplary embodiment of the present application, the time series encoders in steps 104 and 105 are mainly used to extract user blood glucose curve features. Here, the time series encoder module is described in detail, and the structure of the time series encoder is as Figure 7 shown. The time series encoder includes: a data embedding layer, a temporal self-attention mechanism layer, a first dropout layer, a third addition layer, and a fifth normalization layer connected in sequence; the input of the third addition layer is also connected to the output of the data embedding layer.

[0079] The output of the time series encoder includes the key feature and value feature output by the temporal self-attention mechanism layer and the output of the fifth normalization layer.

[0080] Among them, in step 106, the pre-meal blood glucose curve feature input to the time series decoder is the feature (query feature) output by the fifth normalization layer after inputting the pre-meal blood glucose curve into the time series encoder.

[0081] The reference blood glucose curve feature input to the time series decoder includes the key feature and value feature output by the temporal self-attention mechanism layer obtained after inputting the reference blood glucose curve into the time series encoder.

[0082] The main process of time series encoder processing is divided into three steps:

[0083] (d1) Input the blood glucose curves of N users before and after meals Introduce sequence position order information using the data embedding layer to obtain high-dimensional feature S e , and

[0084] Use the data embedding layer to map the original blood glucose curve data into high-dimensional features, and introduce sequence position order information using positional encoding.

[0085] (d2) Input the high-dimensional tensor S obtained in step (d1) e into the temporal self-attention mechanism layer to obtain the key feature K of the blood glucose curve, and the value feature V, and the correlation information feature S a , and Let S a be randomly inactivated to obtain S d , and

[0086] Use the temporal self-attention mechanism layer to further dynamically adjust the attention degree to different positions according to the correlation between positions in the high-dimensional tensor S e , capture its global dependence, and through context modeling, make the output of each position fuse the weighted information of itself and other positions in the sequence to obtain the correlation information and the key and value features of the blood glucose curve.

[0087] Use the first dropout layer to perform random inactivation processing on the correlation information, avoid problems such as overfitting and gradient disappearance during training, and construct a residual connection with the high-dimensional features obtained by the data embedding layer.

[0088] (d3) Construct a residual connection between S obtained in step (d2) d and S e (corresponding to the addition operation in Figure 7 , denoted as the third addition layer) to obtain F r =S d +S e , and after normalization, obtain the blood glucose information of the user before and after meals and

[0089] Use the fifth normalization layer to perform normalization processing on the obtained high-dimensional feature F r to finally obtain the blood glucose feature (query feature) of the blood glucose curve.

[0090] The time series encoder initially obtains blood glucose features through the temporal self-attention mechanism module. The structure of the temporal self-attention mechanism layer is as Figure 8 shown. The temporal self-attention mechanism layer mainly consists of a linear layer SQ, a linear layer SK, a linear layer SV, and operations such as matrix multiplication, scaling, and the SoftMax(·) function. Here, the processing flow of the temporal self-attention mechanism layer is described in detail.

[0091] (e1) Duplicate the input feature S e three times and use them as the inputs of the linear layer SQ, the linear layer SK, and the linear layer SV respectively. Through the above three linear layers, three high-dimensional features Q’, K, and V will be obtained in sequence. Among them, after the reference blood glucose curve is input into the time series encoder, the key feature K and the value feature V output by the linear layer SK and the linear layer SV will be part of the input of the time series decoder.

[0092] (e2) Transpose the high-dimensional feature Q’ and perform matrix multiplication with the high-dimensional feature K, and scale the feature obtained after multiplication through division calculation.

[0093] (e3) Normalize the scaled feature through SoftMax(·) and perform matrix multiplication with the output feature V of the linear layer SV to finally obtain the high-dimensional feature S a , and

[0094] In another exemplary embodiment of the present application, as Figure 9 shown, in step 106, the time series decoder includes a multi-head cross-attention mechanism module, a splicing layer, a third linear layer, a second dropout layer, a fourth addition layer, a sixth normalization layer, a mapping module, a fifth addition layer, a seventh normalization layer, and a fourth linear layer connected in sequence; the input of the fourth addition layer is also connected to the input of the multi-head cross-attention mechanism module, and the input of the fifth addition layer is also connected to the output of the sixth normalization layer.

[0095] The specific processing flow of the time series decoder is divided into 3 steps:

[0096] (f1) Use the key and value extracted from the time series encoder and the blood glucose feature as the query sequence and input it into the multi-head cross-attention mechanism module. Use the attention mechanism to capture the correlation between the query sequence and the key and value sequences to obtain the correlation feature Then splice it through the second splicing layer into a high-dimensional weight After that, perform dimensionality reduction processing through the third linear layer to Finally, obtain after being processed by the second dropout layer and further construct a residual connection with the input feature of the time series encoder (correspondingFigure 9 The first addition operation in (denoted as the fourth addition layer) to obtain attention weights and

[0097] The time series decoder utilizes the multi-head cross-attention mechanism module to capture the correlation between query features and key and value features, obtain the global dependencies and important features of the input sequence, thereby improving the accuracy and efficiency of prediction.

[0098] The second concatenation layer is used to concatenate the correlation information. Since the multi-head cross-attention mechanism module is stacked by multiple independent cross-attention mechanism modules, it is necessary to integrate the correlation features extracted by all cross-attention mechanism modules.

[0099] The third linear layer is used to reduce the dimension of the high-dimensional correlation features, so that the output features are consistent with the input feature scale required by the subsequent structure, thereby removing redundant information while retaining important features.

[0100] The second dropout layer is used to perform dropout processing on the correlation information and construct a residual connection with the query sequence to avoid problems such as overfitting and gradient disappearance during training.

[0101] (f2) The attention weights processed by the sixth normalization layer Input mapping module, the enhanced weights of the information are further combined with F n Construct a residual connection (corresponding to Figure 9 the second addition operation in, denoted as the fifth addition layer) to obtain Y′ = X n + F n , and

[0102] The sixth normalization layer is used to normalize the input features to reduce the internal covariate shift of the features and maintain the consistency of the data distribution.

[0103] The mapping module is used to enhance the information of the processed correlation information above, provide stronger non-linear transformation and local representation capabilities for the sequence features captured by the cross-attention mechanism, and further construct a residual connection with the input of the module to ensure the stability of network training.

[0104] (f3) The normalized weights processed by the seventh normalization layer are input into the fourth linear layer to match the data dimension with the required output scale, and the final predicted result O of the blood glucose curve is obtained, and

[0105] The input features are normalized using the seventh normalization layer to reduce the internal covariate shift of the features and maintain the consistency of the data distribution.

[0106] The correlation features are subjected to data mapping and dimension matching using the fourth linear layer, and the resulting output is the postprandial blood glucose curve.

[0107] The multi-head cross-attention mechanism module of the time series decoder is composed of multiple independent cross-attention mechanism modules stacked together. Here, a single cross-attention mechanism module is described in detail, and its structure is as Figure 10 shown. A single cross-attention mechanism module mainly consists of a linear layer CQ, a linear layer CK, a linear layer CV, and operations such as matrix multiplication, scaling, and the SoftMax(·) function. Here, the processing flow of a single cross-attention mechanism module is described in detail.

[0108] (g1) The blood glucose features key and value extracted from the time series encoder are respectively input into the linear layer CQ, the linear layer CK, and the linear layer CV to sequentially obtain three high-dimensional features and

[0109] (g2) The high-dimensional features and K T are multiplied matrix-wise, and the result is scaled to to obtain and

[0110] (g3) The scaled features are normalized through SoftMax(·) and multiplied matrix-wise with the output feature V′ of the linear layer CV to finally obtain the high-dimensional feature A, and

[0111] Among them, the time series decoder enhances the information of the features through a mapping module. Here, the mapping module is described in detail, and its structure is as Figure 11 shown. The mapping module mainly includes a second convolutional layer, a second activation layer, a third dropout layer, a third convolutional layer, and a fourth dropout layer connected in series.

[0112] The mapping module adopts an overall series-connected structure, and its specific processing flow is as follows:

[0113] (h1) Using the second convolutional layer, the normalized attention weight F n output by the sixth normalization layer is mapped to

[0114] The time series decoder maps the features to a specific scale through the second convolutional layer (with a kernel size of 1×1, a stride of 1, and a padding of 0) to facilitate subsequent network processing.

[0115] (h2) Using the second activation layer, for the output F of the second convolutional layer n (2) introduce non-linearity and map it to

[0116] Using the second activation layer to introduce non-linearity to the features, enhancing the feature expression ability of the network.

[0117] (h3) Using the third dropout layer, map the output F of the second activation layer n (2) to

[0118] Using the third dropout layer to improve the generalization ability and robustness of the model and reduce the risk of overfitting during training.

[0119] (h4) Using the third convolutional layer, map the output F of the third dropout layer n (3) to

[0120] Using the third convolutional layer (with a kernel size of 1×1, a stride of 1, and a padding of 0) to map the features to a specific scale to facilitate subsequent network processing.

[0121] (h5) Using the fourth dropout layer, map the output F of the third convolutional layer n (4) to as the output of the mapping module.

[0122] Using the fourth dropout layer to improve the generalization ability and robustness of the model and reduce the risk of overfitting during training.

[0123] In another exemplary embodiment of the present application, since the present invention adopts a deep network framework, the network needs to be trained first, and then the inference process is completed based on the trained network. Therefore, before performing step 101 "obtain the pre-meal blood glucose curve and pre- and post-meal dining pictures of the user", the post-meal blood glucose trend prediction method based on multi-modal fusion further includes: training the image encoder and the time series encoder, as Figure 12As shown, the training strategies involved in the image encoder and the time series encoder: contrastive learning training for aligning the feature modalities of pre-meal and post-meal images and blood glucose time series features. Among them, the changing trend of post-meal blood glucose is used to supervise the dietary information of pre-meal and post-meal images. In this supervision method, two different modalities are adopted. Therefore, a multi-modal contrastive learning strategy is adopted during the supervision process to complete the self-supervised training of the image modality encoder. This self-supervised method has more advantages compared to the traditional calorie annotation of pre-meal and post-meal images.

[0124] The training process of the image encoder and the time series encoder is specifically as follows:

[0125] (i1) Obtain pre-meal blood glucose curve samples and pre-meal and post-meal dining picture samples of multiple users.

[0126] Obtain the data pairs of blood glucose curve samples - pre-meal and post-meal picture samples Among them and respectively represent the blood glucose curve sample data and the pre-meal and post-meal picture sample data of the i-th user, and N represents the input of N data pairs. Map the N groups of blood glucose curve samples - pre-meal and post-meal picture sample data into tensors (f t , f img ), and where L represents the length of blood glucose data, and H, W, and C respectively represent the height, width, and number of channels of the input picture. The input picture sample data is the pre-meal and post-meal food picture samples, and they are concatenated in the channel dimension during input.

[0127] (i2) Input the pre-meal and post-meal dining picture samples into the image encoder to obtain the first pre-meal and post-meal dining picture sample features.

[0128] Input the pre-meal and post-meal picture samples of N users corresponding to the blood glucose curve samples Extract the user dining features through the image encoder Corresponding to the first pre-meal and post-meal dining picture sample features.

[0129] (i3) Input the pre-meal blood glucose curve samples into the time series encoder to obtain the first pre-meal blood glucose curve sample features.

[0130] Input the blood glucose curve samples of N users before and after meals Obtain the blood glucose features of users before and after meals through the time series encoder Corresponding to the first pre-meal blood glucose curve sample features (query features).

[0131] Extract the high-dimensional features of pre-meal and post-meal images using the image encoder of Extract the features F of the input blood glucose curve using the time series encoder oft i And align the features obtained by the two encoders.

[0132] (i4) Calculate the CLIP loss according to the first pre - and post - meal dining picture sample features and the first pre - meal blood glucose curve sample features.

[0133] Using the blood glucose features obtained in step (i3) and the user dining features obtained in step (i2) Construct an N×N matrix, as Figure 13 shown. The first cell in the first row of the matrix represents the similarity degree between the first group of pre - and post - meal pictures and their corresponding blood glucose curve features. Consider the pre - and post - meal and corresponding blood glucose curve data as positive sample data, that is, the diagonal elements in the matrix are all positive samples, and the other positions are negative sample data. Use the cross - entropy function for training, with the goal of making the inner product at the positive sample positions larger, and construct the CLIP loss function.

[0134] (i5) Back - propagate according to the CLIP loss to optimize the parameters of the image encoder and the time - series encoder until the CLIP loss converges or the number of iterations reaches the first preset number of iterations, and obtain the trained image encoder and the trained time - series encoder.

[0135] In another exemplary embodiment of the present application, before performing step 101 "obtain the user's pre - meal blood glucose curve and pre - and post - meal dining pictures", the post - meal blood glucose trend prediction method based on multi - modal fusion further includes: training the time - series decoder, and the training strategy of the time - series decoder adopts the time - series regression learning training strategy based on post - meal blood glucose curve supervision. As Figure 14 and Figure 15 shown, the specific process of training the time - series decoder is as follows:

[0136] (j1) Obtain pre - meal blood glucose curve samples, pre - and post - meal dining picture samples, and corresponding measured post - meal blood glucose curve results of multiple users.

[0137] (j2) Input the pre - and post - meal dining picture samples into the trained image encoder to obtain the second pre - and post - meal dining picture sample features.

[0138] (j3) According to the second pre - and post - meal dining picture sample features, select the blood glucose curve sample most relevant to the pre - and post - meal dining picture samples from the blood glucose curve database to obtain the reference blood glucose curve sample.

[0139] (j4) Input the reference blood glucose curve sample into the trained time - series encoder to obtain the reference blood glucose curve sample features.

[0140] (j5) Input the pre-meal blood glucose curve sample into the trained time series encoder to obtain the second pre-meal blood glucose curve sample features.

[0141] Extract the high-dimensional features of the pre-meal and post-meal images using the trained image encoder Extract the features F of the input blood glucose curve using the trained time series encoder t i And align the features obtained by the two encoders.

[0142] (j6) Input the reference blood glucose curve sample features and the second pre-meal blood glucose curve sample features into the time series decoder to obtain the predicted results of the post-meal blood glucose curve sample.

[0143] (j7) Calculate the loss error based on the predicted results of the post-meal blood glucose curve sample and the corresponding measured results of the post-meal blood glucose curve.

[0144] Calculate the MSE loss function for the predicted sequence of the post-meal blood glucose curve sample and the corresponding measured sequence of the post-meal blood glucose curve to ensure that the time series decoder learns the correct blood glucose-time dependence relationship.

[0145] (j8) Optimize the parameters of the time series decoder by backpropagation based on the loss error until the loss error converges or reaches the second preset number of iterations to obtain the trained time series decoder.

[0146] In this application, the core of post-meal blood glucose monitoring is to accurately predict the curve of the patient's post-meal blood glucose change over time, which belongs to the category of time series prediction in the field of artificial intelligence. And predicting the time series of post-meal blood glucose based on the time series of pre-meal blood glucose requires considering the impact of the meal situation on blood glucose changes. The assessment of the meal situation is carried out in the way of pre-meal and post-meal images, that is, obtaining meal information from the image modality. To fuse this image modality into the time series modality, first, these two modalities need to be aligned. After the modality alignment, the pre-meal blood glucose time series features and the meal image features are jointly used as the basis for post-meal blood glucose time series prediction, and accurate prediction of blood glucose change trends is achieved by means of relevant time series prediction technologies.

[0147] The present application has the following technical effects: (1) A deep network framework for predicting postprandial blood glucose trends based on multimodal fusion is proposed. This deep network framework takes two types of modal data, namely preprandial blood glucose time series and dining images, as inputs to predict postprandial blood glucose trends. There are two existing blood glucose prediction methods. The first type does not consider the impact of diet on blood glucose and directly uses a model to perform regression fitting on the recorded preprandial and postprandial blood glucose curves. In fact, dietary information cannot be ignored in postprandial blood glucose prediction. Without considering dietary intake at all, it is usually impossible to accurately estimate the postprandial blood glucose curve based solely on the preprandial blood glucose curve. This is the reason for introducing the modality of preprandial and postprandial images in the present application. The second type obtains dietary information by strictly restricting the types and amounts of ingredients consumed by patients. This acquisition method is more used in short-term experiments and is not suitable for a large number of people to participate for a long time. The present invention innovatively uses the modality of preprandial and postprandial images to estimate dietary information. This method first reduces many restrictions on the test users, is closer to the daily life scenarios of users, greatly reduces the complexity of participating in the test, reduces the difficulty of collecting data, and provides the possibility for the collection of big data. Due to the reduction of the quantitative restrictions on dining, the problem studied in the present invention is more inclined to the impact of generalized dietary information on blood glucose.

[0148] (2) A method for supervising the dietary information of preprandial and postprandial images using the postprandial blood glucose change trend is proposed. In this supervision method, two different modalities are adopted. Therefore, a multimodal contrast learning strategy is used to complete the self-supervised training of the image modality encoder during the supervision process. This supervision method has more advantages compared with the traditional calorie annotation of preprandial and postprandial images. First, calorie annotation requires experts in the relevant field to calibrate by comparing the foods in preprandial and postprandial images. The postprandial blood glucose change curve adopted in the present application can be automatically recorded by the blood glucose monitoring device worn by the user, which not only reduces the cumbersome expert annotation but also avoids the subjectivity of expert annotation. Second, in the field of blood glucose monitoring, it is more desirable to directly establish the relationship between dining images and postprandial blood glucose, rather than converting dining images into calorie information. Given that calorie information and postprandial blood glucose changes are not equivalent, the present application directly links preprandial and postprandial images with postprandial blood glucose, which more meets the task requirements of postprandial blood glucose monitoring.

[0149] The present application also provides an application scenario, which applies the above-mentioned postprandial blood glucose trend prediction method based on multimodal fusion. Specifically: The postprandial blood glucose trend prediction method based on multimodal fusion provided in this embodiment can be applied to the postprandial blood glucose prediction scenario. This scenario includes a data collection link, a postprandial blood glucose prediction link, and a blood glucose regulation strategy optimization link; the data collection link is used to collect the user's pre-meal and post-meal dining images and the pre-meal blood glucose curve; the postprandial blood glucose prediction link is used to predict the postprandial blood glucose according to the user's pre-meal and post-meal dining images and the pre-meal blood glucose curve; the blood glucose regulation strategy optimization link is used to optimize the user's blood glucose regulation strategy with reference to the postprandial blood glucose prediction result. The postprandial blood glucose trend prediction method based on multimodal fusion provided in this embodiment belongs to the postprandial blood glucose prediction link.

[0150] In an exemplary embodiment, a computer device is provided. This computer device can be a server or a terminal, and its internal structure diagram can be as Figure 16 shown. This computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of this computer device is used to provide computing and control capabilities. The memory of this computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of this computer device is used to store postprandial blood glucose curve prediction data. The input / output interface of this computer device is used to exchange information between the processor and external devices. The communication interface of this computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it implements a postprandial blood glucose trend prediction method based on multimodal fusion.

[0151] Those skilled in the art can understand, Figure 16The structures shown are only block diagrams of some of the structures related to the solution of this application, and do not constitute a limitation on the computer devices to which the solution of this application is applied. The specific computer devices may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented. In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by the processor, the steps in the above method embodiments are implemented. In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0152] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0153] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0154] In this article, specific examples are used to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation on this application.

Claims

1. A method for predicting postprandial blood glucose trend based on multimodal fusion, characterized in that: The postprandial blood glucose trend prediction method based on multimodal fusion includes: Obtain the user's pre-meal blood sugar curve and pre-meal and post-meal meal pictures; Extracting image features of the meal pictures before and after the meal using the trained image encoder; According to the image features of the meal pictures before and after the meal, the blood glucose curve most relevant to the meal pictures before and after the meal is selected from the blood glucose curve database to obtain a reference blood glucose curve; Inputting the pre-meal blood glucose curve into the trained time series encoder to obtain the pre-meal blood glucose curve features; Inputting the reference blood glucose curve into the trained time series encoder to obtain the reference blood glucose curve features; The pre-meal blood glucose curve features and the reference blood glucose curve features are input into the trained time series decoder to obtain a post-meal blood glucose curve prediction result.

2. The method for predicting postprandial blood glucose trend based on multimodal fusion according to claim 1, characterized in that: The image encoder includes a convolutional layer, a first linear layer, a first normalization layer, a Transformer layer unit and a second normalization layer connected in series; the Transformer layer unit includes a plurality of Transformer layers connected in series; the input of the convolutional layer is the meal pictures before and after the meal; the output of the second normalization layer is the image features of the meal pictures before and after the meal.

3. The method for predicting postprandial blood glucose trend based on multimodal fusion according to claim 2, characterized in that: The Transformer layer includes a third normalization layer, an image self-attention mechanism layer module, a first splicing layer, a second linear layer, a first addition layer, a fourth normalization layer, a multilayer perceptron and a second addition layer connected in series; the input of the first addition layer is also connected to the input of the third normalization layer, and the input of the second addition layer is also connected to the output of the first addition layer.

4. The method for predicting postprandial blood glucose trend based on multimodal fusion according to claim 1, characterized in that: The time series encoder comprises: a data embedding layer, a temporal self-attention mechanism layer, a first random inactivation layer, a third addition layer and a fifth normalization layer connected in sequence; the input of the third addition layer is also connected to the output of the data embedding layer; The output of the time series encoder includes the key features and value features output by the time series self-attention mechanism layer and the output of the fifth normalization layer.

5. The method for predicting postprandial blood glucose trend based on multimodal fusion according to claim 1, characterized in that: The time series decoder includes a multi-head cross-attention mechanism module, a second splicing layer, a third linear layer, a second random inactivation layer, a fourth addition layer, a sixth normalization layer, a mapping module, a fifth addition layer, a seventh normalization layer and a fourth linear layer connected in sequence; the input of the fourth addition layer is also connected to the input of the multi-head cross-attention mechanism module, and the input of the fifth addition layer is also connected to the output of the sixth normalization layer.

6. The method for predicting postprandial blood glucose trend based on multimodal fusion according to claim 1, characterized in that: Before executing the step of "obtaining the user's pre-meal blood glucose curve and pre-meal and post-meal meal pictures", the post-meal blood glucose trend prediction method based on multimodal fusion also includes: training the image encoder and the time series encoder, specifically: Obtain pre-meal blood sugar curve samples and pre-meal and post-meal meal picture samples of multiple users; Input the meal picture samples before and after the meal into the image encoder to obtain the features of the first meal picture samples before and after the meal; Inputting the pre-meal blood glucose curve sample into the time series encoder to obtain the first pre-meal blood glucose curve sample feature; Calculating CLIP loss according to the first pre-meal and post-meal meal picture sample features and the first pre-meal blood glucose curve sample features; Optimize the parameters of the image encoder and the time series encoder according to the CLIP loss back propagation until the CLIP loss converges or the number of iterations reaches a first preset number of iterations, thereby obtaining a trained image encoder and a trained time series encoder.

7. The method for predicting postprandial blood glucose trend based on multimodal fusion according to claim 6, characterized in that: Before executing the step of "obtaining the user's pre-meal blood glucose curve and pre-meal and post-meal meal pictures", the post-meal blood glucose trend prediction method based on multimodal fusion also includes: training the time series decoder, specifically: Obtaining pre-meal blood glucose curve samples, pre-meal and post-meal meal picture samples, and corresponding post-meal blood glucose curve measurement results for multiple users; Inputting the meal picture samples before and after the meal into the trained image encoder to obtain features of the second meal picture samples before and after the meal; According to the characteristics of the second meal picture sample before and after the meal, select the blood glucose curve sample most relevant to the meal picture sample before and after the meal from the blood glucose curve database to obtain a reference blood glucose curve sample; Inputting the reference blood glucose curve sample into the trained time series encoder to obtain the reference blood glucose curve sample features; Inputting the pre-meal blood glucose curve sample into the trained time series encoder to obtain a second pre-meal blood glucose curve sample feature; Inputting the reference blood glucose curve sample features and the second pre-meal blood glucose curve sample features into a time series decoder to obtain a post-meal blood glucose curve sample prediction result; Calculating the loss error according to the postprandial blood glucose curve sample prediction result and the postprandial blood glucose curve actual measurement result; The parameters of the time series decoder are optimized according to the back propagation of the loss error until the loss error converges or reaches a second preset number of iterations, thereby obtaining a trained time series decoder.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the postprandial blood glucose trend prediction method based on multimodal fusion as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the postprandial blood glucose trend prediction method based on multimodal fusion described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the postprandial blood glucose trend prediction method based on multimodal fusion described in any one of claims 1 to 7.

Images