Food cooking state detection method and system based on cross-modal deep learning

Abstract

The invention provides a food cooking state detection method and system based on cross-modal deep learning, and the method comprises the steps: collecting a food cooking data set, and extracting image features through an improved Yolo model; using UMAP to carry out dimensionality reduction on the image features and then carrying out maturity grading; extracting temperature-time sequence features by using a TCN time sequence model; fusing the image features and the temperature-time sequence features; training the ViT model, and using a self-adaptive mechanism improved crown porcupine optimization algorithm CPO to intelligently find an optimal solution for setting of hyper-parameters of the trained ViT model; identifying a cooking state of the food; according to the invention, optimal weight adjustment and weighted fusion of image features and time sequence features are realized through the lightweight multi-modal attention mechanism module; a ViT model hyper-parameter optimization strategy based on a crown porcupine optimization algorithm (CPO) is further introduced, the model detection precision is improved, and real-time monitoring and evaluation of the food cooking state are achieved.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a food cooking state detection method and system based on cross-modal deep learning. Background Art

[0002] In recent years, food safety issues have received much attention. In the food processing process, in addition to the freshness of the ingredients and the hygiene and cleanliness of the processing environment, the maturity of the food during the cooking process is also a key issue of concern.

[0003] Maturity is a comprehensive indicator that includes the maturity of the food's surface during the cooking process, the maturity of the food's center, and the nutrient content of the food in different states of maturity. The standard definition of maturity is generally determined by manual judgment of whether the food has reached a maturity that meets the usage standards.

[0004] The traditional method mainly uses temperature probes to detect the doneness of the center of food; however, manual inspection is time-consuming and laborious, while temperature probes will damage the appearance of food and the cleaning of probe equipment is also a problem.

[0005] In response to the above problems, the existing technology uses machine vision to detect the maturity of food, and the determination of maturity during food cooking is handed over to the target detection algorithm to achieve maturity grading and classification tasks. However, the existing food image recognition only determines the maturity based on the image appearance feature information, and cannot comprehensively assess the maturity of food. How to establish a mapping relationship between the maturity of the food appearance and the maturity of the center, so as to comprehensively evaluate the optimal solution for the maturity of the food during processing, which will help the food industry develop a more intelligent cooking system. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides a food cooking status detection method and system based on cross-modal deep learning. The present invention establishes a mapping relationship between the surface maturity and core maturity of food, thereby comprehensively evaluating the maturity of food during the cooking process.

[0007] The technical solution of the present invention is: a food cooking state detection method based on cross-modal deep learning, comprising the following steps:

[0008] S1), collecting a food cooking data set, and dividing the food cooking data set into a training set and a test set after preprocessing;

[0009] S2), constructing an improved Yolo model and training it, and using the trained improved Yolo model to extract features from the preprocessed image data set to obtain image features containing rich feature information;

[0010] S3), using the UMAP algorithm to reduce the dimension of the extracted image features, visualize the reduced-dimensional data, and grade the appearance maturity of the food cooking process based on the visualized data distribution map;

[0011] S4), constructing a TCN time series model, and extracting temperature-time series features from the food temperature series data set through the TCN time series model;

[0012] S5), fusing the image features and the temperature-time series features through a multimodal fusion layer to obtain multimodal features;

[0013] S6) Using the fused multimodal features to train the ViT model, and using the crested porcupine optimization algorithm (CPO) with an adaptive mechanism to intelligently find the optimal solution for the setting of hyperparameters of the ViT model training;

[0014] S7), constructing a mapping relationship between the food surface maturity value and the food center maturity value according to the food surface image feature information and time series;

[0015] S8) Using the trained ViT model to identify the cooking state in the food image to be detected, and based on the mapping relationship, to achieve real-time detection of the state change during the food cooking process and feedback of the corresponding maturity.

[0016] Preferably, in step S2), the improved Yolo model is: the feature pyramid network FPN of the neck network Neck of the Yolo model is replaced by the path aggregation network PANet; and the ordinary convolution block of the path aggregation network PANet is replaced by a depth-separable convolution block; and at the same time, the CBAM attention mechanism module is inserted in the upsampling and downsampling of the path aggregation network PANet.

[0017] Preferably, in step S2), the improved Yolo model includes an image input layer, a backbone network Backbone, a neck network Neck, and an output layer; wherein the backbone network Backbone adopts CSPDarknet53.

[0018] Preferably, in step S3), the extracted image feature map is reduced in dimension using the UMAP algorithm to visualize the reduced-dimensional data, and the appearance maturity of the food cooking process is graded according to the visualized data distribution map, which specifically includes the following steps:

[0019] S31) construct a graph in high-dimensional space, where each point is connected to its nearest neighbor; and the UMAP algorithm describes the similarity between points through probability distribution. For each point x i , calculate k-nearest neighbors and use the probability distribution p ij To represent the point xi and x j The similarity between them is calculated using the Gaussian kernel function to calculate the probability distribution p ij :

[0020]

[0021] In the formula, d(x i ,x j ) represents point x i and x j The distance between i For the point x i The relevant bandwidth parameters are determined by binary search so that point x i The probability of the k-nearest neighbors is close to a preset value;

[0022] S32) Construct a graph in a low-dimensional space and use probability to describe the similarity between points. For each point y i , calculate its k-nearest neighbors, and use Student's t-distribution to calculate the probability distribution q ij To represent point y i and j The similarity between the probability distribution q ij The calculation formula is:

[0023]

[0024] S33), optimizing the low-dimensional embedding by the fuzzy cross entropy loss function between the minimum high-dimensional graph and the low-dimensional graph, wherein the fuzzy cross entropy loss function C is:

[0025]

[0026] Where C represents the fuzzy cross entropy loss function;

[0027] S34) Use stochastic gradient descent SGD to optimize the above objective function to approximate the projection of the manifold structure of high-dimensional data on the low-dimensional space. The same food forms different clusters in different cooking state dimensions on the two-dimensional image. Based on this, the different cooking states of various types of food can be intuitively judged and graded according to the actual maturity state.

[0028] Preferably, in step S4), the TCN time series model is composed of multiple temporal blocks, and each temporal block includes causal convolution, dilated convolution, residual connection and normalization.

[0029] Preferably, in step S4), the causal convolution is expressed as:

[0030]

[0031] In the formula, y t represents the output of causal convolution, w i is the convolution kernel, b is the bias term, x t-i represents the time series at time point ti; k is the convolution kernel size;

[0032] The dilated convolution is expressed as:

[0033]

[0034] In the formula, Z t is the output of the dilated convolution, and d is the dilation factor.

[0035] Preferably, in step S5), the fusion of the image features and the temperature-time series features includes feature alignment, feature weight adjustment, and splicing operations.

[0036] Preferably, in step S5), the feature alignment uses dynamic time planning DTW to align image features and temperature-time series features.

[0037] Preferably, in step S5), the feature weight adjustment is performed by inserting a lightweight multimodal attention module before the multimodal fusion layer to adjust the optimal weights of the image features and the time series features and perform weighted fusion.

[0038] Preferably, in step S5), the expression of weighted fusion is:

[0039] F fused =α(A I +C IT )+(1-α)(A T +C TI );

[0040] In the formula, F fused represents the weighted fusion feature, α is the learnable weight parameter, A I The self-attention weight matrix representing image features; C IT Represents the cross-attention weight matrix of image features to time series features; A T The self-attention weight matrix representing the time series features; C TI Represents the cross-attention weight matrix of time series features on image features.

[0041] Preferably, the present invention further provides a food cooking state detection system based on cross-modal deep learning, comprising:

[0042] A data collection module, used for collecting a food cooking data set, wherein the food cooking data set includes a food state appearance image data set and a food temperature sequence data set;

[0043] The image feature extraction module is used to extract features from the preprocessed image data set using the improved Yolo model to obtain image features containing rich feature information;

[0044] A maturity grading module is used to reduce the dimension of the image features extracted by the image feature extraction module, visualize the reduced-dimensional data, and grade the appearance maturity of the food during the cooking process based on the visualized data distribution diagram;

[0045] The temperature-time series feature extraction module is used to extract temperature-time series features from the food temperature series dataset through the TCN time series model;

[0046] A feature fusion module is used to obtain multimodal features by fusing image features and temperature-time series features;

[0047] The cooking status detection module is used to identify the cooking status in the food image.

[0048] Preferably, the improved Yolo model is as follows: the feature pyramid network FPN of the neck network Neck of the Yolo model is replaced by the path aggregation network PANet; and the ordinary convolution block of the path aggregation network PANet is replaced by a depth-separable convolution block; and at the same time, the CBAM attention mechanism module is inserted in the upsampling and downsampling of the path aggregation network PANet.

[0049] Preferably, the cooking status detection module utilizes the ViT model to identify the cooking status.

[0050] Preferably, the ViT model is trained using multimodal features, and an improved crested porcupine optimization algorithm (CPO) with an adaptive mechanism is used to intelligently find the optimal solution for setting the hyperparameters of the training ViT model.

[0051] Preferably, the cooking state detection module constructs a mapping relationship between the food surface maturity value and the food center maturity value based on the food surface image information and temperature-time information, thereby realizing real-time detection of state changes during food cooking and feeding back corresponding maturity.

[0052] The beneficial effects of the present invention are:

[0053] 1. The present invention obtains images and temperature information during the food cooking process, uses the improved YOLO model and TCN model to extract image features and time series features respectively, and uses the UMAP algorithm to reduce the dimension of the image features to establish a food maturity assessment system;

[0054] 2. The present invention realizes the optimal weight adjustment and weighted fusion of image features and time series features through a lightweight multimodal attention mechanism module;

[0055] 3. The present invention also introduces a ViT model hyperparameter optimization strategy based on the crested porcupine optimization algorithm (CPO), which improves the model detection accuracy and realizes real-time monitoring and evaluation of food cooking status. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the process of the present invention;

[0057] Figure 2 It is a framework structure diagram of the system of the present invention;

[0058] Figure 3 This is a framework diagram of the improved Yolo model of the present invention. DETAILED DESCRIPTION

[0059] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings:

[0060] Example 1

[0061] like Figure 1 As shown, this embodiment provides a food cooking state detection method based on cross-modal deep learning, comprising the following steps:

[0062] S1), collecting a food cooking data set, and dividing the food cooking data set into a training set and a test set after preprocessing;

[0063] In this embodiment, the food cooking data set includes a food state appearance image data set and a food temperature sequence data set; in this embodiment, the preprocessing of the food state appearance image data includes but is not limited to labeling, flipping, tilting and partially cropping the image to generate a preprocessed image data set, and standardizing the food temperature sequence data set, and dividing the preprocessed image data set into a training set and a test set in a ratio of 7:3.

[0064] S2), constructing an improved Yolo model and training it, and using the trained improved Yolo model to extract features from the preprocessed image data set to obtain image features containing rich feature information;

[0065] In this embodiment, the improved Yolo model is as follows: the feature pyramid network FPN of the neck network Neck of the Yolo model is replaced by the path aggregation network PANet; and the ordinary convolution block of the path aggregation network PANet is replaced by a depth-separable convolution block; and at the same time, the CBAM attention mechanism module is inserted in the upsampling and downsampling of the path aggregation network PANet.

[0066] In this embodiment, the improved Yolo model extracts the image features F of the food state appearance image. I The expression can be expressed as:

[0067] F I =Yolo(I);

[0068] Where I represents the appearance image of the object state.

[0069] In this embodiment, Figure 3 As shown, the improved Yolo model includes an image input layer, a backbone network Backbone, a neck network Neck, and an output layer; wherein the backbone network Backbone adopts CSPDarknet53.

[0070] In this embodiment, since the forward calculation path of the improved Yolo model is long, the image feature information will be diluted once after each convolution layer. However, in the process of cooking food, the feature information that needs to be extracted in this embodiment is to be able to characterize the different maturity of the food appearance. Therefore, in this embodiment, by introducing the path aggregation network PANet in the neck network Neck, in addition to the top-down network structure, a bottom-up path aggregation module is added, and the standard convolution block is replaced with a depth-separable convolution block. After upsampling and downsampling, the CBAM attention mechanism is used for the output feature map to optimize the PANet pyramid. The depth-separable convolution can reduce the amount of calculation and parameters of the model, and CBAM is a lightweight mechanism module that can improve performance while ignoring its overhead.

[0071] S3), using the UMAP algorithm to reduce the dimension of the extracted image features, visualize the reduced-dimensional data, and grade the appearance maturity of the food cooking process based on the visualized data distribution diagram; specifically including the following steps:

[0072] S31) construct a graph in high-dimensional space, where each point is connected to its nearest neighbor; and the UMAP algorithm describes the similarity between points through probability distribution. For each point x i , calculate k-nearest neighbors and use the probability distribution p ij To represent the point x i and x jThe similarity between them is calculated using the Gaussian kernel function to calculate the probability distribution p ij :

[0073]

[0074] In the formula, d(x i ,x j ) represents point x i and x j The distance between i For the point x i The relevant bandwidth parameters are determined by binary search so that point x i The probability of the k-nearest neighbors is close to a preset value;

[0075] S32) Construct a graph in a low-dimensional space and use probability to describe the similarity between points. For each point y i , calculate its k-nearest neighbors, and use Student's t-distribution to calculate the probability distribution q ij To represent point y i and j The similarity between the probability distribution q ij The calculation formula is:

[0076]

[0077] S33), optimizing the low-dimensional embedding by the fuzzy cross entropy loss function between the minimum high-dimensional graph and the low-dimensional graph, wherein the fuzzy cross entropy loss function C is:

[0078]

[0079] In the formula, C represents the fuzzy cross entropy loss function, which is used to measure the similarity p between data points in high-dimensional space. ij The similarity q between these points in the low-dimensional space ij The difference between

[0080] S34), using stochastic gradient descent SGD to optimize the above objective function to approximate the projection of the manifold structure of high-dimensional data on the low-dimensional space, the same food forms different clusters in different cooking state dimensions on the two-dimensional image, based on which the different cooking states of various types of food can be intuitively judged, and graded according to the actual maturity state. In this embodiment, the maturity of the food is divided into three levels: undercooked, mature, and overcooked.

[0081] S4), construct a TCN time series model, and extract the temperature-time series feature F from the food temperature series data set through the TCN time series model T ; The temperature-time series characteristic F T The extraction can be expressed as:

[0082] F T =TCN(T);

[0083] T represents the food temperature sequence dataset; it is a dataset about heating time and food temperature;

[0084] The TCN time series model of this embodiment is composed of multiple temporal blocks, each of which includes causal convolution, dilated convolution, residual connection and normalization.

[0085] Among them, the causal convolution is expressed as:

[0086]

[0087] In the formula, y t represents the output of causal convolution, w i is the convolution kernel, b is the bias term, x t-i represents the time series at time point ti; k is the convolution kernel size;

[0088] Correspondingly, the dilated convolution is expressed as:

[0089]

[0090] In the formula, Z t is the output of the dilated convolution, and d is the dilation factor.

[0091] S5), fusing the image features and the temperature-time series features through a multimodal fusion layer to obtain multimodal features;

[0092] In this embodiment, the fusion of image features and temperature-time series features includes feature alignment, feature weight adjustment, and splicing operations. Among them, the feature alignment refers to the process of mapping features of different modes to the same reference frame. Through this process, the relationship and similarity between different features can be better captured; in this embodiment, the feature alignment uses dynamic time planning DTW to align image features and time series features.

[0093] The feature weight adjustment is performed by inserting a lightweight multimodal attention module before the multimodal fusion layer to adjust the optimal weights of the image features and the temperature-time series features and perform weighted fusion. The expression of the weighted fusion is:

[0094] F fused =α(A i +C iT )+(1-α)(A T +C TI );

[0095] In the formula, F fused represents the weighted fusion feature, α is the learnable weight parameter, A I The self-attention weight matrix representing image features; C IT Represents the cross-attention weight matrix of image features to time series features; A T The self-attention weight matrix representing the time series features; C TI Represents the cross attention weight matrix of time series features on image features, indicating the influence of time series features on image features.

[0096] S6) Using the fused multimodal features to train the ViT model, and using the crested porcupine optimization algorithm (CPO) with an adaptive mechanism to intelligently find the optimal solution for the setting of hyperparameters of the ViT model training;

[0097] In this embodiment, the crested porcupine optimization algorithm CPO realizes the optimization process by simulating four defensive behaviors of crested porcupines. When facing predators, crested porcupines will adopt different defense strategies, which can be divided into two categories:

[0098] Global Exploration: Visual and audio threats.

[0099] Local Exploitation: Smell attacks and physical attacks.

[0100] In order to improve the convergence speed, this embodiment adds an adaptive mechanism to the crested porcupine optimization algorithm CPO. The adaptive mechanism is a method of adjusting parameters through dynamic feedback, which can effectively adjust the different weights of global exploration and local development in the algorithm.

[0101] By using the improved crown porcupine optimization algorithm (CPO) with an adaptive mechanism to automatically find the optimal settings of the hyperparameters of the ViT model, the ViT model can achieve the optimal solution when trained on images of different food cooking states, thereby obtaining a better detection model.

[0102] The specific steps include:

[0103] S61) Initialize algorithm parameter settings: Initialize the population size N, that is, randomly generate N individuals (solutions), each individual X i Random initialization within the search range; maximum number of iterations T max ; Initial search range [L, U]; Initial values ​​and adjustment strategies of adaptive parameters, where the parameters mainly include learning rate α, exploration probability p e and the development probability p d ;

[0104] S62) After setting the initial values ​​of the algorithm-related parameters, calculate the fitness value f(X) of each individual in the current population. i ), find the best individual X in the current population best ;

[0105] S63) For each individual X i , the CPO optimization algorithm will explore the probability p e and development probability p d Perform global exploration and local development with probability 1-p e -p d constant;

[0106] With probability p e Conduct global exploration: X′ i =X i +α×rand×(UL);

[0107] With probability p d Conduct local development: X′ i =X i +α×rand×(X best -X i );

[0108] S64) After the algorithm has explored each individual to varying degrees, the updated individual X′ i Limit search to: X′ i =min[max(X′ i ,L),U], and calculate the updated individual X′ i The fitness value f(X′ i ), if f(X′ i ) is better than f(X i ), then accept the new individual X′ i And update the best individual X in the current population best ;

[0109] S65) dynamically adjusting the adaptive parameters according to the current number of iterations t and the change of the fitness value;

[0110] Adaptively adjust the learning rate α:

[0111] S66) When the number of iterations reaches the maximum T max Or when the fitness value does not improve significantly, stop the iteration and output the best individual X best And its corresponding fitness value f(X′ i )

[0112] S7), constructing a mapping relationship between the food surface maturity value and the food center maturity value according to the food surface image feature information and time series;

[0113] S8) Using the trained ViT model to identify the cooking state in the food image to be detected, and based on the mapping relationship, to achieve real-time detection of the state change during the food cooking process and feedback of the corresponding maturity.

[0114] Example 2

[0115] like Figure 2 As shown, a food cooking state detection system based on cross-modal deep learning is provided, comprising:

[0116] A data collection module, used for collecting a food cooking data set, wherein the food cooking data set includes a food state appearance image data set and a food temperature sequence data set;

[0117] The image feature extraction module is used to extract features from the preprocessed image data set using the improved Yolo model to obtain image features containing rich feature information;

[0118] A maturity grading module is used to reduce the dimension of the image features extracted by the image feature extraction module, visualize the reduced-dimensional data, and grade the appearance maturity of the food during the cooking process based on the visualized data distribution diagram;

[0119] The temperature-time series feature extraction module is used to extract temperature-time series features from the food temperature series dataset through the TCN time series model;

[0120] A feature fusion module is used to obtain multimodal features by fusing image features and temperature-time series features;

[0121] The cooking status detection module is used to identify the cooking status in the food image.

[0122] As a preferred embodiment of this embodiment, the improved Yolo model is: the feature pyramid network FPN of the neck network Neck of the Yolo model is replaced by the path aggregation network PANet; and the ordinary convolution block of the path aggregation network PANet is replaced by a depth-separable convolution block; at the same time, the CBAM attention mechanism module is inserted in the upsampling and downsampling of the path aggregation network PANet.

[0123] As a preferred embodiment of this embodiment, the maturity grading module uses the UMAP algorithm to reduce the dimension of the extracted image features.

[0124] As preferred embodiment of the present invention, the feature fusion of the feature fusion module includes feature alignment, feature weight adjustment, and splicing operations.

[0125] As a preferred embodiment of this embodiment, the cooking status detection module uses the ViT model to identify the cooking status.

[0126] As a preferred embodiment of the present invention, the ViT model is trained using multimodal features, and an improved crested porcupine optimization algorithm CPO using an adaptive mechanism is used to intelligently find the optimal solution for the setting of hyperparameters of the training ViT model.

[0127] As a preferred embodiment of the present invention, the cooking status detection module constructs a mapping relationship between the food surface maturity value and the food center maturity value based on the food surface image information and the temperature-time information, thereby realizing real-time detection of the state changes of the food during cooking and feeding back the corresponding maturity.

[0128] The above embodiments and descriptions are only for illustrating the principles and best embodiments of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, all of which fall within the scope of the present invention to be protected.

Claims

1. A food cooking state detection method based on cross-modal deep learning, characterized in that: The following steps are involved: S1), collecting a food cooking data set, and dividing the food cooking data set into a training set and a test set after preprocessing; S2), constructing an improved Yolo model and training it, and using the trained improved Yolo model to extract features from the preprocessed image data set to obtain image features containing rich feature information; S3), using the UMAP algorithm to reduce the dimension of the extracted image features, visualize the reduced-dimensional data, and grade the appearance maturity of the food cooking process based on the visualized data distribution map; S4), constructing a TCN time series model, and extracting temperature-time series features from the food temperature series data set through the TCN time series model; S5), fusing the image features and the temperature-time series features through a multimodal fusion layer to obtain multimodal features; S6) Using the fused multimodal features to train the ViT model, and using the crested porcupine optimization algorithm (CPO) with an adaptive mechanism to intelligently find the optimal solution for the setting of hyperparameters of the ViT model training; S7), constructing a mapping relationship between the food surface maturity value and the food center maturity value according to the food surface image feature information and time series; S8) Using the trained ViT model to identify the cooking state in the food image to be detected, and based on the mapping relationship, to achieve real-time detection of the state change during the food cooking process and feedback of the corresponding maturity.

2. The method for detecting food cooking status based on cross-modal deep learning according to claim 1, characterized in that: In step S2), the improved Yolo model is: the feature pyramid network FPN of the neck network Neck of the Yolo model is replaced by the path aggregation network PANet; and the ordinary convolution block of the path aggregation network PANet is replaced by a depth-separable convolution block; and at the same time, the CBAM attention mechanism module is inserted in the upsampling and downsampling of the path aggregation network PANet.

3. The method for detecting food cooking status based on cross-modal deep learning according to claim 1, characterized in that: In step S3), the extracted image feature map is reduced in dimension using the UMAP algorithm to visualize the reduced-dimensional data, and the appearance maturity of the food cooking process is graded according to the visualized data distribution map, which specifically includes the following steps: S31) construct a graph in high-dimensional space, where each point is connected to its nearest neighbor; and the UMAP algorithm describes the similarity between points through probability distribution. For each point x i , calculate k-nearest neighbors and use the probability distribution p ij To represent the point x i and x j The similarity between them is calculated using the Gaussian kernel function to calculate the probability distribution p ij : In the formula, d(x i ,x j ) represents point x i and x j The distance between i For the point x i The relevant bandwidth parameters are determined by binary search so that point x i The probability of the k-nearest neighbors is close to a preset value; S32) Construct a graph in a low-dimensional space and use probability to describe the similarity between points. For each point y i , calculate its k-nearest neighbors, and use Student's t-distribution to calculate the probability distribution q ij To represent point y i and j The similarity between the probability distribution q ij The calculation formula is: S33), optimizing the low-dimensional embedding by the fuzzy cross entropy loss function between the minimum high-dimensional graph and the low-dimensional graph, wherein the fuzzy cross entropy loss function C is: Where C represents the fuzzy cross entropy loss function; S34) Use stochastic gradient descent SGD to optimize the above objective function to approximate the projection of the manifold structure of high-dimensional data on the low-dimensional space. The same food forms different clusters in different cooking state dimensions on the two-dimensional image. Based on this, the different cooking states of various types of food can be intuitively judged and graded according to the actual maturity state.

4. The method for detecting food cooking status based on cross-modal deep learning according to claim 1, characterized in that: In step S4), the TCN time series model is composed of multiple temporal blocks, each of which includes causal convolution, dilated convolution, residual connection and normalization.

5. The method for detecting food cooking status based on cross-modal deep learning according to claim 1, characterized in that: In step S5), the fusion of the image features and the temperature-time series features includes feature alignment, feature weight adjustment, and splicing operations.

6. The method for detecting food cooking status based on cross-modal deep learning according to claim 1, characterized in that: In step S5), the feature alignment uses dynamic time planning DTW to align image features and temperature-time series features; The feature weight adjustment is performed by inserting a lightweight multimodal attention module before the multimodal fusion layer to adjust the optimal weights of the image features and the time series features and perform weighted fusion; the expression of the weighted fusion is: F fused =α(A I +C IT )+(1-α)(A t +C TI ); In the formula, F fused represents the weighted fusion feature, α is the learnable weight parameter, A I The self-attention weight matrix representing image features; C IT A represents the cross-attention weight matrix of image features to time series features; T The self-attention weight matrix representing the time series features; C TI Represents the cross-attention weight matrix of time series features on image features.

7. A food cooking state detection system based on cross-modal deep learning, characterized in that: The system uses the detection method described in any one of claims 1 to 6 to detect the cooking status of food, and the system includes: A data collection module, used for collecting a food cooking data set, wherein the food cooking data set includes a food state appearance image data set and a food temperature sequence data set; The image feature extraction module is used to extract features from the preprocessed image data set using the improved Yolo model to obtain image features containing rich feature information; A maturity grading module is used to reduce the dimension of the image features extracted by the image feature extraction module, visualize the reduced-dimensional data, and grade the appearance maturity of the food during the cooking process based on the visualized data distribution diagram; The temperature-time series feature extraction module is used to extract temperature-time series features from the food temperature series dataset through the TCN time series model; A feature fusion module is used to obtain multimodal features by fusing image features and temperature-time series features; The cooking state detection module is used to identify the cooking state in the food image.

8. The food cooking state detection system based on cross-modal deep learning according to claim 7, characterized in that: The improved Yolo model is as follows: the feature pyramid network FPN of the neck network Neck of the Yolo model is replaced by the path aggregation network PANet; and the ordinary convolution block of the path aggregation network PANet is replaced by a depth-separable convolution block; and at the same time, the CBAM attention mechanism module is inserted in the upsampling and downsampling of the path aggregation network PANet.

9. The food cooking state detection system based on cross-modal deep learning according to claim 7, characterized in that: The cooking state detection module uses the ViT model to identify the cooking state; The ViT model is trained using multimodal features, and uses an improved crested porcupine optimization algorithm (CPO) with an adaptive mechanism to intelligently find the optimal solution for setting the hyperparameters of the training ViT model.

10. The food cooking state detection system based on cross-modal deep learning according to claim 7, characterized in that: The cooking state detection module constructs a mapping relationship between the food surface maturity value and the food center maturity value according to the food surface image information and temperature-time information, realizes real-time detection of the state change of the food during cooking and feeds back the corresponding maturity.

Images