Intelligent cooking doneness detection method based on multi-element perception

Abstract

This invention belongs to the field of intelligent cooking and food inspection technology. It discloses an intelligent cooking doneness detection method based on multi-modal perception. A multi-modal collaborative perception system is constructed, including visible vision, infrared temperature, hyperspectral imaging, and an electronic nose, to achieve synchronous acquisition of data across all dimensions of the cooking process. The PELT change point detection algorithm combined with a T-test is used to automatically classify doneness stages and verify their significance. Parallel feature extraction networks are designed for image, time-series, and spectral data, respectively, based on an improved ResNet18 structure to achieve deep extraction of visual, temperature, odor time-series, and hyperspectral features. A cross-modal attention mechanism is introduced to dynamically fuse multi-source features and weighted input to a fully connected classifier, achieving real-time, accurate, and non-destructive identification of cooking doneness. This invention solves the problems of traditional intelligent cooking equipment having single-sensor capabilities, relying on fixed programs, and being unable to dynamically adjust according to the real-time state of the ingredients.

Description

Technical Field

[0001] This invention belongs to the field of intelligent cooking and food testing technology, and relates to an intelligent cooking doneness detection method based on multi-sensor perception. Background Technology

[0002] With the development of smart homes and food industry automation, smart cooking equipment is widely used in home kitchens, chain restaurants, and central kitchens. These devices typically rely on preset electronic recipes and fixed time programs to prepare dishes. While this simplifies the operation process and achieves basic cooking automation, it still has significant shortcomings in dealing with the dynamic complexity of the cooking process.

[0003] Currently, modern cooking equipment, represented by smart microwave ovens, possesses functions such as precise temperature control, adjustable programs, and support for network connectivity and remote control. Some high-end cooking equipment can also achieve limited food status monitoring through built-in cameras or simple sensors. However, most of these devices still follow the control logic of preset time and temperature programs, lacking the ability to perceive the actual physical state and chemical composition changes of ingredients during cooking in real time and from multiple dimensions. They cannot dynamically adjust according to the true doneness of the food. Therefore, when dealing with actual variables such as the initial state of different ingredients, ambient humidity, and fluctuations in equipment power, it is still difficult to guarantee the consistency of cooking results and the optimization of flavor levels and texture.

[0004] The cooking process is essentially a dynamic process in which ingredients undergo a series of physical changes (such as moisture evaporation and protein denaturation) and chemical changes (such as Maillard reactions and lipid oxidation) after being heated. The initial state of the ingredients, environmental conditions, and even minor fluctuations in the equipment itself can cause the actual cooking trajectory to deviate from the preset program. Existing fixed processes lack the ability to sense and adjust to the real-time state of the ingredients, making it difficult to dynamically adjust according to the actual doneness. This is not only a key reason why many smart devices cannot reproduce the "wok hei" (wok aroma) and flavor levels of traditional cooking, but it also directly affects the consistency of the taste, texture, and quality of dishes, becoming a bottleneck restricting the development of intelligent cooking technology towards high-end and precise applications.

[0005] Currently, some studies have attempted to use single-sensor technology for cooking monitoring. For example, patent CN113974420A discloses a control method for intelligent cooking equipment based on infrared thermal imaging, which obtains the temperature distribution inside the pot by collecting and stitching multiple thermal images. However, it only relies on infrared visual information and does not combine multi-dimensional features such as smell or color, making it difficult to comprehensively assess the doneness of the ingredients. Patent CN108742170A proposes an intelligent oven recognition system based on computer vision, which identifies the type, thickness, and fat content of meat through image analysis and is supplemented by temperature sensors to monitor the cooking process. However, its vision system is limited by the closed oven environment and does not integrate chemical change information such as smell, making it difficult to adapt to the doneness judgment in open cooking scenarios. Patent CN111820711A discloses an intelligent stir-fry device and method with multiple pots linked together, which processes different ingredients in stages through multiple independent woks and finally mixes them. However, its control logic mainly relies on preset recipe programs and temperature sensor feedback, lacking a comprehensive perception of the multi-dimensional state of the ingredients in real time, such as color, texture, internal components, and smell. It cannot dynamically adjust and precisely control the ingredients according to their real and multi-dimensional doneness state during the cooking process. While the aforementioned patents have achieved effective monitoring in specific dimensions, none of them have achieved deep integration of multimodal information such as vision, temperature, and odor. They are difficult to adapt to the multidimensional and dynamic changes in the state of ingredients during cooking and cannot meet the need for comprehensive, real-time, and accurate assessment of food doneness.

[0006] Therefore, the existing technology has the following problems: intelligent cooking devices lack the ability to perceive the multi-dimensional ripeness status of ingredients in real time and accurately during the cooking process; ripeness judgment methods based on single sensor information have limitations and instability; existing multi-sensor data fusion methods fail to fully cover the key internal and external factors affecting ripeness (such as appearance color and texture, temperature distribution, internal composition structure, and volatile odors), and the fusion strategy fails to make full use of advanced technologies such as deep learning to adaptively optimize the feature extraction and fusion process, resulting in insufficient model generalization ability and the need to improve prediction accuracy.

[0007] Therefore, developing a method that can comprehensively utilize multi-source information such as vision, temperature, composition, and odor, and achieve real-time, accurate, and non-destructive detection of cooking doneness through advanced data fusion mechanisms, is of great significance for improving the perception and control level of intelligent cooking equipment. This study takes the typical cooking process of making egg pancakes as the specific research object, aiming to establish a general framework for multi-source perception and intelligent recognition methods, providing core technical support for the precise control of intelligent cooking. Summary of the Invention

[0008] The purpose of this invention is to overcome the technical shortcomings of existing intelligent cooking devices, such as their single sensing dimension and inability to dynamically adjust according to the real-time state of ingredients. This invention designs an intelligent cooking doneness detection method based on multi-modal perception. Taking the frying process of an egg pancake as a typical application scenario, this method achieves high-precision identification of cooking doneness by constructing a multi-modal perception system, designing a stage division strategy, designing a targeted deep feature extraction network, and designing a dynamic adaptive feature fusion mechanism. This provides core technology for closed-loop feedback control in intelligent cooking.

[0009] The technical solution of this invention: A smart cooking doneness detection method based on multiple sensing methods, comprising the following steps: Step 1: Design and build a multimodal collaborative sensing system to achieve full-dimensional data collection of cooking status; The multimodal collaborative sensing system includes a visible vision sensing module, a temperature sensing module, a hyperspectral sensing module, and an electronic nose sensing module; The visible vision sensing module uses an industrial camera, which is connected to a bracket via an adjustable universal joint, allowing it to be vertically suspended above the sample being tested. The temperature sensing module uses an infrared thermal imager, which is fixed to the upper right side of the cooking area. The optical axis of the infrared thermal imager is tilted at a 40° angle to the vertical direction towards the center of the cooking area, ensuring that the area where the food is located is completely within its detection field of view. The hyperspectral sensing module integrates a hyperspectral camera and a uniform light source, and is mounted on a bracket. The focal length of the hyperspectral camera is adjusted by the guide rail slide on the bracket. The electronic nose sensing module includes an air chamber, an air inlet, an air pump, an Arduino control unit, an air outlet, and a breadboard placement area. The air chamber contains an air inlet hose, a gas sensor, an air outlet hose, and a baffle, and is externally enclosed by four partitions forming a cuboid device. The air inlet and outlet are respectively located on opposite sides of the partitions. The Arduino control unit is used to collect and process the electrical signals output by the gas sensor. The breadboard placement area is used to arrange the sensor interface circuitry and wiring connections. The air chamber is an airtight, sealed, flat cavity, and a central point is fixed longitudinally along the airflow axis within the flat cavity. A strip-shaped baffle horizontally divides the internal space of the flat cavity into two symmetrical flow diversion channels to form a biomimetic nasal cavity-like flow diversion structure. Multiple gas sensors are symmetrically and alternately distributed on the two side walls of the strip-shaped baffle and the inner wall of the flow diversion channels. The axial ends of the flat cavity are connected to the air inlet and air outlet through air inlet hoses and air outlet hoses, respectively. The air inlet hose is connected to an air pump to drive the gas to be measured to be introduced through the air inlet. Under the obstruction and diversion of the strip-shaped baffle, the gas diffuses evenly along the two symmetrical flow diversion channels and flows fully through each gas sensor, and finally is discharged through the air outlet. The visible vision sensing module, temperature sensing module, hyperspectral sensing module, and electronic nose sensing module are connected to the computer via cables. The computer synchronizes the acquisition actions of the visible vision sensing module, temperature sensing module, hyperspectral sensing module, and electronic nose sensing module by sending a unified hardware trigger pulse, constructing a multimodal fusion dataset composed of images, temperature matrices, spectral reflectance curves, and odor signals that are completely corresponding in time. Furthermore, this multimodal collaborative sensing system has modular and portable structural characteristics. Each sensing module can be flexibly deployed according to the internal space and structural requirements of the actual equipment, thus enabling it to be easily embedded into various cooking equipment such as microwave ovens, ovens, and superheaters, achieving engineering applications.

[0010] Step 2: Design a data-driven adaptive segmentation of maturity stages; The design employs a data-driven adaptive segmentation of cooking doneness stages to characterize the dynamic process of food evolving from its initial state to its target doneness state. Step 1 involves acquiring a temperature matrix and then using an improved PELT change point detection algorithm for identifying cooking doneness stages. This algorithm identifies change points in the temperature change trend and uses these change points as the boundary positions of the doneness stages. During the change point identification process, a candidate segmentation scheme evaluation process is established to constrain and filter each candidate change point, thereby calculating and outputting the initial optimal segmentation scheme for the temperature time series data and the positions of each candidate change point. The improved PELT change point detection algorithm is as follows: Let the temperature time series be: in, This represents the temperature value of the temperature matrix corresponding to the nth sampling time. First, the stage average temperature, temperature change slope, and temperature fluctuation intensity are jointly introduced into its piecewise cost function for any candidate segment interval. Its piecewise cost function is defined as: in, This represents the average temperature of the i-th stage. This represents the slope of the temperature change in the i-th stage. This represents the intensity of temperature fluctuation in the i-th stage. and These are the weighting coefficients; Secondly, an adaptive penalty term related to local temperature change characteristics is set in the optimal piecewise objective function. The optimal piecewise objective function for the entire temperature time series is expressed as: in, , m represents the number of variable points. Represents the penalty term function; To avoid spurious changes caused by local temperature disturbances, a minimum stage duration constraint is introduced during the change point search process, ensuring that the minimum stage duration constraint is satisfied between two adjacent change points: in, A preset minimum stage duration is used to limit the minimum time interval between adjacent maturity points. Furthermore, to ensure clear maturity differentiation between adjacent maturity stages, a stage distinguishability constraint is introduced during the evaluation of candidate segmentation schemes, ensuring that adjacent maturity stages meet this constraint. Wherein, δ is the average temperature difference threshold between adjacent maturity stages, so that adjacent maturity stages reach the preset difference condition in temperature characteristics. Based on the improved PELT change point detection algorithm, after obtaining the initial optimal segmentation scheme and the location of each change point of the temperature time series, the T-test is further used to analyze the significant difference between the temperature time series of adjacent ripeness stages. The t-statistic and its corresponding P-value of adjacent ripeness stages are calculated and compared with the preset significance level to verify whether there is a statistically significant difference between adjacent ripeness stages. When the difference between adjacent ripeness stages does not meet the significance requirement, the corresponding segment boundaries are merged or corrected, and invalid segments without significant distinguishability are removed. Finally, the temperature interval jointly confirmed by the improved PELT change point detection algorithm and the T-test is defined as the corresponding ripeness stage, realizing the adaptive division of ripeness stages of the entire cooking process. The obtained ripeness stages are used as ripeness labels and mapped to the multimodal fusion dataset collected in step 1 to provide a ripeness-labeled training dataset for subsequent model training. Step 3: Design a multivariate data feature extraction network; The multivariate data feature extraction network employs four parallel feature extraction sub-modules: visible visual feature extraction module, temperature feature extraction module, electronic nose feature extraction module, and hyperspectral feature extraction module. These modules extract features based on the structural characteristics of the four modal data collected in step 1, and then use the global average pooling layer at the end of each module to uniformly reduce the extracted high-dimensional features to a 128-dimensional feature vector. The visible visual feature extraction module is a multi-scale spatially enhanced visual feature extraction structure based on ResNet18-2D-CNN. It retains the two-dimensional convolutional and residual backbone networks and connects multi-scale convolutional branch modules. The multi-scale convolutional branch modules are multi-branch parallel structures, including 3×3 convolutional branches, 5×5 convolutional branches, and 7×7 convolutional branches, which are used to extract local detail features, medium-scale region features, and large-scale overall structural features, respectively. The output of the multi-scale convolutional branch modules is concatenated through channels to form a multi-scale feature set, and then fused and compressed through 1×1 convolution to obtain a unified multi-scale visual feature representation, which is then input into the residual backbone network. Finally, a unified dimension visible visual feature vector is output. The temperature feature extraction module is based on an LSTM-based structure. The original single-path temperature time-series modeling method is improved to a three-path parallel LSTM branch: a short-time LSTM branch, a medium-time LSTM branch, and a long-time LSTM branch. The short-time LSTM branch corresponds to a time window that occupies 1 / 6 of the total window length, used to characterize local rapid fluctuations and instantaneous temperature rises. The medium-time LSTM branch corresponds to an equal time window that occupies 1 / 3 of the total window length, used to characterize phased temperature rise trends, plateau formation, and phase transitions. The long-time LSTM branch corresponds to the entire time window, used to characterize the overall thermal evolution trajectory and long-term dependence. Let N be the total time step of each food preparation process within the entire monitoring period. The time window length W of the short-time, medium-time, and long-time LSTM branches is defined using a proportional division method based on the global time step. The formula for calculating the time window length is as follows: in, This represents the time window length of the x-th LSTM branch. This indicates the rounding up operation. The window scaling factor corresponds to the LSTM branch, where s corresponds to the short-time LSTM branch, m corresponds to the medium-time LSTM branch, and l corresponds to the long-time LSTM branch. The outputs of the short-time LSTM branch, the medium-time LSTM branch and the long-time LSTM branch are fused by feature concatenation and then the dimension is unified by fully connected mapping. Then, the trend enhancement module generates trend weights by using fully connected mapping combined with Sigmoid gating, and the fused features are enhanced element by element to finally form a temperature feature vector with a unified dimension. The electronic nose feature extraction module is based on a multi-sensor odor feature extraction structure using LSTM. For inputs from multiple gas sensors, a sensor difference-sensing LSTM structure is constructed. At each time step, the original response vectors of each gas sensor are used as the current input, and sensor difference weights are generated through linear mapping and sigmoid activation. The current input is weighted element-wise to highlight the sensor channels that are more critical for maturity determination at the current moment; simultaneously, the response rate vector between the current moment and the previous moment is calculated. Then, the weighted sensor difference weights are calculated. Response rate of change vector Hidden state from the previous moment The LSTM gating computation process is introduced together, ultimately outputting an electronic nose feature vector of uniform dimension; the specific formulas for the forget gate, input gate, output gate, and candidate memory units are as follows: in, , and Let f(t) represent the forget gate, input gate, and output gate at time t, respectively. , , These represent the weight matrices of the corresponding gates; , , These represent the bias terms of the corresponding gates; Indicates the state of candidate memory cells. The weight matrix represents the candidate memory units. Indicates the bias term. Represents the hyperbolic tangent activation function; The hyperspectral feature extraction module is a band feature extraction structure based on ResNet18-1D-CNN. It performs a dimensionality transformation on ResNet18, replacing all two-dimensional convolutional kernels in the cascaded residual blocks used for processing two-dimensional images with one-dimensional convolutional kernels. Simultaneously, a band filtering module filters the spectral reflectance curves of the original input. This module employs a two-stage sensitive band filtering method combining mutual information pre-filtering and mRMR filtering. Finally, the band filtering weight vector is element-wise multiplied with the original spectral reflectance curve vector to obtain the optimized hyperspectral input. The weighted hyperspectral input then enters a one-dimensional convolutional residual backbone network to extract deep spectral features, ultimately outputting a unified-dimensional hyperspectral feature vector. Step 4: Design a feature dynamic fusion and decision-making model based on cross-modal attention mechanism; The visible visual feature vector, temperature feature vector, electronic nose feature vector, and hyperspectral feature vector extracted in step 3 are dynamically fused and classified. The four modal features correspond to changes in apparent color and texture, changes in heat distribution and temperature evolution, changes in volatile odor, and changes in internal composition and structure during the frying process, respectively. Let the four modal features be... , , and Construct a feature dynamic fusion and decision-making model based on cross-modal attention mechanism to adaptively fuse features of four modalities; First, a unified mapping is performed on the four types of modal features to obtain the mapped modal representations: in, and These are the mapping parameters for the corresponding modes. For the modal features after unified mapping; Subsequently, a cross-modal attention weight generation unit is used to calculate the importance score for each modality: in, , And u are learnable parameters. This represents the importance score of the y-th modality in the current sample; then, the fusion weights corresponding to each modality are obtained through the Softmax function: in, This represents the fusion weight of the y-th modality; After obtaining the fusion weights for each modality, the four types of modal features are dynamically weighted and fused to obtain the fused features: Furthermore, to preserve complementary information in the original features of each modality, a concatenated approach is used to construct joint features: Where [;] denotes vector concatenation; Finally, the joint features are input into the classification decision unit; the classification decision unit consists of a fully connected layer and an output layer, wherein the fully connected layer is used to further integrate the joint features and extract deep discriminative features: in, This is the weight matrix of the fully connected layer. Here, f is the bias term, and f is the activation function. Following the fully connected layer, an output layer is further configured to classify and predict the doneness stages of the egg pancake. Let the output be the predicted probability vector P corresponding to each doneness stage: in, and These are the output layer parameters, and P is the predicted probability distribution of each degree of doneness. The category with the highest probability is taken as the degree of doneness prediction result of the current sample, thereby realizing intelligent recognition of the current degree of doneness during the frying process. Step 5: Design the model training and optimization process to achieve high-performance familiarity recognition; Using the training dataset with maturity labels prepared in section 2, we designed an end-to-end training method for a feature dynamic fusion and decision model based on cross-modal attention mechanism, employing the cross-entropy loss function and the Adam optimizer. Through the designed learning rate strategy and batch training method, we stabilized and optimized the model parameters. Finally, on the independent test set, we designed a method to comprehensively evaluate the model performance using accuracy, precision, recall, and F1 score.

[0011] The beneficial effects of this invention are: (1) The present invention constructs a multimodal collaborative sensing system composed of visible vision, infrared temperature, hyperspectral and electronic nose, which realizes the synchronous acquisition of surface features, surface temperature distribution, spectral composition information and volatile odor evolution during the cooking process of food ingredients. It breaks through the problem of limited sensing dimensions and incomplete state representation of traditional single sensing methods, and improves the information completeness of ripeness detection.

[0012] (2) This invention innovates a data-driven maturity stage division strategy. Based on the traditional PELT change point detection algorithm, this strategy introduces a minimum stage duration constraint, a piecewise cost function combining temperature level, change slope, and fluctuation intensity, a stage distinguishability constraint, and an adaptive penalty term related to local temperature changes. It also combines a T-test to verify the significance of adjacent stages, which can effectively suppress spurious change points caused by short-term temperature disturbances, improve the accuracy of identifying true maturity inflection points, and make maturity stage division more objective, stable, and statistically based.

[0013] (3) The present invention constructs a parallel feature extraction network architecture for heterogeneous data and designs dedicated feature extraction branches for different modal data. Among them, the visible vision branch adopts a multi-scale parallel convolution structure to enhance the representation ability of image spatial features, the temperature branch adopts a multi-timescale LSTM and trend enhancement mechanism to enhance the modeling ability of thermal evolution law, the electronic nose branch adopts a sensor difference perception type LSTM structure to improve the recognition ability of key odor dynamic changes, and the hyperspectral branch adopts a sensitive band screening method combining mutual information and mRMR to enhance the response of key bands, thereby realizing efficient and targeted feature extraction of multi-source heterogeneous data.

[0014] (4) The present invention designs a feature dynamic fusion and decision model based on cross-modal attention mechanism, which can adaptively adjust the contribution weight of each modal feature at different maturity stages according to the current sample state, avoiding the problems of modal contribution imbalance and key feature dilution under traditional fixed weight fusion or simple splicing method, and improving the fusion feature expression ability, model robustness and maturity prediction accuracy. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall process of the method of the present invention. Figure 2 This is a schematic diagram of the structure of the multi-sensor system in this invention.

[0016] Figure 3 This is a schematic diagram of the electronic nose structure of the multi-sensor system in this invention.

[0017] Figure 4 This is a schematic diagram showing the detailed structure of the electronic nasal air chamber in the multi-sensor system of the present invention. Figure 5 This is a diagram showing the results of the ripeness stage division based on temperature data according to the present invention.

[0018] Figure 6 This is a visual image evolution diagram of the egg pancake frying process at each stage of doneness in an embodiment of the present invention.

[0019] Figure 7 This is an infrared thermal imaging evolution diagram of the egg pancake cooking process at each stage of doneness in an embodiment of the present invention.

[0020] Figure 8 This is an electronic nose odor response radar diagram of the egg pancake frying process at each stage of doneness in an embodiment of the present invention.

[0021] Figure 9 This is a hyperspectral reflectance curve of the egg pancake cooking process at different stages of doneness in an embodiment of the present invention.

[0022] Figure 10 This is a diagram showing the overall network structure of the maturity classification model constructed in this invention. Figure 11The figures are the accuracy and loss function curves of the model training process in this embodiment of the invention, where (a) is the accuracy curve and (b) is the loss function curve. Figure 12 This is the classification confusion matrix of the model on the test set in this embodiment of the invention.

[0023] In the diagram: 1-Industrial camera, 2-Infrared thermal imager, 3-Hyperspectral camera, 4-Sample under test, 5-Electronic nose, 6-Computer, 51-Air chamber, 52-Air inlet, 53-Air pump, 54-Arduino, 55-Air outlet, 56-Breadboard placement position, 511-Air inlet hose, 512-Sensor, 513-Air outlet hose, 514-Baffle structure. Detailed Implementation

[0024] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0025] like Figure 1 As shown, a smart cooking doneness detection method based on multiple sensing methods includes the following steps: Step 1: Design of collaborative acquisition of sample preparation and multi-sensor data; To construct the high-quality dataset required for the model, a standardized sample preparation process was first designed. This process used fresh eggs, with 60 g of beaten egg liquid accurately weighed for each sample, and the eggs were fried on one side for a total time of 15 minutes. Simultaneously, a data acquisition scheme with high spatiotemporal synchronization was designed. This scheme synchronously triggered all sensing modules to acquire data at fixed intervals of 30 seconds from the start of heating, obtaining samples at 30 consecutive time points. To ensure statistical reliability, 90 sets of repeated experiments were designed and executed, ultimately obtaining 2700 complete multivariate sample data sets.

[0026] The multi-sensing system, such as Figure 2 As shown, it includes the following modules: Step 1-1: Visible visual perception module; The design employs a DF-100 industrial camera to construct the vision acquisition unit, with a resolution of 1080×720 pixels and a frame rate of 30 fps. To optimize image quality, the camera is fixed 30 cm directly above the sample. The acquired RGB images are saved in JPEG format, and subsequently, through a designed image preprocessing workflow, the images are uniformly cropped to a central area of ​​420×420 pixels to effectively eliminate background interference.

[0027] Steps 1-2: Temperature sensing module; The design employs an Airui Optoelectronics AT61 infrared thermal imager 2 as a non-contact temperature sensing unit, fixed to the upper right side of the cooking area, with the imager's optical axis tilted at a 40° angle to the vertical direction towards the center of the cooking area. To obtain accurate two-dimensional temperature distribution, thermal images are acquired using its 640×512 pixel resolution and 8–14 μm response band. To accurately extract the temperature information of the egg pancake, a region extraction algorithm based on a circular mask is designed. This method generates a circular region of interest with a radius of 200 pixels, centered at the center of the thermal image (coordinates 320, 240). Only the temperature values ​​corresponding to the 640×512 pixels within this region are retained, while data outside the mask is invalidated, thus obtaining an effective temperature matrix characterizing the temperature field of the egg pancake.

[0028] Steps 1-3: Hyperspectral sensing module; The design employs an ImSpector V10E hyperspectral camera to construct the material composition sensing unit. Its spectral range covers 380.2–1020.2 nm, with a spectral resolution of 2.8 nm. To ensure spectral data quality, a rigorous black-and-white correction process was designed before acquisition, with an exposure time set to 10 ms and a gain of 1. The mathematical expression for black-and-white correction calculation is as follows: (1) in, For sample images, and These are black and white correction plate images. The central region of the image is extracted as the region of interest (ROI), and the average spectral reflectance is calculated.

[0029] Steps 1-4: Electronic nose sensing module; An electronic nose based on a multi-sensor array was designed and built. Its structural diagram is shown below. Figure 3 As shown, it includes an air chamber 51, an air inlet 52, an air pump 53, an Arduino, an air outlet 55, and a breadboard placement area. The detailed structure of the electronic nose air chamber is shown below. Figure 4 As shown, the system includes an intake hose 511, a sensor 512, an exhaust hose 513, and a baffle structure 514. Simultaneously, an array of eight gas sensors (MQ-138, TGS2616-C00, TGS2612-D00, TGS816, MQ-136, TGS2603, TGS2602, and MQ-137) is used to target characteristic volatile substances emitted during food cooking. After the system preheats for 5 minutes, the voltage signals (range 0–5 V) of each sensor are collected at 30-second intervals to continuously monitor the dynamic evolution of volatile odor substances during cooking.

[0030] Through the synchronous acquisition of the above modules, a multimodal fusion dataset consisting of images, temperature matrices, spectral reflectance curves, and odor signals that are completely corresponding in time is constructed. Step 2: Data-driven maturity stage segmentation; Based on the heat-receiving characteristics of different ingredients and different cooking requirements, a division rule is established to divide the cooking process into several continuous cooking stages to characterize the dynamic process of the ingredients evolving from the initial state to the target cooking state. For the temperature time series data collected in step 1, an improved PELT change point detection algorithm for identifying cooking cooking stages is used to identify key turning points in the temperature change trend and to use the key turning points as the boundary positions between adjacent cooking stages.

[0031] The improved PELT change point detection algorithm is as follows: Let the temperature time series be: (2) in, This represents the temperature value of the temperature matrix corresponding to the nth sampling time, where n=30 in this embodiment. First, the stage average temperature, temperature change slope, and temperature fluctuation intensity are jointly introduced into its piecewise cost function for any candidate segment interval. Its piecewise cost function is defined as: (3) in, This represents the average temperature of the i-th stage. This represents the slope of the temperature change in the i-th stage. This represents the intensity of temperature fluctuation in the i-th stage. and These are the weighting coefficients; Secondly, an adaptive penalty term related to local temperature change characteristics is set in the optimal piecewise objective function. The optimal piecewise objective function for the entire temperature time series is expressed as: (4) in, , m represents the number of variable points. The penalty term function is represented in this embodiment, where n=30, and the optimal number of change points m=5 is obtained after solving. To avoid spurious changes caused by local temperature disturbances, a minimum stage duration constraint is introduced during the change point search process, ensuring that the minimum stage duration constraint is satisfied between two adjacent change points: (5) in, In this embodiment, the minimum stage duration is preset. =5, to limit the minimum time interval between adjacent maturity points; further, to ensure that adjacent maturity stages have clear maturity distinction, a stage distinguishability constraint is introduced in the evaluation process of candidate segmentation schemes, so that adjacent maturity stages meet the stage distinguishability constraint: (6) Wherein, δ is the average temperature difference threshold between adjacent maturity stages. In this embodiment, δ=3℃, so that adjacent maturity stages reach the preset difference condition in temperature characteristics. Based on the improved PELT algorithm, after obtaining the initial optimal segmentation scheme and the positions of each change point of the temperature time series data, a T-test is further used to analyze the significant differences in temperature data between adjacent stages. The t-statistic and its corresponding p-value of adjacent stages are calculated and compared with a preset significance level to verify whether there is a statistically significant difference between adjacent stages. When the difference between adjacent stages does not meet the significance requirement, the corresponding segment boundaries are merged or corrected to remove invalid segments without significant distinguishing power. Finally, the adjacent temperature intervals confirmed by the improved PELT change point detection and the T-test are defined as the corresponding doneness stages, achieving adaptive division of doneness stages for the entire cooking process. The formula for calculating the T-test statistic is: (7) in, and These represent the average temperatures of two adjacent ripeness stages. and These represent the sample variances for the corresponding stages. and These represent the sample size for the corresponding stage; when the calculated P-value is less than the preset significance level α=0.05, it is determined that there is a statistically significant difference between adjacent stages.

[0032] The process of frying egg pancakes can be divided into six consecutive stages of doneness based on the aforementioned temperature time series data. The doneness division is as follows: Figure 5 As shown in the figure. The various maturity stages obtained are used as maturity labels and mapped to the multimodal fusion dataset collected in step 1 to provide a training dataset with maturity labels for subsequent model training; Step 3: Design a multivariate data feature extraction network; The ResNet18 backbone network is embedded with four dedicated feature extraction sub-modules: visible visual feature extraction module, temperature feature extraction module, electronic nose feature extraction module, and hyperspectral feature extraction module. The overall structure is as follows: Figure 10 As shown.

[0033] Step 3-1: Visible visual feature extraction module; This module takes a cropped and resized RGB visible image of an egg pancake as input. For example... Figure 6 As shown, this is a set of sample data randomly selected from the training set. The image, arranged from left to right, represents a stage with five images at each stage of the egg pancake-making process, captured at fixed 30-second intervals, providing the basic input for subsequent feature extraction. This module includes an input convolutional layer, a multi-scale convolutional branch module, a residual feature extraction module, a global average pooling layer, and a feature mapping layer. The input convolutional layer performs initial shallow feature extraction on the original image, preferably using a 7×7 convolutional kernel. The multi-scale convolutional branch module is located after the input convolutional layer and includes parallel 3×3, 5×5, and 7×7 convolutional branches. The 3×3 convolutional branch extracts subtle local information, edge changes, and small-scale texture features; the 5×5 convolutional branch extracts mid-scale regional structural features and mid-scale color transition features; and the 7×7 convolutional branch extracts large-scale overall contours, overall surface morphology, and global structural distribution features. Three convolutional branches perform convolution operations on the feature maps output from the input convolutional layer, resulting in three sets of feature maps with different receptive fields. These three sets of feature maps are then concatenated along the channel dimension and fused and compressed using a 1×1 convolution to obtain a unified multi-scale visual feature representation. Next, this unified visual feature representation is input into a residual feature extraction module. This module includes a max-pooling layer and four sets of residual convolutional layers, each composed of residual blocks, used to extract visual features layer by layer from shallow edge textures to deep semantic patterns. Finally, the output features are subjected to global average pooling and converted into a 128-dimensional visible visual feature vector through a fully connected mapping layer.

[0034] Step 3-2: Temperature Feature Extraction Module; This module takes time-series temperature data acquired by an infrared thermal imaging thermometer as input. For example... Figure 7 As shown, this is a set of sample data randomly selected from the training set. The image, arranged from left to right, consists of five images per stage, presenting infrared thermal images of the frying process of an egg pancake at fixed 30-second time intervals. This aims to reflect the spatial distribution and temperature changes of the overall temperature field, providing data support for the subsequent feature extraction network. This module constructs a three-way parallel temporal modeling structure—a short-time LSTM branch, a medium-time LSTM branch, and a long-time LSTM branch—based on the temperature time-series data acquired in step 1 and used for cooking stage segmentation in step 2. The formula for calculating the window length is as follows: (8) in, This represents the length of the input window for the x-th LSTM branch. This indicates the rounding up operation. Here, 's' represents the window scaling factor for each branch, 's' corresponds to the short-time LSTM branch, 'm' to the medium-time LSTM branch, and 'l' to the long-time LSTM branch. In this embodiment, the preferred window scaling factor for each branch is... =0.17 accounts for about 1 / 6 of the total time, which can ensure that the short-time branch accurately captures the local rapid temperature fluctuations caused by environmental disturbances or fine-tuning of heating power within a relatively short time window, without being masked by the general trend; =0.33, which accounts for about 1 / 3 of the total time, can extract the core phased trend features that characterize the alternation of food heating slope and isothermal plateau; =1.0 corresponds to the total time length, which is used to extract the macroscopic thermal evolution trajectory and total heat accumulation characteristics throughout the entire cooking process.

[0035] Since each set of temperature data contains 30 consecutive time steps and the sampling interval is 30 seconds, in this embodiment, the input window length of the short-time LSTM branch is fixed at 5 time steps to extract local rapid fluctuation features and instantaneous temperature rise features within the most recent 2.5 minutes; the input window length of the medium-time LSTM branch is fixed at 10 time steps to extract the stage-by-stage temperature rise trend features, plateau formation features, and stage transition features within the most recent 5 minutes; and the input window length of the long-time LSTM branch is fixed at 30 time steps to extract the overall thermal evolution trajectory features and long-term dependence features throughout the entire heating process.

[0036] Let the characteristics of the three LSTM branch outputs be as follows: , and First, feature concatenation is performed on the three outputs: (9) Then, a fully connected mapping layer is used for linear fusion and dimensionality unification to obtain the fused temperature features: (10) in, This is the weight matrix. For bias terms, This is the activation function.

[0037] A trend enhancement module is set up after fusion. This module generates trend weights using a fully connected mapping combined with Sigmoid gating. (11) in, Represents the trend weight vector. This represents the weight matrix of the trend enhancement module. Indicates the bias term. This represents the Sigmoid activation function. This indicates the temperature characteristics after fusion.

[0038] Then the trend weights are multiplied element-wise by the fusion features: (12) in, This indicates the enhanced temperature characteristics. This indicates element-wise multiplication. Represents the trend weight vector. This indicates the temperature characteristics after fusion.

[0039] This enhances the characteristic responses related to the direction of temperature change, heating rate, plateau formation, and phase transition, while suppressing characteristic components that contribute less. Finally, the enhanced temperature features are mapped into a 128-dimensional temperature feature vector.

[0040] Step 3-3: Electronic nose feature extraction module; This module uses the voltage timing signals collected by eight gas sensors in a self-made electronic nose as input. For example... Figure 8 The figure shows a visualization of a set of sample data randomly selected from the training set. This figure displays the average voltage data of five samples at each cooking stage on various sensors, reflecting the voltage signals of the odor sensors at each stage of the cooking process. It also provides data support for the subsequent electronic nose feature extraction network. This module targets the multi-sensor electronic nose time-series signal obtained in step 1. Let the original sensor input vector at time t be: (13) in, This represents the original input vector of the electronic nose sensor array at time t. Let represent the response value of the k-th sensor at time t, and 'a' represent the number of sensors. In this embodiment, there are 8 sensors, so a = 8.

[0041] To enhance the ability of different sensors to represent differences at different times, sensor difference weights are first generated through linear mapping and sigmoid activation: (14) in, This represents the sensor difference weight vector at time t. Represents the linear mapping weight matrix. Indicates the bias term. This represents the Sigmoid activation function. Then, the original input is element-wise weighted to obtain the weighted input vector: (15) Simultaneously, calculate the response rate vector between the current time and the previous time: (16) Then, the weighted input vector rate of change vector and the hidden state at the previous moment A common-input improved LSTM unit. The forget gate, input gate, and output gate are represented as follows: (17) (18) (19) in, , and Let f(t) represent the forget gate, input gate, and output gate at time t, respectively. , , These represent the weight matrices of the corresponding gates; , , These represent the bias terms of the corresponding gates; This means concatenating the weighted input vector, the rate of change vector, and the hidden state from the previous time step.

[0042] Candidate memory units are represented as follows: (20) in, Indicates the state of candidate memory cells. The weight matrix represents the candidate memory units. Indicates the bias term. This represents the hyperbolic tangent activation function.

[0043] The output in the hidden state is: (twenty one) Through the above improvements, the LSTM unit simultaneously considers the sensor's absolute response value, sensor difference weights, and response change rate information during state updates, thereby enhancing its ability to model the response and dynamic changes of key odor sensors. Finally, the temporal features of the electronic nose are globally aggregated and output as a 128-dimensional electronic nose feature vector through a fully connected mapping.

[0044] Steps 3-4: Hyperspectral feature extraction module; This module takes one-dimensional average spectral reflectance data as input. For example... Figure 9As shown, this figure presents a set of randomly selected sample data from the training set. It displays the one-dimensional average spectral reflectance response curves covering various stages of visible light (400-700 nm) and near-infrared light (above 700 nm) within the 380.2 nm to 1020.2 nm band, providing basic input for the subsequent hyperspectral feature extraction network. This module sets up a band filtering module before the one-dimensional convolutional backbone network for the hyperspectral reflectance data acquired and preprocessed in step 1, to reduce the interference of low-contribution and redundant bands on subsequent feature extraction. The band filtering module employs a two-stage sensitive band filtering method combining mutual information pre-screening and mRMR screening. First, the mutual information between each band in the original hyperspectral reflectance vector and its maturity label is calculated. Mutual information is defined as: (twenty two) Where X represents the feature of the band to be evaluated, Y represents the maturity label, p(x,y) represents the joint probability distribution, and p(x) and p(y) represent the marginal probability distributions, respectively. The larger the mutual information value, the stronger the correlation between the corresponding band and the maturity label. Based on this mutual information value, low-correlation bands are removed to obtain the candidate band set.

[0045] Subsequently, based on the principles of maximum correlation and minimum redundancy, the correlation between each candidate band and the maturity label, as well as its redundancy with the selected bands, are comprehensively evaluated within the candidate band set to obtain the mRMR score for each candidate band, which is defined as: (twenty three) in, This represents the current candidate band, y represents the maturity label, and S represents the set of selected bands. This indicates the number of bands in the selected band set, the first item. The first term represents the correlation between candidate bands and maturity labels, while the second term represents the average redundancy between candidate bands and the selected band set. A higher mRMR score indicates lower redundancy for the band while maintaining high discriminative power.

[0046] Furthermore, the mutual information score and mRMR score are normalized separately and then weighted and summed to obtain the comprehensive score for each band: (twenty four) in, The normalized mutual information score, The normalized mRMR score. and As the weighting coefficient, it is preferred to set... =0.5、 =0.5.

[0047] Subsequently, the comprehensive score is mapped to band selection weights using the Sigmoid function: (25) The band selection weight vector is obtained; in this embodiment, q=360. (26) Then, the band selection weight vector is multiplied element-wise with the original hyperspectral reflectance vector to obtain the optimized hyperspectral input: (27) Where R represents the original hyperspectral reflectance vector, This represents the optimized hyperspectral input after band filtering. The above steps enhance the response of sensitive bands and suppress interference from low-contribution and redundant bands. The weighted hyperspectral input enters a one-dimensional convolutional residual backbone network. This network includes an initial convolutional layer and multiple sets of one-dimensional residual modules to extract deep spectral features formed by local modes and their combinations in different bands. Finally, a 128-dimensional hyperspectral feature vector is output after passing through an average pooling layer and a fully connected mapping layer.

[0048] Step 4: Cross-modal attention feature fusion and classification; The visible visual features, temperature features, electronic nose features, and hyperspectral features extracted in step 3 are dynamically fused and classified. Specifically, step 3 outputs visible visual feature vectors, temperature feature vectors, electronic nose feature vectors, and hyperspectral feature vectors, respectively. These four types of features correspond to changes in apparent color and texture, heat distribution and temperature evolution, volatile odor, and internal composition and structure during the frying process of the egg pancake. Let the four modal features be... , , and Since the four types of modal features differ in dimensional structure, numerical range, and expression, simply concatenating them directly can easily lead to an imbalance in information contribution between modalities and dilution of key features. Therefore, this embodiment constructs a feature dynamic fusion and decision-making model based on a cross-modal attention mechanism to adaptively fuse the four types of modal features.

[0049] First, a unified mapping is performed on the four types of modal features to obtain the mapped modal representations: (28) in, and These are the mapping parameters for the corresponding modes. This step unifies the mapped modal features. It reduces the differences in dimensionality and distribution between different modalities, improving the comparability of subsequent fusion processes.

[0050] Subsequently, a cross-modal attention weight generation unit is used to calculate the importance score for each modality: (29) in, , And u are learnable parameters. This represents the importance score of the y-th modality in the current sample. The fusion weights for each modality are then obtained using the Softmax function. (30) in, This represents the fusion weight of the y-th modality. Through the above process, the model can adaptively evaluate the contribution of the four modalities—visual, temperature, electronic nose, and hyperspectral—to maturity discrimination based on the current sample state, rather than using fixed weights for static fusion.

[0051] After obtaining the fusion weights for each modality, the four types of modal features are dynamically weighted and fused to obtain the fused features: (31) Furthermore, to preserve complementary information in the original features of each modality, a concatenated approach is used to construct joint features: (32) Here, [;] represents vector concatenation. This method can highlight key modal information while preserving complementary features between different modalities.

[0052] Finally, the joint features are input into the classification decision unit. The classification decision unit includes a fully connected layer and an output layer, wherein the fully connected layer is used to further integrate the joint features and extract deeper discriminative features. (33) in, This is the weight matrix of the fully connected layer. is the bias term, and f is the activation function. In this embodiment, the joint features obtained by the cross-modal attention mechanism are further integrated through a fully connected layer, resulting in a 512-dimensional feature vector, which is used for classification in the subsequent output layer, enabling the model to learn more complex and discriminative maturity features.

[0053] Following the fully connected layer, an output layer is further configured to classify and predict the doneness stages of the egg pancake. Let the output be the predicted probability vector P corresponding to each doneness stage: (34) in, and These are the output layer parameters, and P is the predicted probability distribution of each ripeness stage. In this embodiment, P is a 6-dimensional vector. The category with the highest probability is taken as the ripeness stage prediction result of the current sample, thereby realizing intelligent recognition of the current ripeness state during the frying process of the egg pancake.

[0054] Step 5: Model training and performance evaluation; To train and validate the designed model, all 2700 datasets were randomly divided into training and test sets in an 8:2 ratio. The training configuration was as follows: batch size of 16 and initial learning rate of 1×10⁻⁶. -4 The training iterations were 100 rounds, the Adam optimizer was selected, and cross-entropy loss was used as the loss function for model training.

[0055] The accuracy versus loss function curves during training are as follows: Figure 11 As shown. The model stabilizes after 20 rounds, with accuracy exceeding 95% on both the training and validation sets. The confusion matrix for the test set is shown below. Figure 12 As shown, the overall accuracy reached 95.93%, the F1 score at each stage was greater than 0.94, and the recall rate was higher than 91%.

Claims

1. A method for intelligent cooking doneness detection based on multiple sensing methods, characterized in that, The steps are as follows: Step 1: Design and build a multimodal collaborative sensing system to achieve full-dimensional data collection of cooking status; The multimodal collaborative sensing system includes a visible vision sensing module, a temperature sensing module, a hyperspectral sensing module, and an electronic nose sensing module; The visible vision sensing module, temperature sensing module, hyperspectral sensing module, and electronic nose sensing module are connected to the computer via cables. The computer sends a unified hardware trigger pulse to synchronize the acquisition actions of the visible vision sensing module, temperature sensing module, hyperspectral sensing module, and electronic nose sensing module, and constructs a multimodal fusion dataset composed of images, temperature matrix, spectral reflectance curves, and odor signals that are completely corresponding in time. Step 2: Design a data-driven adaptive segmentation of maturity stages; Design a data-driven adaptive division of ripeness stages to characterize the dynamic process of food evolving from its initial state to its target ripeness state; The temperature matrix obtained in step 1 is used to identify the change points in the temperature change trend using a PELT change point detection algorithm that is improved for the identification of cooking doneness stages. The change points are used as the boundary positions of the doneness stages. During the change point identification process, a candidate segmentation scheme evaluation process is established to constrain and screen each candidate change point, so as to calculate and output the initial optimal segmentation scheme and the position of each candidate change point of the temperature time series data. Step 3: Design a multivariate data feature extraction network; The multivariate data feature extraction network employs four parallel feature extraction sub-modules: visible visual feature extraction module, temperature feature extraction module, electronic nose feature extraction module, and hyperspectral feature extraction module. These modules extract features based on the structural characteristics of the four modal data collected in step 1, and then use the global average pooling layer at the end of each module to uniformly reduce the extracted high-dimensional features to a 128-dimensional feature vector. Step 4: Design a feature dynamic fusion and decision-making model based on cross-modal attention mechanism; The visible visual feature vector, temperature feature vector, electronic nose feature vector, and hyperspectral feature vector extracted in step 3 are dynamically fused and classified. The four modal features correspond to changes in apparent color and texture, changes in heat distribution and temperature evolution, changes in volatile odor, and changes in internal composition and structure during the frying process, respectively. Let the four modal features be... , , and Construct a feature dynamic fusion and decision-making model based on cross-modal attention mechanism to adaptively fuse features of four modalities; Step 5: Design the model training and optimization process to achieve high-performance familiarity recognition; Using the training dataset with maturity labels prepared in section 2, we designed an end-to-end training method for a feature dynamic fusion and decision model based on cross-modal attention mechanism, employing the cross-entropy loss function and the Adam optimizer. Through the designed learning rate strategy and batch training method, we stabilized and optimized the model parameters. Finally, on the independent test set, we designed a method to comprehensively evaluate the model performance using accuracy, precision, recall, and F1 score.

2. The intelligent cooking doneness detection method based on multiple sensing according to claim 1, characterized in that, The visible vision sensing module uses an industrial camera, which is connected to a bracket via an adjustable universal joint, allowing it to be vertically suspended above the sample being tested. The temperature sensing module uses an infrared thermal imager, which is fixed to the upper right side of the cooking area. The optical axis of the infrared thermal imager is tilted at a 40° angle to the vertical direction towards the center of the cooking area, ensuring that the area where the food is located is completely within its detection field of view. The hyperspectral sensing module integrates a hyperspectral camera and a uniform light source, and is mounted on a bracket. The focal length of the hyperspectral camera is adjusted by the guide rail slide on the bracket. The electronic nose sensing module includes an air chamber, an air inlet, an air pump, an Arduino control unit, an air outlet, and a breadboard placement area. The air chamber contains an air inlet hose, a gas sensor, an air outlet hose, and a baffle, and is externally enclosed by four partitions forming a cuboid device. The air inlet and outlet are respectively located on opposite sides of the partitions. The Arduino control unit is used to collect and process the electrical signals output by the gas sensor. The breadboard placement area is used to arrange the sensor interface circuitry and wiring connections. The air chamber is an airtight, sealed, flat cavity, and a central point is fixed longitudinally along the airflow axis within the flat cavity. A strip-shaped baffle laterally divides the internal space of the flat cavity into two symmetrical flow-diverting channels, forming a biomimetic nasal cavity-like flow-diverting structure. Multiple gas-sensitive sensors are symmetrically and alternately distributed on the two side walls of the strip-shaped baffle and the inner wall of the flow-diverting channels. The axial ends of the flat cavity are connected to the air inlet and air outlet through air inlet hoses and air outlet hoses, respectively. The air inlet hose is connected to an air pump to drive the gas to be measured to be introduced through the air inlet. Under the obstruction and diversion of the strip-shaped baffle, the gas diffuses evenly along the two symmetrical flow-diverting channels and flows fully through each gas-sensitive sensor, and finally is discharged through the air outlet.

3. The intelligent cooking doneness detection method based on multiple sensing according to claim 1, characterized in that, The specific implementation process of step 2 is as follows: The improved PELT change point detection algorithm is as follows: Let the temperature time series be: in, This represents the temperature value of the temperature matrix corresponding to the nth sampling time. First, the stage average temperature, temperature change slope, and temperature fluctuation intensity are jointly introduced into its piecewise cost function for any candidate segment interval. Its piecewise cost function is defined as: in, This represents the average temperature of the i-th stage. This represents the slope of the temperature change in the i-th stage. This represents the intensity of temperature fluctuation in the i-th stage. and These are the weighting coefficients; Secondly, an adaptive penalty term related to local temperature change characteristics is set in the optimal piecewise objective function. The optimal piecewise objective function for the entire temperature time series is expressed as: in, , m represents the number of variable points. Represents the penalty term function; To avoid spurious changes caused by local temperature disturbances, a minimum stage duration constraint is introduced during the change point search process, ensuring that the minimum stage duration constraint is satisfied between two adjacent change points: in, A preset minimum stage duration is used to limit the minimum time interval between adjacent maturity points. Furthermore, to ensure clear maturity differentiation between adjacent maturity stages, a stage distinguishability constraint is introduced during the evaluation of candidate segmentation schemes, ensuring that adjacent maturity stages meet this constraint. Wherein, δ is the average temperature difference threshold between adjacent maturity stages, so that adjacent maturity stages reach the preset difference condition in temperature characteristics. Based on the improved PELT change point detection algorithm, after obtaining the initial optimal segmentation scheme and the location of each change point of the temperature time series, the T-test is further used to analyze the significant difference between the temperature time series of adjacent ripeness stages. The t-statistic and its corresponding P-value of adjacent ripeness stages are calculated and compared with the preset significance level to verify whether there is a statistically significant difference between adjacent ripeness stages. When the difference between adjacent ripeness stages does not meet the significance requirement, the corresponding segment boundaries are merged or corrected, and invalid segments without significant distinguishability are removed. Finally, the temperature interval jointly confirmed by the improved PELT change point detection algorithm and the T-test is defined as the corresponding ripeness stage, realizing the adaptive division of ripeness stages of the entire cooking process. The obtained ripeness stages are used as ripeness labels and mapped to the multimodal fusion dataset collected in step 1 to provide a ripeness-labeled training dataset for subsequent model training.

4. The intelligent cooking doneness detection method based on multiple sensing according to claim 1, characterized in that, The specific implementation process of step 3 is as follows: The visible visual feature extraction module is a multi-scale spatially enhanced visual feature extraction structure based on ResNet18-2D-CNN. It retains the two-dimensional convolutional and residual backbone networks and connects multi-scale convolutional branch modules. The multi-scale convolutional branch modules are multi-branch parallel structures, including 3×3 convolutional branches, 5×5 convolutional branches, and 7×7 convolutional branches, which are used to extract local detail features, medium-scale region features, and large-scale overall structural features, respectively. The output of the multi-scale convolutional branch modules is concatenated through channels to form a multi-scale feature set, and then fused and compressed through 1×1 convolution to obtain a unified multi-scale visual feature representation, which is then input into the residual backbone network. Finally, a unified dimension visible visual feature vector is output. The temperature feature extraction module is based on an LSTM-based structure. The original single-path temperature time-series modeling method is improved to a three-path parallel LSTM branch: a short-time LSTM branch, a medium-time LSTM branch, and a long-time LSTM branch. The short-time LSTM branch corresponds to a time window that occupies 1 / 6 of the total window length, used to characterize local rapid fluctuations and instantaneous temperature rises. The medium-time LSTM branch corresponds to an equal time window that occupies 1 / 3 of the total window length, used to characterize phased temperature rise trends, plateau formation, and phase transitions. The long-time LSTM branch corresponds to the entire time window, used to characterize the overall thermal evolution trajectory and long-term dependence. Let N be the total time step of each food preparation process within the entire monitoring period. The time window length W of the short-time, medium-time, and long-time LSTM branches is defined using a proportional division method based on the global time step. The formula for calculating the time window length is as follows: in, This represents the time window length of the x-th LSTM branch. This indicates the rounding up operation. The window scaling factor corresponds to the LSTM branch, where s corresponds to the short-time LSTM branch, m corresponds to the medium-time LSTM branch, and l corresponds to the long-time LSTM branch. The outputs of the short-time LSTM branch, the medium-time LSTM branch and the long-time LSTM branch are fused by feature concatenation and then the dimension is unified by fully connected mapping. Then, the trend enhancement module generates trend weights by using fully connected mapping combined with Sigmoid gating, and the fused features are enhanced element by element to finally form a temperature feature vector with a unified dimension. The electronic nose feature extraction module is based on a multi-sensor odor feature extraction structure using LSTM. For inputs from multiple gas sensors, a sensor difference-sensing LSTM structure is constructed. At each time step, the original response vectors of each gas sensor are used as the current input, and sensor difference weights are generated through linear mapping and sigmoid activation. The current input is weighted element-wise to highlight the sensor channels that are more critical for maturity determination at the current moment; simultaneously, the response rate vector between the current moment and the previous moment is calculated. Then, the weighted sensor difference weights are calculated. Response rate of change vector Hidden state from the previous moment The LSTM gating computation process is introduced together, ultimately outputting an electronic nose feature vector of uniform dimension; the specific formulas for the forget gate, input gate, output gate, and candidate memory units are as follows: in, , and Let f(t) represent the forget gate, input gate, and output gate at time t, respectively. , , These represent the weight matrices of the corresponding gates; , , These represent the bias terms of the corresponding gates; Indicates the state of candidate memory cells. The weight matrix represents the candidate memory units. Indicates the bias term. Represents the hyperbolic tangent activation function; The hyperspectral feature extraction module is a band feature extraction structure based on ResNet18-1D-CNN. It performs a dimensionality transformation on ResNet18, replacing all two-dimensional convolutional kernels in the cascaded residual blocks used for processing two-dimensional images with one-dimensional convolutional kernels. Simultaneously, a band filtering module filters the spectral reflectance curves of the original input. This module employs a two-stage sensitive band filtering method combining mutual information pre-filtering and mRMR filtering. Finally, the band filtering weight vector is element-wise multiplied with the original spectral reflectance curve vector to obtain the optimized hyperspectral input. The weighted hyperspectral input then enters a one-dimensional convolutional residual backbone network to extract deep spectral features, ultimately outputting a unified-dimensional hyperspectral feature vector.

5. The intelligent cooking doneness detection method based on multiple sensing according to claim 1, characterized in that, The specific implementation process of step 4 is as follows: First, a unified mapping is performed on the four types of modal features to obtain the mapped modal representations: in, and These are the mapping parameters for the corresponding modes. For the modal features after unified mapping; Subsequently, a cross-modal attention weight generation unit is used to calculate the importance score for each modality: in, , And u are learnable parameters. This represents the importance score of the y-th modality in the current sample; then, the fusion weights corresponding to each modality are obtained through the Softmax function: in, This represents the fusion weight of the y-th modality; After obtaining the fusion weights for each modality, the four types of modal features are dynamically weighted and fused to obtain the fused features: Furthermore, to preserve complementary information in the original features of each modality, a concatenated approach is used to construct joint features: Where [;] denotes vector concatenation; Finally, the joint features are input into the classification decision unit; the classification decision unit consists of a fully connected layer and an output layer, wherein the fully connected layer is used to further integrate the joint features and extract deep discriminative features: in, This is the weight matrix of the fully connected layer. Here, f is the bias term, and f is the activation function. Following the fully connected layer, an output layer is further configured to classify and predict the doneness stages of the egg pancake; let the output be the predicted probability vector P corresponding to each doneness stage: in, and These are the output layer parameters, and P is the predicted probability distribution of each degree of doneness. The category with the highest probability is taken as the degree of doneness prediction result of the current sample, thereby realizing intelligent recognition of the current degree of doneness during the frying process.

Images