Menu recommendation method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a menu recommendation method and device, equipment and a storage medium. The method comprises the steps of obtaining image features and smell features of inventory food material information; fusing the image features and the smell features to obtain freshness features of the stock food material information; inputting the freshness characteristics into a pre-trained freshness quantification model to obtain a freshness score of the inventory food material information output by the freshness quantification model and a confidence interval width corresponding to the freshness score; determining a confidence compensation factor according to the confidence interval width; determining the freshness weight of the inventory food material information according to the freshness score and the confidence compensation factor; and determining a target menu according to the inventory food material information and the fresh weight of the inventory food material information, and recommending the target menu. According to the embodiment of the invention, uncertainty in model judgment can be comprehensively considered, the practicability and the intelligent level of a recommendation result are improved, and the recommended menu has robustness while preferentially reducing waste.

Description

Technical Field

[0001] This application relates to the field of smart home technology, and in particular to a recipe recommendation method and apparatus, an electronic device, and a storage medium. Background Technology

[0002] Current intelligent food ingredient management solutions mainly rely on single image recognition technology for static freshness assessment. This method has significant limitations: First, judging the freshness of ingredients solely based on appearance is easily affected by environmental factors and is difficult to detect internal spoilage, resulting in insufficient accuracy. Second, it cannot capture the dynamic spoilage process of ingredients and lacks predictive capabilities. Finally, its recipe recommendation mechanism is usually quite simple, often aiming only to "clear inventory" without comprehensively considering factors such as uncertainties in the model's judgment. This leads to low practicality and insufficient intelligence in the recommendation results, and may even cause misleading results due to misjudgment, affecting the user experience. Summary of the Invention

[0003] This application provides a recipe recommendation method to solve or at least partially solve the above-mentioned problems.

[0004] Accordingly, embodiments of this application also provide a recipe recommendation device, an electronic device, and a storage medium to ensure the implementation and application of the above methods.

[0005] To address the aforementioned problems, this application discloses a recipe recommendation method, the method comprising: Acquire image and odor features of inventory food products; By fusing the image features and the odor features, the freshness features of the stocked food information are obtained; The freshness features are input into a pre-trained freshness measurement model to obtain the freshness score of the inventory food information output by the freshness measurement model and the confidence interval width corresponding to the freshness score. The confidence interval width reflects the credibility of the freshness score. A confidence compensation factor is determined based on the confidence interval width, and the confidence compensation factor is inversely proportional to the confidence interval width. The freshness weight of the inventory food information is determined based on the freshness score and the confidence compensation factor. The target recipe is determined and recommended based on the inventory ingredient information and the freshness weight of the inventory ingredient information.

[0006] This application also discloses a recipe recommendation device, the device comprising: The feature acquisition module is used to acquire image features and odor features of inventory food information; A feature fusion module is used to fuse the image features and the odor features to obtain the freshness features of the inventory food information; The score determination module is used to input the freshness features into a pre-trained freshness measurement model to obtain the freshness score of the inventory food information and the confidence interval width corresponding to the freshness score output by the freshness measurement model. A confidence determination module is used to determine a confidence compensation factor based on the confidence interval width, wherein the confidence compensation factor is inversely proportional to the confidence interval width; The weighting determination module is used to determine the freshness weight of the inventory food information based on the freshness score and the confidence compensation factor. The recipe recommendation module is used to determine target recipes based on the inventory ingredient information and the freshness weight of the inventory ingredient information, and to recommend the target recipes.

[0007] This application also discloses an electronic device, including: a processor; and a memory storing executable code thereon, which, when executed, causes the processor to perform one or more recipe recommendation methods as described in the embodiments of this application.

[0008] This application also discloses a machine-readable medium storing executable code thereon, which, when executed, causes a processor to perform one or more recipe recommendation methods as described in this application.

[0009] Compared with the prior art, the embodiments of this application have the following advantages: In this embodiment, image features and odor features of the stocked ingredients are acquired; the image features and odor features are fused to obtain the freshness features of the stocked ingredients; the freshness features are input into a pre-trained freshness measurement model to obtain the freshness score of the stocked ingredients and the corresponding confidence interval width, where the confidence interval width reflects the reliability of the freshness score; a confidence compensation factor is determined based on the confidence interval width, which is inversely proportional to the confidence interval width; the freshness weight of the stocked ingredients is determined based on the freshness score and the confidence compensation factor; and a target recipe is determined and recommended based on the stocked ingredients and their freshness weight.

[0010] This application significantly improves the accuracy of food freshness determination by fusing multimodal information such as image and odor features from inventory food information. A freshness quantification model is used to process freshness features, obtaining freshness scores for inventory food and their corresponding confidence interval widths. The wider confidence interval enhances the interpretability of the freshness scores, reflecting the system's grasp of the score judgments, avoiding misjudgments and making decisions more reliable, thus increasing user trust. Based on this, the freshness of food can be judged according to the freshness score, allowing users sufficient time to react. Furthermore, the freshness weight of inventory food is determined based on the freshness score and confidence compensation factor, thereby enabling recipe recommendations. This application comprehensively considers the uncertainties in model judgments, improving the practicality and intelligence of the recommendation results, making the recommended recipes both waste-reducing and robust. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating the steps of an embodiment of a recipe recommendation method according to this application; Figure 2 This is a flowchart illustrating an embodiment of a recipe recommendation method according to this application; Figure 3 This is a flowchart illustrating the feature processing of one embodiment of the recipe recommendation method of this application; Figure 4 This is a structural block diagram of an embodiment of a recipe recommendation device according to this application; Figure 5 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0012] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0013] In current technological solutions, intelligent food management systems typically utilize cameras installed inside the refrigerator to capture images of food within their field of view. These images are then analyzed using food type and freshness recognition models to assess the type and freshness of the food. The system combines user preference records with a pre-set recipe database to generate a recommended menu. After the user selects a specific dish from the recommendations, the system provides a complete recipe and a list of any missing ingredients from the refrigerator, based on the information in the recipe database.

[0014] However, current intelligent food management solutions mainly rely on single image recognition technology for static freshness assessment. This method has significant limitations: First, judging the freshness of food solely based on appearance is easily affected by environmental factors and is difficult to detect internal spoilage, resulting in insufficient accuracy. Second, it cannot capture the dynamic spoilage process of food and lacks predictive capabilities. Finally, its recipe recommendation mechanism is usually quite simple, often aiming only to "clear inventory" without comprehensively considering factors such as nutritional balance, user preferences, and uncertainties in model judgment. This leads to low practicality and insufficient intelligence in the recommendation results, and may even cause misleading results due to misjudgment, affecting the user experience.

[0015] Therefore, some embodiments of this application propose a recipe recommendation method, which mainly addresses the following problems existing in the prior art: 1. Problems with insufficient accuracy and robustness: How to overcome the limitations of a single image modality and achieve a more accurate and interference-resistant perception of food freshness.

[0016] 2. The problem of static assessment lacking foresight: How to upgrade from single-point detection to continuous monitoring, realize dynamic prediction of food spoilage trends, and transform passive notification into proactive early warning.

[0017] 3. The problem of single recommendation strategy and low reliability: How can the recommendation system not only reduce waste, but also balance nutrition and preferences, and identify and avoid the uncertainty risk of the model itself, so as to make smarter and more robust decisions?

[0018] Reference Figure 1 This is a flowchart illustrating the steps of an embodiment of a recipe recommendation method according to this application, including the following steps: Step 101: Obtain image features and odor features of the inventory ingredients.

[0019] In this embodiment, the inventory food information refers to the tags corresponding to the food items in the inventory, and the inventory food information is used to manage the inventory food. In one embodiment, the inventory food can be food stored in a smart refrigerator. The smart refrigerator can be equipped with a camera and an odor sensor to collect image data and odor data of the inventory food, thereby obtaining the image features and odor features of the current inventory food information based on the image data and odor data. The image features reflect the current visual features of the food corresponding to the inventory food information and the time-series sequence of visual features over a certain period of time, and the odor features reflect the current odor features of the food corresponding to the inventory food information and the time-series sequence of odor features over a certain period of time.

[0020] Step 102: Fuse the image features and the odor features to obtain the freshness features of the stocked food information.

[0021] The image features and odor features of the acquired inventory food information are fused to obtain the freshness features of the inventory food information, which are then used as input for the subsequent freshness measurement model.

[0022] Step 103: Input the freshness feature into the pre-trained freshness measurement model to obtain the freshness score of the inventory food information output by the freshness measurement model and the confidence interval width corresponding to the freshness score. The confidence interval width reflects the credibility of the freshness score.

[0023] The freshness measurement model is used to determine the freshness of ingredients corresponding to inventory information based on freshness features. Freshness is represented by a freshness score, and the confidence interval width corresponding to the freshness score reflects the reliability of the freshness score, that is, the degree of uncertainty of the freshness measurement model's judgment on that freshness score. The freshness features are input into a pre-trained freshness measurement model to obtain the freshness score of the inventory ingredient information and the corresponding confidence interval width.

[0024] For example, the output of a freshness measurement model can be a continuous freshness score between 0 and 100, along with the confidence interval width for that score. A higher freshness score indicates that the ingredients corresponding to the inventory information are fresher. The size of the confidence interval width reflects the degree of confidence of the freshness measurement model's judgment. A large confidence interval width (e.g., 85 ± 10 points, where 85 is the freshness score and 10 is the confidence interval width) indicates high uncertainty in the current judgment, which may be affected by environmental interference (such as light or obstruction); a small confidence interval width (e.g., 85 ± 2 points, where 85 is the freshness score and 2 is the confidence interval width) indicates a high degree of reliability in the judgment.

[0025] In one embodiment, the freshness quantification model can feed back the uncertainty reflected by the confidence interval width to the user, for example, prompting "The current judgment uncertainty is high, please adjust the lighting and try again".

[0026] In one embodiment, the fresh metric model can be a specially designed Bayesian Neural Network (BNN).

[0027] Step 104: Determine the confidence compensation factor based on the confidence interval width, wherein the confidence compensation factor is inversely proportional to the confidence interval width.

[0028] A confidence compensation factor corresponding to the confidence interval width is determined and used in the calculation of the freshness weight of ingredients corresponding to inventory information. In this embodiment, the confidence compensation factor is inversely proportional to the confidence interval width. Therefore, the higher the uncertainty of the freshness score, i.e., the wider the confidence interval, the smaller the confidence compensation factor, thereby reducing the freshness weight of ingredients corresponding to inventory information and avoiding misleading recipe recommendations due to unreliable freshness score judgments.

[0029] Step 105: Determine the freshness weight of the inventory food information based on the freshness score and the confidence compensation factor.

[0030] The freshness weight of the inventory food information is calculated. This freshness weight combines the freshness score and the confidence compensation factor obtained based on the confidence interval width, which reduces the risk of misjudgment and improves the accuracy of the freshness weight calculation.

[0031] In one embodiment, the less fresh the ingredient, i.e., the lower the freshness score, the higher the freshness weight. For freshness scores with a wide confidence interval (high uncertainty), their freshness weight is appropriately reduced to avoid misleading the recommendation system due to unreliable judgments. For freshness scores with a narrow confidence interval (high certainty), the freshness weight is calculated more heavily based on the freshness score. This allows the recommendation algorithm to assign a higher freshness weight to the inventory information of ingredients that are about to spoil, while reducing the risk of misjudgment.

[0032] Step 106: Determine the target recipe based on the inventory ingredient information and the freshness weight of the inventory ingredient information, and recommend the target recipe.

[0033] In one embodiment, in order to comprehensively consider nutrition and food consumption, a multi-objective optimizer is used when generating the target recipe. This optimizer takes into account the nutritional goals set by the user (such as calorie and protein intake) and the freshness weight of the above-mentioned inventory food information, and prioritizes recommending recipe combinations that can efficiently consume high-freshness weighted ingredients and meet the nutritional goals as the target recipe, thereby achieving the goal of proactively and intelligently reducing food waste.

[0034] This application significantly improves the accuracy of food freshness determination by fusing multimodal information such as image and odor features from inventory food information. A freshness quantification model is used to process freshness features, obtaining freshness scores for inventory food and their corresponding confidence interval widths. The wider confidence interval enhances the interpretability of the freshness scores, reflecting the system's grasp of the score judgments, avoiding misjudgments and making decisions more reliable, thus increasing user trust. Based on this, the freshness of food can be judged according to the freshness score, allowing users sufficient time to react. Furthermore, the freshness weight of inventory food is determined based on the freshness score and confidence compensation factor, thereby enabling recipe recommendations. This application comprehensively considers the uncertainties in model judgments, improving the practicality and intelligence of the recommendation results, making the recommended recipes both waste-reducing and robust.

[0035] Optionally, step 101 includes: According to the preset acquisition frequency, image data and odor data of the food corresponding to the inventory food information are simultaneously acquired at the target time to obtain image data sequence and odor data sequence; Extracting features from the image data sequence yields the image features of the inventory food information; The odor features of the inventory ingredients information are obtained by extracting features from the odor data sequence.

[0036] In this embodiment of the application, image data and odor data of the food corresponding to the inventory food information can be collected at the same target time according to a preset collection frequency. After multiple collections, an image data sequence and odor data sequence with time order and timestamp alignment are obtained for subsequent feature extraction.

[0037] Specifically, images of the stored ingredients can be captured using a mobile device or the refrigerator's built-in camera. These images must include the complete appearance of the ingredients. Simultaneously, the refrigerator's built-in metal-oxide-semiconductor (MOS) odor sensor collects data on the concentration of volatile organic compounds (VOCs) in the ingredients. The data collection process is automatically triggered at a fixed preset collection frequency (e.g., every 12 hours). This preset frequency is sufficient to capture significant changes in most ingredients under normal temperature or refrigeration conditions, indicating spoilage. Each collection includes a pair of data: an image of the ingredient and a set of synchronized odor sensor readings. In one embodiment, after acquiring the image and odor data, preprocessing is required. The image preprocessing module performs standardization operations on the image data, including size normalization, color space conversion, and noise removal, to improve the accuracy of subsequent feature extraction. The odor data undergoes filtering, noise reduction, and concentration normalization preprocessing.

[0038] After n acquisitions, two time series are obtained: an image data sequence (Image_Sequence): [Image_t1, Image_t2, ..., Image_tn], where tn represents the nth acquisition time point (i.e., the target time), and an odor data sequence (Odor_Sequence): [Odor_t1, Odor_t2, ..., Odor_tn], where each Odor_t is a vector containing the concentration readings of various VOCs (such as ethylene, hydrogen sulfide, and ammonia).

[0039] In this embodiment, it is necessary to ensure that the image data sequence and the odor data sequence are strictly aligned in timestamps, i.e., Image_ti and Odor_ti are acquired at the same target time. In practice, there may be a small but significant time difference between the instant the camera captures the image and the stable moment of the odor sensor reading. For ingredients whose freshness changes rapidly, this difference may affect the effectiveness of subsequent data fusion. Therefore, in one embodiment, a time tolerance window (e.g., 100 milliseconds) can be defined to pair image data and odor data. In another embodiment, a time storage window can also be set to determine which recent data needs to be retained and which outdated data needs to be deleted. For example, only image data and odor data from the most recent 30 days can be retained as image data sequence and odor data sequence to manage storage space and ensure the timeliness of the analysis.

[0040] In one embodiment of this application, to ensure strict alignment of image and odor data in terms of timestamps, a central scheduling module (such as the main control microprocessor inside a refrigerator) can be configured to issue a hardware synchronization signal at a target time (e.g., every 12 hours). The hardware synchronization signal is simultaneously sent to the camera and odor sensor via a dedicated control bus (such as I²C (Inter-Integrated Circuit) or SPI (Serial Peripheral Interface)), triggering both to begin data acquisition almost simultaneously. This hardware-level triggering fundamentally ensures that the start time of the acquisition action is consistent.

[0041] After each synchronization trigger, the central scheduling module generates a globally unique master timestamp for this acquisition event (e.g., a system clock synchronized based on the Network Time Protocol (NTP)). When the camera completes the capture and transmission of a frame of image data, the image data packet carries this master timestamp as its metadata. Simultaneously, after being triggered, the odor sensor enters a brief sampling window (e.g., lasting 5 seconds to obtain stable readings), and its output volatile organic compound concentration data vector (i.e., odor data) is also marked with the same master timestamp during packaging. Thus, image data and odor data are bound to the same spatiotemporal event point at the time of generation.

[0042] At the data receiving layer, a first-in-first-out (FIFO) buffer with a master timestamp is maintained. For each master timestamp, the system waits for both the image data packet and the odor data packet corresponding to that master timestamp to arrive before sending it as a complete multimodal data unit into subsequent processing. If a data packet fails to arrive on time due to transmission delays or other reasons, a short-term wait, retransmission request, or marking the acquisition as "incomplete data" and logging it after a timeout can be implemented according to the strategy. However, data from different timestamps will not be forcibly paired, thus ensuring that the image data sequences and odor data sequences used for subsequent analysis are strictly aligned between modalities.

[0043] A clock calibration procedure can also be run periodically to ensure that the local clocks of all modules are synchronized with the master clock, preventing minor drifts during long-term operation. For single acquisition failures caused by sensor malfunctions or transient interference, the subsequent data processing model itself has a certain degree of robustness to missing data, but the system will still record such events and use them as data quality information to potentially affect the confidence interval width of the subsequent freshness quantification model output. Thus, through the combined strategy of hardware triggering, software binding, and verification management, it can be ensured that each index t_i in the image sequence [Image_t1, Image_t2, ...] and the odor sequence [Odor_t1, Odor_t2, ...] precisely corresponds to the same target time in reality.

[0044] Features of the image data sequence and the odor data sequence are extracted separately to obtain the image features and odor features of the inventory food information.

[0045] According to the embodiment of this application, image data and odor data of the food corresponding to the inventory food information are collected simultaneously at the same target time according to the set collection frequency, so as to obtain image data sequence and odor data sequence, and feature extraction is performed for subsequent freshness score determination. Thus, by combining the multimodal data of time series images and odors, the time series data can reflect the dynamic trend of food spoilage, and realize a more accurate judgment of food freshness.

[0046] Optionally, the target time includes the current time, and the step of extracting the image features of the image data sequence to obtain the image features of the inventory food information includes: Determine the current image data at the current moment from the image data sequence; The current image data is input into a preset convolutional neural network to obtain the current image features; The image data sequence is input into a preset time series model to obtain time series image features; The current image features and the time-series image features are used as the image features of the inventory food information; The step of extracting the odor data sequence to obtain the odor features of the inventory food information includes: Determine the current odor data at the current moment from the odor data sequence; Determine the current odor characteristics based on the current odor data; The odor data sequence is input into the preset time series model to obtain time series odor features; The current odor feature and the temporal odor feature are used as the odor features of the inventory food information.

[0047] In this embodiment of the application, key features are extracted from image data sequences and odor data sequences for subsequent prediction of the changing trend of food freshness.

[0048] Specifically, the processing of image data sequences is as follows: Since the image data in the image data sequence is collected for multiple target times, including the current time and historical times, for the current image data Image_tn at the current time, a pre-trained pre-defined convolutional neural network (CNN, such as ResNet (Residual Network) or VGG (Visual Geometry Group)) is used to extract its high-dimensional visual feature vector V_curr as the current image feature. This vector encodes the appearance information of the food, such as color, texture, shape, and spots.

[0049] For the image data sequence [Image_t1, ..., Image_tn], the same CNN is used to extract features from each frame, resulting in an image feature sequence [V_t1, V_t2, ..., V_tn]. Temporal trend analysis is performed on this image feature sequence to capture the dynamic process of food spoilage, rather than just static moments. For example, an apple that scored 80 points yesterday and 75 points today—this "downward trend" itself is important information for judging its impending spoilage. Specifically, the image feature sequence [V_t1, V_t2, ..., V_tn] is input into a pre-defined temporal model (such as LSTM or Transformer). The pre-defined temporal model analyzes the dependencies in the image feature sequence and outputs a contextual feature vector that incorporates historical information. This can be taken as the hidden state of the last time step of the LSTM or the output corresponding to the [CLS] token of the Transformer, as the temporal image feature V_ctx.

[0050] Thus, the visual features V_curr (i.e., current image features) and the temporal context features V_ctx (i.e., temporal image features) based on visual history are obtained as image features for inventory food information.

[0051] The processing of odor data sequences is as follows: Since the odor data in the odor data sequence is collected for multiple target times, including the current time and historical times, the current odor data Odor_tn at the current time is usually already a structured numerical vector (such as sensor readings from four different gases). It can be directly used as the current odor feature S_curr, or it can be used as the current odor feature S_curr after simple standardization.

[0052] For an odor data sequence [Odor_t1, ..., Odor_tn], the same CNN is used to extract features from each frame of the odor data sequence, resulting in an odor feature sequence [S_t1, S_t2, ..., S_tn]. This odor feature sequence [S_t1, S_t2, ..., S_tn] is input into a pre-defined temporal model (such as LSTM or Transformer). The pre-defined temporal model analyzes the dependencies between the odor feature sequences and outputs a contextual feature vector that incorporates historical information. This vector can be the hidden state of the last time step of the LSTM or the output corresponding to the [CLS] token of the Transformer, serving as the temporal odor feature S_ctx. The pre-defined temporal model input to the odor feature sequence and the pre-defined temporal model input to the image feature sequence can be the same type of temporal model or different types of temporal models.

[0053] Thus, the odor feature S_curr (i.e., the current odor feature) and the temporal context feature S_ctx (i.e., the temporal odor feature) based on the odor history are obtained as the odor features of the inventory food information.

[0054] In this application embodiment, features such as current image features, time-series image features, current odor features, and time-series odor features are extracted from image data sequences and odor data sequences. These features reflect the current freshness status of the ingredients and the dynamic freshness change process, making the output of the subsequent freshness measurement model more accurate.

[0055] Optionally, step 102 includes: The current image features, the time-series image features, the current odor features, and the time-series odor features are concatenated to obtain a concatenated vector; The spliced ​​vector is input into a preset fully connected layer to obtain the freshness features of the inventory ingredients information.

[0056] In this embodiment of the application, to determine the freshness at the current moment, the image features and odor features obtained at the current moment need to be fused at the feature layer to form a multimodal fusion feature vector that can more comprehensively represent the state of the ingredients, which is then sent as the freshness feature to the freshness measurement model in the next stage.

[0057] Specifically, for the image and odor features at the current moment, there are four feature vectors, including the current image feature V_curr, the current odor feature S_curr, the time-series image feature V_ctx, and the time-series odor feature S_ctx. It is necessary to integrate information from different sources (images, odors) and different time dimensions (current moment, time-series trend) into a unified and more informative feature vector as the freshness feature.

[0058] In this embodiment, a "concatenation + fully connected layer" approach can be used to obtain freshness features. The four feature vectors—current image feature V_curr, current odor feature S_curr, time-series image feature V_ctx, and time-series odor feature S_ctx—are directly concatenated into a single concatenation vector F_concat = [V_curr; S_curr; V_ctx; S_ctx]. This concatenation vector is then passed through one or more pre-defined fully connected layers, allowing the network to automatically learn the weights and interactions between different features. The network ultimately outputs a fixed-dimensional multimodal fusion feature vector F_fused, representing the freshness features of the stored ingredients. The F_fused vector, which integrates comprehensive feature representations from current multimodal perception and historical time-series information, will be fed into the next stage of the freshness measurement model for accurate quantification and uncertainty estimation of freshness.

[0059] This application embodiment fuses current image features, time-series image features, current odor features, and time-series odor features to obtain unified and more informative freshness features, making the output of the subsequent freshness measurement model more accurate.

[0060] Optionally, step 102 includes: The current image features, the temporal image features, the current odor features, and the temporal odor features are used as food ingredient feature units; The food feature units are processed using a preset attention method to obtain the freshness features of the stocked food information.

[0061] In some other embodiments of this application, the current image feature V_curr, the current odor feature S_curr, the temporal image feature V_ctx, and the temporal odor feature S_ctx can also be fused through an attention mechanism.

[0062] Specifically, the four feature vectors are treated as a set of feature tokens, i.e., a set of food feature units, and these tokens are processed using a cross-modal attention layer or a Transformer encoder. The attention mechanism can dynamically calculate the importance of each feature for the final freshness judgment task. In one example, for judging a vegetable wrapped in plastic wrap, higher attention might be assigned to the odor features S_curr and S_ctx because the food is visually obscured; while for a piece of meat with obvious color changes, the model might pay more attention to the image features V_curr and V_ctx. The representations of all tokens output by the attention layer are aggregated (e.g., averaged) to obtain the final freshness feature F_fused of the inventory food information. The F_fused vector, which integrates the comprehensive feature representation of the current multimodal perception and historical time-series information, will be fed into the next stage of the freshness measurement model for accurate quantification of freshness and uncertainty estimation.

[0063] This application embodiment fuses current image features, time-series image features, current odor features, and time-series odor features to obtain unified and more informative freshness features, making the output of the subsequent freshness measurement model more accurate.

[0064] Optionally, step 103 includes: The freshness feature is input into the freshness measurement model, so that the freshness measurement model performs a preset number of forward propagations on the freshness feature to obtain a freshness score prediction set and a prediction uncertainty estimation set; Calculate the average value of the predicted freshness scores to obtain the freshness score of the inventory ingredients information; Calculate the variance of the freshness score prediction set to obtain the first confidence value; Calculate the average value of the set of prediction uncertainty estimates to obtain the second confidence value; Calculate the positive square root of the sum of the first confidence value and the second confidence value to obtain the target confidence value; The target confidence value, which is a preset multiple, is used as the confidence interval width corresponding to the freshness score.

[0065] In this embodiment of the application, the freshness feature F_fused is input into the freshness measurement model to obtain the freshness score and the corresponding target confidence value output by the freshness measurement model.

[0066] In this embodiment, the freshness metric model is a specially designed Bayesian neural network. Unlike conventional neural networks that output a single, deterministic value, the core of this Bayesian neural network lies in its probabilistic design. Each weight and bias in the Bayesian neural network is not a fixed value, but is modeled as a probability distribution (usually assumed to be a Gaussian distribution). This means that the freshness metric model itself has inherent uncertainty, and its parameters follow a certain distribution, thus making the output of the freshness metric model naturally a probability distribution as well.

[0067] In one alternative embodiment, a freshness measurement model is first trained before using it. Using a labeled real freshness dataset, the model parameters of the freshness measurement model to be trained are optimized using methods such as variational inference. The goal of training is to enable the freshness measurement model to learn to adjust its weight distribution (i.e., model parameters) so that, when inputting training data, its predicted freshness distribution can cover the true label values ​​with a high probability. The training process not only teaches the freshness measurement model the mapping relationship between food features and freshness scores, but also the degree of confidence in this mapping relationship.

[0068] Among them, the labeled real freshness dataset serves as training data and is a multimodal time-aligned sample set, providing a complete mapping from perceptual signals to food quality labels for the real-world spoilage process of food.

[0069] Specifically, the real freshness dataset includes multimodal time-series sensing data, training time-series image sequences, training time-series odor sequences, and metadata. The multimodal time-series sensing data serves as the model's input, simulating all food information collected by the system in actual deployment. The training time-series image sequences are high-quality images collected periodically (e.g., every 12 hours) throughout the entire lifecycle of the same batch of food samples (from fresh to completely spoiled). These images need to cover different lighting conditions, angles, and typical occlusion scenarios to ensure the model's robustness. The training time-series odor sequences are odor sensor data collected strictly synchronously with each training time-series image. They are typically a multidimensional vector containing the response values ​​(e.g., resistance changes, concentration estimates) of various metal oxide semiconductor sensors to target volatile organic compounds (such as ethylene, ethanol, ammonia, and hydrogen sulfide) from which they are collected. Metadata includes information such as the type of food (e.g., tomatoes, beef, lettuce), initial storage conditions (e.g., refrigerated at 4°C, room temperature at 25°C), and the time elapsed since the initial state. This information is crucial for the model to understand the spoilage dynamics of different foods.

[0070] The labeling of real freshness datasets consists of freshness scores defined by experts or reliable physicochemical measurements. These labels map sensory evaluations or physicochemical indicators to objective scores from 0 to 100. For example, trained reviewers might score the freshness of ingredients using a standardized scale based on criteria such as color, texture, odor, and appearance defects. Alternatively, laboratory measurements might be mapped to freshness scores. For vegetables, these measurements could include firmness obtained from a texture analyzer, soluble solids content obtained from a refractometer, and chlorophyll / anthocyanin content obtained from a spectrometer. For meat, measurements could include pH, volatile basic nitrogen content, and total bacterial count. General measurements could include water activity and concentrations of specific spoilage products. Freshness score labeling can also include label confidence, reflecting the uncertainty of the labeling process. This confidence level indicates the degree of certainty the labeler or measurement method itself has about the given freshness score. Label confidence helps the freshness measurement model being trained to better understand the inherent noise in the data. For example, for ingredients that are on the verge of spoilage, different reviewers may give them a score of 75 or 70, in which case the label confidence level is low.

[0071] Therefore, in this embodiment, the labeled real freshness dataset used for model training is organized around food samples. Each sample contains a series of data packets aligned by timestamps, which simultaneously record images, odor sensor data, and corresponding ground truth freshness values. This allows the model to learn both the static features and dynamic evolution patterns of the food's state. For example, a tomato is used as a food sample, and this sample contains a sequence of data packets aligned by timestamps: {timestamp t1:(image 1, sensor data 1, freshness ground truth 1), timestamp t2:(image 2, sensor data 2, freshness ground truth 2),...}.

[0072] In this embodiment, after the freshness features are input into the freshness measurement model, the freshness measurement model performs forward propagation on the freshness features a preset number of times to obtain a freshness score prediction set and a prediction uncertainty estimation set.

[0073] Specifically, during multiple random forward propagations, since the weights of a BNN are probabilistic, each forward propagation is equivalent to randomly sampling from the probability distribution of these weights, thus obtaining a temporary set of network parameters. For example, during the i-th random forward propagation, when the network performs computation on each layer with a probability distribution (such as a Bayesian fully connected layer), it independently samples a specific weight matrix W_i and bias b_i from the distribution of the weights of that layer (such as a Gaussian distribution N(μ_w,σ_w)) for this computation. Therefore, the network parameters used in each forward propagation are independent, random, and slightly different. The parameter set θ_1 (including the weight matrix and bias) used in the first propagation is a different random sample from the parameter set θ_2 (including the weight matrix and bias) used in the second propagation.

[0074] For the i-th propagation, the freshness feature F_fused is calculated using the parameter set θ_i from this random forward propagation, resulting in a pair of output values: one is the instantaneous prediction μ_i of the freshness score, and the other is the instantaneous estimate σ_i of the prediction uncertainty, i.e., the random uncertainty inherent in this prediction. This process is repeated T times (e.g., T=100, i.e., the preset number of times), resulting in two sets: the freshness score prediction set {μ1, μ2, ..., μ_T} and the prediction uncertainty estimate set {σ1, σ2, ..., σ_T}.

[0075] The average of the predicted freshness scores is calculated to obtain the freshness score of the inventory food information. In one embodiment, the final freshness score S of the inventory food information is the average of these T random predictions, calculated as S = (1 / T)*Σ(μ_i). The freshness score S of the inventory food information integrates all possible cognitive states of the freshness measurement model, is a robust point estimate, and represents the best guess of the freshness of the food by the freshness measurement model.

[0076] The overall prediction uncertainty σ_total of the freshness measurement model consists of two parts: cognitive uncertainty (i.e., the first confidence value) and random uncertainty (i.e., the second confidence value). Cognitive uncertainty stems from the freshness measurement model's own uncertainty regarding its parameters and is measured by the variance of the freshness score prediction set from T predictions, i.e., (1 / T)*Σ(μ_i - S). 2 The variance of the freshness score prediction set is calculated to obtain the first confidence value. The first confidence value reflects the degree of uncertainty of the freshness measurement model regarding the freshness score due to the lack of similar training data. The random uncertainty stems from the inherent, uninterpretable noise in the input data and is approximated by the average of the prediction uncertainty estimation set of T predictions, i.e., (1 / T) * Σ(σ_i) 2 ), calculate the average of the set of prediction uncertainty estimates, and obtain the second confidence value.

[0077] The final total variance is the sum of the first and second confidence values, i.e., σ_total 2 = (1 / T) * Σ(μ_i- S) 2 + (1 / T) * Σ(σ_i 2 The total uncertainty σ_total (i.e., the target confidence value) is σ_total. 2The confidence interval width for the freshness score is determined by the square root of the target confidence value, which is then used as the confidence interval width corresponding to the freshness score. In one example, the preset multiple can be 2, so the confidence interval width W is 2*σ_total, and the confidence interval for the freshness score S is [S - 2*σ_total, S + 2*σ_total]. The confidence interval intuitively expresses the degree of confidence the freshness measurement model has in its own judgment. The larger the confidence interval width, the more uncertain the freshness measurement model is; the smaller the confidence interval width, the more reliable the freshness score output by the freshness measurement model is.

[0078] In one embodiment, the system ultimately outputs a quantitative result such as "freshness score: 85±5". This result not only includes the objective freshness score, but also includes the self-evaluation of the freshness score judgment by the freshness measurement model. This allows the system to feed back uncertainty to the user and intelligently weigh the reliability of information in subsequent recipe recommendations, thereby making more robust and humane decisions.

[0079] This application uses a freshness quantification model to process freshness features, obtaining the freshness score of the stored ingredients and its corresponding confidence interval width. The confidence interval width enhances the interpretability of the freshness score, reflects the system's grasp of the score judgment, avoids erroneous guidance caused by misjudgment, makes the decision more reliable, and also enhances user trust.

[0080] Optionally, step 104 includes: Get the scaling factor; The confidence compensation factor is determined using the natural exponential function based on the scaling factor and the confidence interval width.

[0081] In this embodiment, a confidence compensation factor is determined based on the confidence interval width for subsequent calculation of the freshness weight. The confidence compensation factor plays the following role in the calculation of the freshness weight: for freshness scores with large confidence interval widths (i.e., high uncertainty in the freshness score), the freshness weight is appropriately reduced to avoid misleading the recommendation system due to unreliable freshness score judgments; for freshness scores with small confidence interval widths (i.e., high certainty in the freshness score), the freshness weight can be calculated almost entirely based on the freshness score. Therefore, the higher the uncertainty (i.e., the larger the confidence interval width W), the smaller the confidence compensation factor should be, thereby reducing the freshness weight of the inventory food information.

[0082] In this embodiment, to meet the above requirements, the confidence compensation factor can be designed as a decreasing function based on the confidence interval width W. Specifically, a scaling factor β is obtained, where β is greater than 0, to control the attenuation strength of uncertainty on the freshness weight. The larger β is set, the greater the penalty brought by uncertainty. Based on the scaling factor β and the confidence interval width W, the confidence compensation factor is determined using the natural exponential function exp, i.e., the confidence compensation factor can be expressed as exp(-β*W). Thus, when W is small (the judgment is certain), the confidence compensation factor is close to 1, and the freshness weight is almost unaffected; when W is large (the judgment is uncertain), the confidence compensation factor will be significantly less than 1, thereby greatly reducing the final freshness weight.

[0083] This application embodiment improves the accuracy of the freshness weight by setting a confidence compensation factor, thereby incorporating the uncertainty corresponding to the freshness score predicted by the freshness quantification model into the subsequent calculation of the freshness weight.

[0084] Optionally, step 105 includes: Obtain the smoothing factor; Calculate the reciprocal of the sum of the freshness score and the smoothing factor to obtain the intermediate weight value; The product of the intermediate weight value and the confidence compensation factor is used as the freshness weight of the inventory food information.

[0085] In this embodiment, the freshness weight of the inventory ingredients is determined based on the freshness score and confidence compensation factor, and is used in the subsequent recommendation of target recipes. The freshness weight needs to take into account both the freshness of the ingredients and the reliability of the model's judgment.

[0086] In one example, the freshness weight can be calculated using the following formula (1): Freshness weight = [1 / (freshness score + ε)] * (confidence compensation factor) (1) Here, ε is a smoothing factor, which is a very small positive number (such as 0.01) used to prevent division by zero errors when the freshness score is 0.

[0087] Specifically, obtain the freshness score S and the confidence interval width W. For example, for 85±5 points, S=85 and W=5. First, calculate the reciprocal of the sum of the freshness score and the smoothing factor to obtain the intermediate weight value, which is [1 / (freshness score + ε)] in formula (1). This is the basic part of the freshness weight calculation. The lower the freshness score S (the less fresh the ingredients), the larger the value of this item. This makes the less fresh ingredients get a higher freshness weight, so they will be recommended for consumption in the future.

[0088] In formula (1), the confidence compensation factor is the core component that integrates the uncertainty of the freshness score. The confidence compensation factor can be expressed as exp(-β*W). Therefore, the complete formula for calculating the freshness weight is formula (2): Freshness weight = [1 / (freshness score + ε)] * exp(-β * W) (2) Where ε is the smoothing factor, β is the scaling factor, exp is the natural exponential function, and W is the confidence interval width.

[0089] For example, if the freshness score of stale ingredient A is determined to be 20, and the confidence interval width is W=2, meaning the judgment of the freshness score is very certain, then according to formula (2), the freshness weight is calculated as ≈ [1 / (20+0.01)] * exp(-0.1*2) ≈0.05 * 0.82 = 0.041; if the freshness score of ingredient B is uncertain, and the confidence interval width is W=15, meaning the judgment of the freshness score is very uncertain, then according to formula (2), the freshness weight is calculated as ≈ [1 / (20+0.01)]*exp(-0.1*15) ≈ 0.05 * 0.22 = 0.011. Although ingredients A and B have the same freshness score, due to the high uncertainty of the system's judgment on B, its freshness weight is significantly reduced, and the system will prioritize recommending the consumption of ingredient A.

[0090] In this embodiment, the freshness weight of the stored ingredients is determined based on the freshness score and confidence compensation factor. Higher freshness weights are assigned to ingredients corresponding to the more certain information of the stored ingredients that are about to spoil, so that the ingredients corresponding to the information of these stored ingredients are consumed first in the subsequent recommendation of the target recipe.

[0091] Optionally, the inventory food information includes nutritional information of the food, and step 106 includes: Obtain candidate recipes and nutritional targets, wherein the candidate recipes include information on candidate ingredients in stock; The target recipe is determined using a preset target optimizer based on the nutritional objectives, the candidate recipes, the freshness weights corresponding to the candidate stock ingredients, and the nutritional information of the ingredients.

[0092] In this embodiment, a preset target optimizer is used to determine the target recipe based on the freshness weight of the inventory ingredients and the nutritional goals set by the user. The nutritional goals can be the calorie range, protein, carbohydrate, and other macronutrient intake of a meal. The preset target optimizer needs to find an optimal solution among the candidate recipes that simultaneously serves both the goal of minimizing waste and the goal of best aligning with the nutritional objectives.

[0093] Specifically, the process of the preset target optimizer finding an optimal solution among candidate recipes is modeled as a complex multi-objective combinatorial optimization problem. Its decision variables are a set of recipe combinations selected from a large recipe database as the target recipes, and the optimization objectives are twofold: first, to maximize the total freshness weight of the ingredients consumed by the selected recipes; and second, to minimize the total gap between the nutritional information of the ingredients in the recipe combination and the nutritional objectives. At the same time, the optimization calculation must ensure that the total amount of all ingredients required by the recipe combination does not exceed the current real-time inventory.

[0094] In one embodiment, a target function is constructed for a preset target optimizer, as shown in formula (3): Overall score = α * (total freshness weight of ingredients) - (1 - α) * (total nutritional gap) (3) The overall score is the score of the candidate recipe during the optimization process. α is a trade-off parameter between 0 and 1. When α is set close to 1, the optimization process prioritizes "reducing waste," even at the cost of compromising nutritional value to consume high-freshness ingredients. When α is close to 0, the optimization process strictly pursues nutritional goals and may temporarily ignore the urgency of consuming ingredients. The value of α can be set by the user according to their preferences, or dynamically adjusted by the system based on the situation (such as when family food waste is severe).

[0095] The calculation of the total ingredient freshness weight involves matching the ingredient lists required for all recipes in the candidate recipe pool with the inventory information and their freshness weights, and then summing the freshness weights of all consumed ingredients. The higher this value, the stronger the candidate recipe's ability to address the urgent waste problem.

[0096] Total nutritional gap is an error term that measures nutritional fit. It is calculated as the sum of the absolute or squared differences between the estimated total nutritional information (such as calories and protein) of the candidate recipe and the user's nutritional target values. The smaller this value, the more the candidate recipe meets the user's health needs.

[0097] In one embodiment, since the recipe database is so large that it is computationally infeasible to find the optimal solution by exhaustive search, a metaheuristic algorithm, such as a genetic algorithm, is used to efficiently explore this huge search space.

[0098] Specifically, the algorithm starts by randomly generating a "population" containing multiple candidate recipes. Feasibility checks are performed during the random generation of this initial candidate recipe "population." If a candidate recipe requires more of any ingredient than is in stock, a repair procedure is triggered. This involves automatically reducing the recipe portion size or replacing it with a similar ingredient in sufficient stock until all inventory constraints are met, thus ensuring that the starting point for optimization is a set of feasible solutions.

[0099] Subsequently, the algorithm enters an iterative evolutionary process: in each generation, the algorithm evaluates the comprehensive score of each candidate recipe according to formula (3). The candidate recipe with the higher score is selected and has a higher probability of participating in crossover (exchanging parts of each other's recipes) and mutation (randomly replacing a recipe) operations to generate new candidate recipes. When performing crossover and mutation operations to generate a new generation of candidate recipes, it is ensured that these genetic operations are performed within the feasible domain. For example, when a new candidate recipe is generated through crossover, if the candidate recipe causes an overspending of a certain ingredient (such as eggs), the system will intelligently remove the recipe that consumes the most eggs from the candidate recipes or replace it with a recipe that does not contain eggs, thereby fixing the constraint violation in place while creating a new candidate recipe. In an optional embodiment, when evaluating the comprehensive score of each candidate recipe, a penalty term is subtracted from the comprehensive score. This penalty term is proportional to the total overspending of all overspending ingredients. In other words, if a candidate recipe combination violates any inventory constraints, regardless of how well it performs in reducing waste and ensuring nutritional balance, its final score will be significantly lowered, thus leading to its natural elimination in the selection process. Through this mechanism, the search process of the optimization algorithm is strictly limited to the feasible solution space defined by the inventory. The algorithm no longer learns simply "which recipe combination is better," but rather "which recipe combination is optimal under the constraints of existing ingredients."

[0100] After several generations of natural selection, the population eventually converges to a Pareto optimal recommendation that strictly satisfies all hard inventory constraints, serving as the target recipe. At this point, the system will recommend the target recipe: "Based on your current inventory (only 2 eggs left), we suggest making 'Tomato Scrambled Eggs' (consuming tomatoes and eggs) and 'Stir-fried Cabbage' (consuming cabbage). This combination can efficiently consume your soon-to-be-stale tomatoes, and is completely within the inventory limit, while the estimated calorie intake is 480 kcal, which highly matches your goal." This application embodiment uses a preset target optimizer to determine the target recipe based on nutritional goals, candidate recipes, freshness weights corresponding to candidate inventory ingredient information, and ingredient nutritional information. This allows the target recipe to balance nutritional health and user preferences while prioritizing waste reduction.

[0101] In one embodiment, federated learning can be used in the decision-making process of this application embodiment to achieve continuous personalized optimization of target recipe recommendations while protecting user privacy.

[0102] To enable those skilled in the art to more clearly understand the recipe recommendation method shown in the embodiments of this application, through... Figure 2 , Figure 3 This application provides an explanation of a recipe recommendation method illustrated in an embodiment.

[0103] Reference Figure 2 This is a flowchart illustrating an embodiment of a recipe recommendation method according to this application.

[0104] like Figure 2 As shown, multimodal data acquisition is first performed, including image data acquisition and odor sensor data (i.e., odor data) acquisition. The acquired data is then preprocessed, such as image data preprocessing (including size normalization, color space conversion, and noise removal) and odor data preprocessing (including odor data filtering and normalization). The preprocessed data is used to construct time-series data, namely image data sequences and odor data sequences, which are stored as historical data sequences. After extracting image and odor features, multimodal feature fusion is performed to obtain freshness features.

[0105] Specifically, refer to Figure 3 This is a flowchart illustrating the feature processing of an embodiment of a recipe recommendation method according to this application.

[0106] Figure 3 The specific process for obtaining freshness features (i.e., fused feature vectors) is illustrated. The data acquisition process is automatically triggered according to a fixed preset acquisition frequency (e.g., every 12 hours). Each acquisition contains a pair of data: an image of the food and a set of odor data from synchronized odor sensor readings. After n acquisitions, two time series are obtained: an image data sequence (i.e., an image time series) and an odor data sequence (i.e., an odor data time series).

[0107] For the current image data at the current moment, a CNN is used to extract its high-dimensional visual feature vector V_curr as the current image feature. For the image data sequence, the same CNN is used to extract its high-dimensional visual feature vector for each frame in the image data sequence, resulting in an image feature sequence. LSTM or Transformer is then used to perform temporal trend analysis on the image feature sequence, analyzing historical feature sequences and predicting future trends. The hidden state of the last time step of the LSTM or the output corresponding to the [CLS] token of the Transformer can be taken as the temporal image feature V_ctx (i.e., temporal visual context feature).

[0108] The current odor data at the current moment is usually already a structured numerical vector, which can be directly used as the current odor feature S_curr, or it can be used as the current odor feature S_curr after simple standardization. For the odor data sequence, the same CNN is used to extract its features for each frame in the odor data sequence, resulting in an odor feature sequence. LSTM or Transformer is then used to perform temporal trend analysis on the odor feature sequence, analyzing historical feature sequences and predicting future trends. The hidden state of the last time step of the LSTM, or the output corresponding to the [CLS] token of the Transformer, can be taken as the temporal odor feature S_ctx (i.e., temporal odor context feature).

[0109] An attention mechanism or a fully connected layer is used to fuse the current image feature V_curr, the current odor feature S_curr, the time-series image feature V_ctx, and the time-series odor feature S_ctx into a unified, more informative feature vector as the freshness feature, which is then input into the Bayesian neural network in the next stage.

[0110] After obtaining the freshness feature as input, the next stage of the Bayesian neural network RNN ​​is as follows: Figure 2 As shown, the Bayesian neural network outputs a freshness score (ranging from 0 to 100) and a corresponding confidence interval (e.g., 85 ± 3, where 3 is the confidence interval width). After determining the confidence compensation factor based on the confidence interval width, the priority weight (i.e., freshness weight) is calculated according to formula (1). The freshness weight is combined with the user's nutritional goals (such as calorie and protein requirements), and a multi-objective optimization recommendation algorithm is executed using a target optimizer to generate an optimal strategy recommendation list as the target recipe, thereby prioritizing the consumption of high-weight ingredients while meeting the user's nutritional goals.

[0111] This application significantly improves the accuracy of food freshness determination by fusing multimodal information such as image and odor features from inventory food information. A freshness quantification model is used to process freshness features, obtaining freshness scores for inventory food and their corresponding confidence interval widths. The wider confidence interval enhances the interpretability of the freshness scores, reflecting the system's grasp of the score judgments, avoiding misjudgments and making decisions more reliable, thus increasing user trust. Based on this, the freshness of food can be judged according to the freshness score, allowing users sufficient time to react. Furthermore, the freshness weight of inventory food is determined based on the freshness score and confidence compensation factor, thereby enabling recipe recommendations. This application comprehensively considers the uncertainties in model judgments, improving the practicality and intelligence of the recommendation results, making the recommended recipes both waste-reducing and robust.

[0112] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0113] Based on the above embodiments, this embodiment also provides a recipe recommendation device, which can be applied to terminal devices, servers and other electronic devices.

[0114] Reference Figure 4 The diagram shows a structural block diagram of an embodiment of a recipe recommendation device according to this application, which may specifically include the following modules: Feature acquisition module 401 is used to acquire image features and odor features of inventory food information; Feature fusion module 402 is used to fuse the image features and the odor features to obtain the freshness features of the stocked food information; The score determination module 403 is used to input the freshness feature into a pre-trained freshness measurement model to obtain the freshness score of the inventory food information and the confidence interval width corresponding to the freshness score output by the freshness measurement model. The confidence determination module 404 is used to determine a confidence compensation factor based on the confidence interval width, wherein the confidence compensation factor is inversely proportional to the confidence interval width. The weight determination module 405 is used to determine the freshness weight of the inventory food information based on the freshness score and the confidence compensation factor. The recipe recommendation module 406 is used to determine a target recipe based on the inventory ingredient information and the freshness weight of the inventory ingredient information, and to recommend the target recipe.

[0115] Optionally, the feature acquisition module 401 includes: The data acquisition submodule is used to simultaneously acquire image data and odor data of the food corresponding to the inventory food information at a target time according to a preset acquisition frequency, so as to obtain image data sequence and odor data sequence. An image feature extraction submodule is used to extract features from the image data sequence to obtain image features of the inventory food information; The odor feature extraction submodule is used to extract features from the odor data sequence to obtain the odor features of the inventory food information.

[0116] Optionally, the target time includes the current time, and the image feature extraction submodule is specifically used for: Determine the current image data at the current moment from the image data sequence; The current image data is input into a preset convolutional neural network to obtain the current image features; The image data sequence is input into a preset time series model to obtain time series image features; The current image features and the time-series image features are used as the image features of the inventory food information; The odor feature extraction submodule is specifically used for: Determine the current odor data at the current moment from the odor data sequence; Determine the current odor characteristics based on the current odor data; The odor data sequence is input into the preset time series model to obtain time series odor features; The current odor feature and the temporal odor feature are used as the odor features of the inventory food information.

[0117] Optionally, the feature fusion module 402 includes: The feature splicing submodule is used to splice the current image features, the temporal image features, the current odor features, and the temporal odor features to obtain a splicing vector; The first freshness feature acquisition submodule is used to input the spliced ​​vector into a preset fully connected layer to obtain the freshness features of the inventory food information.

[0118] Optionally, the feature fusion module 402 includes: The feature unit acquisition submodule is used to use the current image features, the time-series image features, the current odor features, and the time-series odor features as food feature units; The second freshness feature acquisition submodule is used to process the food feature unit using a preset attention method to obtain the freshness features of the stocked food information.

[0119] Optionally, the score determination module 403 includes: The forward propagation submodule is used to input the freshness features into the freshness measurement model, so that the freshness measurement model performs forward propagation on the freshness features a preset number of times to obtain a freshness score prediction set and a prediction uncertainty estimation set. The freshness score calculation submodule is used to calculate the average value of the freshness score prediction set to obtain the freshness score of the inventory food information; The first confidence value calculation submodule is used to calculate the variance of the freshness score prediction set to obtain the first confidence value; The second confidence value calculation submodule is used to calculate the average value of the prediction uncertainty estimation set to obtain the second confidence value; The target confidence value calculation submodule is used to calculate the positive square root of the sum of the first confidence value and the second confidence value to obtain the target confidence value; The confidence interval width calculation submodule is used to use the target confidence value of a preset multiple as the confidence interval width corresponding to the freshness score.

[0120] Optionally, the confidence determination module 404 includes: The scaling factor acquisition submodule is used to obtain the scaling factor; The compensation factor determination submodule is used to determine the confidence compensation factor using a natural exponential function based on the scaling factor and the confidence interval width.

[0121] Optionally, the weight determination module 405 includes: The smoothing factor acquisition submodule is used to acquire the smoothing factor; The intermediate weight calculation submodule is used to calculate the reciprocal of the sum of the freshness score and the smoothing factor to obtain the intermediate weight value; The freshness weight calculation submodule is used to take the product of the intermediate weight value and the confidence compensation factor as the freshness weight of the inventory food information.

[0122] Optionally, the inventory of ingredients includes nutritional information, and the recipe recommendation module 406 includes: The recipe data acquisition submodule is used to acquire candidate recipes and nutritional targets, wherein the candidate recipes include candidate inventory ingredient information; The multi-objective optimization submodule is used to determine the target recipe using a preset target optimizer based on the nutritional objective, the candidate recipe, the freshness weight corresponding to the candidate inventory ingredient information, and the ingredient nutritional information.

[0123] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.

[0124] This application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In this application, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0125] The embodiments of this disclosure can be implemented as an apparatus configured as desired using any suitable hardware, firmware, software, or any combination thereof, including electronic devices such as terminal devices, servers (clusters), etc. Figure 5 An exemplary apparatus 500 is schematically shown that can be used to implement the various embodiments described in this application.

[0126] In one embodiment, Figure 5 An exemplary device 500 is shown, which includes one or more processors 502, a control module (chipset) 504 coupled to at least one of the processors 502, a memory 506 coupled to the control module 504, a non-volatile memory (NVM) / storage device 508 coupled to the control module 504, one or more input / output devices 510 coupled to the control module 504, and a network interface 512 coupled to the control module 504.

[0127] Processor 502 may include one or more single-core or multi-core processors, and processor 502 may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, device 500 can serve as a terminal device, server (cluster), or other device as described in the embodiments of this application.

[0128] In some embodiments, apparatus 500 may include one or more computer-readable media (e.g., memory 506 or NVM / storage device 508) having instructions 514 and one or more processors 502 that are combined with the one or more computer-readable media and configured to execute the instructions 514 to implement the module and thus perform the actions described in this disclosure.

[0129] In one embodiment, the control module 504 may include any suitable interface controller to provide any suitable interface to at least one of the processors 502 and / or any suitable device or component communicating with the control module 504.

[0130] The control module 504 may include a memory controller module to provide an interface to the memory 506. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0131] Memory 506 may be used, for example, to load and store data and / or instructions 514 for device 500. In one embodiment, memory 506 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, memory 506 may include double data rate type quad synchronous dynamic random access memory (DDR4 SDRAM).

[0132] In one embodiment, the control module 504 may include one or more input / output controllers to provide an interface to the NVM / storage device 508 and (one or more) input / output devices 510.

[0133] For example, NVM / storage device 508 may be used to store data and / or instructions 514. NVM / storage device 508 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drive (HDD), one or more optical disc (CD) drives, and / or one or more digital universal optical disc (DVD) drives).

[0134] NVM / storage device 508 may include storage resources that are physically part of a device on which device 500 is mounted, or that are accessible to the device but do not necessarily have to be part of the device. For example, NVM / storage device 508 may be accessed via a network through one or more input / output devices 510.

[0135] One or more input / output devices 510 may provide an interface for device 500 to communicate with any other suitable device. Input / output devices 510 may include communication components, audio components, sensor components, etc. A network interface 512 may provide an interface for device 500 to communicate via one or more networks. Device 500 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing a wireless network based on communication standards, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof.

[0136] In one embodiment, at least one of the processors 502 may be logically packaged with one or more controllers (e.g., memory controller modules) of the control module 504. In one embodiment, at least one of the processors 502 may be logically packaged with one or more controllers of the control module 504 to form a system-in-package (SiP). In one embodiment, at least one of the processors 502 may be integrated with the logic of one or more controllers of the control module 504 on the same die. In one embodiment, at least one of the processors 502 may be integrated with the logic of one or more controllers of the control module 504 on the same die to form a system-on-a-chip (SoC).

[0137] In various embodiments, device 500 may be, but is not limited to, a server, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). In various embodiments, device 500 may have more or fewer components and / or different architectures. For example, in some embodiments, device 500 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0138] The detection device can use a main control chip as a processor or control module, and sensor data, position information, etc. can be stored in a memory or NVM / storage device. The sensor group can be used as an input / output device, and the communication interface can include a network interface.

[0139] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0140] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0141] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0142] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable recipe recommendation terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0143] These computer program instructions can also be loaded onto a computer or other programmable terminal device, causing a series of operational steps to be performed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0144] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0145] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0146] The above provides a detailed description of a recipe recommendation method and apparatus, an electronic device, and a storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A recipe recommendation method, characterized in that, The method includes: Acquire image and odor features of inventory food products; By fusing the image features and the odor features, the freshness features of the stocked food information are obtained; The freshness features are input into a pre-trained freshness measurement model to obtain the freshness score of the inventory food information output by the freshness measurement model and the confidence interval width corresponding to the freshness score. The confidence interval width reflects the credibility of the freshness score. A confidence compensation factor is determined based on the confidence interval width, and the confidence compensation factor is inversely proportional to the confidence interval width. The freshness weight of the inventory food information is determined based on the freshness score and the confidence compensation factor. The target recipe is determined and recommended based on the inventory ingredient information and the freshness weight of the inventory ingredient information.

2. The method according to claim 1, characterized in that, The image features and odor features used to acquire information on stocked ingredients include: According to the preset acquisition frequency, image data and odor data of the food corresponding to the inventory food information are simultaneously acquired at the target time to obtain image data sequence and odor data sequence; Extracting features from the image data sequence yields the image features of the inventory food information; The odor features of the inventory ingredients information are obtained by extracting features from the odor data sequence.

3. The method according to claim 2, characterized in that, The target time includes the current time, and the step of extracting image features from the image data sequence to obtain the image features of the inventory food information includes: Determine the current image data at the current moment from the image data sequence; The current image data is input into a preset convolutional neural network to obtain the current image features; The image data sequence is input into a preset time series model to obtain time series image features; The current image features and the time-series image features are used as the image features of the inventory food information; The step of extracting the odor data sequence to obtain the odor features of the inventory food information includes: Determine the current odor data at the current moment from the odor data sequence; Determine the current odor characteristics based on the current odor data; The odor data sequence is input into the preset time series model to obtain time series odor features; The current odor feature and the temporal odor feature are used as the odor features of the inventory food information.

4. The method according to claim 3, characterized in that, The process of fusing the image features and the odor features to obtain the freshness features of the stocked food information includes: The current image features, the time-series image features, the current odor features, and the time-series odor features are concatenated to obtain a concatenated vector; The spliced ​​vector is input into a preset fully connected layer to obtain the freshness features of the inventory ingredients information.

5. The method according to claim 3, characterized in that, The process of fusing the image features and the odor features to obtain the freshness features of the stocked food information includes: The current image features, the temporal image features, the current odor features, and the temporal odor features are used as food ingredient feature units; The food feature units are processed using a preset attention method to obtain the freshness features of the stocked food information.

6. The method according to claim 1, characterized in that, The step of inputting the freshness features into a pre-trained freshness measurement model to obtain the freshness score of the inventory food information and the confidence interval width corresponding to the freshness score output by the freshness measurement model includes: The freshness feature is input into the freshness measurement model, so that the freshness measurement model performs a preset number of forward propagations on the freshness feature to obtain a freshness score prediction set and a prediction uncertainty estimation set; Calculate the average value of the predicted freshness scores to obtain the freshness score of the inventory ingredients information; Calculate the variance of the predicted freshness scores to obtain the first confidence value; Calculate the average value of the set of prediction uncertainty estimates to obtain the second confidence value; Calculate the positive square root of the sum of the first confidence value and the second confidence value to obtain the target confidence value; The target confidence value, which is a preset multiple, is used as the confidence interval width corresponding to the freshness score.

7. The method according to claim 1, characterized in that, The step of determining the confidence compensation factor based on the confidence interval width includes: Get the scaling factor; The confidence compensation factor is determined using the natural exponential function based on the scaling factor and the confidence interval width.

8. The method according to claim 7, characterized in that, The step of determining the freshness weight of the inventory food information based on the freshness score and the confidence compensation factor includes: Obtain the smoothing factor; Calculate the reciprocal of the sum of the freshness score and the smoothing factor to obtain the intermediate weight value; The product of the intermediate weight value and the confidence compensation factor is used as the freshness weight of the inventory food information.

9. The method according to claim 1, characterized in that, The inventory ingredient information includes nutritional information. Determining the target recipe based on the inventory ingredient information and its freshness weight includes: Obtain candidate recipes and nutritional targets, wherein the candidate recipes include information on candidate ingredients in stock; The target recipe is determined using a preset target optimizer based on the nutritional objectives, the candidate recipes, the freshness weights corresponding to the candidate stock ingredients, and the nutritional information of the ingredients.

10. A recipe recommendation device, characterized in that, The device includes: The feature acquisition module is used to acquire image features and odor features of inventory food information; A feature fusion module is used to fuse the image features and the odor features to obtain the freshness features of the inventory food information; The score determination module is used to input the freshness features into a pre-trained freshness measurement model to obtain the freshness score of the inventory food information and the confidence interval width corresponding to the freshness score output by the freshness measurement model. A confidence determination module is used to determine a confidence compensation factor based on the confidence interval width, wherein the confidence compensation factor is inversely proportional to the confidence interval width; The weighting determination module is used to determine the freshness weight of the inventory food information based on the freshness score and the confidence compensation factor. The recipe recommendation module is used to determine target recipes based on the inventory ingredient information and the freshness weight of the inventory ingredient information, and to recommend the target recipes.

11. An electronic device, characterized in that, include: processor; and A memory having executable code stored thereon, which, when executed, causes the processor to perform the recipe recommendation method as described in any one of claims 1-9.

12. A machine-readable medium having executable code stored thereon, which, when executed, causes a processor to perform the recipe recommendation method as described in any one of claims 1-9.

Images