Video-based cooking state recognition method and computer program product

Abstract

Embodiments of the present application provide a video-based cooking state recognition method and computer program product, comprising: obtaining a video frame sequence; performing feature extraction on the video frame sequence through a backbone network to obtain a plurality of layer feature maps; performing a learnable time sequence displacement operation on the feature maps to obtain a time sequence feature map; the learnable time sequence displacement operation is used to adaptively adjust the time sequence information interaction strength of the feature maps through a weighted displacement mechanism; performing classification prediction based on the time sequence feature map to obtain a probability distribution of each category; and outputting a final classification result based on the probability distribution of each category. In this way, the transition state sample can be effectively processed without explicitly predicting a stability value, without additional stability label or supervision signal, without an explicit threshold or artificial rule, the robustness and generalization ability of video state recognition are improved, and this method is particularly suitable for business scenarios such as cooking process and industrial monitoring which have hard classification requirements for transition states.

Description

Technical Field

[0001] This invention relates to the field of intelligent cooking technology, and in particular to a video-based cooking status recognition method and computer program product. Background Technology

[0002] Currently, video state recognition tasks often require discrete classification of continuously changing processes, a "hard classification" requirement that is prevalent in practical applications. Taking cooking process state recognition as an example, the continuous changes in the liquid inside the pot need to be classified into discrete categories such as "calm," "bubbling," "stable boiling," and "about to overflow." However, in practice, the same category often contains both typical stable state samples (such as truly calm liquid surface or stable boiling) and transitional or critical state samples (such as the liquid surface just beginning to bubble or the critical state transition stage).

[0003] Most existing video-based state recognition methods rely on predefined or fixed temporal modeling structures, such as spatiotemporal feature modeling methods based on 3D convolution, Temporal Shift Modules (TSMs) based on fixed temporal shifts, and temporal modeling methods based on self-attention mechanisms (such as Video Swin Transformer). These methods typically determine the modeling method and intensity of temporal information during the network structure design stage, lacking the ability to adaptively adjust to the differences in temporal dynamic characteristics of different states. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a video-based cooking state recognition method and computer program product that can adaptively adjust the temporal modeling intensity according to the temporal state of the samples and theoretically characterize "stability differences". Stability is not treated as a directly supervised output quantity, but rather as a latent state variable, which is naturally reflected through the training and optimization process. This allows the model to implicitly distinguish between stable and unstable states based on the degree of aggregation of sequence features in the time dimension without explicitly predicting state stability parameters, thus solving the common technical problem of hard classification of transitional states.

[0005] In a first aspect, embodiments of the present invention provide a video-based cooking state recognition method, the method comprising: acquiring a video frame sequence; wherein the video frame sequence includes multiple frames of images, each frame of image including information of the scene to be recognized; extracting features from the video frame sequence through a backbone network to obtain a multi-layer feature map; performing a learnable temporal shift operation on the feature map to obtain a temporal feature map; wherein the learnable temporal shift operation is used to adaptively adjust the temporal information interaction intensity of the feature map through a weighted shift mechanism; performing classification prediction based on the temporal feature map to obtain the probability distribution of each category; and outputting the final classification result based on the probability distribution of each category; wherein the classification result includes each category and its corresponding probability value.

[0006] In an optional embodiment of this application, the step of performing a learnable temporal displacement operation on the feature map to obtain a temporal feature map includes: obtaining learnable parameters characterizing the intensity of adaptively adjusting temporal modeling, and the channel partitioning ratio of displacement channels and non-displacement channels; wherein, the displacement channels characterize feature channels in the feature map that are displaced in the time dimension, and the non-displacement channels characterize feature channels in the feature map that are not displaced in the time dimension, and the displacement channels include: left-shift channels and right-shift channels; and performing a learnable temporal displacement operation on the feature map based on the channel partitioning ratio and the learnable parameters to obtain a temporal feature map.

[0007] In an optional embodiment of this application, the step of performing learnable temporal shifting operation on the feature map based on the channel partitioning ratio and learnable parameters includes: in the intermediate feature layer of the two-dimensional convolutional network, dividing the feature channels of the feature map into left-shifting channels, right-shifting channels, and non-shifting channels based on the channel partitioning ratio; performing temporal shifting on the left-shifting channels and right-shifting channels of the feature map based on the learnable parameters; wherein, the temporal shifting of the left-shifting channel is a forward shift, and the temporal shifting of the right-shifting channel is a backward shift; or, the temporal shifting of the left-shifting channel is a backward shift, and the temporal shifting of the right-shifting channel is a forward shift.

[0008] In optional embodiments of this application, the learnable parameters include: learnable parameters for the left-shift channel and learnable parameters for the right-shift channel; the step of temporally shifting the left-shift and right-shift channels of the feature map based on the learnable parameters includes: mapping each feature value of the left-shift channel of the feature map to a specified interval using a sigmoid function based on the learnable parameters of the left-shift channel; when the mapped feature value is a first value, performing a temporal shift on the feature value; when the mapped feature value is a second value, not performing a temporal shift on the feature value; mapping each feature value of the right-shift channel of the feature map to a specified interval using a sigmoid function based on the learnable parameters of the right-shift channel; when the mapped feature value is a first value, performing a temporal shift on the feature value; when the mapped feature value is a second value, not performing a temporal shift on the feature value.

[0009] In optional embodiments of this application, the above method further includes: jointly optimizing the learnable parameters through backpropagation and the main task loss function.

[0010] In an optional embodiment of this application, after the above-mentioned step of classifying and predicting the probability distribution of each category based on the temporal feature map, the method further includes: determining the state stability of the video frame sequence based on the probability distribution of each category; wherein, the state stability is used to characterize whether the video frame sequence has a feature center in the semantic space; the state stability of the video frame sequence is used as a latent state variable to model a cooking state recognition model, and the cooking state recognition model is jointly optimized by a classification loss function and a temporal compactness loss function; wherein, the temporal compactness loss function is used to constrain the distance between the feature and the feature center.

[0011] In optional embodiments of this application, the step of determining the state stability of a video frame sequence based on the probability distribution of each category includes: determining whether the time-slice features of the video frame sequence are distributed around the same feature center and whether there is state fluctuation based on the probability distribution of each category; if the samples of the video frame sequence are distributed around the same feature center and the fluctuation is low, then the sample is a stable state sample; if the samples of the video frame sequence are not distributed around the same feature center and the fluctuation is high, then the sample is a transitional state sample.

[0012] In an optional embodiment of this application, the method further includes: determining the feature center by exponential moving average.

[0013] In an optional embodiment of this application, the total loss function of the cooking state recognition model is determined by the following formula: ;in, For the total loss function, For classification loss function, Let be the temporal compactness loss function. The weights are preset for temporal compactness loss; the classification loss function is determined using the following formula: ;in, The preset scaling factor, Input feature vector With the correct category weight vector The angle between them For the edge of the angle, Input feature vector With error categories weight vector The angle between them; the temporal compactness loss function is determined by the following formula: ;in, The number of time slices in the video frame sequence. For time slice index, For the first Feature vectors of each time slice To which the sample belongs The feature center vector of the feature center vector; It is the square of the L2 norm, that is, the square of the Euclidean distance; For all time slices From 1 to Sum.

[0014] Secondly, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, can implement the above-described video-based cooking state recognition method.

[0015] The embodiments of the present invention bring the following beneficial effects: This invention provides a video-based cooking state recognition method and computer program product, which acquires a video frame sequence. The video frame sequence includes multiple frames, each containing information about the scene to be recognized. Features are extracted from the video frame sequence using a backbone network to obtain multi-layer feature maps. A learnable temporal shift operation is performed on the feature maps to obtain a temporal feature map. This learnable temporal shift operation adaptively adjusts the temporal information interaction intensity of the feature maps through a weighted shift mechanism. Classification prediction is performed based on the temporal feature maps to obtain the probability distribution of each category. The final classification result is output based on the probability distribution of each category, where the classification result includes each category and its corresponding probability value. This method effectively handles transitional state samples without explicit prediction of stability values, additional stability labels or supervision signals, or explicit thresholds or manual rules, improving the robustness and generalization ability of video state recognition. It is particularly suitable for business scenarios with hard classification requirements for transitional states, such as cooking processes and industrial monitoring.

[0016] Other features and advantages of this disclosure will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the techniques described above.

[0017] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a video-based cooking state recognition method provided in an embodiment of the present invention; Figure 2 A schematic diagram illustrating a video-based cooking state recognition method provided in an embodiment of the present invention; Figure 3 A schematic diagram of the model architecture of a video-based cooking state recognition method provided in an embodiment of the present invention; Figure 4 A schematic diagram of the reasoning process of a video-based cooking state recognition method provided in an embodiment of the present invention; Figure 5 A schematic diagram illustrating the training process of a video-based cooking state recognition method provided in an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the weighted displacement effect of a learnable temporal displacement provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Currently, in the field of video state recognition and classification, existing technologies mainly include methods based on traditional computer vision, video classification methods based on deep learning, and dedicated technical solutions for specific application scenarios.

[0022] However, the aforementioned existing technologies have the following main shortcomings: 1. Fixed temporal modeling strength: Existing video classification technologies based on methods such as TSM and 3D convolution have their temporal modeling strength determined during the network design stage. They cannot adaptively adjust according to the temporal characteristics of the samples (such as stable and transitional states), which leads to the introduction of unnecessary temporal perturbations to stable samples and insufficient modeling of transitional samples.

[0023] 2. Reliance on preset thresholds and rules: Most existing overflow detection methods rely on preset thresholds (such as point cloud change threshold, fluctuation value threshold, tilt angle threshold, etc.) and fixed rules to determine the state. They lack the ability to adaptively handle transition states, require manual setting of multiple thresholds, have high engineering complexity and poor generalization ability.

[0024] 3. Lack of stability perception mechanism: Existing technologies do not introduce the concept of "state stability" to characterize the temporal consistency differences of samples within a category, and cannot distinguish between typical stable states and transitional states, resulting in low recognition accuracy of transitional state samples.

[0025] 4. Insufficient feature space constraints: Existing technologies do not implicitly reflect stability information through geometric constraints in the feature space, and cannot distinguish samples with different levels of stability through the natural structure of feature distribution.

[0026] Based on this, the present invention provides a video-based cooking state recognition method and computer program product, specifically providing a stability-aware temporal constraint method for learnable temporal displacement, which can adaptively adjust the temporal modeling intensity according to the temporal state of the sample, and implicitly reflect stability information through feature space geometric constraints.

[0027] The purpose of this invention is to propose a video-based cooking state recognition method, comprising: adaptively adjusting the temporal modeling capability through learnable temporal displacement intensity, enabling the network to automatically adjust the temporal information interaction intensity according to sample characteristics; characterizing the temporal consistency of samples within a category through state stability perception modeling, distinguishing between typical stable states and transitional states; and constraining the geometric distribution of stable states in the feature space through a temporal compactness loss function, allowing stability information to be naturally reflected through the training and optimization process. Stability is no longer a supervised label, but rather the result of "whether the central constraint is satisfied". Therefore, it can effectively handle transitional state samples without explicitly predicting stability values, without additional stability labels or supervision signals, and without explicit thresholds or manual rules, improving the robustness and generalization ability of video state recognition. It is particularly suitable for business scenarios with hard classification requirements for transitional states, such as cooking processes and industrial monitoring.

[0028] To facilitate understanding of this embodiment, a video-based cooking state recognition method disclosed in this embodiment of the invention will first be described in detail.

[0029] Example 1: This invention provides a video-based cooking state recognition method, applicable to cooking equipment such as range hoods, stoves, and cookware. The cooking state includes boiling and / or overflow states; that is, this method can be used for both boiling and overflow detection during the cooking process. This invention can identify the continuous changes in liquid within the pot through video analysis, distinguishing between static stable states (e.g., calm liquid surface, empty pot), dynamic stable states (e.g., stable boiling), and critical transition states (e.g., bubbling, about to overflow). Furthermore, it can be extended to state monitoring in industrial processes (e.g., chemical reaction stage identification) and agricultural processes (e.g., crop growth stage identification), i.e., business scenarios in the industrial and agricultural fields with hard classification requirements for transition states, achieving accurate identification of discrete states in continuous processes.

[0030] Based on the above description, see Figure 1 The flowchart shown illustrates a video-based cooking state recognition method, which includes the following steps: Step S102: Obtain a video frame sequence; wherein the video frame sequence includes multiple frames of images, and each frame of images includes information about the scene to be identified.

[0031] See also Figure 2 The diagram shown illustrates a video-based cooking state recognition method. Figure 2 This embodiment demonstrates the complete inference path from the input video frame sequence to the output classification result, intuitively showing the data flow during inference: N-frame video frame sequence → multi-layer feature map → temporal feature map → probability distribution of each category → final distribution result.

[0032] like Figure 2 As shown, during the input phase, a sequence of N video frames can be input. Each frame contains information about the scene to be identified, such as information about the cookware and the state of the liquid in a cooking scene.

[0033] Step S104: Extract features from the video frame sequence using a backbone network to obtain a multi-layer feature map.

[0034] like Figure 2 As shown, in the feature extraction stage, features can be extracted from the input video frame sequence through backbone networks (such as MobileNetV2, ResNet, and ViT (Vision Transformer)) to obtain multi-layer feature map representations.

[0035] See also Figure 3The diagram shows a model architecture for a video-based cooking state recognition method. In this embodiment, a video frame sequence can be input into a backbone network, and multi-layer feature maps can be extracted through the backbone network.

[0036] Step S106: Perform a learnable temporal shift operation on the feature map to obtain a temporal feature map; wherein, the learnable temporal shift operation is used to adaptively adjust the temporal information interaction intensity of the feature map through a weighted shift mechanism.

[0037] like Figure 2 As shown, in the temporal modeling stage, learnable temporal shift operations can be applied to the extracted feature maps. The intensity of temporal information interaction can be adaptively adjusted through a weighted shift mechanism to obtain a weighted shift feature map (i.e., temporal feature map). This allows the network to automatically adjust the temporal modeling intensity according to the characteristics of the samples.

[0038] like Figure 3 As shown, this embodiment can input multi-layer feature maps into the temporal shift module and output temporal feature maps. In this embodiment, the temporal modeling intensity can be adaptively adjusted in the temporal shift module through learnable parameters. The channel partitioning ratio can be preset as needed, enabling the network to automatically adjust the temporal information interaction intensity based on sample characteristics.

[0039] Step S108: Based on the temporal feature map, perform classification prediction to obtain the probability distribution of each category.

[0040] like Figure 2 and Figure 3 As shown, in the classification prediction stage, the temporal feature map can be input into the classification layer (such as Arc Face, Cross Entropy, Center Loss, etc.) to obtain the probability distribution of each category.

[0041] Step S110: Output the final classification result based on the probability distribution of each category; wherein, the classification result includes each category and its corresponding probability value.

[0042] like Figure 2 As shown, in the output stage, the final classification results can be output, including each category and its corresponding probability value, such as: "Boiling without lid but not overflowing: 95%", "About to overflow without lid: 2%", etc.

[0043] In this embodiment, stability is not used as a supervised label; that is, no result is output indicating whether stability is achieved. Instead, the embodiment considers whether the central constraint is satisfied, thus indirectly characterizing stability. Therefore, this embodiment does not require explicit prediction of stability values, additional stability labels or supervisory signals, explicit thresholds or manual rules, and allows stability information to be naturally reflected through the training and optimization process.

[0044] This embodiment can simultaneously solve the state recognition problem of both boiling and overflow cooking states, and can accurately identify the specific state of boiling and / or overflow by determining whether the video sequence is stably present around a certain "state center" in the semantic space. It can identify and respond to more specific transition states, and can also give clear probability predictions for specific states including transition states.

[0045] This invention provides a video-based cooking state recognition method, which acquires a video frame sequence, wherein the video frame sequence includes multiple frames of images, each frame containing information about the scene to be recognized; features are extracted from the video frame sequence through a backbone network to obtain multi-layer feature maps; learnable temporal shift operations are performed on the feature maps to obtain temporal feature maps; wherein the learnable temporal shift operation is used to adaptively adjust the temporal information interaction intensity of the feature maps through a weighted shift mechanism; classification prediction is performed based on the temporal feature maps to obtain the probability distribution of each category; the final classification result is output based on the probability distribution of each category; wherein the classification result includes each category and its corresponding probability value. This method can effectively handle transitional state samples without explicit prediction of stability values, without additional stability labels or supervision signals, and without explicit thresholds or manual rules, improving the robustness and generalization ability of video state recognition. It is particularly suitable for business scenarios with hard classification requirements for transitional states, such as cooking processes and industrial monitoring.

[0046] Example 2: This embodiment provides another video-based cooking state recognition method, which is implemented based on the above embodiment. This embodiment focuses on describing the inference and training processes of the video-based cooking state recognition method. Figure 4 A schematic diagram illustrating the inference process of a video-based cooking state recognition method is shown. Figure 5 This diagram illustrates the training process of a video-based cooking state recognition method. Wherein, Figure 4 and Figure 5 The key technical points of this embodiment are shown in the figure, including: adaptive adjustment of temporal modeling intensity by learnable temporal displacement intensity, implicit representation of stability information by state stability-aware modeling, and temporal compactness loss constraint of the distance between features and feature centers.

[0047] 1. Learnable Shift Strength: like Figure 4 and Figure 5 As shown, in this embodiment, after the backbone network features are extracted, the multi-layer feature map is input into the temporal shift module. The temporal modeling intensity is adaptively adjusted through learnable parameters. The channel division ratio can be preset as needed, which enables the network to automatically adjust the temporal information interaction intensity according to the sample characteristics.

[0048] Typically, displacement channels account for 1 / 4 of the total channels, while non-displacement channels account for 3 / 4. The left-shift and right-shift channels are used for forward and backward swapping of feature channels, respectively, to achieve temporal feature fusion. The left and right shifts occur simultaneously, and the magnitude of the shift intensity can be learned.

[0049] 2. State stability-aware modeling: like Figure 5 As shown, stability information is reflected through implicit modeling, without the need for explicit prediction of stability values; stability is naturally reflected through the training and optimization process, rather than being a supervised output.

[0050] 3. Temporal compactness loss: like Figure 5 As shown, Temporal Center Loss constrains the distance between features of each time slice within the same video and the center of its respective category; features of stable state samples naturally cluster in time, while features of transitional state samples diverge in time; stability is no longer a supervised label, but rather the result of "whether the center constraint is satisfied".

[0051] In summary, through the synergistic effect of these three key technologies, effective processing of transition state samples is achieved without the need for explicit prediction of stability values, additional stability labels or monitoring signals, or setting manual thresholds.

[0052] In some embodiments, learnable parameters characterizing the adaptive adjustment of temporal modeling intensity, as well as the channel partitioning ratio of displacement channels and non-displacement channels, can be obtained; wherein, displacement channels characterize feature channels in the feature map that are displaced in the time dimension, and non-displacement channels characterize feature channels in the feature map that are not displaced in the time dimension, and displacement channels include: left-shift channels and right-shift channels; learnable temporal displacement operations are performed on the feature map based on the channel partitioning ratio and learnable parameters to obtain a temporal feature map.

[0053] In existing technologies, the temporal modeling intensity is determined during the network design phase and cannot be adaptively adjusted based on the temporal characteristics of the samples. This embodiment introduces learnable parameters to control the displacement intensity, enabling the network to automatically perform deep learning based on the data distribution: enhancing temporal information interaction in samples with significant dynamic changes and suppressing unnecessary temporal disturbances in samples with stable states. The channel partitioning ratio can be preset as needed.

[0054] In some embodiments, in the intermediate feature layer of a two-dimensional convolutional network, the feature channels of the feature map can be divided into left-shifted channels, right-shifted channels, and non-shifted channels based on the channel partitioning ratio; the left-shifted and right-shifted channels of the feature map can be temporally shifted based on learnable parameters; wherein the temporal shift of the left-shifted channel is a forward shift and the temporal shift of the right-shifted channel is a backward shift; or, the temporal shift of the left-shifted channel is a backward shift and the temporal shift of the right-shifted channel is a forward shift.

[0055] The temporal shifting module in this embodiment may include a two-dimensional convolutional network. Based on a video model containing a two-dimensional convolutional network, a temporal shifting operation is introduced in the intermediate feature layer: the channel dimension is divided into several subsets, some of which are used for temporal shifting operations, while the remaining channels remain unchanged; the channels used for temporal shifting are shifted forward or backward (usually by 1 frame); the shifting intensity is controlled by a learnable parameter, rather than a fixed value.

[0056] Specifically, each time-series displacement module can be divided into three parts along the channel dimension, and the division ratio can be preset as needed: (1) Left shift channel: accounts for a certain proportion of the total number of channels (typically 1 / 8); (2) Right shift channel: accounts for a certain proportion of the total number of channels (typically 1 / 8); (3) Non-displaced channels: the remaining part of the total number of channels (typically 3 / 4).

[0057] Shifting to the left means using future information in the present, while shifting to the right means using past information in the present.

[0058] In some embodiments, the learnable parameters include: learnable parameters for the left-shifted channel and learnable parameters for the right-shifted channel; based on the learnable parameters for the left-shifted channel, the feature values ​​of each feature in the left-shifted channel of the feature map can be mapped to a specified interval using the sigmoid function; when the mapped feature value is a first value, the feature value is temporally shifted; when the mapped feature value is a second value, the feature value is not temporally shifted; based on the learnable parameters for the right-shifted channel, the feature values ​​of each feature in the right-shifted channel of the feature map can be mapped to a specified interval using the sigmoid function; when the mapped feature value is a first value, the feature value is temporally shifted; when the mapped feature value is a second value, the feature value is not temporally shifted.

[0059] In this embodiment, two learnable parameters can be introduced to control the displacement intensity, including: the learnable parameter shift_weight_left for the left shift channel and the learnable parameter shift_weight_right for the right shift channel.

[0060] (1) shift_weight_left: controls the blending intensity of the left-shifted channel, which is mapped to the [0,1] interval by the sigmoid function. When the value is 0 (i.e., the first value is 0), the original channel value is used completely, and when the value is 1 (i.e., the second value is 1), the shifted value is used completely.

[0061] (2) shift_weight_right: controls the blending intensity of the right-shifted channel, which is also mapped to the [0,1] interval through the sigmoid function; (3) The non-displaced channel retains its original value and does not require weight parameters.

[0062] The mixed intensity of the original features and displacement features can be mathematically represented as a weighted sum of the original features and displacement features.

[0063] See also Figure 6 This diagram illustrates the effect of weighted displacement for learnable temporal displacement. Figure 6 In this context, 't' refers to time. The black blocks indicate padding with zeros. For example, when a feature channel shifts to the right at time t=0, there are no other feature maps to its left, and the feature map at time t=0 is shifted to the right for t=1. Therefore, the feature map that was moved away at t=0 is padded with zeros.

[0064] In some embodiments, learnable parameters can also be jointly optimized with the main task loss function via backpropagation.

[0065] In this embodiment, the temporal displacement intensity parameters are jointly optimized through backpropagation and the main task loss function, enabling the network to automatically perform deep learning based on the data distribution.

[0066] In samples with significant dynamic changes (such as boiling state and transitional state between boiling and non-boiling), the interaction of temporal information is enhanced to increase displacement intensity; in samples with stable states (such as calm liquid surface and empty pot state), unnecessary temporal disturbances are suppressed to reduce displacement intensity. In this way, the network can adaptively adjust the temporal modeling intensity, avoiding overfitting to stable samples and undermodeling of transitional samples.

[0067] The method provided in this embodiment can introduce learnable temporal displacement intensity parameters into the temporal displacement module of the network to control the forward or backward displacement ratio of features in the time dimension, thereby achieving adaptive adjustment of temporal modeling intensity.

[0068] In some embodiments, the state stability of the video frame sequence can also be determined based on the probability distribution of each category; wherein, the state stability is used to characterize whether the video frame sequence has a feature center in the semantic space; the state stability of the video frame sequence is used as a latent state variable to model the cooking state recognition model, and the cooking state recognition model is jointly optimized by the classification loss function and the temporal compactness loss function; wherein, the temporal compactness loss function is used to constrain the distance between the feature and the probability distribution of each category of the feature center.

[0069] like Figure 3 As shown, this embodiment can model the cooking state recognition model based on the probability distributions of each category output by the classification layer. This embodiment can introduce a state stability perception mechanism into the cooking state recognition model, enabling the model to learn and adjust the utilization of temporal information according to the smoothness of state evolution in the video sequence.

[0070] Existing technologies rely on preset thresholds (such as point cloud change thresholds, fluctuation value thresholds, tilt angle thresholds, etc.) and fixed rules to determine the state, lacking the ability to learn and process transitional states. This embodiment introduces a state stability perception mechanism, enabling the cooking state recognition model to learn and adjust the utilization of temporal information based on the smoothness of state evolution in the video sequence. This results in better discrimination capabilities in both stable and transitional state scenarios, without the need for explicit prediction of stability variables or setting manual thresholds.

[0071] To characterize the differences in temporal consistency among different samples within the same category, the method provided in this embodiment can theoretically introduce the concept of "state stability" to describe the degree of self-consistency of a video sample in the temporal dimension. The key innovation lies in the fact that stability is not a directly supervised output quantity, but rather a latent state variable.

[0072] In this embodiment, stability does not refer to whether the image is static or whether noise exists, but rather whether the current video sequence stably exists around a specific "feature center" in the semantic space. Here, the feature center refers to features that are similar to those of video sequences of the same class, but significantly different from those of video sequences of different classes. For a video sample, its state stability reflects whether the features of each time slice are distributed around the same feature center; and whether there are obvious stage changes or state fluctuations.

[0073] In this embodiment, stability does not mean stillness, but rather "semantic consistency". Stable state samples (such as calm liquid surface, stable boiling) have high temporal consistency, and their features are closely distributed around the same center in the time dimension; while transitional state samples (such as the beginning of bubbling, critical switching stage) exhibit greater temporal dispersion, and their features are more dispersed in the time dimension.

[0074] For example, in cooking state recognition: when the state is truly calm, the feature center is clear, the stability is high, and the features naturally converge and have low fluctuations over time; when the state is bubbling, the feature center is unclear (moving between calm and boiling), the stability is low, and the features diverge over time and are difficult to converge; when the state is truly boiling, the feature center is clear, the stability is high, and the features naturally converge and have low fluctuations over time.

[0075] In some embodiments, the distribution of time-slice features of a video frame sequence around the same feature center and the presence of state fluctuations can be determined based on the probability distribution of each category. If the samples of a video frame sequence are distributed around the same feature center and have low fluctuations, then the sample is a stable state sample. If the samples of a video frame sequence are not distributed around the same feature center and have high fluctuations, then the sample is a transitional state sample.

[0076] This embodiment does not attempt to accurately predict the state stability value at a certain moment. Instead, it introduces a state stability perception mechanism to enable the cooking state recognition model to learn and adjust the way it utilizes temporal information based on the smoothness of state evolution in the video sequence.

[0077] This embodiment treats stability variables as latent state variables, whose values ​​are indirectly learned through temporal modeling, feature aggregation, or modulation of the loss function, rather than through explicit regression prediction. Specific implementation methods include, but are not limited to, implicit representation through the loss function structure. Stability information is entirely implicitly represented through the temporal center loss function: 1. Typical stable state samples: Features naturally cluster in time with low fluctuations. Features in each time slice are closely distributed around the class center. The gradient direction of all time steps is consistent. Features converge quickly to the center. Temporal compactness loss is small and optimization is easy.

[0078] 2. Transitional state samples: Features diverge in time and are difficult to converge. Features in each time slice deviate from the class center, are loosely distributed, have conflicting gradient directions, and cannot simultaneously satisfy the central constraint. The temporal compactness loss is large, making optimization difficult.

[0079] The temporal center loss function in this embodiment essentially measures whether samples fluctuate around a stable feature center in the time dimension. Therefore, "stability" does not need to be explicitly predicted, but is implicitly characterized by the behavior of "whether it is easy to be pulled towards the center." Stability is no longer a supervised label, but a result of "whether the center constraint is satisfied"; stability is no longer an output, but an attribute of "how easy it is to optimize." By minimizing the temporal center loss, the network naturally learns the ability to distinguish between stable and transitional states without explicitly predicting the stability value.

[0080] Existing technologies do not implicitly represent stability information through geometric constraints in the feature space, and cannot distinguish samples with different levels of stability based on the natural structure of the feature distribution. This embodiment imposes constraints on the features of each time slice in the video samples through the Temporal Center Loss function, limiting their distance from the center of their respective categories. This allows the network to distinguish samples with different levels of stability in the feature space without explicitly predicting stability variables. The key innovation is that stability is no longer a supervised label, but a result of "whether the center constraint is satisfied"; stability is no longer an output, but an attribute of "how easy it is to optimize".

[0081] The Temporal Center Loss in this embodiment does not predict "stability," but rather forces the network to reveal differences in stability or instability through geometric constraints of temporal consistency.

[0082] In some embodiments, the feature center can also be determined by an exponential moving average.

[0083] For each category, an updatable feature center is maintained to represent the typical time-series state characteristics of that category. This center is updated using an exponential moving average (EMA): ;in, Indicates category At any moment Feature center, Indicates belonging to a category samples At any moment Features For the preset update momentum, To take the average value.

[0084] The loss function in this embodiment consists of two parts: ArcFace classification loss and temporal compactness loss. The overall loss function is defined as follows: .

[0085] in, The total loss function is used to jointly optimize the classification task and temporal compactness constraints; The ArcFace classification loss is used to improve the discriminativeness of features and the separability between classes; Temporal Center Loss is used to constrain the distance between features of each time slice within the same video and the center of their respective categories. This is the weighting coefficient for the temporal compactness loss, used to balance the classification loss and the temporal compactness loss, and is usually set to 0.1.

[0086] ArcFace classification loss enhances inter-class separability by adding margins in the angle space, and its mathematical expression is: .

[0087] in, The classification loss value for ArcFace; This is the scaling factor, used to scale logits and enhance gradients; it is typically set to 64.

[0088] Input feature vector With the correct category weight vector The angle between them. In ArcFace, both the feature vector and the weight vector are L2 normalized ( , Therefore, the dot product equals the cosine similarity, and the calculation formula is: ;in, The feature vector after temporal pooling and L2 normalization ( dimension), For category The weight vector (also L2 normalized).

[0089] The angular edge margin is used to add an extra angle penalty for the correct category, and is usually set to 0.5; This is the cosine similarity score after adding margin. Because the cosine function... The interval is monotonically decreasing, and increasing the angle leads to a decrease in the cosine value, thus reducing the score for the correct category. Therefore, to increase the score for the correct category, it is necessary to decrease... The value of , i.e., reducing the eigenvector With the correct class weight vector The angle between them makes Increase.

[0090] Input feature vector With error categories weight vector The angle between them is calculated using the following formula: (Calculated on the same normalized features and weights); Total number of categories; This indicates that the correct category is excluded when summing. Summation is performed only for error categories.

[0091] In this embodiment, the ArcFace classification loss is achieved through feature normalization ( ) and weight normalization ( Map the features onto a unit hypersphere such that the dot product equals the cosine similarity; add margins in the angle space. This makes the angle for the correct category need to be smaller ( Only by using scaling factors can high scores be obtained, thereby enhancing inter-class separability; Enlarging logits makes the softmax curve steeper and the classification boundaries clearer.

[0092] In this embodiment, constraints can be imposed on the features of each time slice in the video sample to limit the distance between each time slice feature in the same video and the center of its category.

[0093] The temporal compactness loss function can be determined using the following formula: ;in, The loss value is the time-compressibility loss function. The number of time slices in the video frame sequence. For time slice index, For the first Feature vectors of each time slice To which the sample belongs The feature center vector of the feature center vector; It is the square of the L2 norm, that is, the square of the Euclidean distance; For all time slices From 1 to Sum.

[0094] In this embodiment, the feature center can be dynamically updated using the exponential moving average (EMA) method: .

[0095] in, For category The feature center at the epoch-th iteration; For category The feature center at the (1)th iteration; To update momentum, the degree of influence of the new feature on the center is controlled, and it is usually set to 0.1; To belong to category samples The feature vector at the epoch-th iteration; To belong to the category The average of all sample features is calculated.

[0096] In this embodiment, the temporal compactness loss, for steady-state samples, means that features in each time slice are closely distributed around the class center. Small values ​​make optimization easy; for transitional state samples: features in each time slice deviate from the class center and are loosely distributed. The value is large, making optimization difficult; by minimizing this loss, the network naturally learns the ability to distinguish between stable and transitional states without explicitly predicting the stability value.

[0097] In this embodiment, stability can be implicitly measured through a loss function. Whether the same video consistently approaches a stable semantic anchor point over time is equivalent to: a small temporal compactness loss in a stable state indicates high stability; a large temporal compactness loss in a transitional state indicates low stability. This is precisely the "functional definition" of stability, but without the need for explicit modeling of stability variables.

[0098] From a training optimization perspective, stability is directly reflected in the training optimization process: For stable sequences, features converge quickly to the center; the optimization process is easy, and the loss decreases rapidly. For unstable sequences, the center constraint cannot be satisfied simultaneously; the optimization process is difficult, and the loss decreases slowly. "Stability" is directly reflected in the training optimization process, which is closer to the actual learning process of a cooking state recognition model than predicting a scalar.

[0099] The temporal compactness loss provided in this embodiment allows the network to distinguish samples with different levels of stability in the feature space without explicitly predicting stability variables. For typical stable samples (such as calm liquid surfaces or stable boiling), features naturally cluster in time with low fluctuations, forming a compact distribution in the feature space, with features in each time slice tightly clustered around the class center; the temporal compactness loss value is small, making optimization easy. For transitional samples (such as critical states or state transition stages), features diverge in time and are difficult to converge; they are naturally distributed in the class boundary region, with a relatively dispersed feature distribution; the temporal compactness loss value is large, making optimization difficult. This geometric structure enables the network to implicitly identify and process transitional state samples without additional explicit rules or thresholds.

[0100] The method provided in this embodiment of the invention introduces temporal center constraints into the feature space, enabling the cooking state recognition model to implicitly distinguish between stable and unstable states based on the degree of aggregation of sequence features in the time dimension without explicitly predicting state stability parameters. This allows for learnable adjustment of the utilization of temporal information. This embodiment proposes a learnable temporal displacement stability-aware temporal constraint method. Through learnable temporal displacement intensity, state stability-aware modeling, and a temporal compactness loss function, it achieves learnable adjustment of the temporal modeling intensity and effective processing of transitional state samples.

[0101] The method provided in the embodiments of the present invention mainly provides the following: 1. The temporal modeling intensity is learnable: The temporal displacement intensity of existing technologies is fixed, while this invention introduces learnable parameters, enabling the network to automatically adjust the temporal information interaction intensity according to the characteristics of the samples.

[0102] 2. Stability-aware mechanism: Existing technologies rely on preset thresholds and fixed rules, while this invention implicitly reflects stability information through stability-aware modeling. Stability is no longer a supervised label, but the result of "whether the central constraint is satisfied"; stability is no longer an output, but an attribute of "how easy it is to optimize". There is no need to explicitly predict the stability value, no need for additional stability labels or supervision signals, and no need for explicit thresholds.

[0103] 3. Geometric Constraints in Feature Space: Existing technologies do not use geometric constraints in the feature space to distinguish samples with different levels of stability. However, this invention uses a temporal compactness loss function to allow stability information to be naturally reflected in the training and optimization process. The features of stable state samples naturally converge and have low fluctuations in time, while the features of transitional state samples diverge in time and are difficult to converge.

[0104] Example 3: This invention provides a computer program product, including a computer program that, when executed by a processor, can implement the video-based cooking state recognition method provided in the foregoing embodiments.

[0105] The computer program product of the video-based cooking state recognition method provided in this embodiment of the invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0106] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and / or device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0107] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0108] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0109] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0110] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A video-based cooking state recognition method, characterized in that, The method includes: Obtain a video frame sequence; wherein the video frame sequence includes multiple frames of images, and each frame of the image includes information about the scene to be identified; Feature extraction is performed on the video frame sequence using a backbone network to obtain multi-layer feature maps; A learnable temporal shift operation is performed on the feature map to obtain a temporal feature map; wherein, the learnable temporal shift operation is used to adaptively adjust the temporal information interaction intensity of the feature map through a weighted shift mechanism; Based on the temporal feature map, classification prediction is performed to obtain the probability distribution of each category; The final classification result is output based on the probability distribution of each category; wherein the classification result includes each category and its corresponding probability value; The step of performing a learnable temporal shift operation on the feature map to obtain a temporal feature map includes: obtaining learnable parameters characterizing the intensity of adaptively adjusting temporal modeling, and the channel partitioning ratio of shifted channels and non-shifted channels; wherein, the shifted channels characterize feature channels in the feature map that are shifted in the time dimension, and the non-shifted channels characterize feature channels in the feature map that are not shifted in the time dimension, and the shifted channels include: left-shifted channels and right-shifted channels; performing a learnable temporal shift operation on the feature map based on the channel partitioning ratio and the learnable parameters to obtain a temporal feature map; wherein, the learnable parameters are used to map the left-shifted channels and right-shifted channels of the feature map to a specified interval using a sigmoid function, and determining whether to perform temporal shift on the feature value based on the mapped feature value.

2. The method according to claim 1, characterized in that, The steps of performing learnable temporal shift operations on the feature map based on the channel partitioning ratio and the learnable parameters include: In the intermediate feature layer of the two-dimensional convolutional network, the feature channels of the feature map are divided into the left-shifted channel, the right-shifted channel, and the non-shifted channel based on the channel division ratio. The left and right shift channels of the feature map are temporally shifted based on the learnable parameters; wherein the temporal shift of the left shift channel is a forward shift and the temporal shift of the right shift channel is a backward shift; or, the temporal shift of the left shift channel is a backward shift and the temporal shift of the right shift channel is a forward shift.

3. The method according to claim 2, characterized in that, The learnable parameters include: the learnable parameters of the left shift channel and the learnable parameters of the right shift channel; The step of performing temporal shifting of the left and right shifting channels of the feature map based on the learnable parameters includes: Based on the learnable parameters of the left-shifted channel, the feature values ​​of each feature of the left-shifted channel of the feature map are mapped to a specified interval using the sigmoid function; when the mapped feature value is the first value, the feature value is time-shifted; when the mapped feature value is the second value, the feature value is not time-shifted. Based on the learnable parameters of the right-shift channel, the feature values ​​of each feature of the right-shift channel of the feature map are mapped to a specified interval using the sigmoid function; when the mapped feature value is the first value, the feature value is time-shifted; when the mapped feature value is the second value, the feature value is not time-shifted.

4. The method according to claim 1, characterized in that, The method further includes: The learnable parameters are optimized by combining backpropagation with the main task loss function.

5. The method according to any one of claims 1-4, characterized in that, After the step of performing classification prediction based on the time-series feature map to obtain the probability distribution of each category, the method further includes: The state stability of the video frame sequence is determined based on the probability distribution of each category; wherein, the state stability is used to characterize whether the video frame sequence has a feature center in the semantic space; The cooking state recognition model is obtained by modeling the state stability of the video frame sequence as a latent state variable. The cooking state recognition model is then jointly optimized by a classification loss function and a temporal compactness loss function. The temporal compactness loss function is used to constrain the distance between features and feature centers.

6. The method according to claim 5, characterized in that, The step of determining the state stability of the video frame sequence based on the probability distribution of each category includes: Based on the probability distribution of each category, determine whether the time-slice features of the video frame sequence are distributed around the same feature center and whether there are state fluctuations; If the samples of the video frame sequence are distributed around the same feature center and have low fluctuations, then the sample is a stable state sample. If the samples in the video frame sequence are not distributed around the same feature center and fluctuate significantly, then the sample is a transitional state sample.

7. The method according to claim 5, characterized in that, The method further includes: The feature center is determined by an exponential moving average.

8. The method according to claim 5, characterized in that, The total loss function of the cooking state recognition model is determined by the following formula: ;in, Let the total loss function be... Let be the classification loss function. Let be the temporal compactness loss function. These are the preset weighting coefficients for the temporal compactness loss; The classification loss function is determined by the following formula: ;in, The preset scaling factor, Input feature vector With the correct category weight vector The angle between them For the edge of the angle, Input feature vector With error categories weight vector The angle between them; The temporal compactness loss function is determined by the following formula: ;in, The number of time slices in the video frame sequence. For time slice index, For the first Feature vectors of each time slice To which the sample belongs The feature center vector of the feature center vector; It is the square of the L2 norm, that is, the square of the Euclidean distance; For all time slices From 1 to Sum.

9. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, can implement the video-based cooking state recognition method according to any one of claims 1 to 8.

Images