Cooking video playing method and device

Abstract

The application provides a cooking video playing method and device, cooking videos used for assisting users in cooking are cut into multiple video action slices in advance, multiple anchor points and determination thresholds are set for action nodes in the multiple video action slices, the video action slices are played in the cooking process of the user, multiple modal data of the user in the cooking process are collected, the multiple modal data are preprocessed, the multiple modal data are input into a preset cooking action recognition model, and an identification result is output, whether the multiple modal data match a target video action slice currently played is determined based on the identification result and the determination threshold of the video action slice, if the multiple modal data match the target video action slice, the target video action slice is continuously played based on action completion degree and action standard degree, so that the user cooks based on the target video action slice. The traditional time axis linear playing logic is completely restructured, and full-automatic zero-manual adaptive synchronization of the cooking video used for assisting the user in cooking and the actual cooking progress of the user is realized.

Description

Technical Field

[0001] This invention relates to the field of smart kitchen appliance technology, and in particular to a method and apparatus for playing cooking videos. Background Technology

[0002] With the large-scale implementation of the smart home industry, smart kitchen appliances have evolved from single-function devices to intelligent interactive terminals that assist in the entire cooking process. Currently, the market penetration rate of smart kitchen appliances with screens (including integrated cooktops with screens, smart range hoods, built-in steam ovens, and smart refrigerators with screens) in China is relatively high. Most users learn to cook by watching cooking videos on these smart appliances, and a significant proportion of users are novice cooks. Cooking video tutorials have become a core, frequently used function of smart kitchen appliances, and users are placing increasingly higher demands on video playback for ease of use, intelligent features, and strong adaptability to the actual cooking rhythm.

[0003] Currently, the main methods for playing videos on smart kitchen appliances are: 1. Time-axis driven linear playback technology (mainstream commercial basic solution): This technology is the basic solution adopted by the vast majority of products on the market. Its core logic is that the video playback progress is solely determined by the playback duration. The video plays linearly at a fixed rhythm, and users need to manually adjust playback states such as pause, fast forward, rewind, and speed through touch, buttons, etc.

[0004] 2. Voice-assisted playback control technology (an alternative to manual operation): To address the pain point of inconvenient manual operation during cooking, existing technologies have proposed an optimized solution for voice interaction control. Through voice recognition technology, users can perform operations such as play, pause, fast forward, and rewind using voice commands without needing to touch the screen.

[0005] 3. Cooking video structured decomposition and tagging technology (playback controllability optimization solution): To improve the controllability of video playback steps, existing technologies have proposed a structured processing solution for cooking videos. The core of this solution is to break down the complete video into multiple segments according to the cooking steps and mark them with keywords and step numbers. Users can directly jump to the corresponding step segment by voice or touch, without having to drag the progress bar frame by frame.

[0006] However, the existing smart kitchen appliance video playback methods mentioned above cannot achieve a fundamental reconstruction of the video playback logic from time-axis driven to user action completion-driven. The core pain point of the video playback rhythm being out of sync with the user's actual cooking progress has not been fundamentally resolved. Summary of the Invention

[0007] In view of this, the purpose of the present invention is to provide a cooking video playback method and device, which takes the completion and accuracy of the user's cooking actions as the sole driving core of video playback, and achieves strong binding between video playback nodes and cooking actions through standardized anchoring of multiple nodes in the cooking video, multimodal lightweight action recognition and bidirectional matching verification.

[0008] In a first aspect, embodiments of the present invention provide a method for playing cooking videos. The method pre-divides a cooking video used to assist a user in cooking into multiple video action slices, and sets multiple anchor points and judgment thresholds for action nodes in the multiple video action slices. The method includes: playing video action slices during the user's cooking process; collecting multimodal data of the user during the cooking process; preprocessing the multimodal data; inputting the multimodal data into a preset cooking action recognition model; and outputting recognition results; wherein the recognition results include: action type, action completion degree, and action standard degree; determining whether the multimodal data matches the currently played target video action slice based on the recognition results and the judgment thresholds of the video action slices; if the multimodal data matches the target video action slice, continuing to play the target video action slice based on the action completion degree and action standard degree, so that the user can cook based on the target video action slice.

[0009] In optional embodiments of this application, the steps of dividing the cooking video used to assist users in cooking into multiple video action slices and setting multiple anchor points and judgment thresholds for action nodes in the multiple video action slices include: determining action nodes in the cooking video; wherein, action nodes are used to characterize the actions performed by the user during the cooking process; dividing the cooking video into multiple video action slices based on the action nodes; wherein, each video action slice contains one action node; setting a start anchor point, an intermediate verification anchor point, and a completion anchor point for each action node, and setting a judgment threshold for each action node; wherein, visual features, audio features, and semantic features are extracted for each anchor point; constructing an association mapping table based on the action nodes, the features extracted from each anchor point in the action nodes, the judgment thresholds of the action nodes, and the video action slices; and storing the association mapping table and the cooking video.

[0010] In optional embodiments of this application, the aforementioned multimodal data includes: user's limb movement data, kitchen utensil status data, food ingredient status data, and cooking environment status data; preprocessing includes: downsampling processing and feature extraction.

[0011] In an optional embodiment of this application, after the steps of inputting multimodal data into a preset cooking action recognition model and outputting recognition results, the method further includes: determining the confidence level of multiple consecutive frames of multimodal data; if the confidence level of multiple consecutive frames is greater than or equal to a preset confidence threshold, determining that the recognition result is qualified; if the confidence level of multiple consecutive frames is less than the confidence threshold, determining that the recognition result is unqualified.

[0012] In optional embodiments of this application, the above-mentioned identification results further include: action features; the degree of action completion is determined based on the quantification progress of the action features, and the degree of action standardization is determined based on the similarity between the action features and the standard features.

[0013] In an optional embodiment of this application, the step of determining whether multimodal data matches the currently playing target video action slice based on the recognition result and the determination threshold of the video action slice includes: determining the target determination threshold of the target video action slice and other determination thresholds of multiple other video action slices adjacent to the target video action slice based on the association mapping table; determining a first matching degree for matching the action features with the anchor features of the target video action slice; if the first matching degree is greater than or equal to the target determination threshold, determining multiple second matching degrees for matching the action features with the anchor features of multiple other video action slices respectively; if the multiple second matching degrees are all less than the corresponding other determination thresholds, determining that the multimodal data matches the target video action slice.

[0014] In an optional embodiment of this application, the step of continuing to play the target video action slice based on the action completion degree and the action standard degree includes: determining a playback control strategy based on the action completion degree and the action standard degree; and switching the playback state of the target video action slice by using a smooth transition and anti-shake triggering method based on the playback control strategy, and continuing to play the target video action slice.

[0015] In an optional embodiment of this application, after the step of determining whether the multimodal data matches the currently playing target video action slice based on the recognition result and the determination threshold of the video action slice, the method further includes: if the multimodal data does not match the target video action slice, determining the video action slice that matches the multimodal data, and playing the video action slice that matches the multimodal data.

[0016] In optional embodiments of this application, the above method further includes: collecting the user's historical cooking data and constructing a user cooking profile based on the historical cooking data; wherein, the historical cooking data includes: action execution duration, action completion rate, action standardization, and error action type; adjusting the judgment threshold and playback control strategy based on the user cooking profile; wherein, the playback control strategy includes: video playback parameter strategy and explanation trigger strategy.

[0017] Secondly, embodiments of the present invention also provide a cooking video playback device, which pre-divides a cooking video used to assist a user in cooking into multiple video action slices, and sets multiple anchor points and judgment thresholds for action nodes in the multiple video action slices. The device includes: a multimodal data acquisition module, used to play video action slices during the user's cooking process, acquire multimodal data of the user during the cooking process, and preprocess the multimodal data; a cooking action recognition module, used to input the multimodal data into a preset cooking action recognition model and output recognition results; wherein, the recognition results include: action type, action completion degree, and action standard degree; a target video action slice matching module, used to determine whether the multimodal data matches the currently playing target video action slice based on the recognition results and the judgment thresholds of the video action slices; and a target video action slice continued playback module, used to continue playing the target video action slice based on the action completion degree and action standard degree if the multimodal data matches the target video action slice, so that the user can cook based on the target video action slice.

[0018] The embodiments of the present invention bring the following beneficial effects: This invention provides a method and apparatus for playing cooking videos. It uses the user's cooking action completion and accuracy as the sole driving force for video playback. Through standardized anchoring of multiple nodes in the cooking video, multimodal lightweight action recognition, and bidirectional matching verification, it achieves a strong binding between video playback nodes and cooking actions. This completely reconstructs the traditional linear playback logic along the timeline, enabling fully automatic, zero-manual adaptive synchronization between the cooking video assisting the user and the user's actual cooking progress. It fundamentally solves the core pain point of novice cooks where the video rhythm is disconnected from their operation, requiring frequent manual / voice adjustments to the playback progress.

[0019] Other features and advantages of this disclosure will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the techniques described above.

[0020] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0021] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1A flowchart of a cooking video playback method provided in an embodiment of the present invention; Figure 2 A system block diagram of a cooking video playback method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the setting of three layers of anchor points for an action node involving adding salt for seasoning, provided in an embodiment of the present invention. Figure 4 A schematic diagram of an adaptive playback triggering logic provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a cooking video playback device provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Currently, existing methods for playing videos on smart kitchen appliances have the following drawbacks: 1. Existing optimization solutions such as voice control, video segmentation, and fixed speed have not broken through the underlying framework of linear playback and cannot fundamentally adapt to the rhythm differences of different cooking steps for users. The ease of operation and playback control accuracy cannot be balanced.

[0025] 2. Existing video node tags are generated solely based on the video's own content and lack a standardized, multi-dimensional anchoring system. This makes it impossible to achieve a strong binding between video playback nodes and user cooking actions, which can easily lead to mismatches, misjudgments, and missed judgments.

[0026] 3. Existing video structuring solutions can only adapt to a small number of standardized commercial recipe videos, and cannot cover the massive amount of UGC cooking videos and user-made instructional videos on the Internet, resulting in extremely poor content ecosystem adaptability.

[0027] 4. Novice users have a high failure rate in cooking due to the lack of timely operation, burning of ingredients, incorrect seasoning, and loss of oil temperature caused by the video's rhythm being out of sync. This can even lead to safety risks such as dry burning.

[0028] Based on this, the cooking video playback method and device provided in this embodiment of the invention can be applied to the field of smart kitchen appliances and multimedia smart playback. Specifically, it provides a video adaptive playback method for cooking teaching scenarios, which can drive cooking assistance based on action completion and action accuracy.

[0029] To facilitate understanding of this embodiment, a cooking video playback method disclosed in this embodiment of the invention will first be described in detail.

[0030] Example 1: This invention provides a method for playing cooking videos, which follows a six-step closed-loop process: "video preprocessing → hardware acquisition → motion recognition → node binding → playback control → self-learning optimization". All steps are completed locally on the smart kitchen appliance (either cloud processing or local processing is optional).

[0031] This embodiment can pre-divide cooking videos used to assist users in cooking into multiple video action slices, and set multiple anchor points and judgment thresholds for the action nodes in the multiple video action slices.

[0032] In this embodiment, the cooking video is used to assist users in cooking. Anchor point standardization preprocessing can be performed before the cooking video is stored in the library: the cooking video is sliced ​​and multiple anchor points and judgment thresholds are set.

[0033] In some embodiments, action nodes in a cooking video can be identified; whereby action nodes represent actions performed by a user during the cooking process; the cooking video is divided into multiple video action slices based on the action nodes; whereby each video action slice contains one action node; a start anchor point, an intermediate verification anchor point, and a completion anchor point are set for each action node, and a judgment threshold is set for each action node; whereby visual features, audio features, and semantic features are extracted for each anchor point; an association mapping table is constructed based on the action nodes, the features extracted from each anchor point in the action nodes, the judgment threshold of the action nodes, and the video action slices; and the association mapping table and the cooking video are stored.

[0034] In this embodiment, video action slicing can be performed: the entire cooking video is broken down into single, closed-loop key action units with clear input and output by artificial intelligence (AI) algorithms to extract action nodes. The duration of a single node video can be 3-10 seconds. The slicing rule can be that one action node corresponds to one core user operation. For example, "take the oil bottle", "pour oil", and "put the oil bottle" can be 3 independent action nodes.

[0035] In this embodiment, three layers of anchor points can be generated: a start anchor point, an intermediate verification anchor point, and a completion anchor point are automatically generated for each action node. Each layer of anchor points extracts three types of quantifiable features: visual, audio, and semantic. A judgment threshold is also generated for each action node.

[0036] In this embodiment, standardized storage can be performed: action nodes, three-layer anchor point features, judgment thresholds, and video action slice association mapping tables are stored together with video files in the local / networked cloud recipe video library of the kitchen appliance, supporting fast retrieval from local / networked cloud.

[0037] Based on the above description, please refer to Figure 1 The flowchart shown illustrates a method for playing a cooking video, which includes the following steps: Step S102: Play video motion slices during the user's cooking process, collect multimodal data of the user during the cooking process, and preprocess the multimodal data.

[0038] See also Figure 2 The diagram shows a system block diagram of a cooking video playback method. The multimodal acquisition module can acquire multimodal data of the user during the cooking process, the hardware control component can play video action slices during the user's cooking process, and the feature preprocessing unit can preprocess the multimodal data.

[0039] In some embodiments, the multimodal data includes: user's body movement data, kitchen utensil status data, food ingredient status data, and cooking environment status data; preprocessing includes: downsampling and feature extraction.

[0040] In this embodiment, multimodal cooking actions can be collected in real time at the kitchen appliance: through the multimodal acquisition component integrated into the kitchen appliance, the user's body movements, kitchen utensil status, and food / environment status data during the cooking process are collected synchronously. The collected data only retains the cooking workbench and is directly preprocessed at the chip level (downsampling, feature extraction) after collection, reducing the computing load of the main control chip.

[0041] Step S104: Input multimodal data into a preset cooking action recognition model and output recognition results; wherein, the recognition results include: action type, action completion degree and action standard degree.

[0042] The recognition results also include: action features. For example... Figure 2 As shown, multimodal data can be input into the lightweight recognition model (i.e., the cooking action recognition model), and the action features can be obtained by feature dimensionality reduction through the feature fusion unit. The action completion / standard judgment unit can output the action completion and action standard.

[0043] In some embodiments, the confidence level of multiple consecutive frames of multimodal data can also be determined; if the confidence level of multiple consecutive frames is greater than or equal to a preset confidence threshold, the recognition result is determined to be qualified; if the confidence level of multiple consecutive frames is less than the confidence threshold, the recognition result is determined to be unqualified.

[0044] In this embodiment, real-time inference can be performed on the kitchen appliance to identify lightweight cooking actions and determine completion / standardization: the preprocessed multimodal data is input into the lightweight cooking action recognition model, and the model simultaneously outputs three core results: 1. Action type: The cooking action currently being performed by the user (e.g., "pouring oil" or "chopping vegetables"); 2. Action completion rate: Quantitative value from 0-100% (e.g., if the oil can is tilted at 30°, the oil pouring completion rate is 50%). 3. Action standard: A quantitative value of 0-100% (e.g., if the knife grip is incorrect, the standard of cutting vegetables is 60%). The judgment result is considered valid only if the confidence level is ≥0.85 for 3 consecutive frames (i.e., the confidence level threshold can be 0.85), in order to avoid misjudgment in a single frame.

[0045] Step S106: Based on the recognition results and the judgment threshold of the video action slice, determine whether the multimodal data matches the target video action slice currently being played.

[0046] like Figure 2 As shown, a three-layer anchor mapping table (i.e., association mapping table) can be extracted from the recipe video library. The three-layer anchor mapping table is then input into the bidirectional matching verification unit, which can determine whether the multimodal data matches the action slice of the currently playing target video.

[0047] In some embodiments, a target determination threshold for the target video action slice and other determination thresholds for multiple other video action slices adjacent to the target video action slice can be determined based on an association mapping table; a first matching degree is determined for matching the action features with the anchor features of the target video action slice; if the first matching degree is greater than or equal to the target determination threshold, multiple second matching degrees are determined for matching the action features with the anchor features of multiple other video action slices respectively; if multiple second matching degrees are all less than the corresponding other determination thresholds, it is determined that the multimodal data matches the target video action slice.

[0048] In this embodiment, real-time bidirectional matching verification and strong binding between actions and video nodes can be performed on the kitchen appliance side: the recognition results are matched with the associated mapping table, and "forward matching + reverse verification" is used to achieve a 1:1 strong binding between actions and video nodes. (1) Forward matching: Match the user's action features with the anchor point features of the target video action slice. If the first matching degree is greater than or equal to the target judgment threshold, the binding is initially performed. (2) Reverse verification: Match the user's action features with the anchor features of the three adjacent video action slices. If the second matching degree is less than the other judgment thresholds, the binding is finally confirmed. (3) Binding update: The binding node is updated to "completed" only after the anchor point matching is completed and the matching preparation of the next node is triggered.

[0049] Step S108: If the multimodal data matches the target video action slice, continue playing the target video action slice based on the action completion degree and action standard degree, so that the user can cook based on the target video action slice.

[0050] like Figure 2 As shown, the adaptive playback control unit can obtain the action completion degree and action standard degree output by the action completion degree / standard degree judgment unit, and control the video playback module to continue playing the target video action slice based on the action completion degree and action standard degree. The user can perform cooking operations with the assistance of the target video action slice being played. The target video action is synchronized with the user's cooking actions, and the user does not need to frequently adjust the playback progress manually / voice.

[0051] In some embodiments, a playback control strategy can be determined based on the degree of action completion and the degree of action standard; based on the playback control strategy, the playback state of the target video action slice is switched by means of smooth transition and anti-shake triggering, and the playback of the target video action slice continues.

[0052] In this embodiment, full-scene adaptive playback control driven by action completion can be executed in real time on the kitchen appliance: based on the binding result of the bidirectional matching verification unit and the action completion and action standard output by the action completion / standard judgment unit, the corresponding playback control strategy is triggered, covering 6 high-frequency scenarios for novice cooking. All playback state switching adopts smooth transition + anti-shake triggering, without visual abruptness; playback control commands are transmitted through the native system bus of the kitchen appliance, without additional communication delay.

[0053] In some embodiments, if the multimodal data does not match the target video action slice, a video action slice that matches the multimodal data is determined and played.

[0054] If the multimodal data does not match the target video action slice, a new video action slice that matches the multimodal data can be determined and played through the video playback module.

[0055] In some embodiments, the user's historical cooking data can also be collected, and a user cooking profile can be constructed based on the historical cooking data; wherein, the historical cooking data includes: action execution time, action completion rate, action standardization, and error action type; the judgment threshold and playback control strategy are adjusted based on the user cooking profile; wherein, the playback control strategy includes: video playback parameter strategy and explanation trigger strategy.

[0056] like Figure 2 As shown, this embodiment can construct a user cooking profile. The self-learning optimization unit can obtain the user cooking profile and update the parameters of the lightweight recognition model and the bidirectional matching verification unit through the parameter update module.

[0057] This embodiment can be executed in the background of the kitchen appliance, incrementally learning and adapting to user cooking behavior: continuously collecting the user's historical cooking data (action execution time, completion / standardization, error action type), building a user-specific cooking profile, and adaptively adjusting based on the profile: generating parameter feedback data to update judgment thresholds / matching rules, for example: 1. Action judgment threshold (e.g., automatically relax the standard threshold for beginners). 2. Video playback parameters (e.g., automatically slowing down the base speed for elderly users); 3. Explain the triggering strategy (e.g., early warning of high-frequency error nodes); learning data can be stored locally without uploading to the cloud, ensuring user privacy. Alternatively, it can be uploaded to the cloud for sharing, with user-selectable control options.

[0058] This invention provides a method for playing cooking videos, with the user's cooking action completion and accuracy as the sole driving force for video playback. By standardizing and anchoring multiple nodes of the cooking video, using multimodal lightweight action recognition and bidirectional matching verification, a strong binding between video playback nodes and cooking actions is achieved. This completely reconstructs the traditional linear playback logic of the timeline, enabling fully automatic, zero-manual adaptive synchronization between the cooking video assisting the user and the user's actual cooking progress. It fundamentally solves the core pain point of novice cooks where the video rhythm is out of sync with their operation and they need to frequently manually / voice adjust the playback progress.

[0059] Example 2: This invention provides another method for playing cooking videos, which is implemented based on the above embodiments. The focus is on describing the hardware components of the cooking video playback method. These hardware components can include three major modules: "acquisition + processing + playback", and are adapted to the native architecture of kitchen appliances.

[0060] In this embodiment, the real-time acquisition of multimodal cooking actions is completed collaboratively by a multimodal acquisition module integrated into the kitchen appliance. The multimodal acquisition module includes one or more combinations of a vision acquisition module, a human posture acquisition module, a kitchen utensil status acquisition module, a temperature acquisition module, an audio acquisition module, an environmental perception module, and a hardware preprocessing module.

[0061] The visual acquisition module in this embodiment includes at least one of a monocular camera module, a binocular camera module, a depth camera module, and an infrared night vision camera module. It is used to acquire image information of user body movements, food status, kitchen utensil positions, and operating postures within the cooking workbench area, and to crop the working area of ​​the image to remove invalid background information.

[0062] The human posture acquisition module in this embodiment includes at least one of a millimeter-wave radar module, a human key point detection hardware acceleration module, and a gesture recognition sensor, used to acquire the spatial features of the user's hand posture, limb movement trajectory, and operation actions.

[0063] The kitchenware status acquisition module in this embodiment includes at least one of an inertial sensor module, a Hall sensor module, a pressure sensor module, and a weight detection module, used to acquire information on the kitchenware's pick-up and put-down status, in-position status, tilt angle, and weight change.

[0064] The temperature acquisition module in this embodiment includes at least one of an infrared temperature measurement module, an NTC (negative temperature coefficient) temperature sensor, and a thermocouple temperature measurement module, used to acquire information on the temperature of the cookware, the oil temperature, and the thermal state of the cooking area.

[0065] The audio acquisition module in this embodiment includes a microphone array and a noise reduction pickup module, used to collect sound characteristics during the cooking process, including the sound of ingredients entering the pot, the sound of seasonings spilling, the sound of kitchen utensils colliding, and environmental noise.

[0066] The environmental sensing module in this embodiment includes a light sensor, a fume sensor, and a gas sensor, which are used to collect information on the ambient light intensity, fume concentration, and air quality in the kitchen.

[0067] In this embodiment, the hardware preprocessing module synchronously processes the multimodal acquisition data, completing image downsampling, filtering and noise reduction, initial feature extraction, data alignment and normalization. The processed feature data is then transmitted to the main control chip of the kitchen appliance to reduce the subsequent computational load and improve the real-time performance and stability of the system.

[0068] In this embodiment, the binocular depth camera captures visual features of user body movements, food status, and kitchen utensil positions, supporting work area cropping and depth detection.

[0069] In this embodiment, the low-power inertial sensor module for kitchen utensils collects the states of utensils being picked up / put down / tilted, compensating for the omissions in pure visual recognition.

[0070] The core processing module in this embodiment is used to run the lightweight recognition model, matching algorithm, and playback control logic.

[0071] In this embodiment, the playback control module, specifically the kitchen appliance display unit, plays cooking videos to assist the user in cooking, supporting picture-in-picture, fade-in / fade-out, and split-screen prompts. The video hardware decoding unit decodes H.265 / AV1 format videos, supporting speed adjustment and frame interpolation playback without stuttering.

[0072] In this embodiment, the auxiliary module voice interaction unit has the highest priority fallback control and supports three extremely simple commands ("Say it again", "Wait a moment", "Next step").

[0073] Example 3: This invention provides another method for playing cooking videos, which is implemented based on the above embodiments. The focus is on describing the lightweight algorithm and model adaptation algorithm in the cooking video playback method.

[0074] The AI ​​cooking action video slicing algorithm in this embodiment can extract only the key frames of the cooking action and ignore segments without action.

[0075] In this embodiment, the three-layer anchor point feature extraction algorithm performs multimodal feature fusion (visual + audio + semantic) feature dimensionality reduction, retaining only core judgment features (such as the tilt angle of the oil pot).

[0076] The lightweight cooking action recognition model in this embodiment can recognize cooking actions (such as chopping vegetables, stir-frying, and pouring oil) in videos in real time, providing practical support for intelligent recipe guidance, kitchen safety monitoring, or elderly-assisted cooking. Its lightweight design is achieved through three aspects: employing efficient network structures such as MobileNet (e.g., depthwise separable convolution), model pruning (removing redundant parameters), and 8-bit integer quantization (compressing data precision), significantly reducing computational load and model size, enabling the model to run smoothly on low-power terminals such as mobile phones and embedded devices.

[0077] In this embodiment, the bidirectional matching algorithm for actions and video nodes only matches the core features of anchor points and ignores secondary features, thus simplifying the matching time and improving the matching accuracy.

[0078] In this embodiment, the incremental collaborative filtering algorithm of the user cooking behavior self-learning model only learns high-frequency user cooking data and updates it locally incrementally.

[0079] Example 4: This invention provides another method for playing cooking videos, which is implemented based on the above embodiments. The focus is on describing the quantitative determination logic and formula for the three-layer anchor points. (See also...) Figure 3 The diagram illustrates a three-layer anchor point setup for an action node involving adding salt for seasoning. In some embodiments, the action completion degree is determined based on the quantified progress of the action features, and the action standard degree is determined based on the similarity between the action features and standard features.

[0080] In this embodiment, the video nodes and cooking actions are strongly bound together by the quantization of three layers of anchor points. Each anchor point adopts the hard judgment logic of "feature quantization value ≥ judgment threshold". At the same time, it is combined with "multi-feature joint judgment" (all of which are indispensable) to avoid misjudgment caused by the failure of a single feature. The judgment weights of the three types of features are: visual features (0.6) + audio features (0.2) + semantic features (0.2).

[0081] General judgment formula: Anchor point matching degree When the matching degree is ≥0.85, the anchor point is considered to have passed the matching test; in particular, to complete the anchor point, the matching degree must be ≥0.85 for 3 consecutive times.

[0082] The quantitative calculation of action completion based on action completion / standard is as follows: Action completion rate: Calculated based on the "quantified progress of key features", the formula is: Action completion rate Calculation (Example: The key feature for pouring oil is [the tilt angle of the oil can]. The completion threshold is 60°. If the current tilt is 30°, then the completion rate = 30 / 60 × 100% = 50%).

[0083] Action standardization: Calculated based on the similarity between key features and standard features, using the formula: Action Standardization (Example: The standard knife-holding angle for cutting vegetables is 45°, and the current angle is 60°, then the standard percentage = [1 - [60 - 45] / 45] × 100% ≈ 66.7%).

[0084] Example 5: This invention provides another method for playing cooking videos, which is implemented based on the above embodiments. The focus is on describing the logic of verifying accuracy and anti-interference in bidirectional matching.

[0085] 1. Forward matching: Match the completion anchor features of the user's current action with the completion anchor features of the current playback node in the video library. If the matching degree is ≥0.85, the binding is initially performed. 2. Reverse verification: The completion anchor feature of the user's current action is matched with the completion anchor features of the two adjacent nodes in the video library. If the matching degree of all adjacent nodes is <0.7, the binding is finally confirmed; if there is an adjacent node matching degree ≥0.7, a second judgment is triggered (further judgment is made in combination with the action sequence and the status of the ingredients).

[0086] In particular, all judgment thresholds can be adaptively adjusted according to the user's proficiency, and the secondary judgment is a fallback mechanism to avoid mismatch.

[0087] Example 6: This invention provides another method for playing cooking videos, which is implemented based on the above embodiments. The focus is on describing the triggering logic for achieving zero-manual, seamless, and adaptive playback across all scenarios. (You can participate in...) Figure 4 The diagram shows a triggering logic for adaptive playback.

[0088] When the node video finishes playing, if the action completion rate is less than 100%, the video will pause at the node's ending frame.

[0089] Once the action is completed, proceed to the next step. Triggering an action completion rate of 100%, anchor point matching is completed, and the video automatically advances to the next node.

[0090] When subsequent actions are completed ahead of schedule, the action completion rate of the unplayed node is 100%, and the anchor point matching is completed. The video is then automatically fast-forwarded (at double speed).

[0091] When the movements are slightly non-standard, the standard of the movements is 60%-80%, and the completion rate is ≥100%, the video plays normally with a text prompt in the corner.

[0092] When the motion error (moderate / severe) is less than 60% or the completion rate is less than 100%, but the motion deviates from the expected level, pause and add graded frame interpolation for explanation.

[0093] Example 7: This invention provides another method for playing cooking videos, which is implemented based on the above embodiments, focusing on describing the user's cooking profile and its self-learning core logic.

[0094] Based on the feature data of users' cooking profiles, incremental learning is adopted, updating only the parameters of frequently cooked dishes / actions, without performing full learning. The core learning logic is as follows: 1. Proficiency Tag Generation: Based on the average completion time and average standard of user actions, users are divided into 3 tags: Novice (standard of action < 70%, time > 2 times the standard time), Pro (standard of action 80%-90%, time = 0.8-1.2 times the standard time), and Expert (standard of action ≥ 90%, time < 0.8 times the standard time). 2. Parameter adaptive adjustment: Based on the proficiency tag, the anchor point matching threshold and video playback parameters are automatically adjusted. The adjustment rules are as follows: Novice (matching threshold is reduced by 10%, base speed is 0.6-0.8 times), Experienced (default threshold, base speed is 1.0 times), Expert (matching threshold is increased by 5%, base speed is 1.2-1.5 times). 3. Error prediction: Record frequently made errors by users (such as pouring oil or adding salt). When the video plays to the previous node of the error, an alert will automatically pop up to proactively avoid the problem.

[0095] Example 8: This invention provides another method for playing cooking videos, which is implemented based on the above embodiments. The focus is on describing a specific example of a cooking video playback method. In this embodiment, a smart range hood and stove system or a smart integrated stove is used as the carrier, the target dish is a beginner's guide to "scrambled eggs with tomatoes," and the user is a cooking novice (proficiency tag: novice). The entire technical implementation process of this invention is reproduced to verify the feasibility and effectiveness of the solution.

[0096] Prerequisites: The video preprocessing for the "Tomato and Egg Stir-fry" video has been completed, breaking it down into 16 video action nodes (such as "taking tomatoes", "washing tomatoes", "cutting tomatoes", "beating eggs", etc.). Each node generates three layers of anchor points, and the associated mapping table is stored in the local video library / cloud video library (optional). The multimodal acquisition module of the integrated stove (dual-lens camera + kitchenware status acquisition module + temperature sensor) has been activated, and the working area is locked to the cooking worktable. The user's cooking profile is a new profile, configured with the "beginner" tag by default (matching threshold reduced by 10%, base speed 0.7x).

[0097] Node 1: Picking up a tomato: The camera detects the user's hand grabbing a tomato, the anchor point matching is successful, and the "picking up a tomato" video clip is played automatically (0.7x speed). After the user completes the action, the completion rate is 100%, and the process automatically proceeds to the next node. Node 3: Cutting Tomatoes: The user is a novice and cuts vegetables slowly. After the video clip finishes playing, the action is only 30% complete. The integrated stove automatically waits and pauses, stopping at the end frame of cutting tomatoes. The text prompt "Please cut the tomatoes into small pieces" is displayed in the corner of the screen until the user finishes cutting (completion rate = 100%), and then automatically continues. Node 7: Pouring oil: If the user's grip on the oil pitcher is incorrect and the standard of the action is 55% (<60%), the integrated stove will immediately pause the main video and play a 3-second explanatory segment on "standard grip on the pitcher + pouring oil" in picture-in-picture format. After the explanation is completed, the video will remain paused until the user corrects the action (standard of the action = 75%), and then continue to wait. Node 9: Pouring Egg Liquid: The user has completed "cracking eggs" (Node 8) in advance and directly executes the pouring egg liquid. The integrated stove recognizes that the completion rate of Node 9 is 100%, automatically fast forwards to the "pouring egg liquid" node, transitions at 2x speed for 0.5 seconds, and then plays normally. Node 12: Adding Salt: The user previously made scrambled eggs and added too much salt (a frequent error). When the integrated stove plays Node 11 (stir-frying tomatoes), it automatically pops up a warning prompt: "Add salt in small amounts multiple times, half a spoon is recommended" to avoid this problem in advance. Completely uninterrupted: The user did not touch the screen with either hand or issue any voice commands. The integrated stove automatically paused, accelerated, fast-forwarded, and inserted frames to explain the process based on the user's level of completion and accuracy. The video rhythm perfectly matched the user's operating speed, and the user successfully completed the cooking without any issues such as burnt ingredients or seasoning mistakes.

[0098] Instance verification results: Number of manual / voice interventions: 0, completely freeing up hands; Action-node matching accuracy: 100%, no mismatches or misjudgments; Playback state switching latency: ≤50ms, no stuttering or visual abruptness; Cooking failure rate: 0, beginners can successfully complete the cooking.

[0099] Example 9: This invention provides another method for playing cooking videos, which is implemented based on the above embodiments, and focuses on describing some alternative solutions.

[0100] Alternative Solution 1: Entry-level kitchen appliance adaptation solution (minimal configuration, lowest cost) applicable scenarios.

[0101] Entry-level smart kitchen appliances with screens (such as basic smart range hoods and affordable integrated cooktops) do not have binocular cameras, but are only equipped with a single ordinary camera, and have no additional computing power resources.

[0102] Acquisition Module: The kitchen utensil status acquisition module and temperature sensor have been removed, and only a single-lens ordinary camera is used to collect visual features, simplifying it to "single-modal acquisition"; Recognition Model: A minimalist model is adopted, which only recognizes the core action type and completion level, and does not recognize the standard degree for the time being; Matching Algorithm: Dynamic time warping has been removed, and only cosine similarity positive matching is used. The matching threshold has been increased to 0.9 to avoid false judgments; Playback control: Simplify the frame interpolation explanation strategy, only retain the explanation of "severely incorrect" frame interpolations, and do not provide prompts for minor non-standard ones.

[0103] Core functionality: Retain the core function of "action completion-driven playback" to achieve zero manual intervention.

[0104] Alternative Solution 2: Mid-range kitchen appliance adaptation solution (balanced configuration, highest cost performance) applicable scenarios.

[0105] Mid-range smart kitchen appliances with screens (such as mainstream smart integrated stoves and built-in steam ovens) are equipped with a monocular depth camera and have a small amount of additional computing power.

[0106] Acquisition Module: Retain the monocular depth camera + kitchenware status acquisition module (remove the temperature sensor), simplifying it to "dual-modal acquisition" to take into account both visual features and kitchenware status; Recognition Model: Retain the dual-branch model for lightweight recognition, with an accuracy rate of ≥99%; Matching Algorithm: Retain "forward matching + reverse verification", but reverse verification only matches the previous and next adjacent nodes (reducing the number of nodes by 2) to reduce computing power consumption; Playback control: The full-scene playback strategy is retained, but the duration of the sub-segments for frame-interpolated explanations is shortened (1s for mild, 2s for moderate, and 3s for severe).

[0107] Core Functions: Retains all core functions, balancing recognition accuracy and matching stability, offering the best value for money.

[0108] Alternative Solution 3: High-end kitchen appliance adaptation solution (full configuration, maximum functionality) applicable scenarios.

[0109] High-end smart kitchen appliances with screens (such as flagship smart integrated stoves and smart cooking countertops) are equipped with dual-lens depth cameras and millimeter-wave radar, and have ample computing power resources.

[0110] Acquisition Module: Based on the original solution, millimeter-wave radar is added to acquire the three-dimensional trajectory of user body movements, making up for the shortcomings of visual recognition when oil fumes obscure the image, and upgrading it to "four-modal acquisition"; Recognition Model: A lightweight 3D-CNN model is adopted, which can recognize more complex cooking actions (such as tossing and stirring), and the accuracy of completion / standardization judgment is improved to ±1%; Matching Algorithm: "Forward matching + reverse verification" is retained, and action timing prediction is added. Based on the three high-probability nodes under the current action prediction, video segments are preloaded in advance; Playback control: Added AR real-scene guidance, which overlays AR icons of standard actions onto the user's cooking actions, providing real-time correction and more intuitive explanations.

[0111] Core features: Based on the original solution, the recognition accuracy and anti-interference ability are improved, AR real-scene guidance is added, and the user experience is maximized.

[0112] In summary, the methods provided in the embodiments of the present invention mainly provide the following: 1. A playback mechanism driven solely by the user's cooking action completion rate replaces the timeline-driven logic used in existing technologies for decades; it establishes a closed-loop rule of "action completed → node advancement, action incomplete → video waiting, action error → frame insertion correction", and the video playback progress is entirely determined by the user's actual cooking operation status, rather than the preset playback duration.

[0113] This approach fundamentally achieves the ultimate ideal of "video plays wherever the user works," reducing the number of manual / voice interventions required for beginners cooking a single dish from the industry average of 12-18 times to zero, completely freeing up users' hands. It breaks free from the constraints of linear playback logic, with the video playback rhythm perfectly matching the user's operation speed, fully adapting to the slow-paced, multi-pause cooking characteristics of beginners. It avoids user panic caused by trying to keep up with the video's pace, reducing the failure rate of beginners cooking home-style dishes from 42% to less than 8%.

[0114] 2. The cooking video is broken down into a single closed-loop action node, and each node is bound to a three-layer structure of start anchor point - intermediate verification anchor point - completion anchor point. Each anchor point contains three quantifiable judgment dimensions: visual, audio and semantic, forming a standardized binding rule that can be recognized by machines and is unambiguous.

[0115] The anchor point system is standardized and replicable. It can be automatically generated by AI without manual frame-by-frame annotation, which greatly reduces the cost of video preprocessing. The progressive verification of anchor points completely avoids misbinding and playback jump problems caused by single-frame jitter and short pauses by users, and improves the stability of playback state switching by more than 10 times.

[0116] 3. A lightweight cooking action recognition model with multimodal fusion and a two-way matching and verification mechanism. It adopts a lightweight solution, combining binocular vision, kitchenware NFC sensing, and temperature sensing for multimodal acquisition. At the same time, it completes action-node binding through a two-layer logic of forward matching and reverse verification.

[0117] The multimodal acquisition and two-way verification mechanism enables the recognition accuracy of non-standard actions by beginners to reach ≥99.5%, and greatly improves the ability to resist interference from kitchen fumes, light, and noise. The model can simultaneously identify the completion and standardization of actions, providing full-dimensional data support for subsequent adaptive playback control, filling the gap in existing technologies that can only identify the type of action but cannot determine the completion.

[0118] 4. A full-scene adaptive playback control strategy driven by action completion is implemented. For the six high-frequency scenarios of novice cooking, full-scene playback rules are formulated, including waiting and pausing, node advancement, automatic rewind, automatic fast forward, graded frame interpolation explanation, and speed adaptation. At the same time, a dual-layer anti-shake trigger mechanism is set up to achieve seamless and smooth switching of playback states.

[0119] It achieves fully automatic playback control across all cooking scenarios, requiring no manual or voice intervention from the user. The playback state switching latency has been reduced from the industry average of 300ms to less than 50ms. The graded frame interpolation explanation mechanism matches the explanation content to the severity of the user's action error, which corrects the user's operation without excessively interrupting the cooking rhythm, reducing the error rate of novice operation by 65%.

[0120] 5. User cooking behavior self-learning and personalized adaptation mechanism: By continuously learning from users' historical cooking data, a user-specific cooking profile is built, adaptively adjusting action judgment thresholds, waiting time, playback speed, and explanation detail. Simultaneously, it implements action prediction preloading. This enables personalized adaptation tailored to each individual user. For the elderly, children, and cooking novices, the judgment threshold is automatically relaxed, the playback pace is slowed down, and the explanation detail is increased; for experienced cooks, the threshold is automatically tightened and the pace is accelerated, achieving 100% adaptation. Action prediction preloading ensures zero stuttering and zero loading delay during video clip transitions, further improving playback smoothness. For frequently misused actions, early warning prompts are provided, further reducing the error rate for novice cooks by 30%, achieving an upgrade from "passive correction" to "proactive avoidance of pitfalls."

[0121] Example 10: Corresponding to the above method embodiments, this invention provides a cooking video playback device that pre-divides a cooking video used to assist the user in cooking into multiple video action slices, and sets multiple anchor points and judgment thresholds for action nodes in the multiple video action slices. See [link to relevant documentation]. Figure 5 The diagram shows a structural schematic of a cooking video playback device, which includes: The multimodal data acquisition module 51 is used to play video motion slices during the user's cooking process, acquire multimodal data of the user during the cooking process, and preprocess the multimodal data; The cooking action recognition module 52 is used to input multimodal data into a preset cooking action recognition model and output recognition results; wherein, the recognition results include: action type, action completion degree and action standard degree; The target video action slice matching module 53 is used to determine whether the multimodal data matches the currently playing target video action slice based on the recognition result and the judgment threshold of the video action slice; The target video action slice continued playback module 54 is used to continue playing the target video action slice based on the action completion degree and action standard degree if the multimodal data matches the target video action slice, so that the user can cook based on the target video action slice.

[0122] This invention provides a cooking video playback device that uses the user's cooking action completion and accuracy as the sole driving force for video playback. By standardizing and anchoring multiple nodes of the cooking video, using multimodal lightweight action recognition and bidirectional matching verification, it achieves a strong binding between video playback nodes and cooking actions. This can completely reconstruct the traditional linear playback logic of the timeline, enabling fully automatic zero-manual adaptive synchronization between the cooking video assisting the user and the user's actual cooking progress. It fundamentally solves the core pain point of novice cooks where the video rhythm is out of sync with their operation and they need to frequently manually / voice adjust the playback progress.

[0123] The aforementioned device further includes: a cooking video segmentation module for determining action nodes in the cooking video; wherein, action nodes are used to characterize actions performed by the user during the cooking process; the cooking video is segmented into multiple video action slices based on the action nodes; wherein, each video action slice contains one action node; a start anchor point, an intermediate verification anchor point, and a completion anchor point are set for each action node, and a judgment threshold is set for each action node; wherein, visual features, audio features, and semantic features are extracted for each anchor point; an association mapping table is constructed based on the action nodes, the features extracted from each anchor point in the action nodes, the judgment threshold of the action nodes, and the video action slices; and the association mapping table and the cooking video are stored.

[0124] The aforementioned multimodal data includes: user's body movement data, kitchen utensil status data, food ingredient status data, and cooking environment status data; the aforementioned preprocessing includes: downsampling and feature extraction.

[0125] The aforementioned device further includes: a confidence judgment module, used to determine the confidence of multiple consecutive frames of multimodal data; if the confidence of multiple consecutive frames is greater than or equal to a preset confidence threshold, the recognition result is determined to be qualified; if the confidence of multiple consecutive frames is less than the confidence threshold, the recognition result is determined to be unqualified.

[0126] The above recognition results also include: action features; the degree of action completion is determined based on the quantification progress of the action features, and the degree of action standardization is determined based on the similarity between the action features and the standard features.

[0127] The aforementioned target video action slice matching module is used to determine the target judgment threshold of the target video action slice based on the association mapping table, as well as other judgment thresholds of multiple other video action slices adjacent to the target video action slice; determine the first matching degree of the action feature and the anchor point feature of the target video action slice; if the first matching degree is greater than or equal to the target judgment threshold, determine multiple second matching degrees of the action feature and the anchor point features of multiple other video action slices respectively; if multiple second matching degrees are all less than the corresponding other judgment thresholds, determine that the multimodal data matches the target video action slice.

[0128] The aforementioned target video action slice playback module is used to determine the playback control strategy based on the action completion degree and action standard degree; based on the playback control strategy, it switches the playback state of the target video action slice using smooth transition and anti-shake triggering, and continues to play the target video action slice.

[0129] The aforementioned device further includes a multimodal data rematching module, used to determine a video action slice that matches the multimodal data and play the video action slice that matches the multimodal data if the multimodal data does not match the target video action slice.

[0130] The aforementioned device also includes: a judgment threshold and playback control strategy adjustment module, used to collect the user's historical cooking data and construct a user cooking profile based on the historical cooking data; wherein, the historical cooking data includes: action execution duration, action completion rate, action standardization, and error action type; and the judgment threshold and playback control strategy are adjusted based on the user cooking profile; wherein, the playback control strategy includes: video playback parameter strategy and explanation trigger strategy.

[0131] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the cooking video playback device described above can be referred to the corresponding process in the embodiments of the aforementioned cooking video playback method, and will not be repeated here.

[0132] Example 11: This invention also provides an electronic device for running the above-described cooking video playback method; see [link to previous document]. Figure 6 The diagram shows the structure of an electronic device, which includes a memory 100 and a processor 101. The memory 100 is used to store one or more computer instructions, which are executed by the processor 101 to implement the cooking video playback method described above.

[0133] Furthermore, Figure 6 The electronic device shown also includes a bus 102 and a communication interface 103, with the processor 101, the communication interface 103 and the memory 100 connected via the bus 102.

[0134] The memory 100 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 103 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 102 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0135] Processor 101 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 101 or by instructions in software form. Processor 101 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a readily available storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 100, and processor 101 reads information from memory 100 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0136] This invention also provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are called and executed by a processor, they cause the processor to implement the above-described cooking video playback method. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0137] The computer program product of the cooking video playback method and apparatus provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0138] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and / or device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0139] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0140] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0142] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for playing cooking videos, characterized in that, The method involves pre-segmenting cooking videos used to assist users in cooking into multiple video action slices, and setting multiple anchor points and judgment thresholds for action nodes in the multiple video action slices. During the user's cooking process, video motion slices are played, multimodal data of the user during the cooking process is collected, and the multimodal data is preprocessed. The multimodal data is input into a preset cooking action recognition model, and the recognition result is output; wherein, the recognition result includes: action type, action completion degree and action standard degree; Based on the recognition results and the determination threshold of the video action slice, it is determined whether the multimodal data matches the currently playing target video action slice; If the multimodal data matches the target video action slice, the target video action slice continues to be played based on the action completion rate and the action standard rate, so that the user can cook based on the target video action slice.

2. The method according to claim 1, characterized in that, The steps of dividing a cooking video used to assist a user in cooking into multiple video action slices, and setting multiple anchor points and judgment thresholds for action nodes in the multiple video action slices, include: Identify action nodes in a cooking video; wherein the action nodes are used to characterize actions performed by the user during the cooking process; The cooking video is divided into multiple video action slices based on the action node; wherein each video action slice contains one action node. For each action node, a start anchor point, an intermediate verification anchor point, and a completion anchor point are set, and a judgment threshold is set for each action node; wherein, visual features, audio features, and semantic features are extracted for each anchor point; An association mapping table is constructed based on the action node, the features extracted from each anchor point in the action node, the judgment threshold of the action node, and the video action slice; Store the association mapping table and the cooking video.

3. The method according to claim 1, characterized in that, The multimodal data includes: user's body movement data, kitchen utensil status data, food ingredient status data, and cooking environment status data; the preprocessing includes: downsampling and feature extraction.

4. The method according to claim 1, characterized in that, After the step of inputting the multimodal data into a preset cooking action recognition model and outputting the recognition result, the method further includes: Determine the confidence level of consecutive frames of the multimodal data; If the confidence level of multiple consecutive frames is greater than or equal to a preset confidence threshold, the recognition result is determined to be qualified. If the confidence level is less than the confidence threshold for multiple consecutive frames, the recognition result is determined to be unqualified.

5. The method according to claim 2, characterized in that, The recognition result also includes: action features; the action completion degree is determined based on the quantification progress of the action features, and the action standard degree is determined based on the similarity between the action features and standard features.

6. The method according to claim 5, characterized in that, The step of determining whether the multimodal data matches the currently playing target video action slice based on the recognition result and the determination threshold of the video action slice includes: The target judgment threshold of the target video action slice and other judgment thresholds of multiple other video action slices adjacent to the target video action slice are determined based on the association mapping table. Determine the first matching degree between the action features and the anchor point features of the target video action slice; If the first matching degree is greater than or equal to the target determination threshold, determine multiple second matching degrees for matching the action feature with the anchor point features of multiple other video action slices respectively; If multiple second matching degrees are all less than the corresponding other determination thresholds, it is determined that the multimodal data matches the target video action slice.

7. The method according to claim 1, characterized in that, The step of continuing to play the target video action slice based on the action completion degree and the action standard degree includes: A playback control strategy is determined based on the degree of action completion and the degree of action accuracy. Based on the playback control strategy, the playback state of the target video action slice is switched using a smooth transition and anti-shake triggering method, and the playback of the target video action slice continues.

8. The method according to claim 1, characterized in that, After determining whether the multimodal data matches the currently playing target video action slice based on the recognition result and the determination threshold of the video action slice, the method further includes: If the multimodal data does not match the target video action slice, determine the video action slice that matches the multimodal data and play the video action slice that matches the multimodal data.

9. The method according to claim 7, characterized in that, The method further includes: Collect users' historical cooking data and construct user cooking profiles based on the historical cooking data; wherein, the historical cooking data includes: action execution time, action completion rate, action standardization, and error action type; The judgment threshold and the playback control strategy are adjusted based on the user's cooking profile; wherein, the playback control strategy includes: video playback parameter strategy and explanation triggering strategy.

10. A cooking video playback device, characterized in that, The device pre-segments cooking videos used to assist users in cooking into multiple video action slices, and sets multiple anchor points and judgment thresholds for action nodes in the multiple video action slices. The device includes: A multimodal data acquisition module is used to play video motion slices during the user's cooking process, acquire multimodal data of the user during the cooking process, and preprocess the multimodal data; The cooking action recognition module is used to input the multimodal data into a preset cooking action recognition model and output the recognition result; wherein, the recognition result includes: action type, action completion degree and action standard degree; The target video action slice matching module is used to determine whether the multimodal data matches the currently playing target video action slice based on the recognition result and the determination threshold of the video action slice; The target video action slice playback module is used to continue playing the target video action slice based on the action completion degree and the action standard degree if the multimodal data matches the target video action slice, so that the user can cook based on the target video action slice.

Images