Operation guidance model training method and device, equipment and storage medium

Through multimodal training sample and heterogeneous graph fusion technology, the problem of inaccurate identification of cooking assist system under single modal information is solved, more accurate user intention recognition and operation guidance is achieved, and the accuracy of cooking assist system is improved. and user experience.

CN119090024APending Publication Date: 2024-12-06GREE ELECTRIC APPLIANCE INC OF ZHUHAI +1
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411015319.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing cooking assistive systems rely on single modal information (such as text, voice or picture) to accurately identify complex cooking scenarios and user intentions, resulting in the generated operational guidance information being far from user expectations.

Method used

Multimodal training samples are used to extract image, speech and text feature information, and heterogeneous graphs are constructed for attention weight calculation and update, and target operation guidance information is generated using graph attention network and prediction neural network to fuse multimodal feature information.

Benefits of technology

It improves the accuracy of user intention identification, avoids the problem of inaccurate single-modal recognition, and provides more accurate and comprehensive cooking auxiliary services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090024A_ABST
    Figure CN119090024A_ABST
Patent Text Reader

Abstract

The invention relates to an operation guidance model training method and device, equipment and a storage medium. The method comprises the following steps: acquiring a plurality of preset training samples; the plurality of training samples have at least two modal types; for each training sample, extracting sample feature information from the training sample; training a preset operation guidance model by using the sample feature information corresponding to the plurality of training samples; the operation guidance model is used for fusing the multi-modal feature information corresponding to the multiple training samples according to the sample feature information corresponding to the multiple training samples and generating target operation guidance information corresponding to the training samples according to the multi-modal feature information corresponding to the training samples for each training sample. According to the method, the operation guidance model is trained by adopting the multi-modal training samples, so that the operation guidance model can fully utilize information of different modal types, the user intention is comprehensively considered, the identification accuracy of the user intention is improved, and the problem of inaccurate single-modal identification is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of model training technology, and in particular to a training method, device, equipment and storage medium for an operation guidance model. Background Art

[0002] With the rapid development of smart kitchen and smart home technologies, users' demand for convenient, efficient and intelligent cooking assistance systems is growing. When using, users can input information into the cooking assistance system according to their own needs, and the cooking assistance system recognizes the information and generates the operation guidance information required by the user, and the user completes the cooking according to the operation guidance information.

[0003] At present, cooking assistance systems mainly rely on single modal information input by users, for example, only text, only voice or only pictures are input into the cooking assistance system. The cooking assistance system relies on the single modal information input by the user to generate the operation guidance information required by the user. However, the current cooking assistance system relies on single modal information to make it difficult to accurately understand complex cooking scenarios and user intentions, and the recognition accuracy of user intentions is low, which leads to the generated operation guidance information being far from the user's intentions.

[0004] For example: the user took a photo of chopped mushrooms and onions, but the cooking assistance system mistakenly identified the onions in the photo as garlic, resulting in incorrect cooking steps. The user cooked according to the incorrect cooking steps given by the cooking assistance system, which may result in poor taste of the dish or even need to be re-cooked. Another example: the user only inputs the voice message "stir-fry onions" to the cooking assistance system, but due to the user's accent or background noise, the cooking assistance system identifies "onions" as "scallions" and gives incorrect cooking steps, resulting in the user being unable to achieve the expected effect of "stir-frying onions" according to the cooking steps of "stir-frying scallions". Summary of the invention

[0005] The present application provides a training method, device, equipment and storage medium for an operation guidance model to solve the problem that a cooking assistance system is difficult to accurately identify user intentions by relying on single modal information.

[0006] In view of the above technical problems, the technical solution of this application is solved by the following embodiments:

[0007] An embodiment of the present application provides a training method for an operation guidance model, comprising: obtaining a plurality of preset training samples; wherein, among the plurality of training samples, there are at least two modal types; for each of the training samples, extracting sample feature information from the training sample; using the sample feature information respectively corresponding to the plurality of training samples to train a preset operation guidance model; wherein the operation guidance model is used to fuse multimodal feature information respectively corresponding to the plurality of training samples according to the sample feature information respectively corresponding to the plurality of training samples and, for each of the training samples, generate target operation guidance information corresponding to the training sample according to the multimodal feature information corresponding to the training sample.

[0008] Among them, for each of the training samples, extracting sample feature information from the training sample includes: performing preprocessing on the training sample according to the modality type corresponding to the training sample; and extracting sample feature information from the preprocessed training sample according to the modality type corresponding to the training sample.

[0009] Among them, the extracting sample feature information from the preprocessed training sample according to the modality type corresponding to the training sample includes: when the modality type corresponding to the training sample is an image type, using a preset visual feature extraction model to extract visual feature information from the preprocessed training sample; when the modality type corresponding to the training sample is a speech type, using a preset acoustic feature extraction model to encode the preprocessed training sample to obtain acoustic feature information; when the modality type corresponding to the training sample is a text type, using a preset text feature extraction model to encode the preprocessed training sample to obtain text feature information.

[0010] Among them, the operation guidance model includes: a graph attention network and a prediction neural network; the multimodal feature information corresponding to multiple training samples is fused according to the sample feature information corresponding to the multiple training samples, and for each training sample, the target operation guidance information corresponding to the training sample is generated according to the multimodal feature information corresponding to the training sample, including: using the graph attention network to construct a heterogeneous graph according to the sample feature information corresponding to the multiple training samples, and using the heterogeneous graph to fuse the multimodal feature information corresponding to the multiple training samples; for each training sample, using the prediction neural network to generate the target operation guidance information corresponding to the training sample according to the multimodal feature information corresponding to the training sample.

[0011] Among them, constructing a heterogeneous graph based on the sample feature information corresponding to the multiple training samples includes: for each of the training samples, determining the training sample as a node in the heterogeneous graph, and taking the modal type corresponding to the training sample as the node type of the node; among the multiple sample feature information, determining any two of the sample feature information whose correlation is greater than a preset correlation threshold, and establishing an edge between the nodes corresponding to the two sample feature information.

[0012] Among them, the use of the heterogeneous graph to fuse the multimodal feature information corresponding to the multiple training samples includes: for any two connected nodes in the heterogeneous graph, determining the attention weight between the two connected nodes according to the sample feature information corresponding to the two connected nodes; for any two connected nodes in the heterogeneous graph, determining a reference node among the two connected nodes, and normalizing the attention weights corresponding to the two connected nodes according to the attention weights between the reference node and its neighboring nodes; for each node in the heterogeneous graph, using the normalized attention weights corresponding to the node and the nodes connected to it, updating the sample feature information corresponding to the node, and using the updated sample feature information corresponding to the node as the multimodal feature information corresponding to the node.

[0013] Among them, the method of training a preset operation guidance model using sample feature information corresponding to each of the multiple training samples includes: obtaining target operation guidance information generated by the operation guidance model for each of the training samples and obtaining real operation guidance information pre-set for each of the training samples; using a preset classification loss function, according to the target operation guidance information and the real operation guidance information corresponding to each of the multiple training samples, determining the classification loss value of the operation guidance model; using a preset modal loss function, according to the target operation guidance information and the real operation guidance information corresponding to each of the multiple training samples, determining the modal loss value of the operation guidance model; determining the model loss value corresponding to the operation guidance model based on the classification loss value and the modal loss value, and determining the training status of the operation guidance model based on the model loss value corresponding to the operation guidance model.

[0014] An embodiment of the present application also provides a training device for an operation guidance model, comprising: an acquisition module, used to acquire a plurality of preset training samples; wherein, among the plurality of training samples, there are at least two modal types; an extraction module, used to extract sample feature information from each of the training samples; a training module, used to train a preset operation guidance model using the sample feature information respectively corresponding to the plurality of training samples; wherein the operation guidance model is used to fuse the multimodal feature information respectively corresponding to the plurality of training samples according to the sample feature information respectively corresponding to the plurality of training samples and, for each of the training samples, generate the target operation guidance information corresponding to the training sample according to the multimodal feature information corresponding to the training sample.

[0015] An embodiment of the present application also provides a training device for an operation guidance model, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to: execute a training program for the operation guidance model stored in the memory to implement any of the above-described training methods for the operation guidance model.

[0016] An embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are executed to implement the training method of the operation guidance model described in any one of the above items.

[0017] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: the method provided by the embodiment of the present application can obtain a plurality of preset training samples; wherein, among the plurality of training samples, there are at least two modal types; for each of the training samples, sample feature information is extracted from the training sample; and a preset operation guidance model is trained using the sample feature information corresponding to the plurality of training samples; wherein, the operation guidance model is used to fuse the multimodal feature information corresponding to the plurality of training samples according to the sample feature information corresponding to the plurality of training samples, and for each training sample, generate the target operation guidance information corresponding to the training sample according to the multimodal feature information corresponding to the training sample. This embodiment uses multimodal type training samples to train the operation guidance model, so that the operation guidance model can make full use of information of different modal types, comprehensively consider the user's intention, improve the recognition accuracy of the user's intention, and avoid the problem of inaccurate single-modal recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0021] Figure 1 is a flow chart of a training method for an operation guidance model according to an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of a training process of an operation guidance model according to an embodiment of the present application;

[0023] Figure 3 A flowchart of the steps of fusing sample feature information according to an embodiment of the present application;

[0024] Figure 4 A flowchart of the training steps of the operation guidance model according to an embodiment of the present application;

[0025] Figure 5 is a structural diagram of a training device for an operation guidance model according to an embodiment of the present application;

[0026] Figure 6 It is a structural diagram of a training device for an operation guidance model according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0028] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0029] The present application embodiment provides a training method for an operation guidance model. Figure 1 , which is a flow chart of a method for training an operation guidance model according to an embodiment of the present application.

[0030] Step S110, obtaining a plurality of preset training samples; wherein the plurality of training samples have at least two modality types.

[0031] Modality types include, but are not limited to, text, voice, and image types. For example, multiple training samples may include text of a recipe, cooking-related voice, and images of ingredients.

[0032] Specifically, among the multiple training samples obtained, each training sample corresponds to a modality type, and the modality types corresponding to the multiple training samples are different. For example, 50 training samples were obtained in this training, including 20 text training samples, 15 voice training samples, and 15 image training samples. Multimodal training samples can include different expressions of the same user intent, making the expression of user intent more complete.

[0033] Step S120: for each of the training samples, extract sample feature information from the training sample.

[0034] Sample feature information is the key information used to describe training samples.

[0035] For each training sample, the training sample is preprocessed according to the modality type corresponding to the training sample; and sample feature information is extracted from the preprocessed training sample according to the modality type corresponding to the training sample. Different preprocessing methods and different feature extraction methods are adopted according to different modality types, so that the extracted sample feature information can be more accurate.

[0036] Step S130, using the sample feature information corresponding to the multiple training samples to train a preset operation guidance model; wherein the operation guidance model is used to fuse the multimodal feature information corresponding to the multiple training samples according to the sample feature information corresponding to the multiple training samples, and for each training sample, generate the target operation guidance information corresponding to the training sample according to the multimodal feature information corresponding to the training sample.

[0037] The model training method includes, but is not limited to: using a back propagation algorithm to train the operation guidance model until the training of the operation guidance model is completed.

[0038] Multimodal feature information can reflect the key information of features of multiple modal types.

[0039] In an embodiment of the present application, a plurality of preset training samples are obtained; wherein, among the plurality of training samples, there are at least two modal types; for each of the training samples, sample feature information is extracted from the training sample; and a preset operation guidance model is trained using the sample feature information corresponding to the plurality of training samples; wherein the operation guidance model is used to fuse the multimodal feature information corresponding to the plurality of training samples according to the sample feature information corresponding to the plurality of training samples, and for each training sample, generate the target operation guidance information corresponding to the training sample according to the multimodal feature information corresponding to the training sample. This embodiment uses multimodal type training samples to train the operation guidance model, so that the operation guidance model can make full use of information of different modal types, comprehensively consider user intentions, improve the recognition accuracy of user intentions, and avoid the problem of inaccurate single-modal recognition.

[0040] In order to make the embodiments of the present application clearer, the training process of the operation guidance model of the embodiments of the present application will be further described below.

[0041] like Figure 2 , which is a schematic diagram of the training process of the operation guidance model according to an embodiment of the present application.

[0042] Preparation stage of training samples:

[0043] In the embodiment of the present application, training samples of multiple modality types are collected in advance. For example, if the modality types include text type, voice type and image type, training samples of image type, training samples of voice type and training samples of text type are collected.

[0044] Furthermore, annotation information is pre-set for each training sample. Since training samples of different modal types can reflect different functions, when annotating training samples, the annotation information of training samples of different modal types includes information of the same or different content attributes. The annotation information can be the real operation guidance information corresponding to the training sample. The content attributes included in the real operation guidance information of training samples of different modal types are allowed to be different.

[0045] The content attributes of the real operation instruction information include, but are not limited to: dish name, dish category, ingredient type, ingredient quantity, cooking steps, and instruction type. The instruction type refers to the type of operation. For example, if you need to operate the oven, the instruction type is device control; if you need to stir-fry the ingredients, the instruction type is operation instruction.

[0046] Furthermore, for training samples of image types, a dish image dataset can be constructed. Collect multiple dish images. Cover different dishes and ingredients in different dish images. For example, collect hundreds of thousands of images of real dishes. Label each dish image, and the labeling information includes but is not limited to: dish category, ingredient type.

[0047] For example, the labeling information of image 1 includes: dish category: Sichuan cuisine, ingredient types: chili, pepper, tofu, minced meat. Another example: the labeling information of image 2 includes: dish category: Cantonese cuisine, ingredient types: chicken, mushrooms, green vegetables.

[0048] Furthermore, for training samples of speech types, voice commands in cooking scenarios can be collected. Voice command corpora in real cooking scenarios are collected. Among them, voice commands can cover multiple dialects and accents. Each voice command corpus is transcribed and annotated. Transcription refers to converting the voice command corpus into corresponding text content. The annotation information for the voice command corpus includes but is not limited to: cooking steps, instruction type.

[0049] For example, the annotation information of voice command 1 includes: cooking steps: preheat the oven to 200 degrees; command type: device control. Another example: the annotation information of voice command includes: cooking steps: put peppercorns into the pot; command type: operation command.

[0050] Furthermore, for text-type training samples, a standardized recipe database can be collected. A recipe database refers to standardized recipe texts collected from multiple data sources. The recipe text includes but is not limited to structured information such as dish names, cooking methods, and raw material amounts. The recipe text is annotated, and the annotated information includes but is not limited to dish names, cooking methods, and raw material amounts. When annotating, the recipe text can be structuredly annotated.

[0051] Preprocessing stage of training samples:

[0052] In the embodiment of the present application, after training samples of multiple modality types are collected, for each training sample, it is necessary to perform preprocessing on the training sample according to the modality type corresponding to the training sample.

[0053] Furthermore, for training samples of image type, image enhancement processing, image denoising processing and / or image scaling processing can be performed on the training samples. Among them, image enhancement processing includes: contrast adjustment, brightness adjustment, sharpening and other processing on the training samples to improve image quality. Image denoising processing includes: sampling Gaussian filtering, mean filtering and other methods to remove noise in the training samples. Image scaling processing includes: scaling the training samples to a preset uniform size for subsequent processing.

[0054] Furthermore, for the training samples of speech type, the training samples can be subjected to language denoising, speech segmentation and speech feature extraction. Among them, speech denoising includes: removing background noise in the training samples by using spectral subtraction, Wiener filtering and other methods. Speech segmentation includes: dividing the training samples used to represent continuous speech signals into at least one segment of speech, each segment of speech containing a complete instruction. Speech feature extraction is used to extract speech features in each segment of speech in the training samples. Speech features are used to represent the frequency domain information of speech signals and the acoustic characteristics of speech. Types of speech features include: Mel Frequency Cepstrum Coefficient (MFCC).

[0055] For example: the training sample is the voice information "Preheat the oven to 200 degrees and add beef". When performing voice segmentation processing, voice activity detection (VAD) technology is used to identify the starting and ending points of the voice activity, and "Preheat the oven to 200 degrees and add beef" is divided into two sentences "Preheat the oven to 200 degrees" and "Add beef". "Preheat the oven to 200 degrees" can be used as a complete instruction, and the instruction type is device control. "Add beef" can also be used as a complete instruction, and the instruction type is operation type.

[0056] Furthermore, for text type training samples, text normalization processing, word segmentation processing and stop word filtering processing can be performed on the training samples. Among them, text normalization processing: unifying the text format, removing special symbols and redundant information. Word segmentation processing includes: using natural language processing technology (Natural Language Processing, referred to as NLP) to perform word segmentation and part-of-speech tagging on the training samples. Stop word filtering processing includes: removing preset stop words in the training samples so as to retain the cooking steps and ingredient information in the training samples.

[0057] Feature extraction stage of training samples:

[0058] In the embodiment of the present application, after preprocessing each training sample, it is necessary to extract sample feature information from the preprocessed training sample according to the modality type corresponding to the training sample. According to the different modality types corresponding to the training sample, the types of the sample feature information include, but are not limited to, visual feature information, acoustic feature information, and text feature information.

[0059] Furthermore, when the modality type corresponding to the training sample is an image type, a preset visual feature extraction model is used to extract visual feature information from the preprocessed training sample. Visual features can be used to represent information such as ingredients and dishes in the image. Types of visual feature extraction models include, but are not limited to: pre-trained convolutional neural networks (CNN). For example: ResNet, VGG (Visual Geometry Group Network). The process of CNN model processing is the process of feature vectorization, so that a feature vector with a fixed length can be generated by CNN model processing as visual feature information.

[0060] Furthermore, when the modality type corresponding to the training sample is a speech type, the preprocessed training sample is encoded using a preset acoustic feature extraction model to obtain acoustic feature information. Types of acoustic feature extraction models include, but are not limited to, pre-trained sequence-to-sequence models. For example, a Transformer model. The speech features can be encoded into an acoustic feature vector sequence as acoustic feature information through the acoustic feature extraction model.

[0061] Furthermore, when the modality type corresponding to the training sample is a text type, a preset text feature extraction model is used to encode the preprocessed training sample to obtain text feature information. Types of text feature extraction models include, but are not limited to: pre-trained text encoders. For example: language representation model (Bidirectional EncoderRepresentations from Transformers, referred to as BERT). In this way, the word sequence of the recipe text can be encoded into a text feature vector sequence using the text feature extraction model.

[0062] It can be seen that in the embodiment of the present application, the sample feature information extracted from the multimodal training samples is expressed in the form of feature vectors to facilitate the use of the operation guidance model.

[0063] Operation guidance model training phase:

[0064] The operation guidance model includes: a graph attention network and a prediction neural network. Of course, those skilled in the art should know that: if computing power allows, the preprocessing process and the feature extraction process can also be integrated into the operation guidance model.

[0065] Specifically, the sample feature information corresponding to the multiple training samples can be input into the operation guidance model, and the graph attention network in the operation guidance model receives the sample feature information corresponding to the multiple training samples; using the graph attention network, a heterogeneous graph is constructed according to the sample feature information corresponding to the multiple training samples, and using the heterogeneous graph, multimodal feature information corresponding to the multiple training samples is fused; for each training sample, using the prediction neural network, the target operation guidance information corresponding to the training sample is generated according to the multimodal feature information corresponding to the training sample.

[0066] In an embodiment of the present application, a heterogeneous graph is constructed based on sample feature information corresponding to multiple training samples, including: for each training sample, determining the training sample as a node in the heterogeneous graph, and using the modal type corresponding to the training sample as the node type of the node; among the multiple sample feature information, determining any two sample feature information whose correlation is greater than a preset correlation threshold, and establishing an edge between the nodes corresponding to the two sample feature information.

[0067] Furthermore, the number of nodes in the heterogeneous graph is the same as the number of training samples, each node corresponds to a sample feature information, and the node type of each node is the same as the modality type of the corresponding training sample. For example, the node types include: image type, voice type and text type.

[0068] Furthermore, the correlation is used to measure the correlation between two sample feature information (two nodes). The greater the correlation, the higher the correlation between the two sample feature information; the smaller the correlation, the lower the correlation between the two sample feature information. The correlation between two sample feature information can be the similarity between the two sample feature information. The correlation threshold is used to measure whether it is necessary to establish an edge between two nodes. In this way, only nodes with close correlation have a connection relationship, and nodes with loose correlation do not have a connection relationship.

[0069] Furthermore, when establishing an edge, the similarity between any two sample feature information is calculated in the sample feature information corresponding to the plurality of training samples, and if the similarity between the two current sample feature information is greater than a preset similarity threshold (equivalent to a correlation threshold), an edge is established between the two current sample feature information; otherwise, an edge is not established between the two current sample feature information. The embodiment of the present application does not limit the specific similarity algorithm.

[0070] In the embodiment of the present application, after establishing the edge, the edge can be used to represent the attention weight between nodes, and then multiple sample feature information can be fused into multimodal feature information according to the attention weight corresponding to the edge. Furthermore, in the embodiment of the present application, a graph convolution kernel can be designed, and gated graph convolution can be used to perform self-attention fusion on nodes of different modal types on a heterogeneous graph, and at the same time, heterogeneous information of other modal types can be fused, so that multiple sample feature information can be fused into multimodal feature information. Figure 3 , which is a flow chart of the steps of fusing sample feature information according to an embodiment of the present application.

[0071] Step S310, for any two connected nodes in the heterogeneous graph, determine the attention weight between the two connected nodes according to the sample feature information respectively corresponding to the two connected nodes.

[0072] In the embodiment of the present application, the attention weight between the two connected nodes can be determined using the following formula:

[0073] e ij =LeakyReLu(a T ·[W·h i ||W·h j ]);

[0074] Among them, e ij represents the attention weight between the i-th node and the j-th node it is connected to; LeakyReLu represents the activation function; a T represents the transpose of the attention vector to be trained; W represents the weight matrix to be trained; h iIndicates the sample feature information corresponding to the i-th node; h j Indicates the sample feature information corresponding to the jth node; ∥ indicates a concatenation operation, which concatenates the sample feature information corresponding to the ith node and the sample feature information corresponding to the jth node into a vector. [] can indicate that the concatenated vector is taken as a whole.

[0075] Furthermore, the concatenation operation is, for example: concatenating [1,2] and [3,4] into [1,2,3,4].

[0076] Step S320, for any two connected nodes in the heterogeneous graph, determine a reference node among the two corresponding nodes, and normalize the attention weights corresponding to the two connected nodes according to the attention weights between the reference node and its neighboring nodes.

[0077] In the embodiment of the present application, the following formula may be used to normalize the attention weights corresponding to the two connected nodes:

[0078]

[0079] Among them, a ij represents the normalized attention weight between the i-th node and the j-th node it is connected to; exp represents the exponential function; e ij represents the attention weight between the i-th node and the j-th node; the i-th node is the reference node, is the set of neighbor nodes of the ith node, and the set of neighbor nodes includes all nodes connected to the ith node; Σ represents the accumulation operation; e ik represents the attention weight between the i-th node and the k-th node, and the k-th node belongs to Of course, the jth node also belongs to e ik Please refer to e ij The calculation method is obtained.

[0080] Step S330, for each node in the heterogeneous graph, using the normalized attention weights corresponding to the node and the nodes connected to it, update the sample feature information corresponding to the node, and use the updated sample feature information corresponding to the node as the multimodal feature information corresponding to the node.

[0081] In the embodiment of the present application, the following formula may be used to update the sample feature information corresponding to each node in the heterogeneous graph:

[0082]

[0083] Among them, h′ irepresents the updated sample feature information corresponding to the i-th node; σ represents the nonlinear activation function; is the set of neighbor nodes of the ith node; a ik represents the i-th node and The normalized attention weight between the kth nodes in ; W represents the weight matrix to be trained; h k Represents the sample feature information corresponding to the kth node.

[0084] Furthermore, after updating the sample feature information corresponding to the node, a multi-head attention mechanism can be used to use the updated sample feature information corresponding to the i-th node obtained by the above method as an attention head head1, and the above method can be executed in parallel to calculate multiple attention heads corresponding to the i-th node, and splice the multiple attention heads corresponding to the i-th node to obtain the multimodal feature information corresponding to the i-th node. The multimodal feature information spliced ​​by this multi-head attention mechanism reflects more comprehensive and rich information.

[0085] In the embodiment of the present application, the following formula can be used to determine the multimodal feature information:

[0086] h″=concat(head1, head2..., head l );

[0087] Among them, h″ represents the multimodal feature information corresponding to the i-th node; head l represents the lth attention head corresponding to the i-th node; concat represents the concatenation operation, that is, concatenating the attention heads from the 1st to the lth.

[0088] In the embodiment of the present application, the sample feature information of multiple modal types corresponding to a node can be fused into multimodal feature information through the graph attention mechanism, such as: high-level feature information such as food image, cooking voice, recipe text, etc. can be fused into multimodal feature information, thereby realizing cross-modal information fusion. After the multimodal feature information is fused, the prediction neural network is used to generate and output the target operation guidance information corresponding to the node (training sample) according to the multimodal feature information corresponding to the node.

[0089] Furthermore, the prediction neural network can be a fully connected classification network. The fused multimodal feature information is input into the fully connected classification network, and multi-label classification prediction can be performed on labels such as dish type, ingredient type, cooking method, etc. Among them, the fully connected classification network includes: an input layer, a hidden layer, and an output layer. The input layer is used to receive the fused multimodal feature information. The hidden layer is used to extract high-level features from the multimodal feature information through a multi-layer neural network. The output layer is used to output the probability distribution of each label category for multi-label classification. Each label corresponds to a content attribute. Of course, the output layer of the fully connected classification model can also query the target operation guidance information corresponding to the multimodal feature information in a preset database according to the high-level features. Since there is a corresponding relationship between the multimodal feature information and the training sample, the output layer of the fully connected classification model can output the target operation guidance information corresponding to the training sample. Among them, the database is used to store a variety of target operation guidance information. The target operation guidance information includes but is not limited to: dish name, dish category, ingredient type, raw material amount, cooking steps, and instruction type.

[0090] In the embodiment of the present application, the operation guidance model can be trained by using a back propagation training method until the operation guidance model training is completed. Figure 4 , which is a flow chart of the training steps of the operation guidance model according to an embodiment of the present application.

[0091] Step S410, when performing each training, obtaining a plurality of preset training samples; wherein, among the plurality of training samples, there are at least two modality types.

[0092] Step S420: For each of the training samples, extract sample feature information from the training sample.

[0093] Step S430, using the sample feature information corresponding to the multiple training samples to train a preset operation guidance model; wherein the operation guidance model is used to fuse the sample feature information corresponding to the multiple training samples into multimodal feature information and generate target operation guidance information based on the multimodal feature information.

[0094] Step S440, obtaining the target operation guidance information generated by the operation guidance model for each of the training samples and obtaining the real operation guidance information pre-set for each of the training samples.

[0095] The real operation guidance information pre-set for the training sample may be annotation information pre-set for the training sample.

[0096] Step S450, using a preset classification loss function, according to the target operation guidance information and the actual operation guidance information respectively corresponding to each training sample in the plurality of training samples, determines the classification loss value of the operation guidance model.

[0097] In the embodiment of the present application, the operation guidance model is a multi-label classification model, which can perform multi-label classification prediction based on the fused multimodal features, wherein each label in the multi-label corresponds to a content attribute, such as: dish type, ingredient type, cooking method and other multi-labels.

[0098] According to the content included in the target operation guidance information, determine the modality types that exist and the modality types that do not exist in the target operation guidance information; according to the content included in the real operation guidance information, determine the modality types that exist and the modality types that do not exist in the real operation guidance information; using a preset classification loss function, determine the classification loss value of the operation guidance model according to the modality types that exist and the modality types that do not exist in the target operation guidance information and the modality types that exist and the modality types that do not exist in the real operation guidance information.

[0099] In the embodiment of the present application, the classification loss function can adopt the following formula:

[0100]

[0101] in, represents the classification loss value; C represents the number of modal types; y im Represents the label value corresponding to the modality type m in the real operation guidance information corresponding to the i-th node; represents the label value corresponding to the modality type m in the target operation guidance information corresponding to the i-th node; N represents the number of nodes in the heterogeneous graph. Among them, the modality type m in the actual operation guidance information and the modality type m in the target operation guidance information are the same modality type.

[0102] The tag value corresponding to the modality type is used to indicate whether the modality type exists. If the tag value is the first value, it indicates that the modality type m exists; if the tag value is the second value, it indicates that the modality type m does not exist. The first value and the second value are preset values ​​and can be set as required. For example, the first value is 1 and the second value is 0.

[0103] Step S460, using a preset modal loss function, according to the target operation guidance information and the actual operation guidance information respectively corresponding to each training sample in the plurality of training samples, determines the modal loss value of the operation guidance model.

[0104] In the embodiment of the present application, the modal loss function may adopt the following formula:

[0105]

[0106] in, represents the modal loss value; f i represents the real operation guidance information corresponding to the i-th node; Represents the target operation guidance information corresponding to the i-th node.

[0107] Furthermore, the real operation guidance information and the target operation guidance information have the same information format, and the content attributes corresponding to the same position are the same, so that when the real operation guidance information and the target operation guidance information are subtracted, the content attributes at the same position are the same.

[0108] Step S470: Determine the model loss value corresponding to the operation guidance model according to the classification loss value and the modal loss value, and determine the training state of the operation guidance model according to the model loss value corresponding to the operation guidance model.

[0109] In the embodiment of the present application, the model loss value may be a weighted sum of the classification loss value and the modal loss value. For example, the model loss value may be determined by sampling the following formula:

[0110]

[0111] in, is the model loss value corresponding to the operation guidance model; λ is the weight hyperparameter.

[0112] The training status of the operation guidance model refers to whether the operation guidance model meets the conditions for completing training. The conditions for completing training include: the training of the operation guidance model has been iterated for a preset first number of times; or the model loss value of the operation guidance model has been continuously less than a preset loss threshold for a preset second number of times. The first number and the second number are empirical values ​​or values ​​obtained through experiments.

[0113] Furthermore, it is determined whether the model loss value corresponding to the operation guidance model is greater than a preset loss threshold. If so, the parameters in the operation guidance model are adjusted; if not, when the model loss value corresponding to the operation guidance model has been less than the loss threshold for a preset second number of times in a row, it is determined that the operation guidance model has converged and the training is completed.

[0114] If the training of the operation guidance model has not yet iterated for the preset first number of times, or the model loss value of the operation guidance model is greater than or equal to the loss threshold, or the model loss value of the operation guidance model has not been continuously less than the loss threshold for the preset second number of times, then jump to step S410 and continue the next training of the operation guidance model until the training status of the operation guidance model is that the training is completed.

[0115] Furthermore, an AdamW optimizer may be used to determine whether the operation guidance model has reached the conditions for completing training, thereby preventing overfitting and improving the generalization ability of the model.

[0116] In an embodiment of the present application, the operation guidance model is trained by a multi-task joint loss function, which can simultaneously optimize the classification loss and the modal coordination consistency loss, thereby improving the overall performance of the operation guidance model.

[0117] In an embodiment of the present application, multimodal fusion technology can be used to combine information from different modalities to provide more accurate and comprehensive cooking assistance services.

[0118] Furthermore, from an application perspective, the embodiments of the present application have broad application prospects in scenarios such as smart kitchens and smart ovens. By integrating multimodal inputs such as ingredient images, cooking voices, and recipe texts, it is possible to accurately identify dishes, ingredient types, and cooking methods, accurately identify user intent, and provide users with detailed cooking instructions. This multimodal fusion approach not only improves the user's cooking experience, but also provides real-time feedback and personalized suggestions through devices such as voice assistants and smart displays, enhancing the user's interactive experience. At the same time, through mobile terminal deployment, smart speaker integration, and cloud system support, seamless linkage of multiple terminals is achieved, providing strong technical support for the popularization and development of smart kitchens. Overall, high-precision cooking intent recognition is achieved technically through multimodal fusion and graph attention mechanisms, and the user experience of smart cooking is greatly improved through multi-terminal integration and real-time interaction in application.

[0119] Further, from a technical point of view, the embodiment of the present application introduces a graph attention mechanism to fuse multimodal data, which greatly improves the comprehensive utilization efficiency of cross-modal information. By constructing a heterogeneous graph, the image, speech and text features are regarded as different types of nodes, and gated graph convolution is used for fusion. This method can effectively mine and utilize the deep associations between the modalities. In addition, the use of pre-trained models such as ResNet, Transformer and BERT as unimodal encoders ensures the high precision and robustness of feature extraction of each modality. The design of multi-task loss function (classification loss function and modality loss function) can simultaneously optimize multi-label classification and multimodal collaborative consistency, thereby improving the generalization ability and overall performance of the operation guidance model. In particular, the use of multi-head attention mechanism further improves the accuracy of recognition.

[0120] In an embodiment of the present application, after the operation guidance model completes training, the operation guidance model can be subjected to model compression and knowledge distillation, and the final operation guidance model can be deployed to the client or cloud server for user use. Among them, model compression refers to removing unimportant weights or connections in the operation guidance model through model pruning, quantization and decomposition, reducing the modules in the operation guidance model, thereby reducing the number of parameters in the operation guidance model, reducing model complexity and computational complexity. Knowledge distillation refers to training a smaller student model to imitate the output of a large pre-trained model (teacher model). Furthermore, tools such as TensorFlow Lite or Open Neural Network Exchange (ONNX) can be used to deploy the final operation guidance model to the client. An application programming interface (API) can be provided on the mobile terminal and IoT device where the operation guidance model is located, and remote call and interaction functions are provided through the API interface.

[0121] In this way, users can use the operation guidance model to complete cooking, and can make full use of information of multiple modal types in a complex cooking environment to provide accurate cooking guidance in real time, thereby providing users with a convenient interaction method and personalized cooking suggestions, improving the intelligence level and user satisfaction of the cooking assistance system. For example: On a smart kitchen device deployed with an operation guidance model, the user wants to learn how to make a classic Italian pizza. This smart kitchen device is equipped with a camera, voice recognition and touch screen functions, which can provide multimodal interaction and guidance. The user puts the Italian pizza he is making under the camera of the smart kitchen device, so that the smart kitchen device can obtain a real-time image of the pizza. The user tells the smart kitchen device "I want to learn to make Italian pizza" through voice. At this time, the operation guidance model can be used to provide feedback and guidance to the user. By analyzing the real-time image captured by the camera, the types of pizza ingredients, such as dough, tomato sauce, cheese, ham, vegetables, etc., are identified. The user's voice instructions are converted into text through voice recognition technology, and combined with the identified types of ingredients, the user's learning intention can be more comprehensively understood. The operation guidance information for Italian pizza, at this time, the cooking steps, required ingredients and cooking skills of Italian pizza are retrieved and presented to the user from the built-in recipe database.

[0122] The present application also provides a training device for an operation guidance model. Figure 5 , which is a structural diagram of a training device for an operation guidance model according to an embodiment of the present application.

[0123] The training device of the operation guidance model includes:

[0124] The acquisition module 510 is used to acquire a plurality of preset training samples; wherein the plurality of training samples have at least two modality types.

[0125] The extraction module 520 is used to extract sample feature information from each training sample.

[0126] The training module 530 is used to train a preset operation guidance model using the sample feature information corresponding to the multiple training samples; wherein the operation guidance model is used to fuse the multimodal feature information corresponding to the multiple training samples according to the sample feature information corresponding to the multiple training samples, and for each training sample, generate the target operation guidance information corresponding to the training sample according to the multimodal feature information corresponding to the training sample.

[0127] The functions of the device described in the embodiment of the present application have been described in the above method embodiment. Therefore, for any details not provided in the description of this embodiment, please refer to the relevant descriptions in the above embodiment and will not be repeated here.

[0128] The present application also provides a training device for an operation guidance model, such as Figure 6 , which is a structural diagram of a training device for an operation guidance model according to an embodiment of the present application.

[0129] The training device of the operation guidance model includes: a processor 610, a communication interface 620, a memory 630 and a communication bus 640. The processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640.

[0130] The memory 630 is used to store computer programs.

[0131] In one embodiment of the present application, the processor 610, when executing the program stored in the memory 630, implements the training method of the operation guidance model provided by any of the aforementioned method embodiments, including: obtaining a plurality of preset training samples; wherein, among the plurality of training samples, there are at least two modal types; for each of the training samples, extracting sample feature information from the training sample; using the sample feature information corresponding to the plurality of training samples to train the preset operation guidance model; wherein, the operation guidance model is used to fuse the multimodal feature information corresponding to the plurality of training samples according to the sample feature information corresponding to the plurality of training samples, and for each of the training samples, generate the target operation guidance information corresponding to the training sample according to the multimodal feature information corresponding to the training sample.

[0132] Among them, for each of the training samples, extracting sample feature information from the training sample includes: performing preprocessing on the training sample according to the modality type corresponding to the training sample; and extracting sample feature information from the preprocessed training sample according to the modality type corresponding to the training sample.

[0133] Among them, the extracting sample feature information from the preprocessed training sample according to the modality type corresponding to the training sample includes: when the modality type corresponding to the training sample is an image type, using a preset visual feature extraction model to extract visual feature information from the preprocessed training sample; when the modality type corresponding to the training sample is a speech type, using a preset acoustic feature extraction model to encode the preprocessed training sample to obtain acoustic feature information; when the modality type corresponding to the training sample is a text type, using a preset text feature extraction model to encode the preprocessed training sample to obtain text feature information.

[0134] Among them, the operation guidance model includes: a graph attention network and a prediction neural network; the multimodal feature information corresponding to multiple training samples is fused according to the sample feature information corresponding to the multiple training samples, and for each training sample, the target operation guidance information corresponding to the training sample is generated according to the multimodal feature information corresponding to the training sample, including: using the graph attention network to construct a heterogeneous graph according to the sample feature information corresponding to the multiple training samples, and using the heterogeneous graph to fuse the multimodal feature information corresponding to the multiple training samples; for each training sample, using the prediction neural network to generate the target operation guidance information corresponding to the training sample according to the multimodal feature information corresponding to the training sample.

[0135] Among them, constructing a heterogeneous graph based on the sample feature information corresponding to the multiple training samples includes: for each of the training samples, determining the training sample as a node in the heterogeneous graph, and taking the modal type corresponding to the training sample as the node type of the node; among the multiple sample feature information, determining any two of the sample feature information whose correlation is greater than a preset correlation threshold, and establishing an edge between the nodes corresponding to the two sample feature information.

[0136] Among them, the use of the heterogeneous graph to fuse the multimodal feature information corresponding to the multiple training samples includes: for any two connected nodes in the heterogeneous graph, determining the attention weight between the two connected nodes according to the sample feature information corresponding to the two connected nodes; for any two connected nodes in the heterogeneous graph, determining a reference node among the two connected nodes, and normalizing the attention weights corresponding to the two connected nodes according to the attention weights between the reference node and its neighboring nodes; for each node in the heterogeneous graph, using the normalized attention weights corresponding to the node and the nodes connected to it, updating the sample feature information corresponding to the node, and using the updated sample feature information corresponding to the node as the multimodal feature information corresponding to the node.

[0137] Among them, the method of training a preset operation guidance model using sample feature information corresponding to each of the multiple training samples includes: obtaining target operation guidance information generated by the operation guidance model for each of the training samples and obtaining real operation guidance information pre-set for each of the training samples; using a preset classification loss function, according to the target operation guidance information and the real operation guidance information corresponding to each of the multiple training samples, determining the classification loss value of the operation guidance model; using a preset modal loss function, according to the target operation guidance information and the real operation guidance information corresponding to each of the multiple training samples, determining the modal loss value of the operation guidance model; determining the model loss value corresponding to the operation guidance model based on the classification loss value and the modal loss value, and determining the training status of the operation guidance model based on the model loss value corresponding to the operation guidance model.

[0138] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the training method of the operation guidance model provided in any of the aforementioned method embodiments are implemented. Since the training method of the operation guidance model has been described in detail above, for any details not detailed in the description of this embodiment, please refer to the relevant description in the aforementioned embodiment, and no further description will be given here.

[0139] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0140] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0141] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0142] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A training method for an operation guidance model, characterized in that: include: Acquire a plurality of preset training samples; wherein, among the plurality of training samples, there are at least two modality types; For each of the training samples, extracting sample feature information from the training sample; The preset operation guidance model is trained using the sample feature information corresponding to the multiple training samples; wherein the operation guidance model is used to fuse the multimodal feature information corresponding to the multiple training samples according to the sample feature information corresponding to the multiple training samples, and for each training sample, generate the target operation guidance information corresponding to the training sample according to the multimodal feature information corresponding to the training sample.

2. The method according to claim 1, characterized in that For each of the training samples, extracting sample feature information from the training sample includes: Preprocessing the training samples according to the modality type corresponding to the training samples; According to the modality type corresponding to the training sample, sample feature information is extracted from the preprocessed training sample.

3. The method according to claim 2, characterized in that The extracting sample feature information from the preprocessed training sample according to the modality type corresponding to the training sample includes: When the modality type corresponding to the training sample is an image type, extracting visual feature information from the preprocessed training sample using a preset visual feature extraction model; When the modality type corresponding to the training sample is a speech type, using a preset acoustic feature extraction model, encoding the preprocessed training sample to obtain acoustic feature information; When the modality type corresponding to the training sample is a text type, a preset text feature extraction model is used to encode the preprocessed training sample to obtain text feature information.

4. The method according to claim 1, characterized in that: The operation guidance model includes: a graph attention network and a prediction neural network; The step of fusing the multimodal feature information respectively corresponding to the plurality of training samples according to the sample feature information respectively corresponding to the plurality of training samples and generating, for each of the training samples, the target operation guidance information corresponding to the training sample according to the multimodal feature information corresponding to the training sample comprises: Using the graph attention network, constructing a heterogeneous graph according to the sample feature information corresponding to the plurality of training samples, and fusing the multimodal feature information corresponding to the plurality of training samples using the heterogeneous graph; For each of the training samples, the prediction neural network is used to generate target operation guidance information corresponding to the training sample according to the multimodal feature information corresponding to the training sample.

5. The method according to claim 4, characterized in that The constructing a heterogeneous graph according to the sample feature information respectively corresponding to the plurality of training samples comprises: For each of the training samples, determining the training sample as a node in the heterogeneous graph, and using the modality type corresponding to the training sample as the node type of the node; Among the plurality of sample feature information, any two sample feature information whose correlation is greater than a preset correlation threshold are determined, and edges are established between the nodes respectively corresponding to the two sample feature information.

6. The method according to claim 5, characterized in that The method of utilizing the heterogeneous graph to fuse the multimodal feature information corresponding to the plurality of training samples includes: For any two connected nodes in the heterogeneous graph, determining an attention weight between the two connected nodes according to sample feature information respectively corresponding to the two connected nodes; For any two connected nodes in the heterogeneous graph, a reference node is determined between the two connected nodes, and the attention weights corresponding to the two connected nodes are normalized according to the attention weights between the reference node and its neighboring nodes; For each node in the heterogeneous graph, the normalized attention weights corresponding to the node and the nodes connected to the node are used to update the sample feature information corresponding to the node, and the updated sample feature information corresponding to the node is used as the multimodal feature information corresponding to the node.

7. The method according to claim 1, characterized in that The method of training a preset operation guidance model using sample feature information respectively corresponding to a plurality of training samples includes: Obtain target operation guidance information generated by the operation guidance model for each of the training samples and obtain real operation guidance information pre-set for each of the training samples; Determining a classification loss value of the operation guidance model by using a preset classification loss function according to the target operation guidance information and the real operation guidance information respectively corresponding to each training sample in the plurality of training samples; Determining a modal loss value of the operation guidance model according to the target operation guidance information and the real operation guidance information respectively corresponding to each training sample in the plurality of training samples by using a preset modal loss function; A model loss value corresponding to the operation guidance model is determined according to the classification loss value and the modal loss value, and a training state of the operation guidance model is determined according to the model loss value corresponding to the operation guidance model.

8. A training device for an operation guidance model, characterized in that: include: An acquisition module is used to acquire a plurality of preset training samples; wherein, among the plurality of training samples, there are at least two modality types; An extraction module, configured to extract sample feature information from each training sample; A training module is used to train a preset operation guidance model using the sample feature information corresponding to the multiple training samples; wherein the operation guidance model is used to fuse the multimodal feature information corresponding to the multiple training samples according to the sample feature information corresponding to the multiple training samples, and for each training sample, generate the target operation guidance information corresponding to the training sample according to the multimodal feature information corresponding to the training sample.

9. A training device for an operation guidance model, characterized in that: include: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor coupled to the at least one bus; At least one memory connected to the at least one bus, wherein the processor is configured to: execute a training program of the operation guidance model stored in the memory to implement the training method of the operation guidance model described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed to implement the training method of the operation guidance model described in any one of claims 1-7.

Citation Information

Cited By

  • Equipment operation training auxiliary method and system

    CN121544440A

  • An equipment operation training assistance method and system

    CN121544440B