Food material recommendation method and device based on knowledge graph, and storage medium

Abstract

The application discloses a kind of food material recommendation methods based on knowledge graph, comprising the following steps: obtaining real-time audio and video data, real-time text data and historical review text data;It is preprocessed, and obtain effective real-time voice, real-time text, real-time video and historical review text data;Transcription effective real-time voice and video data are voice text data and image text data;According to the voice text data, real-time text data and image text data, obtain real-time vector matrix;The historical vector matrix corresponding to the historical review text data is obtained;Based on the real-time vector matrix and historical vector matrix, real-time semantic similarity and historical semantic similarity are fused to generate food material recommendation list and output.The method effectively improves the accuracy of recommended food material content, and improves the effect of user.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a method, device, and storage medium for recommending food ingredients based on knowledge graphs. Background Technology

[0002] With the widespread application of natural language understanding, multimodal deep learning, recommendation systems, knowledge graphs, and multi-source heterogeneous big data technologies in smart appliances and refrigeration equipment, some problems have also arisen. For example, the extraction of semantic similarity of food texts is limited by a single method and data sparsity. These problems further affect the accuracy of food content recommendations, thereby reducing the user experience.

[0003] Currently, in the fields of smart appliances and refrigeration equipment, content recommendation methods for topics such as food ingredients or health knowledge primarily employ traditional collaborative filtering algorithms, content recommendation algorithms, and fusion recommendation algorithms. While most of these algorithms are semantic-level recommendation algorithms, they still suffer from inherent data sparsity or cold-start issues. Therefore, the methods described above cannot extract optimal textual semantic representation information. Consequently, the industry has yet to propose a truly effective solution for recommending food ingredients, dietary health, and nutritional knowledge using integrated algorithms that combine deep learning, multi-source heterogeneity, multimodal cognition, and knowledge graphs. Summary of the Invention

[0004] The purpose of this invention is to provide a method, device, and storage medium for recommending food ingredients based on knowledge graphs.

[0005] This invention provides a knowledge graph-based food recommendation method, comprising the following steps:

[0006] Acquire real-time audio and video data, real-time text data, and historical comment text data; preprocess the real-time audio and video data, real-time text data, and historical comment text data to obtain real-time speech data, real-time text data, real-time video data, and historical comment text data; transcribe the valid real-time speech data into speech-text data; transcribe the valid real-time video data into image-text data; obtain a real-time vector matrix based on the speech-text data, real-time text data, and image-text data; obtain the historical vector matrix corresponding to the historical comment text data; based on the real-time vector matrix and the historical vector matrix, fuse real-time semantic similarity and historical semantic similarity to generate a food recommendation list; output the food recommendation list.

[0007] As a further improvement of the present invention, the step of "preprocessing the real-time audio and video data, real-time text data, and historical comment text data to obtain real-time voice data, real-time text data, real-time video data, and historical comment text data" specifically includes: performing data cleaning, format parsing, format conversion, and data storage on the real-time audio and video data, real-time text data, and historical comment text data to obtain real-time audio and video data, real-time text data, and historical comment text data; using scripts or third-party tools to separate the effective audio and video data into voice and video to obtain the real-time voice data and real-time video data; preprocessing the real-time voice and video data, including: performing frame segmentation and windowing processing on the real-time voice data, and performing cropping and frame segmentation processing on the real-time video data; preprocessing the real-time text data and historical comment text data, including: word segmentation, removal of stop words, and removal of duplicate words.

[0008] As a further improvement of the present invention, the "transcription of the effective speech data into speech-text data" specifically includes: extracting features from the effective speech data to obtain speech features; inputting the speech features into a speech recognition multi-channel, multi-size deep recurrent convolutional network model to transcribe the speech features into first speech-text data; outputting the alignment relationship between the speech features and the first speech-text data based on a connection-time classification method to obtain second speech-text data; obtaining key features or weight information of the key features of the second speech-text data based on an attention mechanism; combining the second speech-text data and its key features or weight information of the key features through a fully connected layer, and then calculating the score through a classification function to obtain the speech-text data.

[0009] As a further improvement of the present invention, the "extraction of the effective speech data features" specifically includes: extracting the effective speech data features and obtaining its Mel frequency cepstral coefficient features.

[0010] As a further improvement of the present invention, the "transcription of the effective real-time video data into image text data" specifically includes: inputting the real-time video data into a 3D deep recurrent convolutional neural network to calculate and obtain image features; inputting the image features into a multi-channel multi-size temporal convolutional network to transcribe and obtain first image text data; outputting the alignment relationship between the image features and the first image text data based on a connection-time classification method to obtain second image text data; and combining the second image text data through a fully connected layer and then calculating the score through a classification function to obtain the image text data.

[0011] As a further improvement of the present invention, the step of "obtaining a real-time vector matrix based on the voice text data, real-time text data and image text data" specifically includes: performing entity extraction and entity alignment processing on the text data to obtain multiple normalized entities; querying the food knowledge graph based on each entity to obtain the corresponding entity vector; and inputting the entity vector into an attention mechanism for calculation to obtain the real-time vector matrix.

[0012] As a further improvement of the present invention, both the entity extraction and the food knowledge graph are represented using triples; the entity alignment can be achieved using a knowledge representation learning method.

[0013] As a further improvement of the present invention, the step of "obtaining the historical comment vector matrix corresponding to the historical comment text data" specifically includes: performing entity extraction and entity alignment processing on the historical comment text data to obtain multiple normalized historical comment entities; querying the food knowledge graph based on each historical comment entity to obtain the corresponding historical comment entity vector; and inputting the historical comment entity vector into an attention mechanism for calculation to obtain the historical comment vector matrix.

[0014] As a further improvement of the present invention, the historical text data includes user comment data, user clicks on ingredients, and ingredient browsing information.

[0015] As a further improvement of the present invention, the step of "fusing real-time semantic similarity and historical semantic similarity based on the real-time vector matrix and the historical vector matrix to generate a food recommendation list" specifically includes: obtaining the maximum real-time semantic similarity based on the real-time vector matrix; obtaining the maximum historical semantic similarity based on the historical vector matrix; and fusing the real-time semantic similarity and historical semantic similarity into a matrix-form objective optimization function for calculation to generate a food recommendation list.

[0016] As a further improvement of the present invention, the step of "obtaining the maximum real-time semantic similarity based on the real-time vector matrix" specifically includes: obtaining multiple ingredient vector matrices to be recommended based on the ingredient knowledge graph; calculating the semantic similarity between each ingredient vector matrix to be recommended and the real-time vector matrix to obtain the maximum semantic similarity value.

[0017] As a further improvement of the present invention, the step of "obtaining the maximum historical semantic similarity based on the historical vector matrix" specifically includes: obtaining multiple ingredient vector matrices to be recommended based on the ingredient knowledge graph; calculating the semantic similarity between each ingredient vector matrix to be recommended and the historical vector matrix to obtain the maximum semantic similarity value.

[0018] As a further improvement of the present invention, the "output of the ingredient recommendation list" specifically includes: converting the generated ingredient recommendation list into speech for output, and / or converting the generated ingredient recommendation list into speech and transmitting it to the client terminal for output, and / or converting the generated ingredient recommendation list into text for output, and / or converting the generated ingredient recommendation list into text and transmitting it to the client terminal for output, and / or converting the generated ingredient recommendation list into an image for output, and / or converting the generated ingredient recommendation list into an image and transmitting it to the client terminal for output.

[0019] As a further improvement of the present invention, the "acquiring real-time audio and video data, acquiring real-time text data, and acquiring historical comment text data" specifically includes: acquiring the real-time audio and video data collected by the voice acquisition device, and / or acquiring the real-time audio and video data transmitted from the client terminal; acquiring the real-time text data collected by the text acquisition device, and / or acquiring the real-time text data transmitted from the client terminal; acquiring the historical comment text data stored internally, and / or acquiring the historical comment text data stored externally, and / or acquiring the historical comment text data transmitted from the client terminal.

[0020] As a further improvement of the present invention, the "transcription of the speech data into speech text data" further includes: obtaining configuration data stored in an external cache, and performing text transcription on the speech data based on the configuration data using the multi-channel, multi-size, deep recurrent convolutional neural network model.

[0021] The present invention also provides an electrical device, comprising: a memory for storing executable instructions; and a processor for implementing the above-described knowledge graph-based food recommendation method when running the executable instructions stored in the memory.

[0022] The present invention also provides a refrigerator, comprising: a memory for storing executable instructions; and a processor for implementing the above-described knowledge graph-based food recommendation method when running the executable instructions stored in the memory.

[0023] The present invention also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the above-described knowledge graph-based food recommendation method.

[0024] The beneficial effects of this invention are as follows: The method provided by this invention makes comprehensive innovations from multiple aspects, including different data source channels, different data types, multi-source heterogeneous data, fusion features, and diversified output result delivery methods. It not only mines user text semantic feature information from multiple dimensions such as real-time user voice, video, images, text, and user historical comments on ingredients, user preferences for ingredients, and ingredients that users are interested in, but also recommends the best ingredient list by comprehensively considering multiple factors such as semantics and user ingredient recommendation evaluation scores.

[0025] By utilizing real-time, multi-source heterogeneous data and historical data as training datasets, this invention predicts user needs and recommends the best ingredients, ultimately obtaining rich, complementary, and correlated semantic feature information. In other words, this invention not only uncovers the inherent semantic relationships between users and ingredients in both real-time and historical data, but also considers semantically correlated and complementary data such as user ingredient preferences and reviews, thereby revealing the inherent implicit semantic features between ingredients. Therefore, this invention effectively integrates a deep fusion model combining multi-source heterogeneous data, multimodal cognition, knowledge graphs, and natural language understanding, fully utilizing both real-time and historical data to improve the accuracy of recommended ingredient content and enhance user experience. Attached Figure Description

[0026] Figure 1 This is a structural block diagram of the model involved in the knowledge graph-based food recommendation method according to one embodiment of the present invention.

[0027] Figure 2 This is a schematic diagram of the steps of a knowledge graph-based food recommendation method according to one embodiment of the present invention.

[0028] Figure 3 This is a schematic diagram illustrating the steps of acquiring real-time audio and video data, real-time text data, and historical comment text data in one embodiment of the present invention.

[0029] Figure 4 This is a schematic diagram illustrating the preprocessing steps for the real-time audio and video data, real-time text data, and historical comment text data in one embodiment of the present invention.

[0030] Figure 5 This is a schematic diagram illustrating the steps of transcribing the real-time voice data into voice-text data in one embodiment of the present invention.

[0031] Figure 6 This is a schematic diagram illustrating the step of transcribing the real-time video data into image text data in one embodiment of the present invention.

[0032] Figure 7 This is a schematic diagram illustrating the steps of obtaining a real-time vector matrix based on the text data in one embodiment of the present invention.

[0033] Figure 8 This is a schematic diagram illustrating the steps of generating a food recommendation list based on real-time semantic similarity and historical semantic similarity in one embodiment of the present invention. Detailed Implementation

[0034] The present invention will now be described in detail with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.

[0035] It should be noted that the term "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0036] An embodiment of the present invention is a food recommendation method based on a knowledge graph. Although this application provides the following embodiments or processes... Figure 1 The method described herein may involve steps that are based on conventional or non-creative labor, and in steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application.

[0037] like Figure 1 The diagram shown is a structural block diagram of the model involved in the knowledge graph-based food recommendation method provided by this invention. Figure 2 The diagram illustrates the steps of a knowledge graph-based food recommendation method, which includes:

[0038] S1: Obtain real-time audio and video data, real-time text data, and historical comment text data.

[0039] S2: Preprocess the real-time audio and video data, real-time text data, and historical comment text data to obtain real-time voice data, real-time text data, real-time video data, and historical comment text data.

[0040] S3: Transcribe the real-time voice data into voice-text data.

[0041] S4: Transcribe the real-time video data into image text data.

[0042] S5: Obtain the real-time vector matrix based on the spoken text data, real-time text data, and image text data.

[0043] S6: Obtain the historical vector matrix based on the historical comment text data.

[0044] S7: Based on the real-time vector matrix and the historical vector matrix, fuse the real-time semantic similarity and the historical semantic similarity to generate a food recommendation list.

[0045] S8: Output the recommended list of ingredients.

[0046] The method provided by this invention enables smart electronic devices to achieve real-time interaction or message push functions with users based on real-time audio and video data input. For example, in this embodiment, a smart refrigerator is used as an example, and the method is illustrated using a pre-trained deep learning model. Based on the user's audio and video input, the smart refrigerator predicts the user's food score according to the user's real-time audio and video data, real-time text data, and historical comments related to food. Based on the score results, recommended food or a food list can be obtained, and the required food names or food list can be calculated based on the recommended food or food list results.

[0047] like Figure 3 As shown, step S1 specifically includes:

[0048] S11: Acquire the real-time audio and video data collected by the acquisition device, and / or

[0049] The real-time audio and video data is acquired from the client terminal.

[0050] S12: Acquire the real-time text data collected by the acquisition device, and / or

[0051] The real-time text data is obtained from the client terminal.

[0052] S13: Retrieve historical comment text data from internal storage, and / or

[0053] Retrieve historical comment text data from external storage, and / or

[0054] Obtain historical comment text data transmitted from the client terminal.

[0055] The real-time audio and video data mentioned here refers to data that simultaneously includes real-time voice and real-time video. Preprocessing and separating this real-time audio and video data can generate real-time voice and video data. Of course, the real-time voice and video data can also be collected independently using appropriate acquisition devices. Real-time voice refers to the interrogative or commanding statements spoken by the user to the smart electronic device or a client terminal device connected to the smart electronic device. Similarly, it can also be voice information collected by a voice acquisition device. For example, in this embodiment, the user can ask questions such as "What vegetables are in the refrigerator today?" or "What beef ingredients are in the refrigerator today?", or issue commands such as "Delete all ingredients." The real-time video data is real-time video images captured in real-time by the smart electronic device or a client terminal device connected to the smart electronic device. For example, in this real-time method, a camera built into the smart refrigerator captures an image of the user's face, and the lip region feature image is extracted from the facial image to identify the corresponding text content, such as identifying the image text data "What vegetables are in the refrigerator today?".

[0056] The historical comment text data mentioned here refers to users' past comments on ingredients, clicks related to ingredients, and browsing information related to ingredients during previous use. This data reflects users' preferences, interests, or comments on ingredients. Furthermore, it may also include user-inputted historical ingredient comment text data. Specifically, in this embodiment, the user's historical ingredient comment text data may also include: text data of instructions or questions raised by users regarding ingredients in the past. The acquired text data includes information related to the current real-time audio / video data or real-time text data, and may also be explanatory text information issued by users based on the ingredients they have added during previous use. The acquisition of historical comment text data can be used as part of the dataset for pre-training and prediction models, effectively supplementing the single semantic representation of real-time audio / video data and real-time text data, and enriching semantic features.

[0057] As described in step S11, in this embodiment, real-time audio and video of the user can be collected by audio and video acquisition devices such as cameras or webcams installed inside the smart refrigerator. During use, when the user needs to interact with the smart refrigerator, they can simply speak to the smart refrigerator. Furthermore, real-time audio and video data transmitted from the user can also be obtained through a client terminal connected to the smart refrigerator via a wireless communication protocol. The client terminal is an electronic device with information sending capabilities, such as a mobile phone, tablet computer, smart camera, smartwatch, APP, or Bluetooth-enabled smart electronic device. During use, the user can directly speak to the client terminal or directly use the built-in camera of the refrigerator to take pictures. After the client terminal collects the audio and video, it transmits it to the smart refrigerator via wireless communication methods such as Wi-Fi or Bluetooth. This achieves multi-channel real-time audio and video acquisition methods, not limited to speaking directly to the smart refrigerator. When the user has an interaction need, they can send real-time voice through any convenient channel, thereby significantly improving user convenience. In other embodiments of the present invention, one or more of the above-described real-time audio and video data acquisition methods can be used, or the real-time audio and video data can be obtained through other channels based on existing technology. The present invention does not impose specific limitations on this.

[0058] As described in step S12, in this embodiment, real-time text information generated during the interaction between the smart refrigerator and the user can be collected through various real-time text collection devices such as mobile phones, tablets, apps, mini programs, official accounts, web, or customer service, or directly input by the user.

[0059] As described in step S13, in this embodiment, historical review text data stored in the internal memory of the smart refrigerator can be read. Alternatively, historical review text data can be read from an external storage device configured on the smart refrigerator, such as a USB flash drive or SD card. Setting up an external storage device further expands the storage space of the smart refrigerator. Furthermore, the historical review text data can also be obtained from client terminals such as mobile phones or tablets, or application software servers. Implementing multiple channels for obtaining historical review text data can significantly increase the amount of historical review text data, thereby improving the accuracy of subsequent speech recognition and video image recognition. In other embodiments of the present invention, one or more of the above-described methods for obtaining historical review text data can be used, or the historical review text data can be obtained through other channels based on existing technology; the present invention does not impose specific limitations in this regard.

[0060] Furthermore, in this embodiment, the smart refrigerator is equipped with an external cache, in which at least a portion of the historical audio and video data is stored. As usage time increases, the amount of historical comment text data increases. By storing some of the data in the external cache, the internal storage space of the smart refrigerator can be saved. Moreover, when performing neural network calculations, the historical comment text data stored in the external cache can be directly read, thereby improving algorithm efficiency.

[0061] Specifically, in this embodiment, Redis is used as the external cache. Redis is a widely used distributed caching system with a key / value storage structure, and it can be used as a database, cache, and message queue broker. Other external caches such as Memcached may also be used in other embodiments of the invention, and the invention does not impose specific limitations on this.

[0062] In summary, steps S11 to S13 enable the flexible acquisition of real-time audio and video data, real-time text data, and historical user comments on food ingredients through multiple channels. This not only improves the user experience but also ensures sufficient data volume and effectively enhances algorithm efficiency.

[0063] like Figure 4 As shown, step S2 specifically includes the following steps:

[0064] S21: Perform data cleaning on the real-time audio and video data and historical comment text data to obtain real-time audio and video data and historical comment text data.

[0065] S22: Separate the real-time audio and video data into voice and video data to obtain the real-time voice data and real-time video data.

[0066] S23: Preprocess the real-time voice and video data, including: performing frame segmentation and windowing on the real-time voice data, and performing cropping and frame segmentation on the real-time video data.

[0067] S24: Preprocess the real-time text data and historical comment text data, including: word segmentation, removal of stop words, and removal of duplicate words.

[0068] In step S21, the data cleaning of the real-time audio and video data and historical comment text data specifically includes:

[0069] A certain amount of real-time audio and video datasets and user historical food review text data are obtained. For example, they can be imported into a data cleaning model in the form of files for processing. To prevent data import failure, data that does not meet the file import format is parsed and converted. Then, irrelevant data, duplicate data, and outlier and missing values ​​are deleted from the dataset. Information irrelevant to the classification is initially filtered out. The audio and video data is then cleaned, and the cleaned data is output and saved in a specified format, thereby obtaining valid audio and video data and user historical food review text data.

[0070] In step S22, a script or a third-party audio-video separation tool is used to separate the voice and video data of the valid audio and video data, thereby obtaining voice data and video data.

[0071] In this embodiment of the invention, Python language can be used to write the audio and video separation script, or a third-party audio and video separation tool can be used to separate the input audio and video data to achieve the separation of voice and video and obtain classified voice and video data.

[0072] In step S23, the classified speech is segmented according to a specified time period or number of samples, completing the frame-by-frame processing of the speech to obtain speech signal data. Then, through the action of a window function, the originally noisy speech signal exhibits signal enhancement and periodicity characteristics, completing the windowing process, which facilitates better extraction of speech feature parameters in subsequent steps. For example, step S23 also includes cropping effective video data to generate multiple frames. Specifically, a script can be written to first load the video data and read the video information, then decode the video based on the video information to determine how many images are displayed per second, thereby obtaining single-frame image information. The single-frame image information includes the width and height of each frame image. Finally, the video is saved as multiple images. Therefore, after the processing in step S23, effective speech data and image data can be obtained. In other embodiments of the present invention, other video frame-by-frame methods, such as third-party video cropping tools, can also be used, and the present invention does not impose specific limitations on this.

[0073] In step S24, the text data is preprocessed. Furthermore, in this embodiment, various text preprocessing methods such as word segmentation, stop word removal, duplicate word removal, and word frequency statistics can be used to perform the real-time text data and historical comment text data by running scripts or third-party tools. The specific preprocessing content can be selected according to actual needs, and the present invention does not impose specific limitations.

[0074] like Figure 5 As shown, in step S3, it specifically includes:

[0075] S31: Extract the features of the effective speech data to obtain speech features.

[0076] S32: Input the speech features into a speech recognition multi-channel, multi-size deep recurrent convolutional neural network model to transcribe the first speech text data.

[0077] S33: Based on the connection-time classification method, output the alignment relationship between the speech features and the first speech-text data to obtain the second speech-text data.

[0078] S34: Based on the attention mechanism, obtain the key features of the second speech text data or the weight information of the key features.

[0079] S35: The second speech text data and its key features or the weight information of key features are combined through a fully connected layer, and then the score is calculated through a classification function to obtain the speech text data.

[0080] In step S31, extracting the features of the effective speech data specifically includes:

[0081] The speech data features are extracted to obtain its Mel-scale Frequency Cepstral Coefficients (MFCCs). MFCCs are a distinctive component of a speech signal, extracted in the Mel-scale frequency domain. The Mel scale describes the nonlinear characteristics of human ear frequencies, and the parameters of MFCCs take into account the degree of human ear sensitivity to different frequencies, making them particularly suitable for speech discrimination and speaker identification.

[0082] In this embodiment of the invention, perceptual linear predictive features (PLP) or linear predictive coefficient features (LPC) and other feature parameters of the speech data can also be obtained through different algorithm steps to replace MFCC features. The specific adjustments can be made according to the actual application scenario and the model parameters used. This invention does not impose any specific limitations on this.

[0083] The specific algorithm steps involved in the above steps can be referred to the existing technology in this field, and the specific content will not be described in detail here.

[0084] In step S32, the effective speech data is transcribed into text using a network model in automatic speech recognition technology to obtain the first speech-text data.

[0085] In this embodiment, a multi-channel, multi-size deep recurrent convolutional neural network (RNN) model is constructed by increasing the network width to achieve the speech-to-text task. This deep network model consists of multiple layers of deep RRNNs, which typically comprises several convolutional layers and several fully connected layers, including various nonlinear operations and pooling operations. It is primarily used to process grid-structured data, thus enabling the model to filter out the contours between adjacent pixels using filters. Furthermore, this model first extracts speech feature values ​​and then calculates these features, rather than calculating the original speech data values. Therefore, compared to traditional recurrent neural networks, deep RRNNs offer advantages such as lower computational cost and easier characterization of local features. Shared weights and pooling layers provide better time-domain or frequency-domain invariance, and the deeper nonlinear structure also gives the model powerful representation capabilities. Additionally, the multi-channel, multi-size approach allows for the extraction of speech features from different perspectives, acquiring more speech feature information and resulting in better speech recognition accuracy.

[0086] Specifically, in this embodiment, the multi-channel, multi-size deep recurrent convolutional neural network used in step S32 consists of a 3*3 convolutional layer, 32 channels, and a max pooling layer.

[0087] In step S33, the alignment relationship between the input speech feature sequence and the output speech text feature sequence is obtained using the Connectionist temporal classification (CTC) method.

[0088] In this embodiment, it is difficult to establish an accurate mapping relationship between the effective speech data and the text of the first speech-text data, thus increasing the difficulty of subsequent speech recognition. To solve this problem, a temporal classification method is adopted. This method is generally used after using a convolutional network model. It is a completely end-to-end acoustic model training method that does not require pre-alignment of the data. It only requires an input sequence and an output sequence for training, without the need for data alignment or one-to-one annotation. At the same time, it can directly output the probability of sequence prediction. Based on this prediction probability, we can obtain the most likely text output result to obtain the second speech-text data.

[0089] Furthermore, in step S34, the attention mechanism can guide the deep convolutional neural network to focus on more critical feature information while suppressing other non-critical feature information. Therefore, by introducing the attention mechanism, local key features or weight information of the second speech text data can be obtained, thereby further reducing the irregular error alignment phenomenon of the sequence during model training.

[0090] Here, in step S35, based on the second speech text data and its key features or the weight information of key features, the second speech text data is assigned its own weight information through a model that combines a self-attention mechanism and a fully connected layer, so as to better obtain the internal weight information of the text semantic features of the speech text data, thereby enhancing the importance of different parts of the text semantic feature information. Finally, the speech text data is obtained by calculating the score through a classification function.

[0091] like Figure 6 As shown, step S4 specifically includes:

[0092] S41: Input the real-time video data into a 3D deep recurrent convolutional neural network to calculate and obtain image features.

[0093] S42: Input the image features into a multi-channel, multi-size temporal convolutional network for transcription to obtain the first image text data.

[0094] S43: Based on the connection-time classification method, output the alignment relationship between the image features and the first image text data to obtain the second image text data.

[0095] S44: The second image text data is combined through a fully connected layer and then scored using a classification function to obtain the image text data.

[0096] In step S41, considering the speaker's image features, the sentences that may be identified are quite complex, such as varying sentence lengths, different pause positions or word structures, and correlations between their image features. In this embodiment, the real-time video data is input into a 3D convolutional neural network model. By adding information in the time dimension, more expressive features can be extracted. The 3D convolutional neural network model can solve the correlation information between multiple images. It takes continuous multi-frame images as input and captures motion information in the input frames by adding a new dimension of information, thereby better obtaining their image features.

[0097] In step S42, the result generated by the 3D deep recurrent convolutional neural network model in step S41 is input into a multi-channel, multi-size temporal deep recurrent convolutional neural network model. After multi-channel, multi-kernel operation, multiple feature maps with the same number of convolutional kernels are output. For example, taking a convolutional layer with 3-channel input and 2 convolutional kernels as an example, 2 feature maps are output after convolution calculation. Considering the video image recognition method at the sentence level, in this embodiment, the video image lip reading recognition method is implemented using two steps: Pinyin sequence recognition (LipPic to Pinyin, P2P) and Chinese character sequence recognition (Pinyin to Chinese-Character, P2CC). This method implements a Chinese lip reading recognition method. Specifically, the temporal image features generated by the multi-channel, multi-size temporal deep recurrent convolutional neural network model are mapped to the Pinyin sequence of the Pinyin sentence, and then the Pinyin sequence is translated into the Chinese character sequence of the Chinese character sentence, finally obtaining the first image text data. Of course, other Chinese lip reading recognition methods are not specifically limited, as long as they can convert video images into corresponding text data, they are within the protection scope of this invention.

[0098] In steps S43 and S44, similar to the speech data processing method described above, a continuous temporal classification method is used to realize the mapping relationship between the effective video data and the text of the first image text data, thereby obtaining the second image text data. Then, through a model that combines a self-attention mechanism and a fully connected layer, the second image text data is assigned its own weight information and / or association weight information, thereby better obtaining the internal weight information and / or association weight information of the text semantic features of the image text data, enhancing the importance of different parts of the text semantic feature information. Finally, a classification function is used to calculate the score to obtain the image text data. The specific processing procedure is the same as the speech data processing steps described above, and will not be repeated here.

[0099] like Figure 7 As shown, step S5 specifically includes:

[0100] S51: Perform entity extraction and entity alignment processing on the text data to obtain multiple normalized entities.

[0101] S52: Query the food knowledge graph for each entity to obtain the corresponding entity vector.

[0102] S53: Input the entity vector into the attention mechanism to calculate and obtain the real-time vector matrix.

[0103] In step S51, entity extraction, also known as Named Entities Recognition (NER), primarily aims to identify the textual range of named entities and classify them into predefined categories. Academically, this generally includes three main categories: entity, time, and number. It mainly extracts atomic information elements from text data, such as names of people, organizations / institutions, geographical locations, events / dates, and character values. Entities are the most basic elements of a knowledge graph, and the completeness, accuracy, and recall of entity extraction directly affect the quality of the knowledge graph.

[0104] In this embodiment of the invention, entity extraction is performed on text data converted from collected multi-source heterogeneous data. Since different knowledge graphs have different focuses in collecting knowledge, the description of the same entity may differ across knowledge graphs. To determine whether two or more entities from different information sources refer to the same object in the real world, entity alignment processing is required on the extracted entity results. This involves constructing alignment relationships between multiple entities representing the same object and simultaneously fusing and aggregating the information contained within the entities. Entity alignment is a special type of cross-network relationship that requires analysis based on the characteristics of entity alignment relationships. Specifically, knowledge representation learning-based methods can be used to achieve entity alignment, thereby obtaining standardized or required entities, such as food-related entity information.

[0105] In step S52, a knowledge graph is essentially a knowledge base called a semantic network, that is, a knowledge base with a graph structure. The knowledge graph is a relatively general formal description framework for semantic knowledge. It uses nodes to represent semantic symbols and edges to represent the semantic relationships between symbols.

[0106] In this embodiment of the invention, the food entity vectors corresponding to the extracted entities are obtained based on a pre-constructed food knowledge graph. Specifically, according to the knowledge representation method in the knowledge graph, "entity-relationship-entity" triples are used to describe entities and relationships in the real world, forming a network-like knowledge structure through these relationships. Distributed representation learning of the knowledge graph performs distributed representation of entities and relationships in the knowledge graph, resulting in low-dimensional vector representations containing semantic relationships, i.e., obtaining the corresponding food entity vectors. The knowledge representation method further reflects multi-granularity and multi-level semantic relationships such as entities, categories, attributes, and relationships, thereby enriching the semantic information in the text data.

[0107] In step S53, to focus on the semantic information of the relationships between different entities or within entities themselves after entity extraction in the food knowledge graph, as well as the complementary relationships in related food text data, the entity vectors generated in step S52 are input into the attention mechanism. By guiding the entity features in the multi-hop neighborhood of a given entity or node, different attention weights are assigned to different entities. That is, different attention is given to different entities according to actual needs, and then information is obtained from the corresponding entities according to these weights, thereby obtaining the real-time vector matrix. Therefore, based on the food knowledge graph, the vectors of multiple different entities obtained from entity extraction are fused and aggregated, thereby fully exploring the deep semantics in various text data, supplementing the shortcomings of single features in speech and video data, enriching the semantic representation capabilities of text data, and optimizing subsequent text classification capabilities.

[0108] Similarly, the specific operational steps included in step S6 are similar to those in step S5 above:

[0109] Specifically, in this embodiment of the invention, entity extraction and entity alignment are also performed on the user's historical food review text data to obtain standardized historical food review entities; the food knowledge graph is queried based on the entities to obtain the corresponding historical review entity vectors; and the historical review entity vectors are then input into an attention mechanism for calculation to obtain the historical review vector matrix.

[0110] The user's historical food review data includes user historical reviews, user clicks on food, and browsing information. This data contains rich semantic features such as user food preferences and interests. Through attention mechanism processing, the complementary relationships of user food texts and the semantic information of the correlation between entities themselves or each other based on the knowledge graph can be obtained, thereby making up for the problem of data sparsity.

[0111] like Figure 8 As shown, step S7 specifically includes:

[0112] S71: Obtain the maximum real-time semantic similarity based on the real-time vector matrix.

[0113] S72: Obtain the maximum historical semantic similarity based on the historical vector matrix.

[0114] S73: The real-time semantic similarity and historical semantic similarity are fused into a matrix-form objective optimization function to predict the score, thereby generating a list of recommended ingredients.

[0115] Similarity calculations mainly include cosine similarity, Manhattan distance similarity, and Euclidean distance similarity. This invention does not impose specific limitations on the calculation methods. Using spatial distance similarity can reflect the semantic similarity of ingredients, thereby ensuring a better characterization of semantic relationships in subsequent ingredient recommendations.

[0116] In steps S71 and S72, based on the learning method of the food knowledge graph, the vector representations of all entities and relations in the domain to which the recommended food object belongs are obtained. The entity representation of the recommended food object is selected from the entity vector matrix. The vector representation of the recommended food object integrates the entity knowledge related to it in the entire domain, so the vector representation contains the contextual semantic knowledge of the recommended food object.

[0117] In this embodiment, based on the real-time vector matrix, the cosine or Manhattan distance method is used to calculate the real-time semantic similarity between any two ingredients to obtain the real-time semantic similarity with the highest similarity. Similarly, based on the historical vector matrix, the historical semantic similarity between any two ingredients can be calculated, and then the maximum historical semantic similarity is calculated. To ensure that the range of semantic similarity values ​​remains consistent, it is normalized, and the normalized result is the final similarity of the feature vectors of the two ingredients.

[0118] In step S73, the real-time semantic similarity and historical semantic similarity calculated in steps S71 and S72 are fused into a matrix-form objective optimization function to predict the score. Based on the principle that the higher the predicted score, the more interested the user is, a threshold or threshold range is set, and ingredients with predicted scores that meet the threshold or threshold range are recommended to the user. For example, the list of the top N ingredients with scores greater than the threshold is recommended to the user. Specifically, the ingredients with the highest scores can also be recommended to the user, and the specific ingredient recommendation information can be adjusted according to specific needs.

[0119] In summary, the knowledge graph-based food recommendation method provided by this invention can be obtained by sequentially following the above steps. By acquiring real-time audio and video data, real-time text data, and historical food review data, and performing data cleaning, while separating audio and video to generate effective audio and video data respectively, and converting them into corresponding text data, all of which are used as part of the dataset for pre-training and prediction models, thus obtaining more comprehensive text semantic features.

[0120] Furthermore, by constructing a multi-channel, multi-size deep recurrent convolutional network model that integrates connection-based temporal classification and attention mechanisms, as well as a temporal deep recurrent convolutional neural network model, richer high-level semantic feature information was mined and obtained. Finally, real-time and historical vector matrices were obtained based on various text data, and real-time and historical semantic similarities were calculated based on these matrices. These matrices were then fused into a matrix-form objective optimization function to predict scores, generating a food recommendation list, which was output in multiple ways. The overall model structure has excellent semantic representation capabilities for text data, demonstrating good complementarity and correlation in semantic features, thus improving the accuracy of the food recommendation method.

[0121] Step S8 specifically includes:

[0122] The generated ingredient recommendation list is converted into speech and output, and / or

[0123] The generated ingredient recommendation list is converted into voice and transmitted to the client terminal for output, and / or

[0124] Convert the generated ingredient recommendation list into text and output it, and / or

[0125] The generated ingredient recommendation list is converted into text and transmitted to the client terminal for output, and / or

[0126] Convert the generated food recommendation list into an image for output, and / or

[0127] The generated food recommendation list is converted into an image and transmitted to the client terminal for output.

[0128] As described in step S8, in this real-time mode, after obtaining the food recommendation list information through the above steps, it can be converted into speech and broadcast through the built-in sound playback device of the smart refrigerator. Alternatively, the result information can be converted into text and displayed directly through the display device configured on the smart refrigerator, or it can be converted into an image and displayed directly on the large screen of the smart refrigerator. Furthermore, the result information can also be transmitted via voice communication to a client terminal for output. Here, the client terminal is an electronic device with information receiving capabilities, such as transmitting the voice to a mobile phone, smart speaker, Bluetooth headset, or other devices for broadcast, or transmitting the food recommendation list result information in text or image form via SMS, email, or other means to a client terminal such as a mobile phone or tablet, or to an application software installed on the client terminal, for user viewing. This achieves a multi-channel, multi-category classification result information output method. Users are not limited to obtaining relevant information only near the smart refrigerator. Combined with the multi-channel, multi-category real-time voice acquisition method provided by this invention, users can directly interact with the smart refrigerator remotely, offering high convenience and significantly improving the user experience. In other embodiments of the present invention, only one or more of the above-mentioned classification result information output methods may be used, or the food recommendation list information may be output through other channels based on existing technology. The present invention does not impose specific limitations on this.

[0129] In summary, the present invention provides a knowledge graph-based food recommendation method that acquires real-time audio and video data, real-time text data, and historical user review data for food products through multiple channels. After processing the audio and video data, the voice and video data are converted into corresponding voice-text data and image-text data. After combining the audio and video to generate text, the semantic features of the text are fully extracted using a multi-channel, multi-size, deep recurrent convolutional neural network model and a multi-channel, multi-size, temporal, deep recurrent convolutional neural network model. Real-time and historical semantic similarity are calculated based on the text data, and then fused into a matrix-form objective optimization function to predict scores, thereby generating a food recommendation list. The food recommendation results are output through multiple channels. This method not only significantly improves the accuracy of generating food recommendations but also makes the interaction between users and smart refrigerators more convenient and diversified, greatly improving the user experience.

[0130] Based on the same inventive concept, the present invention also provides an electrical device, comprising:

[0131] Memory, used to store executable instructions;

[0132] When the processor runs the executable instructions stored in the memory, it implements the above-described knowledge graph-based food recommendation method.

[0133] Based on the same inventive concept, the present invention also provides a refrigerator, which includes:

[0134] Memory, used to store executable instructions;

[0135] When the processor runs the executable instructions stored in the memory, it implements the above-described knowledge graph-based food recommendation method.

[0136] Based on the same inventive concept, the present invention also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the above-described knowledge graph-based food recommendation method.

[0137] It should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0138] The detailed descriptions listed above are merely specific descriptions of feasible implementations of the present invention and are not intended to limit the scope of protection of the present invention. All equivalent implementations or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.

Claims

1. A food recommendation method based on knowledge graphs, characterized in that, Including the following steps: Acquire real-time audio and video data, acquire real-time text data, and acquire historical comment text data; The real-time audio and video data, real-time text data, and historical comment text data are preprocessed to obtain real-time voice data, real-time text data, real-time video data, and historical comment text data; The real-time voice data was transcribed into voice-to-text data; The real-time video data was transcribed into image and text data. Based on the spoken text data, real-time text data, and image text data, a real-time vector matrix is ​​obtained; Obtain the historical vector matrix corresponding to the historical comment text data; Based on the real-time vector matrix and the historical vector matrix, the real-time semantic similarity and the historical semantic similarity are fused to generate a food recommendation list; Output the recommended list of ingredients; The phrase "based on the real-time vector matrix and the historical vector matrix, fusing real-time semantic similarity and historical semantic similarity to generate a food recommendation list" includes: Based on the real-time vector matrix and the historical vector matrix, the corresponding maximum real-time semantic similarity and maximum historical semantic similarity are calculated respectively. These maximum real-time semantic similarity and maximum historical semantic similarity are then fused into a matrix-form objective optimization function to predict scores, thereby generating a food recommendation list. The real-time vector matrix and the historical vector matrix are obtained through the following methods: Entity extraction and alignment are performed on the voice text data, real-time text data, image text data, and historical comment text data to obtain multiple normalized entities. For each entity, the food knowledge graph is queried to obtain the corresponding entity vector, and the entity vector is input into an attention mechanism to calculate the corresponding real-time vector matrix and historical vector matrix.

2. The food recommendation method based on knowledge graphs according to claim 1, characterized in that, The phrase "preprocessing the real-time audio / video data, real-time text data, and historical comment text data to obtain real-time voice data, real-time text data, real-time video data, and historical comment text data" specifically includes: The real-time audio and video data, real-time text data, and historical comment text data are cleaned, format parsed, format converted, and stored to obtain real-time audio and video data, real-time text data, and historical comment text data; The real-time audio and video data are separated into voice and video using scripts or third-party tools to obtain the real-time voice data and real-time video data. The real-time voice data and real-time video data are preprocessed, including: performing frame segmentation and windowing on the real-time voice data, and performing cropping and frame segmentation on the real-time video data; The real-time text data and historical comment text data are preprocessed, including word segmentation, removal of stop words, and removal of duplicate words.

3. The food recommendation method based on knowledge graphs according to claim 1, characterized in that, The phrase "transing the real-time speech data into speech-text data" specifically includes: Extract features from the real-time speech data to obtain speech features; The speech features are input into a speech recognition multi-channel, multi-size deep recurrent convolutional network model to transcribe the first speech text data. The second speech text data is obtained by outputting the alignment relationship between the speech features and the first speech text data based on the connection-time classification method. Based on the attention mechanism, key features or weight information of the key features of the second speech-text data are obtained; The second speech-text data and its key features or the weights of key features are combined through a fully connected layer, and then the score is calculated using a classification function to obtain the speech-text data.

4. The knowledge graph-based food recommendation method according to claim 3, characterized in that, The phrase "extracting features from the real-time speech data" specifically includes: Extract features from the real-time speech data to obtain its Mel-frequency cepstral coefficient features.

5. The food recommendation method based on knowledge graphs according to claim 1, characterized in that, The phrase "transing the real-time video data into image text data" specifically includes: The real-time video data is input into a 3D deep recurrent convolutional neural network to calculate and obtain image features; The image features are input into a multi-channel, multi-size temporal convolutional network for transcription to obtain the first image text data. The alignment relationship between the image features and the first image text data is output based on the connection-time classification method to obtain the second image text data; The second image text data is combined through a fully connected layer, and then a score is calculated using a classification function to obtain the image text data.

6. The food recommendation method based on knowledge graphs according to claim 1, characterized in that, The method further includes: Both entity extraction and food knowledge graph are represented using triples; The entity alignment is achieved using a knowledge representation learning-based method.

7. The food recommendation method based on knowledge graphs according to claim 1, characterized in that, The method further includes: The historical comment text data includes user comment data, user clicks on ingredients, and ingredient browsing information.

8. The food recommendation method based on knowledge graphs according to claim 1, characterized in that, The phrase "calculating the corresponding maximum real-time semantic similarity and maximum historical semantic similarity based on the real-time vector matrix and the historical vector matrix respectively" specifically includes: Based on the food knowledge graph, multiple food vector matrices to be recommended are obtained; The semantic similarity between each ingredient vector matrix to be recommended and the real-time vector matrix is ​​calculated to obtain the maximum semantic similarity value.

9. The food recommendation method based on knowledge graphs according to claim 1, characterized in that, The phrase "calculating the corresponding maximum real-time semantic similarity and maximum historical semantic similarity based on the real-time vector matrix and the historical vector matrix respectively" specifically includes: Based on the food knowledge graph, multiple food vector matrices to be recommended are obtained; The semantic similarity between each ingredient vector matrix to be recommended and the historical vector matrix is ​​calculated to obtain the maximum semantic similarity value.

10. The food recommendation method based on knowledge graphs according to claim 1, characterized in that, The phrase "outputting the recommended ingredient list" specifically includes: The generated ingredient recommendation list is converted into speech and output, and / or The generated ingredient recommendation list is converted into voice and transmitted to the client terminal for output, and / or Convert the generated ingredient recommendation list into text and output it, and / or The generated ingredient recommendation list is converted into text and transmitted to the client terminal for output, and / or Convert the generated food recommendation list into an image for output, and / or The generated food recommendation list is converted into an image and transmitted to the client terminal for output.

11. The food recommendation method based on knowledge graphs according to claim 1, characterized in that, The phrase "acquiring real-time audio and video data, acquiring real-time text data, and acquiring historical comment text data" specifically includes: Acquire the real-time audio and video data collected by the voice acquisition device, and / or The real-time audio and video data transmitted from the client terminal is obtained; Acquire the real-time text data collected by the text acquisition device, and / or The real-time text data transmitted from the client terminal is obtained; Retrieve the historical comment text data stored internally, and / or Obtain the historical comment text data from external storage, and / or The historical comment text data is obtained from the client terminal.

12. The food recommendation method based on knowledge graphs according to claim 1, characterized in that, The phrase "transing the real-time speech data into speech-text data" also includes: The configuration data stored in the external cache is obtained, and the real-time voice data is used to perform text transcription by executing a multi-channel, multi-size deep recurrent convolutional neural network model based on the configuration data.

13. An electrical appliance, characterized in that, include: Memory, used to store executable instructions; The processor, when executing executable instructions stored in the memory, implements the knowledge graph-based food recommendation method according to any one of claims 1 to 12.

14. A refrigerator, characterized in that, include: Memory, used to store executable instructions; The processor, when executing executable instructions stored in the memory, implements the knowledge graph-based food recommendation method according to any one of claims 1 to 12.

15. A computer-readable storage medium storing executable instructions, characterized in that, When the executable instructions are executed by the processor, they implement the knowledge graph-based food recommendation method according to any one of claims 1 to 12.

Images