Deep learning-based text representation methods, refrigerators and storage media

By extracting and fusing textual and structural semantic features from refrigerator user interactions using deep learning methods, the problem of insufficient text representation in existing technologies is solved, thereby improving the interaction accuracy and response speed of smart refrigerators.

CN116450773BActive Publication Date: 2026-11-17QINDAO HAIER REFRIGERATOR CO LTD +1
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202310290464.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-03-23
Publication Date
2026-11-17
Estimated Expiration
2043-03-23

AI Technical Summary

Technical Problem

Existing text representation methods are sparse and fail to extract deep structural and semantic information from multimodal data, resulting in low accuracy, slow response time, and poor user experience when smart refrigerators interact with users.

Method used

By employing a deep learning-based text representation method, we can extract text features and structural semantic features from the contextual information of the text data, fuse them, and then perform decoding calculations to generate more comprehensive text representation information.

Benefits of technology

It improves the accuracy of text representation and user experience, enhances the accuracy of food information push and intelligent question answering, and reduces memory space usage and computational complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116450773B_ABST
    Figure CN116450773B_ABST
Patent Text Reader

Abstract

This invention provides a deep learning-based text representation method and a refrigerator. The method includes: acquiring text data and extracting text features with contextual information; calculating structural semantic features corresponding to the text features based on the text features; fusing the text features and the structural semantic features to obtain fused features; decoding and calculating the fused features to obtain and output text representation information corresponding to the text data. The deep learning-based text representation method provided by this invention considers both the features of the text itself and its corresponding contextual information, enabling the output text representation information to balance the accuracy of content on entities with the relationships between entities. This enhances the text representation effect, improves the accuracy of text representation, and improves the user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a text representation method based on deep learning and a refrigerator. Background Technology

[0002] Text representation methods have a wide range of applications, especially in smart home appliances such as refrigerators. For example, smart refrigerators can achieve various human-computer interaction scenarios such as intelligent retrieval, intelligent recommendation, and intelligent question answering through multimodal data such as text, voice, and images. These are all closely related to text representation, making text representation a key part of the application of artificial intelligence technology.

[0003] Existing text representation methods include those based on traditional machine learning models, which represent text from a shallow level such as word, character, phrase, or sentence level, or those based on deep learning models, which represent text from the perspective of a single text feature.

[0004] The text representation methods described above suffer from two problems: firstly, the text representation is sparse; secondly, they fail to mine the underlying, deeper structural and semantic information of multimodal data. These problems easily lead to insufficient and incomplete text representations, resulting in low accuracy, slow response time, poor timeliness, and a poor user experience in areas such as food information push, intent recognition, and intelligent question answering during smart refrigerator interactions with users. Summary of the Invention

[0005] One of the objectives of this invention is to provide a text representation method based on deep learning to solve the technical problem that existing text representation methods correspond to each set of text data in isolation, resulting in a single expression of content and an inability to fully and completely reflect the original meaning of the text data.

[0006] One of the objectives of this invention is to provide a refrigerator.

[0007] To achieve one of the above-mentioned objectives, the present invention provides a deep learning-based text representation method, comprising: acquiring text data and extracting text features with contextual information; calculating structural semantic features corresponding to the text features based on the text features; fusing the text features and the structural semantic features to obtain fused features; decoding and calculating the fused features to obtain and output text representation information corresponding to the text data.

[0008] As a further improvement of one embodiment of the present invention, the text data includes voice text data, real-time text data, comment text data, and video text data.

[0009] As a further improvement of one embodiment of the present invention, the "acquiring text data" specifically includes: acquiring real-time data and extracting data features from the real-time data; wherein the real-time data includes at least one of real-time voice data and real-time video data; inputting the data features into a convolutional neural network model to obtain corresponding first text information; aligning the sequence lengths of the first text information and the data features using a connection-time classification algorithm, and performing a fully connected combination calculation on the first text information to obtain the text data.

[0010] As a further improvement of one embodiment of the present invention, the real-time data includes real-time speech data, and the method further includes: enhancing the speech features of the real-time speech data based on an attention mechanism model.

[0011] As a further improvement of one embodiment of the present invention, the step of "inputting the data features into a convolutional neural network model to obtain the corresponding first text information" specifically includes: inputting the data features into a multi-size multi-channel convolutional neural network model or a distillation-diffusion model to obtain text content that matches the data features.

[0012] As a further improvement of one embodiment of the present invention, before "extracting the data features of the real-time data", the method further includes: determining whether the real-time data includes real-time speech data; if so, inputting the real-time speech data into the multi-size multi-channel convolutional neural network model to obtain first text information corresponding to the real-time speech data; and / or, before "extracting the data features of the real-time data", the method further includes: determining whether the real-time data includes real-time video data; if so, inputting the real-time video data into the distillation diffusion model to obtain first text information corresponding to the real-time video data.

[0013] As a further improvement of one embodiment of the present invention, the "extraction of text features with contextual information" specifically includes: extracting triples from the text data to obtain a triple structure, wherein the triple extraction includes extracting two entities and the relationship between the two entities from the text data; inputting the triple structure into a context recognition model to obtain text features with contextual information.

[0014] As a further improvement of one embodiment of the present invention, the step of "calculating the structural semantic features corresponding to the text features based on the text features" specifically includes: extracting several structural features based on the text features, and calculating the structural semantic features based on the structural features; wherein, the several structural features include at least one of a first structural feature representing the grammatical structure of the text features, a second structural feature representing the semantic relationship of the text features, and a third structural feature representing the connection between the grammatical structure and the semantic relationship.

[0015] As a further improvement of one embodiment of the present invention, the step of "calculating the structural semantic features based on the structural features" specifically includes: inputting the text features into a graph attention mechanism model or a graph neural network attention mechanism model, calculating the weight information corresponding to the structural features, and fusing and calculating the structural semantic features based on the weight information.

[0016] As a further improvement of one embodiment of the present invention, the step of "fusing the text features and the structural semantic features to obtain the fused features" specifically includes: performing concatenation on the text features and the structural semantic features to obtain the fused features.

[0017] As a further improvement of one embodiment of the present invention, the step of "decoding and calculating the fusion features, obtaining and outputting the text representation information corresponding to the text data" specifically includes: using a convolutional knowledge base model to decode the fusion features to obtain the decoded features corresponding to the fusion features; sequentially inputting the decoded features into a fully connected layer and a Softmax function with an attention mechanism model to calculate the scores of the text features and the structural semantic features, and filtering out the text representation information corresponding to the text data based on the scores.

[0018] As a further improvement of one embodiment of the present invention, before the step of "combining the decoded features through a fully connected layer and inputting them into the Softmax function of the attention mechanism model", the method further includes: inputting the text data into a transformer model for encoding to obtain text features with key-value pairs; wherein the key-value pairs represent the mapping relationship between the location of the text data and the corresponding text content; fusing the text features with key-value pairs and the decoded features, and performing pooling operations on the fused features to update the decoded features.

[0019] As a further improvement of one embodiment of the present invention, the method further includes: converting the text representation information corresponding to the text data into speech for output, and / or converting the text representation information corresponding to the text data into speech for transmission to the client for output, and / or converting the text representation information corresponding to the text data into text for output, and / or converting the text representation information corresponding to the text data into text for transmission to the client.

[0020] To achieve one of the above-mentioned objectives, the present invention also provides a refrigerator, comprising: a memory for storing executable instructions; and a processor for implementing the steps of any of the deep learning-based text representation methods described above when running the executable instructions stored in the memory.

[0021] To achieve one of the above-mentioned objectives, the present invention also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the steps of any of the deep learning-based text representation methods described above.

[0022] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0023] This invention employs a deep learning-based text representation method. By simultaneously considering the features of the text itself and its corresponding contextual information, it analyzes text feature information more comprehensively. Furthermore, by fusing text features with structural semantic features and using the generated fused features as the basis for decoding calculations, the output text representation information can take into account both the accuracy of content on entities and the relationships between entities. This is beneficial for enhancing the text representation effect, improving the accuracy of text representation, and improving the user experience.

[0024] Furthermore, by employing a convolutional knowledge base model to decode the fused features, the global relationships and transitional characteristics between entities in the text data can be effectively captured, thereby extracting textual features and structural semantic features more fully. Moreover, by using a transformer model to obtain features with key-value pair relationships, memory usage is effectively reduced and computational efficiency is improved. Attached Figure Description

[0025] Figure 1 This is a schematic diagram illustrating the steps of a deep learning-based text representation method according to an embodiment of the present invention.

[0026] Figure 2 This is a schematic diagram illustrating the steps of acquiring real-time data and transcribing the real-time data into first text information in one embodiment of the present invention.

[0027] Figure 3 is a schematic diagram of the steps of judging real-time voice data and transcribing it into first text information in one embodiment of the present invention.

[0028] Figure 4 This is a schematic diagram illustrating the steps of determining real-time video data and transcribing it into first text information in one embodiment of the present invention.

[0029] Figure 5 This is a schematic diagram of a preferred embodiment of a deep learning-based text representation method according to one aspect of the present invention.

[0030] Figure 6 This is a schematic diagram of data transformation in a preferred embodiment of a deep learning-based text representation method according to one embodiment of the present invention. Detailed Implementation

[0031] The present invention will now be described in detail with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.

[0032] It should be noted that the term "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Furthermore, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0033] In form, natural language text is a specific sequence of symbols. The human brain understands the semantics within these symbols according to certain language rules. To enable computers to process these symbol sequences, the language text needs to be quantified or vectorized, that is, represented as text. Text representation is the starting point of natural language processing and a crucial basis for computers to accurately recognize natural language. Therefore, text representation has significant practical importance in the field of natural language processing.

[0034] Based on this, the present invention provides a text representation method based on deep learning, such as... Figure 1 As shown, the specific steps include the following:

[0035] Step S1: Obtain text data and extract text features with contextual information.

[0036] Step S2: Calculate the structural semantic features corresponding to the text features based on the text features.

[0037] Step S3: The text features and the structural semantic features are fused to obtain fused features.

[0038] Step S4: Decode and calculate the fusion features to obtain and output the text representation information corresponding to the text data.

[0039] Thus, by simultaneously considering the features of the text data itself and the contextual information corresponding to the text data, the text feature information can be analyzed more comprehensively. At the same time, by fusing the text feature information and structural semantic features, and using the generated fused features as the basis for decoding calculation, the output text representation information can take into account both the accuracy of the content on the entity and the semantic correlation between entities. This is conducive to enriching and improving the text representation content, improving the accuracy of the text representation, fully reflecting the original meaning of the text data, and enhancing the user experience.

[0040] For step S1, the text data may include voice text data, real-time text data, comment text data, and video text data.

[0041] In this way, by obtaining heterogeneous data from multiple sources, we can ensure the integrity of the data on the one hand, and enrich the text content of the text data from multiple perspectives and types on the other hand.

[0042] The voice text data can refer to text content transcribed from voice data, and similarly, the video text data can refer to text content transcribed from video data.

[0043] like Figure 2 As shown, the "acquiring text data" part in step S1 can specifically include:

[0044] Step S11: Obtain real-time data and extract the data features of the real-time data.

[0045] Step S12: Input the data features into the convolutional neural network model to obtain the corresponding first text information.

[0046] Step S13: Align the sequence lengths of the first text information and the data features using a connection-based temporal classification algorithm, and perform a fully connected combination calculation on the first text information to obtain the text data.

[0047] The real-time data includes real-time voice data, real-time video data, or both real-time voice data and real-time video data.

[0048] In this way, the data features of the real-time data are transcribed into corresponding text information using a convolutional neural network model, reducing ambiguity caused by identical speech containing different semantics. Furthermore, since the speed of speaking or recording video is usually faster than the speed of typing, speech-to-text or video-to-text conversion can improve work efficiency.

[0049] Regarding step S11, in one embodiment, the real-time voice data refers to the user's current interrogative or commanding statements to the smart electronic device or a terminal device connected to the electronic device. Specifically, taking a smart refrigerator as an example, the user can say an interrogative statement such as "Are there any vegetables in the refrigerator?" or a commanding statement such as "Please adjust the temperature of the refrigerator compartment to 2 degrees Celsius".

[0050] On the other hand, the real-time video data is essentially composed of a series of consecutive images. Specifically, it is captured using video capture devices, such as mobile phones, cameras, Bluetooth devices, etc., to obtain a continuous sequence of images.

[0051] In a preferred embodiment, both aspects mentioned above may be included simultaneously, meaning that the acquired real-time data includes both real-time voice data and real-time video data. Data processing and feature extraction can then be performed on these data separately.

[0052] As described in step S1 or S11, in one embodiment, real-time user voice can be collected through voice acquisition devices such as microphones or microphone arrays installed inside the smart refrigerator; real-time user video can also be collected through video acquisition devices such as cameras installed inside the refrigerator; similarly, real-time text data can be obtained through mini-programs, official accounts, or the Web. Furthermore, the text data may also include user preferences for ingredients, information about ingredients that the user is interested in, and some comments posted by the user, such as "I used to like Kung Pao Chicken." The comments cover user interests and hobbies, and may even include information related to the current real-time text data.

[0053] Furthermore, a client terminal connected via a wireless communication protocol can acquire transmitted real-time voice or video from the user. The client terminal is an electronic device with information sending capabilities, such as a mobile phone, tablet, smart speaker, smart bracelet, or Bluetooth headset. During use, when the user needs to interact with the smart refrigerator, they can directly speak to the refrigerator or record video. The client terminal collects the voice and transmits it to the smart refrigerator via wireless communication methods such as Wi-Fi or Bluetooth. In other embodiments of the present invention, one or more of the above data acquisition methods can be used, or the real-time data can be acquired through other channels based on existing technology; specific details will not be elaborated here.

[0054] Furthermore, step S11 may also include preprocessing the data. Specifically, the speech is segmented according to a specified length (time period or number of samples) to complete the framing and windowing processing of the speech. Further, pre-emphasis processing can be performed before framing processing to emphasize the high-frequency components of the speech, thereby eliminating the influence of lip radiation during phonation, compensating for the high-frequency components of the speech signal suppressed by the vocal system, and highlighting high-frequency formants. Moreover, after windowing processing, steps such as filtering audio noise points and enhancing human voice can be performed to enhance the real-time speech data, extract the feature parameters of the real-time speech, and make the real-time speech data meet the input requirements of the subsequent neural network model.

[0055] Furthermore, the real-time video data can be cropped by writing scripts or using third-party video cropping tools. Specifically, video data is loaded and video information is read, the video is decoded based on the video information, and multi-frame images corresponding to the real-time video data are obtained based on the width and height of a single frame image.

[0056] Furthermore, irrelevant and duplicate data in the text dataset are removed, and outliers and missing values ​​are processed to clean the text data, thereby making the text data meet the input requirements of the subsequent neural network model.

[0057] Furthermore, in one implementation, the audio-video separation script can be written using Python, or a third-party audio-video separation tool can be used to perform separation operations on data that simultaneously contains real-time video data and real-time audio data, thereby obtaining effective real-time voice data and real-time video data.

[0058] For step S12, "inputting the data features into the convolutional neural network model to obtain the corresponding first text information" may specifically include:

[0059] The data features are input into a multi-size, multi-channel convolutional neural network model or a distillation-diffusion model to obtain text content that matches the data features.

[0060] Thus, the convolutional neural network model can include various types, allowing for adjustments to the corresponding neural network model to suit different data features. This increases adaptability to data features and improves the algorithm's generalization and recognition capabilities. In one embodiment, it can also be configured to adaptively select a suitable neural network model for processing based on the type of data features, thereby improving the efficiency and accuracy of converting the data features into text.

[0061] It should be emphasized that, in order to improve the accuracy and adaptability of feature extraction for data of different data types or different structural types, this invention provides refined operations for the data types of the real-time data, so as to give full play to the advantages of the two neural network models.

[0062] Specifically, in the first embodiment, as shown in FIG3(a), before "extracting the data features of the real-time data" in step S11, the method may include step S11A; step S12 may further include step S12A corresponding to step S11A. Steps S11A and S12A are specifically as follows:

[0063] Step S11A: Determine whether the real-time data includes real-time voice data.

[0064] If so, proceed to step S12A, whereby the real-time speech data is input into the multi-size multi-channel convolutional neural network model to obtain the first text information corresponding to the real-time speech data.

[0065] Therefore, by using a multi-size, multi-channel convolutional neural network model to process real-time speech data, the channels and sizes in the neural network model can be adaptively selected according to the characteristics of the speech data itself, which can increase the adaptability of speech-to-text conversion and improve the accuracy of speech-to-text conversion.

[0066] In the second embodiment, as shown in FIG3(b), before "extracting the data features of the real-time data" in step S11, the method may further include step S11B; step S12 may further specifically include step S12B corresponding to step S11B. Steps S11B and S12B are specifically as follows:

[0067] Step S11B: Determine whether the real-time data includes real-time video data.

[0068] If so, proceed to step S12B, whereby the real-time video data is input into the distillation-diffusion model to obtain the first text information corresponding to the real-time video data.

[0069] Thus, by using the distillation-diffusion model to process real-time video data, it is possible to quickly sample video data by changing the weight information in the model, thereby reducing the time and cost of video-to-text conversion. In addition, the distillation-diffusion model can generate images that are visually comparable to the original model, allowing for a trade-off between sample diversity and quality, which is beneficial for improving the timeliness and accuracy of video data to text conversion.

[0070] In a preferred embodiment, the two embodiments described above can be combined. That is, when the text data includes both real-time audio data and real-time video data, different neural network models can be used to process them separately and convert them into corresponding text information. In practical applications, considering the complexity of sentences recognized from image text, such as varying sentence lengths, different pause positions or word structures, and correlations between image features, step S12B can specifically include:

[0071] The real-time video data is input into a 3D deep convolutional neural network to calculate image features; the image features are then input into the distillation-diffusion model for transcription to obtain the first text information.

[0072] Specifically, in one embodiment, video processing operations such as cropping and frame segmentation are performed on the real-time video data to obtain video images of local areas. These local video images are then cropped and segmented to obtain multiple consecutive local image frames. These multiple consecutive local image frames are input into a 3D convolutional neural network model. By adding information in the time dimension, more expressive features can be extracted. The 3D convolutional neural network model can resolve the correlation information between multiple images. It takes multiple consecutive frames as input and captures motion information in the input frames by adding a new dimension of information, thereby better obtaining their image features.

[0073] After acquiring the real-time data, the data features of the real-time data can be enhanced using an attention mechanism model. Preferably, the above operation can be performed on real-time speech data to improve the conversion speed and effect. Based on this, the present invention may further include the step of: enhancing the speech features of the real-time speech data based on the attention mechanism model. It is understood that this step can be set at any position after "acquiring real-time data" in step S11 to achieve the corresponding effect.

[0074] In a preferred embodiment, this scheme can also be combined with an embodiment including steps S11A and S12A. In this case, the step of enhancing speech features using the attention mechanism model can be specifically set after step S12A. Furthermore, the above steps are not limited to scenarios where real-time data includes real-time speech data; the present invention does not exclude the application of the attention mechanism model to other types of real-time data.

[0075] Thus, by introducing an attention mechanism, we can focus on the relevant features or weights of the real-time data or real-time speech data converted into the first text information, and ignore irrelevant features or weights, thereby further improving the speed and effect of speech-to-text conversion and reducing computation time and cost.

[0076] Specifically, in the multi-size, multi-channel convolutional neural network model, an attention mechanism is introduced, enabling the neural network model to not only learn the attention mechanism autonomously, focusing on the key or related features of the real-time speech data while ignoring other non-key or unrelated feature information, thus enhancing speech features; but also, the attention mechanism can in turn help the computer understand the neural network. The two complement each other, which is conducive to improving the efficiency and effect of speech-to-text conversion.

[0077] In other embodiments of the present invention, speech feature enhancement of the real-time speech data can also be achieved through other algorithm models, and the present invention does not impose specific limitations on this.

[0078] The multi-size, multi-channel convolutional neural network model described in step S12A is composed of a multi-layer deep convolutional network model. This deep convolutional neural network model typically consists of several convolutional layers and several fully connected layers, and may also include various non-linear operations, pooling operations, etc. This model first extracts the speech features from the real-time speech data, then calculates text features based on these speech features, converting the speech features into corresponding text information.

[0079] Thus, by calculating speech features instead of real-time speech data, the computational load is small, and it is easier to characterize local features. Secondly, the attention mechanism and pooling operation give the model better time-domain or frequency-domain invariance. Furthermore, the deeper nonlinear structure also enables the model to possess powerful text representation capabilities. The distillation-diffusion model described in step S12B is a type of diffusion model that, given a set of data with an unknown distribution, can be used to characterize the underlying data distribution.

[0080] Specifically, the diffusion model can be divided into a forward diffusion process and a backward diffusion process. First, the data distribution is systematically perturbed through the forward diffusion process. Specifically, Gaussian noise is added to the data according to a pre-designed noise schedule until the data distribution tends to the prior distribution, i.e., the standard Gaussian distribution. Then, the data distribution is recovered through a backward diffusion process, i.e., starting from a given prior distribution, a parameterized Gaussian transformation kernel is used to gradually learn and recover the original data distribution.

[0081] Furthermore, given a trained instructor model, i.e., a teacher model, the distillation-diffusion model may include two steps. In the first step, a continuous-time student model is introduced to match the combined output of the two instructor diffusion models to obtain the distillation model. In the second step, the distillation model trained in the first step is progressively transformed into a model with fewer steps. Of course, other distillation algorithms can also be used, and no specific restrictions are placed here.

[0082] In practical applications, due to differences in human speech speed, video shooting frame rate, or character spacing, it is difficult to align the real-time data and the first text information at the word level. In step S13, the present invention uses the connection time sequence classification algorithm to construct a mapping relationship between the real-time data and the first text information.

[0083] Specifically, the Connectionist Temporal Classification (CTC) algorithm is an end-to-end training algorithm that enables a network model to automatically learn the alignment of input and output sequence lengths. Preferably, the connectionist temporal classification algorithm can be used after using a convolutional network model. This algorithm does not require pre-label alignment and annotation of the input data; instead, it is trained based on an input sequence and an output sequence, outputting a mapping relationship.

[0084] In one embodiment of the present invention, the algorithm aligns the sequence lengths of the data features (input sequence) and the first text information (output sequence) to establish a mapping relationship between the real-time data and the first text information. Based on the mapping relationship, the most likely text content can be inferred, thereby improving the accuracy of real-time data to text conversion, shortening the conversion time, and reducing computational complexity.

[0085] To enrich the semantic features of the text, this invention considers the contextual information of the text data while acquiring it. Specifically, the "extracting text features with contextual information" part in step S1 may specifically include:

[0086] The text data is subjected to triple extraction to obtain a triple structure, wherein the triple extraction includes extracting two entities and the relationship between the two entities from the text data; the triple structure is input into a context recognition model to obtain text features with context information.

[0087] In this way, by capturing contextual information from more different ranges, the characteristics of feature information in different feature layers and the relationships between them can be taken into account, which greatly improves the multi-scale expressive ability of the network model, fully considers the impact of contextual information on feature extraction accuracy, reduces semantic differences, and enhances data correlation.

[0088] The context recognition model may include a Long Short-Term Memory (LSTM) network model, a Bi-directional Long Short-Term Memory (BiLSTM) network model, or a Recurrent Neural Network (RNN) model. This invention does not impose specific limitations on this model.

[0089] It should be emphasized that, in the process of obtaining the text features with contextual information, there is no specific restriction on the order in which the extraction of text features and the acquisition of contextual information are performed.

[0090] Specifically, on the one hand, the contextual information of the text data can be obtained first, and then the text features of the text data and the contextual features of the contextual information can be extracted separately, and finally fused to obtain the text features with contextual information. On the other hand, the text features of the text data can be extracted first, and then the contextual features can be extracted based on the text features, and finally fused to obtain the text features with contextual information.

[0091] For step S2, "calculating the structural semantic features corresponding to the text features based on the text features" may specifically include:

[0092] Based on the text features, several structural features are extracted, and the structural semantic features are calculated based on the structural features.

[0093] The structural features include at least one of the following: a first structural feature representing the grammatical structure of the text features, a second structural feature representing the semantic relationship of the text features, and a third structural feature representing the connection between the grammatical structure and the semantic relationship.

[0094] In this way, by acquiring the grammatical structure features, semantic structure features, and the relationship between the two hidden behind the text features, the grammatical structure and semantic relationship are intertwined and complement each other, the grammatical structure relationship and the specific semantic relationship are determined, so that the grammatical structure and semantic relationship are consistent, eliminating ambiguity and enriching the semantic feature information.

[0095] Furthermore, the part "calculating the structural semantic features based on the structural features" may specifically include:

[0096] The text features are input into a graph attention mechanism model or a graph neural network attention mechanism model to calculate the weight information corresponding to the structural features, and the structural semantic features are obtained by fusing and calculating based on the weight information.

[0097] Thus, by introducing an attention mechanism, the convolutional neural network can be guided to focus on more critical information while ignoring other non-critical information, thereby strengthening the weight of key feature information and making the calculated structural semantic features more consistent with the true meaning of the text features, thus enhancing the text features.

[0098] The graph attention mechanism model or graph neural network attention mechanism model can be used to help graph convolutional networks better aggregate neighboring nodes in the graph network. Specifically, the attention mechanism is used to perform a weighted summation of the features of neighboring nodes, and the weights of the features of neighboring nodes depend entirely on the node features and are independent of the model. This invention does not impose specific restrictions on the attention mechanism model used.

[0099] Step S3 can specifically include:

[0100] The text features and the structural semantic features are concatenated to obtain the fused features.

[0101] In this way, by splicing features, the most differentiated information from multiple original feature sets can be obtained during the splicing process. This makes it easier to make better use of the useful characteristics of different features, eliminate redundant information caused by the correlation between different feature sets, and make the subsequent text representation more accurate, semantically clearer and more complete.

[0102] The concatenation mentioned here is an early fusion method, which refers to directly connecting two features. In one embodiment, specifically, assuming the text feature is set as x with dimension p, and the structural semantic feature is set as y with dimension q, then after concatenating the two, the resulting fused feature is x+y with dimension p+q.

[0103] It should be emphasized that the present invention is not limited to the method of splicing to achieve feature fusion. Other fusion methods can also be used, such as late fusion or the add method in early fusion. The corresponding feature fusion method can be selected according to the actual application scenario. The present invention does not impose specific limitations on this.

[0104] like Figure 4 As shown, step S4 can specifically include:

[0105] Step S41: Decode the fused features using a convolutional knowledge base model to obtain the decoded features corresponding to the fused features.

[0106] Step S42: The decoded features are sequentially input into a fully connected layer and a Softmax function with an attention mechanism model to calculate the scores of the text features and the structural semantic features, and the text representation information corresponding to the text data is obtained based on the scores.

[0107] Thus, by uniformly processing the fused features through a convolutional knowledge base model, this method is simple and easy to implement in terms of logic, and the resulting decoded features are more accurate, with computational and prediction performance superior to that of traditional models.

[0108] For step S41, the Convolutional Knowledge Base (ConvKB) model is an embedding model used to construct a knowledge base. This model can capture global relationships and transitional characteristics between entities. In the ConvKB model, the triple structure is transformed into a vector form of triples. Multiple triple vectors are concatenated to form a matrix vector. Then, a convolution operation is performed on the matrix vector, and the vectors generated after the convolution operation are concatenated to produce the decoded features.

[0109] To further enhance text features and enrich text semantics, before the step S42, which involves "combining the decoded features through a fully connected layer and inputting them into the Softmax function of the attention mechanism model," the following steps may be further included:

[0110] The text data is input into a transformer model for encoding to obtain text features with key-value pairs; wherein the key-value pairs represent the mapping relationship between the location of the text data and the corresponding text content; the text features with key-value pairs and the decoded features are fused, and the fused features are pooled to update the decoded features.

[0111] Thus, by fusing the text features with key-value pairs and the decoded features to update the decoded features, the text representation effect is further enhanced, making the text representation more complete, comprehensive, and semantically clearer. Furthermore, text features with key-value pairs help save memory space and improve computational efficiency. Moreover, performing pooling operations on the fused features filters out redundant information, further strengthening the deep semantic information of the text data.

[0112] Specifically, by improving the existing transformer model, a key-value pair module is added to the model to encode the text data and construct a mapping relationship between the storage location of the text data in memory and the stored text content, which is beneficial for quickly obtaining text semantic information.

[0113] For the Softmax function in step S42, in other embodiments of the present invention, other activation functions may be selected according to the model structure, such as the ReLU function and the Sigmoid function. The present invention does not impose specific limitations on this.

[0114] To facilitate users in obtaining the text representation information, the method may further include: converting the text representation information corresponding to the text data into speech for output, and / or converting the text representation information corresponding to the text data into speech for transmission to the client for output, and / or converting the text representation information corresponding to the text data into text for output, and / or converting the text representation information corresponding to the text data into text for transmission to the client.

[0115] Thus, the multi-channel and multi-type information output methods, combined with the multi-channel and multi-modal data source acquisition method provided by this invention, enable users to directly interact with smart devices remotely, offering high convenience and greatly enhancing the user experience. In other embodiments of this invention, only one or more of the above-mentioned text representation information output methods may be used, or the text representation information may be output through other channels based on existing technology; this invention does not impose specific limitations in this regard.

[0116] The various embodiments, examples, or specific examples provided by this invention can be combined with each other to ultimately form multiple better embodiments.

[0117] For example, such as Figure 5 A block diagram illustrating the principle of a deep learning-based text representation method in a preferred embodiment is shown. Figure 6 This corresponds to the transformation process of relevant real-time data or text data involved in the deep learning-based text representation method when implementing this preferred embodiment. The following will combine... Figure 5 and Figure 6 The processing procedure of this preferred embodiment is summarized below.

[0118] First, multimodal data sources are acquired through multiple channels to obtain various types of data 101. Specifically, the various types of data 101 include real-time text data and comment text data, real-time audio data, and real-time video data. The various types of data 101 are then transcribed to generate corresponding text data 102.

[0119] The transcribed text data 102 is input into the context recognition model to calculate its context-informed text features, resulting in context-informed text features 103.

[0120] The text features 103 with contextual information are then processed by a graph attention mechanism model or a graph neural network attention mechanism model to calculate their semantic structure features, resulting in the semantic structure features 104.

[0121] Based on the text feature 102 with contextual information and the semantic structure feature 104, their fusion feature is calculated to obtain the fusion feature 105.

[0122] The fused feature 105 is decoded to obtain the decoded feature 106. At the same time, the text data 102 is encoded using a transformer model with a key-value pair function module to generate text features 107 with key-value pair relationships.

[0123] Based on the decoding feature 106 and the text feature 107 with key-value pair relationship, calculate their fusion feature and update the decoding feature. Then, input the fully connected layer and softmax function in sequence to calculate and output the text representation to obtain the text representation information 108.

[0124] The present invention also provides a refrigerator, comprising: a memory for storing executable instructions; and a processor for implementing the steps of any of the deep learning-based text representation methods described above when running the executable instructions stored in the memory.

[0125] To achieve one of the above-mentioned objectives, the present invention also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the steps of any of the deep learning-based text representation methods described above.

[0126] In summary, this invention acquires multi-source heterogeneous data through multiple channels, performs transcription operations on the heterogeneous data to convert it into text data, and extracts text features and semantic structural features with contextual information. Then, it fuses the text features and semantic structural features to obtain fused features, and uses a simple and easy-to-operate convolutional knowledge base model method to decode them, generating decoded features. This not only reduces computational costs and improves the computational efficiency of text representation, but also extracts text features and structural semantic features more fully.

[0127] Furthermore, an improved transformer model can be used to obtain text features with key-value pair relationships, and the decoded features can be updated accordingly to generate new decoded features, further enhancing the deep semantic information of the text. This method significantly improves the comprehensiveness, sufficiency, and accuracy of text representation by performing feature fusion operations, greatly enhancing the user experience. Moreover, by using an improved transformer model to obtain features with key-value pair relationships, memory usage is effectively reduced and computational efficiency is improved.

[0128] It should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0129] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent embodiments or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.

Claims

1. A text representation method based on deep learning, characterized in that, include: Acquire text data and extract text features with contextual information; Based on the text features, the structural semantic features corresponding to the text features are calculated; The text features and the structural semantic features are fused to obtain the fused features; The fusion features are decoded and calculated to obtain and output the text representation information corresponding to the text data. Specifically, the step of decoding and calculating the fused features to obtain and output the text representation information corresponding to the text data includes: The fused features are decoded using a convolutional knowledge base model to obtain the decoded features corresponding to the fused features; The text data is input into a transformer model for encoding to obtain text features with key-value pairs; wherein the key-value pairs represent the mapping relationship between the location of the text data and the corresponding text content. The text features with key-value pairs and the decoded features are fused, and the fused features are pooled to update the decoded features. The updated decoded features are sequentially input into a fully connected layer and a Softmax function with an attention mechanism model to calculate the scores of the text features and the structural semantic features. Based on the scores, the text representation information corresponding to the text data is obtained.

2. The deep learning-based text representation method according to claim 1, characterized in that, The text data includes voice text data, real-time text data, comment text data, and video text data.

3. The deep learning-based text representation method according to claim 1, characterized in that, The "acquiring text data" specifically includes: Acquire real-time data and extract data features from the real-time data; wherein the real-time data includes at least one of real-time voice data and real-time video data; The data features are input into a convolutional neural network model to obtain the corresponding first text information; The sequence lengths of the first text information and the data features are aligned using a connection-based temporal classification algorithm, and a fully connected combination calculation is performed on the first text information to obtain the text data.

4. The deep learning-based text representation method according to claim 3, characterized in that, The real-time data includes real-time speech data, and the method further includes: enhancing the speech features of the real-time speech data based on an attention mechanism model.

5. The deep learning-based text representation method according to claim 3, characterized in that, The phrase "inputting the data features into a convolutional neural network model to obtain the corresponding first text information" specifically includes: The data features are input into a multi-size, multi-channel convolutional neural network model or a distillation-diffusion model to obtain text content that matches the data features.

6. The deep learning-based text representation method according to claim 5, characterized in that, Before "extracting data features from the real-time data", the method further includes: Determine whether the real-time data includes real-time voice data; If so, the real-time voice data is input into the multi-size multi-channel convolutional neural network model to obtain the first text information corresponding to the real-time voice data; And / or, before "extracting the data features of the real-time data", the method further includes: Determine whether the real-time data includes real-time video data; If so, the real-time video data is input into the distillation-diffusion model to obtain the first text information corresponding to the real-time video data.

7. The deep learning-based text representation method according to claim 1, characterized in that, The "extraction of text features with contextual information" specifically includes: The text data is subjected to triple extraction to obtain a triple structure, wherein the triple extraction includes extracting two entities from the text data and the relationship between the two entities; The triplet structure is input into the context recognition model to obtain text features with context information.

8. The deep learning-based text representation method according to claim 1, characterized in that, The phrase "calculating the structural semantic features corresponding to the text features based on the text features" specifically includes: Based on the text features, several structural features are extracted, and the structural semantic features are calculated based on the structural features. The structural features include at least one of the following: a first structural feature representing the grammatical structure of the text features, a second structural feature representing the semantic relationship of the text features, and a third structural feature representing the connection between the grammatical structure and the semantic relationship.

9. The deep learning-based text representation method according to claim 8, characterized in that, The phrase "calculating the structural semantic features based on the structural features" specifically includes: The text features are input into a graph attention mechanism model or a graph neural network attention mechanism model to calculate the weight information corresponding to the structural features, and the structural semantic features are obtained by fusing and calculating based on the weight information.

10. The deep learning-based text representation method according to claim 1, characterized in that, The phrase "fusing the text features and the structural semantic features to obtain fused features" specifically includes: The text features and the structural semantic features are concatenated to obtain the fused features.

11. The deep learning-based text representation method according to claim 1, characterized in that, The method further includes: Convert the text representation information corresponding to the text data into speech for output, and / or The text representation information corresponding to the text data is converted into text and output.

12. The deep learning-based text representation method according to claim 11, characterized in that, The method further includes: The text representation information corresponding to the text data is converted into speech and transmitted to the client for output, and / or The text representation information corresponding to the text data is converted into text and transmitted to the client for output.

13. A refrigerator, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the steps of the deep learning-based text representation method according to any one of claims 1-12.

14. A computer-readable storage medium, characterized in that, It stores executable instructions, characterized in that, when the executable instructions are executed by a processor, they implement the deep learning-based text representation method according to any one of claims 1-12.

Citation Information

Patent Citations

  • Automatic classification method of text syntax structure and semantic information fused text entity relationship

    CN111177383A

  • Knowledge representation learning method fusing multi-source information

    CN111538848A

  • Speech recognition and classification method, device and equipment, refrigerator and storage medium

    CN115062143A