Book review generation method, system, device and medium based on multimodal data

Abstract

The present application relates to a method, system, device and medium for generating book reviews based on multimodal data. The method forms a vector matrix by using multiple groups of historical text vectors, with each group of historical text vectors as row and column matrix elements, and forms a relationship matrix by using the interaction relationship between every two groups of historical text vectors. Then, a graph structure consisting of the vector matrix and the relationship matrix is ​​constructed, and multiple current text vectors corresponding to the multiple groups of historical text vectors are extracted from the graph structure. The method can fully explore the time series relationship within each group of historical text vectors and the interaction relationship between the multiple groups of historical text vectors, and can better reflect the changes in the evaluation data of e-books in different periods, avoiding the problem of missing logical relationships caused by changes in evaluation data in different periods due to keyword-based search. The method can effectively predict current evaluation data and improve the real-time and accuracy of book review generation.

Description

Technical Field

[0001] The present application relates to the technical field of electronic book review processing, and in particular to a method, system, device and medium for generating book reviews based on multimodal data. Background Art

[0002] An e-book is a publication that digitizes multimedia content such as text, images, audio, and video. An e-book review is an article that analyzes and comments on the content, style, and value of an e-book. It typically includes detailed descriptions and evaluations of the e-book's theme, plot, characters, language, and structure. It aims to help readers understand the e-book's content and characteristics, as well as the author's perspectives and style, thereby guiding readers in selecting appropriate reading materials or providing references for academic research.

[0003] After reading an e-book, users generate reviews in different ways on different platforms. For example, they upload a review in the form of text and images on an e-book reading platform, or upload a review in the form of a video on a short video platform. To expand the influence of e-book articles, authors and e-book promoters often generate book reviews to introduce and promote the e-book articles. There are two main ways to generate book reviews:

[0004] The first method is to use manual writing, but manual writing is more subjective;

[0005] The second approach is to search for e-book reviews on various platforms and generate book reviews based on these reviews. This approach is more objective and efficient than the first approach. However, this approach is largely based on keyword searches, searching for reviews from all time periods after the e-book article was created. Generating book reviews based on these reviews often overlooks the fact that e-book reviews can change over time due to multiple factors, including the audience, social and cultural background, artistic value, and the e-book environment. This approach, based solely on keyword searches, lacks the ability to analyze the dynamic evolution and trends of review data, and therefore cannot provide highly real-time and accurate book reviews. Summary of the Invention

[0006] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0007] The main purpose of the embodiments of the present disclosure is to provide a method, system, device and medium for generating book reviews based on multimodal data, which can obtain book reviews with high real-time performance and high accuracy.

[0008] A first aspect of an embodiment of the present application provides a method for generating book reviews based on multimodal data, the method comprising:

[0009] searching a first set of target e-books based on the keyword; the first set includes multiple sets of historical evaluation data for the target e-books, wherein any two sets of historical evaluation data belong to historical evaluation data of different modalities, and each set of historical evaluation data includes evaluation data collected at multiple historical moments;

[0010] Converting the multiple groups of evaluation data in the first set into multiple groups of historical evaluation texts, extracting multiple groups of historical text vectors from the multiple groups of historical evaluation texts, forming the multiple groups of historical text vectors into a second set, and sampling the multiple groups of historical text vectors in the second set according to a time window, wherein each group of the historical text vectors after sampling includes multiple historical text vectors;

[0011] Generate a graph structure consisting of a vector matrix and a relationship matrix based on the multiple groups of historical text vectors in the second set; wherein each matrix element in the vector matrix is ​​composed of a historical text vector in the second set, and each row matrix element or each column matrix element is composed of a group of historical text vectors, and each matrix element in the relationship matrix represents an interactive relationship between any two row matrix elements or any two column matrix elements in the vector matrix;

[0012] Predicting a plurality of current text vectors corresponding to the plurality of groups of historical text vectors in the second set according to the graph structure, and converting the plurality of current text vectors into a plurality of current evaluation texts;

[0013] A book review of the target electronic book is generated according to the multiple current evaluation texts.

[0014] The present disclosure provides a method for generating book reviews based on multimodal data, which has at least the following beneficial effects:

[0015] This method can more accurately reflect the logical relationship between evaluations in different periods by comprehensively utilizing the multimodal historical evaluation data of the target e-book, thereby generating more real-time and accurate book reviews. Specifically, a vector matrix is ​​formed by using multiple groups of historical text vectors as row and column matrix elements for each group of historical text vectors, and a relationship matrix is ​​formed by using the interaction relationship between each two groups of historical text vectors. Then, a graph structure consisting of a vector matrix and a relationship matrix is ​​constructed, and multiple current text vectors corresponding to multiple groups of historical text vectors are extracted from the graph structure. This embodiment can fully explore the relationship within each group of historical text vectors in the time series and the interaction relationship between multiple groups of historical text vectors, and can better reflect the changes in the evaluation data of e-books in different periods, avoiding the problem of missing logical relationships caused by changes in evaluation data in different periods based on keyword search alone, and can effectively predict current evaluation data, improving the real-time and accuracy of book review generation.

[0016] In some embodiments, the keyword is a sentence having subject information in the target electronic book.

[0017] In some embodiments, the subject information includes at least one of a title, an abstract, and an introduction.

[0018] In some embodiments, extracting multiple groups of historical text vectors from the multiple groups of historical evaluation texts includes:

[0019] Based on Doc2Vec, the corresponding historical text vector is extracted from each group of historical evaluation texts.

[0020] In some embodiments, the different modalities include at least two of a text modality, an image modality, and an audio modality.

[0021] In some embodiments, predicting multiple current text vectors corresponding to the multiple groups of historical text vectors based on the graph structure includes:

[0022] Constructing a first prediction model and a second prediction model, wherein the first prediction model includes any graph neural network and the second prediction model includes any recurrent neural network;

[0023] extracting, from the graph structure according to the first prediction model, graph embedding features obtained by embedding information in the relationship matrix into information in the vector matrix;

[0024] A current text vector corresponding to each group of the historical text vectors is predicted from the graph embedding features according to the second prediction model.

[0025] In some embodiments, the training process of the first prediction model and the second prediction model includes the following steps:

[0026] Build an evaluation network and construct multiple sets of historical text vectors to be trained;

[0027] Generate a graph structure to be trained consisting of a vector matrix to be trained and a relationship matrix to be trained according to the historical text vectors to be trained; the vector matrix to be trained is generated by the multiple groups of historical text vectors to be trained, and the elements in the relationship matrix to be trained are generated by the interaction relationship between every two matrix elements in the vector matrix to be trained;

[0028] Inputting the graph structure to be trained into the first prediction model to obtain embedding features of the graph to be trained output by the first prediction model;

[0029] Inputting the image embedding features to be trained into the second prediction model, obtaining a plurality of current text vectors to be trained corresponding to the plurality of groups of historical text vectors to be trained predicted by the second prediction model;

[0030] Constructing positive samples and negative samples; wherein the negative samples are obtained by concatenating the historical text vector to be trained and the current text vector to be trained, and the positive samples are obtained by concatenating the historical text vector to be trained and a preset correct label;

[0031] Inputting the positive sample and the negative sample into an evaluation model to determine the probability between the positive sample and the negative sample according to the evaluation model;

[0032] According to the probability and a preset evaluation function, back propagation is performed on the first prediction model and the second prediction model until the training process of the first prediction model and the second prediction model is completed.

[0033] A second aspect of the embodiments of the present application provides a book review generation system based on multimodal data, the system comprising:

[0034] a review extraction unit configured to search a first set of target e-books based on a keyword; the first set comprising multiple sets of historical review data for the target e-books, wherein any two sets of historical review data are of different modalities, and each set of historical review data comprises review data collected at multiple historical moments;

[0035] a text vector extraction unit, configured to convert the plurality of groups of evaluation data in the first set into a plurality of groups of historical evaluation texts, extract a plurality of groups of historical text vectors from the plurality of groups of historical evaluation texts, form the plurality of groups of historical text vectors into a second set, and sample the plurality of groups of historical text vectors in the second set according to a time window, wherein each group of the historical text vectors after sampling includes a plurality of historical text vectors;

[0036] a matrix generating unit, configured to generate a graph structure consisting of a vector matrix and a relationship matrix based on the plurality of groups of historical text vectors in the second set; wherein each matrix element in the vector matrix is ​​composed of a historical text vector in the second set, and each row matrix element or each column matrix element is composed of a group of historical text vectors; and each matrix element in the relationship matrix represents an interactive relationship between any two row matrix elements or any two column matrix elements in the vector matrix;

[0037] a text vector extraction unit, configured to predict, based on the graph structure, a plurality of current text vectors corresponding to the plurality of groups of historical text vectors in the second set, and convert the plurality of current text vectors into a plurality of current evaluation texts;

[0038] The book review generating unit is configured to generate a book review of the target electronic book based on the multiple current evaluation texts.

[0039] A third aspect of an embodiment of the present application proposes an electronic device, at least one controller and a memory for communicating with the at least one controller; the memory stores instructions that can be executed by the at least one controller, and the instructions are executed by the controller to enable the controller to perform the book review generation method based on multimodal data as described in the first aspect.

[0040] A fourth aspect of the embodiments of the present application provides a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-mentioned method for generating book reviews based on multimodal data.

[0041] It can be understood that the beneficial effects of the second to fourth aspects compared with the relevant technologies are the same as the beneficial effects of the first aspect compared with the relevant technologies. Please refer to the relevant description in the first aspect and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 is a flowchart of a method for generating book reviews based on multimodal data provided by this application;

[0044] Figure 2 is a schematic diagram of the graph structure proposed in the embodiment of the present application;

[0045] Figure 3 This is a schematic diagram of the training process of the overall system proposed in the embodiment of the present application;

[0046] Figure 4 Schematic diagram of the structure of a book review generation system based on multimodal data proposed in an embodiment of the present application;

[0047] Figure 5 It is a structural diagram of an electronic device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0049] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0051] An e-book is a publication that digitizes multimedia content such as text, images, audio, and video. An e-book review is an article that analyzes and comments on the content, style, and value of an e-book. It typically includes detailed descriptions and evaluations of the e-book's theme, plot, characters, language, and structure. It aims to help readers understand the e-book's content and characteristics, as well as the author's perspectives and style, thereby guiding readers in selecting appropriate reading materials or providing references for academic research.

[0052] After reading an e-book, users generate reviews in different ways on different platforms. For example, they upload a review in the form of text and images on an e-book reading platform, or upload a review in the form of a video on a short video platform. To expand the influence of e-book articles, authors and e-book promoters often generate book reviews to introduce and promote the e-book articles. There are two main ways to generate book reviews:

[0053] The first method is to use manual writing, but manual writing is more subjective;

[0054] The second approach is to search for e-book reviews on various platforms and generate book reviews based on these reviews. This approach is more objective and efficient than the first approach. However, this approach is largely based on keyword searches, searching for reviews from all time periods after the e-book article was created. Generating book reviews based on these reviews often overlooks the fact that e-book reviews can change over time due to multiple factors, including the audience, social and cultural background, artistic value, and the e-book environment. This approach, based solely on keyword searches, lacks the ability to analyze the dynamic evolution and trends of review data, and therefore cannot provide highly real-time and accurate book reviews.

[0055] This application relates to a method for generating book reviews based on multimodal data. This method aims to address the real-time and accuracy issues of existing book review generation techniques. Existing techniques primarily rely on manual writing or keyword-based review generation, which can be highly subjective and ignore the logical relationships between reviews from different time periods. Therefore, a new method is proposed to generate more accurate and real-time book reviews.

[0056] The method of the present application includes the following steps S100 to S500:

[0057] Step S100, based on the keyword, search for a first set of target e-books; the first set includes multiple groups of historical evaluation data of the target e-books, wherein any two groups of historical evaluation data belong to historical evaluation data of different modalities, and each group of historical evaluation data includes evaluation data collected at multiple historical moments.

[0058] Step S200: convert the multiple groups of evaluation data in the first set into multiple groups of historical evaluation texts, extract multiple groups of historical text vectors from the multiple groups of historical evaluation texts, form the multiple groups of historical text vectors into a second set, and sample the multiple groups of historical text vectors in the second set according to the time window, so that each group of historical text vectors contains multiple historical text vectors.

[0059] Step S300: generating a graph structure consisting of a vector matrix and a relationship matrix based on the multiple groups of historical text vectors in the second set; wherein each matrix element in the vector matrix is ​​composed of a historical text vector in the second set, and each row matrix element or each column matrix element is composed of a group of historical text vectors, and each matrix element in the relationship matrix represents an interactive relationship between any two row matrix elements or any two column matrix elements in the vector matrix;

[0060] Step S400, predicting multiple current text vectors corresponding to multiple groups of historical text vectors in the second set based on the graph structure, and converting the multiple current text vectors into multiple current evaluation texts;

[0061] Step S500: Generate a book review for the target electronic book based on the multiple current evaluation texts.

[0062] With respect to step S110, the keyword may be a word or sentence containing subject information in the target e-book, wherein the subject information includes at least one of a title, an abstract, and an introduction.

[0063] Acquiring subject information can be accomplished through a variety of methods. For example, natural language processing (NLP) can be used to extract information such as titles, abstracts, and descriptions from e-book metadata. Machine learning algorithms can also be used to automatically identify and extract this subject information. These methods ensure that the acquired subject information is accurate and representative, providing a reliable foundation for subsequent book review generation.

[0064] The role of keywords is to find a collection of reviews of the target e-book from the entire Internet.

[0065] Keywords can be the main title or a combination of the titles of each chapter.

[0066] Historical review data in different modalities can include at least two of the following: text, image, and audio. This means that review data isn't limited to text; it can also include information in various formats, such as images and audio. For example, text and image reviews uploaded by users on e-book reading platforms (such as QQ Reader), or video reviews uploaded on short video platforms, can all be considered part of the review data. By combining data from multiple modalities, a more comprehensive picture of the target e-book's evaluation information over time can be obtained, thereby improving the accuracy and objectivity of the reviews.

[0067] These evaluation data can come from the same user or different users, there is no limitation here. For example:

[0068] (1) User A uploaded evaluation data of the target e-book at multiple time points on Platform 1;

[0069] (2) User A uploaded evaluation data of the target e-book on Platform 1 and Platform 2 at multiple time points;

[0070] (3) User A and User B have uploaded evaluation data of the target e-book on Platform 1;

[0071] (4) User A and User B have uploaded evaluation data of the target e-book on both Platform 1 and Platform 2.

[0072] The evaluation data here are all historical evaluation data. Since e-book evaluations change over time, the main idea of ​​this application is to use these historical evaluation data to predict current evaluation data, so that the current evaluation can reflect the rules between historical evaluations, and use the current evaluation data to generate target e-book reviews, so that the target e-book reviews can reflect the changes in evaluation data of e-books in different periods, avoiding the problem of missing logical relationships caused by keyword search alone.

[0073] The following describes the generation of the first set and the second set:

[0074] Taking m platforms (e.g., QQ Reading, Douyin, and Xiaohongshu) as an example, the historical evaluation data of the m platforms are combined into an initial data set (i.e., the first set).

[0075] For step S200, the first set is then processed:

[0076] Convert non-text evaluation data into textual evaluation, for example, use image recognition technology to identify text information from image evaluation data, or use audio conversion technology to convert audio evaluation data into text data.

[0077] Then, Word2Vec, Doc2Vec and other technologies are used to vectorize the text to obtain the historical text vector: .

[0078] in, These are the historical text vector sets converted from the data collected by the first platform, the second platform, and the mth platform, and so on.

[0079] Then define the time t, use a window of length a to To take a sample: , these data constitute the second set.

[0080] in:

[0081] 、 ,to .

[0082] This embodiment needs to predict the current text vector at time t+1, that is, to obtain: .

[0083] Regarding step S300, the following describes how to generate the graph structure:

[0084] A third prediction model is set, wherein the third prediction model is mainly used to generate a relationship matrix, and generate a graph structure based on the vector matrix and the relationship matrix, wherein the nodes of the graph structure are represented by the vector matrix and the edges are represented by the relationship matrix. Figure 2 shown.

[0085] The vector matrix (each row consists of a set of historical text vectors) is:

[0086] .

[0087] The solution is a current text vector corresponding to each set of historical text vectors, which is: .

[0088] The relationship matrix will be introduced in the subsequent embodiments;

[0089] Regarding step S400, the steps include the following process:

[0090] (1) Constructing a first prediction model and a second prediction model, wherein the first prediction model includes any graph neural network and the second prediction model includes any recurrent neural network;

[0091] (2) extracting graph embedding features from the graph structure by embedding information in the relationship matrix into information in the vector matrix according to the first prediction model;

[0092] (3) Predict a current text vector corresponding to each group of historical text vectors from the graph embedding features based on the second prediction model.

[0093] After obtaining a graph structure in the previous step, this embodiment uses a first prediction model, which can be any graph neural network. For example, a graph convolutional neural network (GCN) uses a graph convolution operation to extract a graph embedding feature having first interactive information in a vector matrix and second interactive information in a relationship matrix from a graph structure. This graph embedding feature contains the internal relationship of each group of historical text vectors in the time series, as well as the interactive relationship between multiple groups of historical text vectors. It can fully explore the logical relationship between vectors in the vector matrix, determine the law of vector change over time, and thus obtain the latest evaluation data.

[0094] After obtaining the graph embedding features, this embodiment uses a second prediction model to predict the current text vector at the current moment. The second prediction model can be any recurrent neural network, such as a gated recurrent network (GRU). The gated recurrent network (GRU) predicts a current text vector corresponding to each set of historical text vectors from the graph embedding features.

[0095] Specifically, the combination of the first prediction model and the second prediction model enables the effective extraction of graph embedding features from the graph structure, comprising the first interaction information in the vector matrix and the second interaction information in the relationship matrix. Using a recurrent neural network, the current text vector corresponding to each set of historical text vectors is predicted from the graph embedding features. Graph neural networks are able to capture complex relationships between nodes when processing graph structures, while recurrent neural networks excel at processing time series data and can extract useful information from graph embedding features.

[0096] It should be noted that since graph neural networks and recurrent neural networks are commonly used tools in this field, their tools can be obtained on the Internet and will not be described in detail here.

[0097] Before introducing the training process of the first and second prediction models, we will introduce how the third prediction model generates the relationship matrix:

[0098] The relationship matrix can reflect the interaction between every two rows of vector elements or every two columns of vector elements in the vector matrix. Therefore, the generation of the relationship matrix needs to be combined with the training process of the first prediction model and the second prediction model.

[0099] The third prediction model consists of a fully connected layer and a convolutional model. First, the process of generating the initial relationship matrix by the third prediction model includes:

[0100] (1) Use a fully connected layer to map Gaussian noise into a feature representation;

[0101] (2) Use the convolutional model to generate feature representation as dimensional relationship matrix.

[0102] The dimensions are the same as the m groups of historical evaluation data mentioned above.

[0103] Then, the training process of the first prediction model and the second prediction model is as follows:

[0104] (1) Build an evaluation model and construct multiple sets of historical text vectors to be trained;

[0105] (2) Generate a graph structure to be trained consisting of a vector matrix to be trained and a relationship matrix to be trained based on the historical text vectors to be trained; the vector matrix to be trained is generated by multiple groups of historical text vectors to be trained, and the elements in the relationship matrix to be trained are generated by the interaction between every two matrix elements in the vector matrix to be trained;

[0106] (3) Inputting the graph structure to be trained into the first prediction model to obtain the embedding features of the graph to be trained output by the first prediction model;

[0107] (4) Inputting the embedding features of the graph to be trained into the second prediction model, and obtaining a plurality of current text vectors to be trained corresponding to the plurality of groups of historical text vectors to be trained predicted by the second prediction model;

[0108] (5) Construct positive samples and negative samples; the negative samples are obtained by concatenating the historical text vector to be trained and the current text vector to be trained, and the positive samples are obtained by concatenating the historical text vector to be trained and the preset correct label; the correct label is set in advance;

[0109] (6) Inputting positive samples and negative samples into the evaluation model to determine the probability between positive samples and negative samples according to the evaluation model;

[0110] (7) According to the probability and the preset evaluation function, the second prediction model and the first prediction model are back-propagated until the training process of the second prediction model and the first prediction model is completed.

[0111] The forward propagation process includes:

[0112] First, the third prediction model generates the initial graph structure (consisting of a vector matrix and an initial relationship matrix). The graph generated by the first prediction model is then embedded with features, and the second prediction model generates the current text vector. Finally, the evaluation model generates gradient parameters based on the output probabilities and an evaluation function. The evaluation model can be based on a depth-first minimum-maximum evaluation function, which ensures that the output of the third prediction model approximates positive samples.

[0113] The backpropagation process includes:

[0114] The first prediction model, the second prediction model, and the third prediction model are updated according to the gradient parameter, so that the third prediction model can generate a more accurate relationship matrix.

[0115] In some embodiments, since negative samples are concatenated from the historical text vector to be trained and the current text vector to be trained, and positive samples are concatenated from the historical text vector to be trained and the preset correct label, positive and negative samples also form a time series. Therefore, the evaluation model can be composed of a long short-term memory network (LSTM) and a fully connected layer. The input data of the LSTM network is positive and negative samples, and the output features can serve as the input data of the fully connected layer, which finally determines the probability.

[0116] After the training is completed, the first prediction model, the second prediction model, and the third prediction model can be used to generate the current evaluation text. It should be noted that the evaluation model does not participate in the generation process of the current vector when it is actually used.

[0117] In step S500, when multiple current text vectors are generated, the multiple current text vectors capture the complex interactions between multiple groups of historical text vectors, as well as the temporal relationships within the historical text vectors. The multiple current review texts generated from the multiple current text vectors can reflect the historical patterns of the target e-book's reviews.

[0118] Finally, based on multiple current evaluation texts, book reviews can be generated with high real-time and high accuracy.

[0119] The beneficial effects of this application include:

[0120] This method can more accurately reflect the logical relationship between evaluations in different periods by comprehensively utilizing the multimodal historical evaluation data of the target e-book, thereby generating more real-time and accurate book reviews. Specifically, a vector matrix is ​​formed by using multiple groups of historical text vectors as row and column matrix elements for each group of historical text vectors, and a relationship matrix is ​​formed by using the interaction relationship between each two groups of historical text vectors. Then, a graph structure consisting of a vector matrix and a relationship matrix is ​​constructed, and multiple current text vectors corresponding to multiple groups of historical text vectors are extracted from the graph structure. This embodiment can fully explore the relationship within each group of historical text vectors in the time series and the interaction relationship between multiple groups of historical text vectors, and can better reflect the changes in the evaluation data of e-books in different periods, avoiding the problem of missing logical relationships caused by changes in evaluation data in different periods based on keyword search alone, and can effectively predict current evaluation data, improving the real-time and accuracy of book review generation.

[0121] In some embodiments of the present application, Doc2Vec technology is used to extract corresponding historical text vectors from each group of historical evaluation texts. Doc2Vec is an unsupervised learning algorithm that captures the semantic information of a document by mapping the entire document into a vector space of fixed length. In the specific implementation process, it is first necessary to preprocess multiple groups of historical evaluation texts, including removing stop words, segmenting words, and other operations. Then, the Doc2Vec model is trained using the preprocessed text data, and a vector representation of each group of historical evaluation texts is generated by the model. In this way, multiple groups of historical evaluation texts can be converted into multiple groups of historical text vectors.

[0122] The application of Doc2Vec technology effectively converts text data into vector representations, facilitating subsequent computation and processing. Secondly, the text vectors generated by Doc2Vec effectively preserve the semantic information of the text, helping to improve the accuracy and quality of book review generation. Finally, Doc2Vec's unsupervised learning capabilities enable it to process large amounts of historical review text data, making it highly applicable and scalable.

[0123] Step S500 generates a book review of the target e-book based on the multiple current review texts, including the following steps:

[0124] Method 1:

[0125] First, the similarity between multiple current evaluation texts and keywords is determined.

[0126] Then, based on the similarity threshold, several current evaluation texts having a similarity greater than the similarity threshold are selected from the multiple current evaluation texts.

[0127] Finally, a book review of the target e-book is generated based on several current evaluation texts. For example, after the current evaluation texts are combined, a book review of the target e-book can be obtained.

[0128] Method 2: Generate book reviews using evolutionary algorithms.

[0129] Unlike summaries, book reviews are usually used as recommendations, so they often contain the interest characteristics required by the author or recommender to achieve personalization requirements.

[0130] First, define interest information. Interest information needs to include the above keywords as well as words or phrases customized by the author or promoter based on their own interests.

[0131] Then, from the multiple current evaluation texts, the multiple current evaluation texts with more interesting information are determined, for example, by calculating the proportion of the interesting information or calculating the total amount.

[0132] Then, we use evolutionary algorithms to generate book reviews. The process of evolutionary algorithms is as follows:

[0133] (1) Set the maximum number of evolutionary generations to T and the initial population Y(0) to have N individuals, where each individual is a queue for generating book reviews. Set a collection containing multiple current review texts. Each iteration generates a new population.

[0134] (2) Individual evaluation: Calculate the fitness of each individual in the population P(i). The fitness value here can set the amount of interesting information.

[0135] (3) Selection, crossover, and mutation: Apply the mutation operator to the population.

[0136] (4) After the selection, crossover, and mutation iterations of the population P(i), the next generation population P(i+1) is obtained, and the iterative process is repeated.

[0137] (5) Termination condition judgment: If t = T, the individual with the maximum fitness obtained in the evolution process is output as the optimal solution.

[0138] By setting interest information through method 2, while retaining keywords, words or short sentences set by the author or promoter based on their own interests are added. This can select several current evaluation texts with more interest features, and then use evolutionary algorithms to generate better book reviews, thereby improving the accuracy and adaptability of book review generation.

[0139] like Figure 4 One embodiment of the present application provides a book review generation system based on multimodal data, the system comprising:

[0140] The evaluation extraction unit 1100 is configured to search a first set of target e-books based on a keyword; the first set includes multiple sets of historical evaluation data for the target e-books, wherein any two sets of historical evaluation data belong to different modalities, and each set of historical evaluation data includes evaluation data collected at multiple historical moments;

[0141] The text vector extraction unit 1200 is configured to convert the multiple groups of evaluation data in the first set into multiple groups of historical evaluation texts, extract multiple groups of historical text vectors from the multiple groups of historical evaluation texts, form the multiple groups of historical text vectors into a second set, and sample the multiple groups of historical text vectors in the second set according to a time window so that each group of historical text vectors contains multiple historical text vectors;

[0142] The matrix generation unit 1300 is configured to generate a graph structure consisting of a vector matrix and a relationship matrix based on the multiple groups of historical text vectors in the second set; wherein each matrix element in the vector matrix is ​​composed of a historical text vector in the second set, and each row matrix element or each column matrix element is composed of a group of historical text vectors, and each matrix element in the relationship matrix represents an interactive relationship between any two row matrix elements or any two column matrix elements in the vector matrix;

[0143] The text vector extraction unit 1400 is used to predict multiple current text vectors corresponding to multiple groups of historical text vectors in the second set according to the graph structure, and convert the multiple current text vectors into multiple current evaluation texts;

[0144] The book review generating unit 1500 is used to generate a book review of a target electronic book based on a plurality of current evaluation texts.

[0145] It should be noted that the book review generation system based on multimodal data provided in this embodiment and the above-mentioned book review generation method based on multimodal data are based on the same inventive concept. Therefore, the relevant content of the above-mentioned book review generation method based on multimodal data is also applicable to the content of the book review generation system based on multimodal data. Therefore, it will not be repeated here.

[0146] like Figure 5 , an embodiment of the present application further provides an electronic device, the electronic device comprising:

[0147] at least one memory;

[0148] at least one processor;

[0149] at least one program;

[0150] The programs are stored in the memory, and the processor executes at least one program to implement the above-mentioned method for generating book reviews based on multimodal data in the present disclosure.

[0151] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0152] The electronic device according to the embodiment of the present application is described in detail below.

[0153] The processor 1600 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention.

[0154] Memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). Memory 1700 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification via software or firmware, the relevant program code is stored in memory 1700 and invoked by processor 1600 to execute the multimodal data-based book review generation method of the embodiments of the present invention.

[0155] Input / output interface 1800, used for information input and output;

[0156] Communication interface 1900, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0157] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0158] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0159] An embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium and stores computer-executable instructions. The computer-executable instructions are used to enable a computer to execute the above-mentioned book review generation method based on multimodal data.

[0160] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. Furthermore, memory can include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device.

[0161] In some embodiments, the memory may include a memory remotely located relative to the processor, and the remote memory may be connected to the processor via a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0162] The embodiments described in the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are equally applicable to similar technical problems.

[0163] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0164] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0165] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0166] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0167] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0168] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0169] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0170] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0171] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling an electronic device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0172] The above is a specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above-mentioned implementation methods. Technical personnel familiar with the art can also make various equivalent modifications or substitutions without violating the spirit of the embodiments of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the embodiments of the present application.

Claims

1. A book review generation method based on multimodal data, characterized in that: The method comprises: searching a first set of target e-books based on the keyword; the first set includes multiple sets of historical evaluation data for the target e-books, wherein any two sets of historical evaluation data belong to historical evaluation data of different modalities, and each set of historical evaluation data includes evaluation data collected at multiple historical moments; Converting the multiple groups of evaluation data in the first set into multiple groups of historical evaluation texts, extracting multiple groups of historical text vectors from the multiple groups of historical evaluation texts, forming the multiple groups of historical text vectors into a second set, and sampling the multiple groups of historical text vectors in the second set according to a time window, wherein each group of the historical text vectors after sampling includes multiple historical text vectors; Generate a graph structure consisting of a vector matrix and a relationship matrix based on the multiple groups of historical text vectors in the second set; wherein each matrix element in the vector matrix is ​​composed of a historical text vector in the second set, and each row matrix element or each column matrix element is composed of a group of historical text vectors, and each matrix element in the relationship matrix represents an interactive relationship between any two row matrix elements or any two column matrix elements in the vector matrix; Predicting multiple current text vectors corresponding to the multiple groups of historical text vectors in the second set according to the graph structure, and converting the multiple current text vectors into multiple current evaluation texts; predicting multiple current text vectors corresponding to the multiple groups of historical text vectors according to the graph structure includes: Constructing a first prediction model and a second prediction model, wherein the first prediction model includes any type of graph neural network and the second prediction model includes any type of recurrent neural network; extracting, from the graph structure according to the first prediction model, graph embedding features obtained by embedding information in the relationship matrix into information in the vector matrix; and predicting, from the graph embedding features, a current text vector corresponding to each group of the historical text vectors according to the second prediction model; A book review of the target electronic book is generated according to the multiple current evaluation texts.

2. The method for generating book reviews based on multimodal data according to claim 1, wherein: The training process of the first prediction model and the second prediction model includes the following steps: Build an evaluation model and construct multiple sets of historical text vectors to be trained; Generate a graph structure to be trained consisting of a vector matrix to be trained and a relationship matrix to be trained according to the historical text vector to be trained; The vector matrix to be trained is generated by the multiple groups of historical text vectors to be trained, and the elements in the relationship matrix to be trained are generated by the interaction relationship between every two matrix elements in the vector matrix to be trained; Inputting the graph structure to be trained into the first prediction model to obtain embedding features of the graph to be trained output by the first prediction model; Inputting the image embedding features to be trained into the second prediction model, obtaining a plurality of current text vectors to be trained corresponding to the plurality of groups of historical text vectors to be trained predicted by the second prediction model; Constructing positive samples and negative samples; wherein the negative samples are obtained by concatenating the historical text vector to be trained and the current text vector to be trained, and the positive samples are obtained by concatenating the historical text vector to be trained and a preset correct label; Inputting the positive sample and the negative sample into an evaluation model to determine the probability between the positive sample and the negative sample according to the evaluation model; According to the probability and a preset evaluation function, back propagation is performed on the first prediction model and the second prediction model until the training process of the first prediction model and the second prediction model is completed.

3. The method for generating book reviews based on multimodal data according to claim 1, characterized in that: The keyword is a sentence having subject information in the target electronic book.

4. The method for generating book reviews based on multimodal data according to claim 3, wherein: The subject information includes at least one of a title, an abstract, and an introduction.

5. The method for generating book reviews based on multimodal data according to claim 1, wherein: The step of extracting multiple groups of historical text vectors from the multiple groups of historical evaluation texts includes: Based on Doc2Vec, the corresponding historical text vector is extracted from each group of historical evaluation texts.

6. The method for generating book reviews based on multimodal data according to claim 1, wherein: The different modalities include at least two of a text modality, an image modality, and an audio modality.

7. A book review generation system based on multimodal data, characterized in that: The system comprises: a review extraction unit configured to search a first set of target e-books based on a keyword; the first set comprising multiple sets of historical review data for the target e-books, wherein any two sets of historical review data are of different modalities, and each set of historical review data comprises review data collected at multiple historical moments; a text vector extraction unit, configured to convert the plurality of groups of evaluation data in the first set into a plurality of groups of historical evaluation texts, extract a plurality of groups of historical text vectors from the plurality of groups of historical evaluation texts, form the plurality of groups of historical text vectors into a second set, and sample the plurality of groups of historical text vectors in the second set according to a time window, wherein each group of the historical text vectors after sampling includes a plurality of historical text vectors; a matrix generating unit, configured to generate a graph structure consisting of a vector matrix and a relationship matrix based on the plurality of groups of historical text vectors in the second set; wherein each matrix element in the vector matrix is ​​composed of a historical text vector in the second set, and each row matrix element or each column matrix element is composed of a group of historical text vectors; and each matrix element in the relationship matrix represents an interactive relationship between any two row matrix elements or any two column matrix elements in the vector matrix; A text vector extraction unit is configured to predict, based on the graph structure, a plurality of current text vectors corresponding to the plurality of groups of historical text vectors in the second set, and convert the plurality of current text vectors into a plurality of current evaluation texts; the predicting, based on the graph structure, a plurality of current text vectors corresponding to the plurality of groups of historical text vectors comprises: Constructing a first prediction model and a second prediction model, wherein the first prediction model includes any type of graph neural network and the second prediction model includes any type of recurrent neural network; extracting, from the graph structure according to the first prediction model, graph embedding features obtained by embedding information in the relationship matrix into information in the vector matrix; and predicting, from the graph embedding features, a current text vector corresponding to each group of the historical text vectors according to the second prediction model; The book review generating unit is configured to generate a book review of the target electronic book based on the multiple current evaluation texts.

8. An electronic device, characterized in that: include: at least one controller and a memory for communicatively coupling with the at least one controller; The memory stores instructions that can be executed by the at least one controller, and the instructions are executed by the controller to enable the controller to perform the method for generating book reviews based on multimodal data according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the book review generation method based on multimodal data according to any one of claims 1 to 6.

Images