Content generation methods, apparatus, electronic devices, and computer-readable storage media

Abstract

This disclosure provides a content generation method, apparatus, electronic device, and computer-readable storage medium. The method includes: acquiring a chart image feature vector of a chart image to be processed; acquiring a preset prompt instruction text vector; the prompt instruction is used to indicate the content of the chart image to be generated; inputting the chart image feature vector and the prompt instruction text vector into a preset chart image language feature extraction model, whereby the chart image language feature extraction model extracts language feature information from the chart image feature vector based on the prompt instruction information in the prompt instruction text vector, and obtains a chart image language feature vector based on the language feature information; and generating the content of the chart image based on the chart image language feature vector and the prompt instruction text vector. This embodiment improves data analysis efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of chart and image processing technology, and in particular to a content generation method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] In the financial sector, a vast amount of content is generated daily, including financial news from the securities market, stock market reports, industry and company research reports, listed company financial reports and information announcements, and product promotions from funds, trusts, and insurance companies. This content often includes charts, images, and text, and exists in rich text image documents. Many financial institutions collect publicly available financial image documents and extract information for investment analysis and decision support. Charts, in particular, contain rich information and can intuitively display decision-making information, such as data correlation, outlier identification, and trend analysis. Users can analyze data and generate insights based on their understanding of charts to assist in subsequent decision-making. Summary of the Invention

[0003] This disclosure provides a content generation method, apparatus, electronic device, and computer-readable storage medium that can improve data analysis efficiency.

[0004] In a first aspect, embodiments of this disclosure provide a content generation method, the method comprising:

[0005] Obtain the feature vector of the chart image to be processed;

[0006] Obtain the preset prompt instruction text vector; the prompt instruction is used to indicate the content of the chart image to be generated;

[0007] The chart image feature vector and the prompt instruction text vector are input into a preset chart image language feature extraction model. The chart image language feature extraction model extracts language feature information from the chart image feature vector based on the prompt instruction information in the prompt instruction text vector, and obtains the chart image language feature vector based on the language feature information.

[0008] The content of the chart image is generated based on the language feature vector of the chart image and the prompt instruction text vector.

[0009] Secondly, this disclosure provides a content generation apparatus, comprising:

[0010] The first acquisition module is used to acquire the chart image feature vector of the chart image to be processed;

[0011] The second acquisition module is used to acquire the text vector of the preset prompt instruction; the prompt instruction is used to indicate the content of the chart image to be generated;

[0012] The input module is used to input the chart image feature vector and the prompt instruction text vector into a preset chart image language feature extraction model. The chart image language feature extraction model extracts language feature information from the chart image feature vector based on the prompt instruction information in the prompt instruction text vector, and obtains the chart image language feature vector based on the language feature information.

[0013] The generation module is used to generate the content of the chart image based on the language feature vector of the chart image and the prompt instruction text vector.

[0014] Thirdly, this disclosure provides an electronic device that may include:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the content generation method.

[0018] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the content generation method.

[0019] This embodiment of the present disclosure can automatically acquire the chart image feature vector of the chart image to be processed, obtain the information in the chart image, and automatically acquire the prompt instruction text vector of the preset prompt instruction, obtain the prompt instruction information, and extract language feature information from the chart image feature vector according to the prompt instruction information in the prompt instruction text vector, so as to obtain language feature information matching the prompt instruction, generate the chart image language feature vector according to the language feature information, ensuring the accuracy of the language feature information of the chart image represented by the chart image language feature vector, and finally generate the content of the chart image according to the chart image language feature vector and the prompt instruction text vector, realizing the automatic extraction of chart image content, improving data analysis efficiency, and ensuring data analysis accuracy.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0022] Figure 1 A flowchart of the content generation method provided in this embodiment of the disclosure;

[0023] Figure 2 This is a schematic diagram of a content generation method provided in an embodiment of the present disclosure;

[0024] Figure 3 This is a schematic diagram of the Transformer structure for graph image-language feature extraction provided in an embodiment of this disclosure;

[0025] Figure 4 A flowchart illustrating a method for obtaining a language feature extraction model from a chart image, as provided in this embodiment of the disclosure;

[0026] Figure 5 A schematic diagram illustrating a method for obtaining a language feature extraction model for charts and images provided in this embodiment of the present disclosure;

[0027] Figure 6 A schematic diagram of the training data format provided in the embodiments of this disclosure;

[0028] Figure 7 A flowchart illustrating another method for obtaining a chart image language feature extraction model provided in this embodiment of the disclosure;

[0029] Figure 8 A schematic diagram illustrating another method for obtaining a chart image language feature extraction model provided in this embodiment of the disclosure;

[0030] Figure 9 This is a block diagram of the content generation apparatus provided in the embodiments of this disclosure;

[0031] Figure 10 A block diagram of the electronic device provided in the embodiments of this disclosure. Detailed Implementation

[0032] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0034] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0035] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0036] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0037] With the development of large language models (LLMs) and multimodal pre-training techniques, large language models such as GPT-4 have demonstrated powerful capabilities in many fields and tasks, such as context understanding, code generation, language generation, and storytelling. The multimodal pre-training method BLIP-2, by adding a lightweight query transformer (Q-Fromer) between a pre-trained image encoder with frozen parameters and a pre-trained large language model with frozen parameters, bridges the modality gap between visual and language models, and has shown excellent performance in visual question answering, image paraphrasing, and image-text retrieval tasks.

[0038] For a wide range of chart and image data, including those in the financial field, professional data analysts often need to spend a lot of time analyzing the data and interpreting it based on their own experience. In order to potentially help data analysts improve their daily work efficiency, a method for automatically generating chart and image content and key points of analysis is proposed.

[0039] The content generation method of this disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device may include, but is not limited to, in-vehicle devices, user equipment (UE), mobile devices, etc. The user equipment may include, but is not limited to, home computers, smart home appliances, etc. The mobile device may include, but is not limited to, cellular phones, cordless phones, personal digital assistants (PDAs), handheld devices, portable computing devices, smart wearable devices, etc. The content generation method can be implemented by a processor calling computer-readable program instructions stored in memory, or by a server executing a preset program.

[0040] This disclosure provides a content generation method, such as... Figure 1 , Figure 2 As shown, the method may include steps S11-S14:

[0041] S11. Obtain the chart image feature vector of the chart image to be processed;

[0042] S12. Obtain the preset prompt instruction text vector; the prompt instruction is used to indicate the content of the chart image to be generated;

[0043] S13. Input the chart image feature vector and the prompt instruction text vector into the preset chart image language feature extraction model. The chart image language feature extraction model extracts the language feature information in the chart image feature vector based on the prompt instruction information in the prompt instruction text vector. Based on the language feature information, the chart image language feature vector is obtained.

[0044] S14. Generate the content of the chart image based on the chart image language feature vector and the prompt instruction text vector.

[0045] In this embodiment of the disclosure, the content of a chart image can be automatically generated based on a multimodal pre-trained visual language model. The content of the chart image may include, but is not limited to, descriptive content and / or key analytical points.

[0046] In this embodiment of the disclosure, the multimodal pre-trained visual language model may include, but is not limited to: an image encoder based on frozen parameters of a Transformer (i.e., the parameters remain unchanged during training and inference), a chart image language feature extraction model (or represented as: chart image language feature extraction Transformer, chart image-language feature extraction Transformer), and a generative pre-trained large-scale language model with frozen parameters.

[0047] In this embodiment of the disclosure, the image encoder may select a currently developed mature visual model, including but not limited to the visual Transformer model, such as the ViT (Vision Transformer) model.

[0048] In this embodiment of the disclosure, obtaining the chart image feature vector of the chart image to be processed may include:

[0049] Input the chart image into the preset visual model;

[0050] The visual model generates the feature vector I of the chart image.

[0051] In this embodiment of the disclosure, for example, the chart image to be processed can be input into the ViT model to obtain the chart image feature vector I.

[0052] In this embodiment of the disclosure, obtaining the text vector of the preset prompt instruction includes:

[0053] Get the text of the prompt instruction;

[0054] Generate a one-hot vector of the prompt instruction based on its text.

[0055] In this embodiment of the disclosure, the prompt instruction may be one or more input instructions or commands, such as "generate the content analysis points of this chart".

[0056] In this embodiment of the disclosure, the words in the sentence "Key points of content analysis for generating this chart" can be generated into a one-hot vector to obtain the prompt instruction text vector.

[0057] In this embodiment of the disclosure, a chart image language feature vector E (or chart image-language feature vector E) can be obtained based on the chart image feature vector I and the prompt instruction text vector T described above. The chart image language feature vector E contains the language content features corresponding to the chart image.

[0058] In this embodiment of the disclosure, the language feature vector E of the chart image can be obtained based on the above-described chart image-language feature extraction Transformer.

[0059] In this embodiment of the disclosure, after obtaining the chart image feature vector I and the prompt instruction text vector T obtained in the foregoing content, the chart image feature vector I and the prompt instruction text vector T can be input into the chart image language feature vector E to obtain the chart image language feature vector E.

[0060] In this embodiment of the disclosure, the chart image language feature extraction model includes: an image Transformer, a text Transformer, and a learnable parameter matrix;

[0061] The chart image feature vector and the prompt instruction text vector are input into a preset chart image language feature extraction model. The chart image language feature extraction model extracts language feature information from the chart image feature vector based on the prompt instruction information in the prompt instruction text vector, including:

[0062] Input the text vector of the prompt instruction into the Text Transformer, and the Text Transformer will extract the prompt instruction information;

[0063] The chart image feature vector and parameter matrix are input into an image Transformer. The image Transformer modifies the parameters of the parameter matrix according to the prompts. The modified parameter matrix is ​​then used to interact with the chart image feature vector. Based on the parameters of the parameter matrix, language feature information is extracted from the chart image feature vector.

[0064] In this embodiment of the disclosure, obtaining the language feature vector of the chart image based on language feature information may include:

[0065] Based on the language feature information, the parameters of the parameter matrix are modified again, and the modified parameter matrix is ​​output as the language feature vector of the chart image; the parameters in the output parameter matrix are used to indicate the language feature information.

[0066] In this embodiment of the disclosure, in order to clarify the characteristics of the chart image language feature extraction model, it will be referred to as the chart image-language feature extraction Transformer in the following text. The chart image-language feature extraction Transformer is a multimodal model, and the input data may include, but is not limited to, vectors of chart images and vectors of text.

[0067] In this embodiment of the disclosure, the graph image-language feature extraction Transformer 100 may include, but is not limited to: a preset learnable parameter matrix 101, an image Transformer 102, a text Transformer 103, and a loss calculation module 104.

[0068] In this embodiment of the disclosure, the preset learnable parameter matrix 101 can be composed of multiple (e.g., 64) randomly initialized learnable embedded vectors of dimension x (x is a positive integer, e.g., 768). This parameter matrix contains multiple parameters, which are obtained through random initialization before training.

[0069] In this embodiment of the disclosure, the image Transformer 102 has an image Transformer network structure, and the text Transformer 103 has a text Transformer network structure.

[0070] In the embodiments disclosed herein, such as Figure 3 As shown, the image Transformer network structure may include, but is not limited to: multiple first self-attention layers (Self-Attention1), at least one cross-attention layer (Cross-Attention), and at least one first feedforward neural network layer (FeedForward1).

[0071] In embodiments of this disclosure, for example, the image Transformer may include: 12 first self-attention layers, 1 cross-attention layer, and a feedforward neural network layer.

[0072] In this embodiment of the disclosure, the 12 first self-attention layers (Self-Attention1) can be: Self-Attention1-1, Self-Attention1-2, ..., Self-Attention1-12.

[0073] In this embodiment of the disclosure, multiple first self-attention layers are connected in sequence; that is, in two adjacent first self-attention layers, the output of the previous first self-attention layer is connected to the input of the next first self-attention layer, wherein the input of the first first self-attention layer is used to input a learnable chart image language feature extraction vector; the output of the last first self-attention layer is connected to the first input of the cross-attention layer.

[0074] The second input to the cross-attention layer is used to input the feature vector I of the chart image;

[0075] The output of the cross-attention layer is connected to the input of the first feedforward neural network layer;

[0076] The output of the first feedforward neural network layer serves as the output of the image Transformer.

[0077] In this embodiment of the disclosure, the text Transformer network structure may include, but is not limited to, multiple second self-attention layers (Self-Attention2) and at least one second feedforward neural network layer (FeedForward2).

[0078] In this embodiment of the disclosure, for example, the text Transformer network structure may include: 12 second self-attention layers (Self-Attention2) and a second feedforward neural network layer (FeedForward2).

[0079] In this embodiment of the disclosure, the 12 second self-attention layers (Self-Attention2) can be: Self-Attention2-1, Self-Attention2-2, ..., Self-Attention2-12.

[0080] In this embodiment of the disclosure, multiple second self-attention layers are connected in sequence; that is, in two adjacent second self-attention layers, the output of the previous second self-attention layer is connected to the input of the next second self-attention layer, wherein the input of the first second self-attention layer is used to input the prompt instruction text vector; the output of the last second self-attention layer is connected to the input of the second feedforward neural network layer.

[0081] The output of the second feedforward neural network layer serves as the output of the text Transformer.

[0082] In this embodiment of the disclosure, multiple first self-attention layers and multiple second self-attention layers may share parameters.

[0083] In this embodiment, the shared parameters of the 12 self-attention layers of the image Transformer and text Transformer (the shared parameters ensure that the prompt instruction information extracted from the prompt instruction text vector T is consistent with the language feature information extracted from the chart image feature vector I) can be initialized with the 12 layers of self-attention weights of BERT-base-chinese (a BERT (Bidirectional Encoder Representation from Transformers) pre-trained model for Chinese). The weights set are actually the training parameters required for the image Transformer network structure and the text Transformer network structure. The cross-attention layer of the image Transformer network structure is responsible for the interaction between the learnable parameter matrix and the chart image feature vector I. The weights of the cross-attention layer can be set using a random initialization method.

[0084] In this embodiment, the text Transformer 103 can be used to process the prompt instruction text vector T to obtain prompt instruction information. The image Transformer 102 can be used to process the learnable parameter matrix and the chart image feature vector I. Based on the parameters shared with the text Transformer, the parameters of the parameter matrix are modified according to the prompt instruction information. The modified parameter matrix is ​​then used to interact with the chart image feature vector I. Language feature information is extracted from the chart image feature vector I based on the parameters of the parameter matrix. The parameters of the parameter matrix are then modified again based on the language feature information. The language feature information is then represented using the modified parameters of the parameter matrix. Finally, the parameter matrix is ​​output as the chart image language feature vector E.

[0085] In this embodiment of the disclosure, the loss calculation module 104 can be used to calculate the model loss of the chart image-language feature extraction Transformer 100 based on the output of the image Transformer 102 (i.e., the chart image language feature vector E) and the output of the text Transformer network 103 (i.e., the prompt instruction information).

[0086] In this embodiment of the disclosure, before obtaining the chart image language feature vector E through the chart image-language feature extraction Transformer 100, it is necessary to train the learnable parameter matrix 101, the image Transformer 102, and the text Transformer 103 in advance to obtain the chart image-language feature extraction Transformer 100 with the function of extracting the chart image language feature vector E.

[0087] In the embodiments disclosed herein, such as Figure 4 , Figure 5 As shown, the method for obtaining the chart image language feature extraction model (i.e., chart image-language feature extraction Transformer100) may include steps S21-S25:

[0088] S21. Obtain training data; training data includes chart / image data and text data.

[0089] In this embodiment of the disclosure, the chart image data may include: multiple sample pairs and a sample identifier corresponding to each sample pair, each sample pair including a chart image sample and a chart image content sample paired with the chart image sample, the sample identifier being used to indicate the first information similarity between the chart image sample and the paired chart image content sample; the text data includes: the text of a preset prompt instruction sample.

[0090] In this embodiment of the disclosure, the model training of the graph image-language feature extraction Transformer100 can be divided into two stages, including a graph image-language representation learning stage and a graph image-language generative pre-training stage.

[0091] In this embodiment of the disclosure, the chart image-language representation learning stage uses a pair of chart image samples (chart image format) and chart image content samples (text format), as well as the text of a preset prompt instruction sample, as training data. The pair of chart image samples and chart image content samples can be, but is not limited to, a pair of chart images and image chart analysis key point samples. The parameter matrix, image Transformer network structure, and text Transformer network structure in the chart image-language feature extraction Transformer 100 are trained using this training data, so that the learnable parameter matrix can learn to extract visual representations related to the text content of the prompt instruction sample.

[0092] In this embodiment of the disclosure, the training data may include positive samples and negative samples. For example, a chart image and its corresponding analytical points can be used as positive samples, while analytical points that do not correspond to the chart image can be used as negative samples. The sample label in a positive sample can be 1, indicating that the chart image and the paired analytical points are similar or correspond to each other. The sample label in a negative sample can be 0, indicating that the chart image and the paired analytical points are dissimilar or do not correspond to each other. The training data format is as follows: Figure 6 As shown, Figure 6 It contains multiple charts (such as Chart 1, Chart 2, Chart 3, ..., Chart n, where n is a positive integer and there are multiple of each type of chart) and multiple chart analysis points corresponding to the multiple charts (such as Chart 1 analysis point 1, Chart 1 analysis point 2, Chart 3 analysis point 1, Chart 3 analysis point 2, ..., Chart 3 analysis point m, Chart n analysis point 1, Chart n analysis point 2, etc., where m is a positive integer), forming a chart and chart analysis point pair, and each chart and chart analysis point pair corresponds to a label value.

[0093] S22. Generate chart image feature sample vectors based on chart image data, and generate prompt instruction text sample vectors based on text data.

[0094] S23. Input the prompt instruction text sample vector into the preset text Transformer network structure, extract the prompt instruction sample information, and use it as the output result of the text Transformer network structure; input the parameter matrix and the chart image feature sample vector into the preset image Transformer network structure to obtain the output result of the image Transformer network structure.

[0095] In this embodiment of the disclosure, inputting the parameter matrix and the feature sample vector of the chart image into a preset image Transformer network structure to obtain the output result of the image Transformer network structure may include:

[0096] Modify the parameters of the parameter matrix according to the prompt instructions and sample information;

[0097] Extract language feature sample information from the chart image feature sample vector based on the parameter matrix after parameter modification;

[0098] Based on the language feature sample information, the parameters of the parameter matrix are modified again to obtain the parameter matrix after modification, which is used as the output of the image Transformer network structure.

[0099] S24. Calculate the loss value based on the output results of the image Transformer network structure and the output results of the text Transformer network structure.

[0100] In this embodiment of the disclosure, the loss value is calculated based on the output results of the image Transformer network structure and the output results of the text Transformer network structure, including:

[0101] Calculate the second information similarity between the output of the image Transformer network structure and the output of the text Transformer network structure;

[0102] The loss value is calculated by combining the second information similarity and the first information similarity indicated by the sample identifier corresponding to the sample pair corresponding to the chart image sample information extracted from the language feature sample information.

[0103] S25. Obtain the chart image language feature extraction model based on the loss value.

[0104] If the loss value meets the first preset requirement, save the parameters of the text Transformer network structure, the parameters of the image Transformer network structure, and the parameters of the parameter matrix to obtain the chart image language feature extraction model 100.

[0105] If the loss value does not meet the first preset requirement, adjust the parameters of the image Transformer network structure, the parameters of the text Transformer network structure, and the parameters of the parameter matrix, and return to step S21.

[0106] In this embodiment of the disclosure, during the training process, a set of chart image feature extraction vectors E (i.e., the parameter matrix after parameter modification) output by the image transformer and the vector of the [CLS]Token (which can be called a token, used to indicate the smallest unit in the text, such as a word, a character, a phrase, a letter, etc.) output by the text transformer can be used to calculate cosine similarity. Then, cross-entropy loss can be calculated with the first information similarity indicated by the sample label to obtain the loss value. If the loss value meets the first preset requirement, the training ends and a trained chart image-language feature extraction Transformer 100 is obtained. If the loss value does not meet the first preset requirement, the parameters in the image transformer, the text transformer, and the parameters in the learnable reference matrix can be further optimized.

[0107] In the embodiments disclosed herein, such as Figure 7 , Figure 8As shown, before saving the parameters of the text Transformer network structure, the parameters of the image Transformer network structure, and the parameters of the parameter matrix, the method for obtaining the chart image language feature extraction model may further include steps S31-S33:

[0108] S31. Input the chart image language feature vector output by the chart image language feature extraction model into the generative pre-trained language model with preset frozen parameters, and the generative pre-trained language model generates the content of chart image data based on the chart image language feature vector.

[0109] S32. Calculate the model loss of the generative pre-trained language model based on the content of the chart and image data;

[0110] S33. Obtain the chart image language feature extraction model based on the model loss.

[0111] In this embodiment of the disclosure, a language feature extraction model for chart images is obtained based on model loss, including:

[0112] If the model loss meets the second preset requirement, save the parameters of the text Transformer network structure, the parameters of the image Transformer network structure, and the parameters of the parameter matrix to obtain the chart image language feature extraction model.

[0113] If the model loss does not meet the second preset requirement, adjust the parameters of the image Transformer network structure, the parameters of the text Transformer network structure, and the parameters of the parameter matrix, and return to step S21.

[0114] In this embodiment of the disclosure, the content of steps S31-S33 is the scheme of the graph image-language generative pre-training stage of this embodiment of the disclosure. Through this scheme, the generative pre-trained language model based on frozen parameters (i.e. generative pre-trained large language model) can verify whether the graph image-language feature extraction Transformer100 has been trained.

[0115] In this embodiment of the disclosure, the chart image-language generative pre-training stage involves inputting the chart image-language feature vector E output by the chart image-language feature extraction Transformer 100 into a generative pre-training large-scale language model with frozen parameters. Causal language model loss optimization is performed on the output of the generative pre-training large-scale language model and the text annotated with the analysis points of the chart image. This achieves the training of the chart image-language feature extraction Transformer 100. Essentially, the training results of the chart image-language feature extraction Transformer 100 are verified through the generative pre-training language model with frozen parameters, thereby achieving the purpose of training the chart image-language feature extraction Transformer 100.

[0116] In this embodiment of the disclosure, the generative pre-trained large language model uses a causal language model to calculate the model loss. The causal language model employs a diagonal masking matrix, ensuring that each token can only see the token information preceding it, but not the tokens following it. The training objective of the generative pre-trained large language model is to predict the token at the next position based on the tokens preceding it, typically by calculating the probability of word combinations based on probability distributions. The causal language model predicts the token at the current moment based on all previous token information, which can be directly applied to the text generation task of the generative pre-trained large language model.

[0117] In this embodiment of the disclosure, the generative pre-trained large language model with frozen parameters is intended to prevent the generative pre-trained large language model from participating in training. The purpose is to enable the graph image language feature vector E output by the graph image-language feature extraction Transformer100 to be understood by the generative pre-trained large language model and generate text.

[0118] In this embodiment, the generative pre-trained large language model may include, but is not limited to, GPT3 (Generative Pre-trained Transformer 3), OPT (Open Pre-trained Transformer), etc. As a generative pre-trained large language model with frozen parameters, no fine-tuning of the generative pre-trained large language model is required. In the embodiment of this disclosure, only the image-language feature extraction Transformer 100 needs to be trained.

[0119] In this embodiment of the disclosure, generating the content of the chart image based on the chart image language feature vector E and the prompt instruction text vector T includes:

[0120] The image language feature vector E and the prompt instruction text vector T are concatenated to obtain the concatenated vector;

[0121] The concatenated vector is input into a generative pre-trained language model with preset frozen parameters to generate the content of a chart image; the content of the chart image may include, but is not limited to: descriptive content and / or key analytical points.

[0122] In this embodiment of the disclosure, an accurate chart image language feature vector E can be obtained through the aforementioned trained chart image-language feature extraction Transformer 100. The chart image language feature vector E and the prompt instruction text vector T can be concatenated. The chart image language feature vector E and the prompt instruction text vector T can be concatenated directly. For example, the chart image language feature vector E is 1×128 dimensions, the prompt instruction text vector T is 1×100 dimensions, and the concatenated vector is 1×228 dimensions.

[0123] In this embodiment of the disclosure, the concatenated vector can be directly input into a generative pre-trained language model to obtain the content of the chart image.

[0124] Many current modal models have the ability to generate text from images, and can be pre-trained using charts and chart text to enable them to generate chart text. However, these methods require retraining the chart text fusion model and the text generation module together, which requires more labeled data and has a higher training cost compared to the scheme in this disclosure. The scheme in this disclosure uses a frozen parameter image encoder and a frozen parameter generative pre-trained large-scale language model. It uses manually labeled chart image-description and analysis point text pairs (i.e., chart image sample and chart image content sample pairs) and a two-stage training task to train the chart image-language feature extraction Transformer100, aligning the representation of the chart image with the representation of the description and analysis point text, and enabling it to be understood by the generative pre-trained large-scale language model. Ultimately, it achieves the function of automatically generating the chart description content and analysis point content based on the chart image and prompts.

[0125] This disclosure provides a content generation apparatus 200, such as... Figure 9 As shown, it includes:

[0126] The first acquisition module 201 is used to acquire the chart image feature vector of the chart image to be processed;

[0127] The second acquisition module 202 is used to acquire the text vector of the preset prompt instructions; the prompt instructions are used to indicate the content of the chart image to be generated.

[0128] Input module 203 is used to input the chart image feature vector and the prompt instruction text vector into a preset chart image language feature extraction model. The chart image language feature extraction model extracts the language feature information in the chart image feature vector based on the prompt instruction information in the prompt instruction text vector, and obtains the chart image language feature vector based on the language feature information.

[0129] The generation module 204 is used to generate the content of the chart image based on the chart image language feature vector and the prompt instruction text vector.

[0130] This disclosure provides an electronic device 300, such as... Figure 10 As shown, the electronic device 300 includes:

[0131] At least one processor 301; and

[0132] Memory 302 is communicatively connected to at least one processor 301; wherein,

[0133] The memory 302 stores one or more computer programs that can be executed by at least one processor 301, such that the at least one processor 301 can perform the content generation method described above.

[0134] In this embodiment of the disclosure, the electronic device 300 may further include one or more I / O (input / output) interfaces 303 connected between the processor 301 and the memory 302.

[0135] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the above-described content generation method. The computer-readable storage medium may be volatile or non-volatile.

[0136] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the above-described content generation method.

[0137] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0138] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0139] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0140] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0141] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0142] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0143] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0144] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0145] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0146] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A content generation method characterized by, The method comprises: obtaining a chart image feature vector of a chart image to be processed; obtaining a prompt instruction text vector of a preset prompt instruction; the prompt instruction is used to indicate the content of the chart image to be generated; inputting the prompt instruction text vector into a text Transformer included in a chart image language feature extraction model, extracting the prompt instruction information by the text Transformer, wherein the chart image language feature extraction model further includes an image Transformer and a learnable parameter matrix; inputting the chart image feature vector and the parameter matrix into the image Transformer, modifying the parameters of the parameter matrix according to the prompt instruction information by the image Transformer, performing data interaction between the parameter matrix after modification of parameters and the chart image feature vector, extracting language feature information in the chart image feature vector according to the parameters of the parameter matrix, and obtaining a chart image language feature vector according to the language feature information; generating the content of the chart image according to the chart image language feature vector and the prompt instruction text vector.

2. The content generation method of claim 1, wherein, The chart image language feature vector obtained according to the language feature information comprises: modifying the parameters of the parameter matrix again according to the language feature information, and outputting the parameter matrix after modification of parameters again as the chart image language feature vector; wherein the parameters in the output parameter matrix are used to indicate the language feature information.

3. The content generation method of claim 1, wherein, The method for obtaining the chart image language feature extraction model comprises: obtaining training data; the training data comprises chart image data and text data; generating a chart image feature sample vector according to the chart image data, and generating a prompt instruction text sample vector according to the text data; inputting the prompt instruction text sample vector into a preset text Transformer network structure, extracting prompt instruction sample information as the output result of the text Transformer network structure, inputting a parameter matrix and the chart image feature sample vector into a preset image Transformer network structure, and obtaining the output result of the image Transformer network structure; calculating a loss value according to the output result of the image Transformer network structure and the output result of the text Transformer network structure; obtaining the chart image language feature extraction model according to the loss value.

4. The content generation method of claim 3, wherein, The chart image data comprises a plurality of sample pairs and a sample identifier corresponding to each sample pair, each sample pair comprises a chart image sample and a chart image content sample paired with the chart image sample, and the sample identifier is used to indicate the first information similarity between the chart image sample and the paired chart image content sample; the text data comprises a text of a preset prompt instruction sample.

5. The content generation method of claim 4, wherein, The parameter matrix and the chart image feature sample vector are input into a preset image Transformer network structure, and an output result of the image Transformer network structure is obtained, including: parameters of the parameter matrix are modified according to the prompt instruction sample information; language feature sample information in the chart image feature sample vector is extracted according to the modified parameter matrix; parameters of the parameter matrix are modified again according to the language feature sample information, and a parameter matrix modified again is obtained as the output result of the image Transformer network structure.

6. The content generation method of claim 5, wherein, The loss value is calculated according to the output result of the image Transformer network structure and the output result of the text Transformer network structure, including: a second information similarity of the output result of the image Transformer network structure and the output result of the text Transformer network structure is calculated; the loss value is calculated according to the second information similarity, the language feature sample information extracted, and the first information similarity corresponding to the sample pair corresponding to the sample identifier indicated by the chart image sample corresponding to the sample pair.

7. The content generation method of claim 3, wherein, The chart image language feature extraction model is obtained according to the loss value, including: in a case where the loss value meets a first preset requirement, parameters of the text Transformer network structure, parameters of the image Transformer network structure, and parameters of the parameter matrix are saved, and the chart image language feature extraction model is obtained; in a case where the loss value does not meet the first preset requirement, parameters of the image Transformer network structure, parameters of the text Transformer network structure, and parameters of the parameter matrix are adjusted, and the step of obtaining training data is returned.

8. The content generation method of claim 7, wherein, Before the parameters of the text Transformer network structure, the parameters of the image Transformer network structure, and the parameters of the parameter matrix are saved, the method for obtaining the chart image language feature extraction model further includes: the chart image language feature vector output by the chart image language feature extraction model is input into a preset generative pre-training language model with frozen parameters, and content of the chart image data is generated by the generative pre-training language model according to the chart image language feature vector; a model loss of the generative pre-training language model is calculated according to the content of the chart image data; the chart image language feature extraction model is obtained according to the model loss.

9. The content generation method of claim 8, wherein, The chart image language feature extraction model is obtained according to the model loss, including: in a case where the model loss meets a second preset requirement, parameters of the text Transformer network structure, parameters of the image Transformer network structure, and parameters of the parameter matrix are saved, and the chart image language feature extraction model is obtained; In a case where the model loss does not meet the second preset requirement, parameters of the image Transformer network structure, parameters of the text Transformer network structure, and parameters of the parameter matrix are adjusted, and the step of obtaining training data is returned.

10. The content generation method according to any one of claims 1 to 9, characterized by, The image Transformer network structure comprises a plurality of first self-attention layers, at least one cross-attention layer, and at least one first feedforward neural network layer. The plurality of first self-attention layers are sequentially connected; an input of a first first self-attention layer is used for inputting the parameter matrix; and an output of a last first self-attention layer is connected with a first input of the cross-attention layer. A second input of the cross-attention layer is used for inputting the chart image feature vector. An output of the cross-attention layer is connected with an input of the first feedforward neural network layer. An output of the first feedforward neural network layer serves as an output of the image Transformer network structure.

11. The content generation method of claim 10, wherein, The text Transformer network structure comprises a plurality of second self-attention layers and at least one second feedforward neural network layer. The plurality of second self-attention layers are sequentially connected; an input of a first second self-attention layer is used for inputting the prompt instruction text vector. An output of a last second self-attention layer is connected with an input of the second feedforward neural network layer. An output of the second feedforward neural network layer serves as an output of the text Transformer network structure. The plurality of first self-attention layers and the plurality of second self-attention layers share parameters.

12. The content generation method according to any one of claims 1 to 9, characterized by, The chart image feature vector of the chart image to be processed comprises: The chart image is input into a preset visual model. The chart image feature vector is generated by the visual model.

13. The content generation method according to any one of claims 1 to 9, characterized by, The content of the chart image is generated according to the chart image language feature vector and the prompt instruction text vector, comprising: The chart image language feature vector and the prompt instruction text vector are spliced to obtain a spliced vector; The spliced vector is input into a preset frozen parameter generative pre-training language model to generate the content of the chart image; the content of the chart image comprises description content and / or analysis point content.

14. A content generation apparatus characterized by comprising: Comprise: The first acquisition module is configured to acquire a chart image feature vector of a chart image to be processed. The second acquisition module is configured to acquire a prompt instruction text vector of a preset prompt instruction; the prompt instruction is used to indicate content of the chart image to be generated. The input module is configured to input the prompt instruction text vector into a text Transformer included in a chart image language feature extraction model, and extract the prompt instruction information by the text Transformer, wherein the chart image language feature extraction model further includes an image Transformer and a learnable parameter matrix, input the chart image feature vector and the parameter matrix into the image Transformer, modify the parameters of the parameter matrix according to the prompt instruction information by the image Transformer, perform data interaction between the modified parameter matrix and the chart image feature vector, extract language feature information in the chart image feature vector according to the parameters of the parameter matrix, and obtain a chart image language feature vector according to the language feature information. The generation module is configured to generate content of the chart image according to the chart image language feature vector and the prompt instruction text vector.

15. An electronic device, comprising: The content generation method comprises the following steps: obtaining a chart image feature vector of the chart image by inputting the chart image into a chart image feature extraction model, wherein the chart image feature extraction model includes an image Transformer and a learnable parameter matrix, and the chart image feature vector is extracted by the image Transformer according to the parameters of the parameter matrix; extracting language feature information in the chart image feature vector according to the parameters of the parameter matrix, and obtaining a chart image language feature vector according to the language feature information; generating content of the chart image according to the chart image language feature vector and the prompt instruction text vector. The content generation method comprises the following steps:

16. A computer readable storage medium having stored thereon a computer program, characterized in that, at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the content generation method according to any one of claims 1-13. The computer program, when executed by the processor, implements the content generation method according to any one of claims 1-13. The computer program, when executed by the processor, implements the content generation method according to any one of claims 1-13.

Images