Chart structuring method and device based on large language model and model training method and device

Abstract

The invention provides a chart structuring method and device based on a large language model and a model training method and device, relates to the technical field of artificial intelligence such as natural language processing, large language models and deep learning, and can be applied to the fields such as finance, education and research. According to the specific implementation scheme, the method comprises the steps of obtaining to-be-processed charts and text cues; processing the chart and the text cue words by using the large language model to obtain a predicted Token (lexical element) vector sequence output by the large language model; the large language model is used for predicting a matched Token vector sequence based on charts and text cues; decoding the predicted Token vector sequence to obtain text description information and specific characters used for converting the chart into structured data; and converting the text description information into structured data based on the specific characters. According to the embodiment of the invention, the data can be automatically extracted from the chart, the data processing speed and accuracy are improved, and further data processing and analysis are facilitated.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the field of artificial intelligence technology such as natural language processing, large language models, and deep learning, and specifically to a graph structuring method, a model training method, and a device based on a large language model, which can be applied to finance, education, research, and other fields. Background Art

[0002] Data visualization is the key to understanding and analyzing market trends. For example, data visualization can be achieved through charts. However, traditional charts such as pie charts, bar charts, and line charts, although intuitive, lack structured data formats. Currently, there is a lack of effective means for structuring charts. Summary of the invention

[0003] The present disclosure provides a graph structuring method, a model training method and a device based on a large language model.

[0004] According to a first aspect of the present disclosure, a graph structuring method based on a large language model is provided, comprising:

[0005] Get the charts and text prompts to be processed;

[0006] The chart and the text prompt word are processed by a large language model to obtain a predicted Token vector sequence output by the large language model; the large language model is used to predict a matching Token vector sequence based on the chart and the text prompt word;

[0007] Decoding the predicted Token vector sequence to obtain text description information and specific characters used to convert the chart into structured data;

[0008] Based on the specific characters, the text description information is converted into structured data.

[0009] According to a second aspect of the present disclosure, a method for training a large language model is provided, comprising:

[0010] Acquire training data and label data; the training data includes a chart to be processed and text prompt words, and the label data includes a specific character Token vector used to convert the chart to be processed into structured data;

[0011] Using a large language model to process the chart to be processed and the text prompt word to obtain a predicted Token vector sequence output by the large language model;

[0012] Determine a first loss value based on the specific character Token vector and the predicted Token vector sequence;

[0013] The large language model is trained according to the first loss value.

[0014] According to a third aspect of the present disclosure, a graph structuring device based on a large language model is provided, comprising:

[0015] The first acquisition module is used to acquire the chart and text prompt words to be processed;

[0016] A second acquisition module is used to process the chart and the text prompt word using a large language model to obtain a predicted Token vector sequence output by the large language model; the large language model is used to predict a matching Token vector sequence based on the chart and the text prompt word;

[0017] A decoding module, used to decode the predicted Token vector sequence to obtain text description information and specific characters used to convert the chart into structured data;

[0018] The conversion module is used to convert the text description information into structured data based on the specific characters.

[0019] According to a fourth aspect of the present disclosure, a large language model training device is provided, comprising:

[0020] A first acquisition module is used to acquire training data and label data; the training data includes a chart to be processed and a text prompt word, and the label data includes a specific character Token vector used to convert the chart to be processed into structured data;

[0021] A second acquisition module is used to process the to-be-processed chart and the text prompt word using a large language model to obtain a predicted Token vector sequence output by the large language model;

[0022] A determination module, configured to determine a first loss value based on the specific character Token vector and the predicted Token vector sequence;

[0023] A training module is used to train the large language model according to the first loss value.

[0024] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0025] at least one processor; and

[0026] a memory communicatively connected to the at least one processor; wherein,

[0027] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the first and second aspects.

[0028] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the methods described in the first and second aspects above.

[0029] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the steps of the method described in the first and second aspects when executed by a processor.

[0030] According to the technical solution of the present invention, data can be automatically extracted from charts, which improves the speed and accuracy of data processing, not only improves the readability and accessibility of data, but also enables the data to be more widely used in automated reporting, market analysis and decision support systems.

[0031] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0033] Figure 1 It is a flowchart of a diagram structuring method based on a large language model provided by an embodiment of the present disclosure;

[0034] Figure 2 It is a flowchart of a diagram structuring method based on a large language model provided by an embodiment of the present disclosure;

[0035] Figure 3 is a schematic diagram of a system architecture for implementing a diagram structuring method provided by an embodiment of the present disclosure;

[0036] Figure 4 is a flowchart of a method for training a large language model provided by an embodiment of the present disclosure;

[0037] Figure 5 is a block diagram of a graph structuring device based on a large language model provided by an embodiment of the present disclosure;

[0038] Figure 6 is a block diagram of a graph structuring device based on a large language model provided by an embodiment of the present disclosure;

[0039] Figure 7 is a block diagram of a graph structuring device based on a large language model provided by an embodiment of the present disclosure;

[0040] Figure 8 is a block diagram of a large language model training device provided by an embodiment of the present disclosure;

[0041] Fig. 9 is a block diagram of a large language model training device provided by an embodiment of the present disclosure;

[0042] Fig.10 A block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0044] The embodiments of the present disclosure relate to artificial intelligence technology fields such as natural language processing, large language models, and deep learning.

[0045] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0046] Natural Language Processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between humans and computers using natural language. It is a discipline that uses computer technology to analyze, understand and process natural language, that is, it uses computers as a powerful tool for language research, conducts quantitative research on language information with the support of computers, and provides language descriptions that can be used by humans and computers.

[0047] Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important path to artificial intelligence.

[0048] Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is of great help in interpreting data such as text, images, and sounds. The ultimate goal of deep learning is to enable machines to have analytical learning capabilities like humans and to be able to recognize data such as text, images, and sounds.

[0049] An agent is an agent that can perceive the environment and take actions to achieve specific goals. It can be software, hardware or a system with autonomy, adaptability and interaction. The agent perceives changes in the environment (such as through sensors or data input), makes judgments and decisions based on the knowledge and algorithms it has learned, and then performs actions to affect the environment or achieve predetermined goals.

[0050] It should be noted that in the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0051] It should also be noted that in the embodiments of the present disclosure, certain software, components, models and other existing solutions in the industry may be mentioned, which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

[0052] Data visualization is the key to understanding and analyzing market trends. For example, data visualization can be achieved through charts. However, traditional charts such as pie charts, bar charts, and line charts, although intuitive, lack structured data formats. Currently, there is a lack of effective means for structuring charts.

[0053] Based on this, the embodiments of the present disclosure provide a diagram structuring method based on a large language model, a training method and a device for a large language model. The diagram data can be identified and parsed, and the diagram data can be converted into structured data, which can reduce manual intervention, automatically extract data from the diagram, and improve the speed and accuracy of data processing. The present disclosure not only improves the readability and accessibility of the data, but also enables the data to be more widely used in automated reporting, market analysis, and decision support systems.

[0054] Among them, it should be noted that the execution subject of the graph structuring method based on the large language model of the embodiment of the present disclosure can be a graph structuring device, which can be implemented by software and / or hardware, and the device can be configured in an electronic device. The execution subject of the training method of the large language model of the embodiment of the present disclosure can be a training device for the large language model, which can be implemented by software and / or hardware, and the device can be configured in an electronic device. Exemplarily, the electronic device can include but is not limited to a terminal, a server, etc.

[0055] It is worth noting that, in some embodiments, the graph structuring method based on a large language model of the embodiments of the present disclosure can be implemented by an autonomous agent (AutoAgent) based on a large language model (LLM), which can convert graph data into structured data to achieve structured processing of the graph.

[0056] The following describes a graph structuring method, a model training method, and a device based on a large language model according to an embodiment of the present disclosure with reference to the accompanying drawings.

[0057] Figure 1 FIG. 1 is a flow chart of a diagram structuring method based on a large language model provided by an embodiment of the present disclosure. Figure 1 As shown, the graph structuring method based on a large language model may include but is not limited to the following steps.

[0058] In step 101, a chart and text prompt words to be processed are obtained.

[0059] In some embodiments, an input interface for charts and text prompts may be provided to the user, through which the user may input charts and text prompts, and correspondingly, the electronic device may obtain the charts and text prompts to be processed through the interface.

[0060] In some embodiments, the format of the above-mentioned chart to be processed may be an image format, such as JPEG format, but not limited thereto. In some embodiments, the format of the above-mentioned chart to be processed may be PDF format, but not limited thereto.

[0061] In some embodiments, the types of charts may include but are not limited to: pie charts, column charts, area charts, bar charts, line charts, etc. In some embodiments, the chart may be chart data in the financial field, but is not limited thereto, that is, the chart may also be a medical statistics chart in the medical field, or a traffic flow statistics chart in the transportation field, etc.

[0062] In some embodiments, the above-mentioned text prompt word may be a Prompt prompt word. For example, the text prompt word may be used as an input of a large language model to help the model understand and execute the user's instructions or tasks. For example, it may guide the large language model generation process so that the model can be optimized for a specific task or field. That is, in an embodiment of the present disclosure, the text prompt word may not be relevant descriptive information for the specific content in the chart to be processed, that is, the text prompt word may be irrelevant to the specific content in the chart to be processed. For example, the text prompt word may be used to guide the large language model to convert the chart into structured data.

[0063] In step 102, the large language model is used to process the chart and text prompt words to obtain a predicted Token (word unit) vector sequence output by the large language model. The Token can be translated as a word unit, which refers to a symbol used to represent a word or phrase in the natural language processing process, for example, it can refer to the smallest semantic unit that uses a number to represent a word in a language model.

[0064] In an embodiment of the present disclosure, when obtaining a chart and text prompt words to be processed, feature extraction can be performed on the chart to obtain visual features of the chart, and the visual features and the language features of the text prompt words are fused to obtain multimodal fusion features, and the multimodal fusion features are input into a large language model to obtain a predicted Token vector sequence output by the large language model. Exemplarily, the predicted Token vector sequence includes one or more predicted Token vectors.

[0065] In some embodiments, a visual encoder (also called a visual encoding model) can be used to extract features from a chart to obtain visual features of the chart. Exemplarily, the visual encoder can be a model based on VisionTransformer (abbreviated as: ViT), which is a model that applies the Transformer architecture to computer vision tasks. The core idea is to regard the image as a series of image blocks (tokens) and model these image blocks through the Transformer model, thereby achieving modeling of the global dependencies of the image. Exemplarily, the visual encoder can be used to extract visual features of a chart, that is, the visual encoder can divide the chart image into fixed-size image blocks (patches), and each patch is flattened and linearly transformed into a vector. These vectors are regarded as elements in a sequence, similar to words in natural language. These image block vectors are then added to maintain relative position information by adding position encoding, and are input into a standard Transformer encoder, processed by a self-attention mechanism, thereby obtaining the visual features of the chart.

[0066] In some embodiments, the text prompt word can be feature extracted by a text feature extraction algorithm to obtain the language features of the text prompt word. Exemplarily, the text feature extraction algorithm can include but is not limited to any of the following: bag of words model, TF-IDF (Term Frequency-Inverse Document Frequency, a commonly used weighting technology for information retrieval and data mining), word embedding and topic model, etc.

[0067] In some embodiments, the large language model can be used to predict matching Token vector sequences based on graphs and text prompts. Exemplarily, the input of the large language model can be graphs and text prompts, and the output of the large language model can include predicted Token vector sequences. In some embodiments, the large language model can be a multimodal large language model. In some embodiments, the large language model can use an autoregressive generative pre-trained language model. Exemplarily, the autoregressive generative pre-trained language model can learn the probability distribution of data through a generative model, and use an autoregressive method to gradually generate text that conforms to the laws of natural language.

[0068] In some embodiments, the autoregressive generative pre-trained language model can use an attention mechanism to calculate the weights of the visual features of the graph and the language features of the text prompt words when multimodal features are fused. By combining multimodal feature fusion with the autoregressive generative pre-trained language model, the feature extraction effect of the model can be further improved, thereby improving the model's understanding of visual information.

[0069] In step 103, the predicted Token vector sequence is decoded to obtain text description information and specific characters used to convert the graph into structured data.

[0070] In an embodiment of the present disclosure, when a prediction token vector sequence is obtained, the prediction token vector in the prediction token vector sequence can be decoded by a language decoder to obtain corresponding text description information and specific characters. Exemplarily, the specific characters can be characters used to convert a chart into structured data. For example, when a chart needs to be converted into structured tabular data, the specific characters can include but are not limited to one or more of a numeric placeholder, a text placeholder, a thousand separator, a filler, a mandatory display character, a conditional separator, and the like.

[0071] In step 104, the text description information is converted into structured data based on specific characters.

[0072] In an embodiment of the present disclosure, when text description information and specific characters used to convert a chart into structured data are obtained, the text description information can be converted into structured data based on the specific characters, thereby achieving the purpose of extracting, processing and converting the data in the chart into structured data.

[0073] In the above embodiments, by utilizing the deep learning model to identify and understand different types of charts, the data in the charts is extracted and converted into structured data, which facilitates further data processing and analysis. In the entire chart structuring process, manual intervention is greatly reduced, data is automatically extracted from the chart, and the speed and accuracy of data processing are improved. The disclosed embodiments not only improve the readability and accessibility of data, but also enable data to be more widely used in automated reporting, market analysis, and decision support systems.

[0074] Figure 2 FIG. 1 is a flow chart of a diagram structuring method based on a large language model provided by an embodiment of the present disclosure. Figure 2 As shown, in Figure 1 On the basis shown, the graph structuring method based on a large language model may further include the following steps.

[0075] In step 201, when the first Token vector is generated based on the autoregressive generative pre-trained language model, the category of the input data is identified based on the first Token vector, and the input data is the data input to the large language model. That is, the first Token vector can be used to identify the category of the input data.

[0076] In some embodiments, a first token classifier can be designed, which can be used to identify the category of the input data. Exemplarily, the first token classifier can be a binary classifier, which can be used to identify whether the category of the input data is a chart category. Exemplarily, the first token classifier can provide an early termination (EarlyStop) capability to improve the generation performance of the large language model. Exemplarily, when the first Token (i.e., first Token) vector is generated based on the input chart and text prompt words through an autoregressive generative pre-trained language model, the first Token vector can be input into the first token classifier for classification to obtain the category output by the first token classifier, which is the category of the data input into the large language model.

[0077] It should be noted that, in some embodiments, if the category of the input data is identified as a chart category based on the first Token vector, the following step 202 may be performed. If the category of the input data is not identified as a chart category based on the first Token vector, the following step 203 may be performed.

[0078] In step 202, when the category of the input data is a chart category, the autoregressive generative pre-trained language model generates the next Token vector based on the first Token vector until a predicted Token vector sequence is obtained, and the predicted Token vector sequence does not include the first Token vector.

[0079] In an embodiment of the present disclosure, when the category of the input data is a chart category, the autoregressive generative pre-trained language model can continue to generate the next Token vector based on the existing Token vector (such as the first Token vector) until a predicted Token vector sequence is obtained. It is understood that the predicted Token vector sequence may not include the first Token vector. That is, in some embodiments, the first Token vector may not be input into the language decoder for decoding processing.

[0080] In some embodiments, the last hidden layer (Last Hidden Layer) in the autoregressive generative pre-trained language model can deeply fuse the visual features of the chart and the language features of the text prompt words, thereby improving the model's understanding of visual information.

[0081] In step 203, when the category of the input data is not a chart category, the chart structuring task is terminated in advance.

[0082] In an embodiment of the present disclosure, when the category of the input data is not a chart category, the chart structuring task can be terminated in advance, that is, the reasoning of the model can be terminated in advance, that is, the autoregressive generative pre-trained language model will not continue to generate the next Token vector based on the existing Token vector, but terminate the process in advance. Exemplarily, this can output a prompt message to remind the user that the input data is not a chart category, and the chart structuring task is terminated in advance, thereby achieving non-chart generation acceleration and improving the generation performance of large language models.

[0083] In order to facilitate those skilled in the art to more clearly understand the solution of the present disclosure, Figure 3 Describe in detail.

[0084] For example, taking a chart of financial data, such as Figure 3 As shown, the visual encoding model can be used to extract features from the financial data chart to be processed to obtain the visual features of the financial data chart. The Prompt project guides the large language model generation process so that the model can be optimized for specific tasks or fields. For example, assuming that the Prompt text prompt word is a text prompt word for a specific task, the specific task may be to convert a chart into structured table data. Figure 3As shown in the figure, the multimodal feature fusion unit can perform multimodal feature fusion on the visual features of the financial data chart and the language features of the prompt text prompt word to obtain a multimodal fusion feature, and input the multimodal fusion feature into the autoregressive generative pre-trained language model. The last hidden layer (Last HiddenLayer) of the autoregressive generative pre-trained language model performs multimodal deep fusion on the visual features and the language features to generate the first Token vector (such as Figure 3 When T1 in , the first Token vector can be input into the first Token vector classifier. The first Token vector classifier determines that the category of the input data (i.e., the data input into the large language model) is a chart category, then the autoregressive generative pre-trained language model (such as the last hidden layer) can continue to generate the next Token vector (such as Figure 3 Ti in the figure) until the predicted Token vector sequence is output. The obtained predicted Token vector sequence is input into the language decoder, which decodes the predicted Token vector sequence to obtain text description information and specific characters used to convert the graph into structured data, so that the text description information can be converted into structured data based on the specific characters.

[0085] In some embodiments, after the text description information is converted into structured data based on specific characters, the structured data can be output in Markdown (Lightweight Markup Language) format. Figure 3 As shown, the structured data can be output in Markdown, so that the data can be directly used in reports and documents, thereby improving the usability of the data.

[0086] In the above embodiment, the category of the input data can be identified by the first Token vector. When the category of the input data is a chart category, the next Token vector is generated until all predicted Token vectors are obtained, which facilitates the generation of structured data. When the category of the input data is not a chart category, the chart structuring task is terminated in advance, which can accelerate non-chart generation and save computing resources. In addition, the present disclosure can improve the flexibility and adaptability of the model through the integration of Prompt engineering and multimodal feature fusion, especially in professional fields such as financial data analysis.

[0087] Figure 4 FIG. 1 is a flow chart of a method for training a large language model provided by an embodiment of the present disclosure. Figure 4 As shown, the training method of the large language model may include but is not limited to the following steps.

[0088] In step 401, training data and label data are obtained; the training data may include a chart to be processed and text prompt words, and the label data may include a specific character Token vector used to convert the chart to be processed into structured data.

[0089] In some embodiments, the types of the chart to be processed may include but are not limited to: pie charts, column charts, area charts, bar charts, line charts, etc. In some embodiments, the chart may be chart data in the financial field, but is not limited thereto, that is, the chart may also be a medical statistics chart in the medical field, or a traffic flow statistics chart in the transportation field, etc.

[0090] In some embodiments, the specific character may be a character used to convert a chart into structured data. For example, when a chart needs to be converted into structured tabular data, the specific character may include but is not limited to one or more of a number placeholder, a text placeholder, a thousand separator, a filler, a forced display character, a conditional separator, and the like.

[0091] In some embodiments, the above-mentioned text prompt word may be a Prompt prompt word. For example, the text prompt word may be used as an input of a large language model to help the model understand and execute the user's instructions or tasks. For example, it may guide the large language model generation process so that the model can be optimized for a specific task or field. That is, in an embodiment of the present disclosure, the text prompt word may not be relevant descriptive information for the specific content in the chart to be processed, that is, the text prompt word may be irrelevant to the specific content in the chart to be processed. For example, the text prompt word may be used to guide the large language model to convert the chart into structured data.

[0092] In some embodiments, when the text prompt words are text prompt words for multiple different specific tasks, each text prompt word may correspond to a set of label data for a specific task. That is, for multiple different specific tasks, the large language model may be independently trained using the corresponding label data based on the text prompt words of each feature task, so that the large language model can perform multiple different specific tasks.

[0093] In step 402, the large language model is used to process the chart and text prompt words to be processed to obtain a predicted Token vector sequence output by the large language model.

[0094] In some embodiments, feature extraction is performed on the chart to be processed to obtain the visual features of the chart to be processed; the visual features and the language features of the text prompt words are fused to obtain multimodal fusion features; the multimodal fusion features are input into the large language model to obtain a predicted Token vector sequence output by the large language model. Exemplarily, the predicted Token vector sequence includes one or more predicted Token vectors. In some embodiments, feature extraction can be performed on the chart to be processed by a visual encoder (also called a visual encoding model) to obtain the visual features of the chart to be processed. The description of the visual encoder, the language features of the text prompt words, and the large language model can be found in the relevant description in the above step 102, which will not be repeated here.

[0095] In some embodiments, the large language model may use an autoregressive generative pre-trained language model. For the description of the “autoregressive generative pre-trained language model”, please refer to the description of the “autoregressive generative pre-trained language model” in step 102, which will not be repeated here.

[0096] In step 403, a first loss value is determined based on the specific character Token vector and the predicted Token vector sequence.

[0097] In some embodiments, the first loss value may be calculated by using a preset first loss function, a specific character Token vector and a predicted Token vector sequence. Exemplarily, the first loss function may be a cross entropy loss function.

[0098] In step 404, the large language model is trained according to the first loss value.

[0099] In the embodiments of the present disclosure, the parameters of the large language model can be updated using a gradient descent algorithm based on the first loss value to reduce the value of the first loss function. As the number of iterations increases, the value of the first loss function will gradually decrease and eventually converge to a minimum value. Or when the number of iterations reaches a preset number, it can be considered that the model training termination condition is met, thereby obtaining a trained large language model.

[0100] In some embodiments, the training method further includes the following steps: when generating the first Token vector based on the autoregressive generative pre-trained language model, using a classifier to identify the category of the input data based on the first Token vector, the input data is the data input into the large language model; when the category of the input data is a chart category, the autoregressive generative pre-trained language model generates the next Token vector based on the first Token vector until a predicted Token vector sequence is obtained, and the predicted Token vector sequence does not include the first Token vector; or, when the category of the input data is not a chart category, terminating the training of the autoregressive generative pre-trained language model in advance. Among them, the classifier can be a first Token classifier. Its optional implementation method can be referred to above. Figure 2 The description of the relevant steps in the embodiment will not be repeated here.

[0101] In some embodiments, the label data may also include the category label of the input data. In this embodiment, the optional implementation of training the large language model according to the first loss value includes the following steps: obtaining the classification prediction result output by the classifier; determining the second loss value based on the classification prediction result and the category label; and training the large language model according to the first loss value and the second loss value.

[0102] Exemplarily, a second loss value can be calculated according to the classification prediction result and the above-mentioned category label by a preset second loss function. According to the first loss value and the second loss value, a gradient descent algorithm is used to update the parameters of the large language model to implement the training of the large language model, so as to obtain a trained large language model.

[0103] In some embodiments, when training a large language model, the visual encoding model, the classifier, and the language decoder may be jointly trained. Exemplarily, the autoregressive generative pre-trained language model may be fine-tuned using the training data first. After completing the training of the autoregressive generative pre-trained language model, the weights in the autoregressive generative pre-trained language model are fixed. Then, the visual encoding model, the classifier, and the language decoder are trained in sequence.

[0104] In the embodiments of the present disclosure, the large language model generation process is guided by Prompt engineering so that the model can be optimized for specific tasks or fields, and the flexibility and adaptability of the model can be improved by integrating Prompt engineering with multimodal feature fusion, so that the large language model can learn specific tasks so that it has the ability to convert graphs into structured data.

[0105] It should be noted that, in some embodiments, the graph structuring method based on a large language model involved in the embodiments of the present disclosure can be applied to the financial field. For example, the graph structuring method based on a large language model involved in the embodiments of the present disclosure can be used to quickly convert complex graph data into structured table data, so as to facilitate the use of the structured table data to write a market analysis report, that is, to obtain a corresponding financial report.

[0106] In some embodiments, the graph structuring method based on a large language model involved in the embodiments of the present disclosure can be applied to data integration scenarios. For example, when graph data is integrated into a data warehouse, the graph structuring method based on a large language model in the embodiments of the present disclosure can be used to structure the graph data to provide structured input for big data analysis and machine learning models.

[0107] In some embodiments, the diagram structuring method based on a large language model involved in the embodiments of the present disclosure can be applied to the fields of education and research. In the fields of education and research, text prompt words for different specific tasks can be used, and the diagram structuring method based on a large language model can help students and researchers understand complex financial data and use it for academic research.

[0108] In some embodiments, the chart structuring method based on a large language model involved in the embodiments of the present disclosure can be applied to a decision support system. For example, the chart structuring method based on a large language model can be used to perform automated market trend analysis. For example, the chart structuring method can be used to convert market trend chart data within a certain period of time into corresponding structured data, which facilitates market trend analysis based on the structured data, thereby supporting the decision-making process.

[0109] Figure 5 is a block diagram of a diagram structuring device based on a large language model provided by an embodiment of the present disclosure. Figure 5 As shown, the graph structuring device based on a large language model may include: a first acquisition module 501 , a second acquisition module 502 , a decoding module 503 and a conversion module 504 .

[0110] The first acquisition module 501 is used to acquire the chart and text prompt words to be processed.

[0111] The second acquisition module 502 is used to process the chart and text prompt words using the large language model to obtain a predicted Token vector sequence output by the large language model; the large language model is used to predict the matching Token vector sequence based on the chart and text prompt words.

[0112] In some embodiments, the second acquisition module 502 is used to: extract features from the chart to obtain visual features of the chart; fuse the visual features with the language features of the text prompt words to obtain multimodal fusion features; input the multimodal fusion features into the large language model to obtain a predicted Token vector sequence output by the large language model.

[0113] The decoding module 503 is used to decode the predicted Token vector sequence to obtain text description information and specific characters used to convert the chart into structured data.

[0114] The conversion module 504 is used to convert the text description information into structured data based on specific characters.

[0115] In some embodiments, the large language model uses an autoregressive generative pre-trained language model. In an embodiment of the present disclosure, Figure 6 As shown, the graph structuring device based on the large language model may also include a classification module 605 and a termination module 606. Wherein, the classification module 605 is used to identify the category of the input data based on the first Token vector when generating the first Token vector based on the autoregressive generative pre-trained language model, and the input data is the data input to the large language model. The termination module 606 is used to terminate the graph structuring task in advance when the category of the input data is not a graph category. In some embodiments, when the category of the input data is a graph category, the autoregressive generative pre-trained language model generates the next Token vector based on the first Token vector until a predicted Token vector sequence is obtained, and the predicted Token vector sequence does not include the first Token vector. Wherein, Figure 6 601 to 604 and Figure 5 501 to 504 have the same function and structure.

[0116] In some embodiments, Figure 7 As shown, the diagram structuring device based on the large language model may further include an output module 707. The output module 707 is used to output structured data in a lightweight markup language Markdown format. Figure 7 701 to 706 and Figure 6 601 to 606 have the same function and structure.

[0117] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0118] Figure 8 is a block diagram of a training device for a large language model provided by an embodiment of the present disclosure. Figure 8As shown, the training device of the large language model may include a first acquisition module 801 , a second acquisition module 802 , a determination module 803 and a training module 804 .

[0119] The first acquisition module 801 is used to acquire training data and label data; the training data includes the chart to be processed and the text prompt words, and the label data includes the specific character Token vector used to convert the chart to be processed into structured data.

[0120] The second acquisition module 802 is used to process the chart and text prompt words to be processed by using the large language model to obtain a predicted Token vector sequence output by the large language model.

[0121] In some embodiments, the second acquisition module 802 is used to: extract features from the chart to be processed to obtain visual features of the chart to be processed; fuse the visual features with the language features of the text prompt words to obtain multimodal fusion features; input the multimodal fusion features into the large language model to obtain a predicted Token vector sequence output by the large language model.

[0122] The determination module 803 is used to determine the first loss value based on the specific character Token vector and the predicted Token vector sequence.

[0123] The training module 804 is used to train the large language model according to the first loss value.

[0124] In some embodiments, the large language model uses an autoregressive generative pre-trained language model. In an embodiment of the present disclosure, Fig. 9 As shown, the training device of the large language model may also include: a classification module 905 and a termination module 906. Wherein, the classification module 905 is used to use a classifier to identify the category of the input data based on the first Token vector when generating the first Token vector based on the autoregressive generative pre-trained language model, and the input data is the data input to the large language model. The termination module 906 is used to terminate the training of the autoregressive generative pre-trained language model in advance when the category of the input data is not a chart category. In some embodiments, the category of the input data is a chart category, and the autoregressive generative pre-trained language model generates the next Token vector based on the first Token vector until a predicted Token vector sequence is obtained, and the predicted Token vector sequence does not include the first Token vector. Wherein, Fig. 9 901 to 904 and Figure 8 801 to 804 have the same function and structure.

[0125] In some embodiments, the label data also includes a category label of the input data. In an embodiment of the present disclosure, the training module 904 is used to: obtain a classification prediction result output by the classifier; determine a second loss value based on the classification prediction result and the category label; and train the large language model according to the first loss value and the second loss value.

[0126] In some embodiments, when the text prompt word is a plurality of text prompt words for different specific tasks, each text prompt word corresponds to a set of tag data.

[0127] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0128] The disclosed embodiment also provides an agent, which may include an input module, a processing module, and an output module. The input module is used to receive input information (such as a chart and a text prompt word); the processing module is used to determine a target task based on the input information received by the input module, determine a large language model based on the target task, and execute the chart structuring method based on the large language model in the above method embodiment by calling the large language model to obtain output information (such as structured data); the output module 1003 is used to output the output information obtained by the processing module.

[0129] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0130] like Fig.10 , is a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0131] like Fig.10As shown, the electronic device includes: one or more processors 1001, a memory 1002, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.10 A processor 1001 is taken as an example.

[0132] The memory 1002 is a non-transitory computer-readable storage medium provided by the present disclosure. The memory stores instructions executable by at least one processor to enable the at least one processor to execute the graph structuring method based on a large language model and / or the training method of a large language model provided by the present disclosure. The non-transitory computer-readable storage medium of the present disclosure stores computer instructions, which are used to enable a computer to execute the graph structuring method based on a large language model and / or the training method of a large language model provided by the present disclosure.

[0133] The memory 1002, as a non-transient computer-readable storage medium, can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the graph structuring method based on a large language model and / or the training method of a large language model in the embodiments of the present disclosure. The processor 1001 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1002, that is, implementing the graph structuring method based on a large language model and / or the training method of a large language model in the above method embodiments.

[0134] The memory 1002 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1002 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1002 may optionally include a memory remotely arranged relative to the processor 1001, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0135] The electronic device may further include: an input device 1003 and an output device 1004. The processor 1001, the memory 1002, the input device 1003 and the output device 1004 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0136] The input device 1003 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1004 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0137] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0139] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0140] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0141] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0142] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0143] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A graph structuring method based on a large language model, comprising: Get the charts and text prompts to be processed; The chart and the text prompt word are processed by a large language model to obtain a predicted Token vector sequence output by the large language model; the large language model is used to predict a matching Token vector sequence based on the chart and the text prompt word; Decoding the predicted Token vector sequence to obtain text description information and specific characters used to convert the chart into structured data; Based on the specific characters, the text description information is converted into structured data.

2. The method of claim 1, wherein: The process of processing the chart and the text prompt word by using the large language model to obtain a predicted Token vector sequence output by the large language model includes: Performing feature extraction on the chart to obtain visual features of the chart; Fusing the visual features with the language features of the text prompt words to obtain a multimodal fusion feature; The multimodal fusion features are input into the large language model to obtain a predicted Token vector sequence output by the large language model.

3. The method according to claim 1 or 2, wherein: The large language model uses an autoregressive generative pre-trained language model.

4. The method of claim 3, further comprising: When generating a first Token vector based on the autoregressive generative pre-trained language model, identifying a category of input data based on the first Token vector, the input data being data input into the large language model; The category of the input data is a chart category, the autoregressive generative pre-trained language model generates a next Token vector based on the first Token vector until the predicted Token vector sequence is obtained, and the predicted Token vector sequence does not include the first Token vector; or The category of the input data is not a chart category, and the chart structuring task is terminated prematurely.

5. The method of claim 1, further comprising: The structured data is output in a lightweight markup language Markdown format.

6. A method for training a large language model, comprising: Acquire training data and label data; the training data includes a chart to be processed and text prompt words, and the label data includes a specific character Token vector used to convert the chart to be processed into structured data; Using a large language model to process the chart to be processed and the text prompt word to obtain a predicted Token vector sequence output by the large language model; Determine a first loss value based on the specific character Token vector and the predicted Token vector sequence; The large language model is trained according to the first loss value.

7. The method of claim 6, wherein: The step of processing the chart to be processed and the text prompt word using the large language model to obtain a predicted Token vector sequence output by the large language model includes: Extracting features of the chart to be processed to obtain visual features of the chart to be processed; Fusing the visual features with the language features of the text prompt words to obtain a multimodal fusion feature; The multimodal fusion features are input into the large language model to obtain a predicted Token vector sequence output by the large language model.

8. The method according to claim 6 or 7, wherein: The large language model uses an autoregressive generative pre-trained language model.

9. The method of claim 8, further comprising: When generating a first Token vector based on the autoregressive generative pre-trained language model, using a classifier to identify a category of input data based on the first Token vector, the input data being data input into the large language model; The category of the input data is a chart category, the autoregressive generative pre-trained language model generates a next Token vector based on the first Token vector until the predicted Token vector sequence is obtained, and the predicted Token vector sequence does not include the first Token vector; or The category of the input data is not a chart category, and the training of the autoregressive generative pre-trained language model is terminated in advance.

10. The method of claim 9, wherein: The label data also includes a category label of the input data; The training of the large language model according to the first loss value includes: Obtaining the classification prediction result output by the classifier; Determining a second loss value based on the classification prediction result and the category label; The large language model is trained according to the first loss value and the second loss value.

11. The method according to any one of claims 6 to 10, wherein when the text prompt words are text prompt words for a plurality of different specific tasks, each of the text prompt words corresponds to a set of the label data.

12. A graph structuring device based on a large language model, comprising: The first acquisition module is used to acquire the chart and text prompt words to be processed; A second acquisition module is used to process the chart and the text prompt word using a large language model to obtain a predicted Token vector sequence output by the large language model; the large language model is used to predict a matching Token vector sequence based on the chart and the text prompt word; A decoding module, used to decode the predicted Token vector sequence to obtain text description information and specific characters used to convert the chart into structured data; The conversion module is used to convert the text description information into structured data based on the specific characters.

13. A large language model training device, comprising: A first acquisition module is used to acquire training data and label data; the training data includes a chart to be processed and a text prompt word, and the label data includes a specific character Token vector used to convert the chart to be processed into structured data; A second acquisition module is used to process the to-be-processed chart and the text prompt word using a large language model to obtain a predicted Token vector sequence output by the large language model; A determination module, configured to determine a first loss value based on the specific character Token vector and the predicted Token vector sequence; A training module is used to train the large language model according to the first loss value.

14. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 5 and 6 to 11.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5 and 6 to 11.

16. A computer program product comprising a computer program, wherein: The computer program implements the steps of the method according to any one of claims 1 to 5, 6 to 11 when executed by a processor.

Images