Data construction method and device, electronic equipment and storage medium

Abstract

The invention discloses a data construction method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: respectively constructing an original chart data set for each predetermined chart type; aiming at any original chart data set, respectively constructing a code program for generating a chart picture for each chart picture in the original chart data set; performing modification and amplification processing on each code program, and generating a new chart picture based on the code programs after modification and amplification processing; updating the original chart data set according to the new chart pictures, and enabling the number of the chart pictures in the updated original chart data set to be equal to a preset chart number demand threshold value; and constructing chart understanding instruction data used for training the model chart understanding ability according to the chart pictures in the updated original chart data set through a multi-modal large language model. According to the method, the construction efficiency of the chart understanding instruction data is improved, and the quantity of the constructed various types of chart understanding instruction data is balanced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a data construction method, device, electronic device and storage medium. Background Art

[0002] In the modern information age, the visual expression of data is becoming more and more diverse. As a tool for efficient information transmission, charts (such as pie charts, bar charts, or line charts) play an important role in various documents and reports. In order to extract key information from charts, a trained model with chart understanding ability is usually used. To train a model with chart understanding ability, it is necessary to first construct chart understanding instruction data for training the model.

[0003] At present, the method of constructing chart understanding instruction data for training models is usually to collect chart images from the Internet, and then construct chart understanding instruction data for training models through manual annotation. This method of data construction has certain shortcomings: since the images are mainly obtained from the Internet, there is inevitably an imbalance in the image categories, that is, a certain type or several types of charts account for a high proportion in the images, resulting in an imbalance in the categories of the constructed chart understanding instruction data, and due to the limitation on the number of chart-type images on the Internet, the scale of the constructed chart understanding instruction data is limited; through manual annotation, the efficiency of constructing chart understanding instruction data is low. Therefore, a new method for constructing chart understanding instruction data is urgently needed. Summary of the invention

[0004] The present invention provides a data construction method, device, electronic equipment, storage medium and computer program product.

[0005] According to one aspect of the present invention, there is provided a data construction method, comprising:

[0006] Acquire a plurality of predetermined chart types, and construct an original chart data set for each chart type; wherein the number of chart images included in each original chart data set is the same;

[0007] For any original chart data set, construct a code program for generating a chart image for each chart image in the original chart data set;

[0008] Modify and amplify each code program, and generate a new chart image based on the modified and amplified code program;

[0009] updating the original chart data set according to the new chart images so that the number of chart images in the updated original chart data set is equal to a predetermined chart number requirement threshold;

[0010] Through the multimodal large language model, chart understanding instruction data for training the model's chart understanding ability is constructed based on the chart images in the updated original chart dataset.

[0011] Optionally, construct a raw chart dataset for each chart type, including:

[0012] For any chart type, obtain multiple chart images belonging to the chart type;

[0013] According to the difficulty of understanding each chart image, multiple chart images are divided into multiple chart image sets;

[0014] A preset number of chart images are extracted from each chart image set to form an original chart data set.

[0015] Optionally, for any original chart data set, a code program for generating a chart image is constructed for each chart image in the original chart data set, including:

[0016] For any original chart data set, a code program for drawing a chart image is generated for each chart image in the original chart data set through a multimodal large language model;

[0017] The code program at least includes the text, data points, styles contained in the chart image and the drawing tools required for drawing the chart image.

[0018] Optionally, each code program is modified and amplified, and a new chart image is generated based on the modified and amplified code program, including:

[0019] For any code program, the text, data points or styles included in the code program are modified by using a large language model to obtain multiple modified and amplified code programs;

[0020] For any modified and amplified code program, the modified and amplified code program is executed. If the execution is successful, the generated chart image is used as a new chart image; if the execution fails, the modified and amplified code program is discarded.

[0021] Optionally, through the multimodal large language model, according to the chart images in the updated original chart dataset, chart understanding instruction data for training the chart understanding ability of the model is constructed, including:

[0022] For any chart image in the updated original chart dataset, the multimodal large language model is used to extract and interpret key information from the chart image;

[0023] Construct chart understanding instruction data based on chart images, extracted key information and interpretation results.

[0024] Optionally, the method further includes:

[0025] Through the graph understanding instruction data, a newly constructed lightweight model or an existing model that lacks graph understanding capabilities is trained to obtain a model for performing graph understanding tasks.

[0026] According to another aspect of the present invention, there is provided a data construction device, comprising:

[0027] A data set construction module, used to obtain a plurality of predetermined chart types and construct an original chart data set for each chart type; wherein the number of chart images included in each original chart data set is the same;

[0028] A code generation module, for constructing a code program for generating a chart image for each chart image in the original chart data set for any original chart data set;

[0029] An amplification module, used to modify and amplify each code program, and generate a new chart image based on the modified and amplified code program;

[0030] A data set updating module, used for updating the original chart data set according to the new chart images, so that the number of chart images in the original chart data set after the update is equal to a predetermined chart number requirement threshold;

[0031] The data generation module is used to construct chart understanding instruction data for training the chart understanding ability of the model based on the chart images in the updated original chart dataset through a multimodal large language model.

[0032] Optionally, the dataset construction module is specifically used to:

[0033] For any chart type, obtain multiple chart images belonging to the chart type;

[0034] According to the difficulty of understanding each chart image, multiple chart images are divided into multiple chart image sets;

[0035] A preset number of chart images are extracted from each chart image set to form an original chart data set.

[0036] Optionally, the code generation module is specifically used to:

[0037] For any original chart data set, a code program for drawing a chart image is generated for each chart image in the original chart data set through a multimodal large language model;

[0038] The code program at least includes the text, data points, styles contained in the chart image and the drawing tools required for drawing the chart image.

[0039] Optionally, the amplification module is specifically used for:

[0040] For any code program, the text, data points or styles included in the code program are modified by using a large language model to obtain multiple modified and amplified code programs;

[0041] For any modified and amplified code program, the modified and amplified code program is executed. If the execution is successful, the generated chart image is used as a new chart image; if the execution fails, the modified and amplified code program is discarded.

[0042] Optionally, the data generation module is specifically used for:

[0043] For any chart image in the updated original chart dataset, the multimodal large language model is used to extract and interpret key information from the chart image;

[0044] Construct chart understanding instruction data based on chart images, extracted key information and interpretation results.

[0045] Optionally, the data construction device further includes:

[0046] The training module is used to train a newly constructed lightweight model or an existing model that lacks graph understanding capabilities through graph understanding instruction data to obtain a model for performing graph understanding tasks.

[0047] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0048] at least one processor; and

[0049] a memory communicatively connected to at least one processor; wherein,

[0050] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by the at least one processor so that the at least one processor can execute the data construction method of the embodiment of the present invention.

[0051] According to another aspect of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data construction method of an embodiment of the present invention when executed.

[0052] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, and when the computer program is executed by a processor, the steps in the above method are implemented.

[0053] The technical solution of the embodiment of the present invention is to obtain a small number of chart images to form an original chart data set for each chart type, and to construct a code program for generating the chart image for each chart image in the original chart data set for each chart type, and then generate a large number of new chart images through code amplification, and add them to the original chart data, thereby solving the problem of the small number of chart images on the Internet; and the number of chart images included in each original chart data set after supplementation is the same, ensuring the balance of the number of chart images of each type; based on this, through a multimodal large language model, large-scale and classified balanced chart understanding instruction data can be generated according to various types of chart images, and compared with manual labeling, the efficiency of generating chart understanding instruction data is improved.

[0054] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 It is a flowchart of a data construction method provided by an embodiment of the present invention;

[0057] Figure 2 is a flowchart of another data construction method provided by an embodiment of the present invention;

[0058] Figure 3 is a structural schematic diagram of a data construction device provided by an embodiment of the present invention;

[0059] Figure 4 It is a schematic diagram of the structure of an electronic device for implementing the data construction method of an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0061] Embodiment 1

[0062] Figure 1 A flowchart of a data construction method provided in an embodiment of the present invention. This embodiment can be applied to scenarios of constructing diagram understanding instruction data for training model diagram understanding capabilities. The method can be executed by a data construction device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device.

[0063] like Figure 1 As shown, the data construction method includes:

[0064] S101 , obtaining a plurality of predetermined chart types, and constructing an original chart data set for each chart type.

[0065] In the embodiment of the present invention, the chart type includes at least one of the following: a column chart, a bar chart, a line chart, a pie chart, a scatter plot, a box plot, a radar chart, and a table. The chart type can be determined by the user according to actual needs, and specifically, multiple chart types determined by the user can be obtained through human-computer interaction.

[0066] In an embodiment of the present invention, an original chart data set is constructed for each chart type, including: first, determining the data source of the chart image; wherein the data source may include a public data set website, a search engine, and a professional field database. Further, for any chart type, a preset number of chart images belonging to the chart type is obtained from at least one data source, wherein the preset number can be a preset smaller value, for example, the preset number is 9. It should be noted that the reason why the preset number is set to a very small value is that the number of chart images of each chart type at the data source may be unbalanced. If the setting is too large, it may cause the chart images of some chart types to fail to obtain a sufficient number. Further, according to the chart images of each chart type obtained, an original chart data set is constructed; wherein the number of chart images included in each original chart data set is the same, so that the number of chart images in each original chart data set can be balanced. It should be noted that the various types of chart images involved in the present invention are picture data fully authorized by all parties (such as the author or owner of the picture), and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant chart images comply with the relevant laws, regulations and standards of the relevant countries and regions, and take necessary confidentiality measures without violating public order and good customs.

[0067] S102 . For any original chart data set, construct a code program for generating a chart image for each chart image in the original chart data set.

[0068] In an embodiment of the present invention, for any original chart data set, a code program for drawing the chart image is generated for each chart image in the original chart data set through a multimodal large language model; wherein the multimodal large language model refers to a large high-performance model with chart understanding capabilities. Specifically, for the original chart data set corresponding to any chart type, the chart image of the original chart data set is used as the input of the multimodal large language model, and the multimodal large language model is instructed to generate a code program capable of constructing the chart image according to the input chart image; wherein the code program can be constructed based on the Python language, and the code program also includes at least the text, data points, chart styles and drawing tools required for drawing chart images that are recognized by the multimodal large language model from the chart image. It can be understood that the text can be selected from the chart title, legend description, annotation description, etc. in the chart image; the data point can be selected from the numerical point in the chart image, such as the horizontal and vertical coordinate values ​​corresponding to each inflection point in the line chart; the chart style can be selected from the chart color, shape, line thickness font format, etc.

[0069] S103, modifying and amplifying each code program, and generating a new chart image based on the modified and amplified code program.

[0070] In an optional implementation, for any code program, the text, data points or chart styles included in the code program are modified through a large language model to obtain multiple modified and amplified code programs. Exemplarily, a general code modification instruction is constructed; for each code program used to draw a chart image, the code program and the general code modification instruction are input into the large language model together, and according to the output of the large language model, a code program capable of drawing a new chart image is obtained; wherein, the general code modification instruction can be exemplarily: "The function of this code is to draw a chart, and your task is to modify the text, data points, and chart styles in the code. You can modify all text content in the chart, the value and distribution of data points, and the color, thickness, direction, etc. of the chart. Please note that the modified content should be logical as a whole, for example, the value range and chart format should be consistent with the theme expressed in the text."

[0071] By modifying and amplifying the code program corresponding to each chart image in the above manner, a large number of new code programs that can draw chart images can be obtained. It should be noted that the number of new code programs obtained can be greater than or equal to the quantity requirement threshold set by the user. For any modified and amplified code program, the modified and amplified code program is executed. If the execution is successful, the generated chart image is used as a new chart image; if the execution fails, the modified and amplified code program is discarded. In this way, through the above process, each chart type can obtain a large number of chart images belonging to it, thus avoiding the problem of not being able to obtain enough chart images from the data source.

[0072] It should be noted that if a multimodal large language model is used to modify and amplify a code program, the input may include not only the code program and modification instructions, but also chart images corresponding to the code program.

[0073] S104: updating the original chart data set according to the new chart image, so that the number of chart images in the updated original chart data set is equal to a predetermined chart number requirement threshold.

[0074] In the embodiment of the present invention, for any chart type, new chart images belonging to the type are sequentially added to the original chart data set corresponding to the chart type until the number of chart images in the original chart data set is equal to the predetermined chart number requirement threshold. Through this operation, the number of chart images included in the updated original chart data set corresponding to each chart type is the same, achieving the effect of balancing the number of images of each chart type, thereby ensuring that the chart understanding instruction data of different chart types generated subsequently are balanced.

[0075] S105. Using a multimodal large language model, and based on the chart images in the updated original chart data set, construct chart understanding instruction data for training the chart understanding capability of the model.

[0076] In an embodiment of the present invention, for any chart image in the updated original chart data set, key information of the chart image is extracted and interpreted through a multimodal large language model; chart understanding instruction data is constructed based on the chart image, the extracted key information and the interpretation result.

[0077] In an embodiment of the present invention, for each chart type, a small number of chart images are obtained to form an original chart data set. For the original chart data set of each chart type, a code program for generating the chart image is constructed for each chart image in the original chart data set, and then a large number of new chart images are generated by code amplification, and added to the original chart data, thereby solving the problem of the small number of chart images on the Internet; and the number of chart images included in each original chart data set after supplementation is the same, ensuring the balance of the number of chart images of each type; based on this, through a multimodal large language model, large-scale and classified balanced chart understanding instruction data can be generated according to various types of chart images, and compared with manual labeling, the efficiency of generating chart understanding instruction data is improved.

[0078] Embodiment 2

[0079] Figure 2 A flowchart of a data construction method is provided for an embodiment of the present invention. Figure 2 , the method comprises the following steps:

[0080] S201: Acquire a plurality of predetermined chart types, and for any chart type, acquire a plurality of chart images belonging to the chart type.

[0081] Exemplarily, for any chart type, multiple chart images belonging to the chart type are obtained from at least one predetermined data source (eg, a public dataset website, a search engine, a professional field database).

[0082] S202. Divide the plurality of chart images into a plurality of chart image sets according to the difficulty of understanding each chart image.

[0083] In an embodiment of the present invention, the process of determining the difficulty of understanding of each chart image includes: first, for any chart image, determining a difficulty of understanding evaluation index; wherein the difficulty of understanding evaluation index includes the number of data points in the chart image, the diversity of data types, the complexity of the chart structure, the clarity of the title, the rationality of color matching, the clarity of element layout, etc.; according to the value of the determined difficulty of understanding evaluation index and the pre-specified classification criteria, the difficulty of understanding of each chart image is determined. Exemplarily, if the number of data points of a chart image is greater than a first threshold value (e.g., 50), the difficulty level of understanding of the chart image is considered to be high; if the number of data points of a chart image is greater than a second threshold value (e.g., 20) and less than the first threshold value (e.g., 50), the difficulty level of understanding of the chart image is considered to be medium; if the number of data points of a chart image is less than the second threshold value (e.g., 20), the difficulty level of understanding of the chart image is considered to be low.

[0084] On the above basis, according to the difficulty of understanding each chart image, the multiple chart images are divided into multiple chart image sets, wherein the chart images in different chart image sets have different difficulty of understanding. Exemplarily, the chart images with low difficulty of understanding are grouped into one set, the chart images with medium difficulty of understanding are grouped into another set, and the chart images with high difficulty of understanding are grouped into one set.

[0085] S203: extracting a preset number of chart images from each chart image set to form an original chart data set.

[0086] Optionally, an equal amount of some chart images are extracted from each chart image set to form an original chart data set. For example, 3 pictures are extracted from different chart image sets to form an original chart data set. It is understandable that since the original chart data set includes chart images of different difficulty levels, the chart understanding instruction data finally constructed includes chart images of different difficulty levels. When the chart understanding instruction data is used to train the model, the model can better cope with chart understanding tasks in various practical scenarios, improve the generalization ability of the model, and perform better when facing new charts of different difficulty levels.

[0087] S204: for any original chart data set, construct a code program for generating a chart image for each chart image in the original chart data set.

[0088] Optionally, for any original chart data set, a multimodal large language model is used to generate a code program for drawing each chart image in the original chart data set; wherein the code program includes at least the text, data points, style contained in the chart image and the drawing tools required for drawing the chart image.

[0089] S205 , modify and amplify each code program, and generate a new chart image based on the modified and amplified code program.

[0090] Optionally, for any code program, the text, data points or styles included in the code program are modified through a large language model to obtain multiple modified and amplified code programs; for any modified and amplified code program, the modified and amplified code program is executed, and if the execution is successful, the generated chart image is used as a new chart image; if the execution fails, the modified and amplified code program is discarded.

[0091] S206: Update the original chart data set according to the new chart image, so that the number of chart images in the updated original chart data set is equal to a predetermined chart number requirement threshold.

[0092] S207. Using a multimodal large language model, and based on the chart images in the updated original chart data set, construct chart understanding instruction data for training the chart understanding capability of the model.

[0093] The implementation process of steps S206-S207 can refer to the description of the above embodiment.

[0094] S208. Train a newly constructed lightweight model or an existing model that lacks graph understanding capabilities through graph understanding instruction data to obtain a model for performing graph understanding tasks.

[0095] In an embodiment of the present invention, after obtaining the chart understanding instruction data through steps S201-S207, the chart understanding instruction data can be used as a training sample to train a newly constructed lightweight model or an existing model that lacks chart understanding capabilities to obtain a model for performing chart understanding tasks; wherein the lightweight model can be a model that can be deployed on a mobile terminal.

[0096] In an embodiment of the present invention, an original chart data set is composed of chart images of different difficulty levels. After a lightweight model or an existing model is conditioned on chart understanding instruction data generated based on the chart images of different difficulty levels, the lightweight model or the existing model can better cope with chart understanding tasks in various actual scenarios, improve the generalization ability of the model, and perform better when faced with new charts of different difficulty levels.

[0097] Embodiment 3

[0098] Figure 3 Schematic diagram of a data construction device provided by an embodiment of the present invention. This embodiment is applicable to scenarios where diagram understanding instruction data for training model diagram understanding capabilities is constructed. Figure 3 As shown, the data construction device includes:

[0099] The data set construction module 301 is used to obtain a plurality of predetermined chart types and construct an original chart data set for each chart type; wherein the number of chart images included in each original chart data set is the same;

[0100] A code generation module 302 is used to construct a code program for generating a chart image for each chart image in the original chart data set for any original chart data set;

[0101] An amplification module 303 is used to modify and amplify each code program, and generate a new chart image based on the modified and amplified code program;

[0102] The data set updating module 304 is used to update the original chart data set according to the new chart image, so that the number of chart images in the updated original chart data set is equal to a predetermined chart number requirement threshold;

[0103] The data generation module 305 is used to construct diagram understanding instruction data for training the diagram understanding capability of the model according to the diagram images in the updated original diagram data set through the multimodal large language model.

[0104] Optionally, the data set construction module 301 is specifically used for:

[0105] For any chart type, obtain multiple chart images belonging to the chart type;

[0106] According to the difficulty of understanding each chart image, multiple chart images are divided into multiple chart image sets;

[0107] A preset number of chart images are extracted from each chart image set to form an original chart data set.

[0108] Optionally, the code generation module 302 is specifically used for:

[0109] For any original chart data set, a code program for drawing a chart image is generated for each chart image in the original chart data set through a multimodal large language model;

[0110] The code program at least includes the text, data points, styles contained in the chart image and the drawing tools required for drawing the chart image.

[0111] Optionally, the amplification module 303 is specifically used for:

[0112] For any code program, the text, data points or styles included in the code program are modified by using a large language model to obtain multiple modified and amplified code programs;

[0113] For any modified and amplified code program, the modified and amplified code program is executed. If the execution is successful, the generated chart image is used as a new chart image; if the execution fails, the modified and amplified code program is discarded.

[0114] Optionally, the data generation module 305 is specifically used for:

[0115] For any chart image in the updated original chart dataset, the multimodal large language model is used to extract and interpret key information from the chart image;

[0116] Construct chart understanding instruction data based on chart images, extracted key information and interpretation results.

[0117] Optionally, the data construction device further includes:

[0118] The training module is used to train a newly constructed lightweight model or an existing model that lacks graph understanding capabilities through graph understanding instruction data to obtain a model for performing graph understanding tasks.

[0119] The data construction device provided in the embodiment of the present invention can execute the data construction method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0120] According to an embodiment of the present invention, the present invention also provides an electronic device, a readable storage medium and a computer program product.

[0121] Embodiment 4

[0122] Figure 4 The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit implementation of the invention described and / or claimed herein.

[0123] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0124] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0125] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as executing a data construction method.

[0126] In some embodiments, the data construction method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data construction method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the data construction method in any other appropriate manner (e.g., by means of firmware).

[0127] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0128] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data construction device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0129] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0130] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0131] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0132] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0133] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0134] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data construction method, characterized in that: include: Acquire a plurality of predetermined chart types, and construct an original chart data set for each chart type; wherein the number of chart images included in each of the original chart data sets is the same; For any of the original chart data sets, construct a code program for generating the chart image for each chart image in the original chart data set; Performing modification and amplification processing on each of the code programs, and generating a new chart image based on the modified and amplified code programs; Update the original chart data set according to the new chart image, so that the number of chart images in the updated original chart data set is equal to a predetermined chart number requirement threshold; Through the multimodal large language model, according to the chart images in the updated original chart data set, chart understanding instruction data for training the model's chart understanding ability is constructed.

2. The method according to claim 1, characterized in that The method constructs an original chart dataset for each chart type, including: For any of the chart types, obtain multiple chart images belonging to the chart type; According to the difficulty of understanding each of the chart images, the plurality of chart images are divided into a plurality of chart image sets; A preset number of chart images are extracted from each of the chart image sets to form the original chart data set.

3. The method according to claim 1, characterized in that For any of the original chart data sets, a code program for generating the chart image is constructed for each chart image in the original chart data set, including: For any of the original chart data sets, using a multimodal large language model, a code program for drawing the chart image is generated for each chart image in the original chart data set; The code program at least includes the text, data points, style contained in the chart image and the drawing tools required for drawing the chart image.

4. The method according to claim 3, characterized in that: The step of modifying and amplifying each of the code programs and generating a new chart image based on the modified and amplified code programs includes: For any of the code programs, modify the text, data points or styles included in the code program by using a large language model to obtain multiple modified and amplified code programs; For any of the modified and amplified code programs, the modified and amplified code program is executed. If the execution is successful, the generated chart image is used as a new chart image; if the execution fails, the modified and amplified code program is discarded.

5. The method according to claim 1, characterized in that The method of constructing chart understanding instruction data for training the chart understanding ability of the model based on the chart images in the updated original chart data set through the multimodal large language model includes: For any chart image in the updated original chart data set, extract and interpret key information of the chart image through a multimodal large language model; The chart understanding instruction data is constructed according to the chart image, the extracted key information and the interpretation result.

6. The method according to claim 1, characterized in that The method further comprises: Through the graph understanding instruction data, a newly constructed lightweight model or an existing model that lacks graph understanding capabilities is trained to obtain a model for performing graph understanding tasks.

7. A data construction device, characterized in that: include: A data set construction module, used to obtain a plurality of predetermined chart types, and construct an original chart data set for each chart type; wherein the number of chart images included in each of the original chart data sets is the same; A code generation module, for constructing a code program for generating the chart image for each chart image in the original chart data set for any of the original chart data sets; An amplification module, used to modify and amplify each of the code programs, and generate a new chart image based on the modified and amplified code programs; A data set updating module, used for updating the original chart data set according to the new chart image, so that the number of chart images in the original chart data set after the update is equal to a predetermined chart number requirement threshold; The data generation module is used to construct chart understanding instruction data for training the chart understanding ability of the model based on the chart images in the updated original chart data set through a multimodal large language model.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method according to any one of claims 1 to 6 when executed.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images