A chart key information extraction method based on a multi-modal large model

Abstract

The application relates to the technical field of electric digital data processing, in particular to a chart key information extraction method based on a multi-modal large model. The method comprises the following steps: acquiring an initial prompt text; acquiring a target information type according to the type of a target chart and the type of a question input by a user; acquiring position information of target information corresponding to the target information type in the target chart; constructing a target prompt text according to the prompt text corresponding to the target information type in the initial prompt text and the position information of the target information corresponding to the target information type in the target chart; inputting the target prompt text and the target chart into a multi-modal large model to acquire key information of the target chart related to the question input by the user according to the output of the multi-modal large model. The application can accurately extract key information of a chart according to task requirements.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital data processing, in particular to a chart key information extraction method based on a multi-modal large model. BACKGROUND

[0002] The multi-modal chart understanding task is an important research direction in the field of artificial intelligence, and the goal is to accurately extract context information from charts to support the reasoning and decision-making of models. At present, the OCR-based technology is one of the most commonly used enhancement methods in the multi-modal chart understanding task. The OCR technology extracts all potential relevant information by comprehensively extracting the text in the chart for the model. However, this comprehensive extraction strategy has significant limitations in practical applications. First, the OCR lacks pertinence in information extraction and often introduces a large amount of redundant information. For example, in a task that only needs the background information of the coordinate axis, parsing the data points one by one not only increases the computational complexity, but also may interfere with the actual task. These redundant information seriously affects the efficiency of the model, especially when dealing with complex reasoning tasks, which interferes with the judgment of the model. How to accurately extract the key information of the chart according to the task requirements is a problem to be solved. SUMMARY

[0003] The present application aims to provide a chart key information extraction method based on a multi-modal large model to accurately extract the key information of the chart according to the task requirements.

[0004] According to the present application, a chart key information extraction method based on a multi-modal large model is provided, which comprises the following steps:

[0005] S100, obtaining an initial prompt text; the initial prompt text includes a title prompt text, a legend prompt text and a coordinate axis prompt text, the title prompt text includes a text for representing the extraction of the title and a text for representing the output of the title information format, the legend prompt text includes a text for representing the extraction of the legend and a text for representing the output of the legend information format, and the coordinate axis prompt text includes a text for representing the extraction of the coordinate axis and a text for representing the output of the coordinate axis information format.

[0006] S200, obtaining a target information type according to the type of the target chart and the type of the question input by the user; the target information type includes at least one of the title information type, the legend information type and the coordinate axis information type.

[0007] S300, obtaining the position information of the target information corresponding to the target information type in the target chart.

[0008] S400, constructing a target prompt text according to the prompt text in the initial prompt text corresponding to the target information type and position information of the target information corresponding to the target information type in the target graph.

[0009] S500, inputting the target prompt text and the target graph into the multi-modal large model, and acquiring key information related to the question input by the user according to an output of the multi-modal large model.

[0010] Compared with the prior art, the present application has at least the following beneficial effects:

[0011] The initial text of the present application includes prompt texts corresponding to the title, legend and coordinate axis, the prompt text includes text for representing the extraction of the corresponding content and text for representing the format that should be met when the corresponding content is output, and the multi-modal large model can realize the structured output of the corresponding part of the content according to the prompt text therein; on this basis, the present application determines which prompt texts in the initial text are part of the target prompt text according to the type of the target graph and the type of the question input by the user, and this part can represent the extraction demand and output format of the key information that has relatively large relevance with the question input by the user, which is beneficial to improve the accuracy of extracting the key information and expressing the key information; and the target prompt text of the present application further includes position information of the target information corresponding to the target information type in the target graph, based on which the multi-modal large model can quickly locate, which is beneficial to improve the efficiency of the information output by the large model. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0013] Figure 1 The flowchart of the chart key information extraction method based on the multi-modal large model provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0015] According to the present embodiment, as Figure 1As shown, a method for extracting key information from charts based on a multimodal large model is provided. The method includes the following steps:

[0016] S100, Obtain initial prompt text; the initial prompt text includes title prompt text, legend prompt text, and coordinate axis prompt text. The title prompt text includes text used to represent the extracted title and text used to represent the format of the output title information. The legend prompt text includes text used to represent the extracted legend and text used to represent the format of the output legend information. The coordinate axis prompt text includes text used to represent the extracted coordinate axis and text used to represent the format of the output coordinate axis information.

[0017] In this embodiment, the title serves as the theme of the chart, providing the model with overall background semantics, such as the source of the data, the time range, or the direction of the analysis; the legend contains the correspondence between categories and colors, which is an important source of information for classification reasoning tasks. For example, the legend can explain the specific category or group represented by a certain color; the coordinate axis information can provide the unit and range of the data. For example, the x-axis represents the year, and the y-axis represents the percentage change, which can directly guide numerical queries or trend descriptions.

[0018] In this embodiment, the initial prompt text is pre-built.

[0019] In a preferred embodiment, the legend prompt text includes: identifying the legend type and its corresponding color, in the following format: {Type 1 name (color)}|{Type 2 name (color)}|…|{Type k name (color)}|…|{Type p name (color)}, where k is a variable ranging from 1 to p, and p is the number of types in the legend. Based on this preferred embodiment, the multimodal large model can accurately output legend information. In one specific embodiment, the multimodal large model outputs the legend section as: Energy Import (gray)|Agricultural Export (green).

[0020] As a preferred embodiment, the coordinate axis prompt text includes: extracting the x-axis labels and their values, in the following format: x-axis representation: {x-axis label}, its value is: {value1}|{value2}|…|{valuef} x}|…|{value g x Extract the y-axis labels and their values, in the following format: y-axis representation: {y-axis label}, its value: {value1}|{value2}|…|{valuef} y}|…|{value g y}; where f x and f y Both are variables, f x The value range is from 1 to g x g x f represents the number of data points corresponding to the x-axis.y g y , g y is the number of data corresponding to the y-axis. Based on the preferred embodiment, the multi-modal large model can accurately output the coordinate axis information. As an embodiment, the multi-modal large model outputs the coordinate axis part as follows: the x-axis represents the year, and the value is 1990|1995|2000; the y-axis represents the proportion, and the value is 0%|50%|100%.

[0021] S200, obtaining a target information type according to the type of the target chart and the type of the question input by the user; the target information type at least includes one of a title information type, a legend information type, and a coordinate axis information type.

[0022] In this embodiment, the target chart includes a title, a legend, and a coordinate axis. As an embodiment, the type of the target chart is one of a column chart type, a line chart type, a pie chart type, and a scatter chart type.

[0023] As an embodiment, the type of the question input by the user is one of a numerical query type, a data point comparison type, a trend summary type, a semantic description type, and a category inference type. As a preferred embodiment, the process of obtaining the type of the question input by the user includes:

[0024] S201, obtaining a question type field list A; A={A1, A2,…,An}, n is a preset number of question types; Ai is the ith record included in A, i is an integer and its value range is 1 to n; Ai=(T, d1, d2,…,dr(i),…,dR(i)), T is a question type included in Ai, dr(i) is the r(i)th field included in Ai, r(i) is an integer and its value range is 1 to R(i), R(i) is the number of fields included in Ai, and the question type of any di is T; T included in different Ai is different, and (d1, d2,…,dr(i),…,dR(i)) is different. i n i i i i,1 i,2 i,r(i) i,R(i) i i i,r(i) i i i,r(i) i i i i,1 i,2 i,r(i) i,R(i) ​​​​​​​​​​​​​​​​​​​​​The priority of the field with the position number in front is higher than the priority of the field with the position number behind, and different A i The priority of the field with the same position number is the same.

[0025] In this embodiment, A is a pre-constructed list, each T i The corresponding field is known, for example, when T i is a trend summary type, T i The corresponding field includes trend summary, trend analysis, trend summary, trend description, trend summary, trend analysis, trend summary, and trend description.

[0026] In this embodiment, the priority of different fields corresponding to the same problem type is related to the number of times the corresponding field is used by the user when asking questions of the corresponding problem type in the historical time period. The more times a certain field is used by the user when asking questions of a certain problem type, the closer the position number of the field in the record to which the problem type belongs. It should be understood that d i,r(i) The position number in A i is r(i), for example, d i,1 The position number in A i is 1, d i,1 is the field with the most front position number in A i .

[0027] S202, the fields included in A are matched with the question input by the user in order of priority from high to low, until a matching field is obtained.

[0028] In this embodiment, the matching of the field and the question input by the user is performed in order of priority from high to low. Since the field with higher priority corresponds to a field with higher probability of occurrence, matching the field and the question input by the user in order of priority from high to low is beneficial to improve the efficiency of obtaining a matching field.

[0029] In this embodiment, different A i The priority of the field with the same position number is the same, and in the process of matching the fields included in A with the question input by the user in order of priority from high to low, the fields included in A1 with the highest priority are matched first, then the fields included in A2 with the highest priority, and finally the fields included in A n The fields included in A1 with the highest priority are matched first, and if all matching fails, the fields included in A1 with the second highest priority are matched, then the fields included in A2 with the second highest priority, and finally the fields included in A nThe order of the included priority second high field is matched first, and so on, until the matched field is obtained. It should be understood that if the user input question contains the same content as a certain field, the field is determined as a matched field.

[0030] S203, determining the question type of the record to which the matched field belongs as the type of the user input question.

[0031] Based on S201-S203, the type of the user input question can be quickly and accurately obtained.

[0032] As a preferred embodiment, S200 includes:

[0033] S210, obtaining an information type list C; C={C1, C2,…, C h ,…,C H}, C h is the hth record included in C, h ranges from 1 to H, H is the number of records included in C; C h =(C h,1 ,C h,2 ,C h,3 ), C h,1 is the question type included in C h , C h,2 is the chart type included in C h , different C h includes different C h,1 or C h,2 or C h,1 and C h,2 are different, C h,3 is the set of information types corresponding to C h and C h,1 included in C h,2 , C h,3 at least includes one of title information type, legend information type and coordinate axis information type.

[0034] In this embodiment, C is a pre-constructed list, each C h,1 and C h,2 corresponding C h,3 is known, and C h,3 can be determined according to experience. It should be understood that if C h,3 only includes title information type and coordinate axis information type, it means that the key information is title information and coordinate axis information when the multi-modal large model answers the question of type C h,1 about the chart of type C h,2 ; if C h,3The C includes only the legend information type and the coordinate axis information type, indicating that the key information is the legend information and the coordinate axis information when the multi-modal large model answers the question about the type C chart in the type C h,1 The C includes only the legend information type and the coordinate axis information type, indicating that the key information is the legend information and the coordinate axis information when the multi-modal large model answers the question about the type C chart in the type C h,2 The C includes only the legend information type and the coordinate axis information type, indicating that the key information is the legend information and the coordinate axis information when the multi-modal large model answers the question about the type C chart in the type C h,3 The C includes only the legend information type and the coordinate axis information type, indicating that the key information is the legend information and the coordinate axis information when the multi-modal large model answers the question about the type C chart in the type C h,1 The C includes only the legend information type and the coordinate axis information type, indicating that the key information is the legend information and the coordinate axis information when the multi-modal large model answers the question about the type C chart in the type C h,2 The C includes only the legend information type and the coordinate axis information type, indicating that the key information is the legend information and the coordinate axis information when the multi-modal large model answers the question about the type C chart in the type C

[0035] As a specific embodiment, the question type includes a numerical query type, a data point comparison type, a trend summary type, a semantic description type, and a category inference type.

[0036] S220, matching the type of the target chart and the type of the question input by the user in C.

[0037] As a specific embodiment, the type of the target chart and the type of the question input by the user are matched with C1, C2, …, C H in turn, until a record included in C includes a question type and a chart type that are the same as the type of the question input by the user and the type of the target chart, and the record is the matching record.

[0038] S230, determining the information type included in the matching record as the target information type.

[0039] Based on S210-S230, the target information type can be quickly and accurately determined.

[0040] S300, obtaining the position information of the target information corresponding to the target information type in the target chart.

[0041] It should be understood that the target information corresponding to the title information type is the title information, the target information corresponding to the legend information type is the legend information, and the target information corresponding to the coordinate axis information type is the coordinate axis information. As a specific embodiment, the position information of the title, the legend, and the coordinate axis in the target chart is pre-known and is pre-input information.

[0042] S400, constructing a target prompt text according to the prompt text corresponding to the target information type in the initial prompt text and the position information of the target information corresponding to the target information type in the target chart.

[0043] As a specific embodiment, S400 includes:

[0044] S410, if the target information type includes the title information type, the title prompt text and the position information of the target information corresponding to the title information type in the target graph are appended to the preset specified text; if the target information type includes the legend information type, the legend prompt text and the position information of the target information corresponding to the legend information type in the target graph are appended to the preset specified text; if the target information type includes the coordinate axis information type, the coordinate axis prompt text and the position information of the target information corresponding to the coordinate axis information type in the target graph are appended to the preset specified text.

[0045] In this embodiment, the preset specified text is pre-constructed, and optionally includes three parts, wherein the first part is the beginning of the specified text and is fixed; the second part is blank and is used to write the appended content; and the third part is the end of the specified text and is fixed. When writing in the second part, the order of title, legend and coordinate axis can be followed (if the title, legend and coordinate axis related content exist at the same time).

[0046] It should be understood that if the target information type includes the title information type and the legend information type, the title prompt text, the position information of the target information corresponding to the title information type in the target graph, the legend prompt text and the position information of the target information corresponding to the legend information type in the target graph are appended to the preset specified text; if the target information type includes the title information type, the legend information type and the coordinate axis information type, the title prompt text, the position information of the target information corresponding to the title information type in the target graph, the legend prompt text, the position information of the target information corresponding to the legend information type in the target graph, the coordinate axis prompt text and the position information of the target information corresponding to the coordinate axis information type in the target graph are appended to the preset specified text; and so on.

[0047] S420, the specified text after being appended is determined as the target prompt text.

[0048] Based on S410-S420, the target prompt text can represent the extraction demand, output format and position information of the key information related to the question input by the user.

[0049] S500, the target prompt text and the target graph are input into a multi-modal large model, and the key information of the target graph related to the question input by the user is obtained according to the output of the multi-modal large model.

[0050] Those skilled in the art know that any multi-modal large model in the prior art falls within the protection scope of the present application; optionally, the multi-modal large model is Qwen2-VL or OpenLLaMA.

[0051] As a first optional implementation, without new training of the multimodal large model for the chart extraction task, the multimodal large model can directly output according to the target prompt text and the target chart. As a second optional implementation, the multimodal large model is enhanced for training to improve the adaptability of the multimodal large model to the chart extraction task.

[0052] As an optional implementation, the output of the multimodal large model is determined as the key information of the target chart related to the question input by the user.

[0053] As a preferred implementation, the key information of the target chart related to the question input by the user is obtained according to the output of the multimodal large model, including:

[0054] S510, extracting core keywords in the question input by the user to obtain a core keyword set.

[0055] As an implementation, the question input by the user is segmented first, and each word is tagged with a part of speech, and the words of a preset type of part of speech are determined as core keywords; optionally, the preset type of part of speech is noun and verb.

[0056] S520, performing sentence segmentation processing on the output of the multimodal large model to obtain a sentence set.

[0057] S530, obtaining the relevance of each sentence in the sentence set to the core keyword set; wherein the relevance of any sentence in the sentence set to the core keyword set is the number of occurrences of the core keywords in the core keyword set in the sentence.

[0058] In this embodiment, the number of occurrences of the core keywords in the core keyword set in the sentence is the sum of the number of occurrences of each core keyword in the core keyword set in the sentence.

[0059] S540, sorting each sentence in the output of the multimodal large model in order of relevance from high to low.

[0060] In this embodiment, the higher the relevance of a sentence in the output of the multimodal large model to the core keyword set, the higher the relevance of the sentence to the question input by the user, and the higher the probability of the sentence being key information.

[0061] S550, determining the sorted result as the key information of the target chart related to the question input by the user.

[0062] Based on S510-S550, this embodiment realizes the sorting of information in the output of the multimodal large model, so that the information with stronger relevance to the question input by the user in the output of the multimodal large model is presented earlier.

[0063] The initial text of the embodiment includes prompt texts corresponding to the title, legend and coordinate axis, the prompt texts include texts for representing extraction of corresponding content and texts for representing formats that should be met when the corresponding content is output, and the multi-modal large model can realize structured output of the corresponding part of content according to the prompt texts therein; on this basis, the embodiment determines which prompt texts in the initial text as part of the target prompt text according to the type of the target graph and the type of the question input by the user, the part can represent the extraction demand and the output format of the key information with relatively large relevance to the question input by the user, which is conducive to improving the accuracy of the final extraction of key information and the accuracy of the expression of key information of the embodiment; and the target prompt text of the embodiment further includes position information of the target information corresponding to the target information type in the target graph, based on the position information, the multi-modal large model can quickly locate, which is conducive to improving the efficiency of the large model to output information.

[0064] Although some specific embodiments of the present application have been described in detail by examples, those skilled in the art should understand that the above examples are only for illustration, not for limiting the scope of the present application. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.

Claims

1. A chart key information extraction method based on a multi-modal large model, characterized in that, The method comprises the following steps: S100, acquiring initial prompt text; the initial prompt text comprises title prompt text, legend prompt text and coordinate axis prompt text, the title prompt text comprises text for representing extracted titles and text for representing output title information formats, the legend prompt text comprises text for representing extracted legends and text for representing output legend information formats, and the coordinate axis prompt text comprises text for representing extracted coordinate axes and text for representing output coordinate axis information formats; S200, acquiring a target information type according to a type of a target chart and a type of a question input by a user; the target information type comprises at least one of a title information type, a legend information type and a coordinate axis information type; the type of the target chart is one of a column chart type, a line chart type, a pie chart type and a scatter chart type; and the type of the question input by the user is one of a numerical query type, a data point comparison type, a trend summary type, a semantic description type and a category reasoning type; S300, acquiring position information of target information corresponding to the target information type in the target chart; S400, constructing target prompt text according to prompt text corresponding to the target information type in the initial prompt text and the position information of the target information corresponding to the target information type in the target chart; S500, inputting the target prompt text and the target chart into a multi-modal large model, and acquiring key information of the target chart related to the question input by the user according to an output of the multi-modal large model; S200 comprises: S210, Get the information type list C; C={C1,C2,…,C…} h ,…,C H }, C h Let H be the h-th record included in C, where h ranges from 1 to H, and H is the number of records included in C; h =(C h,1 C h,2 C h,3 ), C h,1 C h Includes the following problem types, C h,2 C h Includes various chart types, different C h Includes C h,1 Different or C h,2 Different or C h,1 and C h,2 They are all different, C h,3 C h Including C h,1 and C h,2 A collection of information types with corresponding relationships, C h,3 It must include at least one of the following: title information type, legend information type, and axis information type; S220, matching the type of the target chart and the type of the question input by the user in C; S230, determining an information type included in a matched record as the target information type.

2. The graph key information extraction method based on a multi-modal large model according to claim 1, characterized in that, Acquiring the key information of the target chart related to the question input by the user according to the output of the multi-modal large model comprises: S510, extracting core keywords in the question input by the user to obtain a core keyword set; S520, performing sentence segmentation processing on the output of the multi-modal large model to obtain a sentence set; S530, acquiring a relevance degree of each sentence in the sentence set to the core keyword set; wherein the relevance degree of any sentence in the sentence set to the core keyword set is a number of core keywords in the core keyword set appearing in the sentence; S540, sorting each sentence in the output of the multi-modal large model in a descending order of the relevance degrees; S550, determining a result after the sorting as the key information of the target chart related to the question input by the user. 3.The graph key information extraction method based on a multi-modal large model according to claim 1, wherein, The legend prompt text comprises: identifying a legend type and a corresponding color, and the format is as follows: {type name (color)}|{type name (color)}|…|{type name (color)}|…|{type name (color)}, wherein k is a variable, the value range of k is 1 to p, and p is the number of types in the legend. 4.The graph key information extraction method based on a multi-modal large model according to claim 1, wherein, The coordinate axis prompt text comprises: extracting an x-axis label and a value thereof, in the following format: the x-axis represents: {x-axis label}, and the value is: {value 1}|{value 2}|…|{value f x}|…|{value g x}; and extracting a y-axis label and a value thereof, in the following format: the y-axis represents: {y-axis label}, and the value is: {value 1}|{value 2}|…|{value f y}|…|{value g y}; wherein f x and f y are variables, the value range of f x is 1 to g x , g x is the data quantity corresponding to the x-axis, the value range of f y is 1 to g y , and g y is the data quantity corresponding to the y-axis. 5.The graph key information extraction method based on a multi-modal large model according to claim 1, wherein, S400 comprises: S410, if the target information type includes a title information type, appending the title prompt text and the position information of the target information corresponding to the title information type in the target graph to the preset specified text; if the target information type includes a legend information type, appending the legend prompt text and the position information of the target information corresponding to the legend information type in the target graph to the preset specified text; if the target information type includes a coordinate axis information type, appending the coordinate axis prompt text and the position information of the target information corresponding to the coordinate axis information type in the target graph to the preset specified text; S420, determining the specified text after being appended as the target prompt text.

Images