A chart processing method based on a multi-modal large model

Abstract

The application relates to the technical field of electric digital data processing, in particular to a chart processing method based on a multi-modal large model. The method comprises the following steps: acquiring context key information of a target chart; acquiring a question type of an initial question according to the initial question input by a user; screening information associated with the question type of the initial question and the type of the target chart from the context key information of the target chart according to the question type of the initial question and the type of the target chart; constructing a target question corresponding to the initial question according to the screened information associated with the question type of the initial question and the type of the target chart and the initial question; and inputting the target question and the target chart into a multi-modal large model, and determining an output of the multi-modal large model as an answer corresponding to the initial question. The application can improve the accuracy and efficiency of the output result of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital data processing, in particular to a chart processing method based on a multi-modal large model. BACKGROUND

[0002] The multi-modal chart question answering task is an important research direction in the field of multi-modal artificial intelligence, which is committed to understanding the visual and language fusion of chart data to solve natural language problems. This task includes two categories of objective question answering and subjective question answering. Objective question answering focuses on extracting specific data points, numerical values or classification information, while subjective question answering requires the model to have high-level explanation and reasoning ability, such as trend summary, category analysis and comprehensive expression of chart viewpoints. However, current research mainly focuses on objective tasks, and there is still a lack of comprehensive solutions for subjective question answering tasks, which has the following problems: lacking a context information extraction mechanism for task requirements, resulting in insufficient precision of the context information input to the model, for example, when the context information input to the model includes noise and redundant information, it will interfere with the reasoning process of the model; when the context information input to the model lacks information related to the task, it will increase the complexity of the model reasoning, which will ultimately affect the accuracy and efficiency of the model output results. SUMMARY

[0003] The present application aims to provide a chart processing method based on a multi-modal large model to improve the accuracy and efficiency of the model output results.

[0004] According to the present application, a chart processing method based on a multi-modal large model is provided, which comprises the following steps:

[0005] S100, obtaining the context key information of the target chart; the context key information of the target chart includes the title information, legend information and coordinate axis information of the target chart.

[0006] S200, obtaining the question type of the initial question according to the initial question input by the user.

[0007] S300, screening information associated with the question type of the initial question and the type of the target chart from the context key information of the target chart according to the question type of the initial question and the type of the target chart; the information associated with the question type of the initial question and the type of the target chart at least includes one of the title information, the legend information and the coordinate axis information.

[0008] S400, constructing a target question corresponding to the initial question according to the information associated with the question type of the initial question and the type of the target chart obtained by screening and the initial question;

[0009] S500, input the target question and the target chart into the multi-modal large model, and determine an output of the multi-modal large model as an answer corresponding to the initial question.

[0010] Compared with the prior art, the present application has at least the following beneficial effects:

[0011] The present application first acquires the title information, the legend information and the coordinate axis information of the target chart, filters information associated with the question type of the initial question and the type of the target chart from the information according to the question type of the initial question input by the user and the type of the target chart, and constructs a target question corresponding to the initial question based on the information and the initial question. Compared with the initial question, the target question of the present application further includes the information associated with the question type of the initial question and the type of the target chart filtered out, which can be used to assist the multi-modal large model in quickly understanding the target chart, and the information has filtered out redundant information not associated with the question type of the initial question. Therefore, the multi-modal large model can quickly understand the target chart according to the information attached in the target question, and then quickly and accurately answer the initial question, achieving the purpose of improving the accuracy and efficiency of the output result. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0013] Figure 1 A flowchart of the chart processing method based on the multi-modal large model provided by the embodiments of the present application. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0015] According to the present embodiment, as shown in Figure 1 a chart processing method based on a multi-modal large model is provided, which comprises the following steps:

[0016] S100, acquiring context key information of a target chart; the context key information of the target chart includes title information, legend information and coordinate axis information of the target chart.

[0017] In this embodiment, the target chart includes a title, legend, and axes. The title, legend, and axes are all known information. The title, as the subject of the chart, provides the model with overall contextual semantics, such as the source of the data, the timeframe, or the direction of analysis. The legend contains the correspondence between categories and colors and is an important source of information for classification reasoning tasks. For example, the legend can explain the specific category or grouping represented by a certain color. The axes provide the units and ranges of the data. For example, the x-axis represents years and the y-axis represents percentage changes, which can directly guide numerical queries or trend descriptions.

[0018] S200: Obtain the question type of the initial question according to the initial question input by the user.

[0019] As an optional specific implementation manner, the question type of the initial question is one of a numerical query type, a data point comparison type, a trend summary type, a semantic description type and a category reasoning type.

[0020] As a preferred embodiment, S200 includes:

[0021] S210, obtain the question type field list A; A={A1, A2, ..., A i ,…,A n}, A i is the i-th record included in A, the value range of i is 1 to n, and n is the number of preset question types; A i =(T i ,(d i,1 ,d i,2 ,…,d i,r(i) ,…,d i,R(i) )), T i A i Types of questions included, d i,r(i) A i The r(i)th field is included, and the value range of r(i) is 1 to R(i), and R(i) is A i The number of fields included, either d i,r(i) The question types are all T i ; Different A i Included T i Different, (d i,1 ,d i,2 ,…,d i,r(i) ,…,d i,R(i) ) has a higher priority than the field with the position number in the front. i Fields with the same included position number have the same priority.

[0022] In this embodiment, A is a pre-constructed list, and each T i The corresponding field is known, for example, when T i is a trend summary type, T i The corresponding field includes trend summary, trend analysis, trend summary, trend description, trend summary, trend analysis, trend summary, and trend description.

[0023] In this embodiment, the priority of different fields corresponding to the same problem type is related to the number of times the corresponding field is used by the user to ask questions of the corresponding problem type in the historical time period. The more times a certain field is used by the user to ask questions of a certain problem type, the higher the position sequence number of the field in the record to which the problem type belongs. It should be understood that d i,r(i) The position sequence number in A i is r(i), for example, d i,1 The position sequence number in A i is 1, d i,1 is the field with the highest position sequence number in A i .

[0024] S220, according to the priority of the field from high to low, the field included in A is matched with the initial question until the matched field is obtained.

[0025] In this embodiment, the matching of the field and the initial question is performed according to the priority of the field from high to low. Since the field with higher priority corresponds to a field with higher probability of occurrence, matching the field and the initial question according to the priority of the field from high to low can improve the efficiency of obtaining the matched field.

[0026] In this embodiment, the priority of the fields included in different A i is the same, and the process of matching the fields included in A with the initial question according to the priority of the field from high to low can be performed in the order of first matching the field with the highest priority included in A1, then the field with the highest priority included in A2, and finally the field with the highest priority included in A n , and if all matching fails, the order of first matching the field with the second highest priority included in A1, then the field with the second highest priority included in A2, and finally the field with the second highest priority included in A n , and if all matching fails, the order of first matching the field with the second highest priority included in A1, then the field with the second highest priority included in A2, and finally the field with the second highest priority included in A

[0027] S230, the problem type of the record to which the matched field belongs is determined as the problem type of the initial question.

[0028] Based on S210-S230, the type of the initial question can be quickly and accurately obtained.

[0029] S300, according to the type of the initial question and the type of the target chart, filtering information associated with the type of the initial question and the type of the target chart from the context key information of the target chart; the information associated with the type of the initial question and the type of the target chart at least includes one of title information, legend information and coordinate axis information.

[0030] As a preferred embodiment, S300 includes:

[0031] S310, obtaining a question type related information list B; B={B1, B2,…, B j ,…,B m}, B j is the jth record included in B, j ranges from 1 to m, m is the number of records included in B; B j =(T j ,U j ,E j ), T j is the question type included in B j , U j is the chart type included in B j , different B j includes different T j or different U j or different T j and U j , E j is the set of information types corresponding to T j and U j included in B j , E j at least includes one of title information type, legend information type and coordinate axis information type.

[0032] In this embodiment, B is a pre-constructed list, each T j and U j corresponding E j is known, and E j can be determined according to experience. It should be understood that if E j only includes title information type and coordinate axis information type, it means that the multi-modal large model needs to pay attention to the title information and coordinate axis information when answering the question of type T j about the chart of type U j ; if E j only includes legend information type and coordinate axis information type, it means that the multi-modal large model needs to pay attention to the legend information and coordinate axis information when answering the question of type T j about the chart of type U j .j the type of the chart is T j the legend information and the axis information need to be paid attention to preferentially; if E j the title information type, the legend information and the axis information type are included in E j the type of the chart is T j the title information, the legend information and the axis information need to be paid attention to simultaneously.

[0033] As a specific embodiment, the question type includes a numerical query type, a data point comparison type, a trend summary type, a semantic description type and a category inference type.

[0034] S320, match the question type of the initial question and the type of the target chart with the question types and chart types included in B.

[0035] As a specific embodiment, match the question type of the initial question and the type of the target chart with B1, B2, …, B n included in B in turn, until a record included in B includes the question type and the chart type which are the same as the question type of the initial question and the type of the target chart respectively, and the record is the matched record.

[0036] S330, determine the set of information types included in the matched record as a target set.

[0037] S340, if the target set includes the title information type, determine the title information in the context key information of the target chart as the information associated with the question type of the initial question and the type of the target chart; if the target set includes the legend information type, determine the legend information in the context key information of the target chart as the information associated with the question type of the initial question and the type of the target chart; if the target set includes the axis information type, determine the axis information in the context key information of the target chart as the information associated with the question type of the initial question and the type of the target chart.

[0038] In this embodiment, all the information associated with the question type of the initial question and the type of the target chart is taken as the information associated with the question type of the initial question and the type of the target chart obtained by screening.

[0039] Based on S310-S340, the information associated with the question type of the initial question and the type of the target chart obtained by screening is more relevant to the initial question, and can be used to assist the multimodal large model to quickly understand the target chart, and the information excludes the redundant information which is not associated with the question type of the initial question.

[0040] S400, constructing a target question corresponding to the initial question according to the information associated with the type of the initial question and the type of the target chart and the initial question.

[0041] As a preferred embodiment, S400 comprises:

[0042] S410, obtaining the type of the target chart.

[0043] Optionally, the type of the target chart is one of a column chart type, a line chart type, a pie chart type, and a scatter chart type.

[0044] S420, obtaining a matched template text according to the type of the target chart and the type of the initial question; the matched template text includes a plurality of to-be-embedded sub-areas, and the plurality of to-be-embedded sub-areas include a sub-area for embedding the initial question and a sub-area for embedding the information associated with the type of the initial question and the type of the target chart.

[0045] In this embodiment, the template text is pre-constructed, and a plurality of template texts are pre-constructed, and one matched template text can be uniquely determined according to the type of the target chart and the type of the initial question. If the type of the initial question corresponds to E j If the type of the initial question corresponds to E j If the type of the initial question corresponds to E

[0046] S430, embedding the information associated with the type of the initial question and the type of the target chart and the initial question into the matched template text, and determining the embedded result as the target question corresponding to the initial question.

[0047] In this embodiment, it is known that each to-be-embedded sub-area in the matched template corresponds to which information or initial question, and embedding is performed according to the corresponding relationship in the embedding process. For example, a to-be-embedded sub-area in the matched template corresponds to the title information, and then the title information of the target chart is embedded into the to-be-embedded sub-area in the embedding process.

[0048] S500, inputting the target question and the target chart into a multi-modal large model, and determining the output of the multi-modal large model as the answer corresponding to the initial question.

[0049] In this embodiment, the prompt of the whole of the target question and the target chart constitutes a multimodal large model.

[0050] Those skilled in the art know that any multimodal large model in the prior art falls within the protection scope of the present application; optionally, the multimodal large model is Qwen2-VL or InternLM-XComposer-2.5.

[0051] As a first optional specific implementation, the multimodal large model can directly generate an answer according to the target question and the target chart without new training of the multimodal large model for the chart question answering task. As a second optional specific implementation, the multimodal large model is enhanced trained to improve the adaptability of the multimodal large model to the chart question answering task.

[0052] In this embodiment, the title information, the legend information and the coordinate axis information of the target chart are first acquired, the information associated with the question type of the initial question and the type of the target chart is filtered from the information according to the question type of the initial question and the type of the target chart input by the user, and the target question corresponding to the initial question is constructed based on the information and the initial question. Compared with the initial question, the target question of this embodiment further includes the information associated with the question type of the initial question and the type of the target chart filtered, which can be used to assist the multimodal large model to quickly understand the target chart, and the information in which the redundant information not associated with the question type of the initial question is filtered out. Therefore, the multimodal large model can quickly understand the target chart according to the information attached in the target question, and then quickly and accurately make an answer to the initial question, so as to achieve the purpose of improving the accuracy and efficiency of the output result.

[0053] Although some specific embodiments of the present application have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration, but not for limiting the scope of the present application. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.

Claims

1. A chart processing method based on a multi-modal large model, characterized in that, The method comprises the following steps: S100, obtaining context key information of a target graph; the context key information of the target graph comprises title information, legend information and coordinate axis information of the target graph; S200, obtaining a question type of an initial question according to the initial question input by a user; S300, screening information associated with the question type of the initial question and the type of the target graph from the context key information of the target graph according to the question type of the initial question and the type of the target graph; the information associated with the question type of the initial question and the type of the target graph at least comprises one of the title information, the legend information and the coordinate axis information; S400, constructing a target question corresponding to the initial question according to the information associated with the question type of the initial question and the type of the target graph obtained by screening and the initial question; S500, inputting the target question and the target graph into a multi-modal large model, and determining an output of the multi-modal large model as an answer corresponding to the initial question; the whole formed by the target question and the target graph is a prompt of the multi-modal large model; S300 comprises: S310, acquire a question type related information list B; B={B1, B2,..., B j ,…,B m} B j is the jth record included in B, the value range of j is 1 to m, m is the number of records included in B; B j =(T j ,U j ,E j ), T j is the question type included in B j , U j is the chart type included in B j , T j included in different B j is different or U j is different or T j and U j are all different, E j is the set of information types included in B j which has a corresponding relationship with T j and U j , E j at least includes one of title information type, legend information type and coordinate axis information type; S320, matching the question type of the initial question and the type of the target graph with question types and graph types included in B; S330, determining a set of information types included in the matched record as a target set; S340, if the target set comprises a title information type, determining title information in the context key information of the target graph as the information associated with the question type of the initial question and the type of the target graph; if the target set comprises a legend information type, determining legend information in the context key information of the target graph as the information associated with the question type of the initial question and the type of the target graph; if the target set comprises a coordinate axis information type, determining coordinate axis information in the context key information of the target graph as the information associated with the question type of the initial question and the type of the target graph. 2.The graph processing method based on a multi-modal large model according to claim 1, wherein, S200 comprises: S210, obtain the question type field list A; A={A1,A2,…,A i ,…,A n }, A i is the i-th record included in A, the value range of i is 1 to n, and n is the number of preset question types; A i =(T i ,(d i,1 ,d i,2 ,…,d i,r(i) ,…,d i,R(i) )), T i A i Types of questions included, d i,r(i) A i The r(i)th field is included, and the value range of r(i) is 1 to R(i), and R(i) is A i The number of fields included, either d i,r(i) The question types are all T i ; Different A i Included T i Different, (d i,1 ,d i,2 ,…,d i,r(i) ,…,d i,R(i) ) has a higher priority than the field with the position number in the front. i Fields with the same position number have the same priority; S220, matching the fields included in A with the initial question in order from high to low according to the priority of the fields until a matched field is obtained; S230, determining a question type included in the record to which the matched field belongs as the question type of the initial question. 3.The graph processing method based on a multi-modal large model according to claim 1, wherein, S400 comprises: S410, obtaining the type of the target graph; S420, obtaining a matched template text according to the type of the target graph and the question type of the initial question; the matched template text comprises a plurality of to-be-embedded sub-areas, and the plurality of to-be-embedded sub-areas comprise a sub-area for embedding the initial question and a sub-area for embedding the information associated with the question type of the initial question and the type of the target graph obtained by screening; S430, embedding the information associated with the question type of the initial question and the type of the target graph obtained by screening and the initial question into the matched template text, and determining the embedded result as the target question corresponding to the initial question. 4.The graph processing method based on a multi-modal large model according to claim 1, wherein, The multi-modal large model is Qwen2-VL. 5.The graph processing method based on a multi-modal large model according to claim 1, wherein, The multi-modal large model is InternLM-XComposer-2.

5. 6.The graph processing method based on a multi-modal large model according to claim 1, wherein, The type of the target chart is one of a column chart type, a line chart type, a pie chart type and a scatter chart type. 7.The graph processing method based on a multi-modal large model according to claim 1, wherein, The type of the initial question is one of a numerical query type, a data point comparison type, a trend summary type, a semantic description type and a category inference type.

Images