Query response method, device and equipment

Abstract

The invention provides an inquiry response method, device and equipment. The method is applied to the terminal equipment, and comprises the following steps: acquiring a first image and a first instruction; wherein the content of the first image comprises chart information, and the first instruction at least comprises a first question inquiry aiming at the first image and / or a first modification instruction aiming at the first image; identifying position information of a first element and position information of a second element in the first image to obtain an identification result; converting the first image into first structured data according to the recognition result, and determining first response information for the first instruction according to the first structured data and the first instruction; and responding to the first instruction according to the first response information.

Description

Technical Field

[0001] The present application relates to electronic technology, and is related to but not limited to inquiry and response methods and devices, equipment, and storage media. Background Art

[0002] Digital charts (such as mind maps, bar charts, pie charts, etc.) play an increasingly important role in daily life and office work. However, due to the small display screen on the terminal device, it is very inconvenient for users to read the icons or manually modify or edit the charts. Summary of the invention

[0003] In a first aspect, an embodiment of the present application provides an inquiry and response method, the method comprising: acquiring a first image and a first instruction; wherein the content of the first image comprises chart information, and the first instruction comprises a first question inquiry for the first image and / or a first modification instruction for the first image; identifying position information of a first element and position information of a second element in the first image to obtain an identification result; converting the first image into first structured data according to the identification result, and determining first response information for the first instruction according to the first structured data and the first instruction; and responding to the first instruction according to the first response information.

[0004] In a second aspect, an embodiment of the present application provides an inquiry and response device, the device comprising: an acquisition module, configured to acquire a first image and a first instruction; wherein the content of the first image includes chart information, and the first instruction includes a first question inquiry for the first image and / or a first modification instruction for the first image; a first recognition module, configured to identify position information of a first element and position information of a second element in the first image to obtain a recognition result; a conversion module, configured to convert the first image into first structured data according to the recognition result; a second recognition module, configured to determine first response information for the first instruction according to the first structured data and the first instruction; and a response module, configured to respond to the first instruction according to the first response information.

[0005] In a third aspect, an embodiment of the present application provides a terminal device, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the program, the method described in the first aspect is implemented.

[0006] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the method described in the first aspect when the computer program is executed by a processor.

[0007] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the method described in the first aspect.

[0008] In a sixth aspect, an embodiment of the present application provides a computer program, which enables a terminal device to execute the method described in the first aspect.

[0009] In an embodiment of the present application, a two-stage method is adopted to realize the response to the first instruction (such as the response to the question inquiry of the chart-type image and / or the modification instruction of the chart-type image, etc.); that is, in the first stage, the position information of the first element and the position information of the second element in the input chart-type image (i.e., the first image) are firstly identified to obtain the recognition result; in the second stage, the chart-type image is converted into the first structured data according to the recognition result, and the first response information for the first instruction is determined according to the data and the first instruction input by the user, and the first instruction input by the user is responded to according to the response information; in this way, the user can directly ask questions or modify the original chart content by inputting the first instruction, which is beneficial for the user to conveniently realize the reading or modification / editing of the chart-type image through the terminal device.

[0010] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0012] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0013] Figure 1 A schematic diagram of the implementation flow of the inquiry response method provided in the embodiment of the present application;

[0014] Figure 2 An example diagram of a mind map provided in an embodiment of the present application;

[0015] Figure 3 An example diagram of a bar graph provided in an embodiment of the present application;

[0016] Figure 4 An example of a pie chart provided in an embodiment of the present application;

[0017] Figure 5 A schematic diagram of a mind map provided in an embodiment of the present application;

[0018] Figure 6 A schematic diagram of a bar graph provided in an embodiment of the present application;

[0019] Figure 7 A schematic diagram of a pie chart provided in an embodiment of the present application;

[0020] Figure 8 A schematic diagram of the processing process of an input image using a hybrid network structure based on CNN and Transformer provided in an embodiment of the present application;

[0021] Fig. 9 A diagram showing how to render a bar chart as a pie chart;

[0022] Fig.10 A schematic diagram of the structure of the inquiry response device provided in an embodiment of the present application;

[0023] Fig.11 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0026] In the following description, reference is made to “some embodiments”, “this embodiment”, “embodiments of the present application” and examples, etc., which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0027] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies or terms of the embodiments of the present application. The following related technologies or related terms can be combined arbitrarily with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0028] (1) Bounding Box (bbox)

[0029] Bounding box plays a vital role in the field of computer vision and image processing. A bounding box is a rectangular box that describes the location and range of key features in an image. It can accurately locate key features in an image and provide a basis for subsequent image processing tasks.

[0030] There are three ways to represent the bounding box:

[0031] (1.1) Use the coordinates of the upper left corner and the lower right corner: that is, (xl, yl, xr, yr), where (xl, yl) is the coordinate of the upper left corner of the bounding box, and (xr, yr) is the coordinate of the lower right corner of the bounding box. This method intuitively describes the location information of the bounding box.

[0032] (1.2) Use the coordinates of the center point and the length and width of the bounding box: that is, (x, y, w, h), where (x, y) is the coordinates of the center point of the bounding box, and w and h are the width and height of the bounding box, respectively. This method is more concise when describing the bounding box and is easier to calculate and process.

[0033] (1.3) Use the coordinates of the upper left corner and the length and width of the bounding box: (xl, yl, w, h), where (xl, yl) is the coordinate of the upper left corner of the bounding box, and w and h are the width and height of the bounding box, respectively. This method is more concise in describing the bounding box and is easier to calculate and process.

[0034] (2) Hybrid network structure of CNN and Transformer

[0035] The hybrid network structure of Convolutional Neural Network (CNN) and Transformer is a deep learning architecture that combines the advantages of CNN in local feature extraction and the ability of Transformer in global context modeling, thereby enhancing the model's understanding of data.

[0036] CNN can efficiently extract local features of images or sequence data through convolution kernels and pooling layers. In addition, the parameter sharing and sparse connection characteristics of CNN reduce the number of model parameters and reduce computational costs.

[0037] Through the self-attention mechanism, Transformer can capture the dependencies between all positions in the input sequence, thereby better understanding the global context. And the parallel processing capability of Transformer makes it particularly efficient when processing long sequences.

[0038] The hybrid network structure includes a CNN encoder and a Transformer decoder; wherein, in some embodiments, the CNN encoder part adopts a convolutional neural network structure, including convolution layers, pooling layers, and activation functions. The role of the CNN encoder is to convert the input image or feature map into a more compact feature representation. By stacking multiple convolution layers and pooling layers, higher-level local features are gradually extracted. In some embodiments, the Transformer decoder part adopts the Transformer architecture, including modules such as a multi-head attention mechanism and a feedforward neural network. The role of the Transformer decoder is to use the features extracted by the CNN encoder to generate a target output sequence. The long-distance dependencies between the input features are captured through the multi-head attention mechanism, and the final output is generated in combination with modules such as a feedforward neural network.

[0039] (3) Large Language Model (LLM)

[0040] LLM is a language model constructed by a deep neural network with more than tens of billions of parameters, which is trained using a large amount of unlabeled text through self-supervised learning methods. Its core idea is to learn the patterns and structures of natural language through large-scale unsupervised training, and to simulate the human language cognition and generation process to a certain extent.

[0041] An inquiry response method provided in an embodiment of the present application, Figure 1 A schematic diagram of the implementation flow of the inquiry response method provided in the embodiment of the present application; Figure 1 As shown, the method may include the following steps 101 to 105:

[0042] Step 101: A terminal device acquires a first image and a first instruction; wherein the content of the first image includes chart information, and the first instruction includes a first question inquiry for the first image and / or a first modification instruction for the first image;

[0043] Step 102, the terminal device identifies the position information of the first element and the position information of the second element in the first image to obtain a recognition result;

[0044] Step 103: The terminal device converts the first image into first structured data according to the recognition result;

[0045] Step 104: The terminal device determines first response information for the first instruction according to the first structured data and the first instruction;

[0046] Step 105: The terminal device responds to the first instruction according to the first response information.

[0047] In an embodiment of the present application, a two-stage method is adopted to realize the response to the first instruction (such as the response to the question inquiry of the chart-type image and / or the modification instruction of the chart-type image, etc.); that is, in the first stage, the position information of the first element and the position information of the second element in the input chart-type image (i.e., the first image) are firstly identified to obtain the recognition result; in the second stage, the chart-type image is converted into the first structured data according to the recognition result, and the first response information for the first instruction is determined according to the data and the first instruction input by the user, and the first instruction input by the user is responded to according to the response information; in this way, the user can directly ask questions or modify the original chart content by inputting the first instruction, which is beneficial for the user to conveniently realize the reading or modification / editing of the chart-type image through the terminal device.

[0048] In the embodiment of the present application, there is no restriction on the type of terminal device, and the terminal device can be various devices with computing capabilities. For example, the terminal device can be a handheld terminal (such as a smart phone, a tablet computer, etc.), a laptop computer, etc.

[0049] The following describes further optional implementations and related terms of each of the above steps.

[0050] Step 101: A terminal device obtains a first image and a first instruction; wherein the content of the first image includes chart information, and the first instruction includes a first question inquiry for the first image and / or a first modification instruction for the first image.

[0051] It can be understood that the first image and the first instruction are information input by the user to the terminal device. The content of the first image includes chart information. Exemplarily, in some embodiments, the first image is a chart-type image. For example, the first image includes a mind map, a bar chart, or a pie chart.

[0052] It can be understood that the first question inquiry is used to inquire / ask questions about the relevant content in the first image. The first question inquiry includes but is not limited to one or more of the following: instructing to view the information in the first image, instructing to count the information in the first image, instructing to search for information in the first image, etc. For example, the first image is a bar chart related to mobile phone sales in recent months, and the user asks the terminal device "What is the total sales of mobile phones in a certain quarter" (i.e., an example of the first question inquiry). The first modification instruction is different from the first question inquiry in that the first modification instruction is used to instruct to modify or edit the relevant content / related information in the first image. Therefore, the first modification instruction can also be understood as an editing instruction, which includes but is not limited to one or more of the following: instructing to modify a certain information or some information in the first image, instructing to replace a certain information or some information in the first image, and instructing to add new information. For example, the user inquires the terminal device "change the first column header to AA" (i.e., an example of the first modification instruction).

[0053] Step 102: The terminal device identifies the position information of the first element and the position information of the second element in the first image to obtain a recognition result.

[0054] In some embodiments, the second element is an element related to the first element, that is, the second element is an element related to the first element.

[0055] It can be understood that the first images of different chart types include different first elements. The first elements and second elements of a mind map, a bar chart, and a pie chart are listed below respectively.

[0056] In some embodiments, the first image includes the mind map, the first element in the mind map includes one or more of a root node, an intermediate node, a cluster and a leaf node; wherein the cluster includes one or more leaf nodes having the same parent node; and the second element includes the parent node of the first element.

[0057] For example, Figure 2 As shown, it is a first image 200 of a mind map, and its first elements include: a root node 201, an intermediate node 202, a leaf node 203 and a cluster 204. It should be noted that for the sake of simplicity, Figure 2 Only some leaf nodes and clusters are schematically marked with numbers. For some leaf nodes and clusters that are not marked with numbers, they can be understood by referring to the leaf nodes and clusters indicated by the numbers.

[0058] In other embodiments, the first image includes the bar chart, and the first elements in the bar chart include one or more of the x-axis coordinate title, x-axis coordinate value, y-axis coordinate title, y-axis coordinate value, the bounding box of the column, the overall title of the bar chart, a legend, and other text areas that do not belong to the above-mentioned first elements; the second element of the bounding box of the column includes one or more of other text areas associated with the column, the legend associated with the column, and the x-axis coordinate value of the column.

[0059] For example, Figure 3 As shown, it is a first image 300 of a bar graph, and its first elements include: x-axis coordinate title 301, x-axis coordinate value 302, y-axis coordinate title 303, y-axis coordinate value 304, columnar boundary box 305, the total title of the bar graph 306, legend 307 and other text areas 308 that do not belong to the above first elements. Figure 3 Taking the bounding box 305 of the column shown in as an example, the second elements of the column include: other text areas "22% 3K" associated with the column, the legend 'BB associated with the column, and the x-axis coordinate value "Jan" of the column.

[0060] It should be noted that for the sake of brevity, Figure 3 Only some x-axis coordinate values, y-axis coordinate values, cylindrical boundary boxes, legends and other text areas that do not belong to the above-mentioned first element are schematically marked with labels. For similar content not marked with labels, you can refer to the content indicated by the labels to understand it.

[0061] In some further embodiments, the first image includes the pie chart, and the first elements in the pie chart include: the center position of a circular area, the intersection of any two sectors, a legend, and one or more of other text areas that do not belong to the above-mentioned first elements; the second elements of the legend include other text areas associated with the legend and / or the center point coordinates of the sectors associated with the legend.

[0062] For example, Figure 4 As shown in FIG. 4 , a first image 400 of a pie chart is shown, wherein the first elements include: the center position 401 of the circular area, the intersection 402 of any two sectors, the legend 403, and other text areas 404 that do not belong to the first elements. Figure 4 Taking the legend 403 shown in FIG. 4 as an example, the second element of the legend 403 includes: other text areas 404 associated with the legend 403 and the center point coordinates of the sector 405 associated with the legend 403 .

[0063] It should be noted that for the sake of brevity, Figure 4The intersection points of some sectors, legends and other text areas that do not belong to the above-mentioned first element are only schematically marked with labels. For similar contents not marked with labels, they can be understood by referring to the contents indicated by the labels.

[0064] Step 102: The terminal device identifies the position information of the first element and the position information of the second element in the first image to obtain a recognition result.

[0065] In some embodiments, step 102 may further include: the terminal device extracts a model of pre-trained chart information, identifies the position information of the first element and the position information of the second element in the first image, and obtains a recognition result.

[0066] In an embodiment of the present application, the chart information extraction models corresponding to different types of charts may be the same AI model (ie, an AI model with the same structure and the same values ​​of model parameters).

[0067] In an embodiment of the present application, the chart information extraction models corresponding to different types of charts may also be different AI models (i.e., AI models with the same structure but different values ​​of model parameters). The chart information extraction models corresponding to different types of charts are different AI models. For example, the chart information extraction model corresponding to a mind map is model M1, the chart information extraction model corresponding to a bar chart is model M2, and the chart information extraction model corresponding to a pie chart is model M3. Based on this, in some embodiments, the chart information extraction model used by the terminal device in step 102 is an AI model corresponding to the chart type of the first image.

[0068] In some embodiments, the pre-trained chart information extraction model is a hybrid network structure based on CNN and Transformer; step 102 may further include: inputting the first image into the pre-trained chart information extraction model to obtain the recognition result.

[0069] In some embodiments, the recognition result includes at least one of: the element category to which the first element belongs, a bounding box (bbox) of the first element, and the center point coordinates of the second element.

[0070] In some embodiments, the chart information extraction model is trained based on a pre-constructed first data set, and the pre-constructed first data set includes: multiple second images, annotation information of the first element in the second image, and ID information of the second annotation information in the second image; wherein the second image is a chart-type image. Further, in some embodiments, the multiple second images are images of the same chart type, for example, the multiple second images are all mind maps, or the multiple second images are all bar charts, or the multiple second images are all pie charts.

[0071] It can be understood that the multiple second images are sample data, and the annotation information of the first element in the second image and the ID information of the annotation information of the second element in the second image can be understood as the label information of the second image. In some embodiments, for a pie chart, the label information in the first data set also includes the coordinates of the center point of the sector where the legend is located.

[0072] Step 103: The terminal device converts the first image into first structured data according to the recognition result.

[0073] In some embodiments, step 103 may further include: acquiring first content included in the first element in the first image according to a bounding box of the first element; and arranging the first content according to a structure corresponding to the first structured data to obtain the first structured data.

[0074] In a possible implementation, obtaining the first content contained in the first element in the first image according to the bounding box of the first element includes: intercepting the image area of ​​the bounding box of the first element from the first image to obtain the image of the first element; and inputting the image of the first element into an OCR algorithm to obtain the first content contained in the first element. For example, the first content is text information contained in the image of the first element.

[0075] It can be understood that, if the position information of the first element and the position information of the second element in the first image (that is, the relationship between the elements) are known, it is easy to convert the information / content in the first image into the first structured data. For example, the first structured data is data in Jsonl format or Markdown table (that is, table data in Markdown format).

[0076] For bar charts and pie charts, they can be converted into Markdown table data. In a possible implementation, the first image includes the bar chart, and the first structured data is a Markdown table, wherein the legend in the bar chart is used as the column name of the table header of the Markdown table; the x-axis coordinate value is used as the row name of the Markdown table; and the y-axis coordinate value is filled in the corresponding table as the table value of the Markdown table.

[0077] In a possible implementation, the first image includes the pie chart, and the first structured data is a Markdown table; wherein the legend in the pie chart serves as the column name of the table header of the Markdown table; and the values ​​contained in the sectors in the pie chart are filled in the corresponding table as the table values ​​of the Markdown table.

[0078] In a possible implementation manner, the first image includes the mind map, and the first structured data is data in a Jsonl format.

[0079] Step 104: The terminal device determines first response information for the first instruction according to the first structured data and the first instruction.

[0080] In some embodiments, step 104 may further include: inputting the first structured data and the first instruction into a pre-trained large language model to obtain first response information for the first instruction.

[0081] In some embodiments, the pre-trained large language model may be an LLM, which is a 3B model and has a decoder-only structure.

[0082] In some embodiments, the pre-trained large language model is trained based on a pre-constructed second data set, and the pre-constructed second data set includes: third structured data corresponding to a plurality of third images, a second question query for the third structured data and response information corresponding to the second question query, a second modification instruction for the third structured data, and fourth structured data corresponding to the second modification instruction, wherein the fourth structured data is obtained by modifying the corresponding third structured data according to the second modification instruction; wherein the third image includes chart information. For example, the third image is a chart-type image.

[0083] It can be understood that the third structured data corresponding to the multiple third images, the second question inquiry for the third structured data, and the second modification instruction for the third structured data are sample data, and the response information corresponding to the second question inquiry and the fourth structured data corresponding to the second modification instruction are label information.

[0084] Step 105: The terminal device responds to the first instruction according to the first response information.

[0085] As mentioned above, the first instruction input by the user to the terminal device may include a question for asking / inquiring about the relevant content in the first image (i.e., question inquiry), and may also include an instruction for modifying or editing the relevant content in the first image (i.e., modification instruction). In a possible implementation, the terminal device may respond in different ways according to different types of inquiry / instruction content.

[0086] For example, in some embodiments, the first instruction includes a first question about the first image, and the first response information includes an answer to the first question; step 105 may further include: outputting the first response information on a user interface, or broadcasting the first response information by voice.

[0087] For example, in some other embodiments, the first instruction includes a first modification instruction for the first image, and the first response information includes second structured data obtained by modifying the first structured data according to the first modification instruction; step 105 may further include: rendering and outputting the second structured data according to the chart type of the first image.

[0088] It is understandable that if the first modification instruction does not indicate to convert the first image into another type of chart, the second structured data is rendered and outputted by default according to the chart type of the first image. That is, the terminal device finally outputs an image or chart of the same chart type as the first image.

[0089] For example, in some other embodiments, the first instruction includes a first modification instruction for the first image, and the first response information includes the second structured data and the chart type to be output after modification required by the first modification instruction; step 105 may further include: rendering and outputting the second structured data according to the chart type to be output after modification required by the first modification instruction.

[0090] It is understandable that if the first modification instruction indicates the type of chart to be output after modification, the second structured data is rendered and output according to the type of chart to be output after modification required by the first modification instruction. That is, the terminal device finally outputs an image of the chart type indicated by the first modification instruction.

[0091] In some embodiments, the image rendered and output by the terminal device for the second structured data is editable, that is, the result of the terminal device rendering and outputting the second structured data is in an editable mode, and the terminal device supports editing or modifying the first element in the rendered and output chart-type image.

[0092] The following examples describe possible implementations of the query response method described in one or more of the above embodiments.

[0093] It is understandable that digital charts (such as mind maps, bar charts, pie charts, flow charts, etc.) play an increasingly important role in daily life and office work. On the mobile phone side, due to the small display screen, it is quite inconvenient to modify and read the chart. In view of this problem, a method for intelligent editing and intelligent question and answer of charts on a mobile phone side (i.e., an implementation method of the inquiry and answering method) is provided in an embodiment of the present application. The main features of this method include:

[0094] 1) To address the problem of weak ability to understand images and texts in combination based on multimodal models, the present application embodiment provides a two-stage method. First, a small model on the end is used to extract information from different types of charts and convert them into Markdown format. Then, LLM is used for intelligent editing and question-answering.

[0095] 2) Considering the requirements of low power consumption and low latency on mobile phones, the chart information extraction models all use lightweight construction technology, and LLM uses the 3B model to ensure that the overall solution can be executed on the terminal side.

[0096] The implementation steps of this method are as follows.

[0097] S1: Data collection and annotation.

[0098] The chart types involved in the embodiments of the present application include: mind maps, pie charts and bar charts. All data are annotated in the form of bbox. Bbox is represented by a mind vector: [x, y, w, h], where x and y represent the coordinates of the center point of the bbox; w represents the width; and h represents the height. The collection and annotation methods of these three types of data are as follows.

[0099] For mind maps, a total of 5,000 original images were collected. The images can be from public data on the Internet or photographed data. The annotation rules for the key elements (i.e. the first element) in the mind map are as follows (e.g. Figure 5 The dashed box shown):

[0100] 1)root_node: root node. The root node represents the only main node of the mind map, and the root node has no parent node.

[0101] 2) node: intermediate node. An intermediate node is defined as a node that is neither a root node nor a leaf node.

[0102] 3)leaf_node: leaf node. A leaf node does not contain any child nodes.

[0103] 4)cluster_node: cluster. A cluster consists of leaf nodes with the same parent node.

[0104] When labeling each node, you need to label their unique ID (numbering starts from 1; IDs are not repeated; the order can be arbitrary) and their parent node ID. For the root node, since it has no parent node, its parent node ID is labeled as None. The dataset constructed in this way is called D1.

[0105] A total of 5000 original images were collected for the bar graph. The images can be from public data on the Internet or photographic data. The annotation rules for the key elements (i.e. the first element) in the bar graph are as follows (e.g. Figure 6 The dashed box shown):

[0106] 1)x_axis_title: x-axis coordinate title.

[0107] 2)x_axis_value: x-axis coordinate value.

[0108] 3)y_axis_title: y-axis coordinate title.

[0109] 4)y_axis_value: y-axis coordinate value.

[0110] 5)bar: the bbox of each column.

[0111] 6)title: The overall title of the chart.

[0112] 7)legend: legend.

[0113] 8)text: Other text areas that do not belong to the above categories.

[0114] When labeling each node, you need to label their unique ID (numbering starts from 1; IDs are not repeated; the order can be arbitrary). For the bar type, in addition to labeling its bbox area, you also need to label the text (text), legend (legend), and x-axis coordinate value (x_axis_value) associated with it. The dataset constructed in this way is called D2.

[0115] A total of 5,000 original images were collected for the pie chart. The images can be from public data on the Internet or photographic data. The annotation rules for the key elements (i.e., the first element) in the pie chart are as follows (e.g. Figure 7 The dashed box shown):

[0116] 1)root_point: the center position of the sector.

[0117] 2)key_point: the intersection of any two sectors.

[0118] 3)legend: legend.

[0119] 4)text: other text areas that do not belong to the above categories.

[0120] When labeling each node, you need to label their unique ID (numbering starts from 1; IDs are not repeated; the order can be arbitrary). When labeling the legend, in addition to labeling its bbox area, you also need to label the text ID associated with it and the center point coordinates of the sector corresponding to the legend. The dataset constructed in this way is called D3.

[0121] In addition to the collection and annotation of the above three types of chart data, some Markdown table data needs to be annotated for training LLM. The annotation method of Markdown data is as follows:

[0122] 1) Manually transcribe image tables into Markdown format;

[0123] 2) For each Markdown table, manually write 10 edit requests (i.e., sample data, such as changing the first column header to "Amount") and manually write the corresponding Markdown table (i.e., label information);

[0124] 3) For each Markdown-formatted table, manually write 10 questions (i.e., sample data, such as: What is the total sales volume of mobile phones in the first half of 2024), and manually write the answers to the corresponding questions (i.e., label information).

[0125] The dataset thus constructed is called D4.

[0126] S2: Construction of the graph information extraction model on the client side

[0127] In order to improve the efficiency of the model on the end side, the embodiment of the present application provides a new end-to-end information extraction method for a graph. The design goal of the model is to identify all the key elements in the graph at one time and simultaneously identify the relationship between the key elements.

[0128] Each key element (ie, the first element) is represented as a four-dimensional vector [x, y, w, h], where x and y represent the coordinates of the center point of the key element; w represents the width; and h represents the height.

[0129] If there is a certain relationship between key element A1 and key element A2, their relationship is expressed as [x1, y1, x2, y2], where x1 and y1 represent the coordinates of the center point of key element A1 (i.e., the first element); x2 and y2 represent the coordinates of the center point of key element A2 (i.e., the second element).

[0130] In the embodiment of the present application, a hybrid network structure based on CNN and Transformer is used as the structure of the graph information extraction model, such as Figure 8 As shown in the figure, the input image first passes through CNN to obtain the feature map F. The feature map F is directly flattened according to the spatial position to obtain the Patch Embedding. The Patch Embedding passes through the Transformer Encoder to obtain the Token Embedding of the middle layer.

[0131] The Transformer Decoder outputs the Output Embedding of the final output layer based on the randomly initialized Object Query and Token Embedding. Finally, the decoding result of each Output Embedding is output based on the fully connected layer.

[0132] The decoding result contains two parts. The first part indicates the probability that the current key element belongs to each element category (not shown in the figure); the second part indicates the bbox position of the current key element (i.e. the first element) and the center point coordinates of other key elements related to it (i.e. the second element). When x1 or y1 is a negative number, it means that the coordinate is invalid. Figure 8 The model shown is called N1.

[0133] Figure 8 The scheme shown is similar to the DETR model. However, their important difference is that Figure 8 The AI ​​model shown can simultaneously identify the bbox of each key element (ie, the first element) and the center point coordinates of other elements related to it (ie, the second element) in an end-to-end manner.

[0134] In the embodiment of the present application, the corresponding models are trained for the mind map, bar chart and pie chart respectively. Specifically, the D1 data set constructed in S1 is used to train N1, and a model M1 suitable for the mind map is obtained (i.e., an example of a chart information extraction model); the D2 data set constructed in S1 is used to train N1, and a model M2 suitable for the bar chart is obtained (i.e., an example of a chart information extraction model); the D3 data set constructed in S1 is used to train N1, and a model M3 suitable for the pie chart is obtained (i.e., an example of a chart information extraction model).

[0135] S3: Building a large language model on the client side

[0136] Using 3B pre-trained LLM, we fine-tune 3B LLM using dataset D4 to get model M4. LLM uses a standard Decoder-only structure.

[0137] It should be noted that S1 to S3 are the model building / training process, which can be executed only once. In actual use of the model, it is not necessary to repeat it, and of course it can be updated iteratively.

[0138] S4: Graph Information Extraction

[0139] According to the image type input by the user, use model M1, M2 or M3 to extract information. When inputting an image, the user needs to specify the image type (that is, the chart type of the input image, such as specifying a mind map, bar chart or pie chart). The extracted information is stored in Jsonl format. The storage format is: each row stores a key element, containing four fields. The first field is "bbox", which indicates the position coordinates of the current key element itself; the second field is "relation", which indicates the center coordinates of other elements related to the current key element; the third field is the element type of the key element; the fourth field is the actual content contained in the key element; among them, the first field, the second field and the third field are the outputs of the chart information extraction model; the fourth field is obtained by recognizing the image content in the bbox through the OCR method.

[0140] S5: Chart information conversion

[0141] For bar charts and pie charts, convert their Jsonl format data into Markdown formatted table data. For bar charts, each legend information is used as the column name of the table header; the value of the x-axis is used as the row name; the value of the y-axis is used as the table value to fill in the corresponding table. For pie charts, each legend information is used as the column name of the table header; the value of the sector is used as the table value to fill in the corresponding table.

[0142] For mind maps, the original Jsonl format is used directly without conversion.

[0143] S6: LLM Reasoning

[0144] For bar charts and pie charts, the Markdown-formatted table data extracted from the original chart and the user's original query are input into LLM together. If the user's query is about a related issue in the chart (i.e., the first question query), LLM outputs an answer related to the question. If the user's query is about modifying the relevant content in the chart (i.e., the first modification instruction), LLM does not output any information to the user, but directly converts the modified Markdown table into data in Jsonl format, and then sends the Jsonl format data to the rendering engine for rendering. In some embodiments, the interface rendered and output by the rendering engine is in editable mode, and the user can edit or modify the key elements in the displayed chart.

[0145] Fig. 9 A schematic diagram of rendering a bar chart as a pie chart; Fig. 9 As shown, in interface 901, the user inputs the voice "change the data of 2014 to 480,000, and display the result in a pie chart" (i.e., an example of the first modification instruction). After receiving the user's inquiry, LLM automatically modifies the Markdown data of the current chart and reconverts it to Jsonl for rendering. The output interface after rendering is shown in 902. Interface 902 supports editing or modifying the key elements of the pie chart.

[0146] In an embodiment of the present application, the provided terminal-side chart intelligent editing and question-and-answer method can help users on the mobile phone to conveniently ask questions or modify the original chart content directly through natural language (voice or text), which can greatly facilitate users to quickly edit charts on the mobile phone.

[0147] The terminal-side chart intelligent editing and question-answering method provided in the embodiment of the present application has the following characteristics:

[0148] 1) From a functional perspective, the embodiments of the present application provide a method for using natural language to answer questions or quickly edit charts on low-power devices such as mobile phones;

[0149] 2) The embodiment of the present application provides an end-to-end information extraction solution for mind maps, bar charts, and pie charts. The problem of identifying the relationship between key elements is modeled as a problem of center point prediction. Therefore, the simultaneous identification of the position and relationship of key elements can be achieved by only changing the number of output values ​​of the "detection head" without changing the overall framework of the model;

[0150] 3) The embodiment of the present application uses an encoder based on the Transformer structure to simultaneously output the positions and relationships of key elements, which can effectively avoid the complex problem of deduplication of multiple candidate boxes faced by the traditional CNN-based object detection model;

[0151] In an embodiment of the present application, a two-stage natural language-based chart question-answering and editing method is provided. The method first uses a chart information extraction model to extract chart information into table data in Markdown format, and uses LLM's advantage of understanding Markdown to implement question-answering or editing based on Markdown.

[0152] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.; or, steps in different embodiments may be combined into a new technical solution.

[0153] Based on the foregoing embodiments, an embodiment of the present application provides an inquiry and response device, which includes the modules included and the units included in the modules, which can be implemented by a processor; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be an AI acceleration engine (such as NPU, etc.), GPU, central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP) or field programmable gate array (FPGA), etc.

[0154] Fig.10 A schematic diagram of the structure of the inquiry response device provided in the embodiment of the present application; Fig.10 As shown, the inquiry response device 100 includes:

[0155] The acquisition module 1001 is configured to acquire a first image and a first instruction; wherein the content of the first image includes chart information, and the first instruction includes a first question inquiry for the first image and / or a first modification instruction for the first image;

[0156] A first recognition module 1002 is configured to recognize position information of a first element and position information of a second element in the first image to obtain a recognition result;

[0157] A conversion module 1003, configured to convert the first image into first structured data according to the recognition result;

[0158] A second identification module 1004 is configured to determine first response information for the first instruction according to the first structured data and the first instruction;

[0159] The response module 1005 is configured to respond to the first instruction according to the first response information.

[0160] In some embodiments, the pre-trained chart information extraction model is a hybrid network structure based on CNN and Transformer; the identifying the position information of the first element and the position information of the second element in the first image to obtain the recognition result includes: inputting the first image into the pre-trained chart information extraction model to obtain the recognition result.

[0161] In some embodiments, the recognition result includes at least one of: the element category to which the first element belongs, a bounding box of the first element, and the center point coordinates of the second element.

[0162] In some embodiments, the second element is an element that has a relationship with the first element.

[0163] Exemplarily, in some embodiments, the first image includes a mind map, a bar chart, or a pie chart.

[0164] Exemplarily, in some embodiments, the first image includes the mind map, the first element in the mind map includes one or more of a root node, an intermediate node, a cluster, and a leaf node; wherein the cluster includes one or more leaf nodes having the same parent node; and the second element includes the parent node of the first element.

[0165] Exemplarily, in some other embodiments, the first image includes the bar chart, and the first elements in the bar chart include one or more of the x-axis coordinate title, x-axis coordinate value, y-axis coordinate title, y-axis coordinate value, the bounding box of the column, the overall title of the bar chart, a legend, and other text areas that do not belong to the above-mentioned first elements; the second element of the bounding box of the column includes one or more of other text areas associated with the column, the legend associated with the column, and the x-axis coordinate value of the column.

[0166] Exemplarily, in some further embodiments, the first image includes the pie chart, and the first element in the pie chart includes: one or more of the center position of the circular area, the intersection of any two sectors, a legend, and other text areas that do not belong to the above-mentioned first element; the second element of the legend includes other text areas associated with the legend and / or the center point coordinates of the sectors associated with the legend.

[0167] In some embodiments, converting the first image into first structured data based on the recognition result includes: obtaining first content contained in the first element in the first image based on a bounding box of the first element; arranging the first content according to a structure corresponding to the first structured data to obtain the first structured data.

[0168] In some embodiments, the first image includes the mind map, and the first structured data is data in Jsonl format.

[0169] In some embodiments, the first image includes the bar chart, and the first structured data is a Markdown table; wherein the legend in the bar chart serves as the column name of the table header of the Markdown table; the x-axis coordinate value serves as the row name of the Markdown table; and the y-axis coordinate value serves as the table value of the Markdown table and is filled in the corresponding table.

[0170] In some embodiments, the first image includes the pie chart, and the first structured data is a Markdown table; wherein the legend in the pie chart serves as the column name of the table header of the Markdown table; and the values ​​contained in the sectors in the pie chart are filled in the corresponding table as the table values ​​of the Markdown table.

[0171] In some embodiments, determining first response information for the first instruction based on the first structured data and the first instruction includes: inputting the first structured data and the first instruction into a pre-trained large language model to obtain the first response information.

[0172] In some embodiments, the first instruction includes a first question regarding the first image, and the first response information includes an answer to the first question; responding to the first instruction according to the first response information includes: outputting the first response information on a user interface, or broadcasting the first response information by voice.

[0173] In some other embodiments, the first instruction includes a first modification instruction for the first image, and the first response information includes second structured data obtained by modifying the first structured data with respect to the first modification instruction; and responding to the first instruction according to the first response information includes: rendering and outputting the second structured data according to a chart type of the first image.

[0174] In some further embodiments, the first instruction includes a first modification instruction for the first image, and the first response information includes the second structured data and the type of chart to be output after modification required by the first modification instruction; and responding to the first instruction according to the first response information includes: rendering and outputting the second structured data according to the type of chart to be output after modification required by the first modification instruction.

[0175] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0176] It should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. It may also be implemented in the form of a combination of software and hardware.

[0177] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable the terminal device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0178] The present application embodiment provides a terminal device, Fig.11 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application; Fig.11 As shown, the terminal device 110 includes a memory 1101 and a processor 1102, wherein the memory 1101 stores a computer program that can be run on the processor 1102, and the processor 1102 implements the steps in the method provided in the above embodiment when executing the program.

[0179] It should be noted that the memory 1101 is configured to store instructions and applications executable by the processor 1102, and can also cache data to be processed or processed by various modules in the processor 1102 and the terminal device 110 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0180] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program.

[0181] Optionally, the computer-readable storage medium can be applied to the terminal device in the embodiments of the present application, and the computer program enables the terminal device to execute the various methods of the embodiments of the present application. For the sake of brevity, they are not described here in detail.

[0182] An embodiment of the present application also provides a computer program product, including computer program instructions.

[0183] Optionally, the computer program product may be applied to a terminal device in an embodiment of the present application, and the computer program instructions enable the terminal device to execute the various methods in the embodiment of the present application, which will not be described in detail here for the sake of brevity.

[0184] The embodiment of the present application also provides a computer program.

[0185] Optionally, the computer program may be applied to a terminal device in an embodiment of the present application. When the computer program runs on the terminal device, the terminal device executes the various methods in the embodiment of the present application. For the sake of brevity, they are not described here in detail.

[0186] It should be noted here that the description of the above terminal device, storage medium, computer program product and computer program embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the terminal device, storage medium, computer program product and computer program embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0187] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. For the sake of brevity, this article will not repeat them.

[0188] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0189] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0190] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0191] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0192] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0193] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0194] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling the terminal device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0195] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0196] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0197] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0198] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A query response method, characterized in that: The method is applied to a terminal device, and the method comprises: Acquire a first image and a first instruction; wherein the content of the first image includes chart information, and the first instruction includes at least a first question inquiry for the first image and / or a first modification instruction for the first image; Identify the position information of the first element and the position information of the second element in the first image to obtain a recognition result; converting the first image into first structured data according to the recognition result, and determining first response information for the first instruction according to the first structured data and the first instruction; The first instruction is responded to according to the first response information.

2. The method according to claim 1, characterized in that The recognition result includes at least one of: the element category to which the first element belongs, the bounding box of the first element, and the center point coordinates of the second element.

3. The method according to claim 1 or 2, characterized in that: The second element is an element related to the first element.

4. The method according to any one of claims 1 to 3, characterized in that The first image includes a mind map, a bar chart or a pie chart.

5. The method according to claim 4, characterized in that The first image includes the mind map, wherein the first element in the mind map includes one or more of a root node, an intermediate node, a cluster, and a leaf node; wherein the cluster includes one or more leaf nodes having the same parent node; The second element includes a parent node of the first element.

6. The method according to claim 4, characterized in that The first image includes the bar graph, and the first elements in the bar graph include one or more of an x-axis coordinate title, an x-axis coordinate value, a y-axis coordinate title, a y-axis coordinate value, a bar boundary box, a general title of the bar graph, a legend, and other text areas that do not belong to the first elements; The second element of the bounding box of the column includes one or more of other text areas associated with the column, a legend associated with the column, and an x-axis coordinate value of the column.

7. The method according to claim 4, characterized in that The first image includes the pie chart, and the first elements in the pie chart include: one or more of the center position of a circular area, the intersection of any two sectors, a legend, and other text areas that do not belong to the first elements; The second element of the legend includes other text areas associated with the legend and / or the center point coordinates of the sector associated with the legend.

8. The method according to any one of claims 5 to 7, characterized in that: The converting the first image into first structured data according to the recognition result includes: Acquire, according to the bounding box of the first element, first content included in the first element in the first image; The first content is arranged according to the structure corresponding to the first structured data to obtain the first structured data.

9. The method according to claim 8, characterized in that The first image includes the mind map, and the first structured data is data in Jsonl format.

10. The method according to claim 8, characterized in that The first image includes the bar chart, and the first structured data is a Markdown table; wherein the legend in the bar chart serves as the column name of the table header of the Markdown table; the x-axis coordinate value in the bar chart serves as the row name of the Markdown table; and the y-axis coordinate value in the bar chart serves as the table value of the Markdown table and is filled in the corresponding table.

11. The method according to claim 8, characterized in that The first image includes the pie chart, and the first structured data is a Markdown table; wherein the legend in the pie chart serves as the column name of the table header of the Markdown table; and the values ​​contained in the sectors in the pie chart are filled in the corresponding table as the table values ​​of the Markdown table.

12. The method according to any one of claims 1 to 11, characterized in that The determining, according to the first structured data and the first instruction, first response information for the first instruction includes: The first structured data and the first instruction are input into a pre-trained large language model to obtain the first response information.

13. The method according to any one of claims 1 to 11, characterized in that The first instruction includes a first question inquiry regarding the first image, and the first response information includes an answer to the first question inquiry; The responding to the first instruction according to the first response information includes: The first response information is output on a user interface, or the first response information is broadcasted by voice.

14. The method according to any one of claims 1 to 11, characterized in that The first instruction includes a first modification instruction for the first image, and the first response information includes second structured data obtained by modifying the first structured data according to the first modification instruction; The responding to the first instruction according to the first response information includes: rendering and outputting the second structured data according to the chart type of the first image; Alternatively, the first instruction includes a first modification instruction for the first image, and the first response information includes the second structured data and a chart type to be output after modification required by the first modification instruction; The responding to the first instruction according to the first response information includes: The second structured data is rendered and output according to the type of the chart to be output after modification required by the first modification instruction.

15. An inquiry response device, characterized in that: The device comprises: An acquisition module configured to acquire a first image and a first instruction; wherein the content of the first image includes chart information, and the first instruction includes at least a first question inquiry for the first image and / or a first modification instruction for the first image; A first recognition module is configured to recognize position information of a first element and position information of a second element in the first image to obtain a recognition result; a conversion module, configured to convert the first image into first structured data according to the recognition result; a second identification module, configured to determine first response information for the first instruction according to the first structured data and the first instruction; A response module is configured to respond to the first instruction according to the first response information.

16. A terminal device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 14 is implemented.

Images