Visual question answering on statistical plots

Abstract

TITLE: VISUAL QUESTION ANSWERING ON STATISTICAL PLOTS APPLICANT: PES UNIVERSITY ABSTRACT The present invention discloses a system of visual question answering on statistical plots for data analysis thereby automating decision making capabilities. The system of present invention comprises of an input device and a central processing device. The input device is registered with the system with a unique ID. It is configured to input graphical image comprising of bar plot, line plot, and dot plots and a query in natural language whose answer is extracted from the graphical image. The central processing device interfaced with the input device is adapted to receive graphical image input from the input device and is configured to process the graphical image. The process is characterized in i. detecting the plot elements of the graphical image employing Faster-RCNN and Resnet-101 to obtain json file containing bounding box values; ii. formatting the json file containing bounding box values into a text file by Optical Character Recognition Module pyocr; iii. generation of the Semi-Structured Table comprising of complete data from the graphical image by mapping legend values to legend color, x-ticks to x-axis label and y-ticks to y-axis label based on PlotQA dataset; iv. outputting answer to the query from the Semi-Structured Table employing TAPAS model to answer open-ended type questions and TABFACT to answer Boolean type questions; v. sending the answer to the input device.

Description

FIELD OF THE INVENTION:The present invention relates to question answering system. More specifically, the present invention relates question answering on data analysis of statistical plots.BACKGROUND OF THE INVENTION:Statistical charts are an intuitive and simple way to represent data. They form a way of representing structured data in the form of graphical visualizations. Such graphical visualizations aid people in better interpreting features of data. Object detection in deep learning is a field that focuses on extricating localized datum from binary or color images.Therefore, it is useful to build a model that can localize and pick up visual data in statistical plots. It is one step towards the improvement in localized object detection capabilities.Visual plots are commonly found in research papers, scientific journals, business records e.t.c. Therefore, automation of plot analysis through the means of question-answering aids an individual to draw statistical inferences quickly from them.Statistical plots are used widely by academicians and business employees because they are a simple way to represent data. They can be easily analysed and interpreted.The difficulty with statistical plots is that even though they are images, they contain both structured and unstructured data. In the case of natural images, there are just visual elements to handle. However, that is not the case with statistical plots, since they contain both visual elements in the form of bars / sectors and textual elements in the form of axis labels and ticks. In short they contain multiple objects within an image where the object is a plot element and the image is the graph itself. To add to this, the size of objects that are there in natural images are constrained to small, medium and large in general. In the case of statistical plots, the aspect ratio is much more varied. For example, in the case of bar plots, there could be bars that are extremely small, and bars that are extremely large on the same plot.The measure of the accuracy of prediction in the case of images is normally IOU (intersection over union). The success criteria for a correct prediction in the case of natural images is normally 50 per cent. The same rate is insufficient for statistical plots. This is because we want the prediction for a bar value (in the case of bar plots) to correspond to the actual value as seen on the graph, as close as possible. Therefore the adequacy criteria for a successful prediction are much higher. The above-mentioned factors highlight the differences in natural images and statistical plots.JP2017534956A discloses an embodiment of a multimodal question response (MQA) model is provided to answer a question related to the content of an image. In an embodiment, the model comprises four components: a long short-term storage (LSTM) component for extracting a question expression; a convolutional neural network (CNN) component for extracting a visual representation; an LSTM component for storing a linguistic context in the answer; and a fusion component for combining information from the three components before generating an answer. By constructing a free multilingual image question response (FM-IQA) data set, an embodiment of the MQA model is trained and evaluated.A human determination person evaluates the quality of an answer generated in the data set by the MQA model by a turing test.US9984772B2 discloses a computer-implemented method for predicting answers to questions concerning medical image analytics reports includes splitting a medical image analytics report into a plurality of sentences and generating a plurality of sentence embedding vectors by applying a natural language processing framework to the plurality of sentences. A question related to subject matter included in the medical image analytics report is received and a question embedding vector is generated by applying the natural language processing framework to the question. A subset of the sentence embedding vectors most similar to the question embedding vector is identified by applying a similarity matching process to the sentence embedding vectors and the question embedding vector. A trained recurrent neural network (RNN) is used to determine a predicted answer to the question based on the subset of the sentence embedding vectors.US11270145B2 discloses approaches for interpretable counting for visual question answering include a digital image processor, a language processor, and a counter. The digital image processor identifies objects in an image, maps the identified objects into an embedding space, generates bounding boxes for each of the identified objects, and outputs the embedded objects paired with their bounding boxes. The language processor embeds a question into the embedding space. The scorer determines scores for the identified objects. Each respective score determines how well a corresponding one of the identified objects is responsive to the question. The counter determines a count of the objects in the digital image that are responsive to the question based on the scores. The count and a corresponding bounding box for each object included in the count are output. In some embodiments, the counter determines the count interactively based on interactions between counted and uncounted objects.CN108170816B discloses an invention discloses an intelligent visual question answering model based on a deep neural network and relates to intelligent visual question answering in the artificial intelligence field. A method comprises the steps of intelligent question answering data preprocessing; image deep convolution feature extraction; text question deep feature extraction; intelligent visual question processing; and visual intelligent question answering based on a hub channel. A multi-task learning framework is adopted to perform brand-new deep learning network design so as to solve the two problems that training data is lacked and answer reasons are unknowable in intelligent visual question answering. A novel deep learning network structure is designed, wherein the network can explain reasons for given answers while intelligent visual question answering is performed, and the network structure comprises a visual description module which can describe image content pertinently according to question content. The network structure adopts hub structural design and can introduce data in image description, text question answering and other fields into a visual intelligent question answering task.WO2019148315 discloses an example apparatus for visual question answering includes a receiver to receive an input image and a question. The apparatus also includes an encoder to encode the input image and the question into a query representation including visual attention features. The apparatus includes a knowledge spotter to retrieve a knowledge entry from a visual knowledge base pre-built on a set of question-answer pairs. The apparatus further includes a joint embedder to jointly embed the visual attention features and the knowledge entry to generate visual-knowledge features. The apparatus also further includes an answer generator to generate an answer based on the query representation and the visual-knowledge features.The prior arts mentioned above disclose visual question answering to automatically answer natural language question about an image. There is a need in the state of art to build a Visual Question Answering system which accepts statistical plots(along with questions on the plot with respect to the elements of the plot such as intersection of the curves, area under the curve, median value and few othervarieties of such relational queries), discover relationships between elements of a plot and provide relational reasoning to answer questions on the plot. Thus, an attempt has been made in the state of art to develop a model to generate a system of visual question answering on statistical plots for data analysis with automated decision making capabilities. OBJECT OF THE INVENTION:The main object of the present invention is to develop a system of visual question answering on statistical plots.Another object of the present invention is to develop a system of visual question answering on statistical plots for data analysis.Yet another object of the present invention is to develop a system of visual question answering on statistical plots for data analysis and automating decision making capabilities.Further object of the present invention is to utilize the developed system to answer the questions on statistical charts and automate the decision-making capabilities in several sectors such as financial sector.BRIEF DESCRIPTION OF DRAWINGS:S.No PART NAME PART No.1 Input device 1012 Central Processing Device 102Figure1 depicts high level view of The VQA systemFigure 2 depicts cut open view of the black box and deep delve into the modulesFigure 3 depicts represents [top-left to bottom-right] vertical grouped bar, simple vertical bar, simple horizontal bar, simple line plot and bar plotFigure 4 depicts the object detection modelFigure 5 depicts Skip connections in a Resnet-101 modelFigure 6 depicts bert based model - Tapas to perform table question answeringFigure 7 depicts a statistical plot given as a test input Figure 8 depicts the json output produced by detectron tester, which maps object elements to its class , with a certain confidence and the bounding box tensorFigure 9 depicts the mapping and the unstructured json dataFigure 10 depictsproperly formatted text file which has the bounding box tensors of all the plot elements that were detected in the modelFigure 11 depicts a simple graph give as inputFigure 12 depicts textual output of input graphFigure 13 depicts Tabular CSV format of the input graphFigure 14 depicts The Table Question Answering StageFigure 15 depicts Test Accuracy After Trial - 1 With Parameter Setting on RHSFigure 16 depicts Test Accuracy After Trial - 2 With Parameter Setting on RHSSUMMARY OF THE INVENTION:The present invention discloses a system of visual question answering on statistical plots for data analysis thereby automating decision making capabilities. The system of present invention comprises of an input device and a central processing device.The input device is registered with the system with a unique ID.It is configured to input graphical image comprising of bar plot, line plot, and dot plots and a query in natural language whose answer is extracted from the graphical image.The central processing device interfaced with the input device is adapted to receive graphical image input from the input device and is configured to process the graphical image. The process is characterized in i. detecting the plot elements of the graphical image employing Faster-RCNN and Resnet-101 to obtain json file containing bounding box values;ii. formatting thejson file containing bounding box values into a text file by Optical Character Recognition Module pyocr;iii. generation of the Semi-Structured Table comprising of complete data from the graphical imageby mapping legend values to legend color, x-ticks to x-axis label and y-ticks to y-axis label based on PlotQA dataset;iv. outputting answer to the query from the Semi-Structured Table employingTAPAS model to answer open-ended type questions and TABFACT to answer Boolean type questions;v. sending the answer to the input device.DETAILED DESCRIPTION OF THE INVENTION:The present invention shall disclose a system for visual question answering on statistical plots for data analysis thereby automating decision making capabilities.The present invention comprises of an input device and a central processing device.The input device is registered with the system with a unique ID. It is configured to input graphical image comprising of bar plot, line plot, and dot plots and a query in natural language whose answer is extracted from the graphical image.The input to our model is an image of a statistical plot and a corresponding question on the plot.The questions provided as an input need to be of the type handled by the scope of this model. Important details from the image and the questions need to be mapped (done here through OCR and semi-structured table creation) and the corresponding answer should be predicted (tablequestion answering). The inputs to black-box / VQA system• A graphical plot [bar {vertical / horizontal / grouped}, dot, line]• A question posed [in vocab / out of vocab] related to the plot imageStatistical charts:Statistical charts are one of the inputs to the visual question answering system. The types of charts that we have constrained to are bar charts, line plots and dot / dot line plots. A typical plot in general irrespective of its type will include:• Title of the plot / graph• Xlabel and Ylabel• Legend data if any• Xticks and YticksThere are different types / variants of bar charts like vertical charts, horizontal charts, simple, and group charts. Whereas dot plots and line plots have zero or no variants.The central processing device interfaced with the input device is adapted to receive graphical image input from the input device and is configured to process the graphical image. The central processing device processes the visual image using the object detection model further converting the objects obtained into a consolidated semi-structured format and parse the query, concatenate the inferences from both and produce an output. Thus, it combines the technicalities of object detection capabilities and query language understanding.In the figure 2 we have two inputs being fed into the central processing device which are an input image [the graphical plot in scope] and a Natural language query related to the graphical image. whereas in the lower level design the central processing device is further expanded and cut open. The central processing device constitutes the core of the visual question answering system.There are 4 stages within the central processing device as shown in Figure2 are as follows:- The plot element detection stage- The optical character recognition stage- The Semi-structured table generation stage- Table Question Answering StageThe central processing device of the present invention detects the plot elements of the graphical image employing Faster-RCNN and Resnet-101 to obtain json file containing bounding box values; Visual features from the charts have to be extracted and preprocessed. This is done using techniques like object detection processing and Optical Character Recognition (OCR).The central processing device of the present system generates the Semi-Structured Table comprising of complete data from the graphical image by mapping legend values to legend color, x-ticks to x-axis label and y-ticks to y-axis label based on Plot QA dataset. The design consists of 3 primary components.1. Image Encoding Module / Plot Elements Detection modelThis is a module that takes in the image as its input along with the annotations of the image, correlates them together. This module consists of a deep learning FASTER R-CNN module for object localization, because our main aim is to localize and produce bounding boxes around the plot elements and extricate them rather than classifying an image. This module produces the bounding box annotation of all involved plot elements captured by the object detection model which is fine-tuned and trained by us. Bounding boxes are a vector representation of the plot elements in a graph, which refer to the coordinates of the object (top_x, top_y, bottom_x, bottom_y); we effectively need only these 4 points to draw a bounding box around a plot element. Bounding box values of the plot elements like xlabel, title, ylable, the bar, the line, the dot are all obtained. While an image classification network can tell whether an image contains a certain object or not, it won't say where in the image the object is located. We are in the quest of locating the plot elements and not just classifying the plot. Hence we don't need just a CNN, we need an object detection neural network like R-CNNs we need to decide over the object detection model.Faster R-CNN is our object detection model which is a descendant from the R-CNNs series. The heuristic behind doing so can be inferred from the below observations and also due to the presence of the RPN (Regional Proposal Network is known for locating feature targets accurately).We also need to decide on the feature extractor model for serving as the backbone / feature-extractor for our faster-RCNN Setup. We Chose Resnet-101 to be our feature extractor due to skip connections and residual blocks which can reduce the problem of vanishing gradients in a very deep network like Resnet-101.The central processing device of the present system formats the json file containing bounding box values into a text file by Optical Character Recognition Module pyocr.The central processing device of the present system shall output answer to the query from the Semi-Structured Table employing TAPAS model to answer open-ended type questions and TABFACT to answer Boolean type questions. The central processing device of the present system shall send the answer to the input device.2. Question Encoding Module / Table Question Answering stageThe input to this module is the question (English Language) and the output is a question embedding. The question embedding captures all of the relevant information in the question in a format that is suitable for further modelling. This module needs tabular data on which the question answering can be performed. Hence the output of the image module must be further structured using other utility stages so that a semi structured table is obtained, on which queries can be executed.Deep learning methods and architectures are employed to predict the output both at the image level and questionnaire levels. Inputs are validated by the system to recognize only statistical charts. Any other images will be responded with poor results as they aren't seen by the model previously or rather untrained. The system will be trained on huge amounts of data and the results will be validated against the true answers using suitable methods.3. Result Estimation ModuleThe output of the image module would be a list of (element_type, confidence score, bounding box vector). The length of the list indicates the number of elements in the input plot. Now that bounding box annotations are obtained, next the points of references to the graph elements is passed along with the image to the OCR stage, in which the localized text content is extricated, after which we have utility modules to map the complete image data into its tabular equivalent and generate the csv, now things become simpler because the plot data is no more an image but it is a semi-structured table on which table question answering can be performed, just as in case of the wiki tables questions dataset.The questions that are handled in our work are related to data-retrieval following a simple select, project or select cum project vice-versa, statistical mean, min - max, boolean truths, range based queries, simple summations, summations across an interval from the graph data, trend wise queries, differences with few constraints and restrictions laid on them.The parameters for the object detection model are tuned to provide the most bounding box capturing capability and a proper pre-trained table question answering module will have to execute the queries on the tabular data and respond with an answer to the end user.In the development of our model, questions must be related to answers from fixed vocabulary or answers that can be extracted from the bounding box representation of an image or answers that can be queried from a structured table generated using visual elements. Plots related to bar plot, line plot, and dot plots have been considered.Security RequirementsData Sharing: As there are lots of graphical images to be collected, the data and statistics storage along with logger data will be done to maintain the correct functioning of the model and to reconstruct what went wrong in case of any system failures by constructing checkpoints. The datasets, if artificially generated, need to be secured locally and not be made available for commercial usages.Model security: The model trained needs to be protected on a local machine or on cloud storage if deployed onto the cloud. Modification of ownership to only allow viewable access can be enabled to outsiders, versioning of the model weights can also be done to build over the training and enhance the performance.Advantages of the present invention:Usability: The trained model must be available at ease to use it, just by choosing an image input from the local drive or file-system. Similar to drag and drop or attaching files. The user must be able to navigate through the interface even with minimal exposure towards computing technologies.Reliability: As the product is developed by following practices in deep-learning, machine learning and imaging, there is no certain fixed level of reliability that can be set for the product. Reliability is not completely independent of the inputs to the model, hence reliability varies with respect to the context and type of inputs passed in provided they comply with the assumptions and restrictions of the model.Maintainability: As the product is just a model, maintaining the model isn't a difficult task, it only requires a cloud machine, a drive to support storage on cloud so that it can persist and to reside on and a browser / interface to access.Performance: The model is expected to draw statistical inferences based on the question and the input passed with a good and acceptable accuracy level relative to ground truth or human evaluation.Robustness: The model must be robust enough to handle any of the graph images and questions related to its scope of operation. The model must yield good performance throughout the pipeline for any input from the wide range of possible inputs pertaining to the scope of the model.In one the preferred embodiments the present invention shall disclose a system of visual question answering on statistical plots for data analysis thereby automating decision making capabilities comprises of a. an input device registered with the system with unique ID and is configured to input graphical image comprising of bar plot, line plot, and dot plots and a query in natural language whose answer is extracted from the graphical image;b. a central processing device interfaced with the input device, adapted to receive graphical image input from the input device and configured to process the graphical image characterized in thati. detecting the plot elements of the graphical image employing Faster-RCNN and Resnet-101 to obtain json file containing bounding box values;ii. formatting the json file containing bounding box values into a text file by Optical Character Recognition Module pyocr;iii. generation of Semi-Structured Table comprising of complete data's from the graphical image by mapping legend values to legend color, x-ticks to x-axis label and y-ticks to y-axis label based on PlotQA dataset;iv. outputting answer to the query from the Semi-Structured Table employing TAPAS model to answer open-ended type questions and TABFACT to answer Boolean type questions;v. sending the answer to the input device. In a further embodiment, the present invention shall disclose a system in which the mapping of text is done by associating legend / x-tick / y-tick value bounding box to the closest legend color / x-axis / y-axis boundary from PlotQA dataset.In a further embodiment, the present invention shall disclose a system in which the mapping of visual elements is done by associating each element is associated with axis, and corresponding legend from PlotQA dataset.In another embodiment, the present invention shall disclose a system in which the mapping of color of the visual element is done by associating legend colors, and the legend of closest match from PlotQA dataset.In yet another embodiment, the present invention shall disclose a system in which the mapping of value associated with the bar is done by taking information of height from bounding box representation and associating closest y-tick from PlotQA dataset.In yet another embodiment, the present invention shall disclose a system in which input device is computer, laptop, IPAD, tab and the like.In yet another embodiment, the central processing device is a computer.Working Examples:The image along with its annotation is passed onto a trained model. The weights of the model are saved for further testing and inference, as the image passes through the designed object detection model, the bounding boxes are expected to be drawn around every potential plot element which was learnt by the model. The output of a model will be a json file which maps all the detected objects (plot elements) to the class to which it belongs, with an appropriate confidence value and its bounding box tensor.Figure 7 depicts a statistical plot given as a test input Figure 8 depicts the json output produced by detectron tester, which maps object elements to its class , with a certain confidence and the bounding box tensorFigure 9 depicts the mapping and the unstructured json dataFigure 10 depicts properly formatted text file which has the bounding box tensors of all the plot elements that were detected in the model.As seen above, all the bounding boxes corresponding to the plot elements have been extracted according to their classes. We chose Faster-RCNN as our object detection model which is a descendant from the R-CNNs series. The heuristic behind doing so is due to the presence of the RPN (Regional proposal network is known for locating feature targets accurately) and the inference time.We also need to decide on the feature extractor model for serving as the backbone / feature-extractor for our faster-RCNN Setup. We Chose Resnet-101 to be our feature extractor due to skip connections and residual blocks which can reduce the problem of vanishing gradients in a very deep network like Resnet-101.B) OCR detection Stage- The textual format of the json file is passed into this stage.- The images directory is made accessible to this module.- With the help of bounding box coordinates extracted, we can accurately and locally capture the text information within the bounding boxes rather than passing an entire image into the OCR module.- The captured text is then read using OCR and classified into its category.C) Semi Structured table generationThis phase is the crucial phase of converting the graphical data to its tabular format based on the OCR readings done in accordance with classes. We'll walk through an example to explain the conversion of an image (plot) into a tabular data format (csv). An input graph goes through several transformations in the pipeline but not necessarily in the image format; right after the OCR stage, the input image is no longer required, the image is discarded and its tabular csv format comes into play for the final table QA stage.Figure 11 depicts a simple graph give as inputThe simple bar graph goes through the plot element detection stage, and a text file emerges as the outcome as discussed in the previous stage. This text file will then be passed onto the OCR stage along with the image to extract the textual content within the boxed coordinates. Following which the information is structured in the semi structured table generation phase.D) Table Question Answering Stage• We have made use of Google's TAPAS to answer questions from tabular data• Tapas selects a subset of table cells and applies aggregation / retrieval operations on top of them• It extends BERT architecture with additional embeddings that capture tabular structure, and with two classification layers for selecting cells and predicting a corresponding aggregation operator.• We have added our custom operations and methods to suit the desired output.• It is trained on Wikipedia Tables and provides a pre-trained model for end tasksRESULTS The steps mentioned in our methodology were followed and the results obtained by us are displayed in the following sections along with the heuristic behind how they have been obtained with supporting evidence.1. Plot Element Detection StageThe aim in this stage was to ensure that there was maximal overlap between the bounding boxes proposed by our object detection trained model with saved weight and the actual bounding box locations of the test image which are hidden from the model. AP aka Average precision is the score to look out for, lies between 0 - 100%, more the value of AP, higher the overlap and better the prediction made by the model. This again requires right selection of models, appropriate number of training iterations and Gamma factor. Below we display the results for various parameter settings and the incremental growth achieved by the model.Figure 14 depicts Test Accuracy After Trial - 1 With Parameter Setting on RHSFigure 15 depicts Test Accuracy After Trial - 2 With Parameter Setting on RHSIt can conclusively be seen that the AP increases linearly with the Iterations. Better training produces better results. The maximum iterations we could run was for 2 lakh iterations. Per category b box AP value also tends to increase along with more training. This better AP helps to capture the textual and pictorial elements better in the further stages of the pipeline.2. Table Question Answering StageThe Table QA Stage outputs an answer for the question using the CSV generated from the semi-structured table generation stage. The accuracy of this stage depends on the accuracy of previous stages. The answers are categorized into two classes to arrive at the final accuracy. One type is the floating-point answers. Since the answer cannot be exact, we have allowed an error window of 5%.Values that are within this 5% range will be accepted. The other type are the String answers. Here, we consider an Exact Match metric. Answers that exactly match the ground truth values are considered towards accuracy.Further modifications and alternative embodiments of various aspects of the invention may be apparent to those skilled in the art in view of this description. Accordingly, this description is to be construed as illustrative only and is for the purpose of teaching those skilled in the art the general manner of carrying out the invention. It is to be understood that the forms of the invention shown and described herein are to be taken as the presently preferred embodiments. Elements and materials may be substituted for those illustrated and described herein, parts and processes may be reversed, and certain features of the invention may be utilized independently, all as would be apparent to one skilled in the art after having the benefit of this description of the invention. Changes may be made in the elements described herein without departing from the scope of the invention as described in the following claims.

Claims

1. A system of visual question answering on statistical plots for data analysis thereby automating decision making capabilities comprises of a. an input device registered with the claimed system with unique ID and is configured to input graphical image comprising of bar plot, line plot, and dot plots and a query in natural language whose answer is extracted from the said graphical image; b. a central processing device interfaced with the said input device, adapted to receive graphical image input from the said input device and configured to process the said graphical image characterized in that i. detecting the plot elements of the said graphical image employing Faster-RCNN and Resnet-101 to obtain json file containing bounding box values; ii. formatting the said json file containing bounding box values into a text file by Optical Character Recognition Module pyocr; iii. generation Semi-Structured Table comprising of complete data's from the said graphical image by mapping legend values to legend color, x-ticks to x-axis label and y-ticks to y-axis label based on PlotQA dataset; iv. outputting answer to the said query from the said Semi-Structured Table employing TAPAS model to answer open-ended type questions and TABFACT to answer Boolean type questions; v. sending the said answer to the said input device.

2. The system as claimed in claim 1 wherein the said mapping of text is done by associating legend / x-tick / y-tick value bounding box to the closest legend color / x-axis / y-axis boundary from PlotQA dataset.

3. The system as claimed in claim 1 wherein the said mapping of visual elements is done by associating each element is associated with axis, and corresponding legend from PlotQA dataset.

4. The system as claimed in claim 1 wherein the said mapping of color of the visual element is done by associating legend colors, and the legend of closest match from PlotQA dataset.

5. The system as claimed in claim 1 wherein the said mapping of value associated with the bar is done by taking information of height from bounding box representation and associating closest y-tick from PlotQA dataset.

6. The system as claimed in claim 1, wherein the said input device is computer, laptop, IPAD, tab and the like.

7. The system as claimed in claim 1, wherein the said central processing device is a computer.