Chart analysis method and device and electronic equipment

Abstract

The invention discloses a chart analysis method and device and electronic equipment. The chart analysis method comprises the steps that a target document is obtained; detecting the target document to obtain a target object in the target document and the type and position information of the target object; based on the position information, extracting a target object from the target document; through the text understanding model, the target object is analyzed according to the type of the target object, the text description information is generated, the text description information can describe the target object, content analysis of the picture and / or the table is achieved, convenience is provided for follow-up document verification and maintenance, and meanwhile the user experience is improved. And a basis is provided for subsequently recalling the chart in the database.

Description

Technical Field

[0001] The present application relates to the technical field of graph processing, and in particular to a graph parsing method, device and electronic device. Background Art

[0002] In some scenarios, pictures and tables can express the content to be conveyed more clearly than text, and pictures or tables can better reflect the characteristics of some data sets. Therefore, pictures and tables are used in more and more documents, such as academic articles, educational examination papers, economic manuscripts, etc.

[0003] However, when verifying, maintaining, and other processing the document, it is impossible to accurately detect images and tables, let alone parse the contents of the images and tables. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a chart analysis method, device and electronic device.

[0005] In a first aspect, an embodiment of the present application provides a chart parsing method, comprising:

[0006] Get the target document;

[0007] Detecting the target document to obtain a target object in the target document and type and location information of the target object;

[0008] extracting the target object from the target document based on the location information;

[0009] The target object is analyzed according to the type through a text understanding model to generate text description information, which can describe the target object.

[0010] In a possible implementation manner, obtaining the target document includes:

[0011] Get the document to be parsed;

[0012] Determine whether the format of the document to be parsed is a picture format;

[0013] If the format of the document to be parsed is not a picture format, the document to be parsed is format converted to obtain the target document, and the format of the target document is a picture format.

[0014] In a possible implementation, detecting the target document to obtain the target object in the target document and the type and location information of the target object includes:

[0015] Identify the target document through a detection model to determine a candidate region in the target document;

[0016] For each candidate region, determining the type of content in the candidate region;

[0017] If the type of the content in the candidate area is a target type, the candidate area is determined to be a target object, and the target type includes a picture and a table;

[0018] The position of the target object in the target document is determined.

[0019] In a possible implementation manner, extracting the target object from the target document based on the location information includes:

[0020] extracting content corresponding to the location information from the target document;

[0021] It is determined that the content corresponding to the location information is the target object, and the format of the target object is a picture format.

[0022] In a possible implementation, analyzing the target object according to the type through a text understanding model to generate text description information includes:

[0023] Extracting text features and visual features of the target object through a text understanding model;

[0024] Based on the text features and the visual features, text description information for the target object is generated.

[0025] In a possible implementation, the generating text description information for the target object based on the type, the text feature, and the visual feature includes:

[0026] Based on the type and the text feature, selecting at least one description template, the text feature being at least one;

[0027] Generate text description information for the target object according to the text feature, the visual feature and the description template

[0028] In a possible implementation, the method further includes the following steps:

[0029] Obtain a first training set, wherein the first training set includes an image sample and a text description sample corresponding to each image sample and an image sample and a text description sample corresponding to each table sample;

[0030] Using the first training set to train the first model until the first model meets the accuracy requirement, thereby obtaining a second model;

[0031] Filter out a second training set using the second model;

[0032] The second model is trained based on the first training set and the second training set to obtain the detection model.

[0033] In a possible implementation, the training the second model based on the first training set and the second training set to obtain the detection model includes:

[0034] According to a preset ratio, select part of the training data from the first training set and the second training set respectively to obtain a third training set;

[0035] The second model is trained using the third training set until the second model meets the accuracy requirement, thereby obtaining the detection model.

[0036] In a second aspect, the present application also provides a chart analysis device, including:

[0037] an acquisition module configured to acquire a target document;

[0038] A detection module, configured to detect the target document to obtain a target object in the target document and type and location information of the target object;

[0039] an extraction module, configured to extract the target object from the target document based on the location information;

[0040] A generation module is configured to analyze the target object according to the type through a text understanding model to generate text description information, wherein the text description information can describe the target object.

[0041] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via a bus, and when the machine-readable instructions are executed by the processor, the steps of any one of the above-mentioned chart parsing methods are performed.

[0042] The embodiment of the present application first detects the target document through the detection model to obtain the target object in the target document and the type and location information of the target object, and extracts the target object based on the location information; then, the target object is analyzed according to the type of the target object through the text understanding model to generate text description information that can describe the target object, thereby realizing the content analysis of the image and / or table, providing convenience for the subsequent verification and maintenance of the document, and at the same time, providing a basis for the subsequent recall of the chart in the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 A flowchart of a chart parsing method provided by the present application is shown;

[0045] Figure 2 A flowchart of obtaining a target document in a diagram parsing method provided by the present application is shown;

[0046] Figure 3 A flowchart showing a method for analyzing a graph provided by the present application for detecting a target document and obtaining the type and location information of a target object in the target document is shown;

[0047] Figure 4 A flowchart of generating text description information in a chart parsing method provided by the present application is shown;

[0048] Figure 5 A flowchart of a detection model training in a chart parsing method provided by the present application is shown;

[0049] Figure 6 A schematic diagram of a target document provided by the present application is shown;

[0050] Figure 7 A schematic diagram of the structure of a chart parsing device provided by the present application is shown;

[0051] Figure 8 A structural schematic diagram of an electronic device provided by the present application is shown. DETAILED DESCRIPTION

[0052] Various aspects and features of the present application are described herein with reference to the accompanying drawings.

[0053] It should be understood that various modifications may be made to the embodiments of the present application. Therefore, the above description should not be considered as limiting, but only as an example of an embodiment. Other modifications within the scope and spirit of the present application will occur to those skilled in the art.

[0054] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0055] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0056] It should also be understood that, although the present application has been described with reference to some specific examples, those skilled in the art will be able to realize many other equivalent forms of the present application that have the features described in the claims and are therefore within the scope of protection defined thereby.

[0057] The above and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.

[0058] Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments applied for are merely examples of the present application, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that obscure the present application. Therefore, the specific structural and functional details applied for herein are not intended to be limiting, but merely serve as a basis and representative basis for the claims to teach those skilled in the art to use the present application in a variety of ways with substantially any suitable detailed structure.

[0059] This specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present application.

[0060] To facilitate understanding of the present application, a chart parsing method provided by the present application is first introduced in detail.

[0061] Figure 1 A flowchart of a chart parsing method provided in an embodiment of the present application is shown, wherein specific steps include S101-S104.

[0062] S101, obtaining a target document.

[0063] In a specific implementation, the target document is in a picture format, such as JPEG, PNG, etc. Since the existing document formats include various formats, such as Word, PPT presentation, portable document format, etc. Therefore, it can be referred to Figure 2 The method flow chart shown is used to obtain the target document, wherein the specific steps include S201-S203.

[0064] S201, obtaining the document to be parsed.

[0065] S202: Determine whether the format of the document to be parsed is a picture format.

[0066] S203: If the format of the document to be parsed is not a picture format, the document to be parsed is format converted to obtain a target document, and the format of the target document is a picture format.

[0067] Here, the document to be parsed may be an academic article, an educational test paper, an economic manuscript, etc., that is, the document to be parsed may come from different fields. Based on this, after obtaining the document to be parsed, it is further determined whether the format of the document to be parsed is a picture format.

[0068] Specifically, determine the file extension of the document to be parsed, that is, check the suffix of the document to be parsed to determine the format of the document to be parsed. For example, if the file extension is .doc, then the format of the document to be parsed is determined to be Word; if the file extension is .ppt, then the format of the document to be parsed is determined to be a PPT presentation; if the file extension is .pdf, then the format of the document to be parsed is determined to be a portable document format, etc.

[0069] If the format of the document to be parsed is a non-image format such as Word, PPT presentation, portable document format, etc., the format of the document to be parsed is converted to obtain a target document, and the format of the target document is a picture format.

[0070] S102, detecting the target document to obtain the type and location information of the target object in the target document.

[0071] After obtaining the target document, the target document is detected to obtain the type and location information of the target object in the target document. It should be noted that the target document includes one or more pictures. For example, if the document to be parsed is a Word document including 10 pages, the target document obtained after the format conversion of the document to be parsed includes 10 pictures. Of course, if the target document includes multiple pictures, each picture included in the target document is detected separately.

[0072] As one example, Figure 3A flow chart of a method for detecting a target document and obtaining the type and location information of a target object in the target document is shown, and specific steps include S301-S304.

[0073] S301, identifying the target document through the detection model to determine the candidate region in the target document.

[0074] S302: For each candidate region, determine the type of content in the candidate region.

[0075] S303: If the type of the content in the candidate area is a target type, determine that the candidate area is a target object. The target type includes a picture and a table.

[0076] S304: Determine the position of the target object in the target document.

[0077] When detecting the target document, the images included in the target document are detected one by one through the pre-trained detection model. Here, considering the characteristics of the detection model, the target document needs to be preprocessed to ensure the accuracy of the detection model.

[0078] Optionally, preprocessing the target document includes performing at least one of the following processing on the image included in the target document according to a preset resolution: reduction, enlargement and translation. Of course, if the resolution of the image included in the target document is consistent with the preset resolution, no preprocessing is required.

[0079] After that, the preprocessed target document is identified by the detection model to determine the candidate area in the target document. As one example, the detection model includes a convolutional neural network (CNN), which extracts the features of the image through its learnable convolutional layer, pooling layer and fully connected layer. For example, the convolutional layer uses a filter to slide on the image to capture local features such as edges and textures; the pooling layer can reduce the spatial dimension of the features to extract important features and reduce the amount of calculation; the fully connected layer maps the extracted features to the final classification or regression task, and the above-mentioned convolutional layer, pooling layer and fully connected layer can all learn automatically, so that CNN can effectively perform image recognition.

[0080] After the target document is recognized, a candidate region in the target document is obtained, where a candidate region is any region in the target document, each candidate region can have a different size, and any two candidate regions can have overlapping regions. Usually, there are multiple candidate regions, but there are also scenarios where there is only one candidate region, such as a picture that only includes a table.

[0081] Furthermore, the detection model determines the type of content in each candidate region, where the types include text, formula, picture, table, etc. And if the type of content in the candidate region is a target type, the candidate region is determined to be a target object, where the target type includes pictures and tables.

[0082] Taking into account that there may be overlapping areas between any two candidate areas, therefore, the same picture or the same table may fall into two or more candidate areas. Based on this, the detection model of the embodiment of the present application is also provided with a technical means of non-maximum suppression (NMS) to screen out accurate target objects from multiple candidate areas through NMS.

[0083] Specifically, when it is determined that there is an overlapping area between two candidate areas and the contents of the two candidate areas are of the same type, the target object is calculated using formula (1).

[0084] IoU(A,B)=Area ofUnion(A,B) / Area ofIntersection(A,B) (1)

[0085] IoU(A, B) represents the intersection over union (IoU) value between candidate region A and candidate region B; Area of ​​Union (A, B) represents the overlapping area between candidate region A and candidate region B; Area of ​​Intersection (A, B) represents the total area of ​​candidate region A and candidate region B.

[0086] As one example, when multiple candidate regions are identified, the confidence corresponding to each candidate region can also be obtained. The confidence represents the accuracy of the identified candidate region and the accuracy of its type. The candidate region A in formula (1) is the candidate region with the highest confidence among all candidate regions, and the candidate region B is the candidate region other than the candidate region A among all candidate regions.

[0087] It is worth mentioning that NMS can also be used to achieve the purpose of screening between categories. That is, in the scenario where the type of the content of the same candidate area is determined to be both a picture and a table, the type of the content of the candidate area can also be accurately determined by performing IoU calculation on the candidate area through NMS.

[0088] After the target object is determined, the position of the target object in the target document is determined, such as the pixel coordinates in the picture included in the target document, etc. Of course, the position of each candidate area in the target document can also be determined while identifying each candidate area, which is not limited in the embodiments of the present application.

[0089] S103: extracting a target object from the target document based on the location information.

[0090] After the target objects in the target document and the position information of each target object are determined, the target objects are extracted from the target document based on each position information.

[0091] As one example, when extracting the target object from the target document based on the location information, the content corresponding to the location information is extracted from the target document, for example, based on the pixel coordinates included in the location information, the corresponding area is cropped from the picture included in the target document. Afterwards, the content corresponding to the location information is determined to be the target object, and of course, the format of the target object is also a picture format.

[0092] S104, analyzing the target object according to the type through the text understanding model, and generating text description information, which can describe the target object.

[0093] After obtaining each target object, each target object and its corresponding type are respectively transmitted to the text understanding model, so that the target object is analyzed according to the type through the text understanding model to generate text description information that can describe the target object.

[0094] As one example, Figure 4 A flow chart of a method for generating text description information is shown, wherein specific steps include S401 and S402.

[0095] S401, extracting text features and visual features of the target object through a text understanding model.

[0096] S402: Generate text description information for the target object based on the type, text features and visual features.

[0097] The text understanding model in the embodiment of the present application is also pre-built and trained. The text features and visual features of the target object are extracted through the text understanding model. Among them, the text understanding model includes a Qwen-VL-Max model. When extracting the text features of the target object, after the text understanding model receives the target object, the target object is identified through the Qwen-VL-Max model to obtain the text features such as the title, axis label and legend included in the target object. In addition, before the target object is identified by the Qwen-VL-Max model, the target object is pre-processed in advance, such as scaling the target object so that the resolution of the target object meets the processing requirements of the text understanding model, such as scaling the target object to a resolution of 448×448 to meet the processing requirements of the text understanding model, thereby ensuring the accuracy of the text understanding model's understanding of the target object.

[0098] When extracting the visual features of the target object, the visual encoder included in the text understanding model is used to extract the features of the target object. Specifically, the target object is divided into multiple blocks through the Vision Transformer (ViT) model, and each block is mapped to a high-dimensional space, and a position code is added to each block to retain its position information in the image. Among them, the ViT model is provided with a multi-head self-attention mechanism and a feedforward network, and when training the ViT model, layer normalization and residual connection are used to optimize the training process, so that the trained ViT model can extract richer visual features of the target object, effectively improving the accuracy of understanding the target object.

[0099] After determining the text features and visual features of the target object, a text description information for the target object is generated based on the type, text features and visual features. Optionally, the text features and visual features are fused through the visual-language adapter, i.e., the projector, included in the text understanding model, and the total features obtained by fusion are transmitted to the large language model included in the text understanding model. After the large language model receives the total features, the total features are serialized and input into a network composed of multiple Transformer layers in combination with the position encoding. After being processed by the multiple Transformer layers, an output sequence that integrates the text features and visual features is generated, and then the text description information is obtained through the decoding strategy. Among them, in each Transformer layer, each element in the sequence is analyzed by a multi-head self-attention mechanism, and a nonlinear transformation is performed using a feedforward network, and when each Transformer layer is trained, layer normalization and residual connection are used to optimize the training process, thereby ensuring the stability of the features and the training efficiency of the network.

[0100] As another example, in order to improve the output quality of the text understanding model, when generating text description information for the target object based on type, text features and visual features, at least one description template is selected based on the type and text features, wherein different types correspond to different description templates, there is at least one text feature, and accordingly, there is at least one description template, wherein one text feature can correspond to one or more description templates, and multiple text features can also correspond to one or more description templates.

[0101] Based on this, text description information for the target object is generated according to the text features, visual features and description template. As an example, the description template may include the following content:

[0102] 1. Objects related to the question being answered;

[0103] 2. Object attributes relevant to answering the question;

[0104] 3. Object relationships related to answering questions;

[0105] The question is: Please give a detailed description of the table or photo in the figure.

[0106] The text description information obtained above not only covers the visual content of the picture or table, but also includes the deeper information implied in the picture or table. For example, for a picture, the text description information may describe the background story in the picture, the interaction between characters, or the role and significance of an object in the scene; for a table, the text description information may describe the overall structure of the data in the table, describe the analyzed data features and data relationships, and even describe the possible impact of the data in the table.

[0107] It is worth noting that corresponding prompt words can be set for different text features and / or visual features. When generating text description information for the target object based on the text features, visual features and description templates, text description information for the target object can be generated based on the text features and their corresponding prompt words, visual features and their corresponding prompt words and description templates, thereby making the text description information more detailed, regular and accurate.

[0108] The embodiment of the present application first detects the target document through the detection model to obtain the target object in the target document and the type and location information of the target object, and extracts the target object based on the location information; then, the target object is analyzed according to the type of the target object through the text understanding model to generate text description information that can describe the target object, thereby realizing the content analysis of the image and / or table, providing convenience for the subsequent verification and maintenance of the document, and at the same time, providing a basis for the subsequent recall of the chart in the database.

[0109] As one example, Figure 5 A flow chart of a method for detecting model training is shown, wherein specific steps include S501-S504.

[0110] S501, obtaining a first training set, the first training set including picture samples and text description samples corresponding to each picture sample and picture samples and text description samples corresponding to each table sample.

[0111] S502: Train the first model using the first training set until the first model meets the accuracy requirement, thereby obtaining a second model.

[0112] S503: Filter out a second training set using a second model.

[0113] S504: Train the second model based on the first training set and the second training set to obtain a detection model.

[0114] In a specific implementation, a large number of pictures and tables are first obtained, wherein the pictures and tables come from test papers, articles, news, etc.; then, data cleaning is performed on the obtained pictures and tables, such as deleting incomplete pictures or tables, deleting pictures with a resolution less than a preset threshold, etc., and the pictures and tables are uniformly adjusted to picture formats such as JPG or PNG, and their resolutions are adjusted to a preset resolution, thereby obtaining picture samples and table samples.

[0115] Among them, in order to ensure that the model can be applied to pictures of different sizes, two preset resolutions can be set: 640x640 and 1280x1280. In addition, for larger pictures, slicing operations can be used, that is, the pictures are divided according to the resolution to obtain pictures that meet the preset resolution, thereby improving its generalization ability and robustness in practical applications.

[0116] At the same time, each image sample is labeled with its corresponding text description sample, and each table sample is labeled with its corresponding text description sample. A first training set is formed based on the image samples and the text description samples corresponding to each image sample and the image samples and the text description samples corresponding to each table sample.

[0117] Furthermore, the first model is trained using the first training set. The first model uses two technical means: affine transformation and bilinear interpolation. Here, affine transformation can achieve operations such as scaling and translation of images through a combination of linear transformation and displacement to maintain the flatness of the image; bilinear interpolation is used to calculate the value of new pixels during the scaling process, which ensures the smooth transition and natural appearance of the generated image by weighted averaging the values ​​of four adjacent pixels. The use of affine transformation and bilinear interpolation at the same time can make image scaling both efficient and have good visual effects, thereby improving the accuracy of image recognition.

[0118] Specifically, when adjusting the resolution of an image through affine transformation, the image is scaled according to the following formula (2).

[0119]

[0120] Among them, (x, y) represents the coordinates of the point in the original image; (x', y') represents the coordinates of the point after transformation; s x Indicates the scaling factor in the x direction; s y represents the scaling factor in the y direction; (t x ,t y ) is the translation vector (if any).

[0121] When generating new pixel values ​​by bilinear interpolation, new pixel values ​​are generated by interpolating in two directions, so that smooth scaling of the image can be achieved, thereby effectively reducing distortion and blur during image scaling and improving image quality. Specifically, refer to the following formulas (3) to (5). Among them, formulas (3) and (4) are used to calculate the new pixel values ​​generated after horizontal interpolation, and formula (5) is used to calculate the new pixel values ​​generated after vertical interpolation.

[0122] R1=Q 11 (1-x)+Q 21 ·x (3)

[0123] R2=Q 12 (1-x)+Q 22 ·x (4)

[0124] P=R1·(1-y)+R2·y (5)

[0125] Among them, R1 represents the 11 and Q 21 The new pixel value generated by horizontal interpolation between Q 12 and Q 22 The new pixel value generated by horizontal interpolation between R1 and R2, P represents the new pixel value generated by vertical interpolation between R1 and R2, Q11 ,Q 12 ,Q 21 ,Q 22 Represents the coordinates of the four corner points of the grid; x and y represent the relative positions relative to the four pixels during the interpolation process (the value range is usually between 0 and 1).

[0126] Through the two technical means of affine transformation and bilinear interpolation, the detection model can be applicable to images of different resolutions, that is, it can process a variety of images, which makes the detection model show better adaptability and flexibility in various application scenarios, while ensuring the accuracy of recognition and analysis.

[0127] After performing the above processing on the image samples and table samples in the first training set, the first model is trained using the first training set. Afterwards, it is monitored whether the first model meets the accuracy requirements, such as if the accuracy is greater than 98%. If not, the model parameters of the first model are adjusted and the first model continues to be trained; if it is satisfied, the second model is obtained.

[0128] After the second model is obtained, the second training set is screened out by the second model. Specifically, some image samples and table samples in the first training set are input into the second model to obtain the actual text descriptions corresponding to the image samples and the actual text descriptions corresponding to the table samples. The image samples and their corresponding actual text descriptions and the table samples and their corresponding actual text descriptions constitute the second training set.

[0129] After obtaining the second training set, the second model is trained based on the first training set and the second training set to obtain a detection model. As an example, when the second model is trained based on the first training set and the second training set, part of the training data is selected from the first training set and the second training set according to a preset ratio to obtain a third training set, so as to train the second model using the third training set until the second model meets the accuracy requirement to obtain a detection model.

[0130] Among them, the embodiment of the present application performs incremental learning based on the YOLOv8x model, that is, the retrieval model includes the YOLOv8x model. Incremental learning enables the model to gradually learn and integrate new knowledge from new data while retaining existing knowledge. Corresponding to the present application, the second model is trained based on the first training set and the second training set, so that the trained detection model can learn and integrate the model knowledge corresponding to the second training set, and also retain the model knowledge corresponding to the first training set. This incremental learning method allows the model to be updated online when new data arrives without retraining the entire model, thereby effectively improving the learning efficiency of the model, and also enabling the model to adapt to changes in data distribution, reducing performance degradation caused by data updates, and ensuring the efficiency and stability of the model when processing diversified data and adapting to new information.

[0131] Optionally, the preset ratio can be set to 2:8, that is, the third training set includes 20% of the data in the first training set and 80% of the data in the second training set. In addition, the data in the second training set is calculated by the second model, and the actual text descriptions corresponding to the picture samples and the actual text descriptions corresponding to the table samples calculated by the second model have different accuracies. Based on this, the actual text descriptions corresponding to the picture samples and the actual text descriptions corresponding to the table samples are sorted from high to low according to their accuracy, and part of the data (80% of the data in the third training set) is selected from high to low according to the sorting as the data in the third training set.

[0132] As another example, the detection model can be trained in real time. In subsequent training, each training set can determine the third training set according to other ratios, and the embodiments of the present application will not be described in detail.

[0133] For example, Figure 6 A target document is shown, the target document is a picture, the target document is detected, and the target object in the target document and the type and location information of the target object are obtained. Figure 6 There are two target objects in the example. The first target object is a picture. Figure 6 The second target object type is table ( Figure 6 In the table, Figure 6 The location information of the first target object and the second target object is not shown. Then, based on the location information, the target object is extracted from the target document, that is, Figure 6 The area corresponding to the two rectangular boxes in the figure. Figure 6After the first target object and the second target object are identified, the first target object is analyzed according to the type of the first target object through the text understanding model to generate text description information corresponding to the first target object, and the second target object is analyzed according to the type of the second target object to generate text description information corresponding to the second target object.

[0134] The text description information corresponding to the first target object is: A scene of a child using a telescope to observe the starry sky. In the picture, a child stands in front of a telescope, concentrating on observing the starry sky. The background is a sky full of stars, creating a tranquil and mysterious atmosphere. The child seems to be full of curiosity and desire to explore the starry sky, symbolizing the desire for the unknown world and the love of science.

[0135] The text description information corresponding to the second target object is: multiple government-related policies and measures are listed. The title of the table is "Government-related policies and measures", and the content includes multiple items, each of which describes the government's policies and measures in different fields. For example, in July 2023, the Office of the Inter-ministerial Joint Conference on the Implementation of the State Council's Intellectual Property Strategy issued the "XXX Implementation and Promotion Plan"; on December 17, 2023, the State Council promulgated the "Regulations on the Supervision and Administration of Non-Bank Payment Institutions", which will come into effect on May 1, 2024; in 2023, my country continued, optimized and improved nearly 70 tax and fee preferential policies, coordinated the use of 3.8 trillion yuan of new special debt limits and 680 billion yuan of central budget investment to stimulate the endogenous power and vitality of the economy; in May 2023, the "Implementation Plan for the Coordinated Development of Beijing-Tianjin-Hebei Industries" was issued, the Beijing-Tianjin-Hebei Industrial Chain Supply Chain Conference was held, and the first cross-regional industrial chain map was jointly released... The industrial chains of the three places grew together, and the level of industrial cooperation continued to improve; in October 2023, the National Data Bureau was officially established to coordinate and promote the construction of data infrastructure systems, coordinate the integration, sharing and development and utilization of data resources, and coordinate the construction of digital China, digital economy, and digital society. These items describe in detail the government's specific policies and measures in the fields of intellectual property, financial supervision, tax and fee preferences, coordinated industrial development, and data management.

[0136] Based on the same inventive concept, the second aspect of the present application also provides a chart analysis device corresponding to a chart analysis method. Since the principle of solving the problem by the chart analysis device in the present application is similar to the above-mentioned chart analysis method of the present application, the implementation of the interactive device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0137] Figure 7 The structure diagram of the chart parsing device provided in the embodiment of the present application is shown, which specifically includes:

[0138] An acquisition module 701 is configured to acquire a target document;

[0139] A detection module 702 is configured to detect the target document to obtain a target object in the target document and type and location information of the target object;

[0140] An extraction module 703, configured to extract the target object from the target document based on the location information;

[0141] The generating module 704 is configured to analyze the target object according to the type through a text understanding model, and generate text description information, wherein the text description information can describe the target object.

[0142] In another example, the obtaining module 701 is specifically configured as follows:

[0143] Get the document to be parsed;

[0144] Determine whether the format of the document to be parsed is a picture format;

[0145] If the format of the document to be parsed is not a picture format, the document to be parsed is format converted to obtain the target document, and the format of the target document is a picture format.

[0146] In yet another example, the detection module 702 is specifically configured as follows:

[0147] Identify the target document through a detection model to determine a candidate region in the target document;

[0148] For each candidate region, determining the type of content in the candidate region;

[0149] If the type of the content in the candidate area is a target type, the candidate area is determined to be a target object, and the target type includes a picture and a table;

[0150] The position of the target object in the target document is determined.

[0151] In yet another example, the extraction module 703 is specifically configured as follows:

[0152] extracting content corresponding to the location information from the target document;

[0153] It is determined that the content corresponding to the location information is the target object, and the format of the target object is a picture format.

[0154] In yet another example, the generating module 704 is specifically configured as follows:

[0155] Extracting text features and visual features of the target object through a text understanding model;

[0156] Based on the text features and the visual features, text description information for the target object is generated.

[0157] In yet another example, the generating module 704 is further configured to:

[0158] Based on the type and the text feature, selecting at least one description template, the text feature being at least one;

[0159] Generating text description information for the target object according to the text feature, the visual feature and the description template.

[0160] In yet another example, the chart parsing apparatus further includes a training module 705, which is configured to:

[0161] Obtain a first training set, wherein the first training set includes an image sample and a text description sample corresponding to each image sample and an image sample and a text description sample corresponding to each table sample;

[0162] Using the first training set to train the first model until the first model meets the accuracy requirement, thereby obtaining a second model;

[0163] Filter out a second training set using the second model;

[0164] The second model is trained based on the first training set and the second training set to obtain the detection model.

[0165] In yet another example, the training module 705 is further configured to:

[0166] According to a preset ratio, select part of the training data from the first training set and the second training set respectively to obtain a third training set;

[0167] The second model is trained using the third training set until the second model meets the accuracy requirement, thereby obtaining the detection model.

[0168] The embodiment of the present application first detects the target document through the detection model to obtain the target object in the target document and the type and location information of the target object, and extracts the target object based on the location information; then, the target object is analyzed according to the type of the target object through the text understanding model to generate text description information that can describe the target object, thereby realizing the content analysis of the image and / or table, providing convenience for the subsequent verification and maintenance of the document, and at the same time, providing a basis for the subsequent recall of the chart in the database.

[0169] An embodiment of the present application provides a storage medium, which is a computer-readable medium and stores a computer program. When the computer program is executed by a processor, the method provided by any embodiment of the present application is implemented, including the following steps S11 to S14:

[0170] S11, obtaining the target document;

[0171] S12, detecting the target document to obtain a target object in the target document and type and location information of the target object;

[0172] S13, extracting the target object from the target document based on the location information;

[0173] S14, analyzing the target object according to the type through a text understanding model to generate text description information, where the text description information can describe the target object.

[0174] An embodiment of the present application provides an electronic device. The structural diagram of the electronic device can be as follows: Figure 8 As shown, at least a memory 801 and a processor 802 are included, the memory 801 stores a computer program, and the processor 802 implements the method provided by any embodiment of the present application when executing the computer program on the memory 801. Exemplarily, the steps of the electronic device computer program are as follows S21 to S24:

[0175] S21, obtaining the target document;

[0176] S22, detecting the target document to obtain a target object in the target document and type and location information of the target object;

[0177] S23, extracting the target object from the target document based on the location information;

[0178] S24, analyzing the target object according to the type through a text understanding model to generate text description information, where the text description information can describe the target object.

[0179] The embodiment of the present application first detects the target document through the detection model to obtain the target object in the target document and the type and location information of the target object, and extracts the target object based on the location information; then, the target object is analyzed according to the type of the target object through the text understanding model to generate text description information that can describe the target object, thereby realizing the content analysis of the image and / or table, providing convenience for the subsequent verification and maintenance of the document, and at the same time, providing a basis for the subsequent recall of the chart in the database.

[0180] Optionally, in this embodiment, the above-mentioned storage medium may include, but is not limited to, various media that can store program codes, such as a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk. Optionally, in this embodiment, the processor executes the method steps recorded in the above-mentioned embodiment according to the program code stored in the storage medium. Optionally, the specific examples in this embodiment can refer to the examples described in the above-mentioned embodiment and the optional implementation mode, and this embodiment is not repeated here. Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present application can be implemented with a general computing device, which can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices. Optionally, they can be implemented with executable program codes of computing devices, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from here, or they can be made into individual integrated circuit modules respectively, or multiple modules or steps in them can be made into a single integrated circuit module to implement. In this way, the present application is not limited to any specific combination of hardware and software.

[0181] In addition, although exemplary embodiments have been described herein, the scope includes any and all embodiments based on the present application with equivalent elements, modifications, omissions, combinations (e.g., various embodiments intersecting schemes), adaptations or changes. The elements in the claims will be interpreted broadly based on the language adopted in the claims, and are not limited to the examples described in this specification or during the implementation of the application, and the examples will be interpreted as non-exclusive. Therefore, this specification and examples are intended to be considered as examples only, and the true scope and spirit are indicated by the following claims and the full scope of their equivalents.

[0182] The above description is intended to be illustrative rather than restrictive. For example, the above examples (or one or more of them) can be used in combination with each other. For example, those of ordinary skill in the art can use other embodiments when reading the above description. In addition, in the above-mentioned specific embodiments, various features can be grouped together to simplify the application. This should not be interpreted as an intention that a disclosed feature that does not require protection is necessary for any claim. On the contrary, the subject matter of the present application may be less than all the features of a specific disclosed embodiment. Thus, the following claims are incorporated into the specific embodiments as examples or embodiments, wherein each claim is independently used as a separate embodiment, and it is considered that these embodiments can be combined with each other in various combinations or arrangements. The scope of the present application should be determined with reference to the attached claims and the full scope of equivalent forms granted by these claims.

[0183] Multiple embodiments of the present application are described in detail above, but the present application is not limited to these specific embodiments. Those skilled in the art can make various variations and modifications to the embodiments based on the concept of the present application, and these variations and modifications should all fall within the scope of protection required by the present application.

Claims

1. A chart analysis method, characterized in that: include: Get the target document; Detecting the target document to obtain a target object in the target document and type and location information of the target object; extracting the target object from the target document based on the location information; The target object is analyzed according to the type through a text understanding model to generate text description information, which can describe the target object.

2. The chart analysis method according to claim 1, characterized in that: The obtaining of the target document comprises: Get the document to be parsed; Determine whether the format of the document to be parsed is a picture format; If the format of the document to be parsed is not a picture format, the document to be parsed is format converted to obtain the target document, and the format of the target document is a picture format.

3. The chart analysis method according to claim 1, characterized in that: The detecting the target document to obtain the target object in the target document and the type and location information of the target object includes: Identify the target document through a detection model to determine a candidate region in the target document; For each candidate region, determining the type of content in the candidate region; If the type of the content in the candidate area is a target type, the candidate area is determined to be a target object, and the target type includes a picture and a table; The position of the target object in the target document is determined.

4. The chart analysis method according to claim 1, characterized in that: The step of extracting the target object from the target document based on the location information includes: extracting content corresponding to the location information from the target document; It is determined that the content corresponding to the location information is the target object, and the format of the target object is a picture format.

5. The chart analysis method according to claim 1, characterized in that: The method of analyzing the target object according to the type through the text understanding model to generate text description information includes: Extracting text features and visual features of the target object through a text understanding model; Based on the type, the text feature and the visual feature, text description information for the target object is generated.

6. The chart analysis method according to claim 5, characterized in that: The generating text description information for the target object based on the type, the text feature and the visual feature includes: Based on the type and the text feature, selecting at least one description template, the text feature being at least one; Generating text description information for the target object according to the text feature, the visual feature and the description template.

7. The chart analysis method according to claim 3, characterized in that: The training steps of the detection model are also included: Obtain a first training set, wherein the first training set includes an image sample and a text description sample corresponding to each image sample and an image sample and a text description sample corresponding to each table sample; Using the first training set to train the first model until the first model meets the accuracy requirement, thereby obtaining a second model; Filter out a second training set using the second model; The second model is trained based on the first training set and the second training set to obtain the detection model.

8. The chart analysis method according to claim 7, characterized in that: The step of training the second model based on the first training set and the second training set to obtain the detection model includes: According to a preset ratio, select part of the training data from the first training set and the second training set respectively to obtain a third training set; The second model is trained using the third training set until the second model meets the accuracy requirement, thereby obtaining the detection model.

9. A chart analysis device, characterized in that: include: an acquisition module configured to acquire a target document; A detection module, configured to detect the target document to obtain a target object in the target document and type and location information of the target object; an extraction module, configured to extract the target object from the target document based on the location information; A generation module is configured to analyze the target object according to the type through a text understanding model to generate text description information, wherein the text description information can describe the target object.

10. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via a bus. When the machine-readable instructions are executed by the processor, the steps of the chart parsing method as described in any one of claims 1 to 8 are performed.

Images