Chart extraction method and device and storage medium

Abstract

The invention provides a chart extraction method and device and a storage medium, and relates to the technical field of chart processing. The extraction method comprises the steps of obtaining a first image and identifying a chart category of the first image; determining a second image based on a preset chart category and the first image; identifying and determining a graphic area and a non-graphic area of the second image; masking the non-graphic area, identifying chart data in the non-graphic area, and extracting a first pixel coordinate corresponding to the chart data; constructing a first mapping from a pixel coordinate system to a plane coordinate system; in the graph area, identifying a chart graph, and extracting a second pixel coordinate corresponding to the chart graph; and determining plane coordinates of the chart graph in the plane coordinate system, and generating a chart data packet. According to the method, the target detection model and the multi-modal large language model are combined, and automatic extraction of the chart data is realized by covering the interference elements, establishing the mapping relation and realizing curve separation and coordinate conversion based on the color gradient.

Description

Technical Field

[0001] This application relates to the field of chart processing technology, and in particular to a chart extraction method, apparatus and storage medium. Background Technology

[0002] In scientific research, massive amounts of experimental data and research results are often presented in academic papers in the form of charts and graphs. These charts and graphs contain key trend information and numerical data. However, current scientific literature is mainly stored in PDF or image formats, and the chart and graph data in them is unstructured, making it impossible for machines to directly read, index, or use for secondary analysis. Existing methods for extracting chart and graph data still have the following problems: First, scientific images often contain multiple sub-graphs, and traditional rule-based or simple convolutional neural network methods are difficult to accurately segment and locate specific lines within them; second, traditional optical character recognition technology struggles to understand the logical correspondence between scale values ​​and coordinate axes when processing coordinate axis scales, and is easily affected by text, annotations, or other interference items within the graph; third, when colors are similar or overlap, traditional image processing algorithms (such as color thresholding) struggle to accurately distinguish different data series, leading to confusion in the extracted data points; finally, existing tools are often designed for specific chart styles and lack a universal understanding of different color schemes and coordinate systems (such as logarithmic coordinates).

[0003] With the development of multimodal large language models, their powerful image semantic understanding and zero-shot / few-shot learning capabilities offer new avenues for solving the aforementioned problems. Therefore, there is an urgent need for a graph data extraction method that combines the efficient segmentation capabilities of object detection models with the semantic cognitive abilities of multimodal large language models. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method, apparatus and storage medium for extracting charts. The method combines a target detection model and a multimodal large language model to mask the interference areas in the chart image, establish a mapping relationship from the pixel coordinate system to the data coordinate system, and realize the separation and coordinate transformation of multiple data curves based on color gradient, ultimately realizing the automated extraction of structured data from the chart.

[0005] In a first aspect, a chart extraction method is provided, comprising: acquiring a first image; identifying the chart category of the first image, the chart category including: line chart, bar chart, or pie chart; determining a second image based on a preset chart category and the first image; identifying the graphic region and non-graphic region of the determined second image; masking the non-graphic region; identifying chart data in the non-graphic region and extracting the first pixel coordinates corresponding to the chart data; constructing a first mapping from the pixel coordinate system to the plane coordinate system based on the chart data and the first pixel coordinates; identifying the chart graphic in the graphic region and extracting the second pixel coordinates corresponding to the chart graphic; determining the plane coordinates of the chart graphic in the plane coordinate system based on the second pixel coordinates and the first mapping, and generating a chart data package.

[0006] Optionally, the default chart type is line chart.

[0007] Optionally, identifying chart data in a non-graphic area and extracting the first pixel coordinates corresponding to the chart data includes: identifying at least one scale data area of ​​the coordinate axis in the chart data, and determining the first pixel coordinates of the scale data area in a pixel coordinate system based on the feature pixels in the scale data area.

[0008] Optionally, a first mapping from the pixel coordinate system to the plane coordinate system is constructed based on the chart data and the first pixel coordinates, including: determining a data pair based on the scale content of the scale data area and the first pixel coordinates, wherein the scale content is determined based on a preset language model; determining the scale rules of the plane coordinate system based on multiple scale contents corresponding to the coordinate axes of the plane coordinate system; and determining the first mapping from the pixel coordinate system to the plane coordinate system based on the scale rules and the data pair, wherein the scale rules include linear scale rules and non-linear scale rules.

[0009] Optionally, in the graphics area, identifying the chart graphic and extracting the second pixel coordinates corresponding to the chart graphic includes: identifying and determining the first line in the line graph and the color information of the first line; determining the color gradient of multiple first pixels of the first line based on the color information and a preset color gradient threshold; determining the pixel-level mask of the first line based on the color gradient; and determining the second pixel coordinates of the first line based on the pixel-level mask of the first line.

[0010] Optionally, the line graph further includes: a second line, wherein the second line intersects with the first line; comparing the color information of the intersection with the color gradient of the first line and the color gradient of the second line respectively; when the color information of the intersection belongs to the color gradient of the first line, the intersection is determined as a point on the first line, and a pixel-level mask of the first line is determined based on the intersection and multiple first pixel points on the first line; when the color information of the intersection belongs to the color gradient of the second line, the intersection is determined as a point on the second line, and a pixel-level mask of the second line is determined based on the intersection and multiple second pixel points on the second line; when the color information of the intersection belongs to both the color gradient of the first line and the color gradient of the second line, the intersection is determined to belong to both the first and second lines, and a pixel-level mask of the first line is determined based on the intersection and multiple first pixel points, and a pixel-level mask of the second line is determined based on the intersection and multiple second pixel points.

[0011] Optionally, based on the second pixel coordinates and the first mapping, the planar coordinates of the chart graphic in the planar coordinate system are determined, and a chart data package is generated, including: determining a first set of second pixel coordinates of the first line and a second set of second pixel coordinates of the second line based on the pixel-level mask of the first line and the pixel-level mask of the second line; determining a set of planar coordinates of the chart graphic based on the first set, the second set and the first mapping; and generating a chart data package based on the set of planar coordinates and a preset data structure.

[0012] Optionally, the first image includes multiple sub-images. Before identifying the chart category of the first image, the method further includes segmenting the first image into multiple sub-images and determining at least one second image based on a preset chart category and multiple sub-images.

[0013] Secondly, a chart extraction device is provided, comprising: an identification unit for acquiring a first image and identifying the chart category of the first image, the chart category including: line chart, bar chart, or pie chart; a determination unit for determining a second image based on a preset chart category and the first image; the identification unit is further configured to identify and determine graphic and non-graphic regions of the second image, and to mask the non-graphic regions, identify chart data in the non-graphic regions, and extract the first pixel coordinates corresponding to the chart data; a mapping unit for constructing a first mapping from a pixel coordinate system to a plane coordinate system based on the chart data and the first pixel coordinates; the identification unit is further configured to identify chart graphics in the graphic regions and extract the second pixel coordinates corresponding to the chart graphics; the determination unit is further configured to determine the plane coordinates of the chart graphics in the plane coordinate system based on the second pixel coordinates and the first mapping, and generate a chart data package.

[0014] Thirdly, a computer-readable storage medium is provided having instructions stored thereon, which, when executed by a processor, implement a chart extraction method as described above.

[0015] In summary, the chart extraction method disclosed in this application has at least the following beneficial effects: (1) By cascading the target detection model with the multimodal large model, the layout analysis and sub-graph segmentation of the composite chart are automated, and the chart type identification and screening of the segmented sub-graphs are performed. This achieves full-process automation of automatic positioning, segmentation and identification of target charts, which significantly improves the efficiency of chart extraction without manual intervention. (2) The semantic segmentation technology is used to identify the graphic and non-graphic areas in the chart, and the non-graphic areas such as coordinate axes, scales, and legends are masked at the pixel level, which effectively eliminates the interference of non-data elements on the core data graphic extraction process, thereby ensuring the accuracy of subsequent data extraction. (3) By calling the ability of multimodal large model to fuse optical character recognition and semantic understanding, the scale text can be accurately identified and the scale pattern can be analyzed. Thus, linear or nonlinear models can be selected to construct the mapping relationship between pixels and data, making the method compatible with multiple coordinate axis types such as linear, logarithmic, and exponential, which greatly expands the applicability of the method. (4) By combining the color semantic prior provided by the multimodal large model, the color gradient threshold is defined in the HSV color space and the pixel ownership is determined based on spatial continuity, which effectively solves the problem of separation when the traditional algorithm processes graphics with similar, intersecting or overlapping colors. Attached Figure Description

[0016] The following is a brief introduction to the accompanying drawings used in the description of the embodiments of this application: Figure 1 A flowchart of a chart extraction method provided in an embodiment of this application is shown; Figure 2 Another flowchart of a chart extraction method provided in an embodiment of this application is shown. Detailed Implementation

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, examples of implementation methods of this application will be described below with reference to the accompanying drawings. The accompanying drawings described below are merely some embodiments of this application. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without creative effort. Adjustments and improvements made without departing from the concept of this application are all within the protection scope of this application.

[0018] To keep the drawings simple, each figure only schematically shows the parts relevant to the embodiment, and they do not represent the actual structure of the product. In addition, for the sake of clarity and ease of understanding, some figures only schematically show parts of components with the same structure or function, and there may actually be more or fewer components with the same structure or function.

[0019] In this application, unless otherwise expressly specified and limited, ordinal numbers, such as "first," "second," etc., are used only to distinguish and describe related objects, and should not be construed as indicating or implying the relative importance or order between related objects; furthermore, they do not represent the quantity of related objects. "Multiple" includes two or more, and other quantifiers are similar. " / " is used to describe the relationship between related objects, indicating an "or" relationship between them. "And / or" is used to describe the relationship between related objects, including any combination relationship between them, such as "a and / or b" including: "a alone," "b alone," or "a and b." "One or more" or "at least one" of multiple objects refers to any object or any combination of multiple objects, such as "one or more of a1, a2, a3" or "at least one of a1, a2, a3" including: "a1 alone," "a2 alone," "a3 alone," "a1 and a2," "a1 and a3," "a2 and a3," or "a1, a2 and a3."

[0020] With the widespread use of digital documents such as scientific research literature and technical reports, charts and graphs have become an important form of data presentation. Automatically extracting structured data from charts and graphs is crucial for scientific research analysis and knowledge summarization, and has become a key technological challenge in the field of document image understanding and information extraction.

[0021] Currently, existing chart data extraction methods have significant limitations when dealing with complex real-world scenarios: traditional methods struggle to achieve high-precision segmentation and positioning for complex layouts containing multiple subgraphs; when identifying axis scales, they lack an understanding of the chart's semantics, are easily influenced by text within the chart, and struggle to establish a logical connection between numerical values ​​and coordinates; when graphic colors are similar or overlap, traditional image processing methods struggle to accurately separate different data sequences; furthermore, existing methods are typically designed for specific chart styles and lack generalization ability for scenarios with non-linear coordinates and diverse color schemes, limiting the breadth and reliability of practical applications.

[0022] In recent years, significant breakthroughs have been achieved in multimodal large-scale models based on large-scale pre-training. These models, trained on large datasets of images and text, possess powerful cross-modal understanding capabilities, enabling them to deeply analyze the semantic relationships between visual content and textual information in images. Multimodal large-scale models can not only identify objects in images but also understand their functional roles, infer their logical relationships, and complete reasoning tasks that require combining visual and linguistic knowledge. This capability makes them suitable for scenarios such as chart understanding that require integrated visual feature recognition and semantic logic analysis, providing a new technological foundation for building more intelligent chart data extraction systems.

[0023] In view of this, this application proposes a method for extracting charts. By leveraging the advantages of multimodal large models in semantic parsing and contextual reasoning, and combining the spatial positioning capabilities of computer vision technology, this method systematically improves the accuracy of layout segmentation, semantic understanding and association capabilities, and achieves precise separation of multiple data sequences, significantly enhancing the adaptability and generalization ability for diverse charts.

[0024] The following description is in conjunction with the accompanying drawings.

[0025] Please refer to Figure 1 The document illustrates a flowchart of a chart extraction method provided in an embodiment of this application. This chart extraction method can be implemented using a multimodal large-scale language model and includes at least the following steps: S110, acquire the first image, identify the chart type of the first image, the chart type includes: line chart, bar chart or pie chart; S120, determine the second image based on the preset chart category and the first image; S130, identify and determine the graphic and non-graphic regions of the second image; S140, mask the non-graphic area, identify the chart data in the non-graphic area, and extract the first pixel coordinates corresponding to the chart data; S150, construct the first mapping from the pixel coordinate system to the plane coordinate system based on the chart data and the first pixel coordinates; S160, In the graphics area, identify the chart graphic and extract the second pixel coordinates corresponding to the chart graphic; S170, based on the second pixel coordinates and the first mapping, determine the planar coordinates of the chart graphic in the planar coordinate system, and generate a chart data package.

[0026] The first image can originate from scanned documents, screenshots of scientific literature, or report files containing charts. A preliminary analysis of the first image is performed to identify its overall chart category. This can be done by inputting the first image into a multimodal large-scale model and constructing specific classification prompts to guide the model in analyzing the semantic features of the first image content and determining its chart category. Chart categories can include line charts, bar charts, pie charts, and other charts used for data visualization. When the recognition result of the first image matches the preset chart category, the target image requiring further processing, i.e., the second image, can be determined. For example, based on the model's output, logical judgments can be made to automatically select sub-charts labeled "line chart" as the core object for subsequent processing, while other types of sub-charts can be archived or removed. In some implementations, when the first image itself is a single chart and the chart category matches, the entire image can be used as the second image; if the first image is a composite image containing multiple sub-charts, it can first be divided into several independent sub-charts, and then the sub-charts that match the category can be selected as the second image.

[0027] After determining the second image, identify and distinguish between graphic and non-graphic regions within it. Graphic regions primarily refer to the parts displaying the main data, such as data curves in a line graph or data bars in a bar chart. Non-graphic regions may include elements such as coordinate axes, tick labels, legends, and annotation text. The visual understanding capabilities of the multimodal large model can be utilized to identify the specific location information of non-data elements in the image. For example, based on the coordinate information returned by the multimodal large model, a corresponding binary mask image can be generated, and graphic and non-graphic regions can be identified separately in the second image.

[0028] The chart data within the masked non-graphical area can include axis tick values, legend data, and other content. When using multimodal large model optical character recognition technology, the tick text on the axes can be accurately located and identified, and the position of each tick text in the image can be recorded. When establishing a pixel coordinate system and identifying the first pixel coordinate of the tick text, a mapping relationship from the image pixel coordinate system to the planar coordinate system can be constructed after obtaining the tick text and its pixel coordinates. For example, multimodal large model analysis can be used to analyze the mathematical relationship between the set of tick text and its corresponding pixel positions, and the corresponding mathematical fitting formula can be automatically adapted to calculate the first mapping from the pixel coordinate system to the planar coordinate system using linear regression or nonlinear fitting algorithms. Within the graphic area, when identifying and extracting the second pixel coordinates corresponding to the chart graphic of the displayed data, the extracted second pixel coordinates can be substituted into the constructed first mapping to convert them into planar coordinates. Finally, all the converted planar coordinates, along with the legend of the non-graphical area, can be packaged into a structured chart data package, thus completing the automated extraction process from the chart image to computable data.

[0029] The chart data extraction method in this embodiment can utilize a multimodal large language model to filter target charts and separate target areas through chart category identification and preprocessing steps, thereby effectively eliminating interference from irrelevant information in the charts to be identified. Furthermore, by constructing an adaptive coordinate mapping relationship, it can be compatible with linear and nonlinear coordinate systems in the charts, improving the efficiency of processing complex scientific charts and assisting researchers in completing the task of processing massive amounts of scientific data.

[0030] In some implementations, identifying chart data in non-graphical areas and extracting the first pixel coordinates corresponding to the chart data includes: Identify at least one tick data region of the coordinate axis in the chart data, and determine the first pixel coordinate of the tick data region in the pixel coordinate system based on the feature pixels in the tick data region.

[0031] On the segmented non-graphic region image, coordinate axis regions are identified. Based on the positional information of the coordinate axes, text regions parallel to the coordinate axes are selected. These regions contain scale data areas. The scale content of the text region next to each axis is identified, and optical character recognition (e.g., PaddleOCR) can be used to filter out the rectangular bounding boxes of the scale data regions.

[0032] A representative pixel can be selected from the rectangular area of ​​the scale data region as a feature pixel, and its pixel coordinates are used as the first pixel coordinates of that scale data region. The selection method for the feature pixel is not limited. For example, the geometric center of the rectangular area of ​​the scale data can be taken as the feature pixel; or for the horizontal X-axis, the midpoint of the bottom edge of the rectangular area of ​​the scale data region can be taken as the feature pixel; for the vertical Y-axis, the midpoint of the left side of the rectangular area of ​​the scale data region can be taken as the feature pixel; or the nearest neighbor method can be used, calculating the distance from each pixel in the scale data region to the coordinate axis and selecting the nearest pixel as the feature pixel. After determining the feature pixel for each scale data region, its pixel coordinates are extracted as the first pixel coordinates of the scale data region. The method of selecting feature pixels can be set by inputting commands to the large language model.

[0033] In some implementations, a first mapping from the pixel coordinate system to the planar coordinate system is constructed based on the chart data and the first pixel coordinates, including: Based on the scale content and the first pixel coordinates of the scale data area, a data pair is determined, and the scale content is determined based on a preset language model. Based on the multiple scale values ​​corresponding to the coordinate axes of the planar coordinate system, determine the scale rules of the coordinate axes of the planar coordinate system; Based on the scale pattern and data pairs, the first mapping from the pixel coordinate system to the plane coordinate system is determined. The scale pattern includes linear scale patterns and non-linear scale patterns.

[0034] Based on the above embodiments, a preset language model can be used to recognize characters within the scale data area, convert them into numerical values, and thus obtain the scale content of each scale data area. The scale content of each scale data area and its corresponding first pixel coordinate are considered as a data pair. Taking the X-axis as an example, scale data areas near the X-axis are selected, and the scale content and the horizontal coordinate value of the first pixel coordinate are considered as a data pair. Similarly, data pairs corresponding to other coordinate axes are determined, and the data pairs corresponding to each coordinate axis are considered as a set of data pairs. For example, a multimodal large model can be used to input the Prompt command: "Based on the content and position of the given scale data recognition box, determine what the scale values ​​of the X-axis and Y-axis are respectively, and output JSON format, T{Pixel, Value}, where Pixel uses the coordinates of the lower right corner of the recognition box, and Value is the recognized scale value; if no scale value exists, the output is empty." Taking the X-axis as an example, the data pair set for the X-axis is [(P1,V1),(P2,V2),...,(P m V m )], where P n V represents the x-coordinate of the first pixel. n This represents the scale content. By selecting at least two data pairs from the same coordinate axis data set, and using the difference between the pixel coordinate values ​​and the difference between the scale content, it is possible to determine whether the scale pattern of that coordinate axis is linear or non-linear. Taking the X-axis as an example, the scale content {V1, V2, ..., V...} of the X-axis data pairs can be analyzed. m The pattern between}. Calculate the difference between adjacent values. If all If all values ​​are equal, it indicates that the scale content is uniformly distributed in the data value space, thus determining that the coordinate axis has a linear scale pattern. If... They are not equal; the ratio of adjacent values ​​can be further calculated. If all If all axes are equal, it can be determined that the coordinate axis has a non-linear scale pattern, such as exponential or logarithmic coordinates. "Equal" here refers to approximately equal axes within a small, permissible error range, the specific error range of which can be set by the user. The same method can be used to determine this for other coordinate axes.

[0035] Based on the scale pattern of the coordinate axes, a corresponding mathematical fitting model is selected to calculate the first mapping from the pixel coordinate system to the planar coordinate system. Taking the X-axis as an example, for a linear scale pattern, assuming the pixel coordinate value P... n With scale content V nIf a linear relationship exists, then construct a linear regression equation, referring to Formula 1. Formula 1 Where X is the coordinate value of the X-axis in the planar coordinate system, and x is the coordinate value of the x-axis in the pixel coordinate system; a is the scaling factor, representing the data value corresponding to each pixel on the x-axis in the pixel coordinate system; b is the offset, representing the data value of the planar coordinate X-axis when the x-axis coordinate in the pixel coordinate system is x=0. Based on the data pairs corresponding to the X-axis, the values ​​of a and b can be solved using the least squares method or the two-point method. Substituting the values ​​of a and b into the linear regression equation, the mapping function in the X-axis direction is obtained. For non-linear scale patterns, taking logarithmic coordinates as an example, a fitting formula is constructed, referring to Formula 2. Formula 2 Similarly, the least squares method or the two-point method can be used to calculate the specific values ​​of a and b. Here, c is the base of the logarithmic function, numerically equal to... Substituting the values ​​of a, b, and c into the fitting equation for logarithmic coordinates yields the mapping function along the X-axis. Calculate the mapping function for all coordinate axes to determine the first mapping. Commands can be entered into the multimodal large model to automatically extract data pairs and calculate the mapping function.

[0036] Most existing technologies rely solely on traditional optical character recognition for scale identification, which is prone to false character detection (such as mistaking noise for decimal points or recognizing numbers in the scale data area) and fails to understand the logical incrementing relationship of scale values ​​on the coordinate axis. This solution combines the semantic understanding capabilities of multimodal large models with optical character recognition, transforming character recognition into coordinate logic recognition, thus overcoming the limitation of existing technologies that are only applicable to linear coordinate systems.

[0037] In some implementation methods, please refer to Figure 2 This illustrates another flowchart of a chart extraction method provided in an embodiment of this application. In these embodiments, the preset chart type is a line chart. In the graphics area, the chart graphic is identified, and the second pixel coordinates corresponding to the chart graphic are extracted, including at least the following steps: S210, Identify and determine the first line in the line drawing and the color information of the first line; S220, Based on color information and a preset color gradient threshold, determine the color gradient of multiple first pixels of the first line; S230, determine the pixel-level mask of the first line based on the color gradient; S240, determine the second pixel coordinates of the first line based on the pixel-level mask of the first line.

[0038] A line chart is a two-dimensional chart with a continuous independent variable axis and a dependent variable axis, forming a Cartesian coordinate system. Data points are marked and connected sequentially by line segments to form a continuous trajectory. Examples include line charts and curve charts. Line charts are primarily used to display trends, changes, and relationships.

[0039] When the preset chart type is a line chart, the judgment conditions for the second image can be set based on the content characteristics of the line chart. For example, it can be identified whether the second image has coordinate axis regions: features such as a pair of approximately perpendicular straight line segments, arrows at the ends of the line segments, and scale markings on the line segments; or whether the second image contains continuous or discontinuous line segments, and whether it contains auxiliary elements such as grid lines and legends. If all or most of the above features are met, the second image can be determined to be a line chart. Furthermore, the visual understanding capabilities of the multimodal large model can be used to handle interfering elements within the chart. For example, by sending instructions to the large model, the specific location information of non-data line elements such as coordinate axis regions, legends, text annotations, and background grids in the chart can be identified, and elements that may interfere with subsequent data line recognition can be masked.

[0040] The second image can be converted to the HSV color space, where H is hue, used to distinguish different color categories, typically ranging from 0 to 360°, and 0 to 179 in computer vision libraries; S is saturation, used to represent the purity of a color, ranging from 0 to 100%, and 0 to 255 in computer vision libraries; V is lightness, used to describe the brightness of a color, ranging from 0 to 100%, and 0 to 255 in computer vision libraries. Compared to the traditional RGB color space, changes in brightness in the HSV color space primarily affect the V component without altering the H component representing the color category, effectively distinguishing visually similar colors with different hues. The H hue space is then divided into multiple color intervals to obtain color gradient thresholds. These thresholds can be pre-set; for example, in a multimodal large model, the H hue space can be divided into 15 color intervals, each with a value interval of 12, and the name and corresponding color gradient threshold for each interval can be determined.

[0041] When identifying the first line and its corresponding color information within a graphic region, the color information of each line can be determined by using optical character recognition (OCR) based on a multimodal large model to identify the text labels in the legend within the non-graphic region and by associating them with the color samples of the lines within the graphic region. The color samples can be the color of a pixel within a small segment of the line; alternatively, the number of lines within the non-graphic region, as well as the color range and color gradient threshold corresponding to each line, can be obtained using the semantic perception capabilities of a multimodal large model. For example, when identifying line color information using a multimodal large model, you can query the multimodal large model: "How many data lines are there in the image? What are their colors? Using an equal division of the h hue space in the pixel p(h, s, v) space, with each h_step being 12, divide it into fifteen color intervals. The s and v spaces are 50-255. Return the color name as one of the names in this list ["red", "orange_red", "orange", "yellow_orange", "yellow", "lime_yellow", "lime", "green", "mint_green", "mint", "cyan", "light_blue", "blue", "violet", "purple"]. Return in JSON format, requiring the line and color name."

[0042] The system iterates through each pixel in the graphic region, obtaining its HSV value (h, s, v). For each color gradient, it checks if the pixel's H value falls within its corresponding gradient threshold range. If a pixel meets the threshold condition for a certain color gradient, it can be initially classified into that color gradient. Multiple first pixels within the color interval corresponding to the first line are masked to obtain the pixel-level mask for the first line. For example, for a line of color class k, its pixel value can be labeled as k, thus obtaining the pixel-level mask for each line. After obtaining the independent pixel mask for each line, the system iterates through each mask image, recording the coordinates of the marked pixels to form the pixel coordinate set for that line. The coordinate set of the line is associated with the color information according to the color gradient. The color information of each line and its corresponding coordinate set are listed sequentially, and all coordinate sets are summarized into the second pixel coordinates.

[0043] In some implementations, the line graph further includes a second line, wherein the second line intersects with the first line; Compare the color information of the intersection points with the color gradient of the first line and the color gradient of the second line respectively; When the color information of the intersection point belongs to the color gradient of the first line, the intersection point is determined as a point on the first line, and the pixel-level mask of the first line is determined based on the intersection point and multiple first pixel points on the first line. When the color information of the intersection point belongs to the color gradient of the second line, the intersection point is determined as a point on the second line, and the pixel-level mask of the second line is determined based on the intersection point and multiple second pixel points on the second line. When the color information of the intersection point belongs to both the color gradient of the first line and the color gradient of the second line, it is determined that the intersection point belongs to both the first and second lines. Based on the intersection point and multiple first pixels, the pixel-level mask of the first line is determined, and based on the intersection point and multiple second pixels, the pixel-level mask of the second line is determined.

[0044] When two or more lines of different colors intersect, the pixel colors near the intersection point may appear as a mixture of multiple colors due to rendering or color overlay. In the above embodiment, when classifying the color gradient of pixels, the H values ​​of these mixed-color pixels near the intersection point may simultaneously fall within the color gradient threshold range of both lines. In this case, threshold-based determination may cause the lines to break at the intersection point.

[0045] During pixel classification, all pixels that simultaneously satisfy two or more color category gradient thresholds are labeled (hereinafter referred to as "multicolor pixels"). For each multicolor pixel, its color gradient is determined. For example, taking a multicolor pixel as the center, the distribution of classified pixels within its eight neighborhoods is examined. The color gradient of the multicolor pixel is determined based on the color gradients of the pixels surrounding it. Simultaneously, the continuity of the intersection point to which the multicolor pixel belongs is considered. In a line graph, if the first line and the second line intersect, and the intersection point is continuous on the first line but not on the second line, the multicolor pixels contained at the intersection point are classified as the color gradient of the first line; if the intersection point is not continuous on the first line but continuous on the second line, the multicolor pixels contained at the intersection point are classified as the color gradient of the second line; if the intersection point is continuous on both the first and second lines, the multicolor pixels contained at the intersection point are classified as the color gradient of both the first and second lines. Based on the classification results, a pixel-level mask for the first line is determined based on multiple first pixels of the first line; a pixel-level mask for the second line is determined based on multiple second pixels of the second line. When a pixel is classified into multiple lines, the value of the pixel can be set to 1 in the mask of each line to which it belongs. When extracting the coordinates of the second pixel, the coordinates of the pixel will appear simultaneously in the coordinate set of multiple lines.

[0046] To address common challenges in line graphs such as multiple curve intersections, overlaps, and similar colors, the above embodiments leverage the large model's keen perception of color gradients to generate independent line masks, achieving pixel-level separation of different data sequences. Furthermore, by increasing the assessment of multi-colored pixels and curve continuity, the data confusion and fragmentation issues inherent in traditional color thresholding algorithms can be avoided to some extent, ensuring data integrity, which is crucial for subsequent coordinate transformation and data output.

[0047] In some implementations, based on the second pixel coordinates and the first mapping, the planar coordinates of the chart graphic in the planar coordinate system are determined, and a chart data package is generated, including: A first set of second pixel coordinates of the first line and a second set of second pixel coordinates of the second line are determined based on the pixel-level mask of the first line and the pixel-level mask of the second line; Based on the first set, the second set, and the first mapping, determine the set of planar coordinates of the chart graph; Based on the set of planar coordinates and a preset data structure, generate chart data packages.

[0048] Based on the above embodiments, a pixel-level mask for each line is obtained. For the mask P of the k-th color class... k It can extract the set of all pixels with a value of k and obtain P. k The coordinates of each pixel in the graph are determined. The second pixel coordinates of the first line are extracted as the first set, and the second pixel coordinates of the second line are extracted as the second set. The first and second sets are substituted into the first mapping to obtain the set of planar coordinates of the chart. For the set of planar coordinates, the X-axis values ​​can be sorted in ascending order, and local weighted regression can be used to smooth and remove noise. The planar coordinates of each line are associated with its corresponding legend name, and finally packaged into a chart data package with a preset data structure, such as a CSV or JSON format data file. The above steps can be pre-set in a multimodal large model to achieve automatic mask extraction, conversion, and data output.

[0049] In some implementations, the first image includes multiple sub-images, and before identifying the chart category of the first image, the first image is further divided into multiple sub-images, and at least one second image is determined based on a preset chart category and the multiple sub-images.

[0050] A pre-trained object detection model is used to perform layout analysis and image segmentation on the first image. For example, the YOLO model can be adaptively adjusted by learning from a scientific chart dataset. The adjusted model is then used to perform a full scan of the first image, identifying and locating all sub-image regions within it, and obtaining the bounding boxes of each sub-image. The pixel positions of the top-left and bottom-right corners of the region in the original image can be used as the boundary coordinates. Based on the boundary coordinates, the first image is cropped into multiple independent sub-images, forming a candidate set of sub-images to be processed. The images in the candidate set are then individually identified, and based on the features of a preset chart category, it is determined whether the sub-image belongs to the preset chart category. Sub-images that meet the criteria are selected as the second image.

[0051] This application also provides a chart extraction device, comprising: The recognition unit is used to acquire the first image and recognize the chart type of the first image, including: line chart, bar chart or pie chart; The determining unit is used to determine the second image based on a preset chart category and a first image; The recognition unit is also used to identify and determine the graphic region and non-graphic region of the second image, and to mask the non-graphic region, identify the chart data in the non-graphic region, and extract the first pixel coordinates corresponding to the chart data. The mapping unit constructs the first mapping from the pixel coordinate system to the planar coordinate system based on the chart data and the first pixel coordinates. The recognition unit is also used to recognize chart graphics in the graphics area and extract the second pixel coordinates corresponding to the chart graphics; The determining unit is also used to determine the planar coordinates of the chart graphic in the planar coordinate system based on the second pixel coordinates and the first mapping, and to generate a chart data package.

[0052] The descriptions of obtaining the first pixel coordinates, constructing the mapping function, obtaining the second pixel coordinates, and generating the data packet are the same as those in the above method embodiments, and will not be repeated here.

[0053] The above division of units is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, the above units can be implemented by a processor calling software; for example, the detection device includes a processor connected to a memory containing instructions. The processor calls the instructions stored in the memory to implement any of the above methods or to realize the functions of each unit. The processor can be, for example, a general-purpose processor, such as a central processing unit (CPU), and the memory can be internal or external to the device. Alternatively, the above units can be implemented as hardware circuits. The functions of some or all units can be realized through the design of the hardware circuit, which can be understood as one or more processors. For example, in some embodiments, the hardware circuit is an application-specific integrated circuit (ASIC), which implements the functions of some or all units by designing the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a programmable logic device (PLD), which can include a large number of logic gates. The logical relationships between the logic gates are configured through a configuration file, thereby realizing the functions of some or all units. All units of the above devices can be implemented entirely through processor calling programs, or entirely through hardware circuits, or partially through processor calling programs with the remaining parts implemented through hardware circuits.

[0054] Furthermore, the units in the above devices can be integrated in whole or in part, or they can be implemented independently. In some embodiments, these units are integrated together and implemented as a system on a chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the units in the above devices.

[0055] Furthermore, embodiments of this application also provide a computer-readable storage medium including instructions stored thereon, which, when invoked by a processor, execute any of the detection methods described in the above embodiments. Embodiments of this application also provide a computer program (or computer program product) including instructions, which, when invoked by a processor, execute any of the control methods described in the above embodiments.

[0056] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not described in detail or in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Furthermore, the above embodiments can be freely combined as needed.

Claims

1. A method for extracting charts, characterized in that, include: Acquire a first image and identify the chart category of the first image, wherein the chart category includes: line chart, bar chart, or pie chart; Based on the preset chart category and the first image, determine the second image; Identify and determine the graphic and non-graphic regions of the second image; The non-graphic area is masked, the chart data in the non-graphic area is identified, and the first pixel coordinates corresponding to the chart data are extracted; Based on the chart data and the first pixel coordinates, a first mapping from the pixel coordinate system to the planar coordinate system is constructed. In the graphic area, the chart graphic is identified, and the second pixel coordinates corresponding to the chart graphic are extracted; Based on the second pixel coordinates and the first mapping, the planar coordinates of the chart graphic in the planar coordinate system are determined, and a chart data package is generated.

2. The chart extraction method as described in claim 1, characterized in that, The preset chart category is the line chart.

3. The chart extraction method as described in claim 1, characterized in that, The step of identifying chart data in the non-graphic area and extracting the first pixel coordinates corresponding to the chart data includes: Identify at least one scale data region of the coordinate axis in the chart data, and determine the first pixel coordinate of the scale data region in the pixel coordinate system based on the feature pixels in the scale data region.

4. The chart extraction method as described in claim 3, characterized in that, The step of constructing a first mapping from the pixel coordinate system to the planar coordinate system based on the chart data and the first pixel coordinates includes: Based on the scale content of the scale data area and the first pixel coordinates, a data pair is determined, wherein the scale content is determined based on a preset language model. Based on the multiple scale contents corresponding to the coordinate axes of the plane coordinate system, the scale rules of the coordinate axes of the plane coordinate system are determined; Based on the scale pattern and the data pair, the first mapping from the pixel coordinate system to the planar coordinate system is determined, wherein the scale pattern includes linear scale patterns and non-linear scale patterns.

5. The chart extraction method as described in claim 2, characterized in that, The process of identifying the chart graphic within the graphic area and extracting the second pixel coordinates corresponding to the chart graphic includes: Identify and determine the first line in the line graph and the color information of the first line; Based on the color information and a preset color gradient threshold, the color gradient of multiple first pixels of the first line is determined. Based on the color gradient, determine the pixel-level mask of the first line; The second pixel coordinates of the first line are determined based on the pixel-level mask of the first line.

6. The chart extraction method as described in claim 5, characterized in that, The line graph further includes: a second line, wherein the second line intersects with the first line; Compare the color information of the intersection points with the color gradient of the first line and the color gradient of the second line, respectively. When the color information of the intersection point belongs to the color gradient of the first line, the intersection point is determined as a point on the first line, and a pixel-level mask of the first line is determined based on the intersection point and multiple first pixel points on the first line. When the color information of the intersection point belongs to the color gradient of the second line, the intersection point is determined as a point on the second line, and the pixel-level mask of the second line is determined based on the intersection point and multiple second pixel points on the second line. When the color information of the intersection point belongs to both the color gradient of the first line and the color gradient of the second line, it is determined that the intersection point belongs to both the first line and the second line. Based on the intersection point and multiple first pixels, the pixel-level mask of the first line is determined, and based on the intersection point and multiple second pixels, the pixel-level mask of the second line is determined.

7. The chart extraction method as described in claim 6, characterized in that, The step of determining the planar coordinates of the chart graphic in the planar coordinate system based on the second pixel coordinates and the first mapping, and generating a chart data package, includes: A first set of second pixel coordinates of the first line and a second set of second pixel coordinates of the second line are determined based on the pixel-level mask of the first line and the pixel-level mask of the second line. Based on the first set, the second set, and the first mapping, determine the set of planar coordinates of the chart graphic; The chart data package is generated based on the set of planar coordinates and the preset data structure.

8. The chart extraction method according to any one of claims 1 to 7, characterized in that, The first image includes multiple sub-images. Before identifying the chart category of the first image, the method further includes segmenting the first image into multiple sub-images and determining at least one second image based on a preset chart category and the multiple sub-images.

9. A chart extraction device, characterized in that, include: The recognition unit is used to acquire a first image and recognize the chart category of the first image, wherein the chart category includes: line chart, bar chart, or pie chart; The determining unit is used to determine the second image based on a preset chart category and the first image; The recognition unit is also used to identify and determine the graphic region and non-graphic region of the second image, and to mask the non-graphic region, identify the chart data in the non-graphic region, and extract the first pixel coordinates corresponding to the chart data; The mapping unit constructs a first mapping from the pixel coordinate system to the planar coordinate system based on the chart data and the first pixel coordinates; The recognition unit is also used to recognize the chart graphic in the graphic area and extract the second pixel coordinates corresponding to the chart graphic; The determining unit is further configured to determine the planar coordinates of the chart graphic in the planar coordinate system based on the second pixel coordinates and the first mapping, and generate a chart data package.

10. A computer-readable storage medium, characterized in that, It stores instructions that, when executed by a processor, implement the chart extraction method as described in any one of claims 1 to 8.

Images