Data determination method and device, equipment, storage medium and program product

Abstract

The invention provides a data determination method and device, equipment, a storage medium and a program product. The method comprises the following steps: encoding a chart image to obtain a first chart feature of the chart image; performing data conversion processing on the first chart feature to obtain first structured data corresponding to the chart image; performing decoding processing on the first chart feature to obtain first data; and determining the confidence coefficient of the first structured data based on the first structured data and the first data. According to the invention, the structured data of the chart image and the confidence of the structured data can be output at the same time.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to a data determination method, device, equipment, storage medium and program product. Background Art

[0002] Artificial Intelligence (AI) is a comprehensive technology in computer science. By studying the design principles and implementation methods of various intelligent machines, it enables machines to have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, such as natural language processing technology, machine learning / deep learning, and document question answering.

[0003] With the development of artificial intelligence technology, the demand for document information query is growing. In these documents, charts, as an important information carrier, can often be used to conduct in-depth query and interpretation of the data and information therein. In related technologies, the content in the chart image is first identified and converted into structured data, and then the information in the chart image is acquired through a large language model. Summary of the invention

[0004] The embodiments of the present application provide a data determination method, apparatus, device, storage medium, and program product, which can simultaneously output the structured data of a chart image and the confidence level of the structured data.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a data determination method, the method comprising:

[0007] Performing encoding processing on the chart image to obtain a first chart feature of the chart image;

[0008] Performing data conversion processing on the first chart feature to obtain first structured data corresponding to the chart image;

[0009] Decoding the first chart feature to obtain first data;

[0010] Based on the first structured data and the first data, a confidence level of the first structured data is determined.

[0011] The present application provides a data determination device, the device comprising:

[0012] An encoding module, used for encoding the chart image to obtain a first chart feature of the chart image;

[0013] A first decoding module, configured to perform data conversion processing on the first chart feature to obtain first structured data corresponding to the chart image;

[0014] A second decoding module, used for decoding the first chart feature to obtain first data;

[0015] A confidence determination module is used to determine the confidence of the first structured data based on the first structured data and the first data.

[0016] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0017] A memory for storing computer executable instructions;

[0018] The processor is used to implement the data determination method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0019] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the data determination method provided in the embodiment of the present application when executed by a processor.

[0020] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the data determination method provided in the embodiment of the present application is implemented.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] By encoding the chart image, a first chart feature of the chart image is obtained, and data conversion processing and decoding processing are performed on the first chart feature to obtain first structured data and first data of the chart image. While outputting the first structured data of the chart image, the confidence of the first structured data is determined based on the first data and the first structured data, so that the reliability of the first structured data is measured through the determined confidence, providing more accurate support for subsequent data analysis and decision-making processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a structural diagram of the data determination system architecture provided in an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of the structure of an electronic device for data determination provided in an embodiment of the present application;

[0025] Figure 3 It is a first flow chart of the data determination method provided in an embodiment of the present application;

[0026] Figure 4 is a second flow chart of the data determination method provided in an embodiment of the present application;

[0027] Figure 5 is a third flow chart of the data determination method provided in an embodiment of the present application;

[0028] Figure 6 is a fourth flow chart of the data determination method provided in an embodiment of the present application;

[0029] Figure 7 is a fifth flow chart of the data determination method provided in an embodiment of the present application;

[0030] Figure 8 is a schematic diagram of a line graph image provided in an embodiment of the present application;

[0031] Fig. 9 is a schematic diagram of an image recognition model provided in an embodiment of the present application;

[0032] Fig.10 It is a structural diagram of the data determination model provided in the embodiment of the present application.

[0033] It can be pointed out that the above-mentioned “first” and “second” are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0035] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0036] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0037] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0038] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0039] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0040] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0041] 1) Chart analysis task: Chart analysis task refers to using machine learning models to convert chart images into structured data, and then realizing a series of tasks such as analyzing, interpreting, and predicting the structured data of the chart. It usually involves an in-depth understanding of the structure, content, and relevance of the chart, for example, converting chart images into tabular data in a specific format.

[0042] 2) Optical Character Recognition (OCR): A technology that enables computers to recognize and extract text from images. Through the OCR interface, applications can convert text in photographed or scanned documents and images into machine-processable text data, thereby realizing the creation and editing of electronic documents.

[0043] 3) Structured Data: refers to data with a fixed format and organization. Its characteristics are that there are obvious separations and relationships between data. It usually appears in the form of a table. The data in structured data is stored according to predefined fields and records, such as database tables, spreadsheet XML and JSON files.

[0044] 4) Image Random Channel Switching (RGB Random Channel Switching): A technique that randomly selects the red, green, or blue channels of an RGB image and then swaps these channels. It is usually used for image enhancement to increase data diversity, and can also be used for image recoloring to generate new image versions.

[0045] Embodiments of the present application provide a data determination method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can simultaneously output structured data of a chart image and the confidence level of the structured data.

[0046] The data determination method provided in the embodiment of the present application can be implemented independently by a terminal or a server; it can also be implemented jointly by a terminal and a server, for example, the terminal independently undertakes the data determination method described below, or the terminal sends a data determination request containing a chart image to the server, and the server executes the data determination method based on the received data determination request containing the chart image, encodes the chart image to obtain a first chart feature of the chart image, performs data conversion processing and decoding processing on the first chart feature to obtain first structured data and first data, and determines the confidence of the first structured data based on the first structured data and the first data.

[0047] The electronic device for data determination provided in the embodiments of the present application may be various types of terminals or servers, wherein the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services; the terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart TV, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the present application.

[0048] See also Figure 1 , Figure 1 It is a structural diagram of the data determination system architecture provided in an embodiment of the present application. In the data determination system 10, the terminal 200 is connected to the server 100 via a network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0049] The terminal 200 may be used to obtain a data determination request including a chart image. For example, when a user inputs a chart image through the terminal 200 , the terminal 200 automatically obtains the chart image and automatically generates a data determination request including the chart image.

[0050] In some embodiments, a data determination plug-in may be embedded in the client running in the terminal, and is used to execute the data determination method locally in the client. For example, after the terminal 200 obtains a data determination request including a chart image, it calls the data determination plug-in, encodes the chart image, obtains a first chart feature of the chart image, performs data conversion and decoding on the first chart feature, obtains first structured data and first data, and determines the confidence of the first structured data based on the first structured data and the first data.

[0051] In some embodiments, after the terminal 200 obtains a data determination request containing a chart image, it calls the data determination interface of the server 100 through the network 300, and the server 100 encodes the chart image to obtain a first chart feature of the chart image, performs data conversion processing and decoding processing on the first chart feature respectively to obtain first structured data and first data, determines the confidence of the first structured data based on the first structured data and the first data, and sends the confidence to the terminal 200.

[0052] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device for determining data provided in an embodiment of the present application, taking the electronic device 500 as a server as an example, Figure 2 The electronic device 500 for data determination shown includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 2 Various buses are labeled as bus system 540 .

[0053] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0054] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiment of the present application is intended to include any suitable type of memory. The memory 550 optionally includes one or more storage devices that are physically far away from the processor 510.

[0055] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0056] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0057] A network communication module 552, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0058] In some embodiments, the data determination device provided in the embodiments of the present application can be implemented in software. Figure 2 The data determination device 555 stored in the memory 550 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: an encoding module 5551, a first decoding module 5552, a second decoding module 5553, a confidence determination module 5554 and a training module 5555, wherein the encoding module 5551, the first decoding module 5552, the second decoding module 5553 and the confidence determination module 5554 are used to implement the data determination function provided in the embodiment of the present application, and the training module 5555 is used to train the chart recognition model in the embodiment of the present application. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0059] As mentioned above, the electronic device implementing the data determination method of the embodiment of the present application may be a terminal, a server, or a combination of the two, so the execution subject of each step will not be repeatedly described below, see Figure 3 , Figure 3 is a first flow chart of the data determination method provided in the embodiment of the present application, which will be combined with Figure 3The steps shown are explained.

[0060] In step 101, a chart image is coded to obtain a first chart feature of the chart image.

[0061] As an example, a chart image refers to a chart in the form of an image, including a bar chart image, a line chart image, a pie chart image, and a scatter plot chart image, see Figure 8 , Figure 8 It is a schematic diagram of a line graph image provided in an embodiment of the present application. By calling a chart recognition model, the chart image is encoded and processed to obtain a first chart feature of the chart image.

[0062] See also Figure 4 , Figure 4 is a second flow chart of the data determination method provided in an embodiment of the present application, Figure 3 The illustrated step 101 may be implemented by following the steps 1011 to 1014, which are described in detail below.

[0063] In step 1011, an enhanced image of the chart image and first digital information in the chart image are determined.

[0064] In some embodiments, determining an enhanced image of a chart image can be achieved by: performing Gaussian blur processing on the chart image to obtain a Gaussian blurred chart image; performing Gaussian noise processing on the Gaussian blurred chart image to obtain a Gaussian noisy chart image; performing image random channel switching (RGB random channel switching) processing on the Gaussian noisy chart image to obtain a channel switched chart image; performing pixel enhancement processing on the channel switched chart image to obtain an enhanced image of the chart image.

[0065] For example, Gaussian blur processing can be a Gaussian blur processing with a first probability (such as 0.5) on the chart image; Gaussian noise processing can be a Gaussian noise processing with a second probability (such as 0.5) on the chart image after Gaussian blur; the three RGB channels of the chart image after Gaussian noise are randomly switched to obtain the chart image after channel switching; pixel enhancement processing can be achieved in the following way: for each pixel of the chart image after channel switching, a random number within the range of 0 to 10 is randomly added to the pixel value with a third probability (such as 0.3), and a random number within the range of 0 to 10 is randomly subtracted from the pixel value with a fourth probability (such as 0.3).

[0066] In some embodiments, determining the first digital information in a chart image can be achieved by: performing text recognition processing on the chart image to obtain text information in the chart image; performing digital screening processing on the text information in the chart image to obtain a number list in the chart image, wherein the number list in the chart image contains at least one number; sorting the numbers in the number list in the chart image to obtain a sorted number list; and splicing the numbers in the sorted number list to obtain the first digital information in the chart image.

[0067] For example, the first digital information in the chart image can be obtained through the ORC interface, see Figure 8 , perform text recognition processing on the chart image, and obtain all the text information in the chart image: "Spot price: average price: wheat, 17-05, 17-11, 18-05, 18-11, 19-05, 19-11, 20-05, 21-05, 21-11, 2000, 2100, 2200, 2300, 2400, 2500, 2600, 2700", perform digital screening processing on all the text information in the chart image, and obtain 2000, 2100, 2200, 2300, 2400, 2500, 2600, 2700, and sort the number list according to the order of numbers from small to large, and obtain [2000, 2100, 2200, 2300, 2400, 2500, 2 600, 2700], the numbers in the number list in the sorted chart image are spliced ​​together to obtain the first digital information "2000, 2100, 2200, 2300, 2400, 2500, 2600, 2700" in the chart image, among which, although "17-05, 17-11, 18-05, 18-11, 19-05, 19-11, 20-05, 21-05, 21-11" also contain numbers, they represent date information and are connected with the symbol "-". The model can recognize "17-05, 17-11, 18-05, 18-11, 19-05, 19-11, 20-05, 21-05, 21-11" as dates and sort them.

[0068] In step 1012, image coding processing is performed on the enhanced image to obtain image coding features of the chart image.

[0069] As an example, image coding processing is performed on the enhanced image, and the enhanced image of the image type is converted into an image coding feature in the form of a tensor, wherein the image coding processing on the enhanced image can be implemented by multiple cascaded convolutional layers, or by an image block encoder (such as a Patch encoding module), which is not limited here.

[0070] In some embodiments, the enhanced image is subjected to image coding processing by an image block encoder to obtain image coding features of the chart image, which can be achieved through the following steps: performing image segmentation processing on the enhanced image to obtain at least one image block; performing embedding processing on at least one image block to obtain embedded features of the chart image; performing attention processing on the embedded features of the chart image to obtain image coding features of the chart image.

[0071] As an example, the enhanced image is divided into multiple regular or irregular image blocks of different sizes, and the image blocks are converted into image block vectors. For example, the enhanced image is divided into image blocks of 16 pixels by 16 pixels. The image blocks can be regarded as a 16 by 16 matrix. Each numerical value in the matrix represents the pixel value at the corresponding position of the image block. The image blocks are dimensionally converted to obtain image block vectors with a length of 256 (16 by 16).

[0072] All image block vectors of the chart image are embedded through a fully connected layer to obtain the embedded features of the chart image. Based on multiple cascaded encoding layers, the embedded features of the chart image are subjected to attention processing to obtain the image encoding features of the chart image, wherein each of the above encoding layers includes a self-attention layer (Self-Attention Layers) and a feed-forward network layer (Feed-Forward Networks).

[0073] In an embodiment of the present application, the enhanced image is subjected to image encoding processing through an image block encoder, and each image block can capture local information of different chart images, which helps to extract features of the chart image in a finer granularity and obtain image encoding features of the chart image with richer texture information and structural information.

[0074] In step 1013, the first digital information is subjected to text encoding processing to obtain digital encoding features of the chart image.

[0075] As an example, the first digital information is subjected to text encoding processing, and the first digital information in character form is converted into digital coding features in tensor form, wherein the text encoding processing of the first digital information can be implemented by multiple cascaded convolutional layers, or by a rotational position encoder (such as RoFormer), which is not limited here.

[0076] In some embodiments, the first digital information is subjected to text encoding processing by a rotational position encoder to obtain digital coding features of a chart image, which can be achieved through the following steps: embedding the first digital information to obtain embedded features of the first digital information; rotating the embedded features of the first digital information to obtain embedded features of the rotated first digital information; and performing attention processing on the embedded features of the rotated first digital information through multiple cascaded attention layers to obtain digital coding features of the chart image.

[0077] As an example, the first digital information is segmented to obtain the first digital information after segmentation; the first digital information after segmentation is indexed to obtain the indexed first digital information; the indexed first digital information is embedded to obtain the embedded features of the first digital information; a rotation matrix is ​​obtained, and the embedded features of the first digital information are rotated based on the rotation matrix to obtain the embedded features of the rotated first digital information, wherein the rotation matrix is ​​calculated by the rotation angle, and the rotation angle is a learnable parameter; the embedded features of the rotated first digital information are used as the query value (query) and key value (key) of the attention layer, and the rotation position encoding features of the first digital information are subjected to attention processing through multiple cascaded attention layers to obtain the digital encoding features of the chart image.

[0078] In the embodiment of the present application, by rotating the position encoder and performing text encoding processing on the first digital information, the relative position relationship between each word in the first digital information can be better captured, thereby improving the efficiency of data processing and analysis.

[0079] In step 1014 , a first chart feature of the chart image is determined based on the image encoding feature and the digital encoding feature.

[0080] As an example, when the tensor dimension of the image coding feature and the tensor dimension of the digital coding feature are inconsistent, the image coding feature and the digital coding feature can also be linearly transformed to obtain the image coding feature after the linear transformation and the digital coding feature after the linear transformation, and the first chart feature of the chart image is determined based on the image coding feature after the linear transformation and the digital coding feature after the linear transformation, wherein the tensor dimension of the image coding feature after the linear transformation and the tensor dimension of the digital coding feature are the same.

[0081] In some embodiments, step 1014 can be implemented by the following steps: fusing the image coding feature and the digital coding feature to obtain a fused feature; and performing image coding processing on the fused feature to obtain a first chart feature.

[0082] As an example, the image coding features and the digital coding features are fused to obtain fused features, and the fused features are image coded to obtain the first chart features, wherein the fusion processing can be a splicing processing of the image coding features and the digital coding features, or a summing processing of the image coding features and the digital coding features, which is not limited here, and image coding of the fused features can be achieved by multiple cascaded convolutional layers, which is not limited here.

[0083] For example, the multiple cascaded convolutional layers are three cascaded convolutional layers, and the fused features are image encoded through the three cascaded convolutional layers to obtain the first chart feature, that is, the fused features are encoded through the first convolutional layer to obtain the output data of the first convolutional layer; the output data of the first convolutional layer is encoded through the second convolutional layer to obtain the output data of the second convolutional layer; the output data of the second convolutional layer is encoded through the third convolutional layer to obtain the first chart feature.

[0084] In the embodiment of the present application, by performing image coding processing on the fused features, a first chart feature with richer feature expression can be obtained.

[0085] In step 102, data conversion processing is performed on the first chart feature to obtain first structured data corresponding to the chart image.

[0086] As an example, the data determination method of the embodiment of the present application is implemented by calling a chart recognition model, performing data conversion processing on the first chart feature, obtaining first structured data corresponding to the chart image, and using the first structured data as the output result of the chart recognition model.

[0087] For example, see Figure 8 , by identifying the model pair through the graph Figure 8 The chart image shown is processed for chart recognition to obtain the first structured data “Title: Spot price: Average price: Wheat; 17-05: 2614; 17-11: 2546; 18-05: 2399; 18-11: 2490; 19-05: 2423; 19-11: 2400; 20-05: 2440; 20-11: 2488; 21-05: 2540”.

[0088] See also Figure 5 , Figure 5 is a third flow chart of the data determination method provided in the embodiment of the present application, Figure 3 The illustrated step 102 may be implemented by following the steps 1021 to 1022, which are described in detail below.

[0089] In step 1021, text decoding is performed on the first chart feature to obtain text information of the chart image.

[0090] In some embodiments, text decoding processing is performed on the first chart feature to obtain text information of the chart image, which is achieved through K cascaded decoding layers, where K is a positive integer greater than 1. Step 1021 can be achieved through the following steps: decoding the first chart feature through the first decoding layer to obtain output data of the first decoding layer; decoding the output data of the k-1th decoding layer through the kth decoding layer to obtain output data of the kth decoding layer, wherein k is a successively increasing positive integer, 1<k≤K; and determining the output data of the Kth decoding layer as the text information of the chart image.

[0091] For example, when K is equal to 3, the first chart feature is subjected to text decoding processing through three cascaded decoding layers, that is, the first chart feature is decoded through the first decoding layer to obtain the output data of the first decoding layer; the output data of the first decoding layer is decoded through the second decoding layer to obtain the output data of the second decoding layer; and the output data of the second decoding layer is decoded through the third decoding layer to obtain the text information of the chart image.

[0092] In some embodiments, the decoding layer includes a multi-head self-attention layer and a feedforward neural network layer, and "decoding the first chart features through the first decoding layer to obtain the output data of the first decoding layer" can be achieved by the following steps: based on the multi-head self-attention layer of the first decoding layer, performing attention processing on the first chart features to obtain the output data of the multi-head self-attention layer of the first decoding layer; based on the feedforward neural network layer of the first decoding layer, performing linear transformation processing on the output data of the multi-head self-attention layer of the first decoding layer to obtain the output data of the first decoding layer.

[0093] In some embodiments, the decoding layer includes a multi-head self-attention layer and a feedforward neural network layer, and "decoding the output data of the k-1th decoding layer through the kth decoding layer to obtain the output data of the kth decoding layer" can be achieved by the following steps: based on the multi-head self-attention layer of the kth decoding layer, performing attention processing on the output data of the k-1th decoding layer to obtain the output data of the multi-head self-attention layer of the kth decoding layer; based on the feedforward neural network layer of the kth decoding layer, performing linear transformation processing on the output data of the multi-head self-attention layer of the kth decoding layer to obtain the output data of the kth decoding layer.

[0094] In step 1022, the text information is structured to obtain first structured data.

[0095] As an example, after text decoding processing is performed on the first chart feature, the multi-dimensional first chart feature is decoded into one-dimensional text information, the one-dimensional text information is structured, and the one-dimensional text information is organized into first structured data.

[0096] In some embodiments, step 1022 may be implemented by the following steps: searching for index data of the text information from the index table; and performing structural adjustment processing on the index data of the text information to obtain first structured data.

[0097] As an example, the index table includes the correspondence between different text information and different index data. According to the text information, the index data corresponding to the text information is searched from the index table, and the index data of the text information is structurally adjusted, wherein the structural adjustment processing includes adjusting the semantic word order of the index data of the text information and adjusting the data structure of the text information.

[0098] For example, when the text information is a one-dimensional vector [100, 605, 755, 7], based on the index table, the text information "100" is found to correspond to the index data "wheat", the text information "605" is found to correspond to the index data "price", the text information "755" is found to correspond to the index data "today", and the text information "7" is found to correspond to the index data "2600", and the index data "wheat", "price", "today" and "2600" of the text information are obtained. The semantic word order of the index data of the text information is adjusted to obtain "today", "wheat", "price" and "2600", and the data structure of the text information is adjusted to adjust the index data to structured data of the key-value pair type to obtain "Today's wheat price: 2600".

[0099] In step 103, the first chart feature is decoded to obtain first data.

[0100] As an example, a decoding process is performed on the first chart feature to decode the first chart feature into one-dimensional first data, wherein the first data is used to assist in determining the confidence level.

[0101] In some embodiments, step 103 may be implemented by the following steps: decoding the first graph feature through N cascaded dimension reduction layers to obtain first data, where N is an integer greater than 1.

[0102] Among them, the input data of the first dimension reduction layer is the first chart feature; the input data of the i-th dimension reduction layer is the output data of the i-1-th dimension reduction layer, 1<i≤N; and the output data of the N-th dimension reduction layer is the first data.

[0103] As an example, the first dimension reduction layer is used to perform dimension reduction processing on the first graph feature to obtain the output data of the first dimension reduction layer; the output data of the n-1th dimension reduction layer is performed on the nth dimension reduction layer to obtain the output data of the nth dimension reduction layer, wherein n is a successively increasing positive integer, 1<n≤N; the output data of the Nth dimension reduction layer is determined as the first data.

[0104] For example, when N is equal to 3, the first dimension reduction layer is used to perform dimension reduction processing on the first graph feature to obtain the output data of the first dimension reduction layer; the second dimension reduction layer is used to perform dimension reduction processing on the output data of the first dimension reduction layer to obtain the output data of the second dimension reduction layer; the third dimension reduction layer is used to perform dimension reduction processing on the output data of the second dimension reduction layer to obtain the first data.

[0105] In some embodiments, the dimensionality reduction layer includes a linear transformation layer and an activation layer. "Performing dimensionality reduction processing on the first chart feature through the first dimensionality reduction layer to obtain the output data of the first dimensionality reduction layer" can be achieved by the following steps: performing linear transformation processing on the first chart feature to obtain the first chart feature after linear transformation processing; performing activation processing on the first chart feature after linear transformation processing to obtain the output data of the first dimensionality reduction layer.

[0106] In some embodiments, "performing dimensionality reduction processing on the output data of the n-1th dimensionality reduction layer through the nth dimensionality reduction layer to obtain the output data of the n-1th dimensionality reduction layer" can be achieved by the following steps: performing linear transformation processing on the output data of the n-1th dimensionality reduction layer to obtain the output data of the n-1th dimensionality reduction layer after the linear transformation processing; performing activation processing on the output data of the n-1th dimensionality reduction layer after the linear transformation processing to obtain the output data of the n-1th dimensionality reduction layer.

[0107] As an example, the tensor dimension of the first chart feature after linear transformation is smaller than that of the first chart feature, so as to realize the function of reducing the tensor dimension of the first chart feature. The activation processing of the first chart feature after linear transformation is realized by an activation function, and the above activation function includes a linear rectification function (ReLU, RectifiedLinear Unit) and a hyperbolic tangent function (tanh, Hyperbolic tangent), which are not limited here.

[0108] In step 104 , the confidence level of the first structured data is determined based on the first structured data and the first data.

[0109] As an example, the confidence level is used to characterize the credibility of the prediction information of the chart recognition model. The credibility of the first structured data output by the chart recognition model is determined through the first data. Whether the first structured data output by the model is credible is judged by setting a confidence threshold. When the confidence level of the first structured data is less than or equal to the confidence threshold, it is characterized that the first structured data output by the current chart recognition model is credible. When the confidence level of the first structured data is greater than the confidence threshold, it is characterized that the first structured data output by the current chart recognition model is uncredible.

[0110] For example, the confidence threshold is set to 0.2. When the confidence of the first structured data is 0.15, it indicates that the first structured data is credible; when the confidence of the first structured data is 0.5, it indicates that the first structured data is uncredible.

[0111] See also Figure 6 , Figure 6 is a fourth flow chart of the data determination method provided in an embodiment of the present application, Figure 3 The illustrated step 104 may be implemented by following the steps 1041 to 1043, which are described in detail below.

[0112] In step 1041, the first structured data is digitally screened to obtain second digital information in the first structured data.

[0113] For example, when the first structured data is “Title: Spot price: Average price: Wheat; 17-05: 2614; 17-11: 2546; 18-05: 2399; 18-11: 2490; 19-05: 2423; 19-11: 2400; 20-05: 244020-11: 2488; 21-05: 2540”, the first structured data is digitally filtered to obtain the second digital information “2614; 2546; 2399; 2490; 2423; 2400; 2440; 2488; 2540”.

[0114] In step 1042, the second digital information is normalized to obtain second data.

[0115] As an example, the second digital information includes at least one number, the maximum number and the minimum number in the second digital information are obtained, and the following steps are performed on each number in the second digital information respectively to obtain a normalized number corresponding to each number in the second digital information, and the normalized numbers corresponding to all numbers in the second digital information are used as the second data: obtain the number to be processed, wherein the number to be processed is any number in the second digital information, subtract the minimum number from the number to be processed to obtain a first number, divide the first number by the maximum number to obtain a normalized number corresponding to the number to be processed, wherein the minimum number is the smallest number in the second digital information, and the maximum number is the largest number in the second digital information.

[0116] For example, when the second digital information is "2614; 2546; 2399; 2490; 2423; 2400; 2440; 2488; 2540", the maximum number "2614" and the minimum number "2399" in the second digital information are obtained, and the minimum number "2399" is subtracted from each number in the second digital information to obtain "215; 147; 0; 91; 24; 1; 41; 89; 141", and each number in "215; 147; 0; 91; 24; 1; 41; 89; 141" is divided by the maximum number "2614" to obtain the second data "0.82; 0.05; 0; 0.03; 0.01; 0.00; 0.02; 0.03; 0.05", wherein, for the convenience of display, the second data displayed here is the result of retaining two decimal places, and in actual application, the precision of the second data is not limited.

[0117] In step 1043 , a confidence level is determined based on the second data and the first data.

[0118] As an example, the confidence level is determined by a difference value between the second data and the first data.

[0119] In some embodiments, the second data includes M first values, and the first data includes M second values. Step 1043 can be implemented by the following steps: determining M difference values ​​between the M first values ​​and the M second values; and determining confidence levels based on the M difference values.

[0120] As an example, based on the sorting rules, the M first numerical values ​​in the second data and the M second numerical values ​​in the first data are sorted respectively to obtain the sorted M first numerical values ​​and the sorted M second numerical values; the first difference value between the sorted first numerical value and the sorted first second numerical value is determined; the mth difference value between the sorted mth first numerical value and the sorted mth second numerical value is determined; and so on, to obtain M difference values, and the sum of the M difference values ​​is determined as the confidence of the first structured data, where 1<m≤M, M is a positive integer greater than 1, and one first numerical value corresponds to one difference value.

[0121] For example, when the sorted second data is "0; 0.00; 0.01; 0.02; 0.03; 0.03; 0.05; 0.05; 0.82" and the sorted first data is "0; 0.01; 0.01; 0.03; 0.04; 0.05; 0.06; 0.06; 0.9", the nine difference values ​​between the sorted second data and the sorted first data are calculated to be "0; 0.01; 0; 0.01; 0.01; 0.02; 0.01; 0.01; 0.08", and the sum of the nine difference values ​​0.15 is determined as the confidence of the first structured data.

[0122] The embodiment of the present application determines the confidence of the chart image analysis result (structured data) output by the model through the difference value between the second data and the first data, thereby improving the reliability of the analysis result and helping to provide more accurate support in the subsequent data analysis and decision-making process.

[0123] See also Figure 7 , Figure 7 is a fifth flow chart of the data determination method provided in the embodiment of the present application. The data determination method in the embodiment of the present application is implemented by calling a chart recognition model. The training process of the chart recognition model is implemented by Figure 7 The implementation of steps 201 to 203 shown is described in detail below.

[0124] In step 201, image recognition processing is performed on a chart image sample by using a chart recognition model to obtain second chart features, second structured data and third data.

[0125] The image recognition process includes: encoding the chart image sample to obtain a second chart feature; performing data conversion on the second chart feature to obtain second structured data; and decoding the second chart feature to obtain third data.

[0126] As an example, referring to the specific steps of step 101, the chart image sample is encoded through the chart recognition model to obtain the second chart feature, referring to the specific steps of step 102, the second chart feature is converted into data through the chart recognition model to obtain the second structured data, referring to the specific steps of step 103, the second chart feature is decoded to obtain the third data.

[0127] In step 202, a loss function of a chart recognition model is constructed based on label data corresponding to the chart image sample, the second chart feature, the second structured data, and the third data.

[0128] The label data is the expected value of the structured data corresponding to the chart image sample, and is used to guide the training of the chart recognition model of the embodiment of the present application.

[0129] In some embodiments, step 202 can be implemented by the following steps: determining a first loss based on the second graph feature and label data; determining a second loss based on the second structured data and the third data; and constructing a loss function of the graph recognition model based on the first loss and the second loss.

[0130] As an example, the label data is encoded through a chart recognition model to obtain label data features, the similarity between the second chart features and the label data features is calculated, and the first loss is determined, wherein the similarity calculation method includes calculating the cosine similarity, cross entropy loss, mean square error loss, etc. between the second chart features and the label data features, which are not limited here.

[0131] Referring to step 1043 , the second loss is determined based on the second structured data and the third data, which will not be repeated here.

[0132] By calculating the weighted sum of the first loss and the second loss, integrating the first loss and the second loss, we get the loss function Loss of the graph recognition model. The formula (1.1) of the loss function is:

[0133] Loss=α·Loss1+β·Loss2 (1.1)

[0134] Among them, Loss1 is the first loss, Loss2 is the second loss, α and β are the weight factors of the first loss and the second loss respectively.

[0135] In step 203 , the parameters of the graph recognition model are updated based on the loss function.

[0136] As an example, the parameters of the graph recognition model are updated by minimizing the loss function, and the updated parameters of the graph recognition model are used as the parameters of the trained graph recognition model until the loss function converges.

[0137] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0138] With the continuous advancement of artificial intelligence technology, the demand for document information query is growing. Among them, charts are a kind of document. Users can ask questions about the information in the charts. Related technologies recognize charts as structured data and then use a large language model to realize the function of chart question answering. See Fig. 9 , Fig. 9 It is a schematic diagram of the image recognition model provided in the embodiment of the present application. In the related art, the chart image is encoded by the image encoding model to obtain the encoding features of the chart image, and then the encoding features of the chart image are decoded by the text decoding module to obtain the output text (structured text) corresponding to the chart image. However, the related art can only obtain the output text corresponding to the chart image, and calculate the credibility of the above output text (equivalent to the confidence level mentioned above).

[0139] In order to solve the above problems, an embodiment of the present application proposes a data determination method, which can not only convert a chart image into structured data, but also determine whether the identified structured data is correct.

[0140] See also Fig.10 , Fig.10 This is a schematic diagram of the structure of the data determination model provided in the embodiment of the present application. Figure 1 The model structure for calculating confidence provided in the embodiments of the present application is described.

[0141] Optical character recognition interface (OCR interface): obtain Chinese chart images, and then process all chart images through the OCR interface to obtain all the text on the chart images, filter out the numbers in all the text, and sort the filtered numbers from small to large to obtain a sorted number list (number_list), and concatenate number_list to obtain a number string (number_str, equivalent to the first number information above).

[0142] Image encoding module: First, the chart image is processed with Gaussian blur and Gaussian noise with a probability of 0.5, then the RGB of the chart image is randomly switched, and finally a random value between 0 and 10 is added to each pixel of the chart image with a probability of 0.3, and a random value between 0 and 10 is subtracted from each pixel with a probability of 0.3 to obtain the enhanced image. The enhanced image is passed through a block encoding module (Patch encoding module) to obtain the image encoding feature (embedding_visual).

[0143] Text encoding module: After number_str is processed by the Chinese word segmenter (Chinese tokenizer), the number_str processed by the Chinese tokenizer is obtained, and then the number_str processed by the Chinese tokenizer is encoded by the text encoding module to obtain the text encoding feature (embedding_str, equivalent to the digital encoding feature mentioned above).

[0144] Model prediction results: Unify the dimensions of image coding features and text coding features, and concatenate the unified image coding features and text coding features to obtain the final features (equivalent to the fusion features mentioned above). Perform image coding on the final features to obtain chart image features (E_visual, equivalent to the first chart feature mentioned above), then perform text decoding on E_visual through a text decoder to obtain text features (E_str, equivalent to the text information of the chart image mentioned above), and process E_str through the Chinese tokenizer to obtain the model prediction results (result_str, equivalent to the first structured data mentioned above).

[0145] Auxiliary text decoding module: Through the auxiliary text decoding module, the image code E_visual is decoded to obtain auxiliary features (E_aux, equivalent to the first data above), where the auxiliary decoding module is composed of four layers of perceptrons (four layers of MLP)

[0146] Model prediction results: The first loss (Loss1) between the text feature E_str and the label feature is calculated through the loss function, where the label feature is obtained by encoding the label corresponding to the chart image.

[0147] Auxiliary prediction results: filter out the numbers in result_str to obtain a list of numbers in result_str (number_result, equivalent to the second digital information above), normalize the number_result list to obtain normalized data information (number_norm_sorted, equivalent to the second data above), and calculate the second loss (Loss2) between E_aux and number_norm_sorted through the loss function. The specific normalization process is as follows: obtain the minimum value (number_min) and the maximum value (number_max) in the number_result list, subtract number_min from each number in number_result, and then divide it by number_max to obtain the normalized number_result list (number_norm), sort number_norm from small to large, and obtain number_norm_sorted.

[0148] Calculate loss: sum Loss1 and Loss2 to get the final loss function. Use the back propagation algorithm to guide the learning process of the model and update the model parameters until the model converges.

[0149] When applying and reasoning about the model, Fig.10 The upper branch of the model structure shown in , obtains the model's prediction result for the input chart image, through Fig.10 The lower branch of the model structure shown in the figure calculates Loss2, and uses Loss2 to calculate the credibility of the prediction result of the chart image output by the upper branch. When Loss2 is less than 0.2, it means that the above prediction result is credible. If Loss2 is greater than 0.2, it means that the above prediction is unreliable.

[0150] The data determination method of the embodiment of the present application verifies the confidence of the output result of the model through the specific output value (numerical value) (not verifying the category), because there is a certain relationship between the true value (label value) of the model and the number on the chart image. Figure 7 , Figure 7 The numbers included are: 2000, 2100, 2200, 2300, 2400, 2500, 2600 and 2700. When the model's prediction results are all between 2000 and 2700, it means that the model's prediction results are credible.

[0151] Chart images often contain some worthless text and graphic information. During the model training process, the model can gradually distinguish important information for learning and information that does not need to be processed. This distinction and selection process is crucial for model training and performance improvement.

[0152] The following continues to describe the exemplary structure of the data determination device 555 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in a data determination device 555 of the memory 550 may include:

[0153] The encoding module 5551 is used to perform encoding processing on the chart image to obtain the first chart feature of the chart image.

[0154] The first decoding module 5552 is used to perform data conversion processing on the first chart feature to obtain first structured data corresponding to the chart image.

[0155] The second decoding module 5553 is used to decode the first chart feature to obtain first data.

[0156] The confidence determination module 5554 is used to determine the confidence of the first structured data based on the first structured data and the first data.

[0157] In some embodiments, the encoding module 5551 is also used to determine an enhanced image of the chart image and a first digital information in the chart image; perform image encoding processing on the enhanced image to obtain image encoding features of the chart image; perform text encoding processing on the first digital information to obtain digital encoding features of the chart image; and determine a first chart feature of the chart image based on the image encoding features and the digital encoding features.

[0158] In some embodiments, the encoding module 5551 is further used to fuse the image encoding feature and the digital encoding feature to obtain a fused feature; and perform image encoding processing on the fused feature to obtain the first chart feature.

[0159] In some embodiments, the first decoding module 5552 is further used to perform text decoding processing on the first chart feature to obtain text information of the chart image; and perform structured processing on the text information to obtain the first structured data.

[0160] In some embodiments, the first decoding module 5552 is further used to search for index data of the text information from an index table, wherein the index table includes a correspondence between different text information and different index data; and perform structural adjustment processing on the index data of the text information to obtain the first structured data.

[0161] In some embodiments, the second decoding module 5553 is further used to decode the first chart feature through N cascaded dimensionality reduction layers to obtain the first data, where N is an integer greater than 1; wherein the input data of the first dimensionality reduction layer is the first chart feature; the input data of the i-th dimensionality reduction layer is the output data of the i-1-th dimensionality reduction layer, 1<i≤N; and the output data of the N-th dimensionality reduction layer is the first data.

[0162] In some embodiments, the confidence determination module 5554 is also used to digitally screen the first structured data to obtain second digital information in the first structured data; normalize the second digital information to obtain second data; and determine the confidence based on the second data and the first data.

[0163] In some embodiments, the second data includes M first values, the first data includes M second values, M is an integer greater than 1, and the confidence determination module 5554 is further used to determine M difference values ​​between the M first values ​​and the M second values, wherein one first value corresponds to one difference value; and determine the confidence based on the M difference values.

[0164] In some embodiments, the training module 5555 is used to perform image recognition processing on the chart image sample through a chart recognition model to obtain a second chart feature, second structured data and third data, wherein the image recognition processing includes: encoding the chart image sample to obtain the second chart feature; performing data conversion processing on the second chart feature to obtain the second structured data; decoding the second chart feature to obtain the third data; constructing a loss function of the chart recognition model based on the label data corresponding to the chart image sample, the second chart feature, the second structured data, and the third data; and updating the parameters of the chart recognition model based on the loss function.

[0165] In some embodiments, the training module 5555 is further used to determine a first loss based on the second graph feature and the label data; determine a second loss based on the second structured data and the third data; and integrate the loss function of the graph recognition model based on the first loss and the second loss.

[0166] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the data determination method described in the embodiment of the present application.

[0167] The present application embodiment provides a computer-readable storage medium in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will be caused to execute the data determination method provided by the present application embodiment, for example, Figures 3 to 7 The data determination method shown.

[0168] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0169] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0170] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0171] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0172] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data determination method, characterized in that: The method comprises: Performing encoding processing on the chart image to obtain a first chart feature of the chart image; Performing data conversion processing on the first chart feature to obtain first structured data corresponding to the chart image; Decoding the first chart feature to obtain first data; Based on the first structured data and the first data, a confidence level of the first structured data is determined.

2. The method according to claim 1, characterized in that The encoding process of the chart image to obtain the first chart feature of the chart image includes: determining an enhanced image of the chart image and first digital information in the chart image; Performing image coding processing on the enhanced image to obtain image coding features of the chart image; Performing text encoding processing on the first digital information to obtain digital encoding features of the chart image; Based on the image encoding feature and the digital encoding feature, a first chart feature of the chart image is determined.

3. The method according to claim 2, characterized in that The determining, based on the image coding feature and the digital coding feature, a first chart feature of the chart image comprises: Fusion of the image coding feature and the digital coding feature to obtain a fusion feature; Perform image coding processing on the fused features to obtain the first chart features.

4. The method according to claim 1, characterized in that The performing data conversion processing on the first chart feature to obtain first structured data corresponding to the chart image includes: Performing text decoding processing on the first chart feature to obtain text information of the chart image; The text information is structured to obtain the first structured data.

5. The method according to claim 4, characterized in that The performing structural processing on the text information to obtain the first structured data includes: Searching for index data of the text information from an index table, wherein the index table includes a correspondence between different text information and different index data; The index data of the text information is subjected to structural adjustment processing to obtain the first structured data.

6. The method according to claim 1, characterized in that The decoding process of the first chart feature to obtain first data includes: Decoding the first graph feature through N cascaded dimension reduction layers to obtain the first data, where N is an integer greater than 1; Among them, the input data of the first dimension reduction layer is the first chart feature; the input data of the i-th dimension reduction layer is the output data of the i-1-th dimension reduction layer, 1<i≤N; and the output data of the N-th dimension reduction layer is the first data.

7. The method according to claim 1, characterized in that The determining, based on the first structured data and the first data, the confidence of the first structured data includes: Digitally screening the first structured data to obtain second digital information in the first structured data; Normalizing the second digital information to obtain second data; The confidence level is determined based on the second data and the first data.

8. The method according to claim 7, characterized in that The second data includes M first values, and the first data includes M second values, where M is an integer greater than 1; The determining the confidence level based on the second data and the first data comprises: Determine M difference values ​​between the M first values ​​and the M second values, wherein one first value corresponds to one difference value; The confidence level is determined based on the M difference values.

9. A model training method, characterized in that: The method comprises: Performing image recognition processing on the chart image sample through the chart recognition model to obtain the second chart feature, the second structured data and the third data, wherein the image recognition processing includes: performing encoding processing on the chart image sample to obtain the second chart feature; performing data conversion processing on the second chart feature to obtain the second structured data; performing decoding processing on the second chart feature to obtain the third data; constructing a loss function of the chart recognition model based on the label data corresponding to the chart image sample, the second chart feature, the second structured data, and the third data; Parameters of the graph recognition model are updated based on the loss function.

10. The method according to claim 9, characterized in that The step of constructing a loss function of the chart recognition model based on the label data corresponding to the chart image sample, the second chart feature, the second structured data, and the third data includes: Determining a first loss based on the second graph feature and the label data; Determine a second loss based on the second structured data and the third data; A loss function of the graph recognition model is constructed based on the first loss and the second loss.

11. A data determination device, characterized in that: The device comprises: An encoding module, used for encoding the chart image to obtain a first chart feature of the chart image; A first decoding module, configured to perform data conversion processing on the first chart feature to obtain first structured data corresponding to the chart image; A second decoding module, used for decoding the first chart feature to obtain first data; A confidence determination module is used to determine the confidence of the first structured data based on the first structured data and the first data.

12. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions; A processor, configured to implement the data determination method according to any one of claims 1 to 10 when executing computer executable instructions or computer programs stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the data determination method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the data determination method according to any one of claims 1 to 10 is implemented.

Images