Image processing method and device, electronic equipment and storage medium

Abstract

The invention provides an image processing method, and relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, computer vision, large models, generative models and multi-modal data processing. According to the specific implementation scheme, an image containing a chart and an image description text of the image are extracted from a document; generating an element description text of the chart according to the image and the image description text; and generating table data according to the element description text of the chart. The invention further provides a deep learning model training method and device, electronic equipment and a storage medium.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the field of deep learning, computer vision, large models, generative models, and multimodal data processing technology. More specifically, the present disclosure provides an image processing method, a training method for a deep learning model, an apparatus, an electronic device, a storage medium, and a computer program product. Background Art

[0002] In daily work, a lot of tabular data is used, but a lot of data is presented in the form of pictures such as bar charts, line charts, and pie charts. Therefore, it is necessary to extract the information of these pictures into editable tabular form for further use and presentation. Summary of the invention

[0003] The present disclosure provides an image processing method, a training method for a deep learning model, an apparatus, an electronic device, a storage medium, and a computer program product.

[0004] According to a first aspect, an image processing method is provided, the method comprising: extracting an image containing a chart and image description text of the image from a document; generating element description text of the chart based on the image and the image description text; and generating table data based on the element description text of the chart.

[0005] According to a second aspect, a training method for a deep learning model is provided, the method comprising: extracting a sample image containing a chart and an image description text of the sample image from a sample document; using a deep learning model to convert the chart in the sample image into first tabular data based on the sample image and the image description text of the sample image; determining the loss of the deep learning model based on the difference between the first tabular data and the benchmark tabular data; and adjusting the parameters of the deep learning model based on the loss.

[0006] According to a third aspect, an image processing device is provided, which includes: a first extraction module for extracting an image containing a chart and an image description text of the image from a document; an element description text generation module for generating element description text of the chart based on the image and the image description text; and a table generation module for generating table data based on the element description text of the chart.

[0007] According to a fourth aspect, a training device for a deep learning model is provided, the device comprising: a second extraction module for extracting a sample image containing a chart and an image description text of the sample image from a sample document; a first model processing module for converting the chart in the sample image into first tabular data based on the sample image and the image description text of the sample image using a deep learning model; a loss determination module for determining the loss of the deep learning model based on the difference between the first tabular data and the benchmark tabular data; and a parameter adjustment module for adjusting the parameters of the deep learning model based on the loss.

[0008] According to a fifth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the present disclosure.

[0009] According to a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0010] According to a seventh aspect, a computer program product is provided, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and wherein the computer program implements the method provided according to the present disclosure when executed by a processor.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is a schematic diagram of an exemplary system architecture to which an image processing method and a training method for a deep learning model can be applied according to an embodiment of the present disclosure;

[0014] Figure 2 is a flowchart of an image processing method according to an embodiment of the present disclosure;

[0015] Figure 3 is a schematic diagram of an image processing method according to an embodiment of the present disclosure;

[0016] Figure 4 is a flowchart of a method for training a deep learning model according to an embodiment of the present disclosure;

[0017] Figure 5 is a schematic diagram of a training method for a deep learning model according to an embodiment of the present disclosure;

[0018] Figure 6 is a block diagram of an image processing apparatus according to an embodiment of the present disclosure;

[0019] Figure 7 is a block diagram of a training apparatus for a deep learning model according to an embodiment of the present disclosure;

[0020] Figure 8 It is a block diagram of an electronic device according to at least one of an image processing method and a deep learning model training method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] There are currently several ways to convert graphic charts (bar charts, line charts, pie charts, etc.) into editable tables.

[0023] One way to convert is to manually list the data in the image into a table. However, in the case of a large number of images, manual processing one by one is very slow and cannot meet business needs.

[0024] Another conversion method is to extract text information from the image based on traditional OCR technology. The information extracted in this way is output in a disordered format. For example, after the information on the bar chart is extracted, the names and values ​​of various types of data cannot be associated, and the data proposed is unusable.

[0025] Another conversion method is to use an open source big model to analyze and understand images. This method has a high error rate when converting bar charts, line charts, pie charts, etc. into tables. For example, when there are two lines a and b in a line chart and their colors are close, the information provided by the current multimodal big model may identify the value of line a as line b. Therefore, the information extracted by this method has a low usability.

[0026] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0027] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0028] Figure 1 is a schematic diagram of an exemplary system architecture to which an image processing method and a training method for a deep learning model can be applied according to an embodiment of the present disclosure. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0029] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0030] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices, including but not limited to smart phones, tablet computers, laptops, etc.

[0031] The image processing method provided in the embodiment of the present disclosure can generally be executed by the terminal devices 101, 102, and 103. Accordingly, the image processing apparatus provided in the embodiment of the present disclosure can generally be set in the terminal devices 101, 102, and 103.

[0032] The training method of the deep learning model provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the training device of the deep learning model provided in the embodiment of the present disclosure can generally be set in the server 105.

[0033] Figure 2 is a flowchart of an image processing method according to an embodiment of the present disclosure.

[0034] like Figure 2 As shown, the image processing method 200 includes operations S210 to S230.

[0035] In operation S210, an image including a chart and an image description text of the image are extracted from a document.

[0036] The document can be a Word document, a PDF document, a PPT document, etc. The document can include plain text, tables, and various charts. Charts such as bar charts, line charts, and pie charts can be presented in the form of images.

[0037] In order to extract the information of the chart in the image, the image containing the chart can be extracted first. In addition, the context above and / or below the chart in the document is generally the explanation or usage instructions of the chart, so the image description text content corresponding to the image can be extracted according to the context of the chart.

[0038] In operation S220, element description text of the diagram is generated according to the image and the image description text.

[0039] For example, a fine-tuned multimodal large model can be used to form an image and image description text into an image-text pair as the input of the multimodal large model. The multimodal large model processes the image-text pair to obtain the element description text of the chart.

[0040] Elements in a chart can include subject and content. For example, subject can be the name of different items in the chart, and content can be the numerical value corresponding to the subject. Image description text can include explanation and instructions for the chart, such as what color or shape is used to represent each subject in the chart, the meaning and unit of the horizontal and vertical axes of the chart, etc.

[0041] The multimodal large model identifies themes and content in chart images, associates themes in the images with themes in the image description text, and associates themes in the images with the numerical content of themes to obtain corresponding themes and content.

[0042] For example, the subject in the image is matched with the subject in the image description text according to the color or shape representing the subject in the image and the color or shape of the subject described in the image description text. The accurate content of each numerical value in the image can be determined according to the meaning and unit of the horizontal and vertical coordinates. Then, the subject is matched with the numerical content according to the position of each subject in the image and the position of the numerical value.

[0043] In operation S230, table data is generated according to the element description text of the graph.

[0044] The correspondence between topics and contents can be converted into a table. For example, each topic in the correspondence can be added to the first row of the table (i.e., the header row), that is, each topic is used as the column header of the table, and then the numerical content corresponding to each topic is used as the cell content under the corresponding column header. Similarly, each topic in the correspondence can be added to the first column of the table, that is, each topic is used as the field in the first column of the table, and then the numerical content corresponding to each topic is used as the cell content under the corresponding field.

[0045] The embodiments of the present disclosure can convert a chart in image form into an editable table by extracting an image containing a chart and image description text, generating element description text of the chart based on the image and image description text, and generating table data based on the element description text, thereby improving the efficiency and accuracy of chart information extraction.

[0046] Compared with the problems of slow manual processing, unavailable OCR extracted data, and low processing accuracy of open source models in related technologies, the embodiments of the present disclosure fine-tune the multimodal big model, take the image and image description text as the input of the fine-tuned multimodal big model, and provide the multimodal big model with more text description information about the chart. Therefore, it can improve the accuracy of the multimodal big model in extracting chart information and save a lot of time and labor costs.

[0047] Figure 3 is a schematic diagram of an image processing method according to an embodiment of the present disclosure.

[0048] like Figure 3 As shown, this embodiment includes an extraction module 310 and a multimodal large model 320. The document 301 is input into the extraction module 310 to obtain an image 311 containing a chart and an image description text 312. The image 311 and the image description text 312 are input into the multimodal large model 320 to obtain a table 321.

[0049] According to an embodiment of the present disclosure, a chart in a document is identified; an image containing the chart is extracted from the document according to the position of the chart; and image description text is determined according to context information of the chart.

[0050] The extraction module 310 may include a deep learning-based target detection model, which can identify objects such as text blocks, images, charts, and tables in the document. The identified charts may also be in the form of images. According to the position of the identified chart image, the chart image can be extracted from the document 301. Since the context of the chart generally describes the title, explanation content, and instructions for use of the chart, the image description text 312 can be determined according to the chart context.

[0051] Next, the image 311 and the image description text 312 may be combined into an image-text pair as an input to the multimodal large model 320 . The multimodal large model 320 processes the image-text pair to obtain a table 321 .

[0052] According to an embodiment of the present disclosure, at least one subject and at least one content in a diagram are determined based on an image and an image description text; a correspondence between the at least one subject and the at least one content is determined; and an element description text is generated based on the correspondence.

[0053] The elements in the chart include a theme and content, where the theme can be a project name and the content can be a numerical value corresponding to the theme. The multimodal macro model 320 can extract the theme and the numerical value from the image 311, align the theme extracted from the image 311 with the theme in the image description text 312, and then align the theme in the image 311 with the numerical value to obtain the corresponding theme and content.

[0054] In one example, the chart is a bar chart that describes the average temperature of each month in the past year. The multimodal big model 320 can align the topic "January" with the content "average temperature in January", align the topic "February" with the content "average temperature in February", and so on.

[0055] In another example, the chart may be a combined chart, such as a bar chart describing the average temperature of each month in the past year, and a line chart representing the average humidity of each month in the past year. The multimodal big model 320 may align the topic "January" with the content "average temperature in January" and "average humidity in January", align the topic "February" with the content "average temperature in February" and "average humidity in February", and so on.

[0056] After obtaining the above-mentioned mutually aligned topics and contents, the corresponding relationship between topics and contents can be converted into a table. For example, the topics "January" to "December" can be used as the first column of the table, the multiple average temperatures corresponding to "January" to "December" can be used as the second column of the table, and the multiple average humidity corresponding to "January" to "December" can be used as the third column of the table. In this way, the average temperature and humidity information of each month in the past year can be obtained in the form of a table.

[0057] The embodiments of the present disclosure align the themes and contents in the image-text pairs by using a multimodal large model, and convert the mutually aligned themes and contents into a table, thereby improving the accuracy of table data generation.

[0058] According to an embodiment of the present disclosure, the present disclosure also provides a training method for a deep learning model.

[0059] Figure 4It is a flowchart of a method for training a deep learning model according to an embodiment of the present disclosure.

[0060] like Figure 4 As shown, the training method 400 of the deep learning model includes operations S410 to S440. The deep learning model can be a pre-trained multimodal large model, and the training process of the deep learning model can also be called the fine-tuning process of the multimodal large model. The deep learning model obtained after training can be called a fine-tuned multimodal large model.

[0061] The pre-trained multimodal large model has been trained with a large amount of data and has a certain degree of versatility. Through fine-tuning, the large model can be adapted to specific tasks and data distribution to solve specific tasks.

[0062] In operation S410, a sample image including a chart and an image description text of the sample image are extracted from a sample document.

[0063] For example, the sample document is a Word document, a PDF document, a PPT document, etc. The sample document contains a chart in the form of an image. The chart can be identified from the sample document, and a sample image containing the chart can be extracted. The image description text used to explain the chart can also be extracted based on the context information of the chart in the document.

[0064] In operation S420 , a chart in the sample image is converted into first table data based on the sample image and image description text of the sample image using a deep learning model.

[0065] The deep learning model is a pre-trained multimodal large model. The sample image and the image description text are composed of image-text pairs and input into the multimodal large model. The multimodal large model can align the theme of the chart in the image with the theme in the image description text, and align the theme in the sample image with the numerical content to obtain the corresponding relationship between each theme and the numerical content. The corresponding relationship between the theme and the numerical content is expressed in the form of a table to obtain the first table data.

[0066] In operation S430, a loss of the deep learning model is determined according to a difference between the first table data and the reference table data.

[0067] The above-mentioned reference table data can be accurate table data corresponding to the chart, and the reference table data is used as the expected output of the multimodal large model. According to the difference between the first table data actually output by the multimodal large model and the reference table data, the loss of the multimodal large model can be determined.

[0068] For example, the difference between the content of each cell in the first table data and the content of the corresponding cell in the reference table data can be calculated to determine the loss of each cell. The sum or weighted sum of the losses of each cell can be used as the overall loss of the deep learning model. The above difference can be, for example, the mean square error, the squared error, etc.

[0069] In operation S440, parameters of the deep learning model are adjusted according to the loss.

[0070] According to the above overall loss, the back propagation algorithm, gradient descent algorithm, etc. can be used to adjust the parameters of the multimodal large model to complete a round of training. Repeat the above training process, and after multiple rounds of training, a trained deep learning model, that is, a fine-tuned multimodal large model, can be obtained. The fine-tuned multimodal large model has the ability to convert image-based charts into tables.

[0071] The embodiments of the present disclosure fine-tune the multimodal large model by using sample images containing charts and sample image description texts, so that the multimodal large model has the ability to convert chart images into tables, which can improve the accuracy of chart information extraction.

[0072] Figure 5 It is a schematic diagram of a training method for a deep learning model according to an embodiment of the present disclosure.

[0073] like Figure 5 As shown, this embodiment is an illustration of training a large multimodal model 530.

[0074] The sample document 510 may be a Word document, a PDF document, a PPT document, etc. The sample document 510 may include a chart in the form of an image. A sample image 521 including the chart and a sample image description text 522 may be extracted from the sample document 510.

[0075] The sample image 521 and the sample image description text 522 form an image-text pair 520 and are input into the multimodal large model 530, and the multimodal large model 530 outputs the first tabular data 531. In addition, the sample image 521 is input into the multimodal large model 530 alone to obtain the second tabular data 532, and the second tabular data 532 is corrected to obtain the reference tabular data 533. The difference between the first tabular data 531 and the reference tabular data 533 is calculated to determine the loss 540, and the parameters of the multimodal large model 530 are adjusted according to the loss 540.

[0076] According to an embodiment of the present disclosure, based on the sample image and the image description text of the sample image, a first element description text of the chart in the sample image is generated; based on the first element description text, the chart in the sample image is converted into the first table data.

[0077] For example, the multimodal macromodel 530 can correspond the subject in the sample image 521 to the subject in the sample image description text 522 according to the color or shape representing the subject in the image and the color or shape of the subject described in the sample image description text. According to the meaning and unit of the horizontal and vertical coordinates, the accurate content of each numerical value in the sample image 521 can be determined. Then, according to the position of each numerical value in the sample image 521, the subject is corresponded to the numerical content to obtain the first element description text, and then the first element description text is converted into the first table data 531.

[0078] After the first table data 531 is obtained, the difference between the first table data 531 and the reference table data 533 may be calculated, and the loss 540 may be determined according to the difference.

[0079] According to an embodiment of the present disclosure, the benchmark table data is converted into a second element description text; and the loss of the deep learning model is determined based on the difference between the first element description text and the second element description text.

[0080] For example, the first element description text may be the content of each cell in the first table data 531, and the second element description text may be the content of each cell in the reference table data 533. Based on the difference between the content of each cell in the first table data 531 and the content of each cell in the reference table data 533, the loss 540 is determined.

[0081] The reference table data 533, as supervisory data during the training process of the multimodal large model 530, may be obtained before training the multimodal large model 530. The acquisition of the reference table data 533 is described below.

[0082] In one example, the sample image 521 may be manually processed to obtain the reference table data 533. However, due to the large sample size, manual processing is difficult to meet business needs.

[0083] According to an embodiment of the present disclosure, a deep learning model is used to convert a chart in a sample image into second table data; and the second table data is corrected to obtain reference table data.

[0084] For example, the multimodal macro model 530 may be used to batch process the sample images 521 to obtain the second table data 532. The second table data 532 may then be modified based on a manual or automated application, which can greatly improve the efficiency of obtaining the reference table data 533.

[0085] The above-mentioned correction of the second table data 532 may include filtering out data without accurate numerical values ​​in the sample image 521, for example, only a broken line without a numerical value on the line.

[0086] According to an embodiment of the present disclosure, the multimodal large model 530 processes the sample image 521 by generating a third element description text of the chart in the sample image based on the sample image; and converting the chart in the sample image into second tabular data according to the third element description text.

[0087] For example, the multimodal large model 530 identifies multiple topics and multiple values ​​in the sample image 521, and matches the multiple topics and multiple values ​​according to the positional relationship between the multiple topics and the multiple values ​​in the sample image. Due to the lack of specific diagram explanation content, matching topics and values ​​only based on the positional relationship may result in errors.

[0088] Therefore, the embodiment of the present disclosure corrects the second table data 532 to obtain accurate table data as the reference table data 533 , thereby ensuring that the table data 533 is correct and usable.

[0089] It should be noted that the steps of batch processing the sample images 521 using the multimodal large model 530 to obtain the second table data 532, and correcting the second table data 532 to obtain the reference table data 533 can be performed before the training of the multimodal large model 530. In other words, the reference table data 533 is prepared before the training of the multimodal large model 530.

[0090] The embodiment of the present disclosure uses a multimodal large model to batch process sample images before training to obtain second table data, and corrects the second table data to obtain benchmark table data, thereby improving the efficiency of obtaining benchmark table data.

[0091] During the training process, the embodiments of the present disclosure use image-text pairs as input of the multimodal big model, providing the multimodal big model with text data for explaining the graphs, thereby optimizing the application of the big model in the graph information extraction scenario and improving the accuracy of the multimodal big model in extracting graph information.

[0092] According to an embodiment of the present disclosure, the present disclosure also provides an image processing device and a training device for a deep learning model.

[0093] Figure 6 is a block diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0094] like Figure 6 As shown, the image processing device 600 includes a first extraction module 610 , an element description text generation module 620 and a table generation module 630 .

[0095] The first extraction module 610 is used to extract images containing charts and image description texts of the images from the document.

[0096] The element description text generation module 620 is used to generate element description text of the chart according to the image and the image description text.

[0097] The table generation module 630 is used to generate table data according to the element description text of the chart.

[0098] According to an embodiment of the present disclosure, an element includes a subject and a content. The element description text generation module 620 includes an element determination unit, a correspondence determination unit, and an element description text generation unit.

[0099] The element determination unit is used to determine at least one subject and at least one content in the diagram according to the image and the image description text.

[0100] The correspondence determination unit is used to determine the correspondence between at least one subject and at least one content.

[0101] The element description text generating unit is used to generate the element description text according to the corresponding relationship.

[0102] The first extraction module 610 includes a recognition unit, an image extraction unit, and an image description text extraction unit.

[0103] The recognition unit is used to recognize diagrams in documents.

[0104] The image extraction unit is used for extracting an image containing a chart from a document according to the position of the chart.

[0105] The image description text extraction unit is used to determine the image description text according to the context information of the chart.

[0106] The chart includes at least one of a bar chart, a line chart, and a pie chart.

[0107] Figure 7 is a block diagram of a training device for a deep learning model according to an embodiment of the present disclosure.

[0108] like Figure 7 As shown, the training device 700 of the deep learning model includes a second extraction module 710, a first model processing module 720, a loss determination module 730 and a parameter adjustment module 740.

[0109] The second extraction module 710 is used to extract sample images containing charts and image description texts of the sample images from the sample documents.

[0110] The first model processing module 720 is used to convert the chart in the sample image into first tabular data based on the sample image and the image description text of the sample image by using a deep learning model.

[0111] The loss determination module 730 is used to determine the loss of the deep learning model according to the difference between the first table data and the reference table data.

[0112] The parameter adjustment module 740 is used to adjust the parameters of the deep learning model according to the loss.

[0113] The first model processing module 720 includes a first element description text generating unit and a first table data generating unit.

[0114] The first element description text generating unit is used to generate the first element description text of the chart in the sample image based on the sample image and the image description text of the sample image.

[0115] The first table data generating unit is used to convert the chart in the sample image into first table data according to the first element description text.

[0116] The loss determination module 730 includes a second element description text generation unit and a loss determination unit.

[0117] The second element description text generating unit is used for converting the reference table data into a second element description text.

[0118] The loss determination unit is used to determine the loss of the deep learning model according to the difference between the first element description text and the second element description text.

[0119] The deep learning model training device 700 also includes a second model processing module and a correction module.

[0120] The second model processing module is used to convert the chart in the sample image into second tabular data using the deep learning model.

[0121] The correction module is used to correct the second table data to obtain the reference table data.

[0122] The second model processing module includes a third element description text generating unit and a second table data generating unit.

[0123] The third element description text generating unit is used to generate the third element description text of the chart in the sample image based on the sample image.

[0124] The second table data generating unit is used to convert the chart in the sample image into second table data according to the third element description text.

[0125] The chart includes at least one of a bar chart, a line chart, and a pie chart.

[0126] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0127] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0128] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0129] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0130] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as at least one of the image processing method and the training method of the deep learning model. For example, in some embodiments, at least one of the image processing method and the training method of the deep learning model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of at least one of the image processing method and the training method of the deep learning model described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute at least one of the image processing method and the deep learning model training method in any other appropriate manner (for example, by means of firmware).

[0131] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0133] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0134] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0135] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0136] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0137] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0138] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image processing method, comprising: extracting an image containing a diagram and image description text of the image from a document; Generate element description text of the chart according to the image and the image description text; as well as Generate tabular data based on the element description text of the diagram.

2. The method according to claim 1, wherein: The elements include a theme and content; the element description text of the chart generated according to the image and the image description text includes: Determining at least one subject and at least one content in the chart according to the image and the image description text; determining a correspondence between the at least one subject and the at least one content; and The element description text is generated according to the corresponding relationship.

3. The method according to claim 1, wherein: The step of extracting an image containing a chart and an image description text of the image from a document comprises: identifying a diagram in the document; extracting the image containing the chart from the document according to the position of the chart; and The image description text is determined according to the context information of the chart.

4. The method according to any one of claims 1 to 3, wherein: The chart includes at least one of a bar chart, a line chart, and a pie chart.

5. A method for training a deep learning model, comprising: Extracting a sample image containing a chart and an image description text of the sample image from a sample document; Using a deep learning model, based on the sample image and the image description text of the sample image, converting a chart in the sample image into first table data; determining a loss of the deep learning model based on a difference between the first tabular data and the benchmark tabular data; and Parameters of the deep learning model are adjusted according to the loss.

6. The method according to claim 5, wherein: The converting the chart in the sample image into first table data based on the sample image and the image description text of the sample image using the deep learning model comprises: Based on the sample image and the image description text of the sample image, generating a first element description text of a chart in the sample image; and The chart in the sample image is converted into the first table data according to the first element description text.

7. The method according to claim 6, wherein: Determining the loss of the deep learning model according to the difference between the first table data and the reference table data includes: converting the reference table data into a second element description text; and Determine the loss of the deep learning model based on the difference between the first element description text and the second element description text.

8. The method according to claim 5, further comprising: Using the deep learning model, converting the chart in the sample image into second tabular data; as well as The second table data is corrected to obtain the reference table data.

9. The method according to claim 8, wherein: The converting the chart in the sample image into second table data using the deep learning model comprises: Based on the sample image, generating a third element description text of the chart in the sample image; and According to the third element description text, the chart in the sample image is converted into the second table data.

10. The method according to any one of claims 5 to 9, wherein: The chart includes at least one of a bar chart, a line chart, and a pie chart.

11. An image processing device, comprising: A first extraction module, used for extracting an image containing a chart and an image description text of the image from a document; an element description text generation module, used to generate an element description text of the chart according to the image and the image description text; and The table generation module is used to generate table data according to the element description text of the chart.

12. The device according to claim 11, wherein: The element includes a subject and content; the element description text generation module includes: an element determination unit, configured to determine at least one subject and at least one content in the diagram according to the image and the image description text; a correspondence determination unit, configured to determine a correspondence between the at least one subject and the at least one content; and An element description text generating unit is used to generate the element description text according to the corresponding relationship.

13. The device according to claim 11, wherein: The first extraction module comprises: A recognition unit, used for recognizing a chart in the document; an image extraction unit, configured to extract the image containing the chart from the document according to the position of the chart; and The image description text extraction unit is used to determine the image description text according to the context information of the chart.

14. The device according to any one of claims 11 to 13, wherein: The chart includes at least one of a bar chart, a line chart, and a pie chart.

15. A training device for a deep learning model, comprising: A second extraction module is used to extract a sample image containing a chart and an image description text of the sample image from the sample document; A first model processing module, configured to convert a chart in the sample image into first table data based on the sample image and the image description text of the sample image using a deep learning model; a loss determination module, configured to determine a loss of the deep learning model based on a difference between the first table data and a reference table data; and A parameter adjustment module is used to adjust the parameters of the deep learning model according to the loss.

16. The device according to claim 15, wherein: The first model processing module includes: A first element description text generating unit, configured to generate a first element description text of a chart in the sample image based on the sample image and the image description text of the sample image; and The first table data generating unit is configured to convert the chart in the sample image into the first table data according to the first element description text.

17. The device according to claim 16, wherein: The loss determination module comprises: A second element description text generating unit, configured to convert the reference table data into a second element description text; and A loss determination unit is used to determine the loss of the deep learning model according to the difference between the first element description text and the second element description text.

18. The apparatus according to claim 15, further comprising: A second model processing module, configured to convert the chart in the sample image into second tabular data using the deep learning model; as well as The correction module is used to correct the second table data to obtain the reference table data.

19. The device according to claim 18, wherein: The second model processing module includes: A third element description text generating unit, configured to generate a third element description text of a chart in the sample image based on the sample image; and The second table data generating unit is configured to convert the chart in the sample image into the second table data according to the third element description text.

20. The device according to any one of claims 15 to 19, wherein: The chart includes at least one of a bar chart, a line chart, and a pie chart.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.

23. A computer program product, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images