Image processing method and device, storage medium and program product

Abstract

The invention provides an image processing method, electronic equipment, a computer readable storage medium and a computer program product, and the method comprises the steps: carrying out the recognition of an image comprising a chart, obtaining a first value related to a graphic element in the chart, and enabling the graphic element to be used for displaying the first value in a graphic manner; performing numerical value prediction on the graphic element based on the image to obtain a second numerical value corresponding to the graphic element; and comparing the first numerical value with the second numerical value, and determining whether the first numerical value in the chart is a real numerical value based on the comparison result. Therefore, whether the numerical value in the chart is tampered or not can be determined.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image processing method, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of artificial intelligence, there is a need to answer questions about document information. When a document includes chart data, the information in the chart data needs to be identified first. Therefore, related technologies mainly focus on how to identify charts and improve the accuracy of chart recognition. Summary of the Invention

[0003] Embodiments of the present application provide an image processing method, an electronic device, and a computer-readable storage medium, which can determine whether the values ​​in a chart have been tampered with.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present invention provides an image processing method, which includes:

[0006] Recognizing an image including a chart to obtain a first numerical value associated with a graphic element in the chart, wherein the graphic element is used to graphically display the first numerical value;

[0007] Based on the image, performing a numerical prediction on the graphic element to obtain a second numerical value corresponding to the graphic element;

[0008] The first value and the second value are compared, and based on the comparison result, it is determined whether the first value in the chart is a true value.

[0009] An embodiment of the present application provides an image processing device, the device comprising:

[0010] a data extraction module, configured to extract data from an image including a graphic and data of the graphic, and obtain first data of the graphic in the image;

[0011] a data prediction module, configured to perform data prediction based on a graphic in the image to obtain second data of the graphic;

[0012] The detection module is used to determine the difference between the first data of the graphic and the second data of the graphic, and detect the authenticity of the image based on the difference.

[0013] An embodiment of the present application provides an electronic device, including:

[0014] Memory for storing computer-executable instructions or computer programs;

[0015] The processor is used to implement the image processing method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0016] An embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the image processing method provided by the embodiment of the present application.

[0017] The present invention provides a computer program product including a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the electronic device to perform the image processing method provided in the present invention.

[0018] The embodiments of the present application have the following beneficial effects:

[0019] For an image that includes a chart, while the image is identified to obtain the first numerical value associated with the graphic element in the chart, the graphic element is also numerically predicted to obtain the second numerical value corresponding to the graphic element, thereby determining whether the first numerical value in the chart is a true numerical value by comparing the first numerical value with the second numerical value; thus, compared to the solution in the related art that only identifies the chart in the image to obtain the numerical value of the chart, the present application determines whether the numerical value of the chart is a true numerical value by comparing two numerical values ​​obtained based on different methods, thereby determining whether the numerical value in the chart has been tampered with. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 1 is a schematic diagram of the architecture of an image processing system 100 provided in an embodiment of the present application;

[0021] Figure 2 is a structural diagram of an electronic device provided in an embodiment of the present application;

[0022] Figure 3 Schematic diagram of the image processing method provided in the embodiment of the present application;

[0023] Figure 4 is a schematic diagram of an image including a chart provided in an embodiment of the present application;

[0024] Figure 5 is a schematic diagram of a process for determining a first value provided in an embodiment of the present application;

[0025] Figure 6 is a schematic diagram of a flow chart for verifying a second value provided in an embodiment of the present application;

[0026] Figure 7 is a structural diagram of an image processing system provided in an embodiment of the present application;

[0027] Figure 8 It is a structural diagram of the image processing model provided in the embodiment of the present application;

[0028] Figure 9 1 is a flow chart of the training process of the numerical recognition model and the numerical prediction model provided in the embodiments of the present application;

[0029] Figure 10 Schematic diagram of the training process of the image processing model provided in the embodiment of the present application;

[0030] Figure 11 This is a technical architecture diagram of an image processing method implemented through an image processing model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0032] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0035] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0036] 1) Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. AI technology is an interdisciplinary discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics.

[0037] 2) Client, also known as the user end, refers to the program corresponding to the server that provides local services to users. Except for some applications that can only run locally, it is generally installed on an ordinary client and needs to cooperate with the server to run. That is, there must be corresponding servers and service programs in the network to provide corresponding services. In this way, a specific communication connection needs to be established between the client and the server to ensure the normal operation of the application.

[0038] During the research process, the inventors found that when processing data including charts, related technologies mostly focus on how to identify chart data and improve the accuracy of chart data identification, but do not consider the problem that chart data may be tampered with.

[0039] Based on this, an embodiment of the present application provides an image processing method. For an image including a chart, while identifying the image to obtain a first numerical value associated with a graphic element in the chart, the graphic element is also numerically predicted to obtain a second numerical value corresponding to the graphic element, thereby determining whether the first numerical value in the chart is a true numerical value by comparing the first numerical value with the second numerical value; thus, compared with the solution in the related art of only identifying the chart in the image to obtain the numerical value of the chart, the present application determines whether the numerical value of the chart is a true numerical value by comparing two numerical values ​​obtained based on different methods, thereby determining whether the numerical value in the chart has been tampered with.

[0040] See also Figure 1 , Figure 1 This is an architectural diagram of the image processing system 100 provided in an embodiment of the present application. To implement an application scenario of image processing, a terminal (terminal 400 is shown as an example) is connected to a server 200 via a network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. The terminal 400 is used for users to use a client 401, which is displayed on a display interface (display interface 401-1 is shown as an example). The terminal 400 and the server 200 are connected to each other via a wired or wireless network.

[0041] The terminal 400 is used to send images including charts to the server 200;

[0042] The server 200 is used to receive an image including a chart; identify the image including the chart to obtain a first numerical value associated with a graphic element in the chart, where the graphic element is used to display the first numerical value in a graphical manner; perform numerical prediction on the graphic element based on the image to obtain a second numerical value corresponding to the graphic element; compare the first numerical value and the second numerical value, and determine whether the first numerical value in the chart is a true numerical value based on the comparison result.

[0043] In some embodiments, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Deliver Network), and big data and artificial intelligence platforms. The terminal 400 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a set-top box, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, and a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device, a smart speaker, and a smart watch), etc., but is not limited thereto. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0044] See also Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. In practical applications, the electronic device can be Figure 1 The server 200 or terminal 400 shown, see Figure 2 , Figure 2 The electronic device shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0045] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0046] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0047] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0048] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0049] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0050] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0051] A network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include Bluetooth, Wi-Fi, and Universal Serial Bus (USB).

[0052] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0053] The input processing module 454 is configured to detect one or more user inputs or interactions from one of the one or more input devices 432 and to translate the detected inputs or interactions.

[0054] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2 An image processing device 455 stored in memory 450 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a recognition module 4551, a prediction module 4552, and a comparison module 4553. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0055] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the image processing device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0056] In some embodiments, the terminal or server can implement the image processing method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as an instant messaging APP, a web browser APP; it can also be a small program, that is, a program that can be run only by downloading it into a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plug-in.

[0057] Based on the above description of the image processing system and electronic device provided by the embodiment of the present application, the image processing method provided by the embodiment of the present application is described below. In actual implementation, the image processing method provided by the embodiment of the present application can be implemented by the terminal or the server alone, or by the terminal and the server in collaboration, so that Figure 1 The server 200 in the embodiment of the present application alone performs the image processing method provided by the embodiment of the present application as an example for explanation. Figure 3 , Figure 3 This is a flow chart of the image processing method provided by the embodiment of the present application. Figure 3 The steps shown will be described.

[0058] In step 101 , the server identifies an image including a chart and obtains a first numerical value associated with a graphic element in the chart. The graphic element is used to display the first numerical value in a graphical manner.

[0059] In actual implementation, the image including the chart can be pre-stored locally or obtained from the outside world (such as the Internet); as an example, the process of the server obtaining the image including the chart can be that the server receives an image processing request sent by the terminal, wherein the image processing request carries the image including the chart, thereby parsing the image processing request to determine the image including the chart, and then identifying the image including the chart to obtain the first numerical value associated with the graphic element in the chart; or, the server can also receive an image processing request sent by the terminal, wherein the image processing request carries an image identifier, thereby parsing the image processing request, determining the image identifier, and then obtaining the image including the chart from the local based on the image identifier.

[0060] It should be noted that the chart here can be a bar chart, a line chart image, a pie chart, etc.; and the graphic element in the chart refers to a graphic in the chart image (that is, an image including the chart) that can indicate the marked data. For example, when the chart is a bar chart, the graphic element is the bar graph in the bar chart, and when the chart is a pie chart, the graphic element is the sector in the pie chart; therefore, for the graphic element used to display the first value in a graphical way, when the chart is a bar chart and the graphic element is the bar graph in the bar chart, the first value can be displayed through the bar graph, and when the chart is a pie chart and the graphic element is the sector in the pie chart, the first value can be displayed through the sector.

[0061] For example, see Figure 4 , Figure 4 is a schematic diagram of an image including a chart provided in an embodiment of the present application, based on Figure 4 , Figure 4 The indicated chart image corresponds to a bar graph, each bar graph is a graphic element, and the value on each bar graph is a first value.

[0062] It should be noted that the image including the chart can be obtained through format conversion. For example, for tabular data, the format of the tabular data is converted to obtain a chart image corresponding to the tabular data, that is, an image including the chart; or for text data, the format of the text data is converted to obtain an image including the chart, etc.

[0063] In actual implementation, the first numerical value in the image is presented in the form of text. For the process of recognizing an image including a chart and obtaining the first numerical value associated with the graphic element in the chart, refer to Figure 5 , Figure 5 This is a flow chart of determining a first value provided by an embodiment of the present application, based on Figure 5 , step 101 can be implemented by the following steps.

[0064] Step 1011: Encode the image to obtain image features.

[0065] In actual implementation, the process of encoding an image to obtain image features of the image may include: dividing the image into blocks to obtain at least two image blocks; for each image block, extracting pixel features of the pixels in the image block to obtain pixel features of each pixel; and performing feature splicing on the pixel features of multiple pixels to obtain pixel features of the image block; for each image block, extracting position features of the image block to obtain position features for indicating the position of the image block in the image; fusing the position features of each image block with the pixel features to obtain image features of the image block; and splicing the image features of multiple image blocks to obtain intermediate image features of the image;

[0066] Then, the intermediate image features are multiplied by the first linear transformation matrix to obtain a first query matrix, the intermediate image features are multiplied by the second linear transformation matrix to obtain a first key value matrix, and the intermediate image features are multiplied by the third linear transformation matrix to obtain a first value matrix; wherein the first linear transformation matrix is ​​a linear transformation matrix corresponding to the preset first query matrix, the second linear transformation matrix is ​​a linear transformation matrix corresponding to the preset first key value matrix, and the third linear transformation matrix is ​​a linear transformation matrix corresponding to the preset first value matrix;

[0067] Then, the attention mechanism is used to learn the first query matrix, the first key matrix and the first value matrix to obtain the image features of the image, that is,

[0068]

[0069] Among them, Z is used to indicate the image features of the image, Q is the first query matrix, K is the first key matrix, V is the first value matrix, KT is the transposed matrix of the key-value matrix, d k Used to indicate the number of columns of the first query matrix and the first key-value matrix, that is, the vector dimension.

[0070] Step 1012: Determine text features of the text corresponding to the first value in the image based on the image features of the image.

[0071] In actual implementation, based on the image features of the image, the process of determining the text features of the text corresponding to the first numerical value in the image may be: multiplying the image features with the first linear transformation matrix to obtain a second query matrix, multiplying the image features with the second linear transformation matrix to obtain a second key value matrix, multiplying the image features with the third linear transformation matrix to obtain a second value matrix; multiplying the second query matrix with the transposed matrix of the second key value matrix to obtain a product matrix, and multiplying the product matrix with a pre-set mask matrix to obtain a mask product matrix; multiplying the mask product matrix with the second value matrix to obtain an intermediate text feature; wherein the dimension of the intermediate text feature is the same as the dimension of the image feature;

[0072] Multiply the intermediate text features by the first linear transformation matrix to obtain a third query matrix; then, use the attention mechanism to learn the third query matrix, the second key-value matrix, and the second value matrix to obtain text features. Here, the process of using the attention mechanism to learn the third query matrix, the second key-value matrix, and the second value matrix is ​​similar to the process of using the attention mechanism to learn the first query matrix, the first key-value matrix, and the first value matrix as described above, and will not be repeated in the embodiments of this application.

[0073] Step 1013: Predict and obtain a first numerical value associated with a graphic element in the chart based on the text feature.

[0074] In actual implementation, the process of predicting the first numerical value associated with the graphic element in the chart based on the text features may be: based on the text features and a character sequence including i-1 characters, predicting the i-th character, where i is an integer greater than 0; and determining the first numerical value associated with the graphic element in the chart based on the character sequence composed of the predicted i characters.

[0075] Among them, i is incremental. When the end condition is met, such as the i-th character is the end character, or the number of characters in the character sequence reaches the target number, the first numerical value associated with the graphic element in the chart is determined based on the character sequence composed of the predicted i characters; for example, i can be an integer greater than 0 and less than or equal to I, and I is a positive integer, so that when the predicted character sequence includes I characters, that is, when i is 1, the first numerical value associated with the graphic element in the chart is determined based on the character sequence composed of the predicted i characters.

[0076] It should be noted that the process of predicting the first numerical value associated with the graphic element in the chart based on the text features can specifically include traversing i to perform the following processing: generating the i-th character based on the text features and a character sequence including i-1 characters; then, inserting the i-th character into the i-th position of the character sequence to obtain a character sequence including i characters; if the end condition is met, such as the i-th character is the end character, or the number of characters in the character sequence reaches the target number, determining the first numerical value associated with the graphic element in the chart from the i characters included in the character sequence.

[0077] It should be noted that since the character sequence consisting of the predicted i characters includes all characters in the image, some characters, such as punctuation marks, are not the first numerical values ​​associated with the graphic elements in the chart. Based on this, it is necessary to select the first numerical values ​​associated with the graphic elements in the chart from the i characters included in the character sequence.

[0078] For example, following the above example, see Figure 4 , when the end condition is met, the predicted character sequence is {2014|7.2, 2015|10.5, 2016|12.8, 2017|10.6, 2018|6.7, 2019|6.1, 2020|6.7, 2021|5.8, 2022|5.1}, and from the i characters included in the character sequence, the first numerical value associated with the graphic element in the chart is selected, that is, 7.2, 10.5, 12.8, 10.6, 6.7, 6.1, 6.7, 5.8, and 5.1 are selected.

[0079] Step 102: Based on the image, perform numerical prediction on the graphic element to obtain a second numerical value corresponding to the graphic element.

[0080] In actual implementation, the process of performing numerical prediction on graphic elements based on an image to obtain a second numerical value corresponding to the graphic element may be to segment the image to obtain a sub-image corresponding to each graphic element in the image; for each sub-image corresponding to a graphic element, the sub-image is input into a model for performing numerical prediction on the graphic element to obtain the second numerical value corresponding to the graphic element.

[0081] It should be noted that the model used for numerical prediction of graphic elements is a pre-trained prediction model; after the sub-image is input into the model for numerical prediction of graphic elements, the numerical value corresponding to the sub-image can be determined based on the sub-image, that is, the second numerical value corresponding to the graphic element.

[0082] For example, following the above example, see Figure 4, perform image segmentation on the image to obtain the sub-image corresponding to each graphic element in the image, that is, determine the image corresponding to each columnar graphic, and then perform numerical prediction on the image corresponding to each columnar graphic to obtain the numerical value corresponding to each columnar graphic.

[0083] In actual implementation, the process of performing numerical prediction on the graphic element based on the image to obtain the second numerical value corresponding to the graphic element may also include encoding the image to obtain image features of the image; determining image features corresponding to the graphic element in the image based on the image features; and predicting the second numerical value corresponding to the graphic element based on the image features corresponding to the graphic element.

[0084] Among them, the process of encoding the image here to obtain the image features of the image is the same as the process of encoding the image to obtain the image features of the image in the aforementioned recognition of the image including the chart to obtain the first numerical value associated with the graphic element in the chart; and the process of determining the image features corresponding to the graphic elements in the image based on the image features is similar to the process of determining the text features of the text corresponding to the first numerical value in the image based on the image features in the aforementioned recognition of the image including the chart to obtain the first numerical value associated with the graphic element in the chart, and will not be repeated here; and the process of predicting the second numerical value corresponding to the graphic element based on the image features corresponding to the graphic element is also similar to the process of predicting the first numerical value associated with the graphic element in the chart based on the text features in the aforementioned recognition of the image including the chart to obtain the first numerical value associated with the graphic element in the chart, and will not be repeated here.

[0085] In some embodiments, after a numerical value prediction is performed on a graphic element based on an image and a second numerical value corresponding to the graphic element is obtained, the second numerical value may be verified. Only when the second numerical value passes the verification is the second numerical value used in a subsequent process to determine whether the first numerical value is a true numerical value; specifically, see Figure 6 , Figure 6 This is a flow chart of verifying the second value provided by the embodiment of the present application, based on Figure 6 The process of verifying the second value is implemented by the following steps.

[0086] In step 201 , the server determines a third value of a geometric parameter of a graphic element, and determines a ratio of a second value to a corresponding third value.

[0087] It should be noted that the geometric parameters of a graphical element refer to the area, height, etc. of the graphical element. For example, if the graphical element is a bar chart, the geometric parameter may be the height of the bar chart; if the graphical element is a pie chart, the geometric parameter may be the area of ​​the pie chart. Therefore, after determining the third ratio of the geometric parameters of each graphical element, the second value of the graphical element is divided by the third value to obtain the ratio of the second value to the third value.

[0088] Step 202 : Determine the scale of the chart, and verify the second value based on the scale and ratio of the chart to obtain a verification result.

[0089] It should be noted that the scale is used to indicate the relationship between the graphic elements and the actual data values ​​in the chart, and here it is used to indicate that the actual data value is the second value, so the scale here is used to indicate the relationship between the graphic elements and the second value.

[0090] In actual implementation, the process of determining the scale of the chart can be, when there are multiple graphic elements, clustering the ratios corresponding to the multiple graphic elements to obtain clusters; from the clusters, selecting clusters whose number of ratios is greater than a quantity threshold; determining the average value of the ratios in the selected clusters, and using the average value as the scale of the chart.

[0091] Among them, the quantity threshold can be pre-set, for example, the quantity threshold is set to the number of ratios included in the cluster that includes the largest number of ratios in at least one cluster, that is, the cluster with the largest number of ratios is selected, and the number of ratios included in the cluster is used as the quantity threshold.

[0092] It should be noted that if each second numerical value is predicted accurately, the ratio corresponding to each graphic element should be the same. Therefore, after clustering the ratios corresponding to multiple graphic elements, the number of clusters obtained should be one, and this cluster only includes the same ratios, so that the ratio is directly used as the scale of the chart. If there is an inaccurate prediction of the second numerical value, the ratio corresponding to the graphic element will be different from the other ratios. Therefore, after clustering the ratios corresponding to multiple graphic elements, the number of clusters obtained should be at least two. At the same time, it can be determined that the number of accurately predicted second numerical values ​​is definitely greater than the number of inaccurately predicted second numerical values. Therefore, the cluster with the largest number of ratios is selected, and the average value of the ratios in the selected cluster is determined, so that the average value is used as the scale of the chart.

[0093] In actual implementation, the verification result includes a first verification result and a second verification result, so that the second value is verified based on the scale and ratio of the chart to obtain the verification result. The process can be to determine the difference between the scale and the ratio; if the difference is greater than or equal to the first difference threshold, the verification result is determined to be the first verification result, and the first verification result is used to indicate that the verification of the second value fails; if the difference is less than the first difference threshold, the verification result is determined to be the second verification result, and the second verification result is used to indicate that the verification of the second value passes; thus, after the second value is verified based on the scale of the chart and the ratio, if the verification result is the first verification result, the second value can be corrected based on the scale of the chart.

[0094] It should be noted that, as mentioned above, the scale is the average value of the ratio in the selected cluster. Therefore, the scale is a numerical value. In this way, the scale can be subtracted from the ratio of each graphic element. If the difference is greater than or equal to the first difference threshold, the verification result of the corresponding graphic element is determined to be the first verification result; if the difference is less than the first difference threshold, the verification result of the corresponding graphic element is determined to be the second verification result; wherein, the first difference threshold can also be pre-set, which is not limited in the embodiment of the present application.

[0095] In actual implementation, if the verification result is the first verification result, the process of correcting the second value is based on the scale of the chart. Specifically, the third value of the geometric parameter of the graphic element is multiplied by the scale of the icon to obtain the standard second value, and the current second value is adjusted to the standard second value, thereby completing the process of correcting the second value.

[0096] Step 203 : When the verification result indicates that the second value has passed the verification, compare the first value and the second value.

[0097] In actual implementation, the process of comparing the first and second values ​​in subsequent processing may be such that the first and second values ​​are compared only when the verification result indicates that the second value has passed the verification. In this way, by verifying the second value, and only comparing the second value with the first value when the second value passes the verification, the accuracy of the second value compared with the first value is guaranteed, thereby improving the accuracy of the result of determining whether the first value is a true value.

[0098] It should be noted that if the verification result indicates that the second value has failed verification, if the second value is corrected, the first value and the corrected second value can be compared. In this way, if the verification result indicates that the second value has failed verification, by correcting the second value, the corrected second value can be compared with the first value. This ensures that each first value can be compared, that is, it can be determined whether each first value is a true value, thereby ensuring that whether the values ​​in the chart have been tampered with can be fully determined.

[0099] It should be noted that since the second value is predicted based on the graphic elements, it can be understood that it is easy to tamper with the marked values ​​in the chart, but it is more difficult to tamper with the graphic elements in the chart image. Therefore, the predicted second value can indicate the true value represented by the graphic elements in the chart. In this way, in the subsequent process, the first value can be compared with the second value to determine whether the first value is the true value.

[0100] Step 103 : Compare the first value and the second value, and determine whether the first value in the chart is a true value based on the comparison result.

[0101] In actual implementation, the process of determining whether the first value in the chart is a true value based on the comparison result may be: if the comparison result indicates that the difference between the first value and the second value is greater than or equal to a second difference threshold, then it is determined that the first value in the chart is not a true value; if the comparison result indicates that the difference between the first value and the second value is less than the second difference threshold, then it is determined that the first value in the chart is a true value.

[0102] It should be noted that the second difference threshold can be pre-set, and if it is determined that the first value in the chart is a true value, it means that the value in the chart has not been tampered with or forged, and if it is determined that the first value in the chart is not a true value, it means that the value in the chart has been tampered with or forged.

[0103] For example, see Figure 4 ,by Figure 4 Taking the first column graph in as an example, based on Figure 4 The first value obtained by performing data recognition on the indicated image is 7.2. Figure 4 The second value obtained by numerical prediction of the indicated image is 6.9, and the difference between the first value and the second value is 0.3. At this time, if the second difference threshold is 0.5, the first difference is less than the second difference threshold, that is, the corresponding first value is a true value; if the second difference threshold is 0.1, the first difference is greater than or equal to the second difference threshold, and it is determined that the corresponding first value is not a true value.

[0104] In actual implementation, the number of graphic elements may be at least one, that is, the number of first values ​​may be at least one, and the number of second values ​​may also be at least one. When the number of first values ​​and second values ​​is both one, the process of comparing the first values ​​and the second values ​​is as described above and is not repeated here.

[0105] When there are multiple first values ​​and multiple second values, there may be multiple processes for comparing the first values ​​and the second values. Next, the process of comparing the first values ​​and the second values ​​will be described by taking two of them as examples.

[0106] In some embodiments, there are multiple first and second values, and there is a one-to-one correspondence between the first and second values; the process of comparing the first and second values ​​may be, for each first value, determining the second difference between the first value and the corresponding second value; and the process of determining whether the first value in the chart is a true value based on the comparison result may be, if there is a comparison result representing that the corresponding second difference is greater than or equal to a third difference threshold, then determining that the first value is not a true value; if each comparison result represents that the corresponding second difference is less than the third difference threshold, then determining that the first value is a true value; wherein, as described above, the third difference threshold may be pre-set, and this is not limited in the embodiments of the present application.

[0107] In some embodiments, there are multiple first and second values. Thus, the process of comparing the first and second values ​​may be to sum the multiple first values ​​to obtain a first summation result, and to sum the multiple second values ​​to obtain a second summation result; to subtract the first summation result from the second summation result to obtain a third difference, and to determine the third difference as the comparison result between the first and second values; thus, the process of determining whether the first value in the chart is a true value based on the comparison result may be to determine that the first value is not a true value if the third difference is greater than or equal to a fourth difference threshold; and to determine that the first value is a true value if the third difference is less than the fourth difference threshold. The fourth difference threshold may be pre-set, and this is not limited in the embodiments of the present application.

[0108] In this way, when there are multiple first values ​​and multiple second values, a third difference is directly determined based on the sum of the first values ​​and the sum of the second values, and then based on this third difference, it is determined whether the first value is a true value. Compared with the solution of determining multiple differences based on multiple first values ​​and multiple second values ​​and then determining whether the first value is a true value based on multiple differences, this simplifies the image processing process and improves image processing efficiency.

[0109] In actual implementation, the image processing method provided in the embodiment of the present application can also be realized through a model. Next, the implementation process of the image processing method provided in the embodiment of the present application is explained based on models with different structures.

[0110] In some embodiments, see Figure 7 , Figure 7 This is a structural diagram of the image processing system provided in the embodiment of the present application, based on Figure 7 The image processing system includes a numerical recognition model, a numerical prediction model, and a comparison module. The numerical recognition model includes a coding layer, a decoding layer, and a classification layer, while the numerical prediction model also includes a coding layer, a decoding layer, and a classification layer. The numerical recognition model and the numerical prediction model can be as follows: Figure 7 As shown, they are all located in the image processing system, but they can also be located in different systems, or even dispersed in different systems. This embodiment of the present application does not limit this.

[0111] In actual implementation, the process of recognizing an image including a chart and obtaining a first numerical value associated with a graphic element in the chart may include inputting the image including the chart into an encoding layer of a numerical recognition model to obtain image features of the image; inputting the image features into a decoding layer of the numerical recognition model to obtain text features of a text corresponding to the first numerical value in the image; and inputting the text features into a classification layer of the numerical recognition model to obtain the first numerical value associated with the graphic element in the chart.

[0112] The process of performing numerical prediction on graphic elements based on an image to obtain a second numerical value corresponding to the graphic elements may be: inputting the image including the chart into the encoding layer of the numerical prediction model to obtain image features of the image; inputting the image features into the decoding layer of the numerical prediction model to obtain image features of the graphic elements in the image; and inputting the image features of the graphic elements into the classification layer of the numerical prediction model to obtain the second numerical value corresponding to the graphic elements.

[0113] It should be noted that the process of obtaining the first numerical value based on the numerical recognition model and the process of obtaining the second numerical value based on the numerical prediction model are parallel processing processes; and the process of inputting the image including the chart into the coding layer of the numerical recognition model to obtain the image features of the image is similar to the process of inputting the image including the chart into the coding layer of the numerical prediction model to obtain the image features of the image. Here, the process of obtaining the image features of the image is as described above and will not be repeated in the embodiments of the present application.

[0114] As for the process of inputting image features into the decoding layer of the numerical extraction model to obtain the text features of the text corresponding to the first numerical value in the image, it is similar to the process of determining the text features of the text corresponding to the first numerical value in the image based on the image features as described above, and this will not be elaborated on; and the process of inputting image features into the decoding layer of the numerical prediction model to obtain the image features of the graphic elements in the image is similar to the process of determining the image features corresponding to the graphic elements in the image based on the image features as described above, and this will also not be elaborated on.

[0115] As for the process of inputting text features into the classification layer of the numerical recognition model to obtain the first numerical value associated with the graphic element in the chart, it is similar to the process of predicting the first numerical value associated with the graphic element in the chart based on the text features as described above, and this will not be elaborated on; and the process of inputting the image features of the graphic element into the classification layer of the numerical prediction model to obtain the second numerical value corresponding to the graphic element is similar to the process of predicting the second numerical value corresponding to the graphic element based on the image features of the graphic element as described above, and this will also not be elaborated on.

[0116] In actual implementation, after obtaining the first value through the numerical recognition model and the second value through the numerical prediction model, the first value and the second value are compared through the comparison module in the image processing system, so as to determine whether the first value in the chart is a true value based on the comparison result.

[0117] In some embodiments, see Figure 8 , Figure 8 This is a schematic diagram of the structure of the image processing model provided in the embodiment of the present application, based on Figure 8 , the image processing model includes a coding layer, a first decoding layer, a second decoding layer, a first classification layer and a second classification layer.

[0118] In actual implementation, before identifying the image including the chart and obtaining the first numerical value associated with the graphic element in the chart, the image including the chart is input into the encoding layer of the image processing model to obtain the image features of the image; then, the process of identifying the image including the chart and obtaining the first numerical value associated with the graphic element in the chart can be inputting the image features into the first decoding layer of the image processing model to obtain the text features of the text corresponding to the first numerical value in the image; inputting the text features into the first classification layer of the image processing model to obtain the first numerical value associated with the graphic element in the chart; and the process of numerically predicting the graphic element based on the image to obtain the second numerical value corresponding to the graphic element can be inputting the image features into the second decoding layer of the image processing model to obtain the image features of the graphic element; and inputting the image features of the graphic element into the second classification layer of the image processing model to obtain the second numerical value corresponding to the graphic element.

[0119] It should be noted that there is only one encoding layer in the image processing model, and the first decoding layer is different from the second decoding layer. The process of inputting the image features into the first decoding layer of the image processing model to obtain the text features of the text corresponding to the first value in the image, and the process of inputting the image features into the second decoding layer of the image processing model to perform data prediction on the graphics in the image to obtain the second data of the graphics can be processed in parallel; wherein the first encoding layer and the second encoding layer are two text decoders; accordingly, the process of inputting the text features into the first classification layer of the image processing model to obtain the first value associated with the graphic elements in the chart, and the process of inputting the image features of the graphic elements into the second classification layer of the image processing model to obtain the second value corresponding to the graphic elements can also be processed in parallel.

[0120] It should be noted that after the first value and the second value are obtained through the image processing model, the first value and the second value are compared, so as to determine whether the first value in the chart is a true value based on the comparison result.

[0121] In actual implementation, before the image processing method provided in the embodiment of the present application is realized through a model, the model needs to be trained. Next, the training process of different models is explained respectively.

[0122] In some embodiments, for Figure 7 The training process of the numerical recognition model and the numerical prediction model shown in FIG. Figure 9 , Figure 9 This is a flow chart of the training process of the numerical recognition model and the numerical prediction model provided in the embodiment of the present application, based on Figure 9 , Figure 7 The training process of the numerical recognition model and the numerical prediction model shown is achieved through the following steps.

[0123] In step 301, the server obtains a sample image including a sample chart; wherein the sample image includes a first label and a second label, the first label is used to indicate a non-real value associated with a chart element in the sample chart, and the second label is used to indicate a real value corresponding to the chart element in the sample chart.

[0124] It should be noted that the sample image is obtained by tampering with the original image including the chart. Specifically, the numerical values ​​associated with the chart elements in the chart of the original image are real numerical values, and then these numerical values ​​are tampered with to obtain non-real numerical values, so that the tampered original image is used as the sample image, that is, the first label is the numerical value associated with the tampered chart element;

[0125] For example, see Figure 4 , if Figure 4The image shown is used as the original image. The original image is annotated. Taking the first bar graph as an example, the first bar graph is annotated to obtain the second label, which is the real data 7.2 corresponding to the sample graph, where 7.2 represents the real value of the first bar graph (the height represents the value. When generating a chart, the height of the graph corresponds to the value on the coordinate axis).

[0126] The image is then tampered with, that is, some of the values ​​are modified, and the tampered values ​​are also used as labels (first labels). For example, the 7.2 in the first bar graph is tampered with to 17.2. The tampered 17.2 is not equal to the true value 7.2 indicated by the graph (the value represented by the height of the bar graph), that is, two labels are obtained, the first label 17.2 and the second label 7.2.

[0127] Step 302: Input the sample image into the coding layer of the numerical recognition model to obtain image features of the sample image, and input the sample image into the coding layer of the numerical prediction model to obtain image features of the sample image.

[0128] In step 303, the image features of the sample image are input into the decoding layer of the numerical recognition model to obtain the first sample features associated with the graphic elements in the sample chart, and the image features of the sample image are input into the decoding layer of the numerical prediction model to obtain the second sample features corresponding to the graphic elements.

[0129] It should be noted that the first sample feature here is the text feature of the text corresponding to the first sample value, and the second sample feature is the image feature corresponding to the graphic element.

[0130] In step 304, the first sample feature is input into the classification layer of the numerical recognition model to obtain the first sample value associated with the graphic element in the sample chart, and the second sample feature is input into the classification layer of the numerical prediction model to obtain the second sample value corresponding to the graphic element.

[0131] Step 305 : Determine the loss corresponding to the numerical recognition model based on the first numerical value and the first label of the sample, and determine the loss corresponding to the numerical prediction model based on the second numerical value and the second label of the sample.

[0132] Among them, the process of determining the loss corresponding to the numerical recognition model based on the first numerical value and the first label of the sample can be to take the difference between the first numerical value of the sample and the first label to obtain the loss corresponding to the numerical recognition model; and the process of determining the loss corresponding to the numerical prediction model based on the second numerical value and the second label of the sample can be to take the difference between the second numerical value of the sample and the second label to obtain the loss corresponding to the numerical prediction model.

[0133] Step 306 : Based on the loss corresponding to the numerical recognition model, the numerical recognition model is trained to obtain a trained numerical recognition model; and based on the loss corresponding to the numerical prediction model, the numerical prediction model is trained to obtain a trained numerical prediction model.

[0134] It should be noted that the trained numerical prediction model here is the model mentioned above for performing numerical prediction based on sub-images obtained by image segmentation.

[0135] In other embodiments, for Figure 8 The training process of the image processing model shown in Figure 10 , Figure 10 This is a flow chart of the training process of the image processing model provided in the embodiment of the present application, based on Figure 10 , Figure 8 The training process of the image processing model shown is achieved through the following steps.

[0136] In step 401, the server obtains a sample image including a sample chart; wherein the sample image includes a first label and a second label, the first label is used to indicate a non-real value associated with a chart element in the sample chart, and the second label is used to indicate a real value corresponding to the chart element in the sample chart.

[0137] It should be noted that, as mentioned above, the sample image is obtained by tampering with the original image including the chart. Specifically, the numerical value associated with the chart element in the chart of the original image is a real numerical value, and then the numerical value is tampered with to obtain a non-real numerical value, so that the tampered original image is used as the sample image, that is, the first label is the numerical value associated with the tampered icon element.

[0138] Step 402: Input the sample image into the encoding layer of the image processing model to obtain image features of the sample image.

[0139] Step 403: input the image features to the first decoding layer to obtain first sample features associated with graphic elements in the sample chart, and input the image features to the second decoding layer to obtain second sample features corresponding to the graphic elements.

[0140] It should be noted that the first sample feature here is the text feature of the text corresponding to the first sample value, and the second sample feature is the image feature corresponding to the graphic element.

[0141] Step 404: input the first sample feature into the first classification layer to obtain the sample first value associated with the graphic element in the sample chart, and input the second sample feature into the second classification layer to obtain the sample second value corresponding to the graphic element.

[0142] Step 405 : Determine the total loss based on the first sample value, the second sample value, the first label, and the second label, and train the image processing model based on the total loss to obtain a trained image processing model.

[0143] In actual implementation, the process of determining the total loss based on the first value of the sample, the second value of the sample, the first label and the second label can be: determining the first loss based on the first value of the sample and the first label; determining the second loss based on the second value of the sample and the second label; summing the first loss and the second loss to obtain the total loss; wherein, after obtaining the total loss, backpropagation is performed and the model is iterated until the model converges.

[0144] In some embodiments, regardless of which model is used to implement the image processing method provided in the embodiments of the present application, in the process of obtaining a sample image including a sample chart, data enhancement may be performed on the initial sample image including the sample chart to obtain a sample image including the sample chart. In this way, by performing data enhancement on the sample image, the generalization of the model used in subsequent image processing can be improved.

[0145] In actual implementation, the process of performing data enhancement on the initial sample image including the sample chart to obtain the sample image including the sample chart may be: blurring the initial sample image to obtain a blurred sample image, and adding noise to the blurred sample image to obtain the sample image; or performing channel analysis on the initial sample image to obtain multiple color channels included in the initial sample image, and adjusting the arrangement order of the multiple color channels included in the initial sample image to obtain the sample image.

[0146] It should be noted that, in the process of blurring the initial sample image to obtain a blurred sample image, and adding noise to the blurred sample image to obtain a sample image, the initial non-real image is blurred, that is, the initial sample image is Gaussian blurred to obtain a model sample image, and then Gaussian noise is added to the blurred sample image to obtain a sample image.

[0147] It should be noted that, channel analysis is performed on the initial sample image to obtain multiple color channels included in the initial sample image, such as RGB channels, so that the arrangement order of the multiple color channels included in the initial sample image is adjusted to obtain the sample image. The process can be to select a color channel from the multiple color channels, and then mask the information of other channels except the selected color channel, and only display the information of the selected color channel, that is, the image will be displayed as the color information of the selected channel, and then save the current image to obtain the sample image; or adjust the arrangement order of the multiple color channels included in the initial sample image from RGB to GBR or BRG, etc. to obtain the sample image.

[0148] By applying the above-mentioned embodiments of the present application, for an image including a chart, while the image is identified to obtain a first numerical value associated with a graphic element in the chart, a numerical prediction is also performed on the graphic element to obtain a second numerical value corresponding to the graphic element, thereby determining whether the first numerical value in the chart is a true numerical value by comparing the first numerical value with the second numerical value; thus, compared to the solution in the related art of only identifying the chart in the image to obtain the numerical value of the chart, the present application determines whether the numerical value of the chart is a true numerical value by comparing two numerical values ​​obtained based on different methods, thereby determining whether the numerical value in the chart has been tampered with.

[0149] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0150] When processing data including charts, related technologies mostly focus on how to identify chart data and improve the accuracy of chart data identification, but do not consider the problem that chart data may be tampered with.

[0151] Based on this, an embodiment of the present application provides an image processing method. For an image including a chart, while identifying the image to obtain a first numerical value associated with a graphic element in the chart, the graphic element is also numerically predicted to obtain a second numerical value corresponding to the graphic element, thereby determining whether the first numerical value in the chart is a true numerical value by comparing the first numerical value with the second numerical value; thus, compared with the solution in the related art of only identifying the chart in the image to obtain the numerical value of the chart, the present application determines whether the numerical value of the chart is a true numerical value by comparing two numerical values ​​obtained based on different methods, thereby determining whether the numerical value in the chart has been tampered with.

[0152] In actual implementation, the image processing method provided in the embodiment of the present application is implemented through an image processing model. Specifically, see Figure 11 , Figure 11 This is a technical architecture diagram of an image processing method implemented by an image processing model according to an embodiment of the present application. Figure 11 When the image processing method provided in the embodiment of the present application is implemented through an image processing model, it includes two parts: the first part is the training process, and the second part is the reasoning process.

[0153] For the training process, specifically, a chart image is obtained, and then the chart image is annotated to obtain the true value of the chart image (the second label), for example, continue to refer to Figure 4 , if Figure 4The image shown is used as a chart image. Taking the first bar graph as an example, the annotation value obtained by annotating the bar graph is 2014|7.2, where 7.2 represents the true value of the first bar graph (the value represented by the height, and the height of the graph corresponds to the value on the coordinate axis when the chart is generated); then the image is tampered with, that is, some values ​​are modified, and then the tampered values ​​are added to the annotation file; for example, continuing with the above example, the annotation value 7.2 of the first bar graph is tampered with to 17.2, and the new annotation value becomes 2014|7.2|17.2; wherein the tampered 17.2 is not equal to the 7.2 of the graph (the value represented by the height of the bar graph), that is, two labels are obtained, the first label 17.2 and the second label 7.2.

[0154] Then, data enhancement is performed on the chart image, for example, by applying Gaussian blur and probabilistic Gaussian noise, or randomly switching RGB channels, to obtain an enhanced chart image. The enhanced chart image is then passed through an encoder to obtain an image code; the image code is then passed through two text decoders to obtain two prediction results, namely the first prediction result (the first value of the sample) and the second prediction result (the second value of the sample). The loss is then calculated based on the first prediction result and the first label to obtain the first loss, and the loss is calculated based on the second prediction result and the second label to obtain the second loss; finally, the first loss and the second loss are added together to obtain the final loss (total loss), and the model is then back-propagated and iterated until the model converges.

[0155] For the inference process, after obtaining the image to be detected (including the image of the chart), the image to be detected passes through the image encoder and two text decoders to obtain two model prediction results (a first value and a second value); then the values ​​included in the two model prediction results are compared. If the difference in values ​​is less than 10% (this value is pre-set), it means that the image to be detected has not been tampered with. Otherwise, it means that the image to be detected has been tampered with. Continuing with the above example, taking the first bar graph as an example, if the two model prediction results are 7.2 and 17.2 respectively, the difference between the two values ​​is (17.2-7.2) / 7.2=1.38, which is much greater than 10%. In this way, it means that the current value has been tampered with.

[0156] In this way, the present application outputs text values ​​and values ​​represented by graphics, and then determines whether the chart values ​​have been tampered with by the difference between the two values, thereby being able to accurately determine the tampered values ​​and improving the traceability of image processing results.

[0157] By applying the above-mentioned embodiments of the present application, for an image including a chart, while the image is identified to obtain a first numerical value associated with a graphic element in the chart, a numerical prediction is also performed on the graphic element to obtain a second numerical value corresponding to the graphic element, thereby determining whether the first numerical value in the chart is a true numerical value by comparing the first numerical value with the second numerical value; thus, compared to the solution in the related art of only identifying the chart in the image to obtain the numerical value of the chart, the present application determines whether the numerical value of the chart is a true numerical value by comparing two numerical values ​​obtained based on different methods, thereby determining whether the numerical value in the chart has been tampered with.

[0158] The following continues to describe the exemplary structure of the image processing device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the image processing device 455 of the memory 450 may include:

[0159] an identification module 4551 for identifying an image including a chart, and obtaining a first numerical value associated with a graphic element in the chart, wherein the graphic element is used to display the first numerical value in a graphical manner;

[0160] A prediction module 4552 is configured to perform a numerical prediction on the graphic element based on the image to obtain a second numerical value corresponding to the graphic element;

[0161] The comparison module 4553 is configured to compare the first value and the second value, and determine whether the first value in the chart is a true value based on the comparison result.

[0162] In some embodiments, the first numerical value in the image is presented in the form of text, and the recognition module 4551 is further used to encode the image to obtain image features of the image; based on the image features of the image, determine the text features of the text corresponding to the first numerical value in the image; based on the text features, predict the first numerical value associated with the graphic element in the chart.

[0163] In some embodiments, the prediction module 4552 is further used to predict the i-th character based on the text features and a character sequence including i-1 characters, where i is an integer greater than 0; and determine the first numerical value associated with the graphic element in the chart based on the character sequence composed of the predicted i characters.

[0164] In some embodiments, the prediction module 4552 is further used to perform image segmentation on the image to obtain a sub-image corresponding to each of the graphic elements in the image; for each sub-image corresponding to the graphic element, the sub-image is input into a model for numerical prediction of the graphic element to obtain a second numerical value corresponding to the graphic element.

[0165] In some embodiments, the device also includes a verification module, which is used to determine a third value of the geometric parameter of the graphic element and determine the ratio of the second value to the corresponding third value; determine the scale of the chart, and based on the scale of the chart and the ratio, verify the second value to obtain a verification result; the comparison module 4553 is also used to compare the first value and the second value when the verification result indicates that the verification of the second value has passed.

[0166] In some embodiments, the verification module is further used to cluster the ratios corresponding to multiple graphic elements to obtain clusters when there are multiple graphic elements; select clusters from the clusters whose number of ratios is greater than a quantity threshold; determine the average value of the ratios in the selected clusters, and use the average value as the scale of the chart.

[0167] In some embodiments, the verification result includes a first verification result and a second verification result, and the verification module is further used to determine the difference between the scale and the ratio; if the difference is greater than or equal to a first difference threshold, the verification result is determined to be the first verification result, and the first verification result is used to indicate that the verification of the second value fails; if the difference is less than the first difference threshold, the verification result is determined to be the second verification result, and the second verification result is used to indicate that the verification of the second value passes; if the verification result is the first verification result, the second value is corrected based on the scale of the chart.

[0168] In some embodiments, the comparison module 4553 is further used to determine that the first value in the chart is not a true value if the comparison result indicates that the difference between the first value and the second value is greater than or equal to a second difference threshold; if the comparison result indicates that the difference between the first value and the second value is less than the second difference threshold, determine that the first value in the chart is the true value.

[0169] The embodiment of the present application provides a computer program product, which includes computer-executable instructions or a computer program, which is stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions or the computer program from the computer-readable storage medium, and the processor executes the computer-executable instructions or the computer program, so that the electronic device performs the above-mentioned image processing method of the embodiment of the present application, for example, Figure 3 The image processing method is shown.

[0170] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the image processing method provided by the embodiment of the present application, for example, Figure 3 The image processing method is shown.

[0171] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0172] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0173] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0174] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0175] It should be noted that in the embodiments of the present application, when it comes to obtaining relevant data such as attribute information, category labels, behavioral events, and behavioral event sequences of objects, when the embodiments of the present application are applied to specific products or technologies, user permission or consent must be obtained, and the collection, use, and processing of relevant data must comply with relevant laws, regulations, and standards of relevant countries and regions.

[0176] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Recognizing an image including a chart to obtain a first numerical value associated with a graphic element in the chart, wherein the graphic element is used to graphically display the first numerical value; Based on the image, performing a numerical prediction on the graphic element to obtain a second numerical value corresponding to the graphic element; The first value and the second value are compared, and based on the comparison result, it is determined whether the first value in the chart is a true value.

2. The method according to claim 1, wherein The first numerical value in the image is presented in the form of text, and the identifying the image including the chart to obtain the first numerical value associated with the graphic element in the chart includes: Encoding the image to obtain image features of the image; determining, based on image features of the image, text features of the text corresponding to the first value in the image; Based on the text feature, a first numerical value associated with the graphic element in the chart is predicted.

3. The method according to claim 2, wherein The predicting, based on the text feature, a first numerical value associated with a graphic element in the chart includes: Predicting an i-th character based on the text feature and a character sequence comprising i-1 characters, where i is an integer greater than 0; Based on the predicted character sequence consisting of the i characters, a first numerical value associated with the graphic element in the chart is determined.

4. The method according to claim 1, wherein The performing numerical prediction on the graphic element based on the image to obtain a second numerical value corresponding to the graphic element includes: Performing image segmentation on the image to obtain a sub-image corresponding to each graphic element in the image; For each sub-image corresponding to the graphic element, the sub-image is input into a model for numerical prediction of the graphic element to obtain a second numerical value corresponding to the graphic element.

5. The method according to claim 1, wherein After performing numerical prediction on the graphic element based on the image to obtain a second numerical value corresponding to the graphic element, the method further includes: determining a third value of a geometric parameter of the graphical element, and determining a ratio of the second value to the corresponding third value; determining a scale of the chart, and verifying the second value based on the scale of the chart and the ratio to obtain a verification result; The comparing the first value and the second value includes: When the verification result indicates that the second value has passed the verification, the first value and the second value are compared.

6. The method according to claim 5, wherein Determining the scale of the chart includes: In the case where there are multiple graphic elements, clustering the ratios corresponding to the multiple graphic elements to obtain clusters; Selecting, from the clusters, clusters whose number of ratios is greater than a number threshold; The average value of the ratios in the selected clusters is determined, and the average value is used as the scale of the graph.

7. The method according to claim 5, wherein The verification result includes a first verification result and a second verification result, and the verification of the second value based on the scale of the chart and the ratio to obtain the verification result includes: determining a difference between the scale and the ratio; If the difference is greater than or equal to a first difference threshold, determining the verification result to be the first verification result, where the first verification result is used to indicate that the verification of the second value fails; If the difference is less than the first difference threshold, determining the verification result as the second verification result, where the second verification result is used to indicate that the verification of the second value has passed; After verifying the second value based on the scale of the chart and the ratio, the method further includes: If the verification result is the first verification result, the second value is corrected based on the scale of the chart.

8. An electronic device, characterized in that: include: Memory for storing computer-executable instructions or computer programs; The processor is configured to implement the image processing method according to any one of claims 1 to 7 when executing the computer-executable instructions or computer program stored in the memory.

9. A computer-readable storage medium, characterized in that Computer executable instructions or computer programs are stored, which are used to cause a processor to execute and implement the image processing method according to any one of claims 1 to 7.

10. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the image processing method according to any one of claims 1 to 7 is implemented.

Images