Information processing apparatus, information processing method, and program

Abstract

To detect an altered part of a voucher.SOLUTION: A storage unit 12 that stores a trained model trained to classify each pixel constituting input image data based on the presence or absence of alteration when image data is input, based on training data in which learning image data, which is image data obtained by capturing an image of a voucher and is image data for learning, and information indicating a pixel altered from original image data in the learning image data are associated with each other, an acquisition unit 131 that acquires target image data which is image data to be determined, and a processor that inputs the target image data to the trained model; And an output unit 133 that outputs a candidate portion for modification in the target image data based on the classification result output by the learned model.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] A technology has been proposed that uses a trained model to determine whether an image of a document has been altered (see, for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Beatriz Martinez Tornes, et al. Receipt Dataset for Document Forgery Detection, ICDAR, pp. 454-469 (2023). Summary of the Invention [Problem to be solved by the invention]

[0004] In the prior art, it is possible to determine whether a document has been altered, but it is not possible to detect which part of the document has been specifically altered.

[0005] The present invention has been made in consideration of these points, and aims to make it possible to detect alterations in evidence. [Means for solving the problem]

[0006] The information processing device of the first aspect of the present invention has a memory unit that stores a trained model that has been trained to classify each pixel that makes up the input image data based on whether or not it has been altered, when image data is input, based on training data that associates image data obtained by capturing an image of a document, the image data being image data for learning, with information indicating pixels in the training image data that have been altered from the original image data; an acquisition unit that acquires target image data that is image data to be judged; and an output unit that inputs the target image data to the trained model and outputs candidate locations of alteration in the target image data based on the classification results output by the trained model.

[0007] The training data associates the training image data with information indicating pixels in the training image data that have been modified from the original image data and the type of modification for those pixels, and the trained model is trained to classify each pixel that makes up the image data based on the training image data, based on whether or not it has been modified and the type of modification, and the output unit may output candidate locations for modification in the target image data and the type of modification at those candidate locations.

[0008] The trained model may be trained to classify, when the training image data is input, each pixel constituting the image data into either (1) a pixel with no alteration, or (2) in terms of the type of alteration, a pixel that has been altered by replacing part of an area constituting the input image data with another area, or a pixel that has been altered in a different way.

[0009] The output unit may cause the display unit to display a screen for displaying candidate portions of the target image data for modification in a manner different from other portions.

[0010] The information processing method of the second aspect of the present invention includes an acquisition step executed by a computer to acquire target image data, which is image data to be judged; and a step of inputting the target image data into a trained model that has been trained to classify each pixel that makes up the input image data based on whether or not it has been altered, based on training data that associates image data obtained by capturing an image of a document, which is stored in a memory unit, with information indicating pixels that have been altered from the original image data in the training image data, and outputting candidate locations of alteration in the target image data based on the classification results output by the trained model.

[0011] In a third aspect of the program of the present invention, a computer is caused to execute an acquisition step of acquiring target image data, which is image data to be judged; and a step of inputting the target image data into a trained model that has been trained to classify each pixel constituting the input image data based on whether or not it has been altered, based on training data that associates image data obtained by capturing an image of a document, which is stored in a memory unit, with information indicating pixels in the training image data that have been altered from the original image data, and outputting candidate locations of alteration in the target image data based on the classification results output by the trained model. [Effects of the Invention]

[0012] According to the present invention, it is possible to detect alterations in a voucher. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a diagram for explaining an overview of an information processing system S according to an embodiment. [Figure 2] FIG. 1 is a diagram illustrating an outline of training data used to train a machine learning model. [Figure 3] 1 is a block diagram showing a configuration of an information processing device 1. FIG. [Figure 4]FIG. 10 is a diagram showing an example of a screen displayed by an output unit 133. [Figure 5] FIG. 1 is a diagram illustrating an outline of training data used to train a machine learning model. [Figure 6] FIG. 10 is a diagram for explaining annotation information D3. [Figure 7] FIG. 10 is a diagram for explaining annotation information D3. [Figure 8] 3 is a flowchart showing the flow of processing in the information processing device 1. DETAILED DESCRIPTION OF THE INVENTION

[0014] [Overview of information processing device 1] 1 is a diagram illustrating an overview of an information processing system S according to an embodiment. The information processing system S is a system for detecting alterations in a voucher. The information processing system S includes an information processing device 1 and an information terminal 2.

[0015] The information processing device 1 is an accounting processing device for processing supporting documents. The information processing device 1 determines whether the acquired supporting documents have been altered. The information processing device 1 stores a trained model for determining whether the documents have been altered. As will be described in detail later, the trained model is trained to classify each pixel that makes up the input image data based on whether it has been altered, when image data is input. As an example, the trained model classifies each pixel that makes up the input image data using known semantic segmentation. The trained model may output a degree of certainty that the supporting documents indicated by the input image data have been altered.

[0016] The information terminal 2 is a terminal used by a user of the information processing system S. The information terminal 2 transmits image data of the voucher to the information processing device 1, acquires the judgment result of the information processing device 1, and displays the acquired judgment result on a display unit.

[0017] An overview of the processing in the information processing system S will be described. The information processing device 1 acquires training data ((1) in FIG. 1). As an example, the information processing device 1 acquires training data from the information terminal 2.

[0018] FIG. 2 is a diagram illustrating an overview of training data used by a machine learning model. Training image data D1 and annotation information D2 are associated with each other in the training data. The training image data is image data obtained by capturing images of supporting documents, and serves as training image data. Examples of supporting documents processed by the machine learning model include, but are not limited to, invoices, receipts, sales slips, or delivery notes. The annotation information D2 is information indicating pixels in the training image data that have been altered from the original image data. In FIG. 2, altered pixels in the training image data are indicated by hatching (symbol A) to represent a correspondence between their positions in the annotation information D2 and their positions in the training image data D2. However, as an example, the annotation information D2 is binary information indicating whether each pixel constituting the image data has been altered. As described below, the annotation information D2 may also include information indicating the type of alteration.

[0019] The information processing device 1 trains a machine learning model based on the training data ((2) in FIG. 1). The training of the machine learning model will be described later. Note that, hereinafter, a machine learning model that has completed training will be referred to as a "trained model."

[0020] The information processing device 1 acquires target image data from the information terminal 2 ((3) in FIG. 1). The target image data is image data generated by capturing an image of a document to be determined for alteration. The information processing device 1 inputs the target image data into a trained model and outputs a classification result ((4) in FIG. 1). The classification result includes, as an example, information indicating whether or not each pixel constituting the target image data has been altered. The classification result may further be associated with an inferred certainty that each pixel has been altered.

[0021] Based on the classification result, the information processing device 1 identifies potential alteration locations that are inferred to have been altered in the image data, and outputs the identified potential alteration locations to the information terminal 2 ((5) in FIG. 1).

[0022] The information processing system S is configured to classify the pixels that make up an image based on a trained model that has been trained to classify whether or not each pixel that makes up image data has been altered, making it possible to detect any alterations in the supporting document.

[0023] [Configuration of information processing device 1] 3 is a block diagram showing the configuration of the information processing device 1. The information processing device 1 has a communication unit 11, a storage unit 12, and a control unit 13. The control unit 13 has an acquisition unit 131, a learning unit 132, and an output unit 133.

[0024] The communication unit 11 is a communication interface for transmitting and receiving data to and from other devices via a network. The storage unit 12 is a storage medium including a read-only memory (ROM), a random access memory (RAM), a solid-state drive (SSD), a hard disk drive, etc. The storage unit 12 stores in advance a program to be executed by the control unit 13. The storage unit 12 stores a trained model.

[0025] The control unit 13 is a processor such as a CPU (Central Processing Unit), etc. The control unit 13 executes a program stored in the storage unit 12, thereby functioning as an acquisition unit 131, a learning unit 132, and an output unit 133.

[0026] [Model Training] The acquisition unit 131 acquires training data. The learning unit 132 inputs training image data D1 associated in the acquired training data into a model and outputs a classification result in which each pixel constituting the training image data is classified into a tampered pixel or an untampered pixel. The learning unit 132 updates the parameters of the machine learning model based on the difference between the output classification result and the annotation information associated in the training data. The learning unit 132 updates the parameters of the machine learning model based on the training data until a predetermined termination condition is met. The information processing device 1 stores the generated trained model in the storage unit 12.

[0027] [Detection of Modifications] The acquisition unit 131 acquires target image data. The target image data is image data obtained by capturing or scanning the document to be subjected to the determination of whether or not it has been altered. The acquisition unit 131 acquires the target image data from the information terminal 2.

[0028] The output unit 133 inputs the target image data into the trained model and causes the trained model to output a classification result. As an example, the classification result is information obtained by classifying each pixel constituting the target image data based on whether it has been altered. In this case, the trained model outputs a classification result indicating whether each pixel constituting the target image data has been tampered with. The output unit 133 outputs candidate locations of alteration in the target image data based on the classification result output by the trained model.

[0029] As an example, the output unit 133 displays a screen showing candidate alteration locations on the information terminal 2. FIG. 4 is a diagram showing an example of a screen displayed by the output unit 133 on the information terminal 2. As shown in FIG. 4, the output unit 133 may display a screen on the display unit of the information terminal 2 for displaying candidate alteration locations in the target image data in a manner different from other locations. In the screen shown in FIG. 4, the output unit 133 highlights the candidate alteration locations by surrounding them with a frame. Note that the screen displayed by the output unit 133 may be configured to display candidate alteration locations in a color different from that of locations that are inferred to be unaltered. To make it easier for a user verifying the evidence to identify the candidate alteration locations, the screen displayed by the output unit 133 may be configured to provide an area (R2) for enlarging and displaying an area (R1) designated by the user.

[0030] The output unit 133 may display the degree of certainty (C) that each candidate alteration portion has been altered in association with the candidate alteration portion. As an example, the output unit 133 calculates the average or maximum value of the certainty values ​​of pixels classified into a class indicating that adjacent pixels have been altered as the degree of certainty of the candidate alteration portion to which the pixel belongs. The output unit 133 may also output the degree of certainty that the evidence indicated by the target image data has been altered.

[0031] If the classification result indicates that a pixel has been classified as having been altered, the screen displayed by the output unit 133 may be configured to display a message indicating that alteration is suspected. The screen displayed by the output unit 133 may also be configured to display the probability (certainty) that each candidate alteration location is inferred to have been altered. The screen may also be configured to display the certainty of the candidate alteration location when the mouse pointer is placed on the candidate alteration location.

[0032] In addition, if there are no pixels classified as altered in the classification result, the output unit 133 may notify the information terminal 2 of information indicating that there is no alteration, or may obtain the information and perform subsequent processing (e.g., accounting processing, payment processing, etc.) based on the target image data.

[0033] By configuring the information processing device 1 in this way, it is possible to detect any alterations made to the evidence.

[0034] The information processing device 1 may be configured to cause the trained model to learn training data that further includes the type of alteration made to the evidence, and to output the type of alteration made to the evidence.

[0035] In this case, the training data is associated with training image data D1 and annotation information D3, which indicates pixels in the training image data that have been altered from the original image data and the type of alteration for those pixels. Figure 5 is a diagram showing an overview of the training data used by the machine learning model in this case. In this case, annotation information D3 indicates whether or not each pixel in the image has been altered and the type of alteration. In Figure 5, annotation information D3 is represented two-dimensionally. As an example, annotation information D3 is information configured to take integer values ​​indicating the presence and type of alteration for each pixel. In Figure 5, the area indicated by A1 indicates an altered area where text has been corrected using image editing software, and the area indicated by A2 indicates an altered area where text has been copied from another document. Note that the other areas in Figure 5 indicate that no alterations have been made.

[0036] Annotation information D3 will be described using Figure 6. Figure 6 is a diagram for explaining an example of annotation information. As an example, annotation information D3 has a value of "0" if there is no alteration, a value of "2" if the alteration was made by copying other characters from the same document, and a value of "5" if the alteration was made by correcting the characters using image editing software.

[0037] The learning unit 132 trains a machine learning model based on the acquired training data and stores the generated trained model in the storage unit 12. In this case, the trained model is trained to classify each pixel constituting image data based on the presence or absence of alteration and the type of alteration, based on the training image data. More specifically, when image data is input, the trained model is trained to output a value indicating the presence or absence of alteration and the type of alteration for each pixel constituting the image data. Note that the values ​​indicating the presence or absence of alteration and the type of alteration output by the trained model correspond to values ​​in the annotation information.

[0038] The output unit 133 outputs candidate locations of alterations in the target image data and the types of alterations at the candidate locations. As an example, the output unit 133 causes the information terminal 2 to display a screen that displays candidate locations in different display modes for each type of alteration based on the classification results output by the trained model.

[0039] By configuring the information processing device 1 to output the type of alteration, the person in charge of checking the supporting document can recognize what alterations have been made, thereby reducing the burden on the person in charge of checking the supporting document.

[0040] By optimizing the classification of alteration types, it is expected that the accuracy of detecting alterations in supporting documents will be improved. For example, if the trained model adopts a Transformer-type structure, it is expected to be effective in determining whether supporting documents have been altered based on the characteristics of pixels located far apart in the image data, since it has an attention mechanism.

[0041] Therefore, as shown in Figure 7, the types of alterations indicated by the associated annotation information in the training data may be classified into "portions altered by copying other characters from the same document" and "portions altered other than those mentioned above." The reason for this classification is that it is expected that a trained model will be able to detect alterations in "portions altered by copying other characters from the same document" based on the correlation with the features of other portions.

[0042] In other words, when training image data is input, the trained model in this case is trained to classify each pixel that makes up the image data into one of the following: (1) unaltered pixels, (2) altered pixels that have been altered by replacing part of the area that makes up the input image data with another area, and (3) altered pixels that have been altered in some other way.

[0043] By configuring the information processing device 1 in this way, it is possible to improve the accuracy of detecting whether or not alteration has occurred.

[0044] [Processing flow in information processing device 1] Fig. 8 is a flowchart showing the flow of processing in the information processing device 1. The flowchart shown in Fig. 8 starts from the point when a trained model is generated and preparations are made to acquire image data to be determined.

[0045] The acquisition unit 131 acquires target image data (S01). The output unit 133 inputs the target image data to a trained model and outputs a classification result (S02).

[0046] The output unit 133 determines whether or not there are any pixels classified as having been altered in the classification result (S03). If there are any pixels classified as having been altered (YES in S03), the output unit 133 causes the information terminal 2 to display a screen indicating that there is a possibility of alteration (S04). Then, the information processing device 1 ends the processing.

[0047] If there are no pixels classified as altered in the classification result (NO in S03), the output unit 133 notifies the information terminal 2 of information indicating that alteration is not permitted (S05). Then, the information processing device 1 ends the processing. Note that if there are no pixels classified as altered, the information processing device 1 may execute subsequent processing (journalizing processing, payment processing, etc.).

[0048] [Effects of information processing device 1] The information processing device 1 is configured to classify the pixels that make up the image data based on a trained model that has been trained to output whether or not the image data of the evidence has been altered, making it possible to detect any alterations to the evidence.

[0049] The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by functionally or physically distributing or integrating in any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination also have the effects of the original embodiments. [Explanation of symbols]

[0050] 1. Information processing equipment 2. Information terminal 11 Communications Department 12 Storage section 13 Control Unit 131 Acquisition Department 132 Learning Department 133 Output section

Claims

1. a storage unit that stores a trained model that has been trained to classify each pixel constituting the input image data based on whether or not it has been altered, when the image data is input based on training data that associates training image data, which is image data obtained by capturing an image of a document, with information indicating pixels that have been altered from the original image data in the training image data; an acquisition unit that acquires target image data that is image data to be determined; An output unit that inputs the target image data into the trained model and outputs candidate locations for modification in the target image data based on the classification result output by the trained model; An information processing device having the above.

2. In the teacher data, the learning image data is associated with information indicating pixels that have been modified from original image data in the learning image data and the type of modification for the pixels, The trained model is trained to classify each pixel constituting the image data based on the presence or absence of modification and the type of modification, based on the training image data, the output unit outputs candidate locations of alteration in the target image data and types of alteration in the candidate locations. The information processing device according to claim 1 .

3. When the training image data is input, the trained model is trained to classify each pixel constituting the image data into either (1) a pixel without any modification, or (2) a pixel modified by replacing a part of a region constituting the input image data with another region, or a pixel modified in other ways, in terms of the type of modification. The information processing device according to claim 2 .

4. the output unit causes the display unit to display a screen for displaying candidate portions of the target image data for modification in a manner different from other portions; The information processing device according to claim 1 .

5. The computer executes an acquisition step of acquiring target image data that is image data to be determined; The target image data is input based on training data stored in a storage unit, which is image data of a document captured, and which is image data for learning, and information indicating pixels in the training image data that have been altered from the original image data. When image data is input, the target image data is input to a trained model that has been trained to classify each pixel that makes up the input image data based on whether or not it has been altered, and candidate locations of alterations in the target image data are output based on the classification results output by the trained model. An information processing method comprising:

6. On the computer, an acquisition step of acquiring target image data that is image data to be determined; The target image data is input based on training data stored in a storage unit, which is image data of a document captured, and which is image data for learning, and information indicating pixels in the training image data that have been altered from the original image data. When image data is input, the target image data is input to a trained model that has been trained to classify each pixel that makes up the input image data based on whether or not it has been altered, and candidate locations of alterations in the target image data are output based on the classification results output by the trained model. A program to execute.

Images