Image processing system, image processing method, and storage medium

Abstract

To reduce the processing load of a computer.SOLUTION: A training data acquisition unit (101) of an image processing system (1) acquires training data including, as an input portion, a training target image in which a training target document is shown and a training reference image in which a training reference document is shown, and including, as a correct portion, correct answer information for processing the training target image such that a training target pose of the training target document in the training target image matches a training reference pose of the training reference document in the training reference image. The training unit (102) trains a learning model for image processing on the basis of training data such that correct answer information is output when a training target image and a training reference image are input.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an image processing system, an image processing method, and a program. [Background technology]

[0002] Conventionally, there are known techniques for processing document images in which a document is displayed. For example, Patent Document 1 describes a technique for correcting the posture of a document in a document image by matching a group of feature points extracted from a document image in which the document is displayed with a group of feature points extracted from a sample image in which the document is displayed, and processing the document image so that the positional relationship of the feature points in the document image matches or approaches the positional relationship of the feature points in the sample image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2020 / 008628 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology of Patent Document 1 requires extracting a large number of feature points from a document image, which increases the processing load on a computer that performs image processing. For example, if documents are continuously photographed with a smartphone camera and image processing is performed on the continuously generated document images, the technology of Patent Document 1 increases the processing load on the smartphone. This also applies to computers other than smartphones.

[0005] One of the goals of this disclosure is to reduce the processing load on a computer. [Means for solving the problem]

[0006] The image processing system according to the present disclosure includes a training data acquisition unit that acquires training data including, as input portions, a training target image showing a training target document and a training reference image showing a training reference document, and includes, as a correct answer portion, correct answer information for processing the training target image so that the training target orientation of the training target document in the training target image matches the training reference orientation of the training reference document in the training reference image; and a learning unit that trains an image processing learning model based on the training data so that the correct answer information is output when the training target image and the training reference image are input. [Effects of the Invention]

[0007] The present disclosure can reduce the processing load on a computer. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example of a hardware configuration of an image processing system. [Figure 2] FIG. 10 is a diagram showing an example of a captured image uploaded by a user. [Figure 3] FIG. 2 is a diagram illustrating an example of functions realized by the image processing system. [Figure 4] FIG. 10 is a diagram illustrating an example of a learning model. [Figure 5] FIG. 10 is a diagram illustrating an example of a training database. [Figure 6] FIG. 10 is a diagram illustrating an example of a loss function used during learning. [Figure 7] FIG. 10 is a diagram illustrating an example of learning performed based on a loss function. [Figure 8] FIG. 10 is a diagram illustrating an example of a learning process. [Figure 9] FIG. 10 is a diagram illustrating an example of an estimation process. [Figure 10] FIG. 10 is a diagram illustrating an example of a function realized in a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0009] [1. Overall configuration of image processing system] An example of an embodiment of an image processing system according to the present disclosure will be described. Fig. 1 is a diagram showing an example of the hardware configuration of the image processing system. For example, the image processing system 1 includes a learning terminal 10, a server 20, and a user terminal 30. The learning terminal 10, the server 20, and the user terminal 30 can be connected to a communication network CN such as the Internet or a LAN.

[0010] The learning terminal 10 is a computer that executes learning of the learning model described below. For example, the learning terminal 10 is a personal computer, a server computer, a smartphone, or a tablet. The learning terminal 10 includes a control unit 11, a memory unit 12, a communication unit 13, an operation unit 14, and a display unit 15. The control unit 11 includes at least one processor. The memory unit 12 includes at least one of volatile memory such as RAM and non-volatile memory such as flash memory. The communication unit 13 includes at least one of a communication interface for wired communication and a communication interface for wireless communication. The operation unit 14 is an input device such as a touch panel. The display unit 15 is an LCD display or an organic EL display.

[0011] The server 20 is a server computer that uses a trained learning model. The server 20 includes a control unit 21, a storage unit 22, and a communication unit 23. The hardware configurations of the control unit 21, the storage unit 22, and the communication unit 23 may be similar to those of the control unit 11, the storage unit 12, and the communication unit 13, respectively.

[0012] The user terminal 30 is a user's computer. For example, the user terminal 30 is a personal computer, a smartphone, a tablet, or a wearable terminal. The user terminal 30 includes a control unit 31, a memory unit 32, a communication unit 33, an operation unit 34, a display unit 35, and an imaging unit 36. The hardware configurations of the control unit 31, the memory unit 32, the communication unit 33, the operation unit 34, and the display unit 35 are the same as those of the control unit 11, the memory unit 12, the communication unit 13, the operation unit 14, and the display unit 15, respectively. The imaging unit 36 ​​includes at least one camera.

[0013] The programs stored in the storage units 12, 22, and 32 may be supplied via a communications network CN. The learning terminal 10, the server 20, or the user terminal 30 may also include a reading unit (e.g., an optical disk drive or a memory card slot) that reads a computer-readable information storage medium, or an input / output unit (e.g., a USB port) for inputting and outputting data to and from an external device. For example, a program stored in an information storage medium may be supplied to the learning terminal 10, the server 20, or the user terminal 30 via the reading unit or the input / output unit.

[0014] Furthermore, the image processing system 1 only needs to include at least one computer. For example, the image processing system 1 may include only the learning terminal 10 and the server 20. In this case, the user terminal 30 exists outside the image processing system 1. The image processing system 1 may also include only the learning terminal 10. In this case, the server 20 and the user terminal 30 exist outside the image processing system 1. The image processing system 1 may also include only the server 20. In this case, the learning terminal 10 and the user terminal 30 exist outside the image processing system 1. The image processing system 1 may also include a computer not shown in FIG. 1.

[0015] [2. Image Processing System Overview] In this embodiment, an example is taken of the case where the image processing system 1 is applied to eKYC (electronic Know Your Customer). eKYC is identity verification performed electronically. In eKYC, a user's identity verification document (identification card) is verified. eKYC may be performed in any service. For example, eKYC may be performed in a communication service, a financial service, a payment service, an e-commerce service, an insurance service, or an administrative service.

[0016] For example, the user operates the user terminal 30 to take a photograph of an identification document using the photographing unit 36. The identification document may be of any type. The identification document may be a driver's license, insurance card, resident registration card, personal identification number card, or passport. The user terminal 30 generates a photographed image showing the identification document photographed by the photographing unit 36. The user terminal 30 uploads the photographed image to the server 20.

[0017] FIG. 2 is a diagram showing an example of a captured image uploaded by a user. In this embodiment, it is assumed that the identity verification document needs to be photographed from the front in order for eKYC to be performed properly. If the user does not photograph the identity verification document from the front, the identity verification document may not be oriented properly or may be distorted, as shown in captured image I at the top of FIG. 2. It is required that the identity verification document be oriented properly and not distorted, as shown in captured image I at the bottom of FIG. 2.

[0018] An appropriate orientation of an identification document means that the up-down direction (vertical or lengthwise direction) of the identification document in the photographed image I matches the up-down direction (vertical or lengthwise direction) of the photographed image I, or the angle between them is less than a predetermined angle (for example, 10°). In other words, an appropriate orientation of an identification document means that the left-right direction (horizontal or widthwise direction) of the identification document in the photographed image I matches the left-right direction (vertical or widthwise direction) of the photographed image I, or the angle between them is less than a predetermined angle (for example, 10°).

[0019] Distortion of an identification document means that the shape of the outline of the identification document in photographed image I differs from the shape of the outline of the actual identification document. For example, if a user photographs an identification document from an angle, the identification document shown in photographed image I will be distorted. If the outline of the identification document is a rectangle, the outline of the identification document shown in photographed image I will become a trapezoid, which corresponds to the identification document being distorted. If the outline of the identification document is a rectangle with rounded corners, the outline of the identification document shown in photographed image I will become a trapezoid with rounded corners, which corresponds to the identification document being distorted.

[0020] For example, when the server 20 receives the photographed image I from the user terminal 30, it detects the identification document from the photographed image I by known image processing such as contour extraction processing. In a state like the photographed image I in the upper part of FIG. 2, the server 20 may not be able to detect the identification document. If the server 20 is unable to detect the identification document, it may consider prompting the user to take a photograph of the identification document again. However, in this case, it requires the user's effort, which reduces user convenience. The same applies when the identification document detection is performed on the user terminal 30 side.

[0021] For example, even when a person in charge of eKYC operations visually checks photographed image I, there is a possibility that the person in charge will not be able to properly verify the identity verification document if the photographed image I is in a state like the upper part of Figure 2. It is time-consuming for the person in charge to check photographed image I by rotating it, etc. For this reason, even when eKYC is performed by a person in charge through visual confirmation, it is required that eKYC be performed with photographed image I in a state like the lower part of Figure 2.

[0022] For example, the server 20 may extract a group of feature points from the identification document shown in the captured image I and process the captured image I so that the positional relationship of the group of feature points matches the positional relationship of the identification document in a state suitable for eKYC. In this case, however, the server 20 needs to extract a large number of groups of feature points from the captured image I, which increases the processing load on the server 20. Furthermore, if the identification document is blurred or reflects light, the server 20 may not be able to properly extract the group of feature points. The same applies when the group of feature points is extracted on the user terminal 30 side.

[0023] Therefore, the learning terminal 10 of this embodiment executes learning of a learning model for acquiring a captured image I suitable for eKYC (for example, a captured image I as shown in the lower part of FIG. 2) from a captured image I unsuitable for eKYC (for example, a captured image I as shown in the upper part of FIG. 2). Based on the trained learning model, even if a captured image I unsuitable for eKYC (for example, a captured image I as shown in the upper part of FIG. 2) is uploaded, the server 20 acquires a captured image I suitable for eKYC (for example, a captured image I as shown in the lower part of FIG. 2). This enables the image processing system 1 to appropriately execute eKYC while reducing the processing load on the server 20. Details of the image processing system 1 will be described below.

[0024] [3. Functions realized by image processing systems] FIG. 3 is a diagram showing an example of functions realized by the image processing system 1. As shown in FIG.

[0025] [3-1. Functions realized on the learning device] For example, the learning terminal 10 includes a data storage unit 100, a training data acquisition unit 101, and a learning unit 102. The data storage unit 100 is realized by the storage unit 12. The training data acquisition unit 101 and the learning unit 102 are realized by the control unit 11.

[0026] [Data storage section] The data storage unit 100 stores data necessary for learning the learning model M. The learning model M is a machine learning model used in image processing. The machine learning method itself may be a known method. For example, the learning model M may be a convolutional neural network (e.g., U-Net), a recurrent neural network, a GAN (Generative Adversarial Networks), a vision transformer, or a model of another method.

[0027] For example, the data storage unit 100 stores a learning model M before learning. The learning model M includes a program that indicates processing of data input to the learning model M and parameters referenced by the program. The parameters of the learning model M may be similar to parameters used in known machine learning. For example, the parameters may be weights, biases, or other coefficients referenced by the program of the learning model M.

[0028] For example, the learning model M before learning includes parameters of initial values. The parameters of the learning model M are adjusted by learning, which will be described later. When learning is completed, the data storage unit 100 stores the trained learning model M. The learning model M before learning may be overwritten with the trained learning model M, or the trained learning model M may be stored in the data storage unit 100 separately from the learning model M before learning. In this embodiment, the trained learning model M is uploaded to the server 20.

[0029] FIG. 4 is a diagram showing an example of a learning model M. In this embodiment, the learning model M is a type of convolutional neural network. For example, the learning model M includes an encoder E, a decoder D, a first network N1, and a second network N2. In the example of FIG. 4, the decoder D is included in the second network N2, but the decoder D may be located outside the second network N2. The encoder E may be included in the first network N1.

[0030] The encoder E calculates the features of the input image input to the learning model M. The features are information indicating the features of the input image. For example, the features are a feature map indicating the features of the input image. The features are sometimes called embedded representations. The encoder E calculates the features by referring to its own parameters and performing convolution on the input image. The formula used by the encoder E to perform convolution on the input image may be a known formula. The features may be in any format, for example, a format expressed by pixel values ​​of multiple pixels like an image, a vector, an array, a single number, a combination of multiple numbers, a matrix, or other format.

[0031] For example, the encoder E may include multiple layers. Each layer of the encoder E calculates features at different levels. Each layer of the encoder E calculates features based on features calculated by the previous layer and its own parameters. Each layer of the encoder E may be called a convolutional layer. The encoder E may include layers other than convolutional layers (e.g., activation function layers, pooling layers, or normalization layers). The configuration of the encoder E may be similar to that of a known encoder E. For example, the encoder E may be a module called a target-aware feature extractor.

[0032] The target-aware feature extractor is an encoder E for extracting useful features for a specific task. When the image processing system 1 is used for eKYC, as in this embodiment, the target-aware feature extractor appropriately extracts features of an identification document. The programs and parameters included in the target-aware feature extractor may be similar to well-known programs and parameters. The encoder E may be an encoder E other than the target-aware feature extractor.

[0033] In this embodiment, since two input images are input to the encoder E, the example of FIG. 4 schematically shows two encoders E, but in reality, there is only one encoder E. However, the learning model M may include multiple encoders E. For example, there may be a separate encoder E for processing a certain input image and a separate encoder E for processing another input image. Also, in the example of FIG. 4, four layers are shown in the encoder E, but the number of layers included in the encoder E is not limited to four. For example, the encoder E may include one, two, three, five or more layers. The same applies to the decoder D. In the example of FIG. 4, two decoders D are schematically shown, but in reality, there is only one decoder D. However, the learning model M may include multiple decoders D, and the number of layers of the decoder D is not limited to the example of FIG. 4.

[0034] In this embodiment, a target image showing an identification document and a reference image are input as input images to the learning model M. The target image is an image to be processed. Processing is image processing that changes the posture of the document shown in the target image. Processing can also be called shaping or deformation. For example, processing may be translation, rotation, enlargement, reduction, shearing, or a combination of these. Image processing called affine transformation is also a type of processing. Changing the arrangement of each pixel included in the input image is also a type of processing.

[0035] The document posture is at least one of the orientation, shape, and position of the document in the image. When the positional relationship between the viewpoint of a camera, such as the photographing unit 36, and the document changes, at least one of the orientation, shape, and position of the document in the image changes, so the document posture can also be said to be the positional relationship between the viewpoint and the document. In this embodiment, an identification document, which is an example of a document, is photographed by the user, and therefore the identification document is shown in the target image in a posture that corresponds to the positional relationship between the photographing unit 36 ​​and the identification document at the time of photographing.

[0036] The reference image is an image showing an identification document in a predetermined pose. The predetermined pose is a desired pose for the target image after processing. The predetermined pose can also be called a target pose or an appropriate pose. The reference image can also be called a sample image showing an identification document in a predetermined pose. The learning terminal 10 aims to create a learning model M that realizes processing such that the pose of the identification document shown in the target image becomes or approaches the pose of the identification document shown in the reference image.

[0037] Note that any type of document may be shown in each of the target image and the reference image. The document shown in the target image and the reference image is not limited to an identification document. In this embodiment, the term "identification document" may be replaced with any document. For example, the document may be an estimate, invoice, receipt, contract, report, specification, manual, catalog, or other document. In this embodiment, an example is given in which the type of document shown in the target image and the type of document shown in the reference image are the same, but these document types may be different from each other. For example, a driver's license may be shown in the target image and an insurance card may be shown in the reference image.

[0038] In this embodiment, the target image and the reference image during learning are referred to as the training target image and the training reference image, respectively. In FIG. 4, the flow of the learning process is shown, and the symbol I t ,I r are the training target image and the training reference image, respectively. The target image and the reference image at the time of estimation are called the estimation target image and the estimation reference image, respectively. When there is no particular distinction between the training target image and the estimation target image, they are simply called the target image. When there is no particular distinction between the training reference image and the estimation reference image, they are simply called the reference image.

[0039] When the first network N1 receives target features, which are features of the target image calculated by the encoder E, and reference features, which are features of the reference image calculated by the encoder E, it outputs processing information for processing the target image. In this embodiment, an example is given in which the processing information corresponds to a conversion coefficient referenced in image processing that changes the arrangement of each pixel of the target image. The target image is processed by executing image processing that changes the arrangement of each pixel of the target image based on the processing information. When the target image is processed appropriately, the orientation of the document in the processed target image becomes the same as or approaches the orientation of the document in the reference image.

[0040] For example, the first network N1 is trained on the image I t and each pixel of the training reference image I r The network may be used to identify correspondences between each pixel of the training image I and the corresponding pixel of the training image I. These correspondences are sometimes called mappings. t By changing the arrangement of each pixel in the training image I t The pose of the training document shown in is compared with the training reference image I r The first network N1 assumes or approaches the posture of the training reference document shown in FIG. 1. When such processing is performed, the first network N1 can also be called a mapping network. Details of the processing of the first network N1 will be described in the section on the function of the learning unit 102.

[0041] The second network N2 outputs a segmentation map of the image to be processed based on the decoder D. The segmentation map is information indicating the classification of each pixel in the image. In this embodiment, the segmentation map is an image in which the classification results are visualized, but the segmentation map may be in a format other than an image. For example, the segmentation map indicates at least one of whether or not a document is represented in each pixel and the type of document represented in each pixel.

[0042] For example, the second network N2 outputs a target segmentation map, which is a segmentation map of the target image, based on the target features of the processed target image. The second network N2 outputs a reference segmentation map, which is a segmentation map of the reference image, based on the reference features of the reference image. The second network N2 can also be called a segmentation network that outputs these segmentation maps. Details of the processing of the second network N2 will be described in the function of the learning unit 102.

[0043] For example, the data storage unit 100 stores a training database DB in which multiple pieces of training data to be learned by the learning model M are stored. The training data includes an input portion that is input to the learning model M during learning, and a correct answer portion (output portion) that is the correct answer during learning. The correct answer portion is not limited to the final output of the learning model M, but may also be an output indicating an intermediate result calculated by the learning model M to obtain the final output. The correct answer portion may also be a result obtained from the final output of the learning model M.

[0044] FIG. 5 is a diagram showing an example of the training database DB. For example, the input part of the training data is a training target image I t and training reference image I r Training image I t denotes the training target document in the first pose. Training reference image I r denotes a training target document in a second pose. The second pose is a pose that is different from the first pose. The second pose can also be called an appropriate pose, a correct pose, or a desired pose. The learning model M is generated from the training target image I t The aim is to process the training target documents in from the first posture to the second posture.

[0045] For example, given a training image I t The first pose of the training document is shown by and the other training image I t The first pose of the training target document represented by and may be different from each other. tThe first pose of the training target document indicated by may be a pose that is not suitable for eKYC or may be a pose that is suitable for eKYC. r The second pose of the training reference document shown by and the other training reference image I r The second pose of the training reference document indicated by and may be different from each other. r The second pose of the training reference document indicated by may be a pose appropriate for eKYC, or may not be a pose appropriate for eKYC. In order to train the learning model M on various poses, training target images I t and training reference image I r It is assumed that training data including the above is stored in a training database DB.

[0046] The correct part of the training data is the correct answer information, and the processed training target image I t It may contain the training image I t The training data may include processing information used in processing the training data, or may include other information. In this embodiment, a case is exemplified in which the correct answer portion of the training data is correct processing information, which is processing information that is the correct answer. The correct answer portion of the training data may include information other than the correct processing information.

[0047] In the example of FIG. 5, the correct answer portion of the training data includes, as correct answer processing information, correct answer basic processing information and correct answer final processing information. The correct answer portion of the training data includes, as other information, a correct answer target segmentation map and a correct answer reference segmentation map. In the example of FIG. 5, the symbols for these four pieces of information are marked with a bar. In the following explanation, the bar for each symbol will be expressed in parentheses, such as H(bar), w(bar), sr(bar), and st(bar). In this embodiment, the correct answer portion of the training data includes correct answer processed information T(I t , w(bar)). Details of these five pieces of information will be explained later.

[0048] The data stored in the data storage unit 100 is not limited to the above example. For example, the data storage unit 100 may store a program that indicates the processing during learning. The program may define a formula for calculating the loss function.

[0049] [Training data acquisition section] The training data acquisition unit 101 acquires a training target image I t and training reference image I, where the training reference document is shown. r and as input parts, and the training target image I t The training pose of the training document in r The training target image I is then matched to the training reference pose of the training reference document in t In this embodiment, the training data is stored in a training database DB, and the training data acquisition unit 101 acquires the training data from the training database DB. The training data stored in the training database DB is prepared by a creator who creates the learning model M (for example, a person who operates the learning terminal 10).

[0050] If the training data is stored in a database other than the training database DB, the training data acquisition unit 101 may acquire the training data from the other database. If the training data is stored in a computer or information storage medium other than the learning terminal 10, the training data acquisition unit 101 may acquire the training data from the other computer or information storage medium. The training data acquisition unit 101 can acquire any number of pieces of training data. For example, the training data acquisition unit 101 acquires all or part of the training data stored in the training database DB. The training data acquisition unit 101 may repeat acquiring training data until the values ​​of each loss function described below become sufficiently small.

[0051] [Study Department] The learning unit 102 generates a training target image I based on the training data. tFor example, the learning unit 102 trains the learning model M for image processing so that correct answer information is output when the training target image Ir, which is the input part of the training data, is input. t and training reference image I r The learning model M is input based on the current parameters and the training image I t training target features and training reference image I r The training model M is calculated based on the current parameters and the training target image I t and the training reference features of the training reference image Ir, t The processing information for the processing (for example, the basic processing information H and the final processing information w described below) is calculated and output.

[0052] For example, the learning unit 102 calculates the loss based on the output of the learning model M (for example, the basic processing information H and final processing information w described below), the correct answer part of the training data (for example, the training basic processing information H(bar) and final processing information w(bar) described below), and a predetermined loss function. The learning unit 102 executes learning of the learning model M by adjusting the parameters of the learning model M so as to reduce the loss. When the training data acquisition unit 101 successively acquires each of a plurality of training data, the learning unit 102 calculates the loss based on the training target image I included in each of the training data for each training data. t and training reference image I r is input to the learning model M, the output from the learning model M is obtained, the loss is calculated based on the loss function, and the process of adjusting the parameters so that the loss is small is repeated.

[0053] The learning unit 102 may perform learning of the learning model M based on a known learning algorithm adopted in machine learning techniques. For example, the learning unit 102 may cause the learning model M to learn training data based on an error backpropagation method, a gradient descent method, an ADAM (Adaptive Moment) method, a momentum method, a method using a classifier and a generator adopted in GAN, or other methods. The learning unit 102 may repeat learning of the learning model M until the loss becomes less than a threshold, or may repeat learning of the learning model M until the number of learning times reaches a predetermined number. The learning unit 102 may use the same training data for repeated learning.

[0054] FIG. 6 is a diagram showing an example of a loss function used during learning. FIG. 7 is a diagram showing an example of learning performed based on the loss function. In this embodiment, the learning model M is t The training target feature f t l and training reference image I r The training reference features f r l and the encoder E that computes the training feature f t l and training reference features f r l Based on the training target image I t and a first network N1 that calculates processing information related to the processing of the image data. The learning unit 102 trains the encoder E of the learning model M and the first network N1.

[0055] In addition, the training target feature f t l and training reference features f r l The symbol l (lowercase L) in denotes the number of layers included in the encoder E. In this embodiment, the encoder E includes four layers, so the value of the symbol l is 4. As mentioned above, the encoder E can include any number of layers, so the value of l is not limited to 4. t l and training reference features f r lare features output by the final layer (fourth layer in this embodiment) of the encoder E. The training target feature f t 1 ~f t l-1 and the training reference features f r 1 ~f r l-1 and are calculated.

[0056] In this embodiment, as shown in Figs. 4, 6 and 7, the encoder E is configured to generate training target features f t 1 ~f t l and training reference features f r 1 ~f r l The encoder E includes multiple layers that compute the training target features f based on the parameters of each of the multiple layers. t 1 ~f t l and training reference features f r 1 ~f r l For example, the first layer of the encoder E computes the training image I t Based on the parameters of the first layer, the training target feature f t 1 The second layer of the encoder E computes the training feature f t 1 Based on the parameters of the second layer, the training target feature f t 2 In this way, each layer computes the training feature f t k-1 Based on the parameters and the training target feature f t k The final layer computes the final training target feature f t l Here, k is an arbitrary value between 1 and 1. When k is 1, there is no layer before it, and the training target feature f tk-1 Since there is no , the first layer calculation described above is performed.

[0057] For example, the first layer of the encoder E is the training reference image I r Based on the parameters of the first layer, the training reference features f r 1 The second layer of the encoder E computes the training reference features f r 1 Based on the parameters of the second layer, the training reference features f r 2 Then, the third and subsequent layers of the encoder E use the training reference features f r k-1 Based on the parameters of the training reference feature f r k The final layer computes the final training reference features f r l Output.

[0058] As shown in Figure 6, the first network N1 is trained on the feature f t l and training reference features f r l Based on this, the basic processing information H is the basic processing information that is calculated first, and the intermediate processing information W is the processing information that is calculated intermediately. l-1 ~w 1 For example, the first network N1 calculates the training target feature f calculated by the layer for each layer. t k and training reference features f r k The processing information is calculated based on the above, and the final processing information, which is the final processing information, is calculated. Hereafter, the basic processing information H and the intermediate processing information W are l-1 ~w 1 , and the method for calculating the final processing information w will be explained.

[0059] First, we will explain how to calculate the basic processing information H. The first network N1 uses the training target features f calculated by the final layer of the multiple layers.t l and training reference features f r l The basic processing information H is calculated based on the training target feature f t l and the training reference features f r l The basic processing information H is information indicating the correspondence (for example, the correspondence between pixels in a feature map, or the relationship between a pixel in a feature map and a corresponding pixel in another feature map) or the difference between the training target feature f t l , the training reference features f r l The basic processing information H may be in any format, such as a vector, a matrix, a single numerical value, a combination of multiple numerical values, an array, or other formats.

[0060] For example, the first network N1 is trained on the feature f t l and training reference features f r l When input, the initial posture network calculates the basic processing information H (in Figures 4, 6, and 7, the training target feature f t l and training reference features f r l and the basic processing information H). The initial posture network includes a plurality of neurons. The initial posture network is configured to train the target feature f t l and training reference features f r l The difference between the two is calculated and input to the neuron. When the neuron receives the difference, it performs calculations such as weighted sum, adds bias as needed, and passes the calculation result to other neurons. When multiple neurons perform calculations one after another, the basic processing information H is output as the final output. Through the calculations of each neuron, the initial posture network calculates the training target feature f t l , training reference features f r lBased on the parameters of each neuron, the basic processing information H is calculated. The parameters of each neuron are also adjusted through learning.

[0061] For example, the correct processing information includes correct basic processing information H (bar), which is the correct basic processing information. The learning unit 102 calculates the basic processing information loss L based on the basic processing information H calculated during learning and the correct basic processing information H (bar). H Calculate the basic processing information loss L H In this embodiment, the learning model M is trained based on the basic processing information loss L H The calculation formula is as follows: Basic processing information loss L H The calculation formula for this may be other than the following formula 1. For example, the learning unit 102 multiplies at least one of the basic processing information H and the correct basic processing information H (bar) by a coefficient, and calculates the difference between them as the basic processing information loss L H It may be calculated as:

[0062]

number

[0063] Next, intermediate processing information l-1 ~w 2 The first network N1 calculates the training target feature f calculated by each layer from the layer that is later in the order of the layers. t l and training reference features f r l Based on this, intermediate processing information w l-1 ~w 1 is calculated to calculate the final processing information w, which is the final processing information. The order of layers is the order of layers in the decoder D (the numerical value of k mentioned above). In this embodiment, since the decoder D includes four layers, there are first to fourth orders.

[0064] For example, the first network N1 uses the training feature f calculated by the final layer (e.g., the fourth layer) tl Based on the basic processing information H, the processed training target feature T(f t l ,H) is calculated. t l , H) is the training feature f after calculation by formula T. t l The processed training target features T(f t l ,H) is the training target feature f t l and the basic processing information H into a predetermined calculation formula T. The calculation formula T has two arguments. Here, the first argument is the training target feature f t l The second argument is the basic processing information H.

[0065] For example, if the first argument of formula T is image-format information such as a feature map, and the second argument is information for changing the arrangement of each pixel, then formula T will obtain image-format information in which the arrangement of each pixel has been changed by the second argument. Formula T may be any formula, and may be, for example, a formula that adds, subtracts, multiplies, or divides the first and second arguments based on a predetermined coefficient. If formula T includes a coefficient, the coefficient may be one of the parameters adjusted by learning. That is, the coefficient of formula T may also be adjusted by learning. Formula T may be a formula used in a technique called feature warping (a technique for spatially transforming the features of data such as an image).

[0066] For example, the first network N1 is trained on the processed training target features T(f t l , H) and the training reference features f r l Based on this, intermediate processing information w l-1 Calculate the intermediate processing information w l-1 is the processed training target feature T(f t l ,H) (e.g., training target features f t lThe feature map () after the feature map indicated by is processed by the basic processing information H) and the training reference feature f r l The processed training target feature T(f t l ,H) is the training target feature f t l than the training reference features f r l Approaching intermediate processing information l-1 may be in any format, such as a vector, a matrix, a single number, a combination of multiple numbers, an array, or other format. l-2 ~w 1 The same is true.

[0067] For example, the first network N1 calculates the processed training target features T(f t l , H) and the training reference features f r l Intermediate processing information w showing the correspondence or difference between l-1 Calculate the intermediate processing information w l-1 is the processed training target feature T(f t l ,H) and the training reference feature f r l The first network N1 may represent the correspondence between the pixels of the image represented by and the intermediate processing information w based on a calculation formula using a method other than Cost Volume. l-1 For example, the first network N1 may calculate the processed training target features T(f t l , H) and the training reference features f r l And the difference, intermediate processing information w l-1 It may be calculated as:

[0068] For example, the first network N1 has intermediate processing information w l-1 and the training feature f computed by the penultimate layer (e.g., the third layer)t l-1 Based on this, the processed training target features T(f t l-1 ,w l-1 ) is calculated. The calculation formula T is as described above. The first network N1 calculates the processed training target feature T(f t l-1 ,w l-1 ) and the training reference features f r l-1 Based on this, intermediate processing information w l-2 Calculate the intermediate processing information w l-2 The formula used to calculate the intermediate processing information w l-1 For example, the intermediate processing information w l-2 In the calculation of the above, a calculation formula using a technique called Cost Volume may be used.

[0069] Similarly, the first network N1 executes the same calculations in succession in each layer of the encoder E, starting from the last layer, and generates intermediate processing information w 1 The first network N1 calculates the intermediate processing information w 1 and the training target feature f calculated by the first layer t 1 Based on this, the processed training target features T(f t 1 ,w 1 ) is calculated. The first network N1 calculates the processed training target features T(f t 1 ,w 1 ) and the training reference features f r 1 The final processing information w is calculated based on the intermediate processing information w l-1 ~w 1 For example, a calculation formula using a method called Cost Volume may be used to calculate the final processing information w.

[0070] For example, the correct processing information includes correct final processing information w (bar), which is the final processing information w that is the correct answer. The learning unit 102 calculates the final processing information loss L based on the final processing information w calculated during learning and the correct final processing information w (bar). w Calculate the final processing information loss L w In this embodiment, the final processed information loss L w The calculation formula is as follows: Final processing information loss L w The calculation formula for may be other formulas than the following formula 2. For example, the learning unit 102 multiplies at least one of the final processing information w and the correct final processing information w (bar) by a coefficient, and calculates the difference between them as the final processing information loss L w It may be calculated as:

[0071]

number

[0072] For example, the correct answer information is the processed training image T(I t ,w) t , w(bar)). The learning unit 102 generates a training target image T(I t , w) and the correct processed information T(I t ,w(bar)) and, based on that, the post-processing loss L I Calculate the post-processing loss L I When the final processing information w indicates Cost Volume, the learning unit 102 trains the training target image I t By converting each pixel of the image into the position indicated by the final processing information w, the processed training image T(I t , w). When the final processing information w is a transformation coefficient such as an affine transformation, the learning unit 102 obtains the training target image I t By performing a transformation according to the final processing information w, the processed training target image T(I t ,w).

[0073] In this embodiment, the post-processing loss L I The calculation formula for loss after processing is as shown in Equation 3 below. I The calculation formula for may be other formulas than the following formula 3. For example, the learning unit 102 calculates the training target image T(I t ,w) and the correct processed information T(I t , w(bar)) and at least one of the pixel values ​​of the pixels are multiplied by a coefficient, and the difference between them is the post-processing loss L I For example, the following formula 3 can be used to calculate the post-processing training image T(I t , w) and the correct processed information T(I t , w(bar)) and the pixel value difference, or if each pixel is assigned a label indicating whether it represents a document or not, it may indicate whether the label of each pixel matches.

[0074]

number

[0075] As described above, in this embodiment, the correct information includes correct processing information, which is processing information that is correct. Although the correct basic processing information H (bar) and the correct final processing information w (bar) are used as an example of the correct processing information, only either the correct basic processing information H (bar) or the correct final processing information w (bar) may be used as the correct processing information. The learning unit 102 calculates the processing information loss based on the processing information calculated by the first network N1 during learning and the correct processing information, and trains the learning model M based on the processing information loss.

[0076] For example, as an example of processing information loss, the basic processing information loss L H and final processing information loss L w As an example, the basic processing information loss L H or final processing information loss L w Only one of the above may be used. l-1 ~w1 The learning unit 102 prepares intermediate processing information w that is also correct for the l-1 ~w 1 The loss may be calculated based on the intermediate processing information that is the correct answer, and the learning model M may be trained based on the loss.

[0077] For example, the learning model M may include a decoder D that outputs a segmentation map and other parts for processing the training target image It. The other parts are parts other than the decoder D. For example, the other parts are an encoder E and a first network N1. In this embodiment, the decoder D is included in the second network N2, so the case where the second network N2 generates the segmentation map will be taken as an example. In the example of FIG. 6, the decoder D in a U-net is shown, but the decoder D may be a decoder D in a convolutional neural network other than a U-net, or may be a network in another machine learning method other than a convolutional neural network.

[0078] For example, the decoder D generates the processed training target features T(f t l ,H)~T(f t 1 ,w 1 ) and perform upsampling based on each of the processed training image T(I t , w) may be input to the encoder E, and the processed training target features calculated by each layer of the encoder E may be input to the decoder D. The decoder D may also include multiple layers, just like the encoder E. Each layer of the decoder D performs upsampling based on its own parameters and outputs a segmentation map. By upsampling, the processed training target features T(f t l ,H)~T(f t 1 ,w 1) is restored to its original resolution. In addition to upsampling, the decoder D may also perform a process of restoring the resolution to its original resolution by a technique called transposed convolution or uppooling.

[0079] In Figure 6, the symbol s indicates the segmentation map. The symbol with a hat above st indicates the processed training target feature T(f t l ,H)~T(f t 1 ,w 1 ) is a segmentation map generated from the t (hat), and write it in parentheses. r is the training reference feature f r l ~f r 1 The segmentation map generated from the processed training image T(I t ,w) as the training target segmentation map s t As mentioned above, the training target segmentation map s t (Hat) denotes the processed training image T(I t , w) may be input to an encoder E to obtain the signal.

[0080] For example, the learning unit 102 may use the training object image I t The processed training image T(I t ,w) is the segmentation map of the training target, s t (hat) and the processed training image T(I t ,w) is the first correct segmentation map s t (bar) and the first segmentation map loss L based on s1 Calculate the first segmentation map loss L s1 In this embodiment, the learning model M is trained based on the first segmentation map loss L s1The calculation formula for is as shown in Equation 4 below. CE in Equation 4 below is the cross entropy. The first segmentation map loss L s1 The calculation formula for the first segmentation map loss L may be other than the following formula 4. For example, s1 may be calculated by other calculation methods such as mean square error, other than cross entropy.

[0081]

number

[0082] For example, the decoder D receives training images I t A training object segmentation map s indicating the pose and type of the training object document in t (Hat) can be output as the training target segmentation map s t (Hat) shows the processed training image T(I t ,w) shows the position and type of the training target document. In the example of Figure 6, the type of training target document is shown by a color that is expressed by a pattern or the presence or absence of a pattern. For example, the classification result of the identification document is shown by color, such as red for driver's license, blue for insurance card, and yellow for personal number card. The training target segmentation map s t The part of the (hat) other than the training target document will be a predetermined background color. The part other than the background color (red part, etc.) will be the processed training target image T(I t ,w) is the part where the identification documents are shown.

[0083] For example, the first correct segmentation map s t The (bar) may indicate the correct pose and type. t The correct training document part among the (bars) indicates the correct color type. First segmentation map loss L s1 is the training target segmentation map s t The pixel value (color) of each pixel indicated by (hat) and the first correct segmentation map s tThe smaller these differences are, the better the processed training image T(I t ,w) will ultimately indicate the result you want to achieve.

[0084] For example, the decoder D receives the training reference image I r The training reference features f r l ~f r 1 By upsampling, the training target features f r l ~f r 1 The resolution of each of the training reference images I is restored. r may be input to the encoder E, and the processed training reference features calculated by each layer of the encoder E may be input to the decoder D. The learning unit 102 may r The training reference segmentation map s is the segmentation map of r and the training reference image I r The second correct segmentation map s r (bar) and the second segmentation map loss L based on s2 Calculate the second segmentation map loss L s2 In this embodiment, the second segmentation map loss L s2 The calculation formula for is as shown in the following Equation 5. CE in the following Equation 5 is the cross entropy. The second segmentation map loss L s2 The calculation formula for the second segmentation map loss L may be other than the following formula 5. For example, s2 may be calculated by other calculation methods such as mean square error, other than cross entropy.

[0085]

number

[0086] For example, the decoder D receives the training reference image Ir training reference segmentation maps s indicating the pose and type of the training reference documents in r The training reference segmentation map s r In the training reference image I r In the example of FIG. 6, the types of training reference documents are indicated by colors that are represented by a schematic pattern. The meaning of the colors may be the same as the types of training target documents. The training reference segmentation map s r The parts of the training reference document that are not part of the training reference document will be a predetermined background color. The parts other than the background color (red, etc.) will be the training reference image I r This is the part where identification documents are shown.

[0087] For example, the second ground truth segmentation map s r indicates the correct pose and type. The second correct segmentation map s r The correct training document part indicates the correct color type. The second segmentation map loss L s2 is the training reference segmentation map s r The pixel value (color) of each pixel indicated by and the second correct segmentation map s r The smaller these differences are, the better the training reference segmentation map s r This means that the accuracy is high.

[0088] As described above, the learning unit 102 of this embodiment uses the basic processing information loss L H , final processing information loss L w , loss L after machining I , the first segmentation map loss L s1 , and the second segmentation map loss L s2For example, the learning unit 102 may calculate a total loss, which is the sum of these, and perform learning of the learning model M so that the total loss becomes smaller. The method by which the learning unit 102 performs learning of the learning model M based on the loss may be similar to a known method (e.g., gradient descent). For example, the learning unit 102 may perform learning of the learning model M based on the gradient of the total loss.

[0089] In the example of FIG. 7, the learning unit 102 calculates the basic processing information loss L H , final processing information loss L w , loss L after machining I , and the first segmentation map loss L s1 For example, the learning unit 102 performs learning of the encoder E and the first network N1 based on the basic processing information loss L H , final processing information loss L w , loss L after machining I , and the first segmentation map loss L s1 The encoder E and the first network N1 are trained so that the total loss, which is the sum of the basic processed information loss L H , final processing information loss L w , loss L after machining I , and the first segmentation map loss L s1 A coefficient may be set for at least one of the above.

[0090] In the example of FIG. 7, the training unit 102 calculates the second segmentation map loss L s2 For example, the training unit 102 performs training of the encoder E and the decoder D based on the second segmentation map loss L s2 The training unit 102 may perform training of the encoder E and the decoder D based on the gradient of the second segmentation map loss L s2 Based on the above, training of only one of the encoder E or decoder D may be performed.

[0091] The learning unit 102 calculates the basic processing information loss L H , final processing information loss L w, loss L after machining I , the first segmentation map loss L s1 , and the second segmentation map loss L s2 Alternatively, the learning unit 102 may calculate only a part of the basic processed information loss L H Only the basic processing information loss L H The learning unit 102 may train the learning model M based only on the final processed information loss L w Only the final processing information loss L w The learning unit 102 may learn the learning model M based only on the processed loss L I Calculate only the post-processing loss L I The training unit 102 may train the training model M based only on the first segmentation map loss L s1 , and calculate the first segmentation map loss L s1 The learning model M may be trained based on only the

[0092] [3-2. Functions realized by the server] For example, the server 20 includes a data storage unit 200 and an estimation unit 201. The data storage unit 200 is realized by the storage unit 22. The estimation unit 201 is realized by the control unit 21.

[0093] [Data storage section] The data storage unit 200 stores data necessary for processing the estimation target image. For example, the data storage unit 200 stores a trained learning model M. The data storage unit 200 may store an estimated reference image showing an estimated reference document in a pose suitable for eKYC. For example, the estimated reference image may show an identification document photographed from the front, as in the captured image I at the bottom of Figure 2. The estimated reference image need only show an identification document in a pose that is desired after processing the estimation target image, and is not limited to an identification document photographed from the front. For example, if it is desired to process the estimation target image so that a predetermined distortion occurs, the estimated reference image may show an identification document with the predetermined distortion.

[0094] [Estimation part] The estimation unit 201 inputs an estimation target image showing an estimation target document and an estimated reference image showing an estimated reference document into a trained learning model M, and acquires a processed estimation target image, which is an estimation target image processed so that the estimation target orientation of the estimation target document matches the estimated reference orientation of the estimated reference document. The processing executed when the estimation target image and estimated reference image are input into the trained learning model M is the same as the processing executed when a training target image and training reference image are input into the learning model M during learning. Of the processing of the learning model M described in the function of the learning unit 102, the processing executed during estimation is the processing executed after the training target image and training reference image are input into the learning model M, with "training" replaced with "estimation".

[0095] For example, the estimation unit 201 inputs the estimation target image and the estimated reference image into a learning model M including a trained encoder E and a first network N1, and acquires an edited estimation target image. Based on the decoder D, the estimation unit 201 acquires an estimation target segmentation map, which is a segmentation map corresponding to the edited estimation target image edited using other parts. These processes are also similar to the processes during learning. The estimation unit 201 may estimate what kind of identification document was photographed based on the color shown in the estimation target segmentation map. Furthermore, the estimation unit 201 may output the estimation result to an eKYC officer. The server 20 may perform known image processing for eKYC on the estimation target image edited by the estimation unit 201.

[0096] For example, when the estimation unit 201 inputs an estimation target image and an estimated reference image to the learning model M, the encoder E of the learning model M calculates estimation target features, which are features of the estimation target image, and estimated reference features, which are features of the estimated reference image, based on parameters adjusted by learning. If the encoder E includes multiple layers, the estimation target features and estimated reference features are calculated by each of the multiple layers. The first network N1 of the learning model M calculates intermediate processing information based on the estimation target features and estimated reference features calculated by the final layer, and then calculates intermediate processing information successively based on the estimation target features and estimated reference features calculated by each layer. The first network N outputs the final processing information.

[0097] For example, the estimation unit 201 processes the estimation target image based on the final processing information to obtain the processed estimation target image. In this embodiment, the processed estimation target image is an image in which the arrangement of each pixel of the estimation target image has been changed based on the final processing information. The orientation of the estimation target document shown in the processed estimation target image becomes the same as or approaches the orientation of the estimated reference document shown in the estimated reference image. The estimation unit 201 inputs the estimation target features calculated by each layer to the decoder D, and the decoder D outputs an estimation target segmentation map based on parameters adjusted by learning. The internal processing of the decoder D is as described above.

[0098] [3-3. Functions implemented on user devices] For example, the user terminal 30 includes a data storage unit 300 and a transmission unit 301. The data storage unit 300 is realized by the storage unit 32. The transmission unit 301 is realized by the control unit 31.

[0099] [Data storage section] The data storage unit 300 stores data necessary for generating an estimation target image. For example, the data storage unit 300 stores an estimation target image generated by the imaging unit 36.

[0100] [Transmitter] The transmission unit 301 transmits the estimation target image generated by the imaging unit 36 ​​to the server 20. The transmission unit 301 may transmit the estimation target image stored in the data storage unit 300 to the server 20.

[0101] [4. Processing performed by the image processing system] Hereinafter, as examples of processing executed by the image processing system 1, a learning processing for executing learning of the learning model M and an estimation processing using the learned learning model M will be described.

[0102] [4-1. Learning process] 8 is a diagram showing an example of the learning process. The control unit 11 executes a program stored in the storage unit 12, thereby executing the learning process.

[0103] As shown in Fig. 8, the learning terminal 10 acquires training data from the training database DB (S100). The learning terminal 10 acquires the training target image I, which is the input part of the training data. t and training reference image I r is input to the learning model M (S101). t , training reference image I r , and based on the encoder E, the training target features f t l and training reference features f r l (S102). The learning terminal 10 calculates the training target feature f t l , training reference features f r l , and the first network N1, the basic processing information H is calculated (S103). The learning terminal 10 calculates the training object features f t l ~f t 1 and training reference features f r l ~f r 1 Based on the intermediate processing information l-1 ~w 1and the final processing information w are calculated one after another (S104).

[0104] The learning terminal 10 processes the training target image I based on the final processing information w. t The processed training image (I t , w) (S105). The learning terminal 10 acquires the decoder D and the processed training target features T(f t l ,H)~T(f t 1 ,w 1 ) and the training object segmentation map s based on t The learning terminal 10 acquires the decoder D and the training reference image I r The training reference features f r l ~f r 1 and, based on,training reference segmentation map s r is obtained (S107).

[0105] The learning terminal 10 calculates the basic processing information loss L based on the basic processing information H calculated in S103 and the correct basic processing information H (bar) included in the training data. H (Equation 1) is calculated (S108). The learning terminal 10 calculates the final processed information loss L based on the final processed information w calculated in S104 and the correct final processed information w (bar) included in the training data. w (Equation 2) is calculated (S109). The learning terminal 10 calculates the processed training target image (I t , w) and the correct processed information T(I t ,w(bar)) and, based on that, the post-processing loss L I (Equation 3) is calculated (S110).

[0106] The learning terminal 10 uses the training object segmentation map s acquired in S106 t (hat) and the first correct segmentation map s included in the training data t (bar) and the first segmentation map loss L based ons1 (Equation 4) is calculated (S111). The learning terminal 10 calculates the training reference segmentation map s r and the first correct segmentation map s included in the training data r (bar) and the second segmentation map loss L based on s2 (Equation 5) is calculated (S112).

[0107] The learning terminal 10 performs learning of the learning model M based on each loss calculated in S118 to S112 (S113). The learning terminal 10 determines whether to complete the learning (S114). In S114, the learning terminal 10 may determine whether each loss is less than a threshold, or may determine whether a predetermined number of training data have been learned by the learning model M. If it is not determined that the learning is to be completed (S114: N), the process returns to S100. If it is determined that the learning is to be completed (S114: Y), the learning terminal 10 transmits the learned learning model M to the server 20 (S115), and this process ends. The server 20 records the learned learning model M in the memory unit 22.

[0108] [4-2. Estimation processing] 9 is a diagram showing an example of the estimation process. The estimation process is performed by the control units 21 and 31 executing programs stored in the storage units 22 and 32, respectively. It is assumed that the learning process has been performed before the estimation process is performed.

[0109] As shown in FIG. 9, the user terminal 30 generates an estimation target image based on the image capturing result of the image capturing unit 36, and transmits the estimation target image to the server 20 (S200). The server 20 receives the estimation target image from the user terminal 30 (S201). The server 20 acquires an estimated reference image stored in the storage unit 22 (S202). The estimated reference image shows the estimated reference document in an appropriate pose. The server 20 inputs the estimation target image and the estimated reference image into the trained learning model M (S203).

[0110] The server 20 calculates estimation target features, which are features of the estimation target image, and estimated reference features, which are features of the estimated reference image, based on the estimation target image, the estimated reference image, and the encoder E (S204). The server 20 calculates basic processing information based on the estimation target features, the estimated reference features, and the first network N1 (S205). The server 20 successively calculates intermediate processing information and final processing information based on the estimation target features and the estimated reference features calculated by each layer of the first network N1, the basic processing information, and the encoder E (S206).

[0111] The server 20 processes the estimation target image based on the final processing information and acquires the processed estimation target image (S207). The server 20 acquires an estimation target segmentation map based on the decoder D and the processed estimation target features calculated midway in S206 (S208). The server 20 executes eKYC based on the processed estimation target image acquired in S207 and the estimation target segmentation map acquired in S208 (S209), and this process ends.

[0112] [5. Summary of embodiments] The image processing system 1 of this embodiment processes a training object image I t and training reference image I r as the input part, and the training object image I is used to match the training object pose with the training reference pose. t The image processing system 1 acquires training data including correct answer information for processing the training target image I based on the training data. t and training reference image I rThe image processing learning model M is trained so that correct answer information is output when input is input. This allows the image processing system 1 to create a learning model M that does not need to perform heavy processing such as extracting a group of feature points, thereby reducing the processing load on a computer (e.g., server 20) that uses the learned learning model M after training. For example, if the image processing system 1 is applied to eKYC, even if the personal identification document is blurred or reflecting light, and sufficient features are present in other parts, the image processing system 1 can use the learning model M to recognize the features of other parts and perform appropriate processing. This allows the image processing system 1 to achieve highly accurate processing.

[0113] Also, the learning model M is trained on the feature f t 1 ~f t l and training reference features f r 1 ~f r l and the training target features f t 1 ~f t l and training reference features f r 1 ~f r l and a first network N1 that calculates processing information based on the learning model M. The image processing system 1 trains the encoder E and the first network N1 of the learning model M. This eliminates the need for the image processing system 1 to perform heavy processing such as extracting a group of feature points in order to obtain the processing information, thereby reducing the processing load on the computer (e.g., server 20) that obtains the processing information. For example, even if the identification document is blurred or reflects light, the image processing system 1 can achieve appropriate processing using the encoder E and the first network N1.

[0114] Also, the encoder E is a trained feature f t 1 ~f t l and training reference features f r 1~f r l The first network N1 includes a plurality of layers that calculate the training target features f t 1 ~f t l and training reference features f r 1 ~f r l , and calculates the final processed information. In this way, the image processing system 1 calculates the final processed information based on the training object features f t 1 ~f t l and training reference features f r 1 ~f r l By obtaining final processing information that takes all of these factors into consideration, a learning model M can be created that obtains highly accurate processing information.

[0115] The correct information includes correct processing information (for example, correct basic processing information H (bar)). The image processing system 1 calculates the processing information loss (for example, basic processing information loss L) based on the processing information (for example, basic processing information H) calculated by the first network N1 during learning and the correct processing information. H The image processing system 1 calculates the loss of the processing information, and trains the learning model M based on the loss of the processing information. This allows the image processing system 1 to create a highly accurate learning model M that can obtain correct processing information, which is the desired processing information.

[0116] Also, the encoder E is a trained feature f t 1 ~f t l and training reference features f r 1 ~f r l The first network N1 includes a plurality of layers that calculate the training target feature f t 1 ~f t l and training reference features fr 1 ~f r l The correct processing information includes the correct basic processing information H (bar). The image processing system 1 calculates the basic processing information loss L based on the basic processing information H calculated during learning and the correct basic processing information H (bar). H Calculate the basic processing information loss L H The learning model M is trained based on the above. As a result, the image processing system 1 can create a highly accurate learning model M that can obtain desired correct basic processing information H(bar).

[0117] Also, the encoder E is a trained feature f t 1 ~f t l and training reference features f r 1 ~f r l The first network N1 includes a plurality of layers that calculate the training target features f t 1 ~f t l and training reference features f r 1 ~f r l Based on the intermediate processing information l-1 ~w 1 The correct processed information includes the correct final processed information w (bar). The image processing system 1 calculates the final processed information loss L based on the final processed information w calculated during learning and the correct final processed information w (bar). w Calculate the final processing information loss L w The learning model M is trained based on the above. As a result, the image processing system 1 can create a highly accurate learning model M that can obtain the desired correct final processing information w(bar).

[0118] The correct answer information includes correct answer processed information. The image processing system 1 processes the training object image T(I t , w) and the correct processed information T(I t ,w(bar)) and, based on that, the post-processing loss L I Calculate the post-processing loss L I Based on this, the learning model M is trained. As a result, the image processing system 1 obtains the desired correct processed information T(I t ,w(bar)), a highly accurate learning model M can be created that realizes processing according to the

[0119] The learning model M consists of a decoder D that outputs a segmentation map s and a training image I t The image processing system 1 generates a processed training object image T(I) processed using the other portion. t ,w) training target segmentation map s t (hat) and the processed training image T(I t ,w) is the first correct segmentation map s t (bar) and the first segmentation map loss L based on s1 Calculate the first segmentation map loss L s1 Based on this, the learning model M is trained. As a result, the image processing system 1 can not only process the image, but also create a highly accurate learning model M that can obtain a desired segmentation map s.

[0120] The decoder D also generates a training object segmentation map s indicating the pose and type of the training object document in the training object image It. t (Hat). The first correct segmentation map s t The (bar) indicates the correct pose and type. This allows the image processing system 1 to create a highly accurate learning model M that can estimate a desired pose and type.

[0121] The image processing system 1 also receives training reference images Ir The training reference segmentation map s r and training reference image I r The second correct segmentation map s r (bar) and the second segmentation map loss L based on s2 Calculate the second segmentation map loss L s2 The learning model M is trained based on the second correct segmentation map s r It is possible to create a highly accurate learning model M that can obtain (bar).

[0122] Also, the decoder D uses the training reference image I r training reference segmentation maps s indicating the pose and type of the training reference documents in r The second correct segmentation map s is output. r The (bar) indicates the correct pose and type. This allows the image processing system 1 to create a highly accurate learning model M that can estimate a desired pose and type.

[0123] Furthermore, the image processing system 1 inputs an estimation target image and an estimated reference image into the trained learning model M, and acquires an edited estimation target image in which the estimation target pose matches the estimated reference pose. This allows the image processing system 1 to acquire an edited estimation target image based on the learning model M, which does not require heavy processing such as extraction of a feature point group, thereby reducing the processing load on a computer (e.g., server 20) that uses the trained learning model M. For example, if the image processing system 1 is applied to eKYC, even if the personal identification document is blurred or reflects light, as long as sufficient features are present in other parts, the image processing system 1 can recognize the features of the other parts using the learning model M and perform appropriate editing. This allows the image processing system 1 to achieve highly accurate editing.

[0124] Furthermore, the image processing system 1 inputs the estimation target image and the estimated reference image to a learning model M including a trained encoder E and a first network N1, and acquires an edited estimation target image. This eliminates the need for the image processing system 1 to perform heavy processing such as extracting a group of feature points in order to acquire the edited information, thereby reducing the processing load on the computer (e.g., server 20) that acquires the edited information. For example, even if the identification document is blurred or reflects light, the image processing system 1 can achieve appropriate editing using the encoder E and the first network N1.

[0125] Furthermore, in the image processing system 1, the estimation unit 201 acquires, based on the decoder D, an estimation target segmentation map that is a segmentation map corresponding to the processed estimation target image that has been processed using a portion other than the decoder D. This allows the image processing system 1 to not only process the image but also acquire a desired segmentation map.

[0126] [6. Modifications] The present disclosure is not limited to the above-described embodiments, and may be modified as appropriate without departing from the spirit of the present disclosure.

[0127] 10 is a diagram showing an example of functions realized in the modified example. The image processing system 1 of the modified example includes a training data generation unit 103 and an image generation unit 104. Each of the training data generation unit 103 and the image generation unit 104 is realized by the control unit 11.

[0128] [6-1. Variation 1] For example, personal information may be included in an identification document used as training data. If personal information is learned by the learning model M, it may be undesirable for the individual represented by the personal information. Furthermore, the image processing system 1 does not intend to have the learning model M learn personal information, but rather to use the training target image I tSince the learning model M learns features for appropriate processing of the personal information, personal information may become noise during learning. For this reason, in the first modification, the training target image I is processed to reduce the features of personal information. t and training reference image I r The case where is acquired will be described.

[0129] The image processing system 1 of the first modification includes a training data generation unit 103. The training data generation unit 103 generates training target images I t and training reference image I r Training data is generated by processing personal information contained in an original image that is the basis for the training data. The original image is assumed to be stored in the data storage unit 100. The original image may represent an identification document of a person belonging to a certain organization, or may represent a document other than an identification document.

[0130] For example, the training data generation unit 103 identifies a portion of a document shown in an original image that contains personal information. In Modification 1, it is assumed that the personal information is contained in a predetermined region of the original image. The training data generation unit 103 acquires a training document and a training reference image Ir by performing image processing on the predetermined region of the original image to reduce the features of the personal information.

[0131] Note that the image processing of personal information may be any image processing that makes the personal information difficult to identify. For example, the image processing may be blurring, mosaic processing, masking (filling), cropping, texture application, or other processing. Furthermore, since personal information is basically text, the training data generation unit 103 may perform optical character recognition on the original image to identify the text portion, and then perform image processing on the text portion of the original image, regarding it as personal information.

[0132] For example, the training data generation unit 103 may directly use the image in which the personal information has been processed as the training target image I t The training data generation unit 103 may acquire the training target image I tBy applying image processing such as affine transformation to the training target document, the orientation of the training target document is changed and the training reference image I r The training data generation unit 103 may acquire these training object images I t and training reference image I r is acquired as the input part of the training data. The correct answer part of the training data may be specified by the creator of the learning model M.

[0133] For example, the training data generation unit 103 may directly use the image in which personal information has been processed as a training reference image I r The training data generation unit 103 may acquire the training reference image I r By applying image processing such as affine transformation to the training reference document, the training target image I t The training data generation unit 103 may acquire these training object images I t and training reference image I r is acquired as the input part of the training data. The correct answer part of the training data may be specified by the creator of the learning model M.

[0134] The image processing system 1 of the first modification is a training object image I t and training reference image I r The training data is generated by processing personal information contained in the original image that is the basis of the image. In this way, the image processing system 1 can prevent personal information from being used in a manner that is inappropriate for the individual represented by the personal information. The image processing system 1 can also have the learning model M learn features for appropriate processing instead of personal information.

[0135] [6-2. Variation 2] For example, the learning model M may perform estimation for processing by referring not only to the poses of the training target document and the training reference document, but also to their backgrounds. Various backgrounds may also be shown in the estimation target image taken by the user. For this reason, in Modification 2, a case will be described in which a training target document and a training reference document are generated that allow the learning model M to learn various backgrounds.

[0136] The image processing system 1 of the second modification includes an image generation unit 104. The image generation unit 104 generates a training object image I based on an original document image showing an original document that is the basis of the training object document and the training reference document, and a background image showing a background prepared in advance. t and training reference image I r The original document image and the background image are assumed to be stored in the data storage unit 100. The data storage unit 100 stores original document images showing documents in each of a plurality of poses, and background images showing each of a plurality of backgrounds.

[0137] For example, the original document image is an image showing an original document that is the same type of document as at least one of the training target document and the training reference document. The original document image may be prepared by the person who creates the learning model M, or may be prepared by another person. The original document image may show the original document in any pose. For example, the original document image may show the original document in the same pose as the pose of the training target document, the original document in the same pose as the pose of the training reference document, or the original document in a different pose. The image generation unit 104 performs image processing so as to change the pose of the original document shown in the original document image, and generates the training target image I. t and training reference image I r The image generating unit 104 may perform image processing to generate a predetermined posture, or may perform image processing to generate a random posture.

[0138] For example, the background image shows a background with different colors, patterns, brightness, patterns, objects, or a combination thereof. The background image is also called a texture image. The image generation unit 104 selects one of the plurality of background images and generates a training object image I on the background shown by the selected background image. t The original document image and the background image are synthesized so that the original document shown on the original document image for training is superimposed on the original document image for training. t The image generation unit 104 selects one of the plurality of background images and generates a training reference image I on the background represented by the selected background image.r The training reference image I is obtained by superimposing the original document shown on the original document image for training. r The training object image I generated by the image generation unit 104 is generated. t and training reference image I r Based on this, training data is generated.

[0139] The image processing system 1 of the second modification generates a training object image I based on an original document image showing an original document that is the basis of the training object document and the training reference document, and a background image showing a background prepared in advance. t and training reference image I r This allows the image processing system 1 to train the learning model M on the features of various backgrounds, thereby improving the accuracy of the learning model M. The image processing system 1 can also reduce the effort required to prepare training data, thereby improving convenience for the creator of the learning model M.

[0140] [6-3. Other variations] For example, the above modifications may be combined.

[0141] For example, a function described as being realized by the learning terminal 10 may be realized by another computer such as the server 20. A function described as being realized by the learning terminal 10 may be shared between the learning terminal 10 and another computer. A function described as being realized by the server 20 may be realized by another computer such as the user terminal 30. A function described as being realized by the server 20 may be shared between the server 20 and another computer.

[0142] [7. Notes] For example, the image processing system can also be configured as follows. (1) a training data acquisition unit that acquires training data including, as input portions, a training target image showing a training target document and a training reference image showing a training reference document, and includes, as a correct answer portion, correct answer information for processing the training target image so that the training target orientation of the training target document in the training target image matches the training reference orientation of the training reference document in the training reference image; a learning unit that learns a learning model for image processing based on the training data so that when the training target image and the training reference image are input, the correct answer information is output; An image processing system comprising: (2) The learning model is an encoder for computing training object features of the training object images and training reference features of the training reference images; a first network that calculates processing information for processing the training object image based on the training object features and the training reference features; Including, The learning unit trains the encoder and the first network of the learning model. The image processing system according to (1). (3) the encoder includes a plurality of layers that compute the training subject features and the training reference features; the first network calculates, for each layer, the processed information based on the training target features and the training reference features calculated by the layer, to calculate the final processed information; (2) An image processing system according to (1). (4) The correct answer information includes correct processed information that is the processed information that is the correct answer, The learning unit calculates a processing information loss based on the processing information calculated by the first network during learning and the correct processing information, and trains the learning model based on the processing information loss. An image processing system according to (2) or (3). (5) the encoder includes a plurality of layers that compute the training subject features and the training reference features; The first network calculates basic processing information, which is the processing information calculated based on the training object features and the training reference features calculated by a final layer of the plurality of layers; The correct processing information includes correct basic processing information that is the basic processing information that is correct, The learning unit calculates a basic processing information loss based on the basic processing information calculated during learning and the correct basic processing information, and trains the learning model based on the basic processing information loss. (4) An image processing system according to (4). (6) the encoder includes a plurality of layers that compute the training subject features and the training reference features; the first network calculates, for each layer, from the layer that is later in order among the plurality of layers, intermediate processed information that is intermediate the processed information based on the training object features and the training reference features calculated by the layer, and calculates final processed information that is final the processed information; The correct processing information includes correct final processing information which is the final processing information that is the correct answer, The learning unit calculates a final processing information loss based on the final processing information calculated during learning and the correct final processing information, and trains the learning model based on the final processing information loss. An image processing system according to (4) or (5). (7) The correct answer information includes correct answer processed information regarding the training object image after processing to become the correct answer, The learning unit calculates a post-processing loss based on the training target image processed based on the processing information calculated by the first network during learning and the correct answer post-processing information, and trains the learning model based on the post-processing loss. The image processing system according to any one of (2) to (6). (8) The learning model includes a decoder that outputs a segmentation map and another part for processing the training target image; the learning unit calculates a first segmentation map loss based on a training object segmentation map that is the segmentation map of an processed training object image that is the training object image processed using the other portion and a first correct answer segmentation map that is a correct answer for the processed training object image, and trains the learning model based on the first segmentation map loss; The image processing system according to any one of (1) to (7). (9) the decoder outputs the training subject segmentation map indicating the pose and type of the training subject documents in the training subject images; The first correct segmentation map indicates the correct pose and the correct type. (8) An image processing system according to (8). (10) The learning model includes a decoder that outputs a segmentation map and another part for processing the training target image; the learning unit calculates a second segmentation map loss based on a training reference segmentation map that is the segmentation map of the training reference image and a second correct answer segmentation map that is a correct answer for the training reference image, and trains the learning model based on the second segmentation map loss; The image processing system according to any one of (1) to (9). (11) the decoder outputs the training reference segmentation map indicating the pose and type of the training reference documents in the training reference images; The second correct segmentation map indicates the correct pose and the correct type. The image processing system according to (10). (12) The image processing system further includes a training data generation unit that generates the training data by processing personal information included in an original image that is a source of the training target image and the training reference image. The image processing system according to any one of (1) to (11). (13) The image processing system further includes an image generation unit that generates the training target image and the training reference image based on an original document image showing an original document that is a source of the training target document and the training reference document, and a background image showing a background that is prepared in advance. The image processing system according to any one of (1) to (12). (14) The image processing system further includes an estimation unit that inputs an estimation target image indicating an estimation target document and an estimated reference image indicating an estimated reference document to the trained learning model, and acquires a processed estimation target image, which is the estimation target image processed so that the estimation target orientation of the estimation target document matches the estimated reference orientation of the estimated reference document. The image processing system according to any one of (1) to (13). (15) The learning model is an encoder for computing training object features of the training object images and training reference features of the training reference images; a first network that calculates processing information for processing the training object image based on the training object features and the training reference features; Including, the learning unit trains the encoder and the first network of the learning model; the estimation unit inputs the estimation target image and the estimated reference image to the learning model including the trained encoder and the first network, and acquires the processed estimation target image; (14) An image processing system according to (14). (16) The learning model includes a decoder that outputs a segmentation map and another part for processing the training target image; the learning unit calculates a first segmentation map loss based on a training object segmentation map that is the segmentation map of the training object image processed using the other portion and a first correct answer segmentation map that is a correct answer for the processed training object image, and trains the learning model based on the first segmentation map loss; the estimation unit acquires, based on the decoder, an estimation target segmentation map, which is the segmentation map according to the processed estimation target image processed using the other part; The image processing system according to (14) or (15). [Explanation of symbols]

[0143] 1 Image processing system, CN Communication network, 10 Learning terminal, 11, 21, 31 Control unit, 12, 22, 32 Memory unit, 13, 23, 33 Communication unit, 20 Server, 30 User terminal, 14, 34 Operation unit, 15, 35 Display unit, 36 Photography unit, D Decoder, DB Training database, 100 Data storage unit, 101 Training data acquisition unit, 102 Learning unit, 103 Training data generation unit, 104 Image generation unit, 200 Data storage unit, 201 Estimation unit, 300 Data storage unit, 301 Transmission unit.

Claims

1. a training data acquisition unit that acquires training data including, as input portions, a training target image showing a training target document and a training reference image showing a training reference document, and includes, as a correct answer portion, correct answer information for processing the training target image so that the training target orientation of the training target document in the training target image matches the training reference orientation of the training reference document in the training reference image; a learning unit that learns a learning model for image processing based on the training data so that when the training target image and the training reference image are input, the correct answer information is output; An image processing system comprising:

2. The learning model is an encoder for computing training object features of the training object images and training reference features of the training reference images; a first network that calculates processing information for processing the training object image based on the training object features and the training reference features; Including, The learning unit trains the encoder and the first network of the learning model. The image processing system according to claim 1 .

3. the encoder includes a plurality of layers that compute the training subject features and the training reference features; the first network calculates, for each layer, the processed information based on the training target features and the training reference features calculated by the layer, to calculate the final processed information; The image processing system according to claim 2 .

4. The correct answer information includes correct processed information that is the processed information that is the correct answer, The learning unit calculates a processing information loss based on the processing information calculated by the first network during learning and the correct processing information, and trains the learning model based on the processing information loss.

4. The image processing system according to claim 2 or 3.

5. the encoder includes a plurality of layers that compute the training subject features and the training reference features; The first network calculates basic processing information, which is the processing information calculated based on the training object features and the training reference features calculated by a final layer of the plurality of layers; The correct processing information includes correct basic processing information that is the basic processing information that is correct, The learning unit calculates a basic processing information loss based on the basic processing information calculated during learning and the correct basic processing information, and trains the learning model based on the basic processing information loss. The image processing system according to claim 4 .

6. the encoder includes a plurality of layers that compute the training subject features and the training reference features; the first network calculates, for each layer, intermediate processed information that is intermediate processed information based on the training object features and the training reference features calculated by the layer, starting from the layer in the order of the plurality of layers, and calculates final processed information that is final processed information; The correct processing information includes correct final processing information which is the final processing information that is the correct answer, The learning unit calculates a final processing information loss based on the final processing information calculated during learning and the correct final processing information, and trains the learning model based on the final processing information loss. The image processing system according to claim 4 .

7. The correct answer information includes correct answer processed information regarding the training object image after processing to become the correct answer, The learning unit calculates a post-processing loss based on the training target image processed based on the processing information calculated by the first network during learning and the correct answer post-processing information, and trains the learning model based on the post-processing loss.

4. The image processing system according to claim 2 or 3.

8. The learning model includes a decoder that outputs a segmentation map and another part for processing the training target image; the learning unit calculates a first segmentation map loss based on a training object segmentation map that is the segmentation map of an processed training object image that is the training object image processed using the other portion and a first correct answer segmentation map that is a correct answer for the processed training object image, and trains the learning model based on the first segmentation map loss; 4. The image processing system according to claim 1.

9. the decoder outputs the training subject segmentation map indicating the pose and type of the training subject documents in the training subject images; the first correct segmentation map indicates the correct pose and the correct type; The image processing system according to claim 8 .

10. The learning model includes a decoder that outputs a segmentation map and another part for processing the training target image; the learning unit calculates a second segmentation map loss based on a training reference segmentation map that is the segmentation map of the training reference image and a second correct answer segmentation map that is a correct answer for the training reference image, and trains the learning model based on the second segmentation map loss; 4. The image processing system according to claim 1.

11. the decoder outputs the training reference segmentation map indicating the pose and type of the training reference documents in the training reference images; the second correct segmentation map indicates the correct pose and the correct type; The image processing system according to claim 10.

12. The image processing system further includes a training data generation unit that generates the training data by processing personal information included in an original image that is a source of the training target image and the training reference image.

4. The image processing system according to claim 1.

13. The image processing system further includes an image generation unit that generates the training target image and the training reference image based on an original document image showing an original document that is a source of the training target document and the training reference document, and a background image showing a background that is prepared in advance.

4. The image processing system according to claim 1.

14. The image processing system further includes an estimation unit that inputs an estimation target image indicating an estimation target document and an estimated reference image indicating an estimated reference document to the trained learning model, and acquires a processed estimation target image, which is the estimation target image processed so that the estimation target orientation of the estimation target document matches the estimated reference orientation of the estimated reference document. The image processing system according to claim 1 .

15. The learning model is an encoder for computing training object features of the training object images and training reference features of the training reference images; a first network that calculates processing information for processing the training object image based on the training object features and the training reference features; Including, the learning unit trains the encoder and the first network of the learning model; the estimation unit inputs the estimation target image and the estimated reference image to the learning model including the trained encoder and the first network, and acquires the processed estimation target image; The image processing system according to claim 14.

16. The learning model includes a decoder that outputs a segmentation map and another part for processing the training target image; the learning unit calculates a first segmentation map loss based on a training object segmentation map that is the segmentation map of the training object image processed using the other portion and a first correct answer segmentation map that is a correct answer for the processed training object image, and trains the learning model based on the first segmentation map loss; the estimation unit acquires, based on the decoder, an estimation target segmentation map, which is the segmentation map according to the processed estimation target image processed using the other part; The image processing system according to claim 14.

17. a training data acquisition step of acquiring training data including, as input portions, a training target image showing a training target document and a training reference image showing a training reference document, and including, as a correct answer portion, correct answer information for processing the training target image so that the training target orientation of the training target document in the training target image matches the training reference orientation of the training reference document in the training reference image; a learning step of learning an image processing learning model based on the training data so that when the training target image and the training reference image are input, the correct answer information is output; An image processing method comprising:

18. a training data acquisition unit that acquires training data including, as input portions, a training target image showing a training target document and a training reference image showing a training reference document, and that includes, as a correct answer portion, correct answer information for processing the training target image so that the training target orientation of the training target document in the training target image matches the training reference orientation of the training reference document in the training reference image; a learning unit that learns a learning model for image processing based on the training data so that when the training target image and the training reference image are input, the correct answer information is output; A program that allows a computer to function as a

Images